[
https://issues.apache.org/jira/browse/AIRAVATA-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17368307#comment-17368307
]
ASF subversion and git services commented on AIRAVATA-3322:
-----------------------------------------------------------
Commit 38a5136c296a2c887f7bd131be0b6bbfba1cf071 in airavata's branch
refs/heads/develop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=38a5136 ]
AIRAVATA-3322 Separate queries for counts and data
> Add paging to experiment statistics
> -----------------------------------
>
> Key: AIRAVATA-3322
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3322
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal, Registry API
> Reporter: Marcus Christie
> Assignee: Marcus Christie
> Priority: Major
>
> The getExperimentStatistics API method works well for small numbers of
> experiments but doesn't scale well to thousands of experiments, which some
> popular gateways like [seagrid.org|https://seagrid.org] seeing this level of
> activity. For example, for SEAGrid we had to up the Apache/Django timeout to
> 5 minutes to load statistics for a few months for one compute resource, and
> even then it didn't always load.
> I added slow query logging in the MariaDB but that turned up nothing so I
> don't believe the queries need to be tuned. I think the main culprit is that
> thousands of ExperimentSummary instances are getting populated and returned
> by the method. It would be better to page in experiments and only return the
> first, say, 50 experiments in each status category.
> Currently the ExperimentSummaries are loaded for each ExperimentState and
> they are counted. Instead, we'd need two queries per state, one to load the
> count as a scalar and another to load a page's worth of ExperimentSummaries.
> To preserve backwards compatibility, it might be best to just make a new
> method and then in time phase out the old one.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
