[
https://issues.apache.org/jira/browse/AIRAVATA-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17368475#comment-17368475
]
Marcus Christie commented on AIRAVATA-3322:
-------------------------------------------
I realized that the master branch of the Django portal is using the develop
branch of airavata python client SDK, so commit the updated stubs to the
develop branch of Airavata will break the master branch of the Django portal.
Ultimately, pulling the stubs from a single repo branch isn't going to work
well because we won't be able to use a different set of stubs in the develop
and master branches of the portal code.
I need to think about it some more, but I think the best thing would be to push
the Airavata Python client SDK to PyPI and just pull it from there. That way
the version to pull is in the portals source control (in the requirements.txt
file) and can be updated along with the code.
> Add paging to experiment statistics
> -----------------------------------
>
> Key: AIRAVATA-3322
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3322
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal, Registry API
> Reporter: Marcus Christie
> Assignee: Marcus Christie
> Priority: Major
>
> The getExperimentStatistics API method works well for small numbers of
> experiments but doesn't scale well to thousands of experiments, which some
> popular gateways like [seagrid.org|https://seagrid.org] seeing this level of
> activity. For example, for SEAGrid we had to up the Apache/Django timeout to
> 5 minutes to load statistics for a few months for one compute resource, and
> even then it didn't always load.
> I added slow query logging in the MariaDB but that turned up nothing so I
> don't believe the queries need to be tuned. I think the main culprit is that
> thousands of ExperimentSummary instances are getting populated and returned
> by the method. It would be better to page in experiments and only return the
> first, say, 50 experiments in each status category.
> Currently the ExperimentSummaries are loaded for each ExperimentState and
> they are counted. Instead, we'd need two queries per state, one to load the
> count as a scalar and another to load a page's worth of ExperimentSummaries.
> To preserve backwards compatibility, it might be best to just make a new
> method and then in time phase out the old one.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
