[
https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464388#comment-16464388
]
ASF GitHub Bot commented on DRILL-5270:
---------------------------------------
kkhatua commented on issue #1250: DRILL-5270: Improve loading of profiles
listing in the WebUI
URL: https://github.com/apache/drill/pull/1250#issuecomment-386733307
[DRILL-5270] User latency when 8 web-clients (wget) request for `/profiles`
against a profile store of 123K profiles (max scale range= 2min). The requests
are done in 2 waves
Note: Both caching **and** archiving is enabled and no new profiles have
been written to the store during the 2 min window.
Notice how all the subsequent responses go fast the moment the third
response is complete. The first 3 clients triggered archiving of profiles from
123K down to about 92K, each time trying to build the cache. By the time the
fourth request comes, there is no more archiving, so the requests are served
from cache (and, hence, they are barely 2-3 seconds apart). The second wave of
requests from the 8 clients is now completely served by the cache.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Improve loading of profiles listing in the WebUI
> ------------------------------------------------
>
> Key: DRILL-5270
> URL: https://issues.apache.org/jira/browse/DRILL-5270
> Project: Apache Drill
> Issue Type: Improvement
> Components: Web Server
> Affects Versions: 1.9.0
> Reporter: Kunal Khatua
> Assignee: Kunal Khatua
> Priority: Major
> Fix For: 1.14.0
>
>
> Currently, as the number of profiles increase, we reload the same list of
> profiles from the FS.
> An ideal improvement would be to detect if there are any new profiles and
> only reload from the disk then. Otherwise, a cached list is sufficient.
> For a directory of 280K profiles, the load time is close to 6 seconds on a 32
> core server. With the caching, we can get it down to as much as a few
> milliseconds.
> To render the cache as invalid, we inspect the last modified time of the
> directory to confirm whether a reload is needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
