[
https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464420#comment-16464420
]
ASF GitHub Bot commented on DRILL-5270:
---------------------------------------
kkhatua commented on issue #1250: DRILL-5270: Improve loading of profiles
listing in the WebUI
URL: https://github.com/apache/drill/pull/1250#issuecomment-386733307
[DRILL-5270] User latency when 8 web-clients (wget) request for `/profiles`
against a profile store of 123K profiles (max scale range= 2min). The requests
are done in 2 waves
Note: Both caching **and** archiving is enabled and no new profiles have
been written to the store during the 2 min window.
Notice how all the subsequent responses go fast the moment the third
response is complete. The first 3 clients triggered archiving of profiles from
123K down to about 92K, each time trying to build the cache. By the time the
fourth request comes, there is no more archiving, so the requests are served
from cache (and, hence, they are barely 2-3 seconds apart). The second wave of
requests from the 8 clients is now completely served by the cache.
Backend logging reveals the archiving process:
```
2018-05-01 22:47:37,870 kk127.qa.lab [qtp132047013-85] INFO
o.a.d.e.s.s.s.LocalPersistentStore - Requesting thread: qtp132047013-85-85
2018-05-01 22:47:45,131 kk127.qa.lab [qtp132047013-85] INFO
o.a.d.e.s.s.s.LocalPersistentStore - Found 32935 excess profiles. For now, will
attempt archiving 10000 profiles to maprfs:/drillbit/profiles/archived
2018-05-01 22:48:04,771 kk127.qa.lab [qtp132047013-85] INFO
o.a.d.e.s.s.s.LocalPersistentStore - Archived 10000 profiles to
maprfs:/drillbit/profiles/archived in 19635 ms
2018-05-01 22:48:04,774 kk127.qa.lab [qtp132047013-85] WARN
o.a.d.e.s.s.s.LocalPersistentStore - Took 26902 ms to list & map 300 profiles
(out of 122935 profiles in store)
2018-05-01 22:48:12,310 kk127.qa.lab [qtp132047013-85] INFO
o.a.d.e.s.s.s.LocalPersistentStore - Requesting thread: qtp132047013-85-85
2018-05-01 22:48:18,439 kk127.qa.lab [qtp132047013-85] INFO
o.a.d.e.s.s.s.LocalPersistentStore - Found 22935 excess profiles. For now, will
attempt archiving 10000 profiles to maprfs:/drillbit/profiles/archived
2018-05-01 22:48:38,234 kk127.qa.lab [qtp132047013-85] INFO
o.a.d.e.s.s.s.LocalPersistentStore - Archived 10000 profiles to
maprfs:/drillbit/profiles/archived in 19791 ms
2018-05-01 22:48:38,236 kk127.qa.lab [qtp132047013-85] WARN
o.a.d.e.s.s.s.LocalPersistentStore - Took 25924 ms to list & map 300 profiles
(out of 112935 profiles in store)
2018-05-01 22:48:43,275 kk127.qa.lab [qtp132047013-85] INFO
o.a.d.e.s.s.s.LocalPersistentStore - Requesting thread: qtp132047013-85-85
2018-05-01 22:48:48,911 kk127.qa.lab [qtp132047013-85] INFO
o.a.d.e.s.s.s.LocalPersistentStore - Found 12935 excess profiles. For now, will
attempt archiving 10000 profiles to maprfs:/drillbit/profiles/archived
2018-05-01 22:49:09,757 kk127.qa.lab [qtp132047013-85] INFO
o.a.d.e.s.s.s.LocalPersistentStore - Archived 10000 profiles to
maprfs:/drillbit/profiles/archived in 20842 ms
2018-05-01 22:49:09,759 kk127.qa.lab [qtp132047013-85] WARN
o.a.d.e.s.s.s.LocalPersistentStore - Took 26482 ms to list & map 300 profiles
(out of 102935 profiles in store)
2018-05-01 22:49:14,119 kk127.qa.lab [qtp132047013-85] INFO
o.a.d.e.s.s.s.LocalPersistentStore - Requesting thread: qtp132047013-85-85
2018-05-01 22:49:19,339 kk127.qa.lab [qtp132047013-85] WARN
o.a.d.e.s.s.s.LocalPersistentStore - Took 5217 ms to list & map 300 profiles
(out of 92935 profiles in store)
2018-05-01 22:49:23,656 kk127.qa.lab [qtp132047013-85] INFO
o.a.d.e.s.s.s.LocalPersistentStore - Requesting thread: qtp132047013-85-85
2018-05-01 22:49:24,214 kk127.qa.lab [qtp132047013-85] INFO
o.a.d.e.s.s.s.LocalPersistentStore - Requesting thread: qtp132047013-85-85
2018-05-01 22:49:24,798 kk127.qa.lab [qtp132047013-85] INFO
o.a.d.e.s.s.s.LocalPersistentStore - Requesting thread: qtp132047013-85-85
2018-05-01 22:49:25,365 kk127.qa.lab [qtp132047013-85] INFO
o.a.d.e.s.s.s.LocalPersistentStore - Requesting thread: qtp132047013-85-85
2018-05-01 22:55:12,247 kk127.qa.lab [qtp132047013-92] INFO
o.a.d.e.s.s.s.LocalPersistentStore - Requesting thread: qtp132047013-92-92
2018-05-01 22:55:12,791 kk127.qa.lab [qtp132047013-92] INFO
o.a.d.e.s.s.s.LocalPersistentStore - Requesting thread: qtp132047013-92-92
2018-05-01 22:55:13,276 kk127.qa.lab [qtp132047013-92] INFO
o.a.d.e.s.s.s.LocalPersistentStore - Requesting thread: qtp132047013-92-92
2018-05-01 22:55:13,770 kk127.qa.lab [qtp132047013-92] INFO
o.a.d.e.s.s.s.LocalPersistentStore - Requesting thread: qtp132047013-92-92
2018-05-01 22:55:30,477 kk127.qa.lab [qtp132047013-92] INFO
o.a.d.e.s.s.s.LocalPersistentStore - Requesting thread: qtp132047013-92-92
2018-05-01 22:55:31,018 kk127.qa.lab [qtp132047013-92] INFO
o.a.d.e.s.s.s.LocalPersistentStore - Requesting thread: qtp132047013-92-92
2018-05-01 22:55:31,578 kk127.qa.lab [qtp132047013-92] INFO
o.a.d.e.s.s.s.LocalPersistentStore - Requesting thread: qtp132047013-92-92
2018-05-01 22:55:32,140 kk127.qa.lab [qtp132047013-92] INFO
o.a.d.e.s.s.s.LocalPersistentStore - Requesting thread: qtp132047013-92-92
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Improve loading of profiles listing in the WebUI
> ------------------------------------------------
>
> Key: DRILL-5270
> URL: https://issues.apache.org/jira/browse/DRILL-5270
> Project: Apache Drill
> Issue Type: Improvement
> Components: Web Server
> Affects Versions: 1.9.0
> Reporter: Kunal Khatua
> Assignee: Kunal Khatua
> Priority: Major
> Fix For: 1.14.0
>
>
> Currently, as the number of profiles increase, we reload the same list of
> profiles from the FS.
> An ideal improvement would be to detect if there are any new profiles and
> only reload from the disk then. Otherwise, a cached list is sufficient.
> For a directory of 280K profiles, the load time is close to 6 seconds on a 32
> core server. With the caching, we can get it down to as much as a few
> milliseconds.
> To render the cache as invalid, we inspect the last modified time of the
> directory to confirm whether a reload is needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
