[
https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16379408#comment-16379408
]
ASF GitHub Bot commented on DRILL-5270:
---------------------------------------
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/755#discussion_r171088565
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/store/LocalPersistentStore.java
---
@@ -112,23 +127,65 @@ public static DrillFileSystem
getFileSystem(DrillConfig config, Path root) throw
@Override
public Iterator<Map.Entry<String, V>> getRange(int skip, int take) {
+ //Marking currently seen modification time
+ long currBasePathModified = 0L;
+ try {
+ currBasePathModified =
fs.getFileStatus(basePath).getModificationTime();
+ } catch (IOException ioexcp) {
+ ioexcp.printStackTrace();
+ }
+
+ //Acquiring lock to avoid reloading for request coming in before
completion of profile read
--- End diff --
1. Before reading lock acquirement was enough, with your changes you modify
class fields. Since many threads can access this method, you'll end up with
raise conditions, also class fields can be cached by threads as well... I think
design here should be reconsidered.
2. Guava library has several cache implementations. Can we leverage any of
them instead of using tree set?
Pinging @vlad since he is working on DRILL-6053 which intends to make
changes in the same class to avoid excessive locking to be aware of intended
changes.
> Improve loading of profiles listing in the WebUI
> ------------------------------------------------
>
> Key: DRILL-5270
> URL: https://issues.apache.org/jira/browse/DRILL-5270
> Project: Apache Drill
> Issue Type: Improvement
> Components: Web Server
> Affects Versions: 1.9.0
> Reporter: Kunal Khatua
> Assignee: Kunal Khatua
> Priority: Major
> Fix For: 1.13.0
>
>
> Currently, as the number of profiles increase, we reload the same list of
> profiles from the FS.
> An ideal improvement would be to detect if there are any new profiles and
> only reload from the disk then. Otherwise, a cached list is sufficient.
> For a directory of 280K profiles, the load time is close to 6 seconds on a 32
> core server. With the caching, we can get it down to as much as a few
> milliseconds.
> To render the cache as invalid, we inspect the last modified time of the
> directory to confirm whether a reload is needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
