[
https://issues.apache.org/jira/browse/DRILL-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongze Zhang updated DRILL-5028:
--------------------------------
Description:
We have a Drill cluster with 20+ Nodes and we store all history profiles in
hdfs. Without doing periodically cleans for hdfs, the profiles page gets slower
while serving more queries.
Code from LocalPersistentStore.java uses fs.list(false, basePath) for fetching
the latest 100 history profiles by default, I guess this operation blocks the
page loading (Millions small files can be stored in the basePath), maybe we can
try some other ways to reach the same goal.
was:
We have a Drill cluster with 20+ Nodes and we store all history profiles in
hdfs. Without doing periodically cleans for hdfs, the profiles page gets slower
while serving more queries.
Code from LocalPersistentStore.java uses fs.list(false, basePath) for fetching
the latest 100 history profiles by default, I guess this operation blocks that
page (Millions small files can be stored in the basePath), maybe we can try
some other ways to reach the same goal.
> Opening profiles page from web ui gets very slow when a lot of history files
> have been stored in HDFS or Local FS.
> ------------------------------------------------------------------------------------------------------------------
>
> Key: DRILL-5028
> URL: https://issues.apache.org/jira/browse/DRILL-5028
> Project: Apache Drill
> Issue Type: Improvement
> Components: Functions - Drill
> Affects Versions: 1.8.0
> Reporter: Hongze Zhang
> Fix For: Future
>
>
> We have a Drill cluster with 20+ Nodes and we store all history profiles in
> hdfs. Without doing periodically cleans for hdfs, the profiles page gets
> slower while serving more queries.
> Code from LocalPersistentStore.java uses fs.list(false, basePath) for
> fetching the latest 100 history profiles by default, I guess this operation
> blocks the page loading (Millions small files can be stored in the basePath),
> maybe we can try some other ways to reach the same goal.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
