Hongze Zhang created DRILL-5028:
-----------------------------------
Summary: Opening profiles page from web ui gets very slow when a
lot of history files have been stored in HDFS or Local FS.
Key: DRILL-5028
URL: https://issues.apache.org/jira/browse/DRILL-5028
Project: Apache Drill
Issue Type: Improvement
Components: Functions - Drill
Affects Versions: 1.8.0
Reporter: Hongze Zhang
Fix For: Future
We have a Drill cluster with 20+ Nodes and we store all history profiles in
hdfs. Without doing periodically cleans for hdfs, the profiles page gets slower
while serving more queries.
Code from LocalPersistentStore.java uses fs.list(false, basePath) for fetching
the latest 100 history profiles by default, I guess this operation blocks that
page (Millions small files can be stored in the basePath), maybe we can try
some other ways to reach the same goal.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
