Hongze Zhang created DRILL-5028: ----------------------------------- Summary: Opening profiles page from web ui gets very slow when a lot of history files have been stored in HDFS or Local FS. Key: DRILL-5028 URL: https://issues.apache.org/jira/browse/DRILL-5028 Project: Apache Drill Issue Type: Improvement Components: Functions - Drill Affects Versions: 1.8.0 Reporter: Hongze Zhang Fix For: Future
We have a Drill cluster with 20+ Nodes and we store all history profiles in hdfs. Without doing periodically cleans for hdfs, the profiles page gets slower while serving more queries. Code from LocalPersistentStore.java uses fs.list(false, basePath) for fetching the latest 100 history profiles by default, I guess this operation blocks that page (Millions small files can be stored in the basePath), maybe we can try some other ways to reach the same goal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)