rangareddy commented on issue #4873: URL: https://github.com/apache/hudi/issues/4873#issuecomment-5102171514
This issue was reviewed as part of the JIRA-migrated backlog triage. Findings: Three separate causes were established in this thread, and none of them is a Hudi defect -- together they account for the symptom. 1. **Compaction never ran.** You confirmed on 2022-02-24 that no compaction process was running, and on Glue async compaction does not start on its own. An uncompacted MOR table keeps accumulating log files, and every later update pays to read them -- which is exactly why processing time grew with the amount of data already written. This is the main one. 2. **Hour-level partitioning** gives more than 25,000 partitions for a few years of data; the practical guidance is to stay at or below ~10,000 and use clustering to get hour-level pruning benefits instead. 3. **The record key `(intervalutc, asset, attribute)` has no ordering property**, so bloom min/max range pruning cannot eliminate file groups and the index lookup degenerates to scanning candidates. That is what the shuffle profile shows: 1,237 GB written against 42.6 GB read, with 7.7 TiB of memory spill. Switching to `SIMPLE` helped, as you reported on 2022-02-26, because it stops paying for ineffective bloom pruning. One process note: the follow-ups on 2022-09-12 and 2022-11-03 tagged the wrong GitHub user, so you may never have been notified of them. Closing as stale -- no Hudi-side defect was isolated and there has been no activity since 2022-11-03. If you still run this pipeline, the highest-value changes on a recent version are: make sure compaction actually runs (inline or a dedicated job), and try **bucket index** or `RECORD_LEVEL_INDEX` -- with bucket index there is no index-lookup fan-out at all, which is the cost dominating your run. If it is still slow after that, please open a fresh issue with the Hudi version, compaction settings and Spark UI timings. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
