rangareddy commented on issue #6188: URL: https://github.com/apache/hudi/issues/6188#issuecomment-5103005550
This issue was reviewed as part of the JIRA-migrated backlog triage. Findings: One correction to the record before closing. The 2022-08-29 reply attributed this to a performance gap in the metadata table's bloom index and pointed at #6432 ([HUDI-4586] Improve metadata fetching in bloom index) as the fix in progress. **#6432 was never merged**, so nothing on this thread should be read as "fixed by that PR" -- the change described there did not land. On the report itself, two factors are visible in the configuration as filed and neither is a Hudi defect. Partitioning on `country_iso` with 200 values where US and China dominate gives severe partition skew, so a handful of tasks do most of the work regardless of parallelism settings. And the workload is 100% updates on a 1-master/2-core-node cluster, which is the most index-and-rewrite-intensive shape there is for a COW table. There has been no activity since 2022-08-29 and the recommendations offered then were never tried, so there is no measurement to act on. Closing as stale. If you revisit this on a current version (1.0.x / 1.1.x / 1.2.0), the two changes that matter most for this shape are: use `RECORD_LEVEL_INDEX` or a bucket index instead of bloom, since an all-updates workload on random keys gets no benefit from bloom range pruning; and pick a partition field that distributes more evenly, using clustering to get `country_iso` pruning instead of partitioning on it. A fresh issue with numbers from that configuration would be actionable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
