rangareddy commented on issue #9751: URL: https://github.com/apache/hudi/issues/9751#issuecomment-4834861894
Hi @njalan Could you please try the following things: 1. **Re-bootstrap the MDT cleanly**. Don't carry a jar-swapped 0.7 MDT. Delete `.hoodie/metadata/` and let it rebuild on the next write with metadata enabled, or rebuild the table on a current version. Then confirm in driver logs you see "Listed partitions from metadata" / "Listed files in partition from metadata" and not FS-based listing for the hot paths. 2. **Enable MDT on the read/sync side too**, not just writes — Spark datasource reads, Hive/Glue sync, and cleaner all need `hoodie.metadata.enable=true`. 3. **Disable validation in prod**: `hoodie.metadata.validate=false`. 4. **Keep MDT compacted for streaming:** lower hoodie.metadata.compact.max.delta.commits and ensure MDT cleaning runs, so the files partition doesn't bloat into a listing problem of its own. Avoid pathologically frequent commits (micro-batch sizing). 5. **For partition-discovery cost specifically**, move to a release with the mature indexing subsystem — Record Level Index (0.14+) and partition/column-stats indexes reduce reliance on `getAllPartitionPaths`. 6. **Third-party S3 caveat:** S3A listing/optimizations and the vendor's metric semantics may differ from AWS S3; confirm the LIST count is real listing and not the vendor counting MDT object reads as "list". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
