deepakpanda93 commented on issue #11273: URL: https://github.com/apache/hudi/issues/11273#issuecomment-4851161429
Hello @SuneethaYamani As @ad1happy2go noted, the "compaction" you see on a COPY_ON_WRITE table is the metadata table compacting — the MDT is always Merge-On-Read internally, so it schedules compaction independent of your data table's type. That part is by design. On the slow MDT compaction (~5h with RECORD_INDEX): - It's governed by hoodie.metadata.compact.max.delta.commits (default 10). You can't disable MDT compaction while keeping the metadata table — compaction is what keeps the MDT log files bounded and reads fast. Setting it very high (e.g. 365) avoids the periodic cost but lets MDT log files grow, which then slows MDT reads, so it's not a good long-term setting. - Since RLI (hoodie.index.type=RECORD_INDEX) requires the metadata table, disabling the MDT isn't an option if you want RLI. - Recent 1.x releases include significant MDT / record-index performance improvements (e.g. HUDI-6200 [#8684](https://github.com/apache/hudi/pull/8684) for larger indexes). Please retest MDT compaction time on a recent 1.1.x release with default compact.max.delta.commits. If MDT compaction is still an unacceptable bottleneck on 1.x, please share the MDT compaction metrics (record_index size, delta-commit count, executor resources) and we'll dig into the performance. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
