deepakpanda93 commented on issue #13002:
URL: https://github.com/apache/hudi/issues/13002#issuecomment-4856616444
Hello @gbcoder2020
The HoodieCompactionException here is a wrapper — the actual cause is in the
nested exception:
```
SparkException: Task 2075 in stage 260.0 failed 4 times …
ExecutorLostFailure … Executor heartbeat timed out after 149584 ms
```
So this is an executor-loss / resource issue during metadata-table
compaction, not a Hudi correctness bug. Answering your questions:
1. What caused the upsert to fail? The metadata-table compaction stage ran
~2 hours and executors were lost to heartbeat timeouts (typically GC pressure /
OOM / a slow-or-large compaction / network). Because MDT compaction runs inline
during initMetadataTable, its failure fails the whole upsert.
2. Why did it recover with no config change? It's transient — on the
successful run the cluster had enough headroom (less contention/GC) for the
compaction to complete. Hudi re-attempts the pending MDT compaction on
subsequent writes until it succeeds.
3. The Mar 11 requested → Mar 16 committed compaction: the compaction was
scheduled on Mar 11 but couldn't finish (executor loss), so it stayed pending.
MDT compaction must complete before new MDT delta commits proceed, so each
subsequent write retried it; it finally committed Mar 16 when a run had enough
resources. That roll-forward is expected behavior.
4. How to guard against it:
- Give executors more memory / fewer cores-per-executor to reduce
GC/OOM; raise spark.network.timeout and spark.executor.heartbeatInterval so
transient stalls don't kill executors.
- Keep the MDT compaction bounded:
hoodie.metadata.compact.max.delta.commits (default 10) — tune, but don't set
very high.
- If you use record-level index / large indexes, ensure the MDT
compaction has adequate resources (it scales with index size).
- Consider a recent 1.x release, which includes metadata-table
performance improvements (e.g. HUDI-6200
[#8684](https://github.com/apache/hudi/pull/8684)).
If you can reproduce a metadata compaction failure whose nested cause is a
Hudi exception (not ExecutorLostFailure), please share that full trace. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
