deepakpanda93 commented on issue #7057: URL: https://github.com/apache/hudi/issues/7057#issuecomment-4844372445
Thanks for the detailed root-cause analysis, @KnightChess — it was spot on. This was a race under OCC multi-writer where one job's failed-write cleanup removes a pending replacecommit instant while a concurrent job reads it from a cached timeline in getAllFileGroupsInPendingClusteringPlans, throwing Error getting all file groups in pending clustering. This is now fixed. ClusteringUtils.getClusteringPlan does exactly what you suggested — on a read failure it reloads the active timeline and, if the instant is no longer present (rolled back by a concurrent writer), skips it instead of failing ([#18288](https://github.com/apache/hudi/pull/18288), in 1.2.0): ``` if (metaClientOpt.isPresent() && !metaClientOpt.get().reloadActiveTimeline().containsInstant(pendingReplaceInstant)) { // instant rolled back by a concurrent writer -> skip return Option.empty(); } ``` Supporting changes in 1.x also reduce this class of conflict: clustering now uses a dedicated clustering action rather than replacecommit (HUDI-7905, [#11553](https://github.com/apache/hudi/pull/11553), 1.0.0+), and INSERT OVERWRITE overlapping a pending clustering is now rejected ([#18829](https://github.com/apache/hudi/pull/18829)). Closing as fixed in 1.2.0. If you still hit Error getting all file groups in pending clustering on 1.2.0+, please reopen with the timeline listing and writer configs. Thanks again! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
