taoran92 opened a new issue, #10892: URL: https://github.com/apache/paimon/issues/10892
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. The related PR discussions and original CI logs are linked below. ### Paimon version `2.2-SNAPSHOT`, master commit `0369648333b9278d4d61a0768cf355f3299ec111`. The failure in PR #10838 occurred at head `948175f8e2f74ab499067be5264d58fd099589ee`. ### Compute Engine Observed in CI with: - Spark 3.5.8 / Scala 2.12 / JDK 8. - Spark 3.5.8 / Scala 2.13 / JDK 8. - Spark 4.1.2 / Scala 2.13 / JDK 17. ### Minimal reproduce step The failure has been observed intermittently in the Spark CI suites containing `RemoveOrphanBlobsProcedureTest`, in the test `Paimon procedure: frozen used mark survives stale sidecar recompute`. Three CI occurrences across Spark 3 and Spark 4 are linked below. There are currently no deterministic reproduction steps for the original CI failure. Running the test alone or rerunning CI may pass. ### What doesn't meet your expectations? Expected: `clearShuffleOutputs()` skips an ID that was unregistered after enumeration, invalidates the remaining registered shuffles, and propagates other errors. Actual: it throws `InvocationTargetException` at line 582 before the deletion-result assertions execute. The cause is `SparkException` on Spark 3 or `ShuffleStatusNotFoundException` on Spark 4, reporting that the shuffle ID does not exist. Excerpt from the [PR #10838 CI log](https://github.com/apache/paimon/actions/runs/40827887533/job/123295122241), with intermediate frames omitted: ```text java.lang.reflect.InvocationTargetException: ... at org.apache.paimon.spark.procedure.RemoveOrphanBlobsProcedureTest.clearShuffleOutputs(RemoveOrphanBlobsProcedureTest.scala:582) at org.apache.paimon.spark.procedure.RemoveOrphanBlobsProcedureTest.$anonfun$new$27(RemoveOrphanBlobsProcedureTest.scala:258) ... Cause: org.apache.spark.SparkException: unregisterAllMapAndMergeOutput called for nonexistent shuffle ID 83. at org.apache.spark.MapOutputTrackerMaster.unregisterAllMapAndMergeOutput(MapOutputTracker.scala:849) ``` The helper invalidates shuffle output metadata so that `deleted.collect()` recomputes after Dataset cache eviction. Removing this invalidation would allow Spark to reuse old shuffle outputs and leave that recomputation path untested. ### Anything else? CI occurrences, with dates in UTC: - **PR #10187, 2026-09-26, Spark 4 / Scala 2.13:** the same test fails at `clearShuffleOutputs:582` with `ShuffleStatusNotFoundException` for shuffle ID **82**. The affected ScalaTest run reports 1019 succeeded and 1 failed. [Original CI log](https://github.com/apache/paimon/actions/runs/36246873273/job/108417818936), [maintainer discussion](https://github.com/apache/paimon/pull/10187#issuecomment-5951614767). - **PR #10284, 2026-09-27, Spark 3 / Scala 2.12:** the same test and helper fail with `SparkException` for shuffle ID **83**; 1019 succeeded and 1 failed. [Original CI log](https://github.com/apache/paimon/actions/runs/36344997576/job/108692319567), [maintainer discussion](https://github.com/apache/paimon/pull/10284#issuecomment-5936003621). - **PR #10838, 2026-10-29, Spark 3 / Scala 2.13:** the same test and helper fail for shuffle ID **83**; 1101 succeeded and 1 failed. [Original CI log](https://github.com/apache/paimon/actions/runs/40827887533/job/123295122241). The helper was introduced in #9812. Its [implementation at the inspected commit](https://github.com/apache/paimon/blob/0369648333b9278d4d61a0768cf355f3299ec111/paimon-spark/paimon-spark-ut/src/test/scala/org/apache/paimon/spark/procedure/RemoveOrphanBlobsProcedureTest.scala#L568-L582) enumerates `shuffleStatuses` and invokes `unregisterAllMapAndMergeOutput` for each ID: ```scala // Enumeration and invalidation are separate operations. shuffleStatuses.keys.toSeq.foreach( id => unregister.invoke(mapOutputTracker, Int.box(id))) ``` A concurrent `unregisterShuffle(id)` can remove the entry between enumeration and invocation. `unregisterAllMapAndMergeOutput(id)` then looks up the ID again and throws because the entry is absent. This interleaving is consistent with the CI exception, but the logs do not capture the removal or identify the thread responsible. Proposed fix: catch only the missing-shuffle error for the ID being processed, inside the per-ID loop, and continue with the remaining IDs. A `containsShuffle` check alone is insufficient because the entry can disappear after the check. Add a deterministic regression test that removes an ID after enumeration and then calls the Paimon helper. Verify that the helper skips the removed ID, still invalidates the remaining registered shuffles, and propagates unrelated errors. Retain the existing `stale sidecar recompute` and `sidecar reads start failing` tests. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
