gaoyajun02 commented on PR #58008: URL: https://github.com/apache/spark/pull/58008#issuecomment-5408493351
Addressed all 6 review items in `1eddc72` (2 blocking, 2 non-blocking, 2 nits), plus a related stale-set lifecycle bug surfaced while investigating #2: - **#1 (blocking):** `fallback.enabled` registered default `true` -> `false`. - **#2 (blocking):** `TaskSetManager` holds the `ShuffleMapStage` reference and re-evaluates indeterminacy when the late duplicate arrives (was cached as a `Boolean` at construction). - **#3 (non-blocking):** `isStaleChunk` delegates to a new `intersectsStaleMapIndexes` tracker query — no per-chunk defensive copy. - **#4 (non-blocking):** test matrix now covers the real default (config unset) and an enabled/detect-all-disabled case with runtime indeterminacy toggled post-construction. - **Nits:** "identical output" -> "the same data set"; `[[withSpark]]` -> `[[LocalSparkContext.withSpark]]`. - **Related:** `unregisterAllMapAndMergeOutput` now clears `staleMapIndexes` — without this, stale marks from a previous attempt survived a stage retry and caused spurious chunk fallback on the retried attempt's fresh push data. Added an assertion to the existing propagation test. `core/compile`, `core/Test/compile`, and the full `TaskSetManagerSuite` / `MapOutputTrackerSuite` / `ShuffleBlockFetcherIteratorSuite` pass. PTAL @cloud-fan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
