viirya commented on PR #58097: URL: https://github.com/apache/spark/pull/58097#issuecomment-5347919762
Thanks for the very thorough review, @dongjoon-hyun -- especially catching the two hangs; both were reachable on the opt-in path and I have fixed all four. Pushed as a new commit. - **Live-reduce-partition mapping**: you are right that "reachable" was wrong. Tightening it to a *direct* dependency then broke `LIMIT` (`executeTake`/`collect` wrap the `ShuffledRowRDD` in a `MapPartitionsRDD`), so the discriminator is now an **identity-preserving** narrow chain: `readsShuffleByIdentity` walks `OneToOneDependency` hops only. That admits the collect/take wrapper (1:1, same index) and rejects `coalesce` (a custom `NarrowDependency`) and `union` (a `RangeDependency` offset), which stay fully live and are drained by that operator. Added your `coalesce` case as a regression test. - **`clearAbandoned` reset**: moved out of `write()` entirely. The DAGScheduler now calls a new `PipelinedShuffleManager.onPipelinedProducerStageSubmit` hook when it submits the producer stage -- one point before any map task of the run starts, so it cannot race the run own writers/readers. The scheduler stays transport-agnostic (it calls the trait, not the concrete transport). - **`wants` per-record allocation**: the static live set is now a precomputed `Array[Boolean]` (one array load per record); abandonment stays checked at hand-off, not per record. - **Test session isolation**: the SQL suites now stop and clear any active/default session before `getOrCreate()`. CI is re-triggered by the push. I will follow up once it is green. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
