sunchao commented on PR #58747: URL: https://github.com/apache/spark/pull/58747#issuecomment-5672179300
Thanks @viirya, @dongjoon-hyun, and @peter-toth. Applied the narrow fix in 2dcee6dd08f5323525963b53e950da9b5cf19ec7 and updated the title, description, and tests. The production change moves the existing registration-and-notification block to the top of the acquisition loop. It removes the waiter map and waiter-specific cleanup, and accepts the existing temporary drop in the active-task count after a last-byte release. Preventing the missing-entry crash is sufficient for the Comet acquire/release interleaving that motivated this work; continuous fairness participation is a separate policy decision. I kept `notifyAll()` with registration. A three-task regression demonstrates why: re-registering A can lower B's minimum share enough for B to proceed, even while A must wait again. Silent re-registration leaves B asleep in that schedule. The revised suite includes the public two-consumer regression in both memory modes, checks that the peer retains all 700 bytes after the completed waiter is released, and covers interruption with the original 100-byte reservation still held. Pool-only cases run once. The multiple-waiter and new zero-byte-cleanup cases are removed. The two consumers share one `TaskMemoryManager`; ordinary release can overlap its pending acquisition, while acquisition and task cleanup are serialized by the task-manager monitor. The tests cover that allocator/API schedule. I have not run an end-to-end Comet or pipelined Python UDF query. Validation so far: all five focused cases pass against cached Spark 4.0.1 dependencies, both public crash cases fail against the base pool, and the silent-reinsertion notification control times out. Repository ScalaStyle checks pass for both changed files. The description explains the focused harness and its limits; the full local source build is still resolving dependencies and [CI for the new revision](https://github.com/sunchao/spark/actions/runs/34908529712) is pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
