comphead commented on PR #7655: URL: https://github.com/apache/datafusion-comet/pull/7655#issuecomment-6992919362
Thanks @andygrove, all three points held up when I checked them against the code. Addressed in 51ef4bcf77. **Memory.** The copies do stack. Each broadcast has its own `SharedDecode`, so stage N's idle copy is still alive while stage N+1 decodes and holds its own. I measured a query with four broadcast hash joins in consecutive stages: a 4M-row fact in 32 files, four 1.5M-row dims of 24.4 MB decoded each, `local[8]`, AQE off, Spark 4.0. I sampled `CometArrowAllocator` every millisecond and took the median of 7 runs with the arms interleaved: | arm | live shared copies | peak shared bytes | decodes | query time | | --- | ---: | ---: | ---: | ---: | | `idleTimeout=0` (as `main`) | 1 | 24.4 MB | 13 | 657 ms | | `idleTimeout=1s`, no cap | 4 | 97.5 MB | 4 | 556 ms | | `idleTimeout=1s`, cap 128 MiB | 4 | 97.5 MB | 4 | 564 ms | | `idleTimeout=1s`, cap 50 MiB | 3 | 73.1 MB | 4 | 569 ms | So without a cap, the timeout adds every broadcast that finished in the last second to the executor's peak. The cap was small enough to include here, so the PR still closes #7487: - `spark.comet.exec.broadcastExchange.shareDecoded.maxIdleSize`, 128 MiB by default, caps the idle copies of all broadcasts on an executor. - A copy larger than the cap is freed as soon as no task holds it, so a GB-scale broadcast is never kept idle. - Keeping a copy frees the copies that have been idle longest until the rest fit. The newest copy is the one the next wave is about to read. - The 50 MiB row shows the cap binding: two idle copies plus the held one, and each broadcast is still decoded once. The cap bounds only the idle copies. The copy that running tasks hold is unchanged from `main`. `IdleDecodes` never takes a broadcast's lock. The lock order is always the broadcast's lock and then the registry's, and evicted copies are freed outside both. While writing the cap I also found that an idle entry freed by another path, such as a read with a timeout of 0, could later be closed again by its old expiry. That is a double close. An expiry now frees an entry only while it is still `current`. **Tests.** Added "a task that comes after another completes reads its decode, which the timer then frees". It runs two tasks one after the other with the real expiry thread and a 500 ms timeout, asserts one decode, and waits for the thread to free the copy. The cap has its own test with an injected `IdleDecodes` and a hand-run scheduler. **Defaults.** `batches` on both `CometBroadcastValue` and `SharedDecode` now takes a `SharedDecode.Retention(idleTimeoutMs, maxIdleBytes)` with no default. The tests pass `NoRetention` or an explicit `Retention`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
