andygrove opened a new pull request, #2654: URL: https://github.com/apache/datafusion-ballista/pull/2654
# Which issue does this PR close? Closes #2651. # Rationale for this change The "In-memory shuffle" direction in the shuffle design page cites a RAM-disk experiment: it recovered a large share of the TPC-H gap against a pipelined engine. The page uses that to call in-memory shuffle the largest single win available, and to conclude that the filesystem round trip, not the barrier, is the cost. That measurement was [retracted on #2308](https://github.com/apache/datafusion-ballista/pull/2308#issuecomment-5308605130) shortly after the page merged: the queries had been failing once the RAM disk filled up. The result that held up is #2317's in-memory store, run on TPC-H SF10. It measured neutral at a 1 GiB budget and about 5% faster at 12 GiB. The passthrough writer still wrote files in that experiment, so the size of the win is an open question rather than settled either way. # What changes are included in this PR? The "In-memory shuffle" bullet in `docs/source/contributors-guide/shuffle.md` now: - drops the RAM-disk claim and the conclusion drawn from it; - states #2317's results as the current evidence, including what they don't cover; - links to #2317 alongside #2318. # Are there any user-facing changes? Documentation only. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
