NoahKusaba commented on issue #2578:
URL:
https://github.com/apache/datafusion-ballista/issues/2578#issuecomment-6159656749
> 5\. We could consider choosing the strategy based on number of partitions
as long as we can override behavior with configs.
> If you want to create a PR with the implementation, I'd be happy
to test with TPC-H @ SF1k and 200 partitions, which is the default benchmark
that I use for now.
Thanks for taking the time to read the proposal! I would greatly appreciate
if you had the time to benchmark the PR, but want to take some more time to
refine it and will open when the refactor is merged in.
I understand the concern about memory growing with partition count, the hope
is by having both paths:
SplitBuffered -> Fast path, buffers the batch to each output partitions
coalescer (memory overhead scales with partition count)
DeferredBuffer -> Old path, no memory concerns
This is the logic I'm thinking about:
Worst case memory for splitting (cost) = partitions * batch_size * avg
row bytes + partitions * view columns * 1MB
With budget being configurable it would start Deferred:
cost ≤ budget → switch to split
cost > budget → stay deferred for good
This way at best we can get the speed benefit from split path on low
partition counts, and not regress speed at higher partition counts by switching
back to deferred under memory pressure (with small overhead).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
