rluvaton opened a new pull request, #23345:
URL: https://github.com/apache/datafusion/pull/23345
## Which issue does this PR close?
N/A
## Rationale for this change
this is part of an effort I'm doing to make sort much much faster.
here I'm making it faster when the data is somewhat sorted
in this there are 2 things that we save on in case we get to the fast path:
1. Not dealing with the the loser tree
2. Skip row by row comparison
3. interleave and data copy - as we output the same batch as input
## What changes are included in this PR?
## Are these changes tested?
yes and existing tests
## Are there any user-facing changes?
yes,
- sort may not respect `batch_size` for already sorted input - it will
return the same size as the input stream
- sort may produce small batches between full sorted batches (if we have 3
streams, stream
--------
TODO
- [ ] Need to add benchmark
- [ ] for already sorted
- [ ] for random
- [ ] for reverse ordering (the worse case scenario)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
