Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/1075
One additional thought. This bug was found when sorting 18 GB of data in 8
GB of memory. That is, a case in which the sort must spill.
What happens in the case in which the 18 GB of data is sorted in, say, 20
GB of memory (an in-memory sort)? We don't want the merge limit to force a
spill in this case; kind of defeats the purpose of an in-memory sort.
So:
1. Does the limit affect in memory sort? If so, we need to revise the
solution.
2. Does the in-memory sort suffer from a similar performance issue? If so,
we need to revise the in memory sort.
One possible solution is to:
1. Defer sorting of individual batches until necessary.
2. Sort batches just before spilling.
3. If all batches fit in memory, do a single, combined sort (using an SV4).
---
