jychen7 opened a new issue, #6000: URL: https://github.com/apache/arrow-datafusion/issues/6000
### Is your feature request related to a problem or challenge? This is separated from https://github.com/apache/arrow-datafusion/issues/3516#issuecomment-1254006432. On a high level, #3516 implements the `green` path ```mermaid graph LR; SPME[SortPreservingMergeExec]-->spwn_handler-->SE[SortExec]; SPME[SortPreservingMergeExec]-->streaming_merge-->SPMS[SortPreservingMergeStream]; SE-->execute-->do_sort-->ExternalSort-->insert_batch-->sort_batch-->lexsort_to_indices; insert_batch-->spill-->in_mem_partial_sort-->get_sorted_iter-->lexsort_to_indices; ExternalSort-->sort-->streaming_merge; ExternalSort-->sort-->in_mem_partial_sort; style spwn_handler color:green style SE color:green style execute color:green style do_sort color:green style ExternalSort color:green style insert_batch color:green style sort_batch color:green style lexsort_to_indices color:green style sort color:green style in_mem_partial_sort color:green style get_sorted_iter color:green style spill color:green ``` This should reduce memory usage of sort and hope this could help for https://github.com/apache/arrow-datafusion/issues/5969a ### Describe the solution you'd like Push down the limit in `grey` path above ### Describe alternatives you've considered _No response_ ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
