2010YOUY01 commented on PR #15948: URL: https://github.com/apache/datafusion/pull/15948#issuecomment-2853206011
> > Benchmark test is not included because it was difficult to limit them to shuffle read scope. > > Did you test it locally? Do you have any performance numbers you can share? > > * I wonder if we could consolidate the spill file managing into SpillManager that was introduced by @2010YOUY01 in [refactor: Use SpillManager for all spilling scenarios #15405](https://github.com/apache/datafusion/pull/15405) I believe the SMJ reader is the only remaining component that hasn't been migrated to SpillManager (the write path of SMJ has already been refactored to use SpillManager). Due to the complexity of the SMJ operator, this refactor might be a bit tricky. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
