Hi, I propose adding memory management capabilities to DataFusion, expanding the scale of data it can handle, and increasing its applicable scenarios.
Here's the design doc [1] available for comments, and an experimental PR [2] illustrates the usage of the proposed API through an ExternalSorter example. I want to raise it in this forum for a broader audience and more valuable suggestions. [1] https://docs.google.com/document/d/1BT5HH-2sKq-Jxo51PNE6l9NNd_F-FyyYcyC3SKTnkIA/edit [2] https://github.com/yjshen/arrow-datafusion/pull/3 Best, Yijie
