Flight is great way to share data among processes, and holding shared data in-memory.
I couldn't find any api or design to handle fall back mechanism in case data is not fitting in memory. Cases: 1- Once memory buffer is nearing full, data should spill over to disk. 2- Spilling over disk or memory mapped file. 3- Should it be .arrow file or feather format on Disk. 4- Should it be compressed? any design suggestion? Can we have a feasibility check and performance impact in terms of benchmark #.
