mingmwang commented on issue #4564: URL: https://github.com/apache/arrow-datafusion/issues/4564#issuecomment-1344154902
@alamb @tustvold @andygrove @yahoNanJing I would like to extend the current `DiskManager's` functionality to not only manager the temp spill files but also the shuffle IPC files in Ballista, as well as the Cached data files generated by the System. The current `DiskManager` is only used to hold the temp directories to store the temporary files generated by any spillable `MemoryConsumers`. The `DiskManager` might be disabled in case that we do not want the MemoryConsumer to spill even under memory pressure. But if I extend the capabilities of the `DiskManager`, it means it should never be disabled. the spilling disable logic will have to be moved out from the `DiskManager`. I'm not sure whether I should extend the current `DiskManager's` capabilities or keep it unchanged and add another new `DiskFileManager`. Want to hear your opinions on this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
