alamb opened a new issue, #3941: URL: https://github.com/apache/arrow-datafusion/issues/3941
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** The basic challenge is that DataFusion can use an unbounded amount of memory for running a plan which typically results in DataFusion being killed by some system memory protection limit (e.g. the OOM Killer on Linux). See https://github.com/apache/arrow-datafusion/issues/587 for more details As a first step towards supporting larger datasets in DataFusion, if a plan will exceed the overall budget, it should generate a runtime error rather than exceeding the budget and risking If the hash aggregate exceeds is budget, it will return a runtime error (resource exhausted) **Describe the solution you'd like** 1. The user can define a limit for memory via [`MemoryManagerConfig`](https://docs.rs/datafusion/13.0.0/datafusion/execution/memory_manager/enum.MemoryManagerConfig.html) 2. All operators that consume significant memory (Hash, Join, Sort) will properly account for and request memory from the `MemoryManager` via methods like [`try_grow`](https://docs.rs/datafusion/13.0.0/datafusion/execution/memory_manager/trait.MemoryConsumer.html#method.try_grow) 3. If sufficient memory can not be allocated, the plan should return [ResourcesExhausted](https://docs.rs/datafusion/13.0.0/datafusion/error/enum.DataFusionError.html#variant.ResourcesExhausted) Needed: - [ ] Use `MemoryManager` in `SortExec`, and return errors if the memory budget is exceeded - [ ] Use `MemoryManager` in Aggregate operators, and return errors if the memory budget is exceeded - [ ] Use `MemoryManager` in Join operators, and return errors if the memory budget is exceeded **Describe alternatives you've considered** We can always increase the accuracy of the memory allocation accounting (e.g. `RecordBatch`es internal to operators, etc). However, for this initial epic I would like to get the major consumers of memory instrumented and using the `MemoryManager` interface. Hopefully this will also allow **Additional context** cc @yjshen @crepererum related to issues like https://github.com/influxdata/influxdb_iox/issues/5776 (and some internal issues of our own) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
