yjshen opened a new pull request #1526:
URL: https://github.com/apache/arrow-datafusion/pull/1526
# Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases. You can
link an issue to this PR using the GitHub syntax. For example `Closes #123`
indicates that this PR will close issue #123.
-->
Closes #587 .
# Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in
the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your
changes and offer better suggestions for fixes.
-->
When DataFusion processes a single partition, it will keep allocating memory
until the OS or the container system kills it. To make it worse, concurrently
executing partitions or even simultaneously running plans will compete for
available memory until all memory is exhausted. It is more challenging to meet
the memory requirements for all operators of each partition when it is running.
None of the partitions or plans would run to finish.
Therefore, the ability to control the total memory usage of the process as a
whole, and at the same time, allocate the available memory to each execution
partition is extremely important. Under this guarantee: when the memory is
sufficient, the operator can acquire as much of the memory to do the
computation; when the memory is tight, the operator can be downgraded to use
the disk to store some intermediate results (spilling to disk) and use a
limited memory for execution.
# What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is
sometimes worth providing a summary of the individual changes in this PR.
-->
The proposed memory management architecture is the following:
1. User designates max execution memory by setting
`RuntimeConfig.max_memory` and `RuntimeConfig.memory_fraction` (float64 between
0..1). The actual max memory DataFusion could use `pool_size = max_memory *
memory_fraction`.
2. The entities that take up memory during its execution are called `Memory
Consumers`. Operators or others are encouraged to register themselves to the
memory manager and report its usage through `mem_used()`.
3. There are two kinds of consumers:
- `Controlling` consumers that would acquire memory during its execution
and release memory through `spill` if no more memory is available.
- `Tracking` consumers that exist for reporting purposes to provide a
more accurate memory usage estimation for memory consumers.
4. Controlling and tracking consumers share the pool. Each controlling
consumer could acquire a maximum of
`(pool_size - all_tracking_used) / active_num_controlling_consumers`.
```
Memory Space for the DataFusion Lib / Process of `pool_size`
┌──────────────────────────────────────────────z─────────────────────────────┐
│ z
│
│ z
│
│ Controlling z Tracking
│
│ Memory Consumers z Memory Consumers
│
│ z
│
│ z
│
└──────────────────────────────────────────────z─────────────────────────────┘
```
# Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
Users could limit the max memory used for DataFusion through
`RuntimeConfig::max_memory` and `RuntimeConfig::memory_fraction`
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
# Note
In addition to the proposed memory manager as well as the runtime that
plumbing the execute API, an `ExternalSortExec` is implemented to illustrate
the API usage.
TODOs:
- [ ] Tests for `external sorter`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
