andygrove opened a new pull request, #6194:
URL: https://github.com/apache/datafusion-comet/pull/6194
## Which issue does this PR close?
No issue was filed. This is a small documentation correction found while
working on #6191.
## Rationale for this change
The memory management guide and the tuning guide both say that a native
shuffle runs the pre-shuffle operators and the shuffle writer as two native
execution contexts in the same task, and use that to explain why the memory
pool is shared per task. That stopped being true with #4507, which plans the
shuffle writer together with the native operators that feed it.
I checked this by counting the native plans each task created, using a
temporary print where `CometExecIterator` creates its plan, on the default
Spark 4.1 build:
| Query | Native plans per task |
| --- | --- |
| Scan, filter, native shuffle | 1 |
| Union of two filtered scans, partial aggregate, native shuffle | 2 |
| Filter, `coalesce(1)`, aggregate | 6 in the single task: one per coalesced
input partition, plus the aggregate |
| Filter, native Parquet write | 2 |
| Filter, then a typed `map` that Comet does not support, then an aggregate
| 1, because everything above the `map` stays in Spark |
## What changes are included in this PR?
- The memory management guide notes that a native shuffle is a single plan.
It then lists the operators that do split a task's native work into separate
plans:
- union and coalesce, which read their children through the JVM
- collect limit and take-ordered-and-project, which apply their limit in a
plan of their own
- native Parquet and Iceberg writes, which run the writer as a plan of its
own
- The tuning guide's description of the shared pool now uses union and
coalesce as its example instead of shuffle.
## How are these changes tested?
This is a documentation-only change. The counts above come from a temporary
instrumented run that is not part of this PR. The collect limit,
take-ordered-and-project and Iceberg write cases were checked in the code
rather than measured.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
