This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 0341e5edb3 Publish built docs triggered by
a8fc789b3cb8546592299c736e8a89a69adfb0b5
0341e5edb3 is described below
commit 0341e5edb391666976c5ff7058128d5dd0ec9814
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Thu Sep 24 05:56:54 2026 +0000
Publish built docs triggered by a8fc789b3cb8546592299c736e8a89a69adfb0b5
---
...id-013012468fa27c9e9fdb0a14196d357c619c68a8.svg | 2 +-
.../contributor-guide/memory_management.md.txt | 54 +++++++++++----------
_sources/contributor-guide/tracing.md.txt | 21 ++++----
_sources/user-guide/latest/configs.md.txt | 1 +
_sources/user-guide/latest/tuning.md.txt | 56 ++++++++++++++++++++++
contributor-guide/memory_management.html | 48 +++++++++++--------
contributor-guide/tracing.html | 18 ++++---
searchindex.js | 2 +-
user-guide/latest/configs.html | 6 ++-
user-guide/latest/tuning.html | 53 +++++++++++++++++++-
10 files changed, 190 insertions(+), 71 deletions(-)
diff --git a/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg
b/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg
index f7a19985c6..882fb1c923 100644
--- a/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg
+++ b/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg
@@ -1 +1 @@
-<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg";
xmlns:xlink="http://www.w3.org/1999/xlink"; class="flowchart" style="max-width:
511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 942"
role="graphics-document document"
aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet
ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes
edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes
dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg";
xmlns:xlink="http://www.w3.org/1999/xlink"; class="flowchart" style="max-width:
511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 942"
role="graphics-document document"
aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet
ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes
edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes
dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
diff --git a/_sources/contributor-guide/memory_management.md.txt
b/_sources/contributor-guide/memory_management.md.txt
index 6296fefeee..93ba4e8754 100644
--- a/_sources/contributor-guide/memory_management.md.txt
+++ b/_sources/contributor-guide/memory_management.md.txt
@@ -363,14 +363,14 @@ diverge for several structural reasons:
Freeing memory does not necessarily return pages to the OS.
- **Non-Rust allocations.** Memory allocated by C dependencies through libc
`malloc`, and anything
`mmap`ed, never passes through Rust's `GlobalAlloc`, so neither the memory
pool nor the
- `jemalloc_allocated` metric sees it. In a default build the C dependencies
are libzstd
- (`zstd-sys`, behind the Parquet `zstd` codec), libhdfs (`hdfs-sys`, pulled
in by the default
- `hdfs-opendal` feature), and the TLS stack used for cloud object stores
(`aws-lc-sys`). Building
- with the `jemalloc` or `mimalloc` feature adds the allocator itself
(`tikv-jemalloc-sys`,
- `libmimalloc-sys`). It is worth knowing which dependencies are _not_ C,
because several names
- suggest otherwise: the other Parquet codecs are pure Rust in this build,
`snap` for Snappy,
- `lz4_flex` for LZ4 and `zlib-rs` for gzip, as is `libbz2-rs-sys` despite its
name, so those
- allocations do pass through `GlobalAlloc` and are counted.
+ allocation counters (`native_allocated`, `jemalloc_allocated`) see it. In a
default build the C
+ dependencies are libzstd (`zstd-sys`, behind the Parquet `zstd` codec),
libhdfs (`hdfs-sys`,
+ pulled in by the default `hdfs-opendal` feature), and the TLS stack used for
cloud object stores
+ (`aws-lc-sys`). Building with the `jemalloc` or `mimalloc` feature adds the
allocator itself
+ (`tikv-jemalloc-sys`, `libmimalloc-sys`). It is worth knowing which
dependencies are _not_ C,
+ because several names suggest otherwise: the other Parquet codecs are pure
Rust in this build,
+ `snap` for Snappy, `lz4_flex` for LZ4 and `zlib-rs` for gzip, as is
`libbz2-rs-sys` despite its
+ name, so those allocations do pass through `GlobalAlloc` and are counted.
- **Batches in flight across the FFI boundary.** Reservations stop at the
operator that made them.
Imported JVM batches are reserved only while a reserving operator holds
them, and exported native
batches have usually been released by the time the JVM receives them yet
stay resident until the
@@ -386,9 +386,13 @@ The practical consequence is that `reserved()` is a lower
bound on Comet's real
gap is workload-dependent. `spark.comet.exec.memoryPool.fraction` exists
purely so operators can
hand-tune a margin that covers the gap for their workload.
-To measure the gap on a real query, enable tracing with the `jemalloc` feature
and compare
-`jemalloc_allocated` against the summed `thread_NNN_comet_memory_reserved`
values; see
-[Tracing](tracing.md#analyzing-memory-usage).
+To measure the gap on a real workload, read the executor's periodic memory
usage log, which
+reports the bytes Rust's allocator has handed out next to the pools'
reservations; see
+[Sizing the Overhead from the Memory Usage Log][memory-usage-log]. For a view
per event rather
+than per interval, enable tracing and compare `native_allocated` against
+`comet_memory_reserved_total`; see
[Tracing](tracing.md#analyzing-memory-usage).
+
+[memory-usage-log]:
../user-guide/latest/tuning.md#sizing-the-overhead-from-the-memory-usage-log
## What the container sees
@@ -466,9 +470,9 @@ has bounds _declared reservations_, and the sections above
describe several stru
declared reservations are a lower bound on physical usage. The known gaps,
roughly in order of how
much they matter:
-- **No signal for real native usage.** The only way to observe the gap today
is to enable tracing
- with the `jemalloc` feature and compare `jemalloc_allocated` against summed
reservations after
- the fact. There is no runtime value that an operator, a metric, or a policy
could read.
+- **Real native usage is observed but not acted on.** Comet counts the bytes
Rust's allocator has
+ handed out, and each executor logs that count next to the pools'
reservations. Nothing reads it at runtime, though: no operator, metric, or
policy
+ responds to it, so an executor that outgrows its container is still stopped
only by the kill.
- **`spark.comet.exec.memoryPool.fraction` is a manual proxy for the gap.** It
asks operators to
guess a per-workload margin rather than measuring anything.
- **`CometArrowAllocator` is unbounded** and participates in no budget.
@@ -483,13 +487,14 @@ much they matter:
## Debugging memory issues
-| Tool | What it gives you
|
-| ------------------------------------------------ |
-------------------------------------------------------------------------- |
-| `spark.comet.debug.memory=true` | `LoggingMemoryPool` logs
every register/grow/shrink with the consumer name |
-| `spark.comet.explain.native.enabled=true` | Native plan with
per-operator metrics, including spill counts |
-| [Tracing](tracing.md#analyzing-memory-usage) | `jemalloc_allocated` vs
summed pool reservations; the accounting gap |
-| `TrackConsumersPool` | Names the top 10
consumers in `ResourcesExhausted` messages (always on) |
-| [`thresher`](https://github.com/cetra3/thresher) | Third-party crate that
dumps a jemalloc heap profile at a threshold |
+| Tool
| What it gives you
|
+|
------------------------------------------------------------------------------------------------
|
-------------------------------------------------------------------------------------------
|
+| `spark.comet.debug.memory=true`
| `LoggingMemoryPool` logs every register/grow/shrink with
the consumer name |
+| `spark.comet.explain.native.enabled=true`
| Native plan with per-operator metrics, including spill
counts |
+| [Memory usage
log](../user-guide/latest/tuning.md#sizing-the-overhead-from-the-memory-usage-log)
| Executor-wide native allocation vs pool reservations, logged every 10
seconds by default |
+| [Tracing](tracing.md#analyzing-memory-usage)
| `native_allocated` vs `comet_memory_reserved_total` per
event; the accounting gap over time |
+| `TrackConsumersPool`
| Names the top 10 consumers in `ResourcesExhausted`
messages (always on) |
+| [`thresher`](https://github.com/cetra3/thresher)
| Third-party crate that dumps a jemalloc heap profile at a
threshold |
A checklist for triaging an executor OOM kill:
@@ -498,9 +503,10 @@ A checklist for triaging an executor OOM kill:
treats it as fatal, so the executor is lost either way and the exit code is
what distinguishes
them. A failed task with `SparkOutOfMemoryError` and a surviving executor
is Spark's managed
memory pool, which is the only one of the three that is recoverable at task
level.
-2. Compare `jemalloc_allocated` against the summed pool reservations from a
trace. A large excess
- points at undeclared native allocations; a small excess points at the
budget simply being too
- small, or at the JVM side.
+2. Compare `allocated` against `reserved` in the executor's `Comet native
memory usage` log lines
+ leading up to the kill, or `native_allocated` against
`comet_memory_reserved_total` in a trace.
+ A large excess points at undeclared native allocations; a small excess
points at the budget
+ simply being too small, or at the JVM side.
3. Check `spark.comet.batchSize` against the schema width. Peak memory scales
with
`batch_size * columns`, and wide or deeply nested schemas amplify it.
4. Check whether the operators involved can spill at all. `ShuffledHashJoin`
cannot, so
diff --git a/_sources/contributor-guide/tracing.md.txt
b/_sources/contributor-guide/tracing.md.txt
index 4e7b82b914..507b58c8ce 100644
--- a/_sources/contributor-guide/tracing.md.txt
+++ b/_sources/contributor-guide/tracing.md.txt
@@ -27,20 +27,17 @@ directory with the filename `comet-event-trace.json`.
[Trace Event Format]:
https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/preview?tab=t.0#heading=h.yr4qxyxotyw
-Additionally, enabling the `jemalloc` feature will enable tracing of native
memory allocations.
+Native memory is traced as `native_allocated`. Comet wraps whichever global
allocator the build
+selected and counts the bytes it has handed out. It counts only what Rust code
allocated, so it can be compared against the
+memory pool's reservations without the allocator's own caching in the way. The
same figure appears in
+the executor's periodic memory usage log, which does not need tracing; see
+[Sizing the Overhead from the Memory Usage
Log](../user-guide/latest/tuning.md#sizing-the-overhead-from-the-memory-usage-log).
-```shell
-make release COMET_FEATURES="jemalloc"
-```
-
-The `alloc-accounting` feature adds a second, allocator-independent measure of
native memory. It
-wraps whichever global allocator the build selected and reports the bytes it
has handed out as
-`native_allocated`. Unlike `jemalloc_allocated` it does not require jemalloc,
and it counts only
-what Rust code allocated, so it can be compared against the memory pool's
reservations without the
-allocator's own caching in the way. The two features are independent and can
be combined:
+Enabling the `jemalloc` feature adds a second measure, `jemalloc_allocated`,
which also includes
+jemalloc's own metadata and fragmentation:
```shell
-make release COMET_FEATURES="jemalloc,alloc-accounting"
+make release COMET_FEATURES="jemalloc"
```
Example output:
@@ -165,7 +162,7 @@ samples: they are not an atomic per-query balance, and
neither is a measure of R
| jemalloc_allocated | Native memory usage for the executor
process (requires `jemalloc` feature)
|
| jvm_arrow_allocated | Bytes charged to Comet's Arrow allocator
tree on the JVM, including buffers imported from native over the Arrow C Data
Interface
|
| jvm_arrow_imported | Bytes charged to the Arrow C Data
Interface import allocator, a subset of `jvm_arrow_allocated`. An allocator
charge, not a measure of where the bytes were allocated; see above.
|
-| native_allocated | Bytes handed out by the Rust global
allocator, process-wide (requires `alloc-accounting` feature). Approximate to
within 64 KiB of un-flushed delta per live thread.
|
+| native_allocated | Bytes handed out by the Rust global
allocator, process-wide. Approximate to within 64 KiB of un-flushed delta per
live thread.
|
| comet_memory_reserved_total | Total memory reserved across every live
Comet memory pool, process-wide, whatever the configured pool type. Counts a
pool shared between execution contexts once, so unlike the per-thread counters
it can be compared directly against an allocation counter. |
| thread_NNN_comet_memory_reserved | Memory reserved by Comet's DataFusion
memory pool (summed across all contexts on the thread). NNN is the Rust thread
ID. Do not sum these across threads: a shared pool reports its full reservation
on every thread that references it. |
| thread_NNN_comet_jvm_shuffle | Off-heap memory allocated by Comet for
columnar shuffle. NNN is the Rust thread ID.
|
diff --git a/_sources/user-guide/latest/configs.md.txt
b/_sources/user-guide/latest/configs.md.txt
index ccdefd52e6..0290e84751 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -153,6 +153,7 @@ shuffle, including applications that rely only on remote
reliable storage or dec
| `spark.comet.exec.memoryPool` | The type of memory pool to be used for Comet
native execution when running Spark in off-heap mode. Available pool types are
`greedy_unified` and `fair_unified`. For more information, refer to the [Comet
Tuning
Guide](https://datafusion.apache.org/comet/user-guide/latest/tuning.html). |
fair_unified |
| `spark.comet.exec.memoryPool.fraction` | Fraction of off-heap memory pool
that is available to Comet. Only applies to off-heap mode. For more
information, refer to the [Comet Tuning
Guide](https://datafusion.apache.org/comet/user-guide/latest/tuning.html). |
1.0 |
| `spark.comet.maxTempDirectorySize` | The maximum amount of data (in bytes)
stored inside the temporary directories used by native operators when spilling.
Applied per Spark task, so an executor running N concurrent tasks may use up to
N times this value on shared local disks. Once the limit is reached, further
spills will fail and the query will error out. | 107374182400b |
+| `spark.comet.memory.logInterval` | How often each executor logs its native
memory usage at INFO level while Comet native plans are running: the bytes the
native allocator has handed out, and the bytes reserved in Comet's memory
pools. The difference is native memory that the pools are not accounting for.
The executor logs one line per interval however many tasks are running, and one
more after the last plan finishes. It logs a warning when the native memory
looks larger than the execut [...]
| `spark.comet.tracing.enabled` | Enable fine-grained tracing of events and
memory usage. For more information, refer to the [Comet Tracing
Guide](https://datafusion.apache.org/comet/contributor-guide/tracing.html). |
false |
<!-- prettier-ignore-end -->
<!--END:CONFIG_TABLE-->
diff --git a/_sources/user-guide/latest/tuning.md.txt
b/_sources/user-guide/latest/tuning.md.txt
index 84a3a16ede..368f46ece6 100644
--- a/_sources/user-guide/latest/tuning.md.txt
+++ b/_sources/user-guide/latest/tuning.md.txt
@@ -108,6 +108,7 @@ there is. That includes:
Reserved memory is therefore a lower bound on what Comet really uses, and how
far below it sits depends on the
workload. This is why Comet can stay within the pool's limit and still push
the executor past its container limit.
+Each executor logs how far apart the two are while Comet runs; see [Sizing the
Overhead from the Memory Usage Log].
To leave room for the part that is not counted, set
`spark.comet.exec.memoryPool.fraction` to a value less than
`1.0`, which restricts the amount of memory Comet is allowed to reserve.
@@ -180,10 +181,65 @@ spark.executor.memoryOverheadFactor=0.2
Raise the value further if executors are killed by the cluster manager (on
Kubernetes,
`ExecutorLostFailure` with exit code 137) rather than failing with a
task-level out-of-memory error.
+To measure how much Comet needs rather than guessing, see [Sizing the Overhead
from the Memory Usage Log].
Note that on Kubernetes and YARN the overhead is added to the container size,
so raising it reduces
how many executors fit on a node.
+[Sizing the Overhead from the Memory Usage Log]:
#sizing-the-overhead-from-the-memory-usage-log
+
+### Sizing the Overhead from the Memory Usage Log
+
+While Comet native plans are running, each executor logs its native memory
usage at INFO level,
+one line every 10 seconds for the whole executor:
+
+```
+Comet native memory usage: allocated 5412.3 MiB, reserved 3890.0 MiB (16
native plans, 8 memory pools)
+```
+
+- `allocated` is the memory that Comet's native code has allocated and not yet
freed, whether or not
+ a pool tracks it.
+- `reserved` is the part that Comet's memory pools track. It is charged against
+ `spark.memory.offHeap.size`, so the container already has room for it.
+
+The difference between the two, `allocated - reserved`, is Comet's untracked
native memory. It is
+the part of Comet's footprint that has to fit in
`spark.executor.memoryOverhead`, alongside the
+JVM's own non-heap memory. To size the overhead from it:
+
+1. Run a representative workload and find the line with the largest difference
in each executor's
+ log. Take both figures from the same line: they are sampled together, and
figures from different
+ lines describe different moments. Setting
`spark.comet.memory.logInterval=1s` for this run makes a
+ short-lived peak less likely to fall between samples.
+2. Start from the overhead the executors had before Comet was enabled, which
covers the JVM's own
+ non-heap memory, and add the largest difference seen on any executor.
+3. Add a margin on top. The log can miss the true peak between samples, and
neither figure includes
+ the allocator's fragmentation and retained pages, memory allocated by
native C libraries such as
+ zstd, or Comet's Arrow buffers on the JVM side.
+
+For example, a 16 GiB executor derives an overhead of 1638 MiB. If the largest
difference in its
+log is the 1522.3 MiB in the line above, the overhead needs to be at least
1638 + 1523 = 3161 MiB
+before any margin, so `spark.executor.memoryOverhead=4g` would be a reasonable
setting.
+
+The executor also logs a warning when its native memory looks larger than its
container allows:
+when the difference, plus everything in use in Spark's off-heap memory pool
(which includes Comet's
+reservations), exceeds `spark.memory.offHeap.size` plus the memory overhead.
This counts the part of
+the off-heap pool that nothing has acquired at that moment, which untracked
memory can occupy until
+Spark hands it out, so a quiet log is not a sign that the overhead is large
enough: size it from the
+largest difference as described above. The overhead also has to hold the JVM's
own non-heap memory,
+so by the time the warning appears the executor has likely outgrown its
container. It warns the first time this
+happens, and again each time it happens after dropping back below. The
overhead it uses is
+`spark.executor.memoryOverhead` if set, otherwise
`spark.executor.memoryOverheadFactor` of
+`spark.executor.memory` with a minimum of `spark.executor.minMemoryOverhead`,
as Spark sizes the
+default container. There is no warning in local mode.
+
+Look more closely before raising the overhead if the difference keeps growing
through a run rather
+than levelling off: native memory that is not being released will exhaust any
overhead eventually.
+The executor logs one more line after its last native plan finishes, and an
`allocated` figure there
+that grows from one query to the next points the same way.
+
+`spark.comet.memory.logInterval` is read when an executor starts its first
Comet native plan, so set
+it when the application is submitted. Set it to `0` to turn the log off.
+
### Determining How Much Memory to Allocate
Generally, increasing the amount of memory allocated to Comet will improve
query performance by reducing the
diff --git a/contributor-guide/memory_management.html
b/contributor-guide/memory_management.html
index 41569c2962..9863e79f73 100644
--- a/contributor-guide/memory_management.html
+++ b/contributor-guide/memory_management.html
@@ -922,14 +922,14 @@ retained/dirty page cache all add resident bytes that no
layer above the allocat
Freeing memory does not necessarily return pages to the OS.</p></li>
<li><p><strong>Non-Rust allocations.</strong> Memory allocated by C
dependencies through libc <code class="docutils literal notranslate"><span
class="pre">malloc</span></code>, and anything
<code class="docutils literal notranslate"><span
class="pre">mmap</span></code>ed, never passes through Rust’s <code
class="docutils literal notranslate"><span
class="pre">GlobalAlloc</span></code>, so neither the memory pool nor the
-<code class="docutils literal notranslate"><span
class="pre">jemalloc_allocated</span></code> metric sees it. In a default build
the C dependencies are libzstd
-(<code class="docutils literal notranslate"><span
class="pre">zstd-sys</span></code>, behind the Parquet <code class="docutils
literal notranslate"><span class="pre">zstd</span></code> codec), libhdfs
(<code class="docutils literal notranslate"><span
class="pre">hdfs-sys</span></code>, pulled in by the default
-<code class="docutils literal notranslate"><span
class="pre">hdfs-opendal</span></code> feature), and the TLS stack used for
cloud object stores (<code class="docutils literal notranslate"><span
class="pre">aws-lc-sys</span></code>). Building
-with the <code class="docutils literal notranslate"><span
class="pre">jemalloc</span></code> or <code class="docutils literal
notranslate"><span class="pre">mimalloc</span></code> feature adds the
allocator itself (<code class="docutils literal notranslate"><span
class="pre">tikv-jemalloc-sys</span></code>,
-<code class="docutils literal notranslate"><span
class="pre">libmimalloc-sys</span></code>). It is worth knowing which
dependencies are <em>not</em> C, because several names
-suggest otherwise: the other Parquet codecs are pure Rust in this build, <code
class="docutils literal notranslate"><span class="pre">snap</span></code> for
Snappy,
-<code class="docutils literal notranslate"><span
class="pre">lz4_flex</span></code> for LZ4 and <code class="docutils literal
notranslate"><span class="pre">zlib-rs</span></code> for gzip, as is <code
class="docutils literal notranslate"><span
class="pre">libbz2-rs-sys</span></code> despite its name, so those
-allocations do pass through <code class="docutils literal notranslate"><span
class="pre">GlobalAlloc</span></code> and are counted.</p></li>
+allocation counters (<code class="docutils literal notranslate"><span
class="pre">native_allocated</span></code>, <code class="docutils literal
notranslate"><span class="pre">jemalloc_allocated</span></code>) see it. In a
default build the C
+dependencies are libzstd (<code class="docutils literal notranslate"><span
class="pre">zstd-sys</span></code>, behind the Parquet <code class="docutils
literal notranslate"><span class="pre">zstd</span></code> codec), libhdfs
(<code class="docutils literal notranslate"><span
class="pre">hdfs-sys</span></code>,
+pulled in by the default <code class="docutils literal notranslate"><span
class="pre">hdfs-opendal</span></code> feature), and the TLS stack used for
cloud object stores
+(<code class="docutils literal notranslate"><span
class="pre">aws-lc-sys</span></code>). Building with the <code class="docutils
literal notranslate"><span class="pre">jemalloc</span></code> or <code
class="docutils literal notranslate"><span class="pre">mimalloc</span></code>
feature adds the allocator itself
+(<code class="docutils literal notranslate"><span
class="pre">tikv-jemalloc-sys</span></code>, <code class="docutils literal
notranslate"><span class="pre">libmimalloc-sys</span></code>). It is worth
knowing which dependencies are <em>not</em> C,
+because several names suggest otherwise: the other Parquet codecs are pure
Rust in this build,
+<code class="docutils literal notranslate"><span
class="pre">snap</span></code> for Snappy, <code class="docutils literal
notranslate"><span class="pre">lz4_flex</span></code> for LZ4 and <code
class="docutils literal notranslate"><span class="pre">zlib-rs</span></code>
for gzip, as is <code class="docutils literal notranslate"><span
class="pre">libbz2-rs-sys</span></code> despite its
+name, so those allocations do pass through <code class="docutils literal
notranslate"><span class="pre">GlobalAlloc</span></code> and are
counted.</p></li>
<li><p><strong>Batches in flight across the FFI boundary.</strong>
Reservations stop at the operator that made them.
Imported JVM batches are reserved only while a reserving operator holds them,
and exported native
batches have usually been released by the time the JVM receives them yet stay
resident until the
@@ -944,9 +944,11 @@ outstanding.</p></li>
<p>The practical consequence is that <code class="docutils literal
notranslate"><span class="pre">reserved()</span></code> is a lower bound on
Comet’s real footprint, and the
gap is workload-dependent. <code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.memoryPool.fraction</span></code> exists purely so
operators can
hand-tune a margin that covers the gap for their workload.</p>
-<p>To measure the gap on a real query, enable tracing with the <code
class="docutils literal notranslate"><span class="pre">jemalloc</span></code>
feature and compare
-<code class="docutils literal notranslate"><span
class="pre">jemalloc_allocated</span></code> against the summed <code
class="docutils literal notranslate"><span
class="pre">thread_NNN_comet_memory_reserved</span></code> values; see
-<a class="reference internal" href="tracing.html#analyzing-memory-usage"><span
class="std std-ref">Tracing</span></a>.</p>
+<p>To measure the gap on a real workload, read the executor’s periodic memory
usage log, which
+reports the bytes Rust’s allocator has handed out next to the pools’
reservations; see
+<a class="reference internal"
href="../user-guide/latest/tuning.html#sizing-the-overhead-from-the-memory-usage-log"><span
class="std std-ref">Sizing the Overhead from the Memory Usage Log</span></a>.
For a view per event rather
+than per interval, enable tracing and compare <code class="docutils literal
notranslate"><span class="pre">native_allocated</span></code> against
+<code class="docutils literal notranslate"><span
class="pre">comet_memory_reserved_total</span></code>; see <a class="reference
internal" href="tracing.html#analyzing-memory-usage"><span class="std
std-ref">Tracing</span></a>.</p>
</section>
<section id="what-the-container-sees">
<h2>What the container sees<a class="headerlink"
href="#what-the-container-sees" title="Link to this heading">#</a></h2>
@@ -1017,9 +1019,9 @@ has bounds <em>declared reservations</em>, and the
sections above describe sever
declared reservations are a lower bound on physical usage. The known gaps,
roughly in order of how
much they matter:</p>
<ul class="simple">
-<li><p><strong>No signal for real native usage.</strong> The only way to
observe the gap today is to enable tracing
-with the <code class="docutils literal notranslate"><span
class="pre">jemalloc</span></code> feature and compare <code class="docutils
literal notranslate"><span class="pre">jemalloc_allocated</span></code> against
summed reservations after
-the fact. There is no runtime value that an operator, a metric, or a policy
could read.</p></li>
+<li><p><strong>Real native usage is observed but not acted on.</strong> Comet
counts the bytes Rust’s allocator has
+handed out, and each executor logs that count next to the pools’ reservations.
Nothing reads it at runtime, though: no operator, metric, or policy
+responds to it, so an executor that outgrows its container is still stopped
only by the kill.</p></li>
<li><p><strong><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.memoryPool.fraction</span></code> is a manual
proxy for the gap.</strong> It asks operators to
guess a per-workload margin rather than measuring anything.</p></li>
<li><p><strong><code class="docutils literal notranslate"><span
class="pre">CometArrowAllocator</span></code> is unbounded</strong> and
participates in no budget.</p></li>
@@ -1047,13 +1049,16 @@ reservations are released only when a native operator
decides to spill on its ow
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.explain.native.enabled=true</span></code></p></td>
<td><p>Native plan with per-operator metrics, including spill counts</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal"
href="tracing.html#analyzing-memory-usage"><span class="std
std-ref">Tracing</span></a></p></td>
-<td><p><code class="docutils literal notranslate"><span
class="pre">jemalloc_allocated</span></code> vs summed pool reservations; the
accounting gap</p></td>
+<tr class="row-even"><td><p><a class="reference internal"
href="../user-guide/latest/tuning.html#sizing-the-overhead-from-the-memory-usage-log"><span
class="std std-ref">Memory usage log</span></a></p></td>
+<td><p>Executor-wide native allocation vs pool reservations, logged every 10
seconds by default</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">TrackConsumersPool</span></code></p></td>
+<tr class="row-odd"><td><p><a class="reference internal"
href="tracing.html#analyzing-memory-usage"><span class="std
std-ref">Tracing</span></a></p></td>
+<td><p><code class="docutils literal notranslate"><span
class="pre">native_allocated</span></code> vs <code class="docutils literal
notranslate"><span class="pre">comet_memory_reserved_total</span></code> per
event; the accounting gap over time</p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">TrackConsumersPool</span></code></p></td>
<td><p>Names the top 10 consumers in <code class="docutils literal
notranslate"><span class="pre">ResourcesExhausted</span></code> messages
(always on)</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference external"
href="https://github.com/cetra3/thresher";><code class="docutils literal
notranslate"><span class="pre">thresher</span></code></a></p></td>
+<tr class="row-odd"><td><p><a class="reference external"
href="https://github.com/cetra3/thresher";><code class="docutils literal
notranslate"><span class="pre">thresher</span></code></a></p></td>
<td><p>Third-party crate that dumps a jemalloc heap profile at a
threshold</p></td>
</tr>
</tbody>
@@ -1066,9 +1071,10 @@ code 52 with <code class="docutils literal
notranslate"><span class="pre">java.l
treats it as fatal, so the executor is lost either way and the exit code is
what distinguishes
them. A failed task with <code class="docutils literal notranslate"><span
class="pre">SparkOutOfMemoryError</span></code> and a surviving executor is
Spark’s managed
memory pool, which is the only one of the three that is recoverable at task
level.</p></li>
-<li><p>Compare <code class="docutils literal notranslate"><span
class="pre">jemalloc_allocated</span></code> against the summed pool
reservations from a trace. A large excess
-points at undeclared native allocations; a small excess points at the budget
simply being too
-small, or at the JVM side.</p></li>
+<li><p>Compare <code class="docutils literal notranslate"><span
class="pre">allocated</span></code> against <code class="docutils literal
notranslate"><span class="pre">reserved</span></code> in the executor’s <code
class="docutils literal notranslate"><span class="pre">Comet</span> <span
class="pre">native</span> <span class="pre">memory</span> <span
class="pre">usage</span></code> log lines
+leading up to the kill, or <code class="docutils literal notranslate"><span
class="pre">native_allocated</span></code> against <code class="docutils
literal notranslate"><span
class="pre">comet_memory_reserved_total</span></code> in a trace.
+A large excess points at undeclared native allocations; a small excess points
at the budget
+simply being too small, or at the JVM side.</p></li>
<li><p>Check <code class="docutils literal notranslate"><span
class="pre">spark.comet.batchSize</span></code> against the schema width. Peak
memory scales with
<code class="docutils literal notranslate"><span class="pre">batch_size</span>
<span class="pre">*</span> <span class="pre">columns</span></code>, and wide or
deeply nested schemas amplify it.</p></li>
<li><p>Check whether the operators involved can spill at all. <code
class="docutils literal notranslate"><span
class="pre">ShuffledHashJoin</span></code> cannot, so
diff --git a/contributor-guide/tracing.html b/contributor-guide/tracing.html
index 09832a6edd..8924cc9cec 100644
--- a/contributor-guide/tracing.html
+++ b/contributor-guide/tracing.html
@@ -552,18 +552,16 @@ under the License.
<p>With this feature enabled, each Spark executor will write a JSON event log
file in
Chrome’s <a class="reference external"
href="https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/preview?tab=t.0#heading=h.yr4qxyxotyw";>Trace
Event Format</a>. The file will be written to the executor’s current working
directory with the filename <code class="docutils literal notranslate"><span
class="pre">comet-event-trace.json</span></code>.</p>
-<p>Additionally, enabling the <code class="docutils literal notranslate"><span
class="pre">jemalloc</span></code> feature will enable tracing of native memory
allocations.</p>
+<p>Native memory is traced as <code class="docutils literal notranslate"><span
class="pre">native_allocated</span></code>. Comet wraps whichever global
allocator the build
+selected and counts the bytes it has handed out. It counts only what Rust code
allocated, so it can be compared against the
+memory pool’s reservations without the allocator’s own caching in the way. The
same figure appears in
+the executor’s periodic memory usage log, which does not need tracing; see
+<a class="reference internal"
href="../user-guide/latest/tuning.html#sizing-the-overhead-from-the-memory-usage-log"><span
class="std std-ref">Sizing the Overhead from the Memory Usage
Log</span></a>.</p>
+<p>Enabling the <code class="docutils literal notranslate"><span
class="pre">jemalloc</span></code> feature adds a second measure, <code
class="docutils literal notranslate"><span
class="pre">jemalloc_allocated</span></code>, which also includes
+jemalloc’s own metadata and fragmentation:</p>
<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span>make<span class="w"> </span>release<span
class="w"> </span><span class="nv">COMET_FEATURES</span><span
class="o">=</span><span class="s2">"jemalloc"</span>
</pre></div>
</div>
-<p>The <code class="docutils literal notranslate"><span
class="pre">alloc-accounting</span></code> feature adds a second,
allocator-independent measure of native memory. It
-wraps whichever global allocator the build selected and reports the bytes it
has handed out as
-<code class="docutils literal notranslate"><span
class="pre">native_allocated</span></code>. Unlike <code class="docutils
literal notranslate"><span class="pre">jemalloc_allocated</span></code> it does
not require jemalloc, and it counts only
-what Rust code allocated, so it can be compared against the memory pool’s
reservations without the
-allocator’s own caching in the way. The two features are independent and can
be combined:</p>
-<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span>make<span class="w"> </span>release<span
class="w"> </span><span class="nv">COMET_FEATURES</span><span
class="o">=</span><span class="s2">"jemalloc,alloc-accounting"</span>
-</pre></div>
-</div>
<p>Example output:</p>
<div class="highlight-json notranslate"><div
class="highlight"><pre><span></span><span class="p">{</span><span class="w">
</span><span class="nt">"name"</span><span class="p">:</span><span class="w">
</span><span class="s2">"decodeShuffleBlock"</span><span
class="p">,</span><span class="w"> </span><span class="nt">"cat"</span><span
class="p">:</span><span class="w"> </span><span class="s2">"PERF"</span><span
class="p">,</span><span class="w"> </span><span class="nt">"ph"</span><span
clas [...]
<span class="p">{</span><span class="w"> </span><span
class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span
class="s2">"decodeShuffleBlock"</span><span class="p">,</span><span class="w">
</span><span class="nt">"cat"</span><span class="p">:</span><span class="w">
</span><span class="s2">"PERF"</span><span class="p">,</span><span class="w">
</span><span class="nt">"ph"</span><span class="p">:</span><span class="w">
</span><span class="s2">"E"</span><span class="p"> [...]
@@ -683,7 +681,7 @@ samples: they are not an atomic per-query balance, and
neither is a measure of R
<td><p>Bytes charged to the Arrow C Data Interface import allocator, a subset
of <code class="docutils literal notranslate"><span
class="pre">jvm_arrow_allocated</span></code>. An allocator charge, not a
measure of where the bytes were allocated; see above.</p></td>
</tr>
<tr class="row-even"><td><p>native_allocated</p></td>
-<td><p>Bytes handed out by the Rust global allocator, process-wide (requires
<code class="docutils literal notranslate"><span
class="pre">alloc-accounting</span></code> feature). Approximate to within 64
KiB of un-flushed delta per live thread.</p></td>
+<td><p>Bytes handed out by the Rust global allocator, process-wide.
Approximate to within 64 KiB of un-flushed delta per live thread.</p></td>
</tr>
<tr class="row-odd"><td><p>comet_memory_reserved_total</p></td>
<td><p>Total memory reserved across every live Comet memory pool,
process-wide, whatever the configured pool type. Counts a pool shared between
execution contexts once, so unlike the per-thread counters it can be compared
directly against an allocation counter.</p></td>
diff --git a/searchindex.js b/searchindex.js
index d94d71f6ae..44c9c4dcc2 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[56, "id1"]], "%": [[54, "id1"]], "&":
[[44, "id1"]], "*": [[54, "id2"]], "+": [[54, "id3"]], "-": [[54, "id4"]], "/":
[[54, "id5"]], "1. Format Your Code": [[41, "format-your-code"]], "1. Install
Comet": [[62, "install-comet"], [73, "install-comet"]], "1. Native Operators
(nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and
Verify": [[41, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[62,
"clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[56, "id1"]], "%": [[54, "id1"]], "&":
[[44, "id1"]], "*": [[54, "id2"]], "+": [[54, "id3"]], "-": [[54, "id4"]], "/":
[[54, "id5"]], "1. Format Your Code": [[41, "format-your-code"]], "1. Install
Comet": [[62, "install-comet"], [73, "install-comet"]], "1. Native Operators
(nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and
Verify": [[41, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[62,
"clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index a4b0a6afa8..c87f847ad2 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -1137,7 +1137,11 @@ shuffle, including applications that rely only on remote
reliable storage or dec
<td><p>The maximum amount of data (in bytes) stored inside the temporary
directories used by native operators when spilling. Applied per Spark task, so
an executor running N concurrent tasks may use up to N times this value on
shared local disks. Once the limit is reached, further spills will fail and the
query will error out.</p></td>
<td><p>107374182400b</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.tracing.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.memory.logInterval</span></code></p></td>
+<td><p>How often each executor logs its native memory usage at INFO level
while Comet native plans are running: the bytes the native allocator has handed
out, and the bytes reserved in Comet’s memory pools. The difference is native
memory that the pools are not accounting for. The executor logs one line per
interval however many tasks are running, and one more after the last plan
finishes. It logs a warning when the native memory looks larger than the
executor’s container allows. This is [...]
+<td><p>10000ms</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.tracing.enabled</span></code></p></td>
<td><p>Enable fine-grained tracing of events and memory usage. For more
information, refer to the <a class="reference external"
href="https://datafusion.apache.org/comet/contributor-guide/tracing.html";>Comet
Tracing Guide</a>.</p></td>
<td><p>false</p></td>
</tr>
diff --git a/user-guide/latest/tuning.html b/user-guide/latest/tuning.html
index cfbaf18085..a85734b8e6 100644
--- a/user-guide/latest/tuning.html
+++ b/user-guide/latest/tuning.html
@@ -849,6 +849,7 @@ free rather than returning to the operating system.</p></li>
</ul>
<p>Reserved memory is therefore a lower bound on what Comet really uses, and
how far below it sits depends on the
workload. This is why Comet can stay within the pool’s limit and still push
the executor past its container limit.
+Each executor logs how far apart the two are while Comet runs; see <a
class="reference internal"
href="#sizing-the-overhead-from-the-memory-usage-log">Sizing the Overhead from
the Memory Usage Log</a>.
To leave room for the part that is not counted, set <code class="docutils
literal notranslate"><span
class="pre">spark.comet.exec.memoryPool.fraction</span></code> to a value less
than
<code class="docutils literal notranslate"><span
class="pre">1.0</span></code>, which restricts the amount of memory Comet is
allowed to reserve.</p>
<p>For more details about Spark off-heap memory mode, please refer to <a
class="reference external"
href="https://spark.apache.org/docs/latest/configuration.html";>Spark
documentation</a>.</p>
@@ -902,10 +903,59 @@ size:</p>
</pre></div>
</div>
<p>Raise the value further if executors are killed by the cluster manager (on
Kubernetes,
-<code class="docutils literal notranslate"><span
class="pre">ExecutorLostFailure</span></code> with exit code 137) rather than
failing with a task-level out-of-memory error.</p>
+<code class="docutils literal notranslate"><span
class="pre">ExecutorLostFailure</span></code> with exit code 137) rather than
failing with a task-level out-of-memory error.
+To measure how much Comet needs rather than guessing, see <a class="reference
internal" href="#sizing-the-overhead-from-the-memory-usage-log">Sizing the
Overhead from the Memory Usage Log</a>.</p>
<p>Note that on Kubernetes and YARN the overhead is added to the container
size, so raising it reduces
how many executors fit on a node.</p>
</section>
+<section id="sizing-the-overhead-from-the-memory-usage-log">
+<h3>Sizing the Overhead from the Memory Usage Log<a class="headerlink"
href="#sizing-the-overhead-from-the-memory-usage-log" title="Link to this
heading">#</a></h3>
+<p>While Comet native plans are running, each executor logs its native memory
usage at INFO level,
+one line every 10 seconds for the whole executor:</p>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="n">Comet</span> <span
class="n">native</span> <span class="n">memory</span> <span
class="n">usage</span><span class="p">:</span> <span class="n">allocated</span>
<span class="mf">5412.3</span> <span class="n">MiB</span><span
class="p">,</span> <span class="n">reserved</span> <span
class="mf">3890.0</span> <span class="n">MiB</span> <span
class="p">(</span><span class="mi">16</span> <span clas [...]
+</pre></div>
+</div>
+<ul class="simple">
+<li><p><code class="docutils literal notranslate"><span
class="pre">allocated</span></code> is the memory that Comet’s native code has
allocated and not yet freed, whether or not
+a pool tracks it.</p></li>
+<li><p><code class="docutils literal notranslate"><span
class="pre">reserved</span></code> is the part that Comet’s memory pools track.
It is charged against
+<code class="docutils literal notranslate"><span
class="pre">spark.memory.offHeap.size</span></code>, so the container already
has room for it.</p></li>
+</ul>
+<p>The difference between the two, <code class="docutils literal
notranslate"><span class="pre">allocated</span> <span class="pre">-</span>
<span class="pre">reserved</span></code>, is Comet’s untracked native memory.
It is
+the part of Comet’s footprint that has to fit in <code class="docutils literal
notranslate"><span class="pre">spark.executor.memoryOverhead</span></code>,
alongside the
+JVM’s own non-heap memory. To size the overhead from it:</p>
+<ol class="arabic simple">
+<li><p>Run a representative workload and find the line with the largest
difference in each executor’s
+log. Take both figures from the same line: they are sampled together, and
figures from different
+lines describe different moments. Setting <code class="docutils literal
notranslate"><span class="pre">spark.comet.memory.logInterval=1s</span></code>
for this run makes a
+short-lived peak less likely to fall between samples.</p></li>
+<li><p>Start from the overhead the executors had before Comet was enabled,
which covers the JVM’s own
+non-heap memory, and add the largest difference seen on any executor.</p></li>
+<li><p>Add a margin on top. The log can miss the true peak between samples,
and neither figure includes
+the allocator’s fragmentation and retained pages, memory allocated by native C
libraries such as
+zstd, or Comet’s Arrow buffers on the JVM side.</p></li>
+</ol>
+<p>For example, a 16 GiB executor derives an overhead of 1638 MiB. If the
largest difference in its
+log is the 1522.3 MiB in the line above, the overhead needs to be at least
1638 + 1523 = 3161 MiB
+before any margin, so <code class="docutils literal notranslate"><span
class="pre">spark.executor.memoryOverhead=4g</span></code> would be a
reasonable setting.</p>
+<p>The executor also logs a warning when its native memory looks larger than
its container allows:
+when the difference, plus everything in use in Spark’s off-heap memory pool
(which includes Comet’s
+reservations), exceeds <code class="docutils literal notranslate"><span
class="pre">spark.memory.offHeap.size</span></code> plus the memory overhead.
This counts the part of
+the off-heap pool that nothing has acquired at that moment, which untracked
memory can occupy until
+Spark hands it out, so a quiet log is not a sign that the overhead is large
enough: size it from the
+largest difference as described above. The overhead also has to hold the JVM’s
own non-heap memory,
+so by the time the warning appears the executor has likely outgrown its
container. It warns the first time this
+happens, and again each time it happens after dropping back below. The
overhead it uses is
+<code class="docutils literal notranslate"><span
class="pre">spark.executor.memoryOverhead</span></code> if set, otherwise <code
class="docutils literal notranslate"><span
class="pre">spark.executor.memoryOverheadFactor</span></code> of
+<code class="docutils literal notranslate"><span
class="pre">spark.executor.memory</span></code> with a minimum of <code
class="docutils literal notranslate"><span
class="pre">spark.executor.minMemoryOverhead</span></code>, as Spark sizes the
+default container. There is no warning in local mode.</p>
+<p>Look more closely before raising the overhead if the difference keeps
growing through a run rather
+than levelling off: native memory that is not being released will exhaust any
overhead eventually.
+The executor logs one more line after its last native plan finishes, and an
<code class="docutils literal notranslate"><span
class="pre">allocated</span></code> figure there
+that grows from one query to the next points the same way.</p>
+<p><code class="docutils literal notranslate"><span
class="pre">spark.comet.memory.logInterval</span></code> is read when an
executor starts its first Comet native plan, so set
+it when the application is submitted. Set it to <code class="docutils literal
notranslate"><span class="pre">0</span></code> to turn the log off.</p>
+</section>
<section id="determining-how-much-memory-to-allocate">
<h3>Determining How Much Memory to Allocate<a class="headerlink"
href="#determining-how-much-memory-to-allocate" title="Link to this
heading">#</a></h3>
<p>Generally, increasing the amount of memory allocated to Comet will improve
query performance by reducing the
@@ -1261,6 +1311,7 @@ fallback to Spark works, see <a class="reference
internal" href="understanding-c
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#memory-tuning">Memory Tuning</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#configuring-comet-memory">Configuring Comet Memory</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#configuring-executor-memory-overhead">Configuring Executor Memory
Overhead</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#sizing-the-overhead-from-the-memory-usage-log">Sizing the Overhead from
the Memory Usage Log</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#determining-how-much-memory-to-allocate">Determining How Much Memory to
Allocate</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#batch-size">Batch Size</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#limiting-spill-disk-usage">Limiting Spill Disk Usage</a></li>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
