This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new f95d4c5273 Publish built docs triggered by
a51ff5cd886328c91bfb8155bca12cb0192ec83c
f95d4c5273 is described below
commit f95d4c52738295f4aadc7ee9d6d41a6f62afb838
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Mon Sep 21 00:30:11 2026 +0000
Publish built docs triggered by a51ff5cd886328c91bfb8155bca12cb0192ec83c
---
_sources/contributor-guide/tracing.md.txt | 85 ++++++++++++++++++-------------
contributor-guide/tracing.html | 73 +++++++++++++++-----------
searchindex.js | 2 +-
3 files changed, 95 insertions(+), 65 deletions(-)
diff --git a/_sources/contributor-guide/tracing.md.txt
b/_sources/contributor-guide/tracing.md.txt
index bc5690bc59..4e7b82b914 100644
--- a/_sources/contributor-guide/tracing.md.txt
+++ b/_sources/contributor-guide/tracing.md.txt
@@ -66,7 +66,7 @@ Example trace visualization:
## Analyzing Memory Usage
The `analyze_trace` tool parses a trace log and compares the process-wide
native allocation counter against
-the sum of per-thread Comet memory pool reservations. This is useful for
detecting untracked native memory
+the total memory reserved by Comet's memory pools. This is useful for
detecting untracked native memory
growth where native allocations exceed what the memory pools account for.
Build and run:
@@ -78,46 +78,60 @@ cargo run --bin analyze_trace --
/path/to/comet-event-trace.json
The tool reads counter events from the trace log. Because tracing logs metrics
per thread, `native_allocated`
and `jemalloc_allocated` are process-wide values (the same global allocation
reported from whichever thread
-logs it), while `thread_NNN_comet_memory_reserved` values are per-thread pool
reservations that are summed to
-get the total tracked memory. The tool analyzes `native_allocated` when the
trace contains it, since that
-counts only what Rust code holds from the allocator, and otherwise falls back
to `jemalloc_allocated`. The
-output names the counter it used. A trace with neither counter is rejected.
+logs it). The total tracked memory comes from `comet_memory_reserved_total`,
which covers every pool type and
+counts each pool once process-wide. Every plan registers its pool for this
total, including plans running
+with tracing disabled, because the allocation counter it is compared against
covers them too. The per-thread
+`thread_NNN_comet_memory_reserved` values must not be summed to obtain it: a
shared pool reports its full
+reservation on every thread that references it, so summing multiplies it by
the thread count.
-Sample output:
+The allocation counter and the total are emitted back to back on one thread
when a traced plan finishes
+executing, and the tool compares only samples paired that way, so a fresh
allocation is never measured
+against a stale reservation. Traces recorded before that counter existed are
still analyzed from the
+per-thread sum, and the tool warns that the sum is not the same measure.
+
+The tool analyzes `native_allocated` when the trace contains it, since that
counts only what Rust code holds
+from the allocator, and otherwise falls back to `jemalloc_allocated`. The
output names the counter it used. A
+trace with neither counter is rejected.
+
+Sample output, with the violation table and the per-thread list elided:
```
=== Comet Trace Memory Analysis ===
-Counter events parsed: 193104
-Allocation counter: jemalloc_allocated
-Threads with memory pools: 8
-Peak jemalloc_allocated: 3068.2 MB
-Peak pool total: 2864.6 MB
-Peak excess (jemalloc_allocated - pool): 364.6 MB
+Counter events parsed: 2946
+Allocation counter: native_allocated
+Pool total source: comet_memory_reserved_total (process-wide)
+Peak native_allocated: 395.9 MB
+Peak pool total: 250.9 MB
+Peak excess (native_allocated - pool): 171.2 MB
-WARNING: jemalloc_allocated exceeded pool reservation at 138 sampled points:
+WARNING: native_allocated exceeded pool reservation at 87 sampled points:
- Time (us) jemalloc_allocated pool_total excess
+ Time (us) native_allocated pool_total excess
------------------------------------------------------------------
- 179578 210.8 MB 0.1 MB 210.7 MB
- 429663 420.5 MB 145.1 MB 275.5 MB
- 1304969 2122.5 MB 1797.2 MB 325.2 MB
- 21974838 407.0 MB 42.3 MB 364.6 MB
- 33543599 5.5 MB 0.1 MB 5.3 MB
+ 14662 19.0 MB 0.7 MB 18.3 MB
+ 109895 48.2 MB 22.4 MB 25.8 MB
+ 1623833 94.0 MB 71.4 MB 22.6 MB
+ 3315951 193.8 MB 167.8 MB 26.0 MB
+ 4107671 233.7 MB 199.9 MB 33.8 MB
+ ...
+ 5441547 389.3 MB 218.1 MB 171.2 MB
+ 6445070 235.6 MB 209.8 MB 25.8 MB
--- Final per-thread pool reservations ---
- thread_60_comet_memory_reserved: 0.0 MB
- thread_95_comet_memory_reserved: 0.0 MB
- thread_96_comet_memory_reserved: 0.0 MB
+ thread_60_comet_memory_reserved: 39.2 MB
+ thread_61_comet_memory_reserved: 48.3 MB
+ thread_62_comet_memory_reserved: 27.2 MB
...
- Total: 0.0 MB
+ Total: 245.8 MB
```
-Some excess is expected (allocator metadata and fragmentation for
`jemalloc_allocated`, and non-pool
-allocations like Arrow IPC buffers for either counter). Large or growing
excess may indicate memory that is
-not being tracked by the pool.
+A steady excess is expected, since not every native allocation goes through a
pool (Arrow IPC buffers, for
+instance), and `jemalloc_allocated` additionally includes the allocator's own
metadata and fragmentation. An
+excess that grows over the run, as it does from 26 MB to 171 MB above, is the
signal worth chasing: that is
+memory the pool is not accounting for.
Arrow memory on the JVM side is reported separately, because it is off-heap
and so invisible to
`jvm_heap_used`. Comet imports batches from native over the Arrow C Data
Interface, and Arrow
@@ -145,12 +159,13 @@ samples: they are not an atomic per-query balance, and
neither is a measure of R
## Definition of Labels
-| Label | Meaning
|
-| -------------------------------- |
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
-| jvm_heap_used | JVM heap memory usage of live objects for
the executor process
|
-| jemalloc_allocated | Native memory usage for the executor
process (requires `jemalloc` feature)
|
-| jvm_arrow_allocated | Bytes charged to Comet's Arrow allocator
tree on the JVM, including buffers imported from native over the Arrow C Data
Interface |
-| jvm_arrow_imported | Bytes charged to the Arrow C Data
Interface import allocator, a subset of `jvm_arrow_allocated`. An allocator
charge, not a measure of where the bytes were allocated; see above. |
-| native_allocated | Bytes handed out by the Rust global
allocator, process-wide (requires `alloc-accounting` feature). Approximate to
within 64 KiB of un-flushed delta per live thread. |
-| thread_NNN_comet_memory_reserved | Memory reserved by Comet's DataFusion
memory pool (summed across all contexts on the thread). NNN is the Rust thread
ID. |
-| thread_NNN_comet_jvm_shuffle | Off-heap memory allocated by Comet for
columnar shuffle. NNN is the Rust thread ID.
|
+| Label | Meaning
|
+| -------------------------------- |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
+| jvm_heap_used | JVM heap memory usage of live objects for
the executor process
|
+| jemalloc_allocated | Native memory usage for the executor
process (requires `jemalloc` feature)
|
+| jvm_arrow_allocated | Bytes charged to Comet's Arrow allocator
tree on the JVM, including buffers imported from native over the Arrow C Data
Interface
|
+| jvm_arrow_imported | Bytes charged to the Arrow C Data
Interface import allocator, a subset of `jvm_arrow_allocated`. An allocator
charge, not a measure of where the bytes were allocated; see above.
|
+| native_allocated | Bytes handed out by the Rust global
allocator, process-wide (requires `alloc-accounting` feature). Approximate to
within 64 KiB of un-flushed delta per live thread.
|
+| comet_memory_reserved_total | Total memory reserved across every live
Comet memory pool, process-wide, whatever the configured pool type. Counts a
pool shared between execution contexts once, so unlike the per-thread counters
it can be compared directly against an allocation counter. |
+| thread_NNN_comet_memory_reserved | Memory reserved by Comet's DataFusion
memory pool (summed across all contexts on the thread). NNN is the Rust thread
ID. Do not sum these across threads: a shared pool reports its full reservation
on every thread that references it. |
+| thread_NNN_comet_jvm_shuffle | Off-heap memory allocated by Comet for
columnar shuffle. NNN is the Rust thread ID.
|
diff --git a/contributor-guide/tracing.html b/contributor-guide/tracing.html
index fdf64af12a..09832a6edd 100644
--- a/contributor-guide/tracing.html
+++ b/contributor-guide/tracing.html
@@ -580,7 +580,7 @@ allocator’s own caching in the way. The two features are
independent and can b
<section id="analyzing-memory-usage">
<h2>Analyzing Memory Usage<a class="headerlink" href="#analyzing-memory-usage"
title="Link to this heading">#</a></h2>
<p>The <code class="docutils literal notranslate"><span
class="pre">analyze_trace</span></code> tool parses a trace log and compares
the process-wide native allocation counter against
-the sum of per-thread Comet memory pool reservations. This is useful for
detecting untracked native memory
+the total memory reserved by Comet’s memory pools. This is useful for
detecting untracked native memory
growth where native allocations exceed what the memory pools account for.</p>
<p>Build and run:</p>
<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span><span class="nb">cd</span><span class="w">
</span>native
@@ -589,43 +589,55 @@ cargo<span class="w"> </span>run<span class="w">
</span>--bin<span class="w"> </
</div>
<p>The tool reads counter events from the trace log. Because tracing logs
metrics per thread, <code class="docutils literal notranslate"><span
class="pre">native_allocated</span></code>
and <code class="docutils literal notranslate"><span
class="pre">jemalloc_allocated</span></code> are process-wide values (the same
global allocation reported from whichever thread
-logs it), while <code class="docutils literal notranslate"><span
class="pre">thread_NNN_comet_memory_reserved</span></code> values are
per-thread pool reservations that are summed to
-get the total tracked memory. The tool analyzes <code class="docutils literal
notranslate"><span class="pre">native_allocated</span></code> when the trace
contains it, since that
-counts only what Rust code holds from the allocator, and otherwise falls back
to <code class="docutils literal notranslate"><span
class="pre">jemalloc_allocated</span></code>. The
-output names the counter it used. A trace with neither counter is rejected.</p>
-<p>Sample output:</p>
+logs it). The total tracked memory comes from <code class="docutils literal
notranslate"><span class="pre">comet_memory_reserved_total</span></code>, which
covers every pool type and
+counts each pool once process-wide. Every plan registers its pool for this
total, including plans running
+with tracing disabled, because the allocation counter it is compared against
covers them too. The per-thread
+<code class="docutils literal notranslate"><span
class="pre">thread_NNN_comet_memory_reserved</span></code> values must not be
summed to obtain it: a shared pool reports its full
+reservation on every thread that references it, so summing multiplies it by
the thread count.</p>
+<p>The allocation counter and the total are emitted back to back on one thread
when a traced plan finishes
+executing, and the tool compares only samples paired that way, so a fresh
allocation is never measured
+against a stale reservation. Traces recorded before that counter existed are
still analyzed from the
+per-thread sum, and the tool warns that the sum is not the same measure.</p>
+<p>The tool analyzes <code class="docutils literal notranslate"><span
class="pre">native_allocated</span></code> when the trace contains it, since
that counts only what Rust code holds
+from the allocator, and otherwise falls back to <code class="docutils literal
notranslate"><span class="pre">jemalloc_allocated</span></code>. The output
names the counter it used. A
+trace with neither counter is rejected.</p>
+<p>Sample output, with the violation table and the per-thread list elided:</p>
<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="o">===</span> <span
class="n">Comet</span> <span class="n">Trace</span> <span
class="n">Memory</span> <span class="n">Analysis</span> <span
class="o">===</span>
-<span class="n">Counter</span> <span class="n">events</span> <span
class="n">parsed</span><span class="p">:</span> <span class="mi">193104</span>
-<span class="n">Allocation</span> <span class="n">counter</span><span
class="p">:</span> <span class="n">jemalloc_allocated</span>
-<span class="n">Threads</span> <span class="k">with</span> <span
class="n">memory</span> <span class="n">pools</span><span class="p">:</span>
<span class="mi">8</span>
-<span class="n">Peak</span> <span class="n">jemalloc_allocated</span><span
class="p">:</span> <span class="mf">3068.2</span> <span class="n">MB</span>
-<span class="n">Peak</span> <span class="n">pool</span> <span
class="n">total</span><span class="p">:</span> <span
class="mf">2864.6</span> <span class="n">MB</span>
-<span class="n">Peak</span> <span class="n">excess</span> <span
class="p">(</span><span class="n">jemalloc_allocated</span> <span
class="o">-</span> <span class="n">pool</span><span class="p">):</span> <span
class="mf">364.6</span> <span class="n">MB</span>
+<span class="n">Counter</span> <span class="n">events</span> <span
class="n">parsed</span><span class="p">:</span> <span class="mi">2946</span>
+<span class="n">Allocation</span> <span class="n">counter</span><span
class="p">:</span> <span class="n">native_allocated</span>
+<span class="n">Pool</span> <span class="n">total</span> <span
class="n">source</span><span class="p">:</span> <span
class="n">comet_memory_reserved_total</span> <span class="p">(</span><span
class="n">process</span><span class="o">-</span><span
class="n">wide</span><span class="p">)</span>
+<span class="n">Peak</span> <span class="n">native_allocated</span><span
class="p">:</span> <span class="mf">395.9</span> <span class="n">MB</span>
+<span class="n">Peak</span> <span class="n">pool</span> <span
class="n">total</span><span class="p">:</span> <span
class="mf">250.9</span> <span class="n">MB</span>
+<span class="n">Peak</span> <span class="n">excess</span> <span
class="p">(</span><span class="n">native_allocated</span> <span
class="o">-</span> <span class="n">pool</span><span class="p">):</span> <span
class="mf">171.2</span> <span class="n">MB</span>
-<span class="n">WARNING</span><span class="p">:</span> <span
class="n">jemalloc_allocated</span> <span class="n">exceeded</span> <span
class="n">pool</span> <span class="n">reservation</span> <span
class="n">at</span> <span class="mi">138</span> <span class="n">sampled</span>
<span class="n">points</span><span class="p">:</span>
+<span class="n">WARNING</span><span class="p">:</span> <span
class="n">native_allocated</span> <span class="n">exceeded</span> <span
class="n">pool</span> <span class="n">reservation</span> <span
class="n">at</span> <span class="mi">87</span> <span class="n">sampled</span>
<span class="n">points</span><span class="p">:</span>
- <span class="n">Time</span> <span class="p">(</span><span
class="n">us</span><span class="p">)</span> <span
class="n">jemalloc_allocated</span> <span class="n">pool_total</span>
<span class="n">excess</span>
+ <span class="n">Time</span> <span class="p">(</span><span
class="n">us</span><span class="p">)</span> <span
class="n">native_allocated</span> <span class="n">pool_total</span>
<span class="n">excess</span>
<span
class="o">------------------------------------------------------------------</span>
- <span class="mi">179578</span> <span
class="mf">210.8</span> <span class="n">MB</span> <span
class="mf">0.1</span> <span class="n">MB</span> <span
class="mf">210.7</span> <span class="n">MB</span>
- <span class="mi">429663</span> <span
class="mf">420.5</span> <span class="n">MB</span> <span
class="mf">145.1</span> <span class="n">MB</span> <span
class="mf">275.5</span> <span class="n">MB</span>
- <span class="mi">1304969</span> <span
class="mf">2122.5</span> <span class="n">MB</span> <span
class="mf">1797.2</span> <span class="n">MB</span> <span
class="mf">325.2</span> <span class="n">MB</span>
- <span class="mi">21974838</span> <span
class="mf">407.0</span> <span class="n">MB</span> <span
class="mf">42.3</span> <span class="n">MB</span> <span
class="mf">364.6</span> <span class="n">MB</span>
- <span class="mi">33543599</span> <span
class="mf">5.5</span> <span class="n">MB</span> <span
class="mf">0.1</span> <span class="n">MB</span> <span
class="mf">5.3</span> <span class="n">MB</span>
+ <span class="mi">14662</span> <span
class="mf">19.0</span> <span class="n">MB</span> <span
class="mf">0.7</span> <span class="n">MB</span> <span
class="mf">18.3</span> <span class="n">MB</span>
+ <span class="mi">109895</span> <span
class="mf">48.2</span> <span class="n">MB</span> <span
class="mf">22.4</span> <span class="n">MB</span> <span
class="mf">25.8</span> <span class="n">MB</span>
+ <span class="mi">1623833</span> <span
class="mf">94.0</span> <span class="n">MB</span> <span
class="mf">71.4</span> <span class="n">MB</span> <span
class="mf">22.6</span> <span class="n">MB</span>
+ <span class="mi">3315951</span> <span
class="mf">193.8</span> <span class="n">MB</span> <span
class="mf">167.8</span> <span class="n">MB</span> <span
class="mf">26.0</span> <span class="n">MB</span>
+ <span class="mi">4107671</span> <span
class="mf">233.7</span> <span class="n">MB</span> <span
class="mf">199.9</span> <span class="n">MB</span> <span
class="mf">33.8</span> <span class="n">MB</span>
+ <span class="o">...</span>
+ <span class="mi">5441547</span> <span
class="mf">389.3</span> <span class="n">MB</span> <span
class="mf">218.1</span> <span class="n">MB</span> <span
class="mf">171.2</span> <span class="n">MB</span>
+ <span class="mi">6445070</span> <span
class="mf">235.6</span> <span class="n">MB</span> <span
class="mf">209.8</span> <span class="n">MB</span> <span
class="mf">25.8</span> <span class="n">MB</span>
<span class="o">---</span> <span class="n">Final</span> <span
class="n">per</span><span class="o">-</span><span class="n">thread</span> <span
class="n">pool</span> <span class="n">reservations</span> <span
class="o">---</span>
- <span class="n">thread_60_comet_memory_reserved</span><span
class="p">:</span> <span class="mf">0.0</span> <span class="n">MB</span>
- <span class="n">thread_95_comet_memory_reserved</span><span
class="p">:</span> <span class="mf">0.0</span> <span class="n">MB</span>
- <span class="n">thread_96_comet_memory_reserved</span><span
class="p">:</span> <span class="mf">0.0</span> <span class="n">MB</span>
+ <span class="n">thread_60_comet_memory_reserved</span><span
class="p">:</span> <span class="mf">39.2</span> <span class="n">MB</span>
+ <span class="n">thread_61_comet_memory_reserved</span><span
class="p">:</span> <span class="mf">48.3</span> <span class="n">MB</span>
+ <span class="n">thread_62_comet_memory_reserved</span><span
class="p">:</span> <span class="mf">27.2</span> <span class="n">MB</span>
<span class="o">...</span>
- <span class="n">Total</span><span class="p">:</span> <span
class="mf">0.0</span> <span class="n">MB</span>
+ <span class="n">Total</span><span class="p">:</span> <span
class="mf">245.8</span> <span class="n">MB</span>
</pre></div>
</div>
-<p>Some excess is expected (allocator metadata and fragmentation for <code
class="docutils literal notranslate"><span
class="pre">jemalloc_allocated</span></code>, and non-pool
-allocations like Arrow IPC buffers for either counter). Large or growing
excess may indicate memory that is
-not being tracked by the pool.</p>
+<p>A steady excess is expected, since not every native allocation goes through
a pool (Arrow IPC buffers, for
+instance), and <code class="docutils literal notranslate"><span
class="pre">jemalloc_allocated</span></code> additionally includes the
allocator’s own metadata and fragmentation. An
+excess that grows over the run, as it does from 26 MB to 171 MB above, is the
signal worth chasing: that is
+memory the pool is not accounting for.</p>
<p>Arrow memory on the JVM side is reported separately, because it is off-heap
and so invisible to
<code class="docutils literal notranslate"><span
class="pre">jvm_heap_used</span></code>. Comet imports batches from native over
the Arrow C Data Interface, and Arrow
charges a buffer to whichever allocator owns it, so those imports are taken
against a dedicated
@@ -673,10 +685,13 @@ samples: they are not an atomic per-query balance, and
neither is a measure of R
<tr class="row-even"><td><p>native_allocated</p></td>
<td><p>Bytes handed out by the Rust global allocator, process-wide (requires
<code class="docutils literal notranslate"><span
class="pre">alloc-accounting</span></code> feature). Approximate to within 64
KiB of un-flushed delta per live thread.</p></td>
</tr>
-<tr class="row-odd"><td><p>thread_NNN_comet_memory_reserved</p></td>
-<td><p>Memory reserved by Comet’s DataFusion memory pool (summed across all
contexts on the thread). NNN is the Rust thread ID.</p></td>
+<tr class="row-odd"><td><p>comet_memory_reserved_total</p></td>
+<td><p>Total memory reserved across every live Comet memory pool,
process-wide, whatever the configured pool type. Counts a pool shared between
execution contexts once, so unlike the per-thread counters it can be compared
directly against an allocation counter.</p></td>
</tr>
-<tr class="row-even"><td><p>thread_NNN_comet_jvm_shuffle</p></td>
+<tr class="row-even"><td><p>thread_NNN_comet_memory_reserved</p></td>
+<td><p>Memory reserved by Comet’s DataFusion memory pool (summed across all
contexts on the thread). NNN is the Rust thread ID. Do not sum these across
threads: a shared pool reports its full reservation on every thread that
references it.</p></td>
+</tr>
+<tr class="row-odd"><td><p>thread_NNN_comet_jvm_shuffle</p></td>
<td><p>Off-heap memory allocated by Comet for columnar shuffle. NNN is the
Rust thread ID.</p></td>
</tr>
</tbody>
diff --git a/searchindex.js b/searchindex.js
index f98c66382c..00d6ca6fac 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[56, "id1"]], "%": [[54, "id1"]], "&":
[[44, "id1"]], "*": [[54, "id2"]], "+": [[54, "id3"]], "-": [[54, "id4"]], "/":
[[54, "id5"]], "1. Format Your Code": [[41, "format-your-code"]], "1. Install
Comet": [[62, "install-comet"], [73, "install-comet"]], "1. Native Operators
(nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and
Verify": [[41, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[62,
"clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[56, "id1"]], "%": [[54, "id1"]], "&":
[[44, "id1"]], "*": [[54, "id2"]], "+": [[54, "id3"]], "-": [[54, "id4"]], "/":
[[54, "id5"]], "1. Format Your Code": [[41, "format-your-code"]], "1. Install
Comet": [[62, "install-comet"], [73, "install-comet"]], "1. Native Operators
(nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and
Verify": [[41, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[62,
"clone-iceberg-and-apply- [...]
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
