This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 453bb46fbc Publish built docs triggered by
32fae06d24a1489a1b1d729372585211b4f64185
453bb46fbc is described below
commit 453bb46fbc89a7d88ddea1a1ef38ed520a25954d
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Sun Sep 20 17:38:40 2026 +0000
Publish built docs triggered by 32fae06d24a1489a1b1d729372585211b4f64185
---
.../contributor-guide/memory_management.md.txt | 8 +++++
_sources/contributor-guide/tracing.md.txt | 40 ++++++++++++++++++----
contributor-guide/memory_management.html | 7 ++++
contributor-guide/tracing.html | 28 +++++++++++++++
searchindex.js | 2 +-
5 files changed, 77 insertions(+), 8 deletions(-)
diff --git a/_sources/contributor-guide/memory_management.md.txt
b/_sources/contributor-guide/memory_management.md.txt
index 91199dec35..e7916f9c78 100644
--- a/_sources/contributor-guide/memory_management.md.txt
+++ b/_sources/contributor-guide/memory_management.md.txt
@@ -103,6 +103,14 @@ off-heap bytes in container RSS that neither Spark's
`TaskMemoryManager` nor Com
pool sees. In practice the volume is modest, a batch at a time per stream, but
there is no
ceiling and no backpressure.
+One further child, `CometArrowImportAllocator` (`comet-ffi-imports`), is what
the Arrow C Data
+Interface import path allocates from, so that tracing can report those charges
apart from the rest
+of Comet's Arrow memory. Unlike the others it is process-wide and never
closed, because imported
+buffers are reference counted and routinely outlive the task that imported
them. Its reservation
+is zero, so every byte still escalates to the root and the inventory above is
unchanged by it.
+Being charged there is not the same as having been allocated there; see the
scaladoc on the
+allocator before reading anything into the split.
+
**The JVM shuffle allocator is an ordinary Spark consumer.**
`CometShuffleMemoryAllocator.getInstance`
returns `CometUnifiedShuffleMemoryAllocator`, a Spark `MemoryConsumer` drawing
from
`spark.memory.offHeap.size`, so shuffle pages are arbitrated against Spark's
other consumers in the
diff --git a/_sources/contributor-guide/tracing.md.txt
b/_sources/contributor-guide/tracing.md.txt
index 9757bd8a57..bc5690bc59 100644
--- a/_sources/contributor-guide/tracing.md.txt
+++ b/_sources/contributor-guide/tracing.md.txt
@@ -119,12 +119,38 @@ Some excess is expected (allocator metadata and
fragmentation for `jemalloc_allo
allocations like Arrow IPC buffers for either counter). Large or growing
excess may indicate memory that is
not being tracked by the pool.
+Arrow memory on the JVM side is reported separately, because it is off-heap
and so invisible to
+`jvm_heap_used`. Comet imports batches from native over the Arrow C Data
Interface, and Arrow
+charges a buffer to whichever allocator owns it, so those imports are taken
against a dedicated
+child allocator and reported as `jvm_arrow_imported`, within the
`jvm_arrow_allocated` total.
+
+Both are allocator charges. They report what each allocator is accountable
for, not where the bytes
+were allocated, and their difference is not a bound on the Arrow memory the
JVM allocated itself.
+Ownership and allocation come apart in both directions:
+
+- Bytes the JVM allocated get charged to the import allocator. Arrow's
importer allocates the
+ owning `ArrowArray` struct there, and `BitVectorHelper.loadValidityBuffer`
allocates a validity
+ bitmap there when an imported vector is all-valid or all-null and carries no
validity buffer
+ (512 bytes per 4096 rows).
+- Imported bytes get charged to the root. An ownership transfer re-parents a
charge without moving
+ the payload, so a vector that shares buffers with an import, such as a slice
of a UDF input, can
+ leave the root accountable for memory the producer allocated.
+
+For the same reason neither counter is a count of unique physical bytes, and
`jvm_arrow_imported`
+is not guaranteed to be included in `native_allocated`. Usually the producer
is Rust and the bytes
+are counted in both, but a buffer that Comet exported to native and that
native passed back by
+reference is imported without the Rust allocator ever having handed it out.
Finally, the two
+counters are separate reads of process-wide state, so concurrent tasks can
change them between
+samples: they are not an atomic per-query balance, and neither is a measure of
RSS.
+
## Definition of Labels
-| Label | Meaning
|
-| -------------------------------- |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
-| jvm_heap_used | JVM heap memory usage of live objects for
the executor process
|
-| jemalloc_allocated | Native memory usage for the executor
process (requires `jemalloc` feature)
|
-| native_allocated | Bytes handed out by the Rust global
allocator, process-wide (requires `alloc-accounting` feature). Approximate to
within 64 KiB of un-flushed delta per live thread. |
-| thread_NNN_comet_memory_reserved | Memory reserved by Comet's DataFusion
memory pool (summed across all contexts on the thread). NNN is the Rust thread
ID. |
-| thread_NNN_comet_jvm_shuffle | Off-heap memory allocated by Comet for
columnar shuffle. NNN is the Rust thread ID.
|
+| Label | Meaning
|
+| -------------------------------- |
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
+| jvm_heap_used | JVM heap memory usage of live objects for
the executor process
|
+| jemalloc_allocated | Native memory usage for the executor
process (requires `jemalloc` feature)
|
+| jvm_arrow_allocated | Bytes charged to Comet's Arrow allocator
tree on the JVM, including buffers imported from native over the Arrow C Data
Interface |
+| jvm_arrow_imported | Bytes charged to the Arrow C Data
Interface import allocator, a subset of `jvm_arrow_allocated`. An allocator
charge, not a measure of where the bytes were allocated; see above. |
+| native_allocated | Bytes handed out by the Rust global
allocator, process-wide (requires `alloc-accounting` feature). Approximate to
within 64 KiB of un-flushed delta per live thread. |
+| thread_NNN_comet_memory_reserved | Memory reserved by Comet's DataFusion
memory pool (summed across all contexts on the thread). NNN is the Rust thread
ID. |
+| thread_NNN_comet_jvm_shuffle | Off-heap memory allocated by Comet for
columnar shuffle. NNN is the Rust thread ID.
|
diff --git a/contributor-guide/memory_management.html
b/contributor-guide/memory_management.html
index cdfcfb9490..0452b0156d 100644
--- a/contributor-guide/memory_management.html
+++ b/contributor-guide/memory_management.html
@@ -669,6 +669,13 @@ for both at once, and raises the pod’s memory request by
the same amount.</p>
off-heap bytes in container RSS that neither Spark’s <code class="docutils
literal notranslate"><span class="pre">TaskMemoryManager</span></code> nor
Comet’s native memory
pool sees. In practice the volume is modest, a batch at a time per stream, but
there is no
ceiling and no backpressure.</p>
+<p>One further child, <code class="docutils literal notranslate"><span
class="pre">CometArrowImportAllocator</span></code> (<code class="docutils
literal notranslate"><span class="pre">comet-ffi-imports</span></code>), is
what the Arrow C Data
+Interface import path allocates from, so that tracing can report those charges
apart from the rest
+of Comet’s Arrow memory. Unlike the others it is process-wide and never
closed, because imported
+buffers are reference counted and routinely outlive the task that imported
them. Its reservation
+is zero, so every byte still escalates to the root and the inventory above is
unchanged by it.
+Being charged there is not the same as having been allocated there; see the
scaladoc on the
+allocator before reading anything into the split.</p>
<p><strong>The JVM shuffle allocator is an ordinary Spark consumer.</strong>
<code class="docutils literal notranslate"><span
class="pre">CometShuffleMemoryAllocator.getInstance</span></code>
returns <code class="docutils literal notranslate"><span
class="pre">CometUnifiedShuffleMemoryAllocator</span></code>, a Spark <code
class="docutils literal notranslate"><span
class="pre">MemoryConsumer</span></code> drawing from
<code class="docutils literal notranslate"><span
class="pre">spark.memory.offHeap.size</span></code>, so shuffle pages are
arbitrated against Spark’s other consumers in the
diff --git a/contributor-guide/tracing.html b/contributor-guide/tracing.html
index e0d07cd45f..fdf64af12a 100644
--- a/contributor-guide/tracing.html
+++ b/contributor-guide/tracing.html
@@ -626,6 +626,28 @@ output names the counter it used. A trace with neither
counter is rejected.</p>
<p>Some excess is expected (allocator metadata and fragmentation for <code
class="docutils literal notranslate"><span
class="pre">jemalloc_allocated</span></code>, and non-pool
allocations like Arrow IPC buffers for either counter). Large or growing
excess may indicate memory that is
not being tracked by the pool.</p>
+<p>Arrow memory on the JVM side is reported separately, because it is off-heap
and so invisible to
+<code class="docutils literal notranslate"><span
class="pre">jvm_heap_used</span></code>. Comet imports batches from native over
the Arrow C Data Interface, and Arrow
+charges a buffer to whichever allocator owns it, so those imports are taken
against a dedicated
+child allocator and reported as <code class="docutils literal
notranslate"><span class="pre">jvm_arrow_imported</span></code>, within the
<code class="docutils literal notranslate"><span
class="pre">jvm_arrow_allocated</span></code> total.</p>
+<p>Both are allocator charges. They report what each allocator is accountable
for, not where the bytes
+were allocated, and their difference is not a bound on the Arrow memory the
JVM allocated itself.
+Ownership and allocation come apart in both directions:</p>
+<ul class="simple">
+<li><p>Bytes the JVM allocated get charged to the import allocator. Arrow’s
importer allocates the
+owning <code class="docutils literal notranslate"><span
class="pre">ArrowArray</span></code> struct there, and <code class="docutils
literal notranslate"><span
class="pre">BitVectorHelper.loadValidityBuffer</span></code> allocates a
validity
+bitmap there when an imported vector is all-valid or all-null and carries no
validity buffer
+(512 bytes per 4096 rows).</p></li>
+<li><p>Imported bytes get charged to the root. An ownership transfer
re-parents a charge without moving
+the payload, so a vector that shares buffers with an import, such as a slice
of a UDF input, can
+leave the root accountable for memory the producer allocated.</p></li>
+</ul>
+<p>For the same reason neither counter is a count of unique physical bytes,
and <code class="docutils literal notranslate"><span
class="pre">jvm_arrow_imported</span></code>
+is not guaranteed to be included in <code class="docutils literal
notranslate"><span class="pre">native_allocated</span></code>. Usually the
producer is Rust and the bytes
+are counted in both, but a buffer that Comet exported to native and that
native passed back by
+reference is imported without the Rust allocator ever having handed it out.
Finally, the two
+counters are separate reads of process-wide state, so concurrent tasks can
change them between
+samples: they are not an atomic per-query balance, and neither is a measure of
RSS.</p>
</section>
<section id="definition-of-labels">
<h2>Definition of Labels<a class="headerlink" href="#definition-of-labels"
title="Link to this heading">#</a></h2>
@@ -642,6 +664,12 @@ not being tracked by the pool.</p>
<tr class="row-odd"><td><p>jemalloc_allocated</p></td>
<td><p>Native memory usage for the executor process (requires <code
class="docutils literal notranslate"><span class="pre">jemalloc</span></code>
feature)</p></td>
</tr>
+<tr class="row-even"><td><p>jvm_arrow_allocated</p></td>
+<td><p>Bytes charged to Comet’s Arrow allocator tree on the JVM, including
buffers imported from native over the Arrow C Data Interface</p></td>
+</tr>
+<tr class="row-odd"><td><p>jvm_arrow_imported</p></td>
+<td><p>Bytes charged to the Arrow C Data Interface import allocator, a subset
of <code class="docutils literal notranslate"><span
class="pre">jvm_arrow_allocated</span></code>. An allocator charge, not a
measure of where the bytes were allocated; see above.</p></td>
+</tr>
<tr class="row-even"><td><p>native_allocated</p></td>
<td><p>Bytes handed out by the Rust global allocator, process-wide (requires
<code class="docutils literal notranslate"><span
class="pre">alloc-accounting</span></code> feature). Approximate to within 64
KiB of un-flushed delta per live thread.</p></td>
</tr>
diff --git a/searchindex.js b/searchindex.js
index 55f2d6fb31..f98c66382c 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[56, "id1"]], "%": [[54, "id1"]], "&":
[[44, "id1"]], "*": [[54, "id2"]], "+": [[54, "id3"]], "-": [[54, "id4"]], "/":
[[54, "id5"]], "1. Format Your Code": [[41, "format-your-code"]], "1. Install
Comet": [[62, "install-comet"], [73, "install-comet"]], "1. Native Operators
(nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and
Verify": [[41, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[62,
"clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[56, "id1"]], "%": [[54, "id1"]], "&":
[[44, "id1"]], "*": [[54, "id2"]], "+": [[54, "id3"]], "-": [[54, "id4"]], "/":
[[54, "id5"]], "1. Format Your Code": [[41, "format-your-code"]], "1. Install
Comet": [[62, "install-comet"], [73, "install-comet"]], "1. Native Operators
(nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and
Verify": [[41, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[62,
"clone-iceberg-and-apply- [...]
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
