This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 7e7766a081 Publish built docs triggered by
5ca149928f7743bfe7a96feadea5e0f9bed1412f
7e7766a081 is described below
commit 7e7766a0814465a46079fe27090697382b0a7487
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Mon Sep 21 06:54:16 2026 +0000
Publish built docs triggered by 5ca149928f7743bfe7a96feadea5e0f9bed1412f
---
.buildinfo | 2 +-
...id-013012468fa27c9e9fdb0a14196d357c619c68a8.svg | 1 +
...id-24951e49e88a2a575316ecedeb5309d49a140d7a.svg | 1 +
...id-d0bb31b07009ce49e46219a49cdd8e6531dd24ce.svg | 1 +
contributor-guide/ci.html | 16 ++++++++
contributor-guide/memory_management.html | 47 ++++++++++++++++++++++
6 files changed, 67 insertions(+), 1 deletion(-)
diff --git a/.buildinfo b/.buildinfo
index c2bc25f3aa..b2a695f0bf 100644
--- a/.buildinfo
+++ b/.buildinfo
@@ -1,4 +1,4 @@
# Sphinx build info version 1
# This file hashes the configuration used when building these files. When it
is not found, a full rebuild will be done.
-config: e33ca1ce50b275ab64dabb2fb0347e98
+config: eb8ea1a0b32c78350beb1996fa5f6c03
tags: 645f666f9bcd5a90fca523b33c5a78b7
diff --git a/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg
b/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg
new file mode 100644
index 0000000000..6bd32b44e8
--- /dev/null
+++ b/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg
@@ -0,0 +1 @@
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg";
xmlns:xlink="http://www.w3.org/1999/xlink"; class="flowchart" style="max-width:
511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 942"
role="graphics-document document"
aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet
ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes
edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes
dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
diff --git a/_images/mermaid-24951e49e88a2a575316ecedeb5309d49a140d7a.svg
b/_images/mermaid-24951e49e88a2a575316ecedeb5309d49a140d7a.svg
new file mode 100644
index 0000000000..92fc4294f2
--- /dev/null
+++ b/_images/mermaid-24951e49e88a2a575316ecedeb5309d49a140d7a.svg
@@ -0,0 +1 @@
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg";
xmlns:xlink="http://www.w3.org/1999/xlink"; class="flowchart" style="max-width:
1170.02px; background-color: transparent;" viewBox="0 0 1170.015625 1229"
role="graphics-document document"
aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet
ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes
edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes
dash{to{stroke-dashoffset:0;}}#my-svg .ed [...]
\ No newline at end of file
diff --git a/_images/mermaid-d0bb31b07009ce49e46219a49cdd8e6531dd24ce.svg
b/_images/mermaid-d0bb31b07009ce49e46219a49cdd8e6531dd24ce.svg
new file mode 100644
index 0000000000..f848f9b61f
--- /dev/null
+++ b/_images/mermaid-d0bb31b07009ce49e46219a49cdd8e6531dd24ce.svg
@@ -0,0 +1 @@
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg";
xmlns:xlink="http://www.w3.org/1999/xlink"; class="flowchart" style="max-width:
1661px; background-color: transparent;" viewBox="0 0 1661 660"
role="graphics-document document"
aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet
ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes
edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes
dash{to{stroke-dashoffset:0;}}#my-svg .edge-animatio [...]
\ No newline at end of file
diff --git a/contributor-guide/ci.html b/contributor-guide/ci.html
index e4fd820df1..1c48d21a3e 100644
--- a/contributor-guide/ci.html
+++ b/contributor-guide/ci.html
@@ -559,6 +559,22 @@ mechanics behind the configuration are documented in
preflight checks first (license headers, Markdown formatting, workflow
linting, the CI config
checks), computes which heavy jobs the changed files are relevant to, and fans
out to those jobs.
Which jobs run also depends on the event:</p>
+<object data="../_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg"
type="image/svg+xml">
+ <p class="warning">flowchart LR
+ PR([pull request]) --> PRTIER
+ PR -. with label .-> QUEUE
+ PR -. with label .-> NIGHTLY
+ PR -. with label .-> S34
+ MQ([merge queue]) --> PRTIER
+ MQ --> QUEUE
+ CRON([schedule<br>06:00 UTC]) --> NIGHTLY
+ PUSH([push to main]) --> CACHE
+
+ PRTIER["PR tier<br>Linux build, lint, Rust tests<br>TPC-H /
TPC-DS<br>Comet suites, Spark 4.1"]
+ QUEUE["Queue tier, on top of the PR tier<br>Spark SQL, Spark
4.1<br>Iceberg 1.11<br>macOS build and Comet
suites<br>Benchmark check, Delta gate<br>PyArrow UDF, Spark 4.0 /
4.1 / 4.2"]
+ NIGHTLY["Nightly tier<br>Comet suites, Spark 3.4 / 3.5 / 4.0 /
4.2<br>Spark SQL, Spark 3.5 / 4.0<br>Iceberg 1.8 / 1.9 / 1.10"]
+ S34["Neither tier<br>Spark SQL, Spark 3.4"]
+ CACHE["Cache-refresh-only mode<br>the four cache-writing jobs,
plus Lint"]</p></object>
<p>Suite by suite:</p>
<div class="pst-scrollable-table-container"><table class="table">
<thead>
diff --git a/contributor-guide/memory_management.html
b/contributor-guide/memory_management.html
index 0452b0156d..2a4408aabb 100644
--- a/contributor-guide/memory_management.html
+++ b/contributor-guide/memory_management.html
@@ -681,6 +681,36 @@ returns <code class="docutils literal notranslate"><span
class="pre">CometUnifie
<code class="docutils literal notranslate"><span
class="pre">spark.memory.offHeap.size</span></code>, so shuffle pages are
arbitrated against Spark’s other consumers in the
same task like any other allocation.</p>
<p>Which allocator each call site uses, and who ends up charged for the
bytes:</p>
+<object data="../_images/mermaid-24951e49e88a2a575316ecedeb5309d49a140d7a.svg"
type="image/svg+xml">
+ <p class="warning">flowchart LR
+ subgraph SITES["JVM Arrow allocation sites"]
+ NU["NativeUtil<br>FFI structs, imports, exports"]
+ UDF["CometUdfBridge<br>JVM UDF inputs and result"]
+ CGO["CometBatchKernelCodegenOutput<br>codegen UDF output"]
+ SR["StreamReader<br>shuffle and IPC reads"]
+ NAS["CometNativeArrowSource<br>stream and readerBatchIter"]
+ CACHE["ArrowCachedBatchSerializer"]
+ PY["CometArrowPythonRunnerBase"]
+ BC["Utils broadcast-coalesce"]
+ end
+
+ ROOT["CometArrowAllocator<br>RootAllocator, no limit<br>no
allocation listener"]
+ SHUF["Comet JVM shuffle
pages<br>CometUnifiedShuffleMemoryAllocator"]
+ NPOOL["Comet native memory pool<br>declared reservations
only"]
+ TMM["Spark off-heap execution pool<br>TaskMemoryManager"]
+ NOBODY["accounted by nobody"]
+
+ NU --> ROOT
+ UDF --> ROOT
+ CGO --> ROOT
+ SR --> ROOT
+ NAS --> ROOT
+ CACHE --> ROOT
+ PY --> ROOT
+ BC --> ROOT
+ ROOT --> NOBODY
+ SHUF --> TMM
+ NPOOL -->|"CometTaskMemoryManager over JNI"| TMM</p></object>
<section id="constraints-on-a-comet-memory-consumer">
<h3>Constraints on a Comet memory consumer<a class="headerlink"
href="#constraints-on-a-comet-memory-consumer" title="Link to this
heading">#</a></h3>
<p><code class="docutils literal notranslate"><span
class="pre">CometTaskMemoryManager</span></code> is the one place where Comet
code acts as a Spark <code class="docutils literal notranslate"><span
class="pre">MemoryConsumer</span></code>, and the
@@ -925,6 +955,23 @@ hard ceiling on the sum of everything in the container.
That cgroup counts, amon
<li><p>page cache charged to the cgroup by the container’s file I/O, including
spill files.</p></li>
</ul>
<p>Everything the cgroup counts, and who accounts for each part:</p>
+<object data="../_images/mermaid-d0bb31b07009ce49e46219a49cdd8e6531dd24ce.svg"
type="image/svg+xml">
+ <p class="warning">flowchart TB
+ subgraph CG["pod cgroup memory.max, kernel OOM kill above this"]
+ subgraph SEEN["visible to Spark's accounting"]
+ HEAP["JVM heap<br>execution and
storage<br>spark.executor.memory"]
+ TUNG["Spark Tungsten off-heap<br>TaskMemoryManager"]
+ SHUFP["Comet JVM shuffle
pages<br>CometUnifiedShuffleMemoryAllocator"]
+ NATRES["Comet native heap, reserved<br>operators that call
try_grow<br>declared to Spark over JNI, never measured"]
+ end
+ subgraph NONE["accounted by nobody"]
+ NATUND["Comet native heap, undeclared<br>kernels, array
builders, decompression<br>Parquet metadata, object_store, tokio"]
+ ARROWR["Comet JVM Arrow<br>CometArrowAllocator,
unbounded"]
+ NONHEAP["JVM non-heap<br>metaspace, code cache, thread
stacks<br>GC structures, Netty direct buffers"]
+ PAGEC["page cache charged to the cgroup<br>file I/O,
including spill files"]
+ FRAG["allocator overhead<br>fragmentation,
padding<br>jemalloc retained and dirty pages"]
+ end
+ end</p></object>
<p>Spark’s accounting covers the first group, though not in the same sense
throughout it. The JVM
heap, Tungsten pages and Comet’s shuffle pages are allocated by JVM code that
reports what it
allocated. A native reservation is a number an operator declared before
allocating: <code class="docutils literal notranslate"><span
class="pre">try_grow</span></code>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
