This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new bce1c8770 Publish built docs triggered by
dffcb1bdb4ab2be1e724b649f99d07f7e70ff425
bce1c8770 is described below
commit bce1c8770d8ac97891227fdd36f180d77abaed08
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Thu Feb 12 21:39:25 2026 +0000
Publish built docs triggered by dffcb1bdb4ab2be1e724b649f99d07f7e70ff425
---
_sources/contributor-guide/debugging.md.txt | 28 +++++++++++++++++++++++++++-
_sources/user-guide/latest/configs.md.txt | 1 +
contributor-guide/debugging.html | 25 ++++++++++++++++++++++++-
searchindex.js | 2 +-
user-guide/latest/configs.html | 20 ++++++++++++--------
5 files changed, 65 insertions(+), 11 deletions(-)
diff --git a/_sources/contributor-guide/debugging.md.txt
b/_sources/contributor-guide/debugging.md.txt
index db5bdfc59..1d2447d2e 100644
--- a/_sources/contributor-guide/debugging.md.txt
+++ b/_sources/contributor-guide/debugging.md.txt
@@ -127,7 +127,7 @@ To build Comet with this feature enabled:
make release COMET_FEATURES=backtrace
```
-Start Comet with `RUST_BACKTRACE=1`
+Set `RUST_BACKTRACE=1` for the Spark worker/executor process, or for
`spark-submit` if running in local mode.
```console
RUST_BACKTRACE=1 $SPARK_HOME/spark-shell --jars
spark/target/comet-spark-spark3.5_2.12-$COMET_VERSION.jar --conf
spark.plugins=org.apache.spark.CometPlugin --conf spark.comet.enabled=true
--conf spark.comet.exec.enabled=true
@@ -188,3 +188,29 @@ This produces output like the following:
Additionally, you can place a `log4rs.yaml` configuration file inside the
Comet configuration directory specified by the `COMET_CONF_DIR` environment
variable to enable more advanced logging configurations. This file uses the
[log4rs YAML configuration
format](https://docs.rs/log4rs/latest/log4rs/#configuration-via-a-yaml-file).
For example, see:
[log4rs.yaml](https://github.com/apache/datafusion-comet/blob/main/conf/log4rs.yaml).
+
+### Debugging Memory Reservations
+
+Set `spark.comet.debug.memory=true` to log all calls that grow or shrink
memory reservations.
+
+Example log output:
+
+```
+[Task 486] MemoryPool[ExternalSorter[6]].try_grow(256232960) returning Ok
+[Task 486] MemoryPool[ExternalSorter[6]].try_grow(256375168) returning Ok
+[Task 486] MemoryPool[ExternalSorter[6]].try_grow(256899456) returning Ok
+[Task 486] MemoryPool[ExternalSorter[6]].try_grow(257296128) returning Ok
+[Task 486] MemoryPool[ExternalSorter[6]].try_grow(257820416) returning Err
+[Task 486] MemoryPool[ExternalSorterMerge[6]].shrink(10485760)
+[Task 486] MemoryPool[ExternalSorter[6]].shrink(150464)
+[Task 486] MemoryPool[ExternalSorter[6]].shrink(146688)
+[Task 486] MemoryPool[ExternalSorter[6]].shrink(137856)
+[Task 486] MemoryPool[ExternalSorter[6]].shrink(141952)
+[Task 486] MemoryPool[ExternalSorterMerge[6]].try_grow(0) returning Ok
+[Task 486] MemoryPool[ExternalSorterMerge[6]].try_grow(0) returning Ok
+[Task 486] MemoryPool[ExternalSorter[6]].shrink(524288)
+[Task 486] MemoryPool[ExternalSorterMerge[6]].try_grow(0) returning Ok
+[Task 486] MemoryPool[ExternalSorterMerge[6]].try_grow(68928) returning Ok
+```
+
+When backtraces are enabled (see earlier section) then backtraces will be
included for failed allocations.
diff --git a/_sources/user-guide/latest/configs.md.txt
b/_sources/user-guide/latest/configs.md.txt
index ea32a0da6..94f47f2a5 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -136,6 +136,7 @@ These settings can be used to determine which parts of the
plan are accelerated
| `spark.comet.convert.csv.enabled` | When enabled, data from Spark
(non-native) CSV v1 and v2 scans will be converted to Arrow format. This is an
experimental feature and has known issues with non-UTC timezones. | false |
| `spark.comet.convert.json.enabled` | When enabled, data from Spark
(non-native) JSON v1 and v2 scans will be converted to Arrow format. This is an
experimental feature and has known issues with non-UTC timezones. | false |
| `spark.comet.convert.parquet.enabled` | When enabled, data from Spark
(non-native) Parquet v1 and v2 scans will be converted to Arrow format. This
is an experimental feature and has known issues with non-UTC timezones. | false
|
+| `spark.comet.debug.memory` | When enabled, log all native memory pool
interactions. For more information, refer to the Comet Debugging Guide
(https://datafusion.apache.org/comet/contributor-guide/debugging.html). | false
|
| `spark.comet.exec.onHeap.enabled` | Whether to allow Comet to run in on-heap
mode. Required for running Spark SQL tests. It can be overridden by the
environment variable `ENABLE_COMET_ONHEAP`. | false |
| `spark.comet.exec.onHeap.memoryPool` | The type of memory pool to be used
for Comet native execution when running Spark in on-heap mode. Available pool
types are `greedy`, `fair_spill`, `greedy_task_shared`,
`fair_spill_task_shared`, `greedy_global`, `fair_spill_global`, and
`unbounded`. | greedy_task_shared |
| `spark.comet.memoryOverhead` | The amount of additional memory to be
allocated per executor process for Comet, in MiB, when running Spark in on-heap
mode. | 1024 MiB |
diff --git a/contributor-guide/debugging.html b/contributor-guide/debugging.html
index 723f6c146..07f69ee46 100644
--- a/contributor-guide/debugging.html
+++ b/contributor-guide/debugging.html
@@ -550,7 +550,7 @@ Eventually it will be cleaned up when you reboot possibly
after a software updat
<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span>make<span class="w"> </span>release<span
class="w"> </span><span class="nv">COMET_FEATURES</span><span
class="o">=</span>backtrace
</pre></div>
</div>
-<p>Start Comet with <code class="docutils literal notranslate"><span
class="pre">RUST_BACKTRACE=1</span></code></p>
+<p>Set <code class="docutils literal notranslate"><span
class="pre">RUST_BACKTRACE=1</span></code> for the Spark worker/executor
process, or for <code class="docutils literal notranslate"><span
class="pre">spark-submit</span></code> if running in local mode.</p>
<div class="highlight-console notranslate"><div
class="highlight"><pre><span></span><span class="go">RUST_BACKTRACE=1
$SPARK_HOME/spark-shell --jars
spark/target/comet-spark-spark3.5_2.12-$COMET_VERSION.jar --conf
spark.plugins=org.apache.spark.CometPlugin --conf spark.comet.enabled=true
--conf spark.comet.exec.enabled=true</span>
</pre></div>
</div>
@@ -602,6 +602,29 @@ Eventually it will be cleaned up when you reboot possibly
after a software updat
<p>Additionally, you can place a <code class="docutils literal
notranslate"><span class="pre">log4rs.yaml</span></code> configuration file
inside the Comet configuration directory specified by the <code class="docutils
literal notranslate"><span class="pre">COMET_CONF_DIR</span></code> environment
variable to enable more advanced logging configurations. This file uses the <a
class="reference external"
href="https://docs.rs/log4rs/latest/log4rs/#configuration-via-a-yaml-file";>log4rs
YAML [...]
For example, see: <a class="reference external"
href="https://github.com/apache/datafusion-comet/blob/main/conf/log4rs.yaml";>log4rs.yaml</a>.</p>
</section>
+<section id="debugging-memory-reservations">
+<h3>Debugging Memory Reservations<a class="headerlink"
href="#debugging-memory-reservations" title="Link to this heading">#</a></h3>
+<p>Set <code class="docutils literal notranslate"><span
class="pre">spark.comet.debug.memory=true</span></code> to log all calls that
grow or shrink memory reservations.</p>
+<p>Example log output:</p>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="p">[</span><span
class="n">Task</span> <span class="mi">486</span><span class="p">]</span> <span
class="n">MemoryPool</span><span class="p">[</span><span
class="n">ExternalSorter</span><span class="p">[</span><span
class="mi">6</span><span class="p">]]</span><span class="o">.</span><span
class="n">try_grow</span><span class="p">(</span><span
class="mi">256232960</span><span class="p">)</span> [...]
+<span class="p">[</span><span class="n">Task</span> <span
class="mi">486</span><span class="p">]</span> <span
class="n">MemoryPool</span><span class="p">[</span><span
class="n">ExternalSorter</span><span class="p">[</span><span
class="mi">6</span><span class="p">]]</span><span class="o">.</span><span
class="n">try_grow</span><span class="p">(</span><span
class="mi">256375168</span><span class="p">)</span> <span
class="n">returning</span> <span class="n">Ok</span>
+<span class="p">[</span><span class="n">Task</span> <span
class="mi">486</span><span class="p">]</span> <span
class="n">MemoryPool</span><span class="p">[</span><span
class="n">ExternalSorter</span><span class="p">[</span><span
class="mi">6</span><span class="p">]]</span><span class="o">.</span><span
class="n">try_grow</span><span class="p">(</span><span
class="mi">256899456</span><span class="p">)</span> <span
class="n">returning</span> <span class="n">Ok</span>
+<span class="p">[</span><span class="n">Task</span> <span
class="mi">486</span><span class="p">]</span> <span
class="n">MemoryPool</span><span class="p">[</span><span
class="n">ExternalSorter</span><span class="p">[</span><span
class="mi">6</span><span class="p">]]</span><span class="o">.</span><span
class="n">try_grow</span><span class="p">(</span><span
class="mi">257296128</span><span class="p">)</span> <span
class="n">returning</span> <span class="n">Ok</span>
+<span class="p">[</span><span class="n">Task</span> <span
class="mi">486</span><span class="p">]</span> <span
class="n">MemoryPool</span><span class="p">[</span><span
class="n">ExternalSorter</span><span class="p">[</span><span
class="mi">6</span><span class="p">]]</span><span class="o">.</span><span
class="n">try_grow</span><span class="p">(</span><span
class="mi">257820416</span><span class="p">)</span> <span
class="n">returning</span> <span class="n">Err</span>
+<span class="p">[</span><span class="n">Task</span> <span
class="mi">486</span><span class="p">]</span> <span
class="n">MemoryPool</span><span class="p">[</span><span
class="n">ExternalSorterMerge</span><span class="p">[</span><span
class="mi">6</span><span class="p">]]</span><span class="o">.</span><span
class="n">shrink</span><span class="p">(</span><span
class="mi">10485760</span><span class="p">)</span>
+<span class="p">[</span><span class="n">Task</span> <span
class="mi">486</span><span class="p">]</span> <span
class="n">MemoryPool</span><span class="p">[</span><span
class="n">ExternalSorter</span><span class="p">[</span><span
class="mi">6</span><span class="p">]]</span><span class="o">.</span><span
class="n">shrink</span><span class="p">(</span><span
class="mi">150464</span><span class="p">)</span>
+<span class="p">[</span><span class="n">Task</span> <span
class="mi">486</span><span class="p">]</span> <span
class="n">MemoryPool</span><span class="p">[</span><span
class="n">ExternalSorter</span><span class="p">[</span><span
class="mi">6</span><span class="p">]]</span><span class="o">.</span><span
class="n">shrink</span><span class="p">(</span><span
class="mi">146688</span><span class="p">)</span>
+<span class="p">[</span><span class="n">Task</span> <span
class="mi">486</span><span class="p">]</span> <span
class="n">MemoryPool</span><span class="p">[</span><span
class="n">ExternalSorter</span><span class="p">[</span><span
class="mi">6</span><span class="p">]]</span><span class="o">.</span><span
class="n">shrink</span><span class="p">(</span><span
class="mi">137856</span><span class="p">)</span>
+<span class="p">[</span><span class="n">Task</span> <span
class="mi">486</span><span class="p">]</span> <span
class="n">MemoryPool</span><span class="p">[</span><span
class="n">ExternalSorter</span><span class="p">[</span><span
class="mi">6</span><span class="p">]]</span><span class="o">.</span><span
class="n">shrink</span><span class="p">(</span><span
class="mi">141952</span><span class="p">)</span>
+<span class="p">[</span><span class="n">Task</span> <span
class="mi">486</span><span class="p">]</span> <span
class="n">MemoryPool</span><span class="p">[</span><span
class="n">ExternalSorterMerge</span><span class="p">[</span><span
class="mi">6</span><span class="p">]]</span><span class="o">.</span><span
class="n">try_grow</span><span class="p">(</span><span class="mi">0</span><span
class="p">)</span> <span class="n">returning</span> <span class="n">Ok</span>
+<span class="p">[</span><span class="n">Task</span> <span
class="mi">486</span><span class="p">]</span> <span
class="n">MemoryPool</span><span class="p">[</span><span
class="n">ExternalSorterMerge</span><span class="p">[</span><span
class="mi">6</span><span class="p">]]</span><span class="o">.</span><span
class="n">try_grow</span><span class="p">(</span><span class="mi">0</span><span
class="p">)</span> <span class="n">returning</span> <span class="n">Ok</span>
+<span class="p">[</span><span class="n">Task</span> <span
class="mi">486</span><span class="p">]</span> <span
class="n">MemoryPool</span><span class="p">[</span><span
class="n">ExternalSorter</span><span class="p">[</span><span
class="mi">6</span><span class="p">]]</span><span class="o">.</span><span
class="n">shrink</span><span class="p">(</span><span
class="mi">524288</span><span class="p">)</span>
+<span class="p">[</span><span class="n">Task</span> <span
class="mi">486</span><span class="p">]</span> <span
class="n">MemoryPool</span><span class="p">[</span><span
class="n">ExternalSorterMerge</span><span class="p">[</span><span
class="mi">6</span><span class="p">]]</span><span class="o">.</span><span
class="n">try_grow</span><span class="p">(</span><span class="mi">0</span><span
class="p">)</span> <span class="n">returning</span> <span class="n">Ok</span>
+<span class="p">[</span><span class="n">Task</span> <span
class="mi">486</span><span class="p">]</span> <span
class="n">MemoryPool</span><span class="p">[</span><span
class="n">ExternalSorterMerge</span><span class="p">[</span><span
class="mi">6</span><span class="p">]]</span><span class="o">.</span><span
class="n">try_grow</span><span class="p">(</span><span
class="mi">68928</span><span class="p">)</span> <span
class="n">returning</span> <span class="n">Ok</span>
+</pre></div>
+</div>
+<p>When backtraces are enabled (see earlier section) then backtraces will be
included for failed allocations.</p>
+</section>
</section>
</section>
diff --git a/searchindex.js b/searchindex.js
index f7837cb8c..ba308ee28 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Format Your Code": [[12,
"format-your-code"]], "1. Install Comet": [[22, "install-comet"]], "1. Native
Operators (nativeExecs map)": [[4, "native-operators-nativeexecs-map"]], "2.
Build and Verify": [[12, "build-and-verify"]], "2. Clone Spark and Apply Diff":
[[22, "clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[4,
"sink-operators-sinks-map"]], "3. Comet JVM Operators": [[4,
"comet-jvm-operators"]], "3. Run Clippy (Recommended)": [[12 [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Format Your Code": [[12,
"format-your-code"]], "1. Install Comet": [[22, "install-comet"]], "1. Native
Operators (nativeExecs map)": [[4, "native-operators-nativeexecs-map"]], "2.
Build and Verify": [[12, "build-and-verify"]], "2. Clone Spark and Apply Diff":
[[22, "clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[4,
"sink-operators-sinks-map"]], "3. Comet JVM Operators": [[4,
"comet-jvm-operators"]], "3. Run Clippy (Recommended)": [[12 [...]
\ No newline at end of file
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index 8ed9d3752..5845485ec 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -794,35 +794,39 @@ under the License.
<td><p>When enabled, data from Spark (non-native) Parquet v1 and v2 scans will
be converted to Arrow format. This is an experimental feature and has known
issues with non-UTC timezones.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.onHeap.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.debug.memory</span></code></p></td>
+<td><p>When enabled, log all native memory pool interactions. For more
information, refer to the Comet Debugging Guide
(https://datafusion.apache.org/comet/contributor-guide/debugging.html).</p></td>
+<td><p>false</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.onHeap.enabled</span></code></p></td>
<td><p>Whether to allow Comet to run in on-heap mode. Required for running
Spark SQL tests. It can be overridden by the environment variable <code
class="docutils literal notranslate"><span
class="pre">ENABLE_COMET_ONHEAP</span></code>.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.onHeap.memoryPool</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.onHeap.memoryPool</span></code></p></td>
<td><p>The type of memory pool to be used for Comet native execution when
running Spark in on-heap mode. Available pool types are <code class="docutils
literal notranslate"><span class="pre">greedy</span></code>, <code
class="docutils literal notranslate"><span
class="pre">fair_spill</span></code>, <code class="docutils literal
notranslate"><span class="pre">greedy_task_shared</span></code>, <code
class="docutils literal notranslate"><span
class="pre">fair_spill_task_shared</span></code> [...]
<td><p>greedy_task_shared</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.memoryOverhead</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.memoryOverhead</span></code></p></td>
<td><p>The amount of additional memory to be allocated per executor process
for Comet, in MiB, when running Spark in on-heap mode.</p></td>
<td><p>1024 MiB</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.parquet.write.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.parquet.write.enabled</span></code></p></td>
<td><p>Whether to enable native Parquet write through Comet. When enabled,
Comet will intercept Parquet write operations and execute them natively. This
feature is highly experimental and only partially implemented. It should not be
used in production.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.scan.csv.v2.enabled</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.scan.csv.v2.enabled</span></code></p></td>
<td><p>Whether to use the native Comet V2 CSV reader for improved performance.
Default: false (uses standard Spark CSV reader) Experimental: Performance
benefits are workload-dependent.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.sparkToColumnar.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.sparkToColumnar.enabled</span></code></p></td>
<td><p>Whether to enable Spark to Arrow columnar conversion. When this is
turned on, Comet will convert operators in <code class="docutils literal
notranslate"><span
class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code>
into Arrow columnar format before processing. This is an experimental feature
and has known issues with non-UTC timezones.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code></p></td>
<td><p>A comma-separated list of operators that will be converted to Arrow
columnar format when <code class="docutils literal notranslate"><span
class="pre">spark.comet.sparkToColumnar.enabled</span></code> is true.</p></td>
<td><p>Range,InMemoryTableScan,RDDScan</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.testing.strict</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.testing.strict</span></code></p></td>
<td><p>Experimental option to enable strict testing, which will fail tests
that could be more comprehensive, such as checking for a specific fallback
reason. It can be overridden by the environment variable <code class="docutils
literal notranslate"><span
class="pre">ENABLE_COMET_STRICT_TESTING</span></code>.</p></td>
<td><p>false</p></td>
</tr>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
