This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 290e67e9b Publish built docs triggered by
7022983198f00469c6bb830b38f13c9e83513528
290e67e9b is described below
commit 290e67e9be6faf700ead302f86967035f25e1162
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Tue Feb 17 19:23:44 2026 +0000
Publish built docs triggered by 7022983198f00469c6bb830b38f13c9e83513528
---
_sources/contributor-guide/roadmap.md.txt | 19 +++++++++++++++----
_sources/user-guide/latest/configs.md.txt | 1 +
contributor-guide/roadmap.html | 14 +++++++++++---
searchindex.js | 2 +-
user-guide/latest/configs.html | 16 ++++++++++------
5 files changed, 38 insertions(+), 14 deletions(-)
diff --git a/_sources/contributor-guide/roadmap.md.txt
b/_sources/contributor-guide/roadmap.md.txt
index 3abe6f3f1..ce9c41416 100644
--- a/_sources/contributor-guide/roadmap.md.txt
+++ b/_sources/contributor-guide/roadmap.md.txt
@@ -26,11 +26,11 @@ helpful to have a roadmap for some of the major items that
require coordination
### Iceberg Integration
-Iceberg integration is still a work-in-progress ([#2060]), with major
improvements expected in the next few
-releases. The default `auto` scan mode now uses `native_iceberg_compat`
instead of `native_comet`, enabling
-support for complex types.
+Iceberg tables reads are now fully native, powered by a scan operator backed
by Iceberg-rust ([#2528]). We anticipate
+major improvements expected in the next few releases, including bringing
Iceberg table format V3 features (_e.g._,
+encryption) to the reader.
-[#2060]: https://github.com/apache/datafusion-comet/issues/2060
+[#2528]: https://github.com/apache/datafusion-comet/pull/2528
### Spark 4.0 Support
@@ -40,6 +40,17 @@ more Spark SQL tests and fully implementing ANSI support
([#313]) for all suppor
[#313]: https://github.com/apache/datafusion-comet/issues/313
[#1637]: https://github.com/apache/datafusion-comet/issues/1637
+### Dynamic Partition Pruning
+
+Iceberg table scans support Dynamic Partition Pruning (DPP) filters generated
by Spark's `PlanDynamicPruningFilters`
+optimizer rule ([#3349]). However, we still need to bring this functionality
to our Parquet reader. Furthermore,
+Spark's `PlanAdaptiveDynamicPruningFilters` optimizer rule runs after Comet's
rules, so DPP with Adaptive Query
+Execution requires a redesign of Comet's plan translation. We are focused on
implementing DPP to keep Comet competitive
+with benchmarks that benefit from this feature like TPC-DS. This effort can be
tracked at [#3510].
+
+[#3349]: https://github.com/apache/datafusion-comet/pull/3349
+[#3510]: https://github.com/apache/datafusion-comet/issues/3510
+
### Removing the native_comet scan implementation
The `native_comet` scan implementation is now deprecated and will be removed
in a future release ([#2186], [#2177]).
diff --git a/_sources/user-guide/latest/configs.md.txt
b/_sources/user-guide/latest/configs.md.txt
index a4e389b43..6ae0ec6fb 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -139,6 +139,7 @@ These settings can be used to determine which parts of the
plan are accelerated
| `spark.comet.debug.memory` | When enabled, log all native memory pool
interactions. For more information, refer to the Comet Debugging Guide
(https://datafusion.apache.org/comet/contributor-guide/debugging.html). | false
|
| `spark.comet.exec.onHeap.enabled` | Whether to allow Comet to run in on-heap
mode. Required for running Spark SQL tests. It can be overridden by the
environment variable `ENABLE_COMET_ONHEAP`. | false |
| `spark.comet.exec.onHeap.memoryPool` | The type of memory pool to be used
for Comet native execution when running Spark in on-heap mode. Available pool
types are `greedy`, `fair_spill`, `greedy_task_shared`,
`fair_spill_task_shared`, `greedy_global`, `fair_spill_global`, and
`unbounded`. | greedy_task_shared |
+| `spark.comet.exec.respectDataFusionConfigs` | Development and testing
configuration option to allow DataFusion configs set in Spark configuration
settings starting with `spark.comet.datafusion.` to be passed into native
execution. | false |
| `spark.comet.memoryOverhead` | The amount of additional memory to be
allocated per executor process for Comet, in MiB, when running Spark in on-heap
mode. | 1024 MiB |
| `spark.comet.parquet.write.enabled` | Whether to enable native Parquet write
through Comet. When enabled, Comet will intercept Parquet write operations and
execute them natively. This feature is highly experimental and only partially
implemented. It should not be used in production. | false |
| `spark.comet.scan.csv.v2.enabled` | Whether to use the native Comet V2 CSV
reader for improved performance. Default: false (uses standard Spark CSV
reader) Experimental: Performance benefits are workload-dependent. | false |
diff --git a/contributor-guide/roadmap.html b/contributor-guide/roadmap.html
index 91709dd5f..be85cf9ef 100644
--- a/contributor-guide/roadmap.html
+++ b/contributor-guide/roadmap.html
@@ -463,15 +463,23 @@ helpful to have a roadmap for some of the major items
that require coordination
<h2>Major Initiatives<a class="headerlink" href="#major-initiatives"
title="Link to this heading">#</a></h2>
<section id="iceberg-integration">
<h3>Iceberg Integration<a class="headerlink" href="#iceberg-integration"
title="Link to this heading">#</a></h3>
-<p>Iceberg integration is still a work-in-progress (<a class="reference
external"
href="https://github.com/apache/datafusion-comet/issues/2060";>#2060</a>), with
major improvements expected in the next few
-releases. The default <code class="docutils literal notranslate"><span
class="pre">auto</span></code> scan mode now uses <code class="docutils literal
notranslate"><span class="pre">native_iceberg_compat</span></code> instead of
<code class="docutils literal notranslate"><span
class="pre">native_comet</span></code>, enabling
-support for complex types.</p>
+<p>Iceberg tables reads are now fully native, powered by a scan operator
backed by Iceberg-rust (<a class="reference external"
href="https://github.com/apache/datafusion-comet/pull/2528";>#2528</a>). We
anticipate
+major improvements expected in the next few releases, including bringing
Iceberg table format V3 features (<em>e.g.</em>,
+encryption) to the reader.</p>
</section>
<section id="spark-4-0-support">
<h3>Spark 4.0 Support<a class="headerlink" href="#spark-4-0-support"
title="Link to this heading">#</a></h3>
<p>Comet has experimental support for Spark 4.0, but there is more work to do
(<a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/1637";>#1637</a>), such
as enabling
more Spark SQL tests and fully implementing ANSI support (<a class="reference
external"
href="https://github.com/apache/datafusion-comet/issues/313";>#313</a>) for all
supported expressions.</p>
</section>
+<section id="dynamic-partition-pruning">
+<h3>Dynamic Partition Pruning<a class="headerlink"
href="#dynamic-partition-pruning" title="Link to this heading">#</a></h3>
+<p>Iceberg table scans support Dynamic Partition Pruning (DPP) filters
generated by Spark’s <code class="docutils literal notranslate"><span
class="pre">PlanDynamicPruningFilters</span></code>
+optimizer rule (<a class="reference external"
href="https://github.com/apache/datafusion-comet/pull/3349";>#3349</a>).
However, we still need to bring this functionality to our Parquet reader.
Furthermore,
+Spark’s <code class="docutils literal notranslate"><span
class="pre">PlanAdaptiveDynamicPruningFilters</span></code> optimizer rule runs
after Comet’s rules, so DPP with Adaptive Query
+Execution requires a redesign of Comet’s plan translation. We are focused on
implementing DPP to keep Comet competitive
+with benchmarks that benefit from this feature like TPC-DS. This effort can be
tracked at <a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/3510";>#3510</a>.</p>
+</section>
<section id="removing-the-native-comet-scan-implementation">
<h3>Removing the native_comet scan implementation<a class="headerlink"
href="#removing-the-native-comet-scan-implementation" title="Link to this
heading">#</a></h3>
<p>The <code class="docutils literal notranslate"><span
class="pre">native_comet</span></code> scan implementation is now deprecated
and will be removed in a future release (<a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/2186";>#2186</a>, <a
class="reference external"
href="https://github.com/apache/datafusion-comet/issues/2177";>#2177</a>).
diff --git a/searchindex.js b/searchindex.js
index 9904bc795..1d278c792 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Format Your Code": [[12,
"format-your-code"]], "1. Install Comet": [[22, "install-comet"]], "1. Native
Operators (nativeExecs map)": [[4, "native-operators-nativeexecs-map"]], "2.
Build and Verify": [[12, "build-and-verify"]], "2. Clone Spark and Apply Diff":
[[22, "clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[4,
"sink-operators-sinks-map"]], "3. Comet JVM Operators": [[4,
"comet-jvm-operators"]], "3. Run Clippy (Recommended)": [[12 [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Format Your Code": [[12,
"format-your-code"]], "1. Install Comet": [[22, "install-comet"]], "1. Native
Operators (nativeExecs map)": [[4, "native-operators-nativeexecs-map"]], "2.
Build and Verify": [[12, "build-and-verify"]], "2. Clone Spark and Apply Diff":
[[22, "clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[4,
"sink-operators-sinks-map"]], "3. Comet JVM Operators": [[4,
"comet-jvm-operators"]], "3. Run Clippy (Recommended)": [[12 [...]
\ No newline at end of file
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index c640704da..465e3e6ba 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -806,27 +806,31 @@ under the License.
<td><p>The type of memory pool to be used for Comet native execution when
running Spark in on-heap mode. Available pool types are <code class="docutils
literal notranslate"><span class="pre">greedy</span></code>, <code
class="docutils literal notranslate"><span
class="pre">fair_spill</span></code>, <code class="docutils literal
notranslate"><span class="pre">greedy_task_shared</span></code>, <code
class="docutils literal notranslate"><span
class="pre">fair_spill_task_shared</span></code> [...]
<td><p>greedy_task_shared</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.memoryOverhead</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.respectDataFusionConfigs</span></code></p></td>
+<td><p>Development and testing configuration option to allow DataFusion
configs set in Spark configuration settings starting with <code class="docutils
literal notranslate"><span class="pre">spark.comet.datafusion.</span></code> to
be passed into native execution.</p></td>
+<td><p>false</p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.memoryOverhead</span></code></p></td>
<td><p>The amount of additional memory to be allocated per executor process
for Comet, in MiB, when running Spark in on-heap mode.</p></td>
<td><p>1024 MiB</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.parquet.write.enabled</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.parquet.write.enabled</span></code></p></td>
<td><p>Whether to enable native Parquet write through Comet. When enabled,
Comet will intercept Parquet write operations and execute them natively. This
feature is highly experimental and only partially implemented. It should not be
used in production.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.scan.csv.v2.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.scan.csv.v2.enabled</span></code></p></td>
<td><p>Whether to use the native Comet V2 CSV reader for improved performance.
Default: false (uses standard Spark CSV reader) Experimental: Performance
benefits are workload-dependent.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.sparkToColumnar.enabled</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.sparkToColumnar.enabled</span></code></p></td>
<td><p>Whether to enable Spark to Arrow columnar conversion. When this is
turned on, Comet will convert operators in <code class="docutils literal
notranslate"><span
class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code>
into Arrow columnar format before processing. This is an experimental feature
and has known issues with non-UTC timezones.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code></p></td>
<td><p>A comma-separated list of operators that will be converted to Arrow
columnar format when <code class="docutils literal notranslate"><span
class="pre">spark.comet.sparkToColumnar.enabled</span></code> is true.</p></td>
<td><p>Range,InMemoryTableScan,RDDScan</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.testing.strict</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.testing.strict</span></code></p></td>
<td><p>Experimental option to enable strict testing, which will fail tests
that could be more comprehensive, such as checking for a specific fallback
reason. It can be overridden by the environment variable <code class="docutils
literal notranslate"><span
class="pre">ENABLE_COMET_STRICT_TESTING</span></code>.</p></td>
<td><p>false</p></td>
</tr>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
