This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 9300fbff4f Publish built docs triggered by
7203fb596a92c037dd758caa6bd006486612cc8a
9300fbff4f is described below
commit 9300fbff4f12c4ae8d0fdf53445bb942b48f15b7
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Tue May 19 00:45:25 2026 +0000
Publish built docs triggered by 7203fb596a92c037dd758caa6bd006486612cc8a
---
_sources/contributor-guide/bug_triage.md.txt | 8 +++-----
_sources/user-guide/latest/configs.md.txt | 1 -
_sources/user-guide/latest/datasources.md.txt | 12 ++++--------
contributor-guide/bug_triage.html | 8 +++-----
searchindex.js | 2 +-
user-guide/latest/configs.html | 4 ----
user-guide/latest/datasources.html | 15 ++++++---------
7 files changed, 17 insertions(+), 33 deletions(-)
diff --git a/_sources/contributor-guide/bug_triage.md.txt
b/_sources/contributor-guide/bug_triage.md.txt
index 9e51f44637..477829bae4 100644
--- a/_sources/contributor-guide/bug_triage.md.txt
+++ b/_sources/contributor-guide/bug_triage.md.txt
@@ -73,8 +73,7 @@ help contributors find bugs in their area of expertise.
| `area:ffi` | Arrow FFI / JNI boundary |
| `area:ci` | CI/CD, GitHub Actions, build tooling |
-The following pre-existing labels also serve as area indicators:
`native_datafusion`,
-`native_iceberg_compat`, `spark 4`, `spark sql tests`.
+The following pre-existing labels also serve as area indicators: `spark 4`,
`spark sql tests`.
## Triage Process
@@ -109,9 +108,8 @@ Periodically review open bugs to ensure priorities are
still accurate:
crashes, because crashes are at least visible.
2. **User-reported over test-only.** A bug hit by a real user on a real
workload takes priority
over one found only in test suites.
-3. **Core path over experimental.** Bugs in the default scan mode
(`native_comet`) or widely-used
- expressions take priority over bugs in experimental features like
`native_datafusion` or
- `native_iceberg_compat`.
+3. **Core path over experimental.** Bugs in widely-used expressions and
operators take priority over
+ bugs in experimental features.
4. **Production safety over feature completeness.** Fixing a data corruption
bug is more important
than adding support for a new expression.
diff --git a/_sources/user-guide/latest/configs.md.txt
b/_sources/user-guide/latest/configs.md.txt
index 85f06a149b..573ea229da 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -47,7 +47,6 @@ Comet provides the following configuration settings.
| `spark.comet.parquet.read.parallel.io.enabled` | Whether to enable Comet's
parallel reader for Parquet files. The parallel reader reads ranges of
consecutive data in a file in parallel. It is faster for large files and row
groups but uses more resources. | true |
| `spark.comet.parquet.read.parallel.io.thread-pool.size` | The maximum number
of parallel threads the parallel reader will use in a single executor. For
executors configured with a smaller number of cores, use a smaller number. | 16
|
| `spark.comet.parquet.respectFilterPushdown` | Whether to respect Spark's
PARQUET_FILTER_PUSHDOWN_ENABLED config. This needs to be respected when running
the Spark SQL test suite but the default setting results in poor performance in
Comet when using the new native scans, disabled by default | false |
-| `spark.comet.scan.impl` | The implementation of Comet's Parquet scan to use.
Available scans are `native_datafusion`, and `native_iceberg_compat`.
`native_datafusion` is a fully native implementation, and
`native_iceberg_compat` is a hybrid implementation that supports some
additional features, such as row indexes and field ids. `auto` (default)
chooses the best available scan based on the scan schema. It can be overridden
by the environment variable `COMET_PARQUET_SCAN_IMPL`. | auto |
<!-- prettier-ignore-end -->
<!--END:CONFIG_TABLE-->
diff --git a/_sources/user-guide/latest/datasources.md.txt
b/_sources/user-guide/latest/datasources.md.txt
index 8808bce6dd..9e1896141c 100644
--- a/_sources/user-guide/latest/datasources.md.txt
+++ b/_sources/user-guide/latest/datasources.md.txt
@@ -61,11 +61,9 @@ Comet supports most standard storage systems, such as local
file system and obje
Apache DataFusion Comet native reader seamlessly scans files from remote HDFS
for [supported formats](#supported-spark-data-sources)
-### Using experimental native DataFusion reader
+### Building Comet with HDFS support
-Unlike to native Comet reader the Datafusion reader fully supports nested
types processing. This reader is currently experimental only
-
-To build Comet with native DataFusion reader and remote HDFS support it is
required to have a JDK installed
+To build Comet with remote HDFS support it is required to have a JDK installed.
Example:
Build a Comet for `spark-4.1` provide a JDK path in `JAVA_HOME`
@@ -76,11 +74,10 @@ export JAVA_HOME="/opt/homebrew/opt/openjdk@17"
make release PROFILES="-Pspark-4.1" COMET_FEATURES=hdfs RUSTFLAGS="-L
$JAVA_HOME/libexec/openjdk.jdk/Contents/Home/lib/server"
```
-Start Comet with experimental reader and HDFS support as
[described](installation.md/#run-spark-shell-with-comet-enabled)
+Start Comet with HDFS support as
[described](installation.md/#run-spark-shell-with-comet-enabled)
and add additional parameters
```shell
---conf spark.comet.scan.impl=native_datafusion \
--conf spark.hadoop.fs.defaultFS="hdfs://namenode:9000" \
--conf spark.hadoop.dfs.client.use.datanode.hostname = true \
--conf dfs.client.use.datanode.hostname = true
@@ -158,7 +155,6 @@ JAVA_HOME="/opt/homebrew/opt/openjdk@17" make release
PROFILES="-Pspark-4.1" COM
withSQLConf(
CometConf.COMET_ENABLED.key -> "true",
CometConf.COMET_EXEC_ENABLED.key -> "true",
- CometConf.COMET_NATIVE_SCAN_IMPL.key -> CometConf.SCAN_NATIVE_DATAFUSION,
SQLConf.USE_V1_SOURCE_LIST.key -> "parquet",
"fs.defaultFS" -> "hdfs://namenode:9000",
"dfs.client.use.datanode.hostname" -> "true") {
@@ -169,7 +165,7 @@ JAVA_HOME="/opt/homebrew/opt/openjdk@17" make release
PROFILES="-Pspark-4.1" COM
}
```
-Or use `spark-shell` with HDFS support as described
[above](#using-experimental-native-datafusion-reader)
+Or use `spark-shell` with HDFS support as described
[above](#building-comet-with-hdfs-support)
## S3
diff --git a/contributor-guide/bug_triage.html
b/contributor-guide/bug_triage.html
index 14f3c9f5e7..759156133c 100644
--- a/contributor-guide/bug_triage.html
+++ b/contributor-guide/bug_triage.html
@@ -602,8 +602,7 @@ help contributors find bugs in their area of expertise.</p>
</tbody>
</table>
</div>
-<p>The following pre-existing labels also serve as area indicators: <code
class="docutils literal notranslate"><span
class="pre">native_datafusion</span></code>,
-<code class="docutils literal notranslate"><span
class="pre">native_iceberg_compat</span></code>, <code class="docutils literal
notranslate"><span class="pre">spark</span> <span class="pre">4</span></code>,
<code class="docutils literal notranslate"><span class="pre">spark</span> <span
class="pre">sql</span> <span class="pre">tests</span></code>.</p>
+<p>The following pre-existing labels also serve as area indicators: <code
class="docutils literal notranslate"><span class="pre">spark</span> <span
class="pre">4</span></code>, <code class="docutils literal notranslate"><span
class="pre">spark</span> <span class="pre">sql</span> <span
class="pre">tests</span></code>.</p>
</section>
<section id="triage-process">
<h2>Triage Process<a class="headerlink" href="#triage-process" title="Link to
this heading">#</a></h2>
@@ -640,9 +639,8 @@ is more important than whether it crashes.</p></li>
crashes, because crashes are at least visible.</p></li>
<li><p><strong>User-reported over test-only.</strong> A bug hit by a real user
on a real workload takes priority
over one found only in test suites.</p></li>
-<li><p><strong>Core path over experimental.</strong> Bugs in the default scan
mode (<code class="docutils literal notranslate"><span
class="pre">native_comet</span></code>) or widely-used
-expressions take priority over bugs in experimental features like <code
class="docutils literal notranslate"><span
class="pre">native_datafusion</span></code> or
-<code class="docutils literal notranslate"><span
class="pre">native_iceberg_compat</span></code>.</p></li>
+<li><p><strong>Core path over experimental.</strong> Bugs in widely-used
expressions and operators take priority over
+bugs in experimental features.</p></li>
<li><p><strong>Production safety over feature completeness.</strong> Fixing a
data corruption bug is more important
than adding support for a new expression.</p></li>
</ol>
diff --git a/searchindex.js b/searchindex.js
index 2b8bc87e2c..63e34e9baf 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Format Your Code": [[34,
"format-your-code"]], "1. Install Comet": [[36, "install-comet"], [44,
"install-comet"]], "1. Native Operators (nativeExecs map)": [[23,
"native-operators-nativeexecs-map"]], "2. Build and Verify": [[34,
"build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[36,
"clone-iceberg-and-apply-diff"]], "2. Clone Spark and Apply Diff": [[44,
"clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[23,
"sink-operators-sinks [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Format Your Code": [[34,
"format-your-code"]], "1. Install Comet": [[36, "install-comet"], [44,
"install-comet"]], "1. Native Operators (nativeExecs map)": [[23,
"native-operators-nativeexecs-map"]], "2. Build and Verify": [[34,
"build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[36,
"clone-iceberg-and-apply-diff"]], "2. Clone Spark and Apply Diff": [[44,
"clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[23,
"sink-operators-sinks [...]
\ No newline at end of file
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index 24063b0620..3d461c0880 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -576,10 +576,6 @@ under the License.
<td><p>Whether to respect Spark’s PARQUET_FILTER_PUSHDOWN_ENABLED config. This
needs to be respected when running the Spark SQL test suite but the default
setting results in poor performance in Comet when using the new native scans,
disabled by default</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.scan.impl</span></code></p></td>
-<td><p>The implementation of Comet’s Parquet scan to use. Available scans are
<code class="docutils literal notranslate"><span
class="pre">native_datafusion</span></code>, and <code class="docutils literal
notranslate"><span class="pre">native_iceberg_compat</span></code>. <code
class="docutils literal notranslate"><span
class="pre">native_datafusion</span></code> is a fully native implementation,
and <code class="docutils literal notranslate"><span
class="pre">native_iceberg_compat</spa [...]
-<td><p>auto</p></td>
-</tr>
</tbody>
</table>
</div>
diff --git a/user-guide/latest/datasources.html
b/user-guide/latest/datasources.html
index b2c831e480..0656c40808 100644
--- a/user-guide/latest/datasources.html
+++ b/user-guide/latest/datasources.html
@@ -541,10 +541,9 @@ converted into Arrow format, allowing native execution to
happen after that.</p>
<h3>HDFS<a class="headerlink" href="#hdfs" title="Link to this
heading">#</a></h3>
<p>Apache DataFusion Comet native reader seamlessly scans files from remote
HDFS for <a class="reference internal"
href="#supported-spark-data-sources">supported formats</a></p>
</section>
-<section id="using-experimental-native-datafusion-reader">
-<h3>Using experimental native DataFusion reader<a class="headerlink"
href="#using-experimental-native-datafusion-reader" title="Link to this
heading">#</a></h3>
-<p>Unlike to native Comet reader the Datafusion reader fully supports nested
types processing. This reader is currently experimental only</p>
-<p>To build Comet with native DataFusion reader and remote HDFS support it is
required to have a JDK installed</p>
+<section id="building-comet-with-hdfs-support">
+<h3>Building Comet with HDFS support<a class="headerlink"
href="#building-comet-with-hdfs-support" title="Link to this heading">#</a></h3>
+<p>To build Comet with remote HDFS support it is required to have a JDK
installed.</p>
<p>Example:
Build a Comet for <code class="docutils literal notranslate"><span
class="pre">spark-4.1</span></code> provide a JDK path in <code class="docutils
literal notranslate"><span class="pre">JAVA_HOME</span></code>
Provide the JRE linker path in <code class="docutils literal
notranslate"><span class="pre">RUSTFLAGS</span></code>, the path can vary
depending on the system. Typically JRE linker is a part of installed JDK</p>
@@ -552,10 +551,9 @@ Provide the JRE linker path in <code class="docutils
literal notranslate"><span
make<span class="w"> </span>release<span class="w"> </span><span
class="nv">PROFILES</span><span class="o">=</span><span
class="s2">"-Pspark-4.1"</span><span class="w"> </span><span
class="nv">COMET_FEATURES</span><span class="o">=</span>hdfs<span class="w">
</span><span class="nv">RUSTFLAGS</span><span class="o">=</span><span
class="s2">"-L </span><span class="nv">$JAVA_HOME</span><span
class="s2">/libexec/openjdk.jdk/Contents/Home/lib/server"</span>
</pre></div>
</div>
-<p>Start Comet with experimental reader and HDFS support as <a
class="reference internal"
href="installation.html#run-spark-shell-with-comet-enabled"><span class="std
std-ref">described</span></a>
+<p>Start Comet with HDFS support as <a class="reference internal"
href="installation.html#run-spark-shell-with-comet-enabled"><span class="std
std-ref">described</span></a>
and add additional parameters</p>
-<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span>--conf<span class="w">
</span>spark.comet.scan.impl<span class="o">=</span>native_datafusion<span
class="w"> </span><span class="se">\</span>
---conf<span class="w"> </span>spark.hadoop.fs.defaultFS<span
class="o">=</span><span class="s2">"hdfs://namenode:9000"</span><span
class="w"> </span><span class="se">\</span>
+<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span>--conf<span class="w">
</span>spark.hadoop.fs.defaultFS<span class="o">=</span><span
class="s2">"hdfs://namenode:9000"</span><span class="w"> </span><span
class="se">\</span>
--conf<span class="w">
</span>spark.hadoop.dfs.client.use.datanode.hostname<span class="w">
</span><span class="o">=</span><span class="w"> </span><span
class="nb">true</span><span class="w"> </span><span class="se">\</span>
--conf<span class="w"> </span>dfs.client.use.datanode.hostname<span class="w">
</span><span class="o">=</span><span class="w"> </span><span
class="nb">true</span>
</pre></div>
@@ -625,7 +623,6 @@ Input<span class="w"> </span><span class="o">[</span><span
class="m">3</span><sp
<div class="highlight-scala notranslate"><div
class="highlight"><pre><span></span><span class="w"> </span><span
class="n">withSQLConf</span><span class="p">(</span>
<span class="w"> </span><span class="nc">CometConf</span><span
class="p">.</span><span class="nc">COMET_ENABLED</span><span
class="p">.</span><span class="n">key</span><span class="w"> </span><span
class="o">-></span><span class="w"> </span><span
class="s">"true"</span><span class="p">,</span>
<span class="w"> </span><span class="nc">CometConf</span><span
class="p">.</span><span class="nc">COMET_EXEC_ENABLED</span><span
class="p">.</span><span class="n">key</span><span class="w"> </span><span
class="o">-></span><span class="w"> </span><span
class="s">"true"</span><span class="p">,</span>
-<span class="w"> </span><span class="nc">CometConf</span><span
class="p">.</span><span class="nc">COMET_NATIVE_SCAN_IMPL</span><span
class="p">.</span><span class="n">key</span><span class="w"> </span><span
class="o">-></span><span class="w"> </span><span
class="nc">CometConf</span><span class="p">.</span><span
class="nc">SCAN_NATIVE_DATAFUSION</span><span class="p">,</span>
<span class="w"> </span><span class="nc">SQLConf</span><span
class="p">.</span><span class="nc">USE_V1_SOURCE_LIST</span><span
class="p">.</span><span class="n">key</span><span class="w"> </span><span
class="o">-></span><span class="w"> </span><span
class="s">"parquet"</span><span class="p">,</span>
<span class="w"> </span><span
class="s">"fs.defaultFS"</span><span class="w"> </span><span
class="o">-></span><span class="w"> </span><span
class="s">"hdfs://namenode:9000"</span><span class="p">,</span>
<span class="w"> </span><span
class="s">"dfs.client.use.datanode.hostname"</span><span class="w">
</span><span class="o">-></span><span class="w"> </span><span
class="s">"true"</span><span class="p">)</span><span class="w">
</span><span class="p">{</span>
@@ -636,7 +633,7 @@ Input<span class="w"> </span><span class="o">[</span><span
class="m">3</span><sp
<span class="w"> </span><span class="p">}</span>
</pre></div>
</div>
-<p>Or use <code class="docutils literal notranslate"><span
class="pre">spark-shell</span></code> with HDFS support as described <a
class="reference internal"
href="#using-experimental-native-datafusion-reader">above</a></p>
+<p>Or use <code class="docutils literal notranslate"><span
class="pre">spark-shell</span></code> with HDFS support as described <a
class="reference internal"
href="#building-comet-with-hdfs-support">above</a></p>
</section>
</section>
<section id="s3">
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
