This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new a7d1cf4985 Publish built docs triggered by
691e2ae948432dadbb8be752000d2fe1b2feade8
a7d1cf4985 is described below
commit a7d1cf4985badf64b41579b355be8fcc932ccc82
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Mon Jul 13 18:17:46 2026 +0000
Publish built docs triggered by 691e2ae948432dadbb8be752000d2fe1b2feade8
---
_sources/user-guide/latest/configs.md.txt | 2 +-
_sources/user-guide/latest/datasources.md.txt | 8 +++-----
searchindex.js | 2 +-
user-guide/latest/configs.html | 2 +-
user-guide/latest/datasources.html | 7 +++----
5 files changed, 9 insertions(+), 12 deletions(-)
diff --git a/_sources/user-guide/latest/configs.md.txt
b/_sources/user-guide/latest/configs.md.txt
index ef766171db..eeecf2cb4b 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -31,7 +31,7 @@ Comet provides the following configuration settings.
| `spark.comet.scan.icebergNative.dataFileConcurrencyLimit` | The number of
Iceberg data files to read concurrently within a single task. Higher values
improve throughput for tables with many small files by overlapping I/O latency,
but increase memory usage. Values between 2 and 8 are suggested. | 1 |
| `spark.comet.scan.icebergNative.enabled` | Whether to enable native Iceberg
table scan using iceberg-rust. When enabled, Iceberg tables are read directly
through native execution, bypassing Spark's DataSource V2 API for better
performance. | true |
| `spark.comet.scan.unsignedSmallIntSafetyCheck` | Parquet files may contain
unsigned 8-bit integers (UINT_8) which Spark maps to ShortType. When this
config is true (default), Comet falls back to Spark for ShortType columns
because we cannot distinguish signed INT16 (safe) from unsigned UINT_8 (may
produce different results). Set to false to allow native execution of ShortType
columns if you know your data does not contain unsigned UINT_8 columns from
improperly encoded Parquet files. F [...]
-| `spark.hadoop.fs.comet.libhdfs.schemes` | Defines filesystem schemes (e.g.,
hdfs, webhdfs) that the native side accesses via libhdfs, separated by commas.
Valid only when built with hdfs feature enabled. | |
+| `spark.hadoop.fs.comet.libhdfs.schemes` | Defines filesystem schemes (e.g.,
hdfs, webhdfs) that the native side accesses via libhdfs, separated by commas.
Valid only when built with hdfs-opendal feature enabled. | |
<!-- prettier-ignore-end -->
<!--END:CONFIG_TABLE-->
diff --git a/_sources/user-guide/latest/datasources.md.txt
b/_sources/user-guide/latest/datasources.md.txt
index adcb1a8f86..9d68748a75 100644
--- a/_sources/user-guide/latest/datasources.md.txt
+++ b/_sources/user-guide/latest/datasources.md.txt
@@ -71,7 +71,7 @@ Provide the JRE linker path in `RUSTFLAGS`, the path can vary
depending on the s
```shell
export JAVA_HOME="/opt/homebrew/opt/openjdk@17"
-make release PROFILES="-Pspark-4.1" COMET_FEATURES=hdfs RUSTFLAGS="-L
$JAVA_HOME/libexec/openjdk.jdk/Contents/Home/lib/server"
+make release PROFILES="-Pspark-4.1" RUSTFLAGS="-L
$JAVA_HOME/libexec/openjdk.jdk/Contents/Home/lib/server"
```
Start Comet with HDFS support as
[described](installation.md/#run-spark-shell-with-comet-enabled)
@@ -110,7 +110,7 @@ Arguments: [id#0, first_name#1, personal_info#4]
Input [3]: [id#0, first_name#1, personal_info#4]
-25/01/30 16:50:44 INFO fs-hdfs-0.1.12/src/hdfs.rs: Connecting to Namenode
(hdfs://namenode:9000)
+25/01/30 16:50:44 INFO opendal::services::hdfs: Connecting to Namenode
(hdfs://namenode:9000)
+---+----------+-----------------+
|id |first_name|personal_info |
+---+----------+-----------------+
@@ -124,8 +124,6 @@ Input [3]: [id#0, first_name#1, personal_info#4]
Verify the scan type should be `CometNativeScan`.
-More on [HDFS
Reader](https://github.com/apache/datafusion-comet/blob/main/native/hdfs/README.md)
-
### Local HDFS development
- Configure local machine network. Add hostname to `/etc/hosts`
@@ -145,7 +143,7 @@ docker compose -f kube/local/hdfs-docker-compose.yml up
- Build a project with HDFS support
```shell
-JAVA_HOME="/opt/homebrew/opt/openjdk@17" make release PROFILES="-Pspark-4.1"
COMET_FEATURES=hdfs RUSTFLAGS="-L
/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home/lib/server"
+JAVA_HOME="/opt/homebrew/opt/openjdk@17" make release PROFILES="-Pspark-4.1"
RUSTFLAGS="-L
/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home/lib/server"
```
- Run local test
diff --git a/searchindex.js b/searchindex.js
index 889cf0f084..1de3872ee8 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[51, "id1"]], "%": [[49, "id1"]], "&":
[[39, "id1"]], "*": [[49, "id2"]], "+": [[49, "id3"]], "-": [[49, "id4"]], "/":
[[49, "id5"]], "1. Format Your Code": [[36, "format-your-code"]], "1. Install
Comet": [[57, "install-comet"], [66, "install-comet"]], "1. Native Operators
(nativeExecs map)": [[25, "native-operators-nativeexecs-map"]], "2. Build and
Verify": [[36, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[57,
"clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[51, "id1"]], "%": [[49, "id1"]], "&":
[[39, "id1"]], "*": [[49, "id2"]], "+": [[49, "id3"]], "-": [[49, "id4"]], "/":
[[49, "id5"]], "1. Format Your Code": [[36, "format-your-code"]], "1. Install
Comet": [[57, "install-comet"], [66, "install-comet"]], "1. Native Operators
(nativeExecs map)": [[25, "native-operators-nativeexecs-map"]], "2. Build and
Verify": [[36, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[57,
"clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index 3a30d639c0..9114bd6bfb 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -771,7 +771,7 @@ under the License.
<td><p>true</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.hadoop.fs.comet.libhdfs.schemes</span></code></p></td>
-<td><p>Defines filesystem schemes (e.g., hdfs, webhdfs) that the native side
accesses via libhdfs, separated by commas. Valid only when built with hdfs
feature enabled.</p></td>
+<td><p>Defines filesystem schemes (e.g., hdfs, webhdfs) that the native side
accesses via libhdfs, separated by commas. Valid only when built with
hdfs-opendal feature enabled.</p></td>
<td><p></p></td>
</tr>
</tbody>
diff --git a/user-guide/latest/datasources.html
b/user-guide/latest/datasources.html
index b50b87f153..99b0530659 100644
--- a/user-guide/latest/datasources.html
+++ b/user-guide/latest/datasources.html
@@ -788,7 +788,7 @@ converted into Arrow format, allowing the Comet pipeline to
take over after that
Build a Comet for <code class="docutils literal notranslate"><span
class="pre">spark-4.1</span></code> provide a JDK path in <code class="docutils
literal notranslate"><span class="pre">JAVA_HOME</span></code>
Provide the JRE linker path in <code class="docutils literal
notranslate"><span class="pre">RUSTFLAGS</span></code>, the path can vary
depending on the system. Typically JRE linker is a part of installed JDK</p>
<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span><span class="nb">export</span><span
class="w"> </span><span class="nv">JAVA_HOME</span><span
class="o">=</span><span
class="s2">"/opt/homebrew/opt/openjdk@17"</span>
-make<span class="w"> </span>release<span class="w"> </span><span
class="nv">PROFILES</span><span class="o">=</span><span
class="s2">"-Pspark-4.1"</span><span class="w"> </span><span
class="nv">COMET_FEATURES</span><span class="o">=</span>hdfs<span class="w">
</span><span class="nv">RUSTFLAGS</span><span class="o">=</span><span
class="s2">"-L </span><span class="nv">$JAVA_HOME</span><span
class="s2">/libexec/openjdk.jdk/Contents/Home/lib/server"</span>
+make<span class="w"> </span>release<span class="w"> </span><span
class="nv">PROFILES</span><span class="o">=</span><span
class="s2">"-Pspark-4.1"</span><span class="w"> </span><span
class="nv">RUSTFLAGS</span><span class="o">=</span><span class="s2">"-L
</span><span class="nv">$JAVA_HOME</span><span
class="s2">/libexec/openjdk.jdk/Contents/Home/lib/server"</span>
</pre></div>
</div>
<p>Start Comet with HDFS support as <a class="reference internal"
href="installation.html#run-spark-shell-with-comet-enabled"><span class="std
std-ref">described</span></a>
@@ -823,7 +823,7 @@ Arguments:<span class="w"> </span><span
class="o">[</span>id#0,<span class="w">
Input<span class="w"> </span><span class="o">[</span><span
class="m">3</span><span class="o">]</span>:<span class="w"> </span><span
class="o">[</span>id#0,<span class="w"> </span>first_name#1,<span class="w">
</span>personal_info#4<span class="o">]</span>
-<span class="m">25</span>/01/30<span class="w"> </span><span
class="m">16</span>:50:44<span class="w"> </span>INFO<span class="w">
</span>fs-hdfs-0.1.12/src/hdfs.rs:<span class="w"> </span>Connecting<span
class="w"> </span>to<span class="w"> </span>Namenode<span class="w">
</span><span class="o">(</span>hdfs://namenode:9000<span class="o">)</span>
+<span class="m">25</span>/01/30<span class="w"> </span><span
class="m">16</span>:50:44<span class="w"> </span>INFO<span class="w">
</span>opendal::services::hdfs:<span class="w"> </span>Connecting<span
class="w"> </span>to<span class="w"> </span>Namenode<span class="w">
</span><span class="o">(</span>hdfs://namenode:9000<span class="o">)</span>
+---+----------+-----------------+
<span class="p">|</span>id<span class="w"> </span><span
class="p">|</span>first_name<span class="p">|</span>personal_info<span
class="w"> </span><span class="p">|</span>
+---+----------+-----------------+
@@ -833,7 +833,6 @@ Input<span class="w"> </span><span class="o">[</span><span
class="m">3</span><sp
</pre></div>
</div>
<p>Verify the scan type should be <code class="docutils literal
notranslate"><span class="pre">CometNativeScan</span></code>.</p>
-<p>More on <a class="reference external"
href="https://github.com/apache/datafusion-comet/blob/main/native/hdfs/README.md";>HDFS
Reader</a></p>
</section>
<section id="local-hdfs-development">
<h3>Local HDFS development<a class="headerlink" href="#local-hdfs-development"
title="Link to this heading">#</a></h3>
@@ -854,7 +853,7 @@ Input<span class="w"> </span><span class="o">[</span><span
class="m">3</span><sp
<li><p>Check the local namenode is up and running on <code class="docutils
literal notranslate"><span
class="pre">http://localhost:9870/dfshealth.html#tab-overview</span></code></p></li>
<li><p>Build a project with HDFS support</p></li>
</ul>
-<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span><span class="nv">JAVA_HOME</span><span
class="o">=</span><span
class="s2">"/opt/homebrew/opt/openjdk@17"</span><span class="w">
</span>make<span class="w"> </span>release<span class="w"> </span><span
class="nv">PROFILES</span><span class="o">=</span><span
class="s2">"-Pspark-4.1"</span><span class="w"> </span><span
class="nv">COMET_FEATURES</span><span class="o">=</span>hdfs<span class=" [...]
+<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span><span class="nv">JAVA_HOME</span><span
class="o">=</span><span
class="s2">"/opt/homebrew/opt/openjdk@17"</span><span class="w">
</span>make<span class="w"> </span>release<span class="w"> </span><span
class="nv">PROFILES</span><span class="o">=</span><span
class="s2">"-Pspark-4.1"</span><span class="w"> </span><span
class="nv">RUSTFLAGS</span><span class="o">=</span><span class="s2">" [...]
</pre></div>
</div>
<ul class="simple">
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
