This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 84e4c13de Publish built docs triggered by
e823163100dfa72ed48a142bc2950395f458ac69
84e4c13de is described below
commit 84e4c13de32df64c2b7093bb0cd1dc3383af52d4
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Tue Apr 22 23:06:43 2025 +0000
Publish built docs triggered by e823163100dfa72ed48a142bc2950395f458ac69
---
_sources/user-guide/datasources.md.txt | 36 ++++++++++++++++++++++++++
searchindex.js | 2 +-
user-guide/datasources.html | 46 ++++++++++++++++++++++++++++++++++
3 files changed, 83 insertions(+), 1 deletion(-)
diff --git a/_sources/user-guide/datasources.md.txt
b/_sources/user-guide/datasources.md.txt
index 5634df8e0..1e5a7df24 100644
--- a/_sources/user-guide/datasources.md.txt
+++ b/_sources/user-guide/datasources.md.txt
@@ -111,5 +111,41 @@ Verify the native scan type should be `CometNativeScan`.
More on [HDFS Reader](../../../native/hdfs/README.md)
+### Local HDFS development
+
+- Configure local machine network. Add hostname to `/etc/hosts`
+```commandline
+127.0.0.1 localhost namenode datanode1 datanode2 datanode3
+::1 localhost namenode datanode1 datanode2 datanode3
+```
+
+- Start local HDFS cluster, 3 datanodes, namenode url is `namenode:9000`
+```commandline
+docker compose -f kube/local/hdfs-docker-compose.yml up
+```
+
+- Check the local namenode is up and running on
`http://localhost:9870/dfshealth.html#tab-overview`
+- Build a project with HDFS support
+```commandline
+JAVA_HOME="/opt/homebrew/opt/openjdk@11" make release PROFILES="-Pspark-3.5"
COMET_FEATURES=hdfs RUSTFLAGS="-L
/opt/homebrew/opt/openjdk@11/libexec/openjdk.jdk/Contents/Home/lib/server"
+```
+
+- Run local test
+```scala
+
+ withSQLConf(
+ CometConf.COMET_ENABLED.key -> "true",
+ CometConf.COMET_EXEC_ENABLED.key -> "true",
+ CometConf.COMET_NATIVE_SCAN_IMPL.key -> CometConf.SCAN_NATIVE_DATAFUSION,
+ SQLConf.USE_V1_SOURCE_LIST.key -> "parquet",
+ "fs.defaultFS" -> "hdfs://namenode:9000",
+ "dfs.client.use.datanode.hostname" -> "true") {
+ val df = spark.read.parquet("/tmp/2")
+ df.show(false)
+ df.explain("extended")
+ }
+ }
+```
+Or use `spark-shell` with HDFS support as described
[above](#using-experimental-native-datafusion-reader)
## S3
In progress
diff --git a/searchindex.js b/searchindex.js
index 9b8b86d69..96e10d2f9 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Install Comet": [[10, "install-comet"]],
"2. Clone Spark and Apply Diff": [[10, "clone-spark-and-apply-diff"]], "3. Run
Spark SQL Tests": [[10, "run-spark-sql-tests"]], "ANSI mode": [[12,
"ansi-mode"]], "API Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "ASF Links": [[11, null]], "Adding
Spark-side Tests for the New Expression": [[0,
"adding-spark-side-tests-for-the-new-expression"]], "Adding a New Expression":
[[ [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Install Comet": [[10, "install-comet"]],
"2. Clone Spark and Apply Diff": [[10, "clone-spark-and-apply-diff"]], "3. Run
Spark SQL Tests": [[10, "run-spark-sql-tests"]], "ANSI mode": [[12,
"ansi-mode"]], "API Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "ASF Links": [[11, null]], "Adding
Spark-side Tests for the New Expression": [[0,
"adding-spark-side-tests-for-the-new-expression"]], "Adding a New Expression":
[[ [...]
\ No newline at end of file
diff --git a/user-guide/datasources.html b/user-guide/datasources.html
index 332fc0470..5c8814bec 100644
--- a/user-guide/datasources.html
+++ b/user-guide/datasources.html
@@ -328,6 +328,11 @@ under the License.
Using experimental native DataFusion reader
</a>
</li>
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#local-hdfs-development">
+ Local HDFS development
+ </a>
+ </li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
@@ -468,6 +473,47 @@ Input<span class="w"> </span><span class="o">[</span><span
class="m">3</span><sp
<p>Verify the native scan type should be <code class="docutils literal
notranslate"><span class="pre">CometNativeScan</span></code>.</p>
<p>More on <span class="xref myst">HDFS Reader</span></p>
</section>
+<section id="local-hdfs-development">
+<h3>Local HDFS development<a class="headerlink" href="#local-hdfs-development"
title="Link to this heading">¶</a></h3>
+<ul class="simple">
+<li><p>Configure local machine network. Add hostname to <code class="docutils
literal notranslate"><span class="pre">/etc/hosts</span></code></p></li>
+</ul>
+<div class="highlight-commandline notranslate"><div
class="highlight"><pre><span></span>127.0.0.1 localhost namenode
datanode1 datanode2 datanode3
+::1 localhost namenode datanode1 datanode2 datanode3
+</pre></div>
+</div>
+<ul class="simple">
+<li><p>Start local HDFS cluster, 3 datanodes, namenode url is <code
class="docutils literal notranslate"><span
class="pre">namenode:9000</span></code></p></li>
+</ul>
+<div class="highlight-commandline notranslate"><div
class="highlight"><pre><span></span>docker compose -f
kube/local/hdfs-docker-compose.yml up
+</pre></div>
+</div>
+<ul class="simple">
+<li><p>Check the local namenode is up and running on <code class="docutils
literal notranslate"><span
class="pre">http://localhost:9870/dfshealth.html#tab-overview</span></code></p></li>
+<li><p>Build a project with HDFS support</p></li>
+</ul>
+<div class="highlight-commandline notranslate"><div
class="highlight"><pre><span></span>JAVA_HOME="/opt/homebrew/opt/openjdk@11"
make release PROFILES="-Pspark-3.5" COMET_FEATURES=hdfs
RUSTFLAGS="-L
/opt/homebrew/opt/openjdk@11/libexec/openjdk.jdk/Contents/Home/lib/server"
+</pre></div>
+</div>
+<ul class="simple">
+<li><p>Run local test</p></li>
+</ul>
+<div class="highlight-scala notranslate"><div
class="highlight"><pre><span></span><span class="w"> </span><span
class="n">withSQLConf</span><span class="p">(</span>
+<span class="w"> </span><span class="nc">CometConf</span><span
class="p">.</span><span class="nc">COMET_ENABLED</span><span
class="p">.</span><span class="n">key</span><span class="w"> </span><span
class="o">-></span><span class="w"> </span><span
class="s">"true"</span><span class="p">,</span>
+<span class="w"> </span><span class="nc">CometConf</span><span
class="p">.</span><span class="nc">COMET_EXEC_ENABLED</span><span
class="p">.</span><span class="n">key</span><span class="w"> </span><span
class="o">-></span><span class="w"> </span><span
class="s">"true"</span><span class="p">,</span>
+<span class="w"> </span><span class="nc">CometConf</span><span
class="p">.</span><span class="nc">COMET_NATIVE_SCAN_IMPL</span><span
class="p">.</span><span class="n">key</span><span class="w"> </span><span
class="o">-></span><span class="w"> </span><span
class="nc">CometConf</span><span class="p">.</span><span
class="nc">SCAN_NATIVE_DATAFUSION</span><span class="p">,</span>
+<span class="w"> </span><span class="nc">SQLConf</span><span
class="p">.</span><span class="nc">USE_V1_SOURCE_LIST</span><span
class="p">.</span><span class="n">key</span><span class="w"> </span><span
class="o">-></span><span class="w"> </span><span
class="s">"parquet"</span><span class="p">,</span>
+<span class="w"> </span><span
class="s">"fs.defaultFS"</span><span class="w"> </span><span
class="o">-></span><span class="w"> </span><span
class="s">"hdfs://namenode:9000"</span><span class="p">,</span>
+<span class="w"> </span><span
class="s">"dfs.client.use.datanode.hostname"</span><span class="w">
</span><span class="o">-></span><span class="w"> </span><span
class="s">"true"</span><span class="p">)</span><span class="w">
</span><span class="p">{</span>
+<span class="w"> </span><span class="kd">val</span><span class="w">
</span><span class="n">df</span><span class="w"> </span><span
class="o">=</span><span class="w"> </span><span class="n">spark</span><span
class="p">.</span><span class="n">read</span><span class="p">.</span><span
class="n">parquet</span><span class="p">(</span><span
class="s">"/tmp/2"</span><span class="p">)</span>
+<span class="w"> </span><span class="n">df</span><span
class="p">.</span><span class="n">show</span><span class="p">(</span><span
class="kc">false</span><span class="p">)</span>
+<span class="w"> </span><span class="n">df</span><span
class="p">.</span><span class="n">explain</span><span class="p">(</span><span
class="s">"extended"</span><span class="p">)</span>
+<span class="w"> </span><span class="p">}</span>
+<span class="w"> </span><span class="p">}</span>
+</pre></div>
+</div>
+<p>Or use <code class="docutils literal notranslate"><span
class="pre">spark-shell</span></code> with HDFS support as described <a
class="reference internal"
href="#using-experimental-native-datafusion-reader">above</a></p>
+</section>
</section>
<section id="s3">
<h2>S3<a class="headerlink" href="#s3" title="Link to this heading">¶</a></h2>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
