This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-ballista.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 0ecfbb7f Publish built docs triggered by
46cf5d29a6423c08972e320dc258868c1ea0403a
0ecfbb7f is described below
commit 0ecfbb7f22e74fbcbd857bc500f2e97e39b74355
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Thu Dec 5 15:59:28 2024 +0000
Publish built docs triggered by 46cf5d29a6423c08972e320dc258868c1ea0403a
---
_sources/user-guide/tuning-guide.md.txt | 18 +++++++++++++-----
searchindex.js | 2 +-
user-guide/tuning-guide.html | 19 ++++++++++++++-----
3 files changed, 28 insertions(+), 11 deletions(-)
diff --git a/_sources/user-guide/tuning-guide.md.txt
b/_sources/user-guide/tuning-guide.md.txt
index b1b61240..22955b44 100644
--- a/_sources/user-guide/tuning-guide.md.txt
+++ b/_sources/user-guide/tuning-guide.md.txt
@@ -32,7 +32,7 @@ For example, if there is a table "customer" that consists of
200 Parquet files,
200 partitions and the table scan and certain subsequent operations will also
have 200 partitions. Conversely, if the
table only has a single Parquet file then there will be a single partition and
the work will not be able to scale even
if the cluster has resource available. Ballista supports repartitioning within
a query to improve parallelism.
-The configuration setting `ballista.shuffle.partitions`can be set to the
desired number of partitions. This is
+The configuration setting `datafusion.execution.target_partitions`can be set
to the desired number of partitions. This is
currently a global setting for the entire context. The default value for this
setting is 16.
Note that Ballista will never decrease the number of partitions based on this
setting and will only repartition if
@@ -41,11 +41,17 @@ the source operation has fewer partitions than this setting.
Example: Setting the desired number of shuffle partitions when creating a
context.
```rust
-let config = BallistaConfig::builder()
- .set("ballista.shuffle.partitions", "200")
- .build()?;
+use ballista::extension::{SessionConfigExt, SessionContextExt};
-let ctx = BallistaContext::remote("localhost", 50050, &config).await?;
+let session_config = SessionConfig::new_with_ballista()
+ .with_target_partitions(200);
+
+let state = SessionStateBuilder::new()
+ .with_default_features()
+ .with_config(session_config)
+ .build();
+
+let ctx: SessionContext = SessionContext::remote_with_state(&url,state).await?;
```
## Configuring Executor Concurrency Levels
@@ -75,6 +81,8 @@ processes. The default is `pull-based`.
The scheduler provides a REST API for monitoring jobs. See the
[scheduler documentation](scheduler.md) for more information.
+> This is optional scheduler feature which should be enabled with rest-api
feature
+
To download a query plan in dot format from the scheduler, submit a request to
the following API endpoint
```
diff --git a/searchindex.js b/searchindex.js
index 1be93103..87252aff 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"<a name=\"complex\"/>Run a Complex Query":
[[15, "run-a-complex-query"]], "<a name=\"docker\"/> Run Docker Container":
[[15, "run-docker-container"]], "<a name=\"hello\"/>Run a \u201cHello,
World!\u201d Query": [[15, "run-a-hello-world-query"]], "<a
name=\"jdbc\"/>Download the FlightSQL JDBC Driver": [[15,
"download-the-flightsql-jdbc-driver"]], "<a name=\"prereq\"/>Prerequisites":
[[15, "prerequisites"]], "<a name=\"tool\"/>Use the Driver in your Favorite
[...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"<a name=\"complex\"/>Run a Complex Query":
[[15, "run-a-complex-query"]], "<a name=\"docker\"/> Run Docker Container":
[[15, "run-docker-container"]], "<a name=\"hello\"/>Run a \u201cHello,
World!\u201d Query": [[15, "run-a-hello-world-query"]], "<a
name=\"jdbc\"/>Download the FlightSQL JDBC Driver": [[15,
"download-the-flightsql-jdbc-driver"]], "<a name=\"prereq\"/>Prerequisites":
[[15, "prerequisites"]], "<a name=\"tool\"/>Use the Driver in your Favorite
[...]
\ No newline at end of file
diff --git a/user-guide/tuning-guide.html b/user-guide/tuning-guide.html
index 1e0149d9..60be94d2 100644
--- a/user-guide/tuning-guide.html
+++ b/user-guide/tuning-guide.html
@@ -339,16 +339,22 @@ single partition (in the future, Ballista will support
splitting files into part
200 partitions and the table scan and certain subsequent operations will also
have 200 partitions. Conversely, if the
table only has a single Parquet file then there will be a single partition and
the work will not be able to scale even
if the cluster has resource available. Ballista supports repartitioning within
a query to improve parallelism.
-The configuration setting <code class="docutils literal notranslate"><span
class="pre">ballista.shuffle.partitions</span></code>can be set to the desired
number of partitions. This is
+The configuration setting <code class="docutils literal notranslate"><span
class="pre">datafusion.execution.target_partitions</span></code>can be set to
the desired number of partitions. This is
currently a global setting for the entire context. The default value for this
setting is 16.</p>
<p>Note that Ballista will never decrease the number of partitions based on
this setting and will only repartition if
the source operation has fewer partitions than this setting.</p>
<p>Example: Setting the desired number of shuffle partitions when creating a
context.</p>
-<div class="highlight-rust notranslate"><div
class="highlight"><pre><span></span><span class="kd">let</span><span class="w">
</span><span class="n">config</span><span class="w"> </span><span
class="o">=</span><span class="w"> </span><span
class="n">BallistaConfig</span><span class="p">::</span><span
class="n">builder</span><span class="p">()</span>
-<span class="w"> </span><span class="p">.</span><span
class="n">set</span><span class="p">(</span><span
class="s">"ballista.shuffle.partitions"</span><span
class="p">,</span><span class="w"> </span><span
class="s">"200"</span><span class="p">)</span>
-<span class="w"> </span><span class="p">.</span><span
class="n">build</span><span class="p">()</span><span class="o">?</span><span
class="p">;</span>
+<div class="highlight-rust notranslate"><div
class="highlight"><pre><span></span><span class="k">use</span><span class="w">
</span><span class="n">ballista</span><span class="p">::</span><span
class="n">extension</span><span class="p">::{</span><span
class="n">SessionConfigExt</span><span class="p">,</span><span class="w">
</span><span class="n">SessionContextExt</span><span class="p">};</span>
-<span class="kd">let</span><span class="w"> </span><span
class="n">ctx</span><span class="w"> </span><span class="o">=</span><span
class="w"> </span><span class="n">BallistaContext</span><span
class="p">::</span><span class="n">remote</span><span class="p">(</span><span
class="s">"localhost"</span><span class="p">,</span><span class="w">
</span><span class="mi">50050</span><span class="p">,</span><span class="w">
</span><span class="o">&</span><span class="n">config</span>< [...]
+<span class="kd">let</span><span class="w"> </span><span
class="n">session_config</span><span class="w"> </span><span
class="o">=</span><span class="w"> </span><span
class="n">SessionConfig</span><span class="p">::</span><span
class="n">new_with_ballista</span><span class="p">()</span>
+<span class="w"> </span><span class="p">.</span><span
class="n">with_target_partitions</span><span class="p">(</span><span
class="mi">200</span><span class="p">);</span>
+
+<span class="kd">let</span><span class="w"> </span><span
class="n">state</span><span class="w"> </span><span class="o">=</span><span
class="w"> </span><span class="n">SessionStateBuilder</span><span
class="p">::</span><span class="n">new</span><span class="p">()</span>
+<span class="w"> </span><span class="p">.</span><span
class="n">with_default_features</span><span class="p">()</span>
+<span class="w"> </span><span class="p">.</span><span
class="n">with_config</span><span class="p">(</span><span
class="n">session_config</span><span class="p">)</span>
+<span class="w"> </span><span class="p">.</span><span
class="n">build</span><span class="p">();</span>
+
+<span class="kd">let</span><span class="w"> </span><span
class="n">ctx</span><span class="p">:</span><span class="w"> </span><span
class="nc">SessionContext</span><span class="w"> </span><span
class="o">=</span><span class="w"> </span><span
class="n">SessionContext</span><span class="p">::</span><span
class="n">remote_with_state</span><span class="p">(</span><span
class="o">&</span><span class="n">url</span><span class="p">,</span><span
class="n">state</span><span class="p">).</span> [...]
</pre></div>
</div>
</section>
@@ -374,6 +380,9 @@ processes. The default is <code class="docutils literal
notranslate"><span class
<h2>Viewing Query Plans and Metrics<a class="headerlink"
href="#viewing-query-plans-and-metrics" title="Link to this heading">ΒΆ</a></h2>
<p>The scheduler provides a REST API for monitoring jobs. See the
<a class="reference internal" href="scheduler.html"><span class="std
std-doc">scheduler documentation</span></a> for more information.</p>
+<blockquote>
+<div><p>This is optional scheduler feature which should be enabled with
rest-api feature</p>
+</div></blockquote>
<p>To download a query plan in dot format from the scheduler, submit a request
to the following API endpoint</p>
<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="n">http</span><span
class="p">:</span><span class="o">//</span><span
class="n">localhost</span><span class="p">:</span><span
class="mi">50050</span><span class="o">/</span><span class="n">api</span><span
class="o">/</span><span class="n">job</span><span class="o">/</span><span
class="p">{</span><span class="n">job_id</span><span class="p">}</span><span
class="o">/</span><span class="n">dot</span>
</pre></div>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
