This is an automated email from the ASF dual-hosted git repository.
alamb pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/arrow-site.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 15b51915080 DataFusion 18 docs (#313)
15b51915080 is described below
commit 15b51915080cda587b4129410caef7c77e489360
Author: Andy Grove <[email protected]>
AuthorDate: Mon Feb 13 11:46:11 2023 -0700
DataFusion 18 docs (#313)
---
datafusion/searchindex.js | 2 +-
datafusion/user-guide/configs.html | 36 ++++++++++++++++++++++++++--------
datafusion/user-guide/dataframe.html | 21 +++++++++++---------
datafusion/user-guide/expressions.html | 4 ++--
4 files changed, 43 insertions(+), 20 deletions(-)
diff --git a/datafusion/searchindex.js b/datafusion/searchindex.js
index 84ecfff9ed2..8410fe40b0c 100644
--- a/datafusion/searchindex.js
+++ b/datafusion/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"docnames": ["contributor-guide/communication",
"contributor-guide/index", "contributor-guide/quarterly_roadmap",
"contributor-guide/roadmap", "contributor-guide/specification/index",
"contributor-guide/specification/invariants",
"contributor-guide/specification/output-field-name-semantic", "index",
"user-guide/cli", "user-guide/configs", "user-guide/dataframe",
"user-guide/example-usage", "user-guide/expressions", "user-guide/faq",
"user-guide/introduction", "user-guide [...]
\ No newline at end of file
+Search.setIndex({"docnames": ["contributor-guide/communication",
"contributor-guide/index", "contributor-guide/quarterly_roadmap",
"contributor-guide/roadmap", "contributor-guide/specification/index",
"contributor-guide/specification/invariants",
"contributor-guide/specification/output-field-name-semantic", "index",
"user-guide/cli", "user-guide/configs", "user-guide/dataframe",
"user-guide/example-usage", "user-guide/expressions", "user-guide/faq",
"user-guide/introduction", "user-guide [...]
\ No newline at end of file
diff --git a/datafusion/user-guide/configs.html
b/datafusion/user-guide/configs.html
index 9a36ed979fe..dad50b57121 100644
--- a/datafusion/user-guide/configs.html
+++ b/datafusion/user-guide/configs.html
@@ -410,42 +410,62 @@ Environment variables are read during <code
class="docutils literal notranslate"
<td><p>true</p></td>
<td><p>Should DataFusion repartition data using the aggregate keys to execute
aggregates in parallel using the provided <code class="docutils literal
notranslate"><span class="pre">target_partitions</span></code> level”</p></td>
</tr>
-<tr class="row-odd"><td><p>datafusion.optimizer.repartition_joins</p></td>
+<tr
class="row-odd"><td><p>datafusion.optimizer.repartition_file_min_size</p></td>
+<td><p>10485760</p></td>
+<td><p>Minimum total files size in bytes to perform file scan
repartitioning.</p></td>
+</tr>
+<tr class="row-even"><td><p>datafusion.optimizer.repartition_joins</p></td>
<td><p>true</p></td>
<td><p>Should DataFusion repartition data using the join keys to execute joins
in parallel using the provided <code class="docutils literal notranslate"><span
class="pre">target_partitions</span></code> level”</p></td>
</tr>
+<tr class="row-odd"><td><p>datafusion.optimizer.repartition_file_scans</p></td>
+<td><p>false</p></td>
+<td><p>When set to true, file groups will be repartitioned to achieve maximum
parallelism. Currently supported only for Parquet format in which case multiple
row groups from the same file may be read concurrently. If false then each row
group is read serially, though different files may be read in parallel.</p></td>
+</tr>
<tr class="row-even"><td><p>datafusion.optimizer.repartition_windows</p></td>
<td><p>true</p></td>
<td><p>Should DataFusion repartition data using the partitions keys to execute
window functions in parallel using the provided <code class="docutils literal
notranslate"><span class="pre">target_partitions</span></code> level”</p></td>
</tr>
-<tr class="row-odd"><td><p>datafusion.optimizer.skip_failed_rules</p></td>
+<tr class="row-odd"><td><p>datafusion.optimizer.repartition_sorts</p></td>
+<td><p>true</p></td>
+<td><p>Should DataFusion execute sorts in a per-partition fashion and merge
afterwards instead of coalescing first and sorting globally With this flag is
enabled, plans in the form below “SortExec: [a@0 ASC]”, ”
CoalescePartitionsExec”, ” RepartitionExec: partitioning=RoundRobinBatch(8),
input_partitions=1”, would turn into the plan below which performs better in
multithreaded environments “SortPreservingMergeExec: [a@0 ASC]”, ”
SortExec: [a@0 ASC]”, ” RepartitionExec: partit [...]
+</tr>
+<tr class="row-even"><td><p>datafusion.optimizer.skip_failed_rules</p></td>
<td><p>true</p></td>
<td><p>When set to true, the logical plan optimizer will produce warning
messages if any optimization rules produce errors and then proceed to the next
rule. When set to false, any rules that produce errors will cause the query to
fail</p></td>
</tr>
-<tr class="row-even"><td><p>datafusion.optimizer.max_passes</p></td>
+<tr class="row-odd"><td><p>datafusion.optimizer.max_passes</p></td>
<td><p>3</p></td>
<td><p>Number of times that the optimizer will attempt to optimize the
plan</p></td>
</tr>
-<tr
class="row-odd"><td><p>datafusion.optimizer.top_down_join_key_reordering</p></td>
+<tr
class="row-even"><td><p>datafusion.optimizer.top_down_join_key_reordering</p></td>
<td><p>true</p></td>
<td><p>When set to true, the physical plan optimizer will run a top down
process to reorder the join keys</p></td>
</tr>
-<tr class="row-even"><td><p>datafusion.optimizer.prefer_hash_join</p></td>
+<tr class="row-odd"><td><p>datafusion.optimizer.prefer_hash_join</p></td>
<td><p>true</p></td>
<td><p>When set to true, the physical plan optimizer will prefer HashJoin over
SortMergeJoin. HashJoin can work more efficiently than SortMergeJoin but
consumes more memory</p></td>
</tr>
-<tr
class="row-odd"><td><p>datafusion.optimizer.hash_join_single_partition_threshold</p></td>
+<tr
class="row-even"><td><p>datafusion.optimizer.hash_join_single_partition_threshold</p></td>
<td><p>1048576</p></td>
<td><p>The maximum estimated size in bytes for one input side of a HashJoin
will be collected into a single partition</p></td>
</tr>
-<tr class="row-even"><td><p>datafusion.explain.logical_plan_only</p></td>
+<tr class="row-odd"><td><p>datafusion.explain.logical_plan_only</p></td>
<td><p>false</p></td>
<td><p>When set to true, the explain statement will only print logical
plans</p></td>
</tr>
-<tr class="row-odd"><td><p>datafusion.explain.physical_plan_only</p></td>
+<tr class="row-even"><td><p>datafusion.explain.physical_plan_only</p></td>
<td><p>false</p></td>
<td><p>When set to true, the explain statement will only print physical
plans</p></td>
</tr>
+<tr
class="row-odd"><td><p>datafusion.sql_parser.parse_float_as_decimal</p></td>
+<td><p>false</p></td>
+<td><p>When set to true, sql parser will parse float as decimal type</p></td>
+</tr>
+<tr
class="row-even"><td><p>datafusion.sql_parser.enable_ident_normalization</p></td>
+<td><p>true</p></td>
+<td><p>When set to true, sql parser will normalize ident(convert ident to
lowercase when not quoted)</p></td>
+</tr>
</tbody>
</table>
</section>
diff --git a/datafusion/user-guide/dataframe.html
b/datafusion/user-guide/dataframe.html
index 39af087c84e..7e1e20c1e14 100644
--- a/datafusion/user-guide/dataframe.html
+++ b/datafusion/user-guide/dataframe.html
@@ -380,31 +380,34 @@ execution. The plan is evaluated (executed) when an
action method is invoked, su
<tr class="row-odd"><td><p>join</p></td>
<td><p>Join this DataFrame with another DataFrame using the specified columns
as join keys.</p></td>
</tr>
-<tr class="row-even"><td><p>limit</p></td>
+<tr class="row-even"><td><p>join_on</p></td>
+<td><p>Join this DataFrame with another DataFrame using arbitrary
expressions.</p></td>
+</tr>
+<tr class="row-odd"><td><p>limit</p></td>
<td><p>Limit the number of rows returned from this DataFrame.</p></td>
</tr>
-<tr class="row-odd"><td><p>repartition</p></td>
+<tr class="row-even"><td><p>repartition</p></td>
<td><p>Repartition a DataFrame based on a logical partitioning scheme.</p></td>
</tr>
-<tr class="row-even"><td><p>sort</p></td>
+<tr class="row-odd"><td><p>sort</p></td>
<td><p>Sort the DataFrame by the specified sorting expressions. Any expression
can be turned into a sort expression by calling its <code class="docutils
literal notranslate"><span class="pre">sort</span></code> method.</p></td>
</tr>
-<tr class="row-odd"><td><p>select</p></td>
+<tr class="row-even"><td><p>select</p></td>
<td><p>Create a projection based on arbitrary expressions. Example: <code
class="docutils literal notranslate"><span
class="pre">df..select(vec![col("c1"),</span> <span
class="pre">abs(col("c2"))])?</span></code></p></td>
</tr>
-<tr class="row-even"><td><p>select_columns</p></td>
+<tr class="row-odd"><td><p>select_columns</p></td>
<td><p>Create a projection based on column names. Example: <code
class="docutils literal notranslate"><span
class="pre">df.select_columns(&["id",</span> <span
class="pre">"name"])?</span></code>.</p></td>
</tr>
-<tr class="row-odd"><td><p>union</p></td>
+<tr class="row-even"><td><p>union</p></td>
<td><p>Calculate the union of two DataFrames, preserving duplicate rows. The
two DataFrames must have exactly the same schema.</p></td>
</tr>
-<tr class="row-even"><td><p>union_distinct</p></td>
+<tr class="row-odd"><td><p>union_distinct</p></td>
<td><p>Calculate the distinct union of two DataFrames. The two DataFrames must
have exactly the same schema.</p></td>
</tr>
-<tr class="row-odd"><td><p>with_column</p></td>
+<tr class="row-even"><td><p>with_column</p></td>
<td><p>Add an additional column to the DataFrame.</p></td>
</tr>
-<tr class="row-even"><td><p>with_column_renamed</p></td>
+<tr class="row-odd"><td><p>with_column_renamed</p></td>
<td><p>Rename one column by applying a new projection.</p></td>
</tr>
</tbody>
diff --git a/datafusion/user-guide/expressions.html
b/datafusion/user-guide/expressions.html
index deed5a6ef78..3465eeaa35e 100644
--- a/datafusion/user-guide/expressions.html
+++ b/datafusion/user-guide/expressions.html
@@ -383,8 +383,8 @@
<p>DataFrame methods such as <code class="docutils literal notranslate"><span
class="pre">select</span></code> and <code class="docutils literal
notranslate"><span class="pre">filter</span></code> accept one or more logical
expressions and there are many functions
available for creating logical expressions. These are documented below.</p>
<p>Expressions can be chained together using a fluent-style API:</p>
-<div class="highlight-rust notranslate"><div
class="highlight"><pre><span></span><span class="c1">// create the expression
`(a > 5) AND (b < 7)`</span>
-<span class="n">col</span><span class="p">(</span><span
class="s">"a"</span><span class="p">).</span><span
class="n">gt</span><span class="p">(</span><span class="n">lit</span><span
class="p">(</span><span class="mi">5</span><span class="p">)).</span><span
class="n">and</span><span class="p">(</span><span class="n">col</span><span
class="p">(</span><span class="s">"b"</span><span
class="p">).</span><span class="n">lt</span><span class="p">(</span><span
class="n">lit</ [...]
+<div class="highlight-rust notranslate"><div
class="highlight"><pre><span></span><span class="c1">// create the expression
`(a > 6) AND (b < 7)`</span>
+<span class="n">col</span><span class="p">(</span><span
class="s">"a"</span><span class="p">).</span><span
class="n">gt</span><span class="p">(</span><span class="n">lit</span><span
class="p">(</span><span class="mi">6</span><span class="p">)).</span><span
class="n">and</span><span class="p">(</span><span class="n">col</span><span
class="p">(</span><span class="s">"b"</span><span
class="p">).</span><span class="n">lt</span><span class="p">(</span><span
class="n">lit</ [...]
</pre></div>
</div>
<section id="identifiers">
