This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion.git
The following commit(s) were added to refs/heads/asf-site by this push:
new eb208989aa Publish built docs triggered by
66b059e041e332293c21bde857865cfdf04f0772
eb208989aa is described below
commit eb208989aa51590918387d1355acec43ea8b3786
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Thu Jul 9 21:29:49 2026 +0000
Publish built docs triggered by 66b059e041e332293c21bde857865cfdf04f0772
---
.../library-user-guide/upgrading/55.0.0.md.txt | 41 ++++++++++++++++++++++
_sources/user-guide/configs.md.txt | 2 +-
library-user-guide/upgrading/55.0.0.html | 38 ++++++++++++++++++++
searchindex.js | 2 +-
user-guide/configs.html | 2 +-
5 files changed, 82 insertions(+), 3 deletions(-)
diff --git a/_sources/library-user-guide/upgrading/55.0.0.md.txt
b/_sources/library-user-guide/upgrading/55.0.0.md.txt
index 26a26b69b5..181e0e0b7f 100644
--- a/_sources/library-user-guide/upgrading/55.0.0.md.txt
+++ b/_sources/library-user-guide/upgrading/55.0.0.md.txt
@@ -67,6 +67,47 @@ let df = df.fill_null(&ScalarValue::from(0), &["a", "c"])?;
let df = df.fill_null(&ScalarValue::from(0), &[])?;
```
+### `FileScanConfig::partitioned_by_file_group` removed
+
+`FileScanConfig::partitioned_by_file_group` and
+`FileScanConfigBuilder::with_partitioned_by_file_group(...)` have been removed.
+Use `FileScanConfig::output_partitioning` and
+`FileScanConfigBuilder::with_output_partitioning(...)` instead.
+
+**Who is affected:**
+
+- Users who accessed `FileScanConfig::partitioned_by_file_group` directly.
+- Users who called
+ `FileScanConfigBuilder::with_partitioned_by_file_group(true)`.
+
+**Migration guide:**
+
+If your file groups are organized by table partition column values, declare
hash
+output partitioning over those partition columns:
+
+```rust,ignore
+use datafusion_datasource::file_scan_config::{
+ FileScanConfigBuilder, output_partitioning_from_partition_fields,
+};
+
+let output_partitioning = output_partitioning_from_partition_fields(
+ source.table_schema().table_schema(),
+ source.table_schema().table_partition_cols(),
+ file_groups.len(),
+);
+
+let config = FileScanConfigBuilder::new(object_store_url, source)
+ .with_file_groups(file_groups)
+ .with_output_partitioning(output_partitioning)
+ .build();
+```
+
+`output_partitioning_from_partition_fields` returns
+`Some(Partitioning::Hash(...))` when partition columns are present and `None`
+otherwise. If you construct the partitioning manually, pass
+`Some(Partitioning::Hash(partition_exprs, partition_count))` to
+`with_output_partitioning(...)`.
+
### User `SpillFile` traits instead of [`RefCountedTempFile`]
Spill file APIs now use the `datafusion_execution::SpillFile` trait instead of
diff --git a/_sources/user-guide/configs.md.txt
b/_sources/user-guide/configs.md.txt
index 03340c366d..f6e072b59b 100644
--- a/_sources/user-guide/configs.md.txt
+++ b/_sources/user-guide/configs.md.txt
@@ -160,7 +160,7 @@ The following configuration settings are available:
| datafusion.optimizer.repartition_joins |
true | Should DataFusion repartition data using the join
keys to execute joins in parallel using the provided `target_partitions` level
[...]
| datafusion.optimizer.allow_symmetric_joins_without_pruning |
true | Should DataFusion allow symmetric hash joins for
unbounded data sources even when its inputs do not have any ordering or
filtering If the flag is not enabled, the SymmetricHashJoin operator will be
unable to prune its internal buffers, resulting in certain join types - such as
Full, Left, LeftAnti, LeftSemi, Right, RightAnti, and RightSemi - being
produced only at the end of the execut [...]
| datafusion.optimizer.repartition_file_scans |
true | When set to `true`, datasource partitions will be
repartitioned to achieve maximum parallelism. This applies to both in-memory
partitions and FileSource's file groups (1 group is 1 partition). For
FileSources, only Parquet and CSV formats are currently supported. If set to
`true` for a FileSource, all files will be repartitioned evenly (i.e., a single
large file might be partitioned in [...]
-| datafusion.optimizer.preserve_file_partitions | 0
| Minimum number of distinct partition values required
to group files by their Hive partition column values (enabling Hash
partitioning declaration). How the option is used: -
preserve_file_partitions=0: Disable it. - preserve_file_partitions=1: Always
enable it. - preserve_file_partitions=N, actual file partitions=M: Only enable
when M >= N. This threshold preserves I/O parallelism when [...]
+| datafusion.optimizer.preserve_file_partitions | 0
| Minimum number of distinct partition values required
to group files by their Hive partition column values (enabling output
partitioning declaration). How the option is used: -
preserve_file_partitions=0: Disable it. - preserve_file_partitions=1: Always
enable it. - preserve_file_partitions=N, actual file partitions=M: Only enable
when M >= N. This threshold preserves I/O parallelism wh [...]
| datafusion.optimizer.repartition_windows |
true | Should DataFusion repartition data using the
partitions keys to execute window functions in parallel using the provided
`target_partitions` level
[...]
| datafusion.optimizer.repartition_sorts |
true | Should DataFusion execute sorts in a per-partition
fashion and merge afterwards instead of coalescing first and sorting globally.
With this flag is enabled, plans in the form below `text "SortExec: [a@0 ASC]",
" CoalescePartitionsExec", " RepartitionExec: partitioning=RoundRobinBatch(8),
input_partitions=1", ` would turn into the plan below which performs better in
multithreaded enviro [...]
| datafusion.optimizer.subset_repartition_threshold | 4
| Partition count threshold for subset satisfaction
optimization. When the current partition count is >= this threshold, DataFusion
will skip repartitioning if the required partitioning expression is a subset of
the current partition expression such as Hash(a) satisfies Hash(a, b). When the
current partition count is < this threshold, DataFusion will repartition to
increase parallelism e [...]
diff --git a/library-user-guide/upgrading/55.0.0.html
b/library-user-guide/upgrading/55.0.0.html
index 832e242dae..e7262d9073 100644
--- a/library-user-guide/upgrading/55.0.0.html
+++ b/library-user-guide/upgrading/55.0.0.html
@@ -486,6 +486,43 @@ to the main branch and are awaiting release in this
version.</p>
</pre></div>
</div>
</section>
+<section id="filescanconfig-partitioned-by-file-group-removed">
+<h3><code class="docutils literal notranslate"><span
class="pre">FileScanConfig::partitioned_by_file_group</span></code> removed<a
class="headerlink" href="#filescanconfig-partitioned-by-file-group-removed"
title="Link to this heading">#</a></h3>
+<p><code class="docutils literal notranslate"><span
class="pre">FileScanConfig::partitioned_by_file_group</span></code> and
+<code class="docutils literal notranslate"><span
class="pre">FileScanConfigBuilder::with_partitioned_by_file_group(...)</span></code>
have been removed.
+Use <code class="docutils literal notranslate"><span
class="pre">FileScanConfig::output_partitioning</span></code> and
+<code class="docutils literal notranslate"><span
class="pre">FileScanConfigBuilder::with_output_partitioning(...)</span></code>
instead.</p>
+<p><strong>Who is affected:</strong></p>
+<ul class="simple">
+<li><p>Users who accessed <code class="docutils literal notranslate"><span
class="pre">FileScanConfig::partitioned_by_file_group</span></code>
directly.</p></li>
+<li><p>Users who called
+<code class="docutils literal notranslate"><span
class="pre">FileScanConfigBuilder::with_partitioned_by_file_group(true)</span></code>.</p></li>
+</ul>
+<p><strong>Migration guide:</strong></p>
+<p>If your file groups are organized by table partition column values, declare
hash
+output partitioning over those partition columns:</p>
+<div class="highlight-rust notranslate"><div
class="highlight"><pre><span></span><span class="k">use</span><span class="w">
</span><span class="n">datafusion_datasource</span><span
class="p">::</span><span class="n">file_scan_config</span><span
class="p">::{</span>
+<span class="w"> </span><span class="n">FileScanConfigBuilder</span><span
class="p">,</span><span class="w"> </span><span
class="n">output_partitioning_from_partition_fields</span><span
class="p">,</span>
+<span class="p">};</span>
+
+<span class="kd">let</span><span class="w"> </span><span
class="n">output_partitioning</span><span class="w"> </span><span
class="o">=</span><span class="w"> </span><span
class="n">output_partitioning_from_partition_fields</span><span
class="p">(</span>
+<span class="w"> </span><span class="n">source</span><span
class="p">.</span><span class="n">table_schema</span><span
class="p">().</span><span class="n">table_schema</span><span
class="p">(),</span>
+<span class="w"> </span><span class="n">source</span><span
class="p">.</span><span class="n">table_schema</span><span
class="p">().</span><span class="n">table_partition_cols</span><span
class="p">(),</span>
+<span class="w"> </span><span class="n">file_groups</span><span
class="p">.</span><span class="n">len</span><span class="p">(),</span>
+<span class="p">);</span>
+
+<span class="kd">let</span><span class="w"> </span><span
class="n">config</span><span class="w"> </span><span class="o">=</span><span
class="w"> </span><span class="n">FileScanConfigBuilder</span><span
class="p">::</span><span class="n">new</span><span class="p">(</span><span
class="n">object_store_url</span><span class="p">,</span><span class="w">
</span><span class="n">source</span><span class="p">)</span>
+<span class="w"> </span><span class="p">.</span><span
class="n">with_file_groups</span><span class="p">(</span><span
class="n">file_groups</span><span class="p">)</span>
+<span class="w"> </span><span class="p">.</span><span
class="n">with_output_partitioning</span><span class="p">(</span><span
class="n">output_partitioning</span><span class="p">)</span>
+<span class="w"> </span><span class="p">.</span><span
class="n">build</span><span class="p">();</span>
+</pre></div>
+</div>
+<p><code class="docutils literal notranslate"><span
class="pre">output_partitioning_from_partition_fields</span></code> returns
+<code class="docutils literal notranslate"><span
class="pre">Some(Partitioning::Hash(...))</span></code> when partition columns
are present and <code class="docutils literal notranslate"><span
class="pre">None</span></code>
+otherwise. If you construct the partitioning manually, pass
+<code class="docutils literal notranslate"><span
class="pre">Some(Partitioning::Hash(partition_exprs,</span> <span
class="pre">partition_count))</span></code> to
+<code class="docutils literal notranslate"><span
class="pre">with_output_partitioning(...)</span></code>.</p>
+</section>
<section id="user-spillfile-traits-instead-of-refcountedtempfile">
<h3>User <code class="docutils literal notranslate"><span
class="pre">SpillFile</span></code> traits instead of <a class="reference
external"
href="https://docs.rs/datafusion-execution/latest/datafusion_execution/disk_manager/struct.RefCountedTempFile.html";><code
class="docutils literal notranslate"><span
class="pre">RefCountedTempFile</span></code></a><a class="headerlink"
href="#user-spillfile-traits-instead-of-refcountedtempfile" title="Link to this
heading">#</a></h3>
<p>Spill file APIs now use the <code class="docutils literal
notranslate"><span class="pre">datafusion_execution::SpillFile</span></code>
trait instead of
@@ -954,6 +991,7 @@ metadata and schema fingerprint match.</p>
<ul class="pst-show_toc_level nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#datafusion-55-0-0">DataFusion 55.0.0</a><ul class="nav section-nav
flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#dataframe-fill-null-now-borrows-its-arguments"><code class="docutils
literal notranslate"><span class="pre">DataFrame::fill_null</span></code> now
borrows its arguments</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#filescanconfig-partitioned-by-file-group-removed"><code class="docutils
literal notranslate"><span
class="pre">FileScanConfig::partitioned_by_file_group</span></code>
removed</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#user-spillfile-traits-instead-of-refcountedtempfile">User <code
class="docutils literal notranslate"><span class="pre">SpillFile</span></code>
traits instead of <code class="docutils literal notranslate"><span
class="pre">RefCountedTempFile</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#dialect-available-replaced-by-dialect-available"><code class="docutils
literal notranslate"><span class="pre">Dialect::AVAILABLE</span></code>
replaced by <code class="docutils literal notranslate"><span
class="pre">Dialect::available()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#spill-record-batch-by-size-removed"><code class="docutils literal
notranslate"><span class="pre">spill_record_batch_by_size</span></code>
removed</a></li>
diff --git a/searchindex.js b/searchindex.js
index b488863517..87c49e93b7 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles":{"!=":[[74,"op-neq"]],"!~":[[74,"op-re-not-match"]],"!~*":[[74,"op-re-not-match-i"]],"!~~":[[74,"id19"]],"!~~*":[[74,"id20"]],"#":[[74,"op-bit-xor"]],"%":[[74,"op-modulo"]],"&":[[74,"op-bit-and"]],"(relation,
name) tuples in logical fields and logical columns are
unique":[[15,"relation-name-tuples-in-logical-fields-and-logical-columns-are-unique"]],"*":[[74,"op-multiply"]],"+":[[74,"op-plus"]],"-":[[74,"op-minus"]],"/":[[74,"op-divide"]],"1.
Array Literal Con [...]
\ No newline at end of file
+Search.setIndex({"alltitles":{"!=":[[74,"op-neq"]],"!~":[[74,"op-re-not-match"]],"!~*":[[74,"op-re-not-match-i"]],"!~~":[[74,"id19"]],"!~~*":[[74,"id20"]],"#":[[74,"op-bit-xor"]],"%":[[74,"op-modulo"]],"&":[[74,"op-bit-and"]],"(relation,
name) tuples in logical fields and logical columns are
unique":[[15,"relation-name-tuples-in-logical-fields-and-logical-columns-are-unique"]],"*":[[74,"op-multiply"]],"+":[[74,"op-plus"]],"-":[[74,"op-minus"]],"/":[[74,"op-divide"]],"1.
Array Literal Con [...]
\ No newline at end of file
diff --git a/user-guide/configs.html b/user-guide/configs.html
index 81d15c3a0c..661dcca620 100644
--- a/user-guide/configs.html
+++ b/user-guide/configs.html
@@ -867,7 +867,7 @@ example, to configure <code class="docutils literal
notranslate"><span class="pr
</tr>
<tr
class="row-odd"><td><p>datafusion.optimizer.preserve_file_partitions</p></td>
<td><p>0</p></td>
-<td><p>Minimum number of distinct partition values required to group files by
their Hive partition column values (enabling Hash partitioning declaration).
How the option is used: - preserve_file_partitions=0: Disable it. -
preserve_file_partitions=1: Always enable it. - preserve_file_partitions=N,
actual file partitions=M: Only enable when M >= N. This threshold preserves
I/O parallelism when file partitioning is below it. Note: This may reduce
parallelism, rooting from the I/O level, [...]
+<td><p>Minimum number of distinct partition values required to group files by
their Hive partition column values (enabling output partitioning declaration).
How the option is used: - preserve_file_partitions=0: Disable it. -
preserve_file_partitions=1: Always enable it. - preserve_file_partitions=N,
actual file partitions=M: Only enable when M >= N. This threshold preserves
I/O parallelism when file partitioning is below it. Note: This may reduce
parallelism, rooting from the I/O leve [...]
</tr>
<tr class="row-even"><td><p>datafusion.optimizer.repartition_windows</p></td>
<td><p>true</p></td>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
