alamb commented on code in PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#discussion_r3857607896
##########
content/blog/2026-08-25-datafusion-55.0.0.md:
##########
@@ -0,0 +1,718 @@
+---
+layout: post
+title: Apache DataFusion 55.0.0 Released
+date: 2026-08-25
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+We are proud to announce the release of [DataFusion 55.0.0]. This post
+highlights some of the many improvements since [DataFusion 54.0.0], such as
+significant performance increases, range partitioning, `MERGE INTO` support,
and
+runtime row-group pruning. The complete list of changes is available in the
+[changelog]. This release represents roughly 10 weeks of development and 877
+commits. Thanks to the [175 contributors] (a new record!) for making it
+possible.
+
+[DataFusion 55.0.0]: https://crates.io/crates/datafusion/55.0.0
+[DataFusion 54.0.0]:
https://datafusion.apache.org/blog/2026/06/12/datafusion-54.0.0/
+[changelog]:
https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md
+[175 contributors]:
https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md#credits
+
+<img
+src="/blog/images/datafusion-55.0.0/commits_contributors.svg"
+width="100%"
+class="img-fluid"
+alt="Bar charts showing total commits, commits per day, and unique
contributors for DataFusion releases 53.0.0, 54.0.0, and 55.0.0."
+/>
+
+**Figure 1**: Development activity over the last three DataFusion releases:
+total commits, commits per day, and unique contributors, computed from each
+release's [changelog] and release dates.
+
+## Performance Improvements 🚀
+
+In this release, we focused our optimizations on making DataFusion faster
across
+the board rather than further optimizing our already great ClickBench numbers
+(DataFusion is already the fastest in some cases — see the
+[appendix]), as ClickBench represents only a tiny fraction of what our actual
users
+do (e.g. its files have no page index and contain only integer and string
columns).
+
+Here is a representative sample of the performance improvements in this
release;
+see the [full list in the appendix][perf appendix].
+
+| Improvement | Representative Result
| Area |
+|-----------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+| Runtime row-group pruning for TopK | [4.2x faster
`topk_tpch`
Q8](https://github.com/apache/datafusion/pull/22450#issuecomment-4720594338)
| Sort / TopK |
+| Faster `IN` list evaluation | [up to 14.7x faster
for small primitive lists](https://github.com/apache/datafusion/pull/23014),
[9.7x faster for `UInt8`](https://github.com/apache/datafusion/pull/23011)
| Expressions |
+| Prune unread Parquet leaves for nested columns | [Reduces reads from
1.35 TB to 30.9 GB in a production Comet
query](https://github.com/apache/datafusion/pull/24090)
| Scan / IO
|
+| Fewer object store requests for CSV | [70% faster TPC-H CSV
with simulated
latency](https://github.com/apache/datafusion/pull/22962#issuecomment-4721729807)
| Scan / IO |
+| Faster `SortPreservingMerge` tie-breaker | [8% faster `sort_tpch`
Q6](https://github.com/apache/datafusion/pull/23107#issuecomment-4776877963)
| Sorting |
+| Native `GROUP BY` on `FixedSizeBinary` (e.g. UUIDs) | [~5% faster grouping
200M
UUIDs](https://github.com/apache/datafusion/pull/23646#pullrequestreview-4900163566),
[much less
memory](https://github.com/apache/datafusion/pull/23646#issuecomment-4996436765)
| Aggregation |
+
+
+### Sort Pushdown + TopK Pruning
+
+The multi-release [Sort Pushdown effort] continues to optimize `ORDER BY` and
+`ORDER BY ... LIMIT` (TopK) queries. In DataFusion 55, as a dynamic filter
+threshold tightens, the Parquet reader re-evaluates the threshold against the
+remaining row groups and drops those that can no longer contribute ([#22450]),
+and compound `ORDER BY` queries are now supported. Together these reduce the
+total `topk_tpch` suite runtime by ~43%; see our
+[Optimizing for Almost Sorted Data] blog post for more details. Thanks to
+[@zhuqi-lucas] for driving this work, with reviews from [@adriangb].
+
+[Sort Pushdown effort]: https://github.com/apache/datafusion/issues/23036
+[Optimizing for Almost Sorted Data]:
https://datafusion.apache.org/blog/2026/07/20/sort-pushdown/
+
+### Aggregation Improvements
+
+**Complete Multi-Column `GROUP BY` Type Coverage**:
+DataFusion's column-wise `GROUP BY` storage (`GroupValuesColumn`) has
+type-specific fast paths, but previously any unsupported column type forced
+the entire grouping onto a slower row-encoded fallback. For example, this
+query to deduplicate a table of UUIDs used to hit the slow path:
+
+```sql
+SELECT count(*) FROM (SELECT uuid, id FROM 'uuids.parquet' GROUP BY uuid, id);
+```
+
+DataFusion 55 completes the type coverage ([#22715]), so the query above now
runs about 5%
+faster on 200M UUIDs, and uses much less memory (see [#23645]). Thanks to
+[@zhuqi-lucas], [@tohuya6], and [@maxburke] for this work.
+
+### Faster Functions
+
+DataFusion ships hundreds of built-in functions, so speeding them up improves
performance
+for many workloads. This release optimizes dozens of functions — up to 24x
faster
+for [`find_in_set`][find_in_set] and 100x for
[`approx_distinct`][approx_distinct]
+with low-cardinality inputs and many groups ([#22768]). It also includes
+dictionary-encoding preservation for many string functions ([#23743],
+[#23930], [#24100]) and new `IN` list specializations, such as bitmap filters
for small integer types ([#19241]). See the
+[full list in the appendix][perf appendix].
+Thanks to the many contributors who drove this work, especially
+[@andygrove], [@geoffreyclaude], [@neilconway], [@lyne7-sc], [@theirix], and
+[@haohuaijin].
+
+[perf appendix]: #appendix-full-list-of-performance-improvements
+
+### Planner Improvements
+
+**Unified Distribution and Sorting Enforcement**:
+The [`EnforceDistribution`][EnforceDistribution] and
+[`EnforceSorting`][EnforceSorting] physical optimizer passes are
+now merged into a single [`EnsureRequirements`][EnsureRequirements] pass with
idempotent sort
+pushdown ([#21976]), fixing longstanding ordering issues between the two passes
+and enabling the sort pushdown work described above.
+Thanks to [@zhuqi-lucas] for this work, with reviews from [@2010YOUY01] and
+[@alamb].
+
+[EnforceDistribution]:
https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_distribution/struct.EnforceDistribution.html
+[EnforceSorting]:
https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_sorting/struct.EnforceSorting.html
+[EnsureRequirements]:
https://docs.rs/datafusion/55.0.0/datafusion/physical_optimizer/ensure_requirements/struct.EnsureRequirements.html
+
+**Smarter Join Planning**:
+DataFusion 55 now converts inner joins to more efficient semi joins when
equivalent ([#22652]),
+eliminates `LEFT`/`RIGHT` joins with redundant sides ([#23566]), handles
+intermediate projections in outer join elimination ([#22534]), and reorders
+predicates in conjunctions using a cost heuristic ([#22343]).
+Thanks to [@neilconway] and [@simonvandel] for driving this work.
+
+**Better Scalar UDF Metadata APIs**:
+Scalar UDFs can now declare that they are *strict* (they return `NULL` when any
+input is `NULL`) ([#23148]), letting the optimizer eliminate outer joins for
+queries that filter on a function result, and *strictly order preserving*
+(sorted input yields identically sorted output) ([#23807]), letting the
+optimizer eliminate redundant sorts on expressions such as custom casts.
+Thanks to [@lyne7-sc] and [@rluvaton] for this work, with reviews from
+[@alamb], [@kosiew], and [@getChan].
+
+**Faster Optimizer**:
+The optimizer continues to get faster, with improvements such as selective
+subquery traversal and in-place rewrites ([#22298]), collapsing chained
+projections ([#22389]), avoiding re-inlining expensive common subexpressions
+([#23459]), and a faster `PushDownFilter` rule that modifies plans in place
+rather than copying them ([#20002], [#21668]).
+Thanks to [@adriangb], [@Dandandan], [@fordN], and [@joroKr21] for this work.
+
+### Scan Improvements
+
+**Pruning Unread Parquet Leaves for Nested Columns**:
+
+Systems that embed DataFusion — such as [DataFusion Comet], [delta-rs], and
+Iceberg integrations — often hand DataFusion a table schema that includes only
the nested
+subfields the query needs. For example, given a file whose `events` column
+physically holds four subfields, a table might declare only two of them:
+
+```sql
+-- events column is ARRAY<STRUCT<id BIGINT, name VARCHAR, payload VARCHAR,
trace VARCHAR>>
+-- Table definition only refers to the first two subfields, id and name
+CREATE EXTERNAL TABLE events (
+ events ARRAY<STRUCT<id BIGINT, name VARCHAR>>
+)
+STORED AS PARQUET LOCATION 'events.parquet';
+```
+
+DataFusion correctly reconciles these schemas, but prior to DataFusion 55, all
+four leaves were read from the file and decoded, including the large `payload`
+and `trace` subfields, which were then thrown away. The Comet project reported
+a production query where this extra decoding caused 1.35 TB of reads, whereas
+plain Spark read only 30.9 GB for the same pruned schema. DataFusion 55 closes
+that gap by not reading the undeclared `payload` and `trace` leaves from the
+file at all ([#24090]). Thanks to [@mbutrovich] for this work, with reviews
from
+[@adriangb].
+
+[DataFusion Comet]: https://datafusion.apache.org/comet/
+[delta-rs]: https://github.com/delta-io/delta-rs
+
+**Other Scan Improvements**:
+DataFusion 55 also skips loading the page index (and an expensive
+[`ParquetMetaData`](https://docs.rs/parquet/latest/parquet/file/metadata/struct.ParquetMetaData.html)
clone) when a file has no page index ([#24150]), supports
+file-level Parquet row selections ([#22940]), and lowers the default
+`repartition_file_min_size` from 10 MiB to 1 MiB for better parallelism on
+small files ([#22439]).
+Thanks to [@alamb], [@haohuaijin], and [@adriangb].
+
+## Stability Improvements 🛡️
+
+The community also improved DataFusion's handling of larger-than-memory
+aggregate workloads (e.g. [#23657], [#23965], [#24061]), building on a
+refactoring of the aggregation path into dedicated streams (epic [#22710]).
+Sorts under memory pressure are more resilient: when a spill
+merge cannot reserve enough memory, DataFusion now re-spills the largest stream
+in smaller batches rather than failing ([#22945]), and caps the merge fan-in to
+bound memory use ([#23066]). Thanks to [@2010YOUY01], [@EmilyMatt],
+[@yinli-systems], [@Rachelint], and [@pepijnve] (who fixed a subtle lost-wakeup
+bug in the spill pool, [#23522]) for this work.
+
+## New Features ✨
+
+### `file_row_index()` and `input_file_name()`
+
+DataFusion 55 adds [`file_row_index`][file_row_index] ([#22604]) and
[`input_file_name`][input_file_name] ([#22978]) functions
+to expose Parquet virtual columns:
+
+```sql
+> select *, input_file_name(), file_row_index() from '/tmp/foo.parquet';
++---------+-------------------+------------------+
+| column1 | input_file_name() | file_row_index() |
++---------+-------------------+------------------+
+| 100 | tmp/foo.parquet | 0 |
+| 200 | tmp/foo.parquet | 1 |
++---------+-------------------+------------------+
+```
+
+Such functions are useful for change data capture, debugging, and
+Spark-compatible workloads. Thanks to [@mbutrovich] and [@AdamGS] for this work
+(reviving earlier work from [@jkylling]), with reviews from [@adriangb],
+[@comphead], and [@niebayes].
+
+### Range Partitioning
+
+DataFusion 55 adds native *range partitioning* support, which maps rows to
partitions by key ranges (rather than hash
+values). Query inputs are often range partitioned in real-world scenarios,
such as time-series data
+written as one file per day or hour. DataFusion uses range partitioning
information to
+avoid expensive repartitioning operations and push more specific dynamic
filters
+to scans.
+
+Data that is range partitioned declares an ordering and a list of split
+points. Partition `i` holds the keys that fall between split point `i-1` and
+split point `i`:
+
+```text
+ordering = [date ASC NULLS LAST]
+split_points = [(2022-01-01), (2023-01-01)]
+
+partition 0: date < 2022-01-01
+partition 1: 2022-01-01 <= date < 2023-01-01
+partition 2: date >= 2023-01-01
+```
+
+For more details, please see the documentation for
+[`Partitioning::Range`][Partitioning::Range], the planning epic ([#22395]), and
+the design discussion ([#21992]). Thanks to [@gene-bordegaray], [@saadtajwar],
[@peterxcli], [@stuhood],
+[@gmhelmold], [@mattp5657], [@mithuncy], [@JSOD11], [@EdsonPetry],
+[@Rich-T-kid], and [@blinding-pixels] for driving this substantial community
+effort.
+
+### `MERGE INTO` Planner Support
+
+`MERGE INTO` (SQL:2003) is a widely used DML statement for upsert and
+conditional update workloads, and a key building block for table formats such
+as Apache Iceberg and Delta Lake. DataFusion 55 adds the logical plan types
+([#20763]) along with SQL planner and physical planner support, and a new
+[`TableProvider::merge_into`][TableProvider::merge_into] hook ([#22988]) so
table implementations can
+execute merge operations:
+
+```sql
+MERGE INTO target t
+USING source s
+ON t.id = s.id
+WHEN MATCHED AND s.deleted THEN DELETE
+WHEN MATCHED THEN UPDATE SET name = s.name
+WHEN NOT MATCHED THEN INSERT (id, name) VALUES (s.id, s.name);
+```
+
+Built-in table providers do not yet implement the hook, but custom
+[`TableProvider`][TableProvider] implementations (such as lakehouse table
formats) can now plug
+in their own merge execution.
+Thanks to [@wirybeaver] for implementing this feature, with reviews from
+[@alamb] and [@kosiew].
+
+
+### Pluggable Spill Backends
+
+DataFusion spills to disk when a query exceeds its memory budget, but the spill
+infrastructure was previously hardwired to OS-level temporary files. DataFusion
+55 introduces a pluggable [`SpillFile`][SpillFile] trait and
+[`TempFileFactory`][TempFileFactory] ([#21882],
+[#22230]) so hosts can route spill data through their own storage layers — for
+example, extensions like ParadeDB can now integrate spilling into the
+Postgres buffer pool. Implement [`TempFileFactory`][TempFileFactory] and
install
+it on the [`RuntimeEnv`][RuntimeEnv]:
+
+```rust
+let runtime = RuntimeEnvBuilder::new()
+ .with_disk_manager_builder(
+ // register a custom TempFileFactory
+ DiskManagerBuilder::default()
+ .with_temp_file_factory(Arc::new(MyTempFileFactory::new())),
+ )
+ .build_arc()?;
+let ctx = SessionContext::new_with_config_rt(SessionConfig::new(), runtime);
+```
+
+See the [`object_store_spill.rs`][object_store_spill.rs] example for a
complete implementation that
+spills to an [`ObjectStore`][ObjectStore] such as S3. Thanks to [@pantShrey]
for this work,
+with reviews from [@alamb].
+
+### Extensibility for Distributed Engines
+
+Several new APIs make it easier to build distributed systems such as
+[datafusion-distributed], [DataFusion Ballista], and [DataFusion Python] on
+top of DataFusion:
+
+- **Dynamic filter propagation across network boundaries**: new
+ [`ExecutionPlan::apply_expressions`][ExecutionPlan::apply_expressions] and
+
[`ExecutionPlan::dynamic_expressions_produced`][ExecutionPlan::dynamic_expressions_produced]
methods let engines discover
+ which plan nodes produce dynamic filters and re-wire them across stage
+ boundaries ([#24018], [#24068]). Thanks to [@jayshrivastava].
+- **`FFI_QueryPlanner`**: foreign libraries can now provide a custom query
+ planner over the FFI boundary — for example, connecting a distributed
+ planner to a [`SessionContext`][SessionContext] in Python ([#24028]). Thanks
to [@timsaucer].
+- **Self-serializing execution plans**: built-in
[`ExecutionPlan`][ExecutionPlan]s were ported
+ to per-type [`try_to_proto`][try_to_proto] /
[`try_from_proto`][try_from_proto] hooks ([#23494]), putting built-in and
+ third-party plans on the same code path. Thanks to [@adriangb].
+- **Window accumulator state access**:
[`BoundedWindowAggExec`][BoundedWindowAggExec] can now expose
+ finalized accumulator state to an observer callback, enabling incremental /
+ prefix-scan use cases ([#24035]). Thanks to [@avantgardnerio], with reviews
Review Comment:
Added in 024399b
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
