gene-bordegaray commented on code in PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#discussion_r3861835162
##########
content/blog/2026-08-25-datafusion-55.0.0.md:
##########
@@ -0,0 +1,714 @@
+---
+layout: post
+title: Apache DataFusion 55.0.0 Released
+date: 2026-08-25
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+We are proud to announce the release of [DataFusion 55.0.0]. This post
+highlights some of the many improvements since [DataFusion 54.0.0], such as
+significant performance increases, range partitioning, `MERGE INTO` support,
and
+runtime row-group pruning. The complete list of changes is available in the
+[changelog]. This release represents roughly 10 weeks of development and 877
+commits. Thanks to the [175 contributors] (a new record!) for making it
+possible.
+
+[DataFusion 55.0.0]: https://crates.io/crates/datafusion/55.0.0
+[DataFusion 54.0.0]:
https://datafusion.apache.org/blog/2026/06/12/datafusion-54.0.0/
+[changelog]:
https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md
+[175 contributors]:
https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md#credits
+
+<img
+src="/blog/images/datafusion-55.0.0/commits_contributors.svg"
+width="100%"
+class="img-fluid"
+alt="Bar charts showing total commits, commits per day, and unique
contributors for DataFusion releases 53.0.0, 54.0.0, and 55.0.0."
+/>
+
+**Figure 1**: Development activity over the last three DataFusion releases:
+total commits, commits per day, and unique contributors, computed from each
+release's [changelog] and release dates.
+
+## Performance Improvements 🚀
+
+In this release, we focused our optimizations on making DataFusion faster
across
+the board rather than further optimizing our already great ClickBench numbers
+(DataFusion is already the fastest in some cases — see the
+[appendix]), as ClickBench represents only a tiny fraction of what our actual
users
+do (e.g. its files have no page index and contain only integer and string
columns).
+
+Here is a representative sample of the performance improvements in this
release;
+see the [full list in the appendix][perf appendix].
+
+| Improvement | Representative Result
| Area |
+|-----------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+| Runtime row-group pruning for TopK | [4.2x faster
`topk_tpch`
Q8](https://github.com/apache/datafusion/pull/22450#issuecomment-4720594338)
| Sort / TopK |
+| Faster `IN` list evaluation | [up to 14.7x faster
for small primitive lists](https://github.com/apache/datafusion/pull/23014),
[9.7x faster for `UInt8`](https://github.com/apache/datafusion/pull/23011)
| Expressions |
+| Prune unread Parquet leaves for nested columns | [Reduces reads from
1.35 TB to 30.9 GB in a production Comet
query](https://github.com/apache/datafusion/pull/24090)
| Scan / IO
|
+| Fewer object store requests for CSV | [70% faster TPC-H CSV
with simulated
latency](https://github.com/apache/datafusion/pull/22962#issuecomment-4721729807)
| Scan / IO |
+| Faster `SortPreservingMerge` tie-breaker | [8% faster `sort_tpch`
Q6](https://github.com/apache/datafusion/pull/23107#issuecomment-4776877963)
| Sorting |
+| Native `GROUP BY` on `FixedSizeBinary` (e.g. UUIDs) | [~5% faster grouping
200M
UUIDs](https://github.com/apache/datafusion/pull/23646#pullrequestreview-4900163566),
[much less
memory](https://github.com/apache/datafusion/pull/23646#issuecomment-4996436765)
| Aggregation |
+
+
+### Sort Pushdown + TopK Pruning
+
+The multi-release [Sort Pushdown effort] continues to optimize `ORDER BY` and
+`ORDER BY ... LIMIT` (TopK) queries. In DataFusion 55, as a dynamic filter
+threshold tightens, the Parquet reader re-evaluates the threshold against the
+remaining row groups and drops those that can no longer contribute ([#22450]),
+and compound `ORDER BY` queries are now supported. Together these reduce the
+total `topk_tpch` suite runtime by ~43%; see our
+[Optimizing for Almost Sorted Data] blog post for more details. Thanks to
+[@zhuqi-lucas] for driving this work, with reviews from [@adriangb].
+
+[Sort Pushdown effort]: https://github.com/apache/datafusion/issues/23036
+[Optimizing for Almost Sorted Data]:
https://datafusion.apache.org/blog/2026/07/20/sort-pushdown/
+
+### Aggregation Improvements
+
+**Complete Multi-Column `GROUP BY` Type Coverage**:
+DataFusion's column-wise `GROUP BY` storage (`GroupValuesColumn`) has
+type-specific fast paths, but previously any unsupported column type forced
+the entire grouping onto a slower row-encoded fallback. For example, this
+query to deduplicate a table of UUIDs used to hit the slow path:
+
+```sql
+SELECT count(*) FROM (SELECT uuid, id FROM 'uuids.parquet' GROUP BY uuid, id);
+```
+
+DataFusion 55 completes the type coverage ([#22715]), so the query above now
runs about 5%
+faster on 200M UUIDs, and uses much less memory (see [#23645]). Thanks to
+[@zhuqi-lucas], [@tohuya6], and [@maxburke] for this work.
+
+### Faster Functions
+
+DataFusion ships hundreds of built-in functions, so speeding them up improves
performance
+for many workloads. This release optimizes dozens of functions — up to 24x
faster
+for [`find_in_set`][find_in_set] and 100x for
[`approx_distinct`][approx_distinct]
+with low-cardinality inputs and many groups ([#22768]). It also includes
+dictionary-encoding preservation for many string functions ([#23743],
+[#23930], [#24100]) and new `IN` list specializations, such as bitmap filters
for small integer types ([#19241]). See the
+[full list in the appendix][perf appendix].
+Thanks to the many contributors who drove this work, especially
+[@andygrove], [@geoffreyclaude], [@neilconway], [@lyne7-sc], [@theirix], and
+[@haohuaijin].
+
+[perf appendix]: #appendix-full-list-of-performance-improvements
+
+### Planner Improvements
+
+**Unified Distribution and Sorting Enforcement**:
+The [`EnforceDistribution`][EnforceDistribution] and
+[`EnforceSorting`][EnforceSorting] physical optimizer passes are
+now merged into a single [`EnsureRequirements`][EnsureRequirements] pass with
idempotent sort
+pushdown ([#21976]), fixing longstanding ordering issues between the two passes
+and enabling the sort pushdown work described above.
+Thanks to [@zhuqi-lucas] for this work, with reviews from [@2010YOUY01] and
+[@alamb].
+
+[EnforceDistribution]:
https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_distribution/struct.EnforceDistribution.html
+[EnforceSorting]:
https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_sorting/struct.EnforceSorting.html
+[EnsureRequirements]:
https://docs.rs/datafusion/55.0.0/datafusion/physical_optimizer/ensure_requirements/struct.EnsureRequirements.html
+
+**Smarter Join Planning**:
+DataFusion 55 now converts inner joins to more efficient semi joins when
equivalent ([#22652]),
+eliminates `LEFT`/`RIGHT` joins with redundant sides ([#23566]), handles
+intermediate projections in outer join elimination ([#22534]), and reorders
+predicates in conjunctions using a cost heuristic ([#22343]).
+Thanks to [@neilconway] and [@simonvandel] for driving this work.
+
+**Scalar UDF Strictness Metadata**:
+Scalar UDFs can now declare that they are *strict* (they return `NULL` when any
+input is `NULL`) ([#23148]). The optimizer uses this metadata to prove that
+filters reject `NULL`s, unlocking outer join elimination for queries that
+filter on the result of a function call.
+Thanks to [@lyne7-sc] for implementing this feature, with reviews from [@alamb]
+and [@kosiew].
+
+**Faster Optimizer**:
+The optimizer continues to get faster, with improvements such as selective
+subquery traversal and in-place rewrites ([#22298]), collapsing chained
+projections ([#22389]), avoiding re-inlining expensive common subexpressions
+([#23459]), and a faster `PushDownFilter` rule that modifies plans in place
+rather than copying them ([#20002], [#21668]).
+Thanks to [@adriangb], [@Dandandan], [@fordN], and [@joroKr21] for this work.
+
+### Scan Improvements
+
+**Pruning Unread Parquet Leaves for Nested Columns**:
+
+Systems that embed DataFusion — such as [DataFusion Comet], [delta-rs], and
+Iceberg integrations — often hand DataFusion a table schema that includes only
the nested
+subfields the query needs. For example, given a file whose `events` column
+physically holds four subfields, a table might declare only two of them:
+
+```sql
+-- events column is ARRAY<STRUCT<id BIGINT, name VARCHAR, payload VARCHAR,
trace VARCHAR>>
+-- Table definition only refers to the first two subfields, id and name
+CREATE EXTERNAL TABLE events (
+ events ARRAY<STRUCT<id BIGINT, name VARCHAR>>
+)
+STORED AS PARQUET LOCATION 'events.parquet';
+```
+
+DataFusion correctly reconciles these schemas, but prior to DataFusion 55, all
+four leaves were read from the file and decoded, including the large `payload`
+and `trace` subfields, which were then thrown away. The Comet project reported
+a production query where this extra decoding caused 1.35 TB of reads, whereas
+plain Spark read only 30.9 GB for the same pruned schema. DataFusion 55 closes
+that gap by not reading the undeclared `payload` and `trace` leaves from the
+file at all ([#24090]). Thanks to [@mbutrovich] for this work, with reviews
from
+[@adriangb].
+
+[DataFusion Comet]: https://datafusion.apache.org/comet/
+[delta-rs]: https://github.com/delta-io/delta-rs
+
+**Other Scan Improvements**:
+DataFusion 55 also skips loading the page index (and an expensive
+[`ParquetMetaData`](https://docs.rs/parquet/latest/parquet/file/metadata/struct.ParquetMetaData.html)
clone) when a file has no page index ([#24150]), supports
+file-level Parquet row selections ([#22940]), and lowers the default
+`repartition_file_min_size` from 10 MiB to 1 MiB for better parallelism on
+small files ([#22439]).
+Thanks to [@alamb], [@haohuaijin], and [@adriangb].
+
+## Stability Improvements 🛡️
+
+The community also improved DataFusion's handling of larger-than-memory
+aggregate workloads (e.g. [#23657], [#23965], [#24061]), building on a
+refactoring of the aggregation path into dedicated streams (epic [#22710]).
+Sorts under memory pressure are more resilient: when a spill
+merge cannot reserve enough memory, DataFusion now re-spills the largest stream
+in smaller batches rather than failing ([#22945]), and caps the merge fan-in to
+bound memory use ([#23066]). Thanks to [@2010YOUY01], [@EmilyMatt],
+[@yinli-systems], [@Rachelint], and [@pepijnve] (who fixed a subtle lost-wakeup
+bug in the spill pool, [#23522]) for this work.
+
+## New Features ✨
+
+### `file_row_index()` and `input_file_name()`
+
+DataFusion 55 adds [`file_row_index`][file_row_index] ([#22604]) and
[`input_file_name`][input_file_name] ([#22978]) functions
+to expose Parquet virtual columns:
+
+```sql
+> select *, input_file_name(), file_row_index() from '/tmp/foo.parquet';
++---------+-------------------+------------------+
+| column1 | input_file_name() | file_row_index() |
++---------+-------------------+------------------+
+| 100 | tmp/foo.parquet | 0 |
+| 200 | tmp/foo.parquet | 1 |
++---------+-------------------+------------------+
+```
+
+Such functions are useful for change data capture, debugging, and
+Spark-compatible workloads. Thanks to [@mbutrovich] and [@AdamGS] for this work
+(reviving earlier work from [@jkylling]), with reviews from [@adriangb],
+[@comphead], and [@niebayes].
+
+### Range Partitioning
+
+DataFusion 55 adds native *range partitioning* support, which maps rows to
partitions by key ranges (rather than hash
Review Comment:
ya think this is good summary, too much to fit into a non dedicated post 😆
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
