gene-bordegaray commented on code in PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#discussion_r3797587969
##########
content/blog/2026-08-16-datafusion-55.0.0.md:
##########
@@ -0,0 +1,517 @@
+---
+layout: post
+title: Apache DataFusion 55.0.0 Released
+date: 2026-08-16
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+We are proud to announce the release of [DataFusion 55.0.0]. This post
highlights
+some of the major improvements since [DataFusion 54.0.0]. Notable additions
+include range partitioning, `MERGE INTO` support, per-partition TopK for window
+functions, and runtime row-group pruning for TopK queries, alongside
significant
+aggregation, function, and planning performance improvements. The complete list
+of changes is available in the [changelog]. This release represents roughly 9
+weeks of development and 877 commits. Thanks to the [175 contributors]
+(a new record!) for making it possible.
+
+[DataFusion 55.0.0]: https://crates.io/crates/datafusion/55.0.0
+[DataFusion 54.0.0]:
https://datafusion.apache.org/blog/2026/06/12/datafusion-54.0.0/
+[changelog]:
https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md
+[175 contributors]:
https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md#credits
+
+## Performance Improvements 🚀
+
+<!-- TODO: generate updated performance chart for 55.0.0 and place it in
+ content/images/datafusion-55.0.0/performance_over_time_clickbench.png -->
+<img
+src="/blog/images/datafusion-55.0.0/performance_over_time_clickbench.png"
+width="100%"
+class="img-fluid"
+alt="Performance over time"
+/>
+
+**Figure 1**: Average and median normalized execution times for DataFusion
55.0.0 on ClickBench queries, compared to previous releases.
+Query times are normalized using the ClickBench definition. See the
+[DataFusion Benchmarking
Page](https://alamb.github.io/datafusion-benchmarking/)
+for more details.
+
+We continue to make significant performance improvements in DataFusion, as
+explained below. This release skips more work at runtime using statistics and
+dynamic filters, and makes window functions, aggregation, and many built-in
+functions faster.
+
+### Sort Pushdown: Runtime Row-Group Pruning for TopK Queries
+
+The multi-release [Sort Pushdown effort] makes `ORDER BY` and
+`ORDER BY ... LIMIT` (TopK) queries on Parquet skip work end-to-end: skip the
+sort, skip row groups via min/max statistics, and skip rows via dynamic
filters.
+DataFusion 55 lands the next phase: as a TopK query runs and its dynamic filter
+threshold tightens, the Parquet reader now re-evaluates the threshold against
+the remaining row groups at every row-group boundary and drops those that can
+no longer contribute — zero IO and zero decode for the skipped row groups
+([#22450]). This release also adds multi-column lexicographic statistics
+reordering ([#23888]), so compound `ORDER BY` queries benefit too. In the
+`topk_tpch` benchmark suite, 5 of 11 queries got 3-4x faster with no
+regressions, reducing total suite runtime by 44%.
+Thanks to [@zhuqi-lucas] for driving this work, with reviews from [@adriangb].
+
+[Sort Pushdown effort]: https://github.com/apache/datafusion/issues/23036
+
+### Per-Partition TopK for Window Functions
+
+A common analytics pattern selects the top N rows per group using a window
+function:
+
+```sql
+SELECT * FROM (
+ SELECT ROW_NUMBER() OVER (PARTITION BY category ORDER BY revenue DESC) AS
rn, *
+ FROM sales
+) WHERE rn <= 5;
+```
+
+DataFusion previously sorted the *entire* input to evaluate the window
+function, even though only a handful of rows per partition survive the filter.
+DataFusion 55 recognizes this pattern and uses a new `PartitionedTopKExec`
+operator that keeps only the top N rows per partition, dramatically reducing
+sorting and memory for high-cardinality inputs. The optimization applies to
+`ROW_NUMBER` and `RANK`, resolving a feature request first filed in 2023
+([#6899]).
+Thanks to [@SubhamSinghal] for implementing this feature, with reviews from
+[@2010YOUY01] and [@kosiew]. Related PRs: [#21479], [#22885], [#23096]
+
+### Aggregation Improvements
+
+**Complete Multi-Column `GROUP BY` Type Coverage**:
+DataFusion's column-wise `GROUP BY` storage (`GroupValuesColumn`) has
+type-specific fast paths, but previously any unsupported column type dragged
+the entire grouping onto a slower row-encoded fallback. DataFusion 55 completes
+the type coverage ([#22715]): new specializations were added for `Decimal256`,
+`Float16`, and `Interval` ([#23849], [#23785], [#23786]), and a new generic
+`Rows`-backed `GroupColumn` keeps mixed schemas on the column-wise path
+([#23523]).
+Thanks to [@zhuqi-lucas] and [@tohuya6] for this work.
+
+**Split Aggregation Streams**:
+The hash aggregation state machine previously handled partial aggregation,
+final aggregation, and streaming cases in one shared implementation. DataFusion
+55 splits these semantically distinct paths into dedicated streams ([#22729]),
+making the code easier to optimize and extend (part of epic [#22710]).
+Thanks to [@2010YOUY01] for this work, with reviews from [@Rachelint] and
+[@alamb].
+
+### Faster Functions
+
+DataFusion ships hundreds of built-in functions, so speeding them up pays off
+across many workloads. This release optimizes many, including [find_in_set]
+(up to 24x faster), [trunc] (10x), [replace] (2x), [regexp_instr] (40%),
+[regexp_match], [round], [date_trunc], [date_part], [get_field], [upper], and
+[string_trim], plus dictionary-encoding preservation for many string functions
+([#23743], [#23930], [#24100]) and a 100x improvement to [approx_distinct] for
+low-cardinality inputs with many groups ([#22768]). The `approx_distinct`
+aggregate also gained support for many more types, including `Decimal`,
+`Interval`, `Duration`, `Struct`, `Map`, and `Union`, thanks to [@mkleen].
+Thanks to the many contributors who drove this work, especially [@andygrove],
+[@neilconway], [@lyne7-sc], [@theirix], and [@haohuaijin].
+
+**Faster `IN` List Evaluation**:
+`IN` list membership checks can run millions of times during a scan, especially
+with dynamic filter pushdown. DataFusion 55 adds exact lookup strategies
+selected by type and list size, including bitmap filters for small integer
+types and branchless filters for small primitive lists ([#19241]).
+Thanks to [@geoffreyclaude] for driving this work, with contributions from
+[@alamb]. Related PRs: [#23012], [#23014], [#23299]
+
+### Planner Improvements
+
+**Unified Distribution and Sorting Enforcement**:
+The `EnforceDistribution` and `EnforceSorting` physical optimizer passes are
+now merged into a single `EnsureRequirements` pass with idempotent sort
+pushdown ([#21976]), fixing longstanding ordering issues between the two passes
+and enabling the sort pushdown work described above.
+Thanks to [@zhuqi-lucas] for this work, with reviews from [@2010YOUY01] and
+[@alamb].
+
+**Smarter Join Planning**:
+DataFusion 55 converts inner joins to semi joins when equivalent ([#22652]),
+eliminates `LEFT`/`RIGHT` joins with redundant sides ([#23566]), handles
+intermediate projections in outer join elimination ([#22534]), and reorders
+predicates in conjunctions using a cost heuristic ([#22343]).
+Thanks to [@neilconway] and [@simonvandel] for this work.
+
+**Scalar UDF Strictness Metadata**:
+Scalar UDFs can now declare that they are *strict* (they return `NULL` when any
+input is `NULL`) ([#23148]). The optimizer uses this metadata to prove that
+filters reject `NULL`s, unlocking outer join elimination for queries that
+filter on the result of a function call.
+Thanks to [@lyne7-sc] for implementing this feature, with reviews from [@alamb]
+and [@kosiew].
+
+**Faster Optimizer Passes**:
+The logical optimizer now skips subquery traversal when there are no subqueries
+and rewrites plans in place ([#22298]), collapses chained projections in a
+single pass ([#22389]), and avoids re-inlining expensive common subexpressions
+during projection pushdown ([#23459]).
+Thanks to [@adriangb], [@Dandandan], and [@fordN] for this work.
+
+### Scan Improvements
+
+**Pruning Unread Parquet Leaves for Nested Columns**:
+When a table declares a nested column narrower than the Parquet file's physical
+type, DataFusion previously read every leaf of the column and dropped the extra
+subfields in memory. DataFusion 55 derives the projection mask through casts,
+so only the leaves that are actually accessed are read — one production query
+reported by the [DataFusion Comet] project went from reading 1.35 TB to reading
+only the required data ([#24090]).
+Thanks to [@mbutrovich] for this work, with reviews from [@adriangb].
+
+[DataFusion Comet]: https://datafusion.apache.org/comet/
+
+**Other Scan Improvements**:
+DataFusion 55 also skips loading the page index (and an expensive
+`ParquetMetaData` clone) when a file has no page index ([#24150]), supports
+file-level Parquet row selections ([#22940]), and lowers the default
+`repartition_file_min_size` from 10 MiB to 1 MiB for better parallelism on
+small files ([#22439]).
+Thanks to [@alamb], [@haohuaijin], and [@adriangb].
+
+## New Features ✨
+
+### Range Partitioning
+
+DataFusion previously supported only hash and round-robin repartitioning.
+DataFusion 55 adds *range partitioning*, where rows are distributed to
+partitions based on ordered split points ([#22395], design discussion
+[#21992]). Range partitioning preserves ordering across partitions, which is a
+natural fit for pre-sorted data and for distributed engines that shuffle by
+range.
Review Comment:
maybe a quick example or link to documented example to show how split points
are used 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
