Inf1mum opened a new pull request, #219: URL: https://github.com/apache/datafusion-site/pull/219
Add a self-contained blog post about the "TopK per partition" optimization for window functions (`PartitionedTopKExec` / `datafusion.optimizer.enable_window_topn`), as requested in apache/datafusion#27233: https://github.com/apache/datafusion/issues/27233 ## About the post The post expands the draft section written for the DataFusion 55 release post (the draft in the issue, by Andrew Lamb, who also wrote the request) into a full informative post following the "low-key DataFusion pitch" pattern of the [Dynamic Filters](https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/) and [Sort Pushdown](https://datafusion.apache.org/blog/2026/07/20/sort-pushdown/) posts. Structure: - The "top N per group" use case (including the h2o/db-benchmark query-8 shape) - Background: the default plan sorts the entire input, O(N log N) time and O(N) memory - How to use it in DataFusion 55: `SET datafusion.optimizer.enable_window_topn = true`, before/after `EXPLAIN` output, which queries the `WindowTopN` rule applies to, and how to turn it off - How it works: per-partition bounded heaps, shared `RowConverter`/`MemoryReservation`, ROW_NUMBER vs RANK retention semantics, emission order - Benchmarks with the verified numbers from the introducing PR (#21479), including the honest high-cardinality regression cliff and why the feature is behind a flag - Status and roadmap: `ROW_NUMBER` + `RANK` in 55, `DENSE_RANK` on main (#23869), the open enable-by-default EPIC (#24608 / #27197) — stated accurately as of today - Remaining work, acknowledgements, and references ## Coordination / credits Per the issue's note that blog posts help raise author visibility: the post credits SubhamSinghal (feature implementation), 2010YOUY01 and kosiew (reviews), jayzhan211 (memory optimizations), saadtajwar, stuhood, viirya, and zhuqi-lucas (further contributions), and Andrew Lamb (draft section this post grew out of), with links to their GitHub profiles. ## Facts and validation - All technical claims, the benchmark table, and version status are taken from the issue, the linked draft, the 55.0.0 source (`datafusion/physical-optimizer/src/window_topn.rs`, `datafusion/physical-plan/src/sorts/partitioned_topk.rs`, `config.rs`), the `window_topn.slt` sqllogictest EXPLAIN output, and the linked PRs — no invented numbers. The "enable by default" work is described as in-flight (the PR is still open at the time of writing). - Front matter and structure mirror recent posts in this repo (e.g. `2026-07-20-sort-pushdown.md`, `2026-10-24-point-queries.md`). - Local validation: the site's Docker-based build was unavailable in this environment, so the post was validated with a standalone Pelican build (processed 1 article, HTML renders title, tables, and code blocks correctly), and the `typos` check passes. The front matter is identical in shape to sibling posts, and CI's pelican build will be the authoritative check. ## Review Review would be most valuable from the feature authors, in particular SubhamSinghal (https://github.com/SubhamSinghal), jayzhan211 (https://github.com/jayzhan211), and alamb (https://github.com/alamb), to check that the technical description and benchmark framing match their intent. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
