This is an automated email from the ASF dual-hosted git repository.
alamb pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-site.git
The following commit(s) were added to refs/heads/main by this push:
new 9ecf01b Add blog: Optimizing for Almost Sorted Data: Sort Pushdown in
Apache DataFusion (#186)
9ecf01b is described below
commit 9ecf01ba5ec102ea3e95ee37188bcb21eb77e605
Author: Qi Zhu <[email protected]>
AuthorDate: Tue Jul 21 01:11:36 2026 +0800
Add blog: Optimizing for Almost Sorted Data: Sort Pushdown in Apache
DataFusion (#186)
* Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip I/O
A walkthrough of the sort pushdown work landed and in flight on Apache
DataFusion. Covers:
- Why SortExec is expensive and what `Exact` / `Inexact` ordering mean
at runtime (static `fetch` vs `TopK` dynamic filter).
- Phase 1 (#19064): the `PushdownSort` rule + reverse row-group case.
- Phase 2 (#21182): statistics-based file sort that upgrades
`Unsupported` to `Exact`, eliminating the `SortExec` on
non-overlapping ASC scans. Includes the BufferExec compensation
(#21426) so the SPM above doesn't lose its implicit memory buffer.
- Reverse scans: today's row-group reverse (Inexact, #18817) and the
community decision to wait for arrow-rs page-level reverse (#9937)
before pursuing Exact reverse, after memory-profile pushback on the
original row-group-level proposal.
- Benchmarks: 2.1×-49× on the ASC-LIMIT sort_pushdown suite.
- What's next: the dynamic / TopK-driven path (#21351 merged, #21733,
#21712, #21956, #21580) including the precise RG-pruning vs
mid-stream-early-return distinction, and the EnsureRequirements
unification (#21976).
- Links into the prior dynamic filters and limit pruning posts so the
series reads as a coherent thread.
* Push draft date to 2026-05-25
* Add Phase 3 section covering #21956 runtime reorder pipeline
The post previously only mentioned #21956 in passing. The PR is
landing the full mechanism — `try_pushdown_sort` decision tree,
two flags on `ParquetSource`, three composable runtime steps
(file reorder + RG reorder + reverse), and a `sort_prefix`-
preserving short-circuit — so cover it as a dedicated phase
between Phase 2 and the existing reverse-scan section.
- TL;DR: add a Phase 3 bullet alongside Phase 1 and Phase 2.
- Phase 1: replace the in-flight `#21956` aside with a cross-link
to the new section.
- Phase 2: keep the caveat about function-wrapped sorts but note
that #21956's `Inexact` path now covers them via monotonicity
inference.
- New `## Phase 3` section with two SVG diagrams: a decision tree
for the three protocol outcomes, and a three-step pipeline for
the `Inexact` runtime. Covers the two-flag design, the nested
file/RG layers, when EXPLAIN surfaces each flag, and four
scenarios where Phase 3 does not fire (aggregations, multi-
column secondary keys, function-wrapped sorts without a
declared ordering, source declares a forward prefix of the
request).
- "What's Next": rename the old "Phase 3 — filter + sort" bullet
to "Filtered reverse TopK end-to-end" so the label doesn't
clash with the new section, and add a follow-up bullet
referencing #22198 for multi-column / function-wrapped reorder.
* Add empirical note: why we keep an out-of-tree RG-level Exact reverse
Add an ### Empirical note subsection inside "Reverse Scans for
`ORDER BY ... DESC`" that records what we found running an in-house
RG-level `Exact` reverse against upstream `Inexact` + `TopK`:
- `LIMIT N` does not propagate as a static stop signal in the
Inexact path. The dynamic filter pushdown can stats-prune
*subsequent* row groups once the threshold tightens, but inside
the row group `TopK` is currently reading the sort column has to
be fully decoded so the filter can be evaluated row by row.
`LIMIT 10` on a 1M-row row group is still ~1M sort-column
decodes regardless of N. LIMIT only saves work on non-sort
columns inside that row group and on whole subsequent row
groups the threshold prunes.
- `SortExec` stays on top of `Inexact`, so the final ordering
pass and per-row heap maintenance are both extra costs the
`Exact` path (which deletes `SortExec`) does not pay.
Then explain why we run RG-level `Exact` in production but did
not upstream it: parquet does not allow partial row-group reads,
so any RG-level `Exact` implementation peaks at one whole row
group (~128 MB) of decoded data in memory — the same constraint
that closed `#18817`. Our runtime advantage comes from skipping
heap / filter / `SortExec` overhead, not from decoding less.
Frame the page-level `Exact` reverse work in arrow-rs `#9937` as
the path that keeps the runtime win we measured while bringing
peak memory back into the streaming regime via `OffsetIndex`
page-level seek.
* Correct internal RG-Exact description and trim arrow-rs #9937 duplication
Two corrections in the empirical-note / reverse-scans section:
1. The internal RG-level `Exact` reverse path was incorrectly
described as "walks pages from the back, decodes each batch,
reverses row-wise, stops the moment K rows have been delivered."
That is actually the page-level `Exact` shape (arrow-rs #9937),
not the in-house RG-level implementation. Parquet does not allow
partial row-group reads, so the in-house path has to decode the
entire target row group, reverse the buffer in memory, take the
first K rows, and stop — same memory profile as #18817's
proposal. The runtime advantage over `Inexact` + `TopK` comes
from removing the per-row heap maintenance, dynamic-filter
evaluation, and `SortExec` final ordering pass, not from
decoding less sort-column data. Sort col decode on the target
row group is the same on both paths.
2. The arrow-rs #9937 paragraph I previously added duplicated the
technical detail already present in the long-standing
"That primitive is the page-level reverse traversal..."
paragraph. Replaced with a one-sentence bridge ("The fix that
keeps both the runtime win and a streaming memory profile is
page-level `Exact` reverse via arrow-rs #9937, described next.")
so the existing paragraph carries the explanation without
repetition.
* Mark #21956 as landed (merged via merge queue)
* Reframe blog from Phase N → capability-based sections
All three sort-pushdown PRs have now landed, so the chronological
'Phase 1/2/3' framing is less useful for readers than a capability
breakdown. Sections are now:
- The PushdownSort Rule (#19064)
- Sort Elimination via Statistics (#21182)
- Runtime Reorder for TopK Convergence (#21956)
- Reverse Scans for ORDER BY ... DESC (unchanged)
In-body Phase references replaced with PR numbers or capability names;
anchor links updated; references section restructured.
* Clarify two-layer composability: same key (min), not 'nest by
construction'
The previous wording 'nest by construction' could be read as a code-
enforced property. It's actually a logical consequence of using the
same sort key (min) at both file and row-group level: a file's
min(col) is just the minimum over its row groups' min(col) values, so
the most-promising file contains the most-promising row group. The
rewritten paragraph spells that out and ties it to why TopK's dynamic
filter tightens fast.
* Strip inline PR refs from narrative; collect all links in References
Match the dynamic-filter blog's style: narrative talks about
capabilities/mechanisms, not 'PR #21956 did X / PR #19064 introduced
Y'. The 81 inline PR/issue references in the body were dropping the
reader out of the narrative; they belong in a single Issues-and-PRs
list at the end.
Changes:
- TL;DR: drop 6 inline PR refs from the 4 capability bullets
- Body sections (PushdownSort Rule, Sort Elimination, Runtime Reorder,
Reverse Scans, Empirical Note): drop ~30 inline refs to historical
PRs; replace with capability names or 'the rule' / 'the runtime
reorder path' style descriptions
- What's Next: switch from [#NNNNN] format to named markdown links
(matching dynamic-filter's Future Work style)
- Issues for new contributors: same conversion
- References section: rewrite using full URLs (no link-ref
indirection); split into 'Landed' vs 'In flight / open' for
clarity
Net: ~90 lines removed, all PR/issue numbers now consolidated at
the bottom of the post.
* Restructure: split into merged-features / bottlenecks / roadmap
The previous draft mixed merged work, in-flight work, and runtime-cost
analysis into a single 'Reverse Scans' section and a sprawling 'What's
Next' section. Reorganize so the post answers three clear questions in
sequence:
1. What's merged today? (Sort Elimination via Statistics + benchmark,
Runtime Reorder for TopK Convergence, Reverse Scans for DESC) —
unchanged content, just kept tight.
2. Where do those merged features still leave performance on the
table? New 'Current Bottlenecks' section with three explicitly
numbered bottlenecks: SortExec stays / sort column fully decoded
inside open RG / file-granular scheduling can't close the tap
mid-file. Pulls in the runtime-cost content that used to be buried
in an 'Empirical note' subsection.
3. How does each next-step optimization remove a specific bottleneck?
New 'Roadmap' section maps page-level Exact reverse to bottlenecks
1+2, row-group-level dynamic early termination to bottleneck 3, and
shows the in-flight 17x-60x pipeline benchmark as a preview of what
stacking these mechanisms can deliver.
Smaller follow-ups (EnsureRequirements, OFFSET pushdown, multi-column
reorder) collected at the end of the roadmap section as a short
'Other follow-ups' bullet list.
* Tighten verbose paragraphs; drop unrelated limit-pruning blog reference
- 'Why not full Exact reverse' paragraph: cut reviewer attribution and
forward-pointers that were already in the bottleneck/roadmap
sections that follow.
- TL;DR: trim Runtime Reorder + Reverse Scans bullets to capability
and impact; drop implementation mechanics like 'stamps two flags'
and 'three-step pipeline'.
- 'The PushdownSort Rule' section: cut three paragraphs of
'covered in X below' forward-references that were repeating
the section TOC.
- Function-wrapped parenthetical in Sort Elimination: 4 lines to 2.
- Single-partition vs multi-partition edge case: drop the trailing
'which is why the example is drawn that way' tangent.
- 'What this change does not affect' note: trimmed redundant prose.
- Remove all references to the limit-pruning blog (intro mention,
link definition, References section bullet) — that work is about
static LIMIT as an I/O optimization, separate problem from sort
ordering.
* Fold reverse as case of Inexact runtime reorder; make two-trigger
asymmetry explicit
Per code in datasource-parquet/src/source.rs:849-870, the reversed-
satisfies branch is 'strictly more powerful than the column-in-schema
check' — it runs the request through EquivalenceProperties's full
reasoning machinery and handles function monotonicity, constants from
filters, equivalence relationships, and multi-column composite
orderings. The blog had been treating reverse as just step 3 of the
runtime pipeline, which undersold its standalone reach.
Structural changes:
- Drop the standalone 'Reverse Scans for ORDER BY DESC' H2 section;
reverse is now a case of the Inexact runtime reorder path.
- Rename Runtime Reorder section to 'Runtime Reorder for TopK and
DESC Queries'; intro now lists three classes that fall outside
Exact (unsorted, overlapping, DESC).
- 'try_pushdown_sort' subsection rewritten as 'Two independent
triggers for Inexact', describing column-in-schema vs reversed-
satisfies as separate signals with the latter being strictly more
powerful.
- 'Three runtime steps' subsection: step 3 now explicitly notes when
steps 1-2 are skipped and only the iteration reverse runs.
- New 'ORDER BY DESC in practice' subsection right after the 3-step
pipeline, explaining the RGs-descending-x-rows-ascending stream.
- Move reverse-scan.svg from the deleted Reverse Scans section into
the Roadmap > Page-level Exact reverse subsection where it
illustrates the 128 MB vs 1 MB peak comparison directly.
Accuracy fix:
- 'Multi-column reorder follow-ups' bullet was inaccurate — said the
reorder 'only fires on plain columns'. The reverse path does
handle function-wrapped and multi-column cases via
EquivalenceProperties; only the stats reorder step is restricted.
Updated wording to scope the limitation correctly.
* Drop EnsureRequirements + OFFSET pushdown — both unrelated to sort
pushdown
EnsureRequirements (#21976) is a rule-unification effort for
EnforceDistribution+EnforceSorting. Touches the same area but isn't
a sort pushdown optimization.
OFFSET pushdown (#21828) is about LIMIT/OFFSET pruning. Same kind of
tangent as the limit-pruning blog reference removed earlier — it's
LIMIT optimization, not sort pushdown.
The remaining 'Multi-column and function-wrapped reorder follow-ups'
bullet is actually directly about sort pushdown's reorder step
(#22198), so it stays. With the other two removed, 'Other follow-ups'
collapsed to a single point — promoted to its own subsection
'Extending the stats reorder step' for clarity.
Also dropped the corresponding entries from the References section.
* Roadmap RG-level early termination: split into non-overlap vs overlap
(k-way merge)
The 'switch work queue from PartitionedFile to RG descriptors' fix is
sufficient for non-overlapping ranges (post-reorder), where the first
file globally has the smallest values and subsequent files are
already stats-pruned. For overlapping ranges, the next smallest value
could sit in any of several open files — matching the non-overlap
efficiency requires explicit k-way merge across open files' next-RG
mins. The dynamic filter does this implicitly (RGs with max < threshold
are dropped), but explicit comparison closes the tap earlier when the
filter tightens slowly.
* Rewrite blog around #22450: three-layer pruning + topk_tpch headline
Restructure the post so the merged Runtime Row-Group Dynamic Pruning
(#22450) is the centerpiece, matching the community-talk narrative:
- TL;DR now leads with the three-layer pruning stack (file + RG + row)
and the topk_tpch headline (5/11 queries 3-4x, 0 regressions, -44%
total).
- Exact path keeps credit attribution explicit: EnforceSorting already
handled the simple on-disk-sorted case; Phase 2 closes the
wrong-on-disk-order gap. BufferExec story expanded with the SPM
stall diagnosis.
- Inexact path split into Tier 1 (file-level, already had early stop)
vs Tier 2 (RG-level, no early stop until #22450) so the gap that
#22450 closes is named explicitly.
- New section: #22450 mechanics - architecture-3-eras, transition()
drain/decide/drive, RowGroupPruner watch/rebuild/prune, cascading
prune walkthrough.
- New section: Three-layer pruning stack with the same DynamicFilter
driving all three layers, plus the topk_tpch benchmark table.
- Old "Current Bottlenecks" + "Roadmap" merged into "Future Directions":
A) page-level Exact reverse (arrow-rs #9937), B) page-level dynamic
prune at RG boundary (#23216).
- Updated landed-PRs list to include #20839 (push-based decoder, the
prerequisite for #22450) and #22450 itself.
Pulls in 9 PNGs from the community-call deck for the #22450-specific
diagrams (architecture, transition, pruner, cascade, three-layer stack,
topk_tpch results, page-level future).
* Update content/blog/2026-07-05-sort-pushdown.md
Co-authored-by: Andrew Lamb <[email protected]>
* Update content/blog/2026-07-05-sort-pushdown.md
Co-authored-by: Andrew Lamb <[email protected]>
* Update content/blog/2026-07-05-sort-pushdown.md
Co-authored-by: Andrew Lamb <[email protected]>
* Update content/blog/2026-07-05-sort-pushdown.md
Co-authored-by: Andrew Lamb <[email protected]>
* Update content/blog/2026-07-05-sort-pushdown.md
Co-authored-by: Andrew Lamb <[email protected]>
* Update content/blog/2026-07-05-sort-pushdown.md
Co-authored-by: Andrew Lamb <[email protected]>
* Update content/blog/2026-07-05-sort-pushdown.md
Co-authored-by: Andrew Lamb <[email protected]>
* Address alamb review round 2 (46 comments)
Tighten intro to be engine-agnostic and drop time-tied phrasing; use
plain 'sort' instead of SortExec in engine-agnostic sections while
keeping SortExec in DataFusion-internals sections. Remove several
implementation-detail paragraphs (bail-out conditions, plan-walk order,
three-eras evolution history). Rename PR-numbered sections to concept
names (Row Group Filter early stop, Runtime Row-Group Dynamic Pruning,
Example of pruning row groups using TopK dynamic predicates). Relabel
Layer 0/A/B as Layer 1/2/3 consistently. Move sort_pushdown /
topk_tpch benchmark descriptions into their section intros. Reframe
benchmark comparisons as DataFusion 54 sort optimizations disabled vs
enabled. Collapse Future Directions to a summary sentence with issue
links. Add Get Involved section and Massive sponsorship shout-out.
Net -86 lines (625 -> 539).
* Remove transition_anatomy and pruner_loop figures
Both figures had visual overflow issues that alamb flagged (Step 2/3
code text overflowing box boundaries, WATCH circle cutoff). Their
captions were also flagged as content that should live in the blog
text rather than embedded in the diagram. Since the surrounding
markdown text already covers the drain/decide/drive loop and the
watch/rebuild/prune pipeline, dropping the figures resolves both
concerns without losing content.
* Revert "Remove transition_anatomy and pruner_loop figures"
This reverts commit 9b48aa84dff0155731da614bdf1eea5904089a6f.
* Regenerate transition_anatomy and pruner_loop figures
Address alamb's figure-annotation feedback:
transition_anatomy.png:
- Widened Step 2 / Step 3 code boxes so
`into_builder().with_row_groups(...)`
and `Data -> set_active_reader` fit inside their boxes without overflow.
- Removed the bottom captions that duplicated content already in the blog
text.
pruner_loop.png:
- Widened WATCH shape so `1 atomic load` fits without cutoff.
- Removed the `Errors fall back to keep` line (too much detail per alamb).
- Same three shapes + arrows + descriptor cards; only sizing tweaked.
* Move Three-Layer Pruning section earlier
Per alamb's suggestion in #3544371141 — moved the multi-layer pruning
strategy section from after Runtime Row-Group Dynamic Pruning to right
after How DataFusion Tracks Ordering, so readers see the overall
framework (three layers, one dynamic filter) before diving into each
layer's implementation. Reworded the opener from 'sort-based
optimizations described above' to 'techniques described below' since
it's now leading rather than summarizing. Promoted the topk_tpch
benchmark section to top-level ## since its parent section moved.
* Add @alamb as second author
Extensive review + collaborative shaping across multiple rounds
warranted co-authorship.
* Wordsmith into, add blog post
* wordsmith
* wordsmith
* wordsmith
* wordsmith
* wordsmith
* wordsmith
* wordsmith
* nits
* make headings more consistent
* Drop PR-number-as-concept + stale (Future B) label
- Rewrote 'unconditional value of [#22450]' as 'what runtime row-group
dynamic pruning fundamentally provides'.
- Dropped '(Future B)' — the A/B Future Directions split no longer exists.
* Update content/blog/2026-07-05-sort-pushdown.md
Co-authored-by: Andrew Lamb <[email protected]>
* Update figure 2, also wordsmit intro
* refine
* shorten
* upate
* tweak
* move buffering without sorting to an appendix"
* rework
* more
* reorder
* improve narrative
* exits
* convert
* convert + tighten
* update
* rework
* hone
* remove redundancy
* remove redundancy
* typos
* updated with related work
* cit
* move
* cleanup
* more
* update
* small tweaks to alamb edits: #22450 attribution + DF version + alt-text
- Row-Level Early Stopping (line ~334): attribute the "skip projection
column fetch" behavior to arrow-rs `RowFilter` (which is where it
actually lives) and clarify that #22450's role is keeping the
threshold up-to-date at RG boundaries so this fires more often.
- Conclusion: replace "In DataFusion 54, we implemented" with "Over
the last few DataFusion releases" — #22450 merged after 54.0.0 was
tagged and will ship in a later release.
- Decision-tree figure alt-text: `try_pushdown_sort` -> `PushdownSort`
to match the struct name used in the surrounding prose.
* add link to raw sort_tpch benchmark numbers (addresses alamb review)
* move merged arrow-rs#10158 to Major PRs; clarify #23067 uses it
* chore: update publish date to 2026-07-20
* update date, fix bold
* wordsmith
* wordsmith
* wordsmith
* cut
* wordsmit
* wordsmit
* remove layer1/2/3 terminology
* more
* fix links
---------
Co-authored-by: Andrew Lamb <[email protected]>
---
content/blog/2026-07-20-sort-pushdown.md | 528 +++++++++++++++++++++
content/images/sort-pushdown/arch_one_glance.png | Bin 0 -> 132456 bytes
content/images/sort-pushdown/benchmark.svg | 78 +++
content/images/sort-pushdown/buffer-exec-stall.svg | 160 +++++++
content/images/sort-pushdown/buffer-exec.svg | 89 ++++
content/images/sort-pushdown/desc_walk_file.svg | 80 ++++
content/images/sort-pushdown/desc_walk_rg.svg | 61 +++
content/images/sort-pushdown/future_page_level.png | Bin 0 -> 152064 bytes
.../images/sort-pushdown/phase1-file-reorder.svg | 88 ++++
.../images/sort-pushdown/phase2-stats-overlap.svg | 89 ++++
content/images/sort-pushdown/plan-diff.svg | 53 +++
content/images/sort-pushdown/pr21956-decision.svg | 87 ++++
.../sort-pushdown/pr21956-runtime-pipeline.svg | 71 +++
content/images/sort-pushdown/pruner_loop.png | Bin 0 -> 124223 bytes
content/images/sort-pushdown/pruning_stack.svg | 84 ++++
content/images/sort-pushdown/reverse-scan.svg | 100 ++++
content/images/sort-pushdown/rg_cascade.svg | 113 +++++
content/images/sort-pushdown/topk_tpch_bench.svg | 121 +++++
.../images/sort-pushdown/transition_anatomy.png | Bin 0 -> 130980 bytes
19 files changed, 1802 insertions(+)
diff --git a/content/blog/2026-07-20-sort-pushdown.md
b/content/blog/2026-07-20-sort-pushdown.md
new file mode 100644
index 0000000..3fdebe4
--- /dev/null
+++ b/content/blog/2026-07-20-sort-pushdown.md
@@ -0,0 +1,528 @@
+---
+layout: post
+title: Optimizing for Almost Sorted Data: Sort Pushdown in Apache DataFusion
+date: 2026-07-20
+author: Qi Zhu and Andrew Lamb
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*[Qi Zhu](https://github.com/zhuqi-lucas)
([Massive](https://www.massive.com/)); [Andrew Lamb](https://github.com/alamb)
([InfluxData](https://www.influxdata.com/))*
+
+[Apache DataFusion] uses sortedness automatically — even when data is only
+partially sorted or when no ordering was declared. This post explains how
+plan-time sort pushdown, runtime scan reordering, and row-group pruning driven
+by [dynamic filters][dyn-filters-blog] make that possible.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]:
https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Are Real Datasets Sorted?
+
+Sorting data is prohibitively expensive for many workloads, so maintaining it
in fully
+sorted order is often not practical.
+However, many real datasets are at least partly sorted when stored:
time-series files by
+ingestion time, event logs by event id, partitioned tables by partition key,
and
+data lakes based on [Apache Iceberg] and similar formats in write order.
+
+Sortedness only helps if the query engine can detect and use it. Two common
+cases make that hard:
+
+1. The ordering is undeclared in the file. For example, the writer did not set
Parquet
+
[sorting_columns](https://github.com/apache/parquet-format/blob/8a5e04bdecf100e8e981daacfa117e8b5aadacb9/src/main/thrift/parquet.thrift#L1044),
+ or the table was not created with DataFusion's
+ [WITH
ORDER](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table)
clause.
+2. Files are individually sorted, but the engine is scanning multiple files
+ and does not know a global ordering at plan time.
+
+In both cases, queries with `ORDER BY` or `ORDER BY ... LIMIT N` pay for a full
+scan and sometimes a blocking full sort, which buffers every row and can
+dominate latency and peak memory on large scans.
+
+Using min/max statistics for *predicate* pushdown is well-known and widely
+implemented across databases, as covered in [@XiangpengHao]'s
+post on [Parquet pruning][parquet-pruning-blog]. Using the same
+statistics to *reason about sort order* and to prune `ORDER BY ... LIMIT`
+(top-k) queries is also increasingly common, for example in the
+[Pruning in Snowflake: Working Smarter, Not Harder] paper (see [Related
Work](#related-work) for more).
+This post describes several such techniques for a general audience,
+including the less-common case of *discovering* a
+global sort order from per-file statistics,
+deleting redundant sorts, and biasing scan order toward the
+most-promising data.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+[@XiangpengHao]: https://github.com/XiangpengHao
+[parquet-pruning-blog]:
https://datafusion.apache.org/blog/2025/03/20/parquet-pruning
+[Pruning in Snowflake: Working Smarter, Not Harder]:
https://dl.acm.org/doi/10.1145/3722212.3724447
+
+## Exact vs. Inexact Ordering
+
+DataFusion has long skipped sorts when it knows the data is **exactly**
sorted, as covered in
+[@akurmustafa's earlier post][ordering-analysis]:
+if the table declares an ordering (via `WITH ORDER` or Parquet
+`sorting_columns`) and the file listing matches it, the redundant sort is
removed.
+
+This post is about the messier real-world cases
+where sortedness exists but is *inexact* or not provable up front:
+
+- Files listed in the "wrong" order on disk (each file is internally
+ sorted, but the files are not globally ordered).
+- Ordering is known, but the sort key ranges **overlap** across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+This post describes two primary techniques:
+
+1. **Statistics-based sort elimination** (`Exact`): avoids sorts entirely by
+ reordering files by min/max statistics to create a global ordering. This
+ extends DataFusion's existing statistics-based file-group ordering
+ ([#9593]) and is similar to prior work on concatenating disjoint ranges.
+2. **Runtime reorder and dynamic pruning** (`Inexact`): reorders the scan to
read
+ the most-promising data first, then re-checks the `TopK` dynamic filter at
file
+ and row-group boundaries.
+
+These techniques are implemented in DataFusion and together result in:
+
+- **Sort elimination**: 2×–49× faster on `ORDER BY` and `ORDER BY ... LIMIT`
+ queries where the file list was in the wrong disk order.
+- **Runtime row-group pruning**: 5 of 11 benchmark queries run ≥2× faster
+ (up to ~4×) with zero regressions; total runtime drops by 44%.
+
+The rest of this post walks through each technique in detail.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#9593]: https://github.com/apache/datafusion/pull/9593
+[ordering-analysis]:
https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## Tracking Ordering
+
+
+The
[PushdownSort](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs)
+optimizer rule classifies each scan below a sort as either `Exact`,
+`Inexact`, or `Unsupported`, and records this information on DataFusion's
+[FileScanConfig](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html):
+
+- **`Exact`** — the optimizer is *certain* of the output order,
+ and removes redundant
[SortExec](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html)
operators entirely.
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it, so the sort is kept but the scan is biased to read the
most-promising data first.
+- **`Unsupported`** — the optimizer cannot determine the ordering, so no sort
is removed.
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="PushdownSort
decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid"
/><br/>
+*Figure: the `PushdownSort` rule asks each scan whether it can satisfy the
+required ordering, returning `Exact` (drop the sort), `Inexact` (bias the scan
+and keep the sort), or `Unsupported` (the sort stays).*
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With **`Exact`** ordering knowledge, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold is more likely to
+ tighten quickly and prune the rest with statistics.
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+
+## Exact: Sort Elimination via Statistics
+
+DataFusion can eliminate sorts entirely when it can use Parquet min/max
statistics to reorder files and prove a global ordering
+(relevant PRs: [#19064] and [#21182]) as shown below.
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before /
after: SortExec eliminated once ordering is Exact" width="100%"
class="img-fluid"/>
+*Figure: EXPLAIN output before and after `PushdownSort` eliminates the sort.
The `SortExec` is removed and the scan's ordering claim is upgraded to `Exact`.*
+
+
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`, `c.parquet`. Each
is
+internally sorted by `ts` and the time ranges do not overlap, but the files
were
+written by different jobs. Without this optimization, a query engine is forced
to
+use a full sort for `ORDER BY ts` queries, even though it could simply read the
+files one after another and produce the correct result.
+
+The `Exact` path requires each file to be
+*individually* sorted on the key, and DataFusion knows this from the declared
+ordering (`WITH ORDER` or Parquet `sorting_columns`).
+Concatenating the files in min/max range order yields a globally sorted stream
+only when that per-file precondition holds. When no per-file ordering is
declared, the
+`Inexact` path (described next) applies instead.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check overlap**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `Sort`. Note that this requires some additional performance
optimizations
+ described in [appendix on buffering without
sorting](#appendix-buffering-without-sorting).
+
+This is an instance of what Graefe calls *virtual concatenation*, and is
similar to
+LSM "trivial move" compaction (see [Related Work](#related-work)). It is less
common
+to discover the disjoint ranges from per-file
+min/max statistics rather than from a declared partitioning scheme.
+
+[graefe2006]: https://dl.acm.org/doi/10.1145/1132960.1132964
+[graefe1993]: https://dl.acm.org/doi/10.1145/152610.152611
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File
reorder: rearranging files within a partition by min/max statistics so the file
list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: reordering by per-file `min/max` puts the file list in range order.*
+
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting
non-overlapping ranges via min/max statistics" width="100%" class="img-fluid"
/><br/>
+*Figure: `PushdownSort` sorts files by `min` and checks adjacency,
+upgrading to `Exact` only when the ranges don't overlap. Left: non-overlapping
+ranges are safe to upgrade to `Exact` and the sort is removed; right:
+overlapping ranges keep the sort and fall through to the `Inexact` path
described next.*
+
+We measured the single-core performance of statistics-based sort elimination
with DataFusion's
+[sort_pushdown](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown)
benchmark suite
+by setting `--partitions 1`. The results are as follows:
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown
benchmark: 2×–49× speedup across four queries" width="100%" class="img-fluid"
/><br/>
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+*Figure: `sort_pushdown` results. All four queries are `ASC` `ORDER BY` where
the
+file list on disk is in the reverse of sort-key order; a naive engine sorts
even
+though the files are already individually sorted.*
+
+While DataFusion can avoid sorting for all four queries, the benefit is most
+dramatic for `LIMIT` queries:
+
+- **Full-scan queries (Q1, Q3)** result in a ~2× speedup as the scan is now a
single-pass streaming read.
+- **`LIMIT` queries (Q2, Q4)** result in a 27×–49× speedup because `LIMIT N`
turns into a streaming read with early stopping.
+
+## Inexact: Scan Reordering Makes Dynamic Filters More Effective
+
+It is not possible to eliminate the sort when the files have
+overlapping sort key ranges. However, it is still possible to use sorted or
+partly sorted data to optimize `LIMIT` queries. By reordering the read,
+data that is most likely to improve the `TopK` dynamic filter is seen first and
+the filter is more effective at pruning files, row groups, and rows that cannot
+possibly contribute to the final result.
+
+This `Inexact` path is used when Parquet min/max statistics are available
+and the sort expression is supported by DataFusion's
+[equivalence-properties][ordering-analysis], including plain columns, monotonic
+functions<sup id="fn1">[1](#footnote1)</sup>, constants inferred from filters,
+and multi-column orderings. The same dynamic filter is used to drive three
layers of pruning:
+
+<img src="/blog/images/sort-pushdown/pruning_stack.svg" alt="Three-layer
pruning: file-level, row-group-level, row-level, all driven by the same TopK
dynamic filter" width="100%" class="img-fluid" /><br/>
+*Figure: the Parquet reader applies three pruning layers, all driven by the
same `TopK` dynamic filter.*
+
+* **File-level pruning** ([file_pruner] + [EarlyStoppingStream]) — skips whole
files before they are opened.
+* **Row-group-level pruning** ([#22450]) — skips whole row groups at each
boundary, before any pages are fetched.
+* **Row-level filtering** ([RowFilter]) — skips decoding the remaining columns
in a surviving row group.
+
+[file_pruner]:
https://docs.rs/datafusion-pruning/latest/datafusion_pruning/struct.FilePruner.html
+[EarlyStoppingStream]:
https://github.com/apache/datafusion/blob/e104138b4d45d3acfb76223cd968385f6764477b/datafusion/datasource-parquet/src/opener/early_stop.rs
+[RowFilter]:
https://docs.rs/parquet/latest/parquet/arrow/arrow_reader/struct.RowFilter.html
+
+When `Inexact` applies, the Parquet opener reorders
+the scan before and during execution using three steps:
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg"
alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse"
width="100%" class="img-fluid" /><br/>
+
+*Figure: the Parquet opener applies file-level reordering, row-group-level
reordering, and an optional iteration reverse.*
+
+1. **File-level reordering** — the file list is sorted by `min(col)`,
+ so the most-promising file is picked first across all partitions.
+2. **Row-group-level reordering** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — for `DESC` queries, the file and row-group order is
reversed.
+
+**File-Level Pruning**: once files are ordered "most-promising first", the
+`TopK`'s heap fills quickly and its dynamic filter threshold tightens. The
+[FilePruner] then cuts low-value files before opening them, as shown in the
+following example.
+
+[FilePruner]:
https://github.com/apache/datafusion/blob/e104138b4d45d3acfb76223cd968385f6764477b/datafusion/pruning/src/file_pruner.rs
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.svg" alt="File-level
reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+
+*Figure: after reordering files by their sort key, the low sort-key value
files (`file_d` and `file_c`,
+whose sort-key values are least promising for this `DESC LIMIT` query)
+end up at the tail of the read queue and are pruned by the file-level pruner
before they
+are ever opened.*
+
+**Row-Group-Level Pruning**: When a file is first opened, DataFusion prunes
+row groups based on predicates and statistics, and then determines
+the order to scan the row groups. During the scan, immediately before
DataFusion
+reads the next row group, it checks whether the `TopK` dynamic filter has
changed. If so,
+it re-evaluates the remaining row groups against the new threshold and
+prunes any that cannot contribute to the final result (added in [#22450]). See
the
+[Appendix: Decoder Loop and Decision
Point](#appendix-decoder-loop-and-decision-point) for more details.
+
+
+<img src="/blog/images/sort-pushdown/rg_cascade.svg" alt="Cascading prune: one
row group fills the heap, threshold snaps, all subsequent row groups are pruned
in a single pass" width="100%" class="img-fluid" /><br/>
+*Figure: for `ORDER BY x DESC LIMIT 10`, after reading the first row group,
DataFusion prunes the rest of the row groups in a single pass (walkthrough
below).*
+
+For the example shown above, the file has 10 row groups, each containing rows
with values
+`[0..10)`, `[10..20)`, ..., `[90..100)` and the query is `ORDER BY x DESC
LIMIT 10`.
+
+1. The row groups are first reordered by their maximum values, so the
highest-value row group is read first.
+2. Row group 9 (values `[90..100)`) is opened and read, filling
+ the heap with 10 values (the minimum heap value is `90`). The `TopK`
dynamic filter threshold is updated to `90`.
+3. At the next row-group boundary, the pruner sees the dynamic filter has been
updated
+ and re-evaluates the filter on all remaining row groups. Since row group 8
+ through row group 0 have `max < 90`, they are all pruned and skipped in a
single pass.
+4. The scan ends early, having read only one row group instead of all 10.
+
+Note that for the row groups that were skipped, no data is fetched or decoded.
+This example shows the core benefit provided by runtime row-group dynamic
+pruning: reading a single row group can cascade-eliminate every remaining row
+group.
+
+**Row-Level Early Stopping**: Even if the row group cannot be pruned with
statistics,
+the tightened dynamic filter often rules out all rows after evaluating just the
+filter columns. When it does, arrow-rs's `RowFilter` skips fetching the
projection
+columns for that row group entirely. [#22450] makes this fire more often
+by keeping the threshold up-to-date at every row-group boundary.
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.svg" alt="Row-group-level
reorder — filter column still read for every row group before row-group filter
early stop" width="100%" class="img-fluid" /><br/>
+
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+
+## Benchmark: sort_tpch
+
+To test the overall benefit of this work, we used DataFusion's
+[sort_tpch](https://github.com/apache/datafusion/blob/main/benchmarks/src/sort_tpch.rs)
+benchmark that runs 11 queries over the TPC-H SF1 dataset, each with
+`ORDER BY ... LIMIT 100`. The data is stored in Parquet files, sorted by
+`l_orderkey`. For the largest table, `lineitem`, `l_orderkey` is a `BIGINT`
with ~1.5M
+distinct values, so per-row-group `min/max` ranges are cleanly disjoint.
+We compared DataFusion with the sort optimizations enabled (the default) and
disabled.
+
+<img src="/blog/images/sort-pushdown/topk_tpch_bench.svg" alt="sort_tpch
benchmark results: 5 of 11 queries ≥2× faster (up to ~4×), 0 regressions, total
-44%" width="100%" class="img-fluid"/>
+
+| Metric | Value |
+| ----------------------------------- | ---------------------------------- |
+| Total wall-clock (sum of 11 queries) | 248.8 ms → 139.1 ms (**−44%**) |
+| Queries with ≥2× speedup | **5 of 11** (Q2, Q4, Q8, Q9, Q10) |
+| Queries with regression | **0** |
+| Best single-query speedup | **~4×** |
+
+*Per-query raw numbers are available in the [#22450 benchmark
run][topk-tpch-raw].*
+
+[topk-tpch-raw]:
https://github.com/apache/datafusion/pull/22450#issuecomment-4765161078
+
+The five queries that improved use `l_orderkey` as the **first** sort key
column, so
+row-group-level pruning can cascade-prune aggressively. The other queries have
multi-column
+sorts with
+low-cardinality or unsorted columns (`l_linenumber`, `l_comment`,
+`l_shipmode`), whose per-row-group ranges overlap heavily. Even when
`l_orderkey`
+appears later as a tie-breaker, the leading key controls row-group-level
disjointness,
+so row-level filtering is still partially effective.
+
+
+## Conclusion and Future Work
+
+Use cases where the query sort key (e.g. `ORDER BY time DESC LIMIT 10`) is
+aligned with the physical layout (e.g. the data is ordered by `time`) are
common
+in time-series, partitioned tables, and ingestion-ordered event logs. Over the
+last few DataFusion releases, we implemented several optimizations that make
+these workloads up to ~4× faster without slowing down queries for which they
+don't apply. We hope you enjoy using them and welcome your feedback.
+
+Two follow-ups are open: page-level `Exact` reverse would let `DESC` queries
drop the sort
+but needs lower-level support in the Parquet reader
([arrow-rs#9937](https://github.com/apache/arrow-rs/pull/9937))
+for reading pages in reverse order.
+We are also considering implementing page-level dynamic pruning at
+row-group boundaries
([#23216](https://github.com/apache/datafusion/issues/23216))
+using the same pattern one level deeper in Parquet.
+
+## Acknowledgements
+
+Thank you to [@adriangb], [@xudong963], [@2010YOUY01], and
+[@Dandandan] for reviewing the design and the patches across many
+iterations. The DataFusion community's willingness to engage deeply
+with optimizer changes, including ones that touch foundational
+code like the Parquet inner decode loop, is what made this work
+possible.
+
+We also thank [Massive](https://www.massive.com/) and
[InfluxData](https://www.influxdata.com/) for sponsoring this
+work.
+
+<a id="footnote1"></a><sup>[1](#fn1)</sup> For example, if the source declares
+`ts DESC`, reversing that ordering gives `ts ASC`, which can satisfy
+`date_trunc('day', ts) ASC`.
+
+[@adriangb]: https://github.com/adriangb
+[@xudong963]: https://github.com/xudong963
+[@2010YOUY01]: https://github.com/2010YOUY01
+[@Dandandan]: https://github.com/Dandandan
+
+## Get Involved
+
+- **Try it out**: Run your `ORDER BY` / `ORDER BY ... LIMIT N` queries on your
own data and share what you see.
+- **Work on a [good first
issue](https://github.com/apache/datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22)**,
or pick up one of the
[follow-ups](https://github.com/apache/datafusion/issues/23036) listed in the
umbrella issue.
+- **File issues or join the conversation**:
[GitHub](https://github.com/apache/datafusion/) for bugs and feature requests,
[Slack or
Discord](https://datafusion.apache.org/contributor-guide/communication.html)
for discussion.
+- Learn more by visiting the
[DataFusion](https://datafusion.apache.org/index.html) project page.
+
+## Related Work
+
+The ideas in this blog build on a long line of research, and several other
+systems implement closely related techniques.
+
+**Exploiting and proving sort order.** Reasoning about orderings to
+eliminate redundant sorts goes back to "interesting orders" in System R
+([Selinger et al., SIGMOD 1979][selinger1979]) and was formalized by
+[Simmen, Shekita, and Malkemus (SIGMOD 1996)][simmen1996]. Those techniques
+derive orderings from *schema* — declared keys, indexes, and functional
+dependencies — whereas we derive them from per-file min/max *statistics*.
+Concatenating disjoint key ranges rather than merging them is Graefe's
+*virtual concatenation* ([ACM Computing Surveys 2006][graefe2006]) — an idea
+rooted further back in range-partitioned parallel sort
+([Graefe, ACM Computing Surveys 1993][graefe1993]). It also appears as
+"trivial move" compaction in LSM trees ([RocksDB][rocksdb-trivial]). For
+*declared* partition bounds, the "concatenate, don't merge" optimization
+appears in [TimescaleDB's OrderedAppend][timescale-append] and
+[PostgreSQL's ordered partition scans][pg-append].
+
+**Top-k early termination and threshold pushdown.** Using a running
+threshold to stop early is described in [Fagin's Threshold Algorithm
+(PODS 2001)][fagin2001], with index choice for faster convergence
+explored in [IO-Top-k (VLDB 2006)][iotopk2006]; see
+[Ilyas et al. (ACM Computing Surveys 2008)][ilyas2008] for a survey.
+Pushing a *live* top-k boundary value into a columnar scan to skip blocks
+via min/max is described for Snowflake in
+[Pruning in Snowflake: Working Smarter, Not Harder] and ships in
+[ClickHouse][ch-topn] (granule-level top-N skipping), [DuckDB][duckdb-topn]
+(dynamic Top-N table filters), [PolarDB-IMCI][polardb-topk] (self-sharpening
+runtime filters), and InfluxDB IOx ([ProgressiveEvalExec][iox-progressive]).
+
+**Reverse scans and data skipping.** Reading physically-ordered data in
+reverse to satisfy `DESC ... LIMIT` is a well-known technique in traditional
+databases (backward/descending index scans in [PostgreSQL][pg-backward] and
[Oracle][oracle-backward]).
+Min/max block skipping itself originates with
+[Small Materialized Aggregates (Moerkotte, VLDB 1998)][sma1998] and is
+now ubiquitous across columnar storage formats and query engines.
+
+[selinger1979]: https://dl.acm.org/doi/10.1145/582095.582099
+[simmen1996]: https://dl.acm.org/doi/10.1145/233269.233320
+[fagin2001]: https://arxiv.org/abs/cs/0204046
+[iotopk2006]: https://www.vldb.org/conf/2006/p475-bast.pdf
+[ilyas2008]: https://dl.acm.org/doi/10.1145/1391729.1391730
+[sma1998]: https://www.vldb.org/conf/1998/p476.pdf
+[rocksdb-trivial]:
https://github.com/facebook/rocksdb/wiki/Compaction-Trivial-Move
+[timescale-append]:
https://www.tigerdata.com/blog/ordered-append-postgresql-optimization
+[pg-append]:
https://www.postgresql.org/message-id/2401607.SfZhPQhbS4@ronan_laptop
+[pg-backward]: https://www.postgresql.org/docs/current/indexes-ordering.html
+[oracle-backward]:
https://docs.oracle.com/en/database/oracle/oracle-database/19/tgsql/optimizer-access-paths.html
+[ch-topn]:
https://clickhouse.com/blog/clickhouse-top-n-queries-granule-level-data-skipping
+[duckdb-topn]: https://duckdb.org/2024/10/25/topn
+[polardb-topk]:
https://www.alibabacloud.com/blog/how-does-the-imci-of-polardb-for-mysql-achieve-ultimate-topk-query-performance_600006
+[iox-progressive]:
https://www.influxdata.com/blog/query-optimization-progressive-evaluation-influxdb/
+
+
+## References
+
+Umbrella issue tracking the entire effort:
+
+* **[EPIC] Sort Pushdown · skip sorts and skip IO for ORDER BY / TopK queries:
[#23036](https://github.com/apache/datafusion/issues/23036)** — phase-by-phase
status of all the PRs and follow-ups.
+
+Major PRs:
+
+* `MinMaxStatistics` foundation:
[#9593](https://github.com/apache/datafusion/pull/9593)
+* `PushdownSort` rule + row-group reverse:
[#19064](https://github.com/apache/datafusion/pull/19064)
+* Reverse-output redesign:
[#19446](https://github.com/apache/datafusion/pull/19446),
[#19557](https://github.com/apache/datafusion/pull/19557)
+* Sort elimination via statistics:
[#21182](https://github.com/apache/datafusion/pull/21182)
+* `BufferExec` capacity for sort elimination:
[#21426](https://github.com/apache/datafusion/pull/21426)
+* Push-based Parquet decoder (DataFusion owns the loop):
[#20839](https://github.com/apache/datafusion/pull/20839)
+* Morsel-style work scheduling:
[#21351](https://github.com/apache/datafusion/pull/21351)
+* Runtime reorder for `TopK` convergence:
[#21956](https://github.com/apache/datafusion/pull/21956)
+* **Runtime row-group dynamic pruning ([#22450])** — the centerpiece of this
post.
+* `peek_next_row_group` API (arrow-rs):
[arrow-rs#10158](https://github.com/apache/arrow-rs/pull/10158) — enables
per-RG `fully_matched` RowFilter skip.
+
+In flight / open:
+
+* Page-level reverse (arrow-rs):
[arrow-rs#9937](https://github.com/apache/arrow-rs/pull/9937), discussion in
[arrow-rs#9934](https://github.com/apache/arrow-rs/issues/9934)
+* Per-RG `fully_matched` RowFilter skip on top of [#22450] (uses
arrow-rs#10158): [#23067](https://github.com/apache/datafusion/issues/23067)
+* Page-level dynamic prune at RG boundary:
[#23216](https://github.com/apache/datafusion/issues/23216)
+* Multi-column / function-wrapped stats reorder follow-ups:
[#22198](https://github.com/apache/datafusion/issues/22198)
+
+Benchmark suites:
[sort_pushdown](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown),
[sort_tpch](https://github.com/apache/datafusion/blob/main/benchmarks/src/sort_tpch.rs).
+
+
+## Appendix: Buffering Without Sorting
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls
when SortExec is removed in multi-partition plans" width="100%"
class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing `SortExec` was not always faster in multi-partition plans: the deleted
+sort had also acted as an implicit buffer. The fix was explicit buffering in
+some plans; see
+[#21426](https://github.com/apache/datafusion/pull/21426)
+for details, as shown in the following figure.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces
the deleted SortExec with a bounded streaming buffer per partition"
width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is
[BufferExec](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs):
+a bounded per-partition prefill buffer that restores the greedy parallel I/O
+driver role without sorting.
+
+
+
+## Appendix: Decoder Loop and Decision Point
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition()
loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%"
class="img-fluid" /><br/>
+*Figure: the decoder loop has three steps. Step 2 (DECIDE) is what
+[#22450] adds — it only fires at row-group boundaries.*
+
+The loop body reads: **drain** the current row group's batches until
+it's exhausted; **decide** at the boundary whether any of the
+remaining row groups can be dropped based on the live threshold; then
+**drive** the decoder into the next row group and repeat. Inside a
+row group, only drain and drive run — no decision point.
+
+<img src="/blog/images/sort-pushdown/pruner_loop.png" alt="RowGroupPruner:
watch (cheap), rebuild (expensive, only if changed), prune (cheap)"
width="100%" class="img-fluid" /><br/>
+*Figure: the pruner has a cheap "check if the filter changed" step, a
+moderately expensive "rebuild the predicate if so" step, and a cheap
+"apply the predicate to remaining row groups" step.*
+
+The pruner does expensive work only when it can help: a cheap epoch check
+detects dynamic-filter changes, and only then rebuilds the pruning predicate.
+The predicate is then applied to remaining row groups' min/max statistics using
+metadata only.
+
diff --git a/content/images/sort-pushdown/arch_one_glance.png
b/content/images/sort-pushdown/arch_one_glance.png
new file mode 100644
index 0000000..7e8dafc
Binary files /dev/null and b/content/images/sort-pushdown/arch_one_glance.png
differ
diff --git a/content/images/sort-pushdown/benchmark.svg
b/content/images/sort-pushdown/benchmark.svg
new file mode 100644
index 0000000..4eed0a7
--- /dev/null
+++ b/content/images/sort-pushdown/benchmark.svg
@@ -0,0 +1,78 @@
+<svg xmlns="http://www.w3.org/2000/svg"; viewBox="0 0 820 380">
+ <style>
+ text { font-family: 'Segoe UI', Arial, sans-serif; }
+ .title { font-size: 14px; font-weight: 700; fill: #222; }
+ .axis-label { font-size: 11px; fill: #555; }
+ .qlabel { font-size: 12px; font-weight: 600; fill: #222; }
+ .qdesc { font-size: 9px; fill: #777; font-family: 'Courier New',
monospace; }
+ .bartext-dark { font-size: 10px; font-weight: 700; fill: #333; }
+ .bartext-on { font-size: 10px; font-weight: 700; fill: #fff; }
+ .speedup { font-size: 13px; font-weight: 700; fill: #27ae60; }
+ .legend { font-size: 11px; fill: #444; }
+ .grid { stroke: #eee; stroke-width: 1; }
+ </style>
+
+ <text class="title" x="410" y="22" text-anchor="middle">sort_pushdown
benchmark (single partition, release, reversed-name data)</text>
+
+ <!-- gridlines + y-axis ticks (log scale: 1 / 10 / 100 / 1000 ms) -->
+ <line class="grid" x1="160" y1="60" x2="645" y2="60"/>
+ <line class="grid" x1="160" y1="147" x2="645" y2="147"/>
+ <line class="grid" x1="160" y1="233" x2="645" y2="233"/>
+ <line x1="160" y1="60" x2="160" y2="320" stroke="#888" stroke-width="1"/>
+ <text class="axis-label" x="153" y="64" text-anchor="end">1000ms</text>
+ <line x1="154" y1="60" x2="160" y2="60" stroke="#888"/>
+ <text class="axis-label" x="153" y="151" text-anchor="end">100ms</text>
+ <line x1="154" y1="147" x2="160" y2="147" stroke="#888"/>
+ <text class="axis-label" x="153" y="237" text-anchor="end">10ms</text>
+ <line x1="154" y1="233" x2="160" y2="233" stroke="#888"/>
+ <text class="axis-label" x="153" y="323" text-anchor="end">1ms</text>
+ <line x1="154" y1="320" x2="160" y2="320" stroke="#888"/>
+
+ <!-- bars per query: HEAD (gray) and PR (green), side-by-side -->
+ <!-- Q1: 259 -> 122, full ORDER BY -->
+ <text class="qlabel" x="225" y="345" text-anchor="middle">Q1</text>
+ <text class="qdesc" x="225" y="358" text-anchor="middle">ORDER BY full</text>
+ <rect x="180" y="111" width="40" height="209" fill="#bdc3c7"/>
+ <text class="bartext-dark" x="200" y="124" text-anchor="middle">259</text>
+ <rect x="230" y="139" width="40" height="181" fill="#27ae60"/>
+ <text class="bartext-on" x="250" y="152" text-anchor="middle">122</text>
+ <text class="speedup" x="225" y="103" text-anchor="middle">2.1×</text>
+
+ <!-- Q2: 80 -> 3, ORDER BY LIMIT -->
+ <text class="qlabel" x="345" y="345" text-anchor="middle">Q2</text>
+ <text class="qdesc" x="345" y="358" text-anchor="middle">ORDER BY
LIMIT</text>
+ <rect x="300" y="155" width="40" height="165" fill="#bdc3c7"/>
+ <text class="bartext-dark" x="320" y="168" text-anchor="middle">80</text>
+ <rect x="350" y="279" width="40" height="41" fill="#27ae60"/>
+ <text class="bartext-on" x="370" y="292" text-anchor="middle">3</text>
+ <text class="speedup" x="345" y="147" text-anchor="middle">27×</text>
+
+ <!-- Q3: 700 -> 313, SELECT * ORDER BY -->
+ <text class="qlabel" x="465" y="345" text-anchor="middle">Q3</text>
+ <text class="qdesc" x="465" y="358" text-anchor="middle">SELECT * ORDER
BY</text>
+ <rect x="420" y="73" width="40" height="247" fill="#bdc3c7"/>
+ <text class="bartext-dark" x="440" y="87" text-anchor="middle">700</text>
+ <rect x="470" y="104" width="40" height="216" fill="#27ae60"/>
+ <text class="bartext-on" x="490" y="117" text-anchor="middle">313</text>
+ <text class="speedup" x="465" y="66" text-anchor="middle">2.2×</text>
+
+ <!-- Q4: 342 -> 7, SELECT * LIMIT -->
+ <text class="qlabel" x="585" y="345" text-anchor="middle">Q4</text>
+ <text class="qdesc" x="585" y="358" text-anchor="middle">SELECT * ORDER BY
LIMIT</text>
+ <rect x="540" y="100" width="40" height="220" fill="#bdc3c7"/>
+ <text class="bartext-dark" x="560" y="113" text-anchor="middle">342</text>
+ <rect x="590" y="247" width="40" height="73" fill="#27ae60"/>
+ <text class="bartext-on" x="610" y="260" text-anchor="middle">7</text>
+ <text class="speedup" x="585" y="92" text-anchor="middle">49×</text>
+
+ <!-- y-axis label -->
+ <text class="axis-label" x="78" y="190" text-anchor="middle"
transform="rotate(-90 78 190)">latency (ms, log scale)</text>
+
+ <!-- legend -->
+ <rect x="660" y="80" width="14" height="14" fill="#bdc3c7"/>
+ <text class="legend" x="680" y="92">main (before)</text>
+ <rect x="660" y="105" width="14" height="14" fill="#27ae60"/>
+ <text class="legend" x="680" y="117">sort pushdown</text>
+ <text class="legend" x="660" y="145" fill="#777">Lower is better</text>
+ <text class="legend" x="660" y="160" fill="#777">--partitions 1,
release</text>
+</svg>
diff --git a/content/images/sort-pushdown/buffer-exec-stall.svg
b/content/images/sort-pushdown/buffer-exec-stall.svg
new file mode 100644
index 0000000..bbcc9a2
--- /dev/null
+++ b/content/images/sort-pushdown/buffer-exec-stall.svg
@@ -0,0 +1,160 @@
+<svg xmlns="http://www.w3.org/2000/svg"; viewBox="0 0 1080 460">
+ <style>
+ text { font-family: 'Segoe UI', Arial, sans-serif; }
+ .title { font-size: 14px; font-weight: 700; fill: #222; }
+ .header-bad { font-size: 18px; font-weight: 800; fill: #c0392b; }
+ .header-good { font-size: 18px; font-weight: 800; fill: #27ae60; }
+ .panel { font-size: 13px; font-weight: 700; fill: #1d1d1f; }
+ .lbl { font-size: 12px; fill: #555; }
+ .lbl-poll { font-size: 11px; fill: #c0392b; font-style: italic; }
+ .lbl-fast { font-size: 12px; fill: #27ae60; font-weight: 700; }
+ .small { font-size: 11px; fill: #666; font-family: 'Courier New',
monospace; }
+ .stall { font-size: 13px; font-weight: 700; fill: #c0392b; }
+ .arr { fill: none; stroke: #555; stroke-width: 1.4; marker-end: url(#arr);
}
+ .arr-good { fill: none; stroke: #27ae60; stroke-width: 1.6; marker-end:
url(#arr-g); }
+ .arr-blocked { fill: none; stroke: #c0392b; stroke-width: 1.4;
stroke-dasharray: 4,3; marker-end: url(#arr-r); }
+ .arr-prefill { fill: none; stroke: #c0392b; stroke-width: 1.2;
stroke-dasharray: 3,3; marker-end: url(#arr-r); }
+ </style>
+ <defs>
+ <marker id="arr" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6"
markerHeight="6" orient="auto">
+ <path d="M0,0 L10,5 L0,10 Z" fill="#555"/>
+ </marker>
+ <marker id="arr-g" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6"
markerHeight="6" orient="auto">
+ <path d="M0,0 L10,5 L0,10 Z" fill="#27ae60"/>
+ </marker>
+ <marker id="arr-r" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6"
markerHeight="6" orient="auto">
+ <path d="M0,0 L10,5 L0,10 Z" fill="#c0392b"/>
+ </marker>
+ </defs>
+
+ <!-- ============ LEFT PANEL: WITHOUT BufferExec ============ -->
+ <text class="header-bad" x="270" y="30" text-anchor="middle">✗ Without
BufferExec</text>
+ <text class="lbl" x="270" y="50" text-anchor="middle">SPM polls I/O directly
— k-way merge stalls on any slow partition</text>
+
+ <!-- Source side -->
+ <rect x="20" y="70" width="120" height="320" rx="6" fill="#fafafa"
stroke="#bbb" stroke-width="1"/>
+ <text class="panel" x="80" y="92" text-anchor="middle">DataSource</text>
+ <text class="small" x="80" y="108" text-anchor="middle">(I/O bound)</text>
+
+ <!-- P0: waiting on I/O -->
+ <rect x="160" y="110" width="220" height="46" rx="4" fill="#fff5f5"
stroke="#c0392b" stroke-width="1.5"/>
+ <text class="lbl" x="172" y="130">Partition 0</text>
+ <text class="stall" x="370" y="138" text-anchor="end">⏳ I/O in flight</text>
+ <text class="small" x="370" y="152" text-anchor="end">no batch ready</text>
+
+ <!-- P1: ready -->
+ <rect x="160" y="170" width="220" height="46" rx="4" fill="#f5f5f7"
stroke="#bbb"/>
+ <text class="lbl" x="172" y="190">Partition 1</text>
+ <rect x="290" y="180" width="20" height="26" fill="#5b8def"/>
+ <text class="small" x="370" y="200" text-anchor="end">ready</text>
+
+ <!-- P2: ready -->
+ <rect x="160" y="230" width="220" height="46" rx="4" fill="#f5f5f7"
stroke="#bbb"/>
+ <text class="lbl" x="172" y="250">Partition 2</text>
+ <rect x="290" y="240" width="20" height="26" fill="#5b8def"/>
+ <text class="small" x="370" y="260" text-anchor="end">ready</text>
+
+ <!-- PN: ready -->
+ <rect x="160" y="290" width="220" height="46" rx="4" fill="#f5f5f7"
stroke="#bbb"/>
+ <text class="lbl" x="172" y="310">Partition N</text>
+ <rect x="290" y="300" width="20" height="26" fill="#5b8def"/>
+ <text class="small" x="370" y="320" text-anchor="end">ready</text>
+
+ <!-- Arrows source → partitions -->
+ <line class="arr" x1="140" y1="133" x2="158" y2="133"/>
+ <line class="arr" x1="140" y1="193" x2="158" y2="193"/>
+ <line class="arr" x1="140" y1="253" x2="158" y2="253"/>
+ <line class="arr" x1="140" y1="313" x2="158" y2="313"/>
+
+ <!-- Blocked arrows partitions → SPM -->
+ <line class="arr-blocked" x1="380" y1="133" x2="430" y2="190"/>
+ <line class="arr-blocked" x1="380" y1="193" x2="430" y2="195"/>
+ <line class="arr-blocked" x1="380" y1="253" x2="430" y2="220"/>
+ <line class="arr-blocked" x1="380" y1="313" x2="430" y2="245"/>
+
+ <!-- SPM blocked -->
+ <rect x="430" y="170" width="90" height="100" rx="6" fill="#fff5f5"
stroke="#c0392b" stroke-width="1.8"/>
+ <text class="panel" x="475" y="194" text-anchor="middle"
fill="#c0392b">SPM</text>
+ <text class="small" x="475" y="212" text-anchor="middle">k-way merge</text>
+ <text class="stall" x="475" y="234" text-anchor="middle">STALLED</text>
+ <text class="small" x="475" y="252" text-anchor="middle">waits for P0</text>
+
+ <!-- Output: nothing -->
+ <text class="stall" x="475" y="300" text-anchor="middle">↓</text>
+ <text class="stall" x="475" y="320" text-anchor="middle">no progress</text>
+
+ <!-- Divider -->
+ <line x1="555" y1="20" x2="555" y2="440" stroke="#ddd" stroke-width="1.5"
stroke-dasharray="2,4"/>
+
+ <!-- ============ RIGHT PANEL: WITH BufferExec ============ -->
+ <text class="header-good" x="820" y="30" text-anchor="middle">✓ With
BufferExec</text>
+ <text class="lbl" x="820" y="50" text-anchor="middle">Background prefill —
SPM always has rows in hand</text>
+
+ <!-- Source side -->
+ <rect x="575" y="70" width="100" height="320" rx="6" fill="#fafafa"
stroke="#bbb" stroke-width="1"/>
+ <text class="panel" x="625" y="92" text-anchor="middle">DataSource</text>
+ <text class="small" x="625" y="108" text-anchor="middle">(I/O bound)</text>
+
+ <!-- BufferExec container -->
+ <rect x="700" y="78" width="280" height="304" rx="6" fill="#fff"
stroke="#5b8def" stroke-width="1.8"/>
+ <text class="panel" x="840" y="98" text-anchor="middle"
fill="#5b8def">BufferExec (bounded)</text>
+
+ <!-- P0 with queue -->
+ <rect x="715" y="110" width="250" height="46" rx="4" fill="#f5f5f7"
stroke="#bbb"/>
+ <text class="lbl" x="728" y="130">Partition 0</text>
+ <rect x="820" y="120" width="18" height="26" fill="#5b8def"/>
+ <rect x="842" y="120" width="18" height="26" fill="#5b8def"/>
+ <rect x="864" y="120" width="18" height="26" fill="#5b8def"/>
+ <rect x="886" y="120" width="18" height="26" fill="none" stroke="#bbb"
stroke-dasharray="2,2"/>
+ <rect x="908" y="120" width="18" height="26" fill="none" stroke="#bbb"
stroke-dasharray="2,2"/>
+ <text class="lbl-fast" x="960" y="140" text-anchor="end">ready</text>
+
+ <!-- P1 -->
+ <rect x="715" y="170" width="250" height="46" rx="4" fill="#f5f5f7"
stroke="#bbb"/>
+ <text class="lbl" x="728" y="190">Partition 1</text>
+ <rect x="820" y="180" width="18" height="26" fill="#5b8def"/>
+ <rect x="842" y="180" width="18" height="26" fill="#5b8def"/>
+ <rect x="864" y="180" width="18" height="26" fill="none" stroke="#bbb"
stroke-dasharray="2,2"/>
+ <rect x="886" y="180" width="18" height="26" fill="none" stroke="#bbb"
stroke-dasharray="2,2"/>
+ <rect x="908" y="180" width="18" height="26" fill="none" stroke="#bbb"
stroke-dasharray="2,2"/>
+ <text class="lbl-fast" x="960" y="200" text-anchor="end">ready</text>
+
+ <!-- P2 -->
+ <rect x="715" y="230" width="250" height="46" rx="4" fill="#f5f5f7"
stroke="#bbb"/>
+ <text class="lbl" x="728" y="250">Partition 2</text>
+ <rect x="820" y="240" width="18" height="26" fill="#5b8def"/>
+ <rect x="842" y="240" width="18" height="26" fill="#5b8def"/>
+ <rect x="864" y="240" width="18" height="26" fill="#5b8def"/>
+ <rect x="886" y="240" width="18" height="26" fill="#5b8def"/>
+ <rect x="908" y="240" width="18" height="26" fill="none" stroke="#bbb"
stroke-dasharray="2,2"/>
+ <text class="lbl-fast" x="960" y="260" text-anchor="end">ready</text>
+
+ <!-- PN -->
+ <rect x="715" y="290" width="250" height="46" rx="4" fill="#f5f5f7"
stroke="#bbb"/>
+ <text class="lbl" x="728" y="310">Partition N</text>
+ <rect x="820" y="300" width="18" height="26" fill="#5b8def"/>
+ <rect x="842" y="300" width="18" height="26" fill="#5b8def"/>
+ <rect x="864" y="300" width="18" height="26" fill="none" stroke="#bbb"
stroke-dasharray="2,2"/>
+ <rect x="886" y="300" width="18" height="26" fill="none" stroke="#bbb"
stroke-dasharray="2,2"/>
+ <rect x="908" y="300" width="18" height="26" fill="none" stroke="#bbb"
stroke-dasharray="2,2"/>
+ <text class="lbl-fast" x="960" y="320" text-anchor="end">ready</text>
+
+ <!-- Background poll arrows source → queues -->
+ <line class="arr-prefill" x1="675" y1="133" x2="818" y2="133"/>
+ <line class="arr-prefill" x1="675" y1="193" x2="818" y2="193"/>
+ <line class="arr-prefill" x1="675" y1="253" x2="818" y2="253"/>
+ <line class="arr-prefill" x1="675" y1="313" x2="818" y2="313"/>
+
+ <!-- Drain arrows queues → SPM (off-panel arrow indicator) -->
+ <text class="lbl-fast" x="1010" y="220" text-anchor="middle">→ SPM</text>
+ <text class="lbl-fast" x="1010" y="238" text-anchor="middle">unblocked</text>
+
+ <!-- Legend -->
+ <rect x="20" y="412" width="1040" height="36" rx="4" fill="#fafafa"
stroke="#ddd"/>
+ <line class="arr-blocked" x1="35" y1="430" x2="75" y2="430"/>
+ <text class="lbl" x="82" y="434">SPM polls partition (blocked when no
batch)</text>
+ <line class="arr-prefill" x1="430" y1="430" x2="470" y2="430"/>
+ <text class="lbl" x="477" y="434">background prefill keeps queue warm</text>
+ <rect x="790" y="420" width="14" height="14" fill="#5b8def"/>
+ <text class="lbl" x="810" y="432">buffered batch (ready to drain)</text>
+</svg>
diff --git a/content/images/sort-pushdown/buffer-exec.svg
b/content/images/sort-pushdown/buffer-exec.svg
new file mode 100644
index 0000000..fe8a744
--- /dev/null
+++ b/content/images/sort-pushdown/buffer-exec.svg
@@ -0,0 +1,89 @@
+<svg xmlns="http://www.w3.org/2000/svg"; viewBox="0 0 920 380">
+ <style>
+ text { font-family: 'Segoe UI', Arial, sans-serif; }
+ .title { font-size: 14px; font-weight: 700; fill: #222; }
+ .panel { font-size: 14px; font-weight: 700; fill: #1d1d1f; }
+ .panel-buf { font-size: 13px; font-weight: 800; fill: #5b8def; }
+ .panel-spm { font-size: 14px; font-weight: 800; fill: #27ae60; }
+ .lbl { font-size: 12px; fill: #555; }
+ .lbl-poll { font-size: 11px; fill: #c0392b; font-style: italic; }
+ .small { font-size: 11px; fill: #666; font-family: 'Courier New',
monospace; }
+ .arr { fill: none; stroke: #555; stroke-width: 1.4; marker-end: url(#arr);
}
+ .arr-bg { fill: none; stroke: #c0392b; stroke-width: 1.2;
stroke-dasharray: 3,3; marker-end: url(#arr-r); }
+ </style>
+ <defs>
+ <marker id="arr" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6"
markerHeight="6" orient="auto">
+ <path d="M0,0 L10,5 L0,10 Z" fill="#555"/>
+ </marker>
+ <marker id="arr-r" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6"
markerHeight="6" orient="auto">
+ <path d="M0,0 L10,5 L0,10 Z" fill="#c0392b"/>
+ </marker>
+ </defs>
+
+ <text class="title" x="460" y="26" text-anchor="middle">BufferExec —
per-partition bounded queues</text>
+
+ <!-- ============ LEFT: Upstream box ============ -->
+ <rect x="30" y="60" width="120" height="280" rx="8" fill="#fafafa"
stroke="#bbb" stroke-width="1"/>
+ <text class="panel" x="90" y="92" text-anchor="middle">DataSource</text>
+ <text class="lbl" x="90" y="112" text-anchor="middle">(I/O bound)</text>
+
+ <!-- ============ MIDDLE: BufferExec ============ -->
+ <rect x="200" y="60" width="500" height="280" rx="8" fill="#fff"
stroke="#5b8def" stroke-width="2"/>
+ <text class="panel-buf" x="450" y="84" text-anchor="middle">BufferExec</text>
+ <text class="small" x="450" y="100" text-anchor="middle">capacity:
sort_pushdown_buffer_capacity · default 1 GB</text>
+
+ <!-- Partition 0 -->
+ <rect x="220" y="118" width="460" height="48" rx="6" fill="#f5f5f7"
stroke="#bbb"/>
+ <text class="lbl" x="234" y="139">Partition 0</text>
+ <rect x="400" y="129" width="22" height="28" fill="#5b8def"/>
+ <rect x="426" y="129" width="22" height="28" fill="#5b8def"/>
+ <rect x="452" y="129" width="22" height="28" fill="#5b8def"/>
+ <rect x="478" y="129" width="22" height="28" fill="none" stroke="#bbb"
stroke-dasharray="2,2"/>
+ <rect x="504" y="129" width="22" height="28" fill="none" stroke="#bbb"
stroke-dasharray="2,2"/>
+ <text class="small" x="660" y="146" text-anchor="end">bounded queue</text>
+
+ <!-- Partition 1 -->
+ <rect x="220" y="178" width="460" height="48" rx="6" fill="#f5f5f7"
stroke="#bbb"/>
+ <text class="lbl" x="234" y="199">Partition 1</text>
+ <rect x="400" y="189" width="22" height="28" fill="#5b8def"/>
+ <rect x="426" y="189" width="22" height="28" fill="#5b8def"/>
+ <rect x="452" y="189" width="22" height="28" fill="none" stroke="#bbb"
stroke-dasharray="2,2"/>
+ <rect x="478" y="189" width="22" height="28" fill="none" stroke="#bbb"
stroke-dasharray="2,2"/>
+ <rect x="504" y="189" width="22" height="28" fill="none" stroke="#bbb"
stroke-dasharray="2,2"/>
+
+ <!-- ... -->
+ <text class="lbl" x="450" y="252" text-anchor="middle">…</text>
+
+ <!-- Partition N -->
+ <rect x="220" y="272" width="460" height="48" rx="6" fill="#f5f5f7"
stroke="#bbb"/>
+ <text class="lbl" x="234" y="293">Partition N</text>
+ <rect x="400" y="283" width="22" height="28" fill="#5b8def"/>
+ <rect x="426" y="283" width="22" height="28" fill="none" stroke="#bbb"
stroke-dasharray="2,2"/>
+ <rect x="452" y="283" width="22" height="28" fill="none" stroke="#bbb"
stroke-dasharray="2,2"/>
+ <rect x="478" y="283" width="22" height="28" fill="none" stroke="#bbb"
stroke-dasharray="2,2"/>
+ <rect x="504" y="283" width="22" height="28" fill="none" stroke="#bbb"
stroke-dasharray="2,2"/>
+
+ <!-- Background poll arrows from upstream -->
+ <line class="arr-bg" x1="150" y1="142" x2="398" y2="142"/>
+ <line class="arr-bg" x1="150" y1="202" x2="398" y2="202"/>
+ <line class="arr-bg" x1="150" y1="296" x2="398" y2="296"/>
+
+ <!-- Drain arrows to SPM -->
+ <line class="arr" x1="680" y1="142" x2="750" y2="142"/>
+ <line class="arr" x1="680" y1="202" x2="750" y2="202"/>
+ <line class="arr" x1="680" y1="296" x2="750" y2="296"/>
+
+ <!-- ============ RIGHT: SPM box ============ -->
+ <rect x="750" y="60" width="140" height="280" rx="8" fill="#fff"
stroke="#27ae60" stroke-width="2"/>
+ <text class="panel-spm" x="820" y="92" text-anchor="middle">SPM</text>
+ <text class="lbl" x="820" y="112" text-anchor="middle">k-way merge</text>
+
+ <!-- Legend -->
+ <rect x="20" y="350" width="880" height="26" rx="4" fill="#fafafa"
stroke="#eee"/>
+ <line class="arr-bg" x1="40" y1="363" x2="80" y2="363"/>
+ <text class="lbl-poll" x="86" y="367">background prefill</text>
+ <line class="arr" x1="280" y1="363" x2="320" y2="363"/>
+ <text class="lbl" x="326" y="367">consumer drains on demand (no I/O
stall)</text>
+ <rect x="650" y="356" width="14" height="14" fill="#5b8def"/>
+ <text class="lbl" x="670" y="367">buffered batch</text>
+</svg>
diff --git a/content/images/sort-pushdown/desc_walk_file.svg
b/content/images/sort-pushdown/desc_walk_file.svg
new file mode 100644
index 0000000..7e41177
--- /dev/null
+++ b/content/images/sort-pushdown/desc_walk_file.svg
@@ -0,0 +1,80 @@
+<svg xmlns="http://www.w3.org/2000/svg"; viewBox="0 0 1010 445">
+ <style>
+ text { font-family: 'Segoe UI', Arial, sans-serif; }
+ .title { font-size: 18px; font-weight: 700; fill: #2c3e7a; }
+ .section { font-size: 13px; font-weight: 700; }
+ .group { font-size: 11px; font-style: italic; fill: #888; }
+ .fname { font-size: 13px; font-weight: 700; }
+ .range { font-size: 11px; font-family: 'Courier New', monospace; }
+ .banner { font-size: 14px; font-weight: 700; fill: #1b5e20; }
+ .banner-sub { font-size: 12px; font-style: italic; fill: #2e7d32; }
+ .annot { font-size: 13px; font-weight: 700; fill: #333; }
+ .annot-sub { font-size: 12px; font-style: italic; fill: #666; }
+ .reorder { font-size: 12px; font-style: italic; fill: #555; }
+ .arrow { fill: none; stroke: #555; stroke-width: 2; marker-end:
url(#arr); }
+ </style>
+ <defs>
+ <marker id="arr" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6"
markerHeight="6" orient="auto">
+ <path d="M0,0 L10,5 L0,10 Z" fill="#555"/>
+ </marker>
+ </defs>
+
+ <text class="title" x="505" y="32" text-anchor="middle">File-level reorder
(per partition)</text>
+
+ <!-- top banner: early stop available -->
+ <rect x="250" y="50" width="510" height="40" rx="8" fill="#e8f5e9"
stroke="#27ae60" stroke-width="1.5"/>
+ <text class="banner" x="505" y="75" text-anchor="middle"
font-size="13">✓ EARLY STOP available · file_pruner +
EarlyStoppingStream</text>
+
+ <!-- ============ BEFORE ============ -->
+ <text class="section" x="40" y="125" fill="#666">BEFORE</text>
+
+ <text class="group" x="70" y="150">group 0</text>
+ <rect x="65" y="158" width="135" height="50" rx="6" fill="#e8eefc"
stroke="#5b8def" stroke-width="1.5"/>
+ <text class="fname" x="132" y="182" text-anchor="middle"
fill="#1f4fa3">file_d</text>
+ <text class="range" x="132" y="200" text-anchor="middle"
fill="#5b8def">[10..20)</text>
+ <rect x="215" y="158" width="135" height="50" rx="6" fill="#e8eefc"
stroke="#5b8def" stroke-width="1.5"/>
+ <text class="fname" x="282" y="182" text-anchor="middle"
fill="#1f4fa3">file_b</text>
+ <text class="range" x="282" y="200" text-anchor="middle"
fill="#5b8def">[50..60)</text>
+
+ <text class="group" x="425" y="150">group 1</text>
+ <rect x="420" y="158" width="135" height="50" rx="6" fill="#e8eefc"
stroke="#5b8def" stroke-width="1.5"/>
+ <text class="fname" x="487" y="182" text-anchor="middle"
fill="#1f4fa3">file_c</text>
+ <text class="range" x="487" y="200" text-anchor="middle"
fill="#5b8def">[0..10)</text>
+ <rect x="570" y="158" width="135" height="50" rx="6" fill="#e8eefc"
stroke="#5b8def" stroke-width="1.5"/>
+ <text class="fname" x="637" y="182" text-anchor="middle"
fill="#1f4fa3">file_a</text>
+ <text class="range" x="637" y="200" text-anchor="middle"
fill="#5b8def">[30..40)</text>
+
+ <!-- reorder arrow -->
+ <line class="arrow" x1="770" y1="165" x2="770" y2="290"/>
+ <text class="reorder" x="790" y="215">reorder files by</text>
+ <text class="reorder" x="790" y="232">min/max statistics</text>
+
+ <!-- ============ AFTER ============ -->
+ <text class="section" x="40" y="280" fill="#1b5e20">AFTER</text>
+
+ <text class="group" x="70" y="305">group 0</text>
+ <rect x="65" y="313" width="135" height="50" rx="6" fill="#e8f5e9"
stroke="#27ae60" stroke-width="1.5"/>
+ <text class="fname" x="132" y="337" text-anchor="middle"
fill="#1b5e20">file_b</text>
+ <text class="range" x="132" y="355" text-anchor="middle"
fill="#2e7d32">[50..60)</text>
+ <rect x="215" y="313" width="135" height="50" rx="6" fill="#f4f4f4"
stroke="#dddddd" stroke-width="1.5"/>
+ <text class="fname" x="282" y="337" text-anchor="middle"
fill="#bcbcbc">file_d</text>
+ <text class="range" x="282" y="355" text-anchor="middle"
fill="#c8c8c8">[10..20)</text>
+
+ <text class="group" x="425" y="305">group 1</text>
+ <rect x="420" y="313" width="135" height="50" rx="6" fill="#e8f5e9"
stroke="#27ae60" stroke-width="1.5"/>
+ <text class="fname" x="487" y="337" text-anchor="middle"
fill="#1b5e20">file_a</text>
+ <text class="range" x="487" y="355" text-anchor="middle"
fill="#2e7d32">[30..40)</text>
+ <rect x="570" y="313" width="135" height="50" rx="6" fill="#f4f4f4"
stroke="#dddddd" stroke-width="1.5"/>
+ <text class="fname" x="637" y="337" text-anchor="middle"
fill="#bcbcbc">file_c</text>
+ <text class="range" x="637" y="355" text-anchor="middle"
fill="#c8c8c8">[0..10)</text>
+
+ <!-- after annotations -->
+ <text class="annot" x="745" y="325">high-value files come first</text>
+ <text class="annot-sub" x="745" y="345">→ TopK heap fills fast</text>
+ <text class="annot-sub" x="745" y="362">→ threshold tightens</text>
+
+ <!-- bottom banner: never loaded -->
+ <rect x="40" y="385" width="925" height="50" rx="8" fill="#e8f5e9"
stroke="#27ae60" stroke-width="1.5"/>
+ <text class="banner" x="502" y="408" text-anchor="middle">✓
low-value files (dimmed) are NEVER LOADED</text>
+ <text class="banner-sub" x="502" y="426" text-anchor="middle">file_pruner +
EarlyStoppingStream skip their metadata and scan entirely</text>
+</svg>
diff --git a/content/images/sort-pushdown/desc_walk_rg.svg
b/content/images/sort-pushdown/desc_walk_rg.svg
new file mode 100644
index 0000000..945ce29
--- /dev/null
+++ b/content/images/sort-pushdown/desc_walk_rg.svg
@@ -0,0 +1,61 @@
+<svg xmlns="http://www.w3.org/2000/svg"; viewBox="0 0 1010 405">
+ <style>
+ text { font-family: 'Segoe UI', Arial, sans-serif; }
+ .title { font-size: 18px; font-weight: 700; fill: #2c3e7a; }
+ .section { font-size: 13px; font-weight: 700; fill: #666; }
+ .range { font-size: 13px; font-family: 'Courier New', monospace; }
+ .banner { font-size: 13px; font-weight: 700; fill: #1b5e20; }
+ .reorder { font-size: 12px; font-style: italic; fill: #555; }
+ .a-good { font-size: 12px; font-weight: 700; fill: #1b5e20; }
+ .a-good-sub { font-size: 11px; font-style: italic; fill: #2e7d32; }
+ .a-bad { font-size: 12px; font-weight: 700; fill: #c0392b; }
+ .a-bad-sub { font-size: 11px; font-style: italic; fill: #c0392b; }
+ .a-hi { font-size: 12px; font-weight: 700; fill: #b9770e; }
+ .arrow { fill: none; stroke: #555; stroke-width: 2; marker-end:
url(#arr); }
+ </style>
+ <defs>
+ <marker id="arr" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6"
markerHeight="6" orient="auto">
+ <path d="M0,0 L10,5 L0,10 Z" fill="#555"/>
+ </marker>
+ </defs>
+
+ <text class="title" x="505" y="30" text-anchor="middle">Row-group early stop
(inside one file)</text>
+
+ <!-- top banner -->
+ <rect x="225" y="48" width="560" height="38" rx="8" fill="#e8f5e9"
stroke="#27ae60" stroke-width="1.5"/>
+ <text class="banner" x="505" y="72" text-anchor="middle">✓ #22450
· row groups skip projection columns at runtime</text>
+
+ <!-- ============ Original ============ -->
+ <text class="section" x="40" y="122">Original</text>
+ <rect x="65" y="132" width="130" height="50" rx="6" fill="#e8eefc"
stroke="#5b8def" stroke-width="1.5"/>
+ <text class="range" x="130" y="162" text-anchor="middle"
fill="#1f4fa3">[10..20)</text>
+ <rect x="210" y="132" width="130" height="50" rx="6" fill="#e8eefc"
stroke="#5b8def" stroke-width="1.5"/>
+ <text class="range" x="275" y="162" text-anchor="middle"
fill="#1f4fa3">[50..60)</text>
+ <rect x="355" y="132" width="130" height="50" rx="6" fill="#e8eefc"
stroke="#5b8def" stroke-width="1.5"/>
+ <text class="range" x="420" y="162" text-anchor="middle"
fill="#1f4fa3">[0..10)</text>
+ <rect x="500" y="132" width="130" height="50" rx="6" fill="#e8eefc"
stroke="#5b8def" stroke-width="1.5"/>
+ <text class="range" x="565" y="162" text-anchor="middle"
fill="#1f4fa3">[30..40)</text>
+
+ <!-- reorder arrow -->
+ <line class="arrow" x1="770" y1="150" x2="770" y2="285"/>
+ <text class="reorder" x="788" y="205">reorder by min/max</text>
+ <text class="reorder" x="788" y="222">statistics, then reverse</text>
+
+ <!-- ============ Decoder reads ============ -->
+ <text class="section" x="40" y="298">Decoder reads</text>
+ <rect x="65" y="308" width="130" height="50" rx="6" fill="#fdecdc"
stroke="#e67e22" stroke-width="1.5"/>
+ <text class="range" x="130" y="338" text-anchor="middle"
fill="#b9770e">[50..60)</text>
+ <rect x="210" y="308" width="130" height="50" rx="6" fill="#fdecdc"
stroke="#e67e22" stroke-width="1.5"/>
+ <text class="range" x="275" y="338" text-anchor="middle"
fill="#b9770e">[30..40)</text>
+ <rect x="355" y="308" width="130" height="50" rx="6" fill="#fdecdc"
stroke="#e67e22" stroke-width="1.5"/>
+ <text class="range" x="420" y="338" text-anchor="middle"
fill="#b9770e">[10..20)</text>
+ <rect x="500" y="308" width="130" height="50" rx="6" fill="#fdecdc"
stroke="#e67e22" stroke-width="1.5"/>
+ <text class="range" x="565" y="338" text-anchor="middle"
fill="#b9770e">[0..10)</text>
+
+ <!-- annotations -->
+ <text class="a-hi" x="650" y="316">high-value row groups first →
heap fills fast</text>
+ <text class="a-good" x="650" y="338">✓ later row groups
short-circuit projection columns</text>
+ <text class="a-good-sub" x="650" y="354">(RowSelection empty →
arrow-rs selects_any())</text>
+ <text class="a-bad" x="650" y="376">✗ but the filter column is
still read for each row group</text>
+ <text class="a-bad-sub" x="650" y="392">(the sort column is fetched +
decoded to check the threshold)</text>
+</svg>
diff --git a/content/images/sort-pushdown/future_page_level.png
b/content/images/sort-pushdown/future_page_level.png
new file mode 100644
index 0000000..e41f781
Binary files /dev/null and b/content/images/sort-pushdown/future_page_level.png
differ
diff --git a/content/images/sort-pushdown/phase1-file-reorder.svg
b/content/images/sort-pushdown/phase1-file-reorder.svg
new file mode 100644
index 0000000..00d8e24
--- /dev/null
+++ b/content/images/sort-pushdown/phase1-file-reorder.svg
@@ -0,0 +1,88 @@
+<svg xmlns="http://www.w3.org/2000/svg"; viewBox="-25 0 845 320">
+ <style>
+ text { font-family: 'Segoe UI', Arial, sans-serif; }
+ .title { font-size: 14px; font-weight: 700; fill: #222; }
+ .sub { font-size: 12px; fill: #555; }
+ .filename { font-size: 11px; font-weight: 600; fill: #fff; }
+ .range { font-size: 11px; fill: #333; font-family: 'Courier New',
monospace; }
+ .label { font-size: 11px; fill: #555; }
+ .arrow { fill: none; stroke: #555; stroke-width: 1.6; marker-end:
url(#arrow); }
+ .verdict-good { font-size: 11px; font-weight: 600; fill: #27ae60; }
+ .verdict-bad { font-size: 11px; font-weight: 600; fill: #c0392b; }
+ </style>
+ <defs>
+ <marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6"
markerHeight="6" orient="auto">
+ <path d="M0,0 L10,5 L0,10 Z" fill="#555"/>
+ </marker>
+ </defs>
+
+ <!-- Title -->
+ <text class="title" x="410" y="22" text-anchor="middle">File rearrangement
by min/max statistics</text>
+
+ <!-- BEFORE column -->
+ <text class="sub" x="20" y="55">Before — directory order:</text>
+
+ <!-- file boxes (before) -->
+ <rect x="20" y="65" width="160" height="36" rx="4" fill="#5b8def"/>
+ <text class="filename" x="100" y="80" text-anchor="middle">a.parquet</text>
+ <text class="range" x="100" y="95" text-anchor="middle" fill="#fff">ts ∈
[200, 300]</text>
+
+ <rect x="20" y="111" width="160" height="36" rx="4" fill="#5b8def"/>
+ <text class="filename" x="100" y="126" text-anchor="middle">b.parquet</text>
+ <text class="range" x="100" y="141" text-anchor="middle" fill="#fff">ts ∈
[100, 200]</text>
+
+ <rect x="20" y="157" width="160" height="36" rx="4" fill="#5b8def"/>
+ <text class="filename" x="100" y="172" text-anchor="middle">c.parquet</text>
+ <text class="range" x="100" y="187" text-anchor="middle" fill="#fff">ts ∈
[0, 100]</text>
+
+ <text class="verdict-bad" x="100" y="220"
text-anchor="middle">validated_output_ordering() = Unsupported</text>
+ <text class="label" x="100" y="238" text-anchor="middle">→ Sort
required</text>
+
+ <!-- arrow -->
+ <line class="arrow" x1="210" y1="130" x2="320" y2="130"/>
+ <text class="label" x="265" y="120" text-anchor="middle">PushdownSort</text>
+ <text class="label" x="265" y="146" text-anchor="middle">sort by
min(ts)</text>
+
+ <!-- AFTER column -->
+ <text class="sub" x="350" y="55">After — sorted by stats:</text>
+
+ <rect x="350" y="65" width="160" height="36" rx="4" fill="#27ae60"/>
+ <text class="filename" x="430" y="80" text-anchor="middle">c.parquet</text>
+ <text class="range" x="430" y="95" text-anchor="middle" fill="#fff">ts ∈ [0,
100]</text>
+
+ <rect x="350" y="111" width="160" height="36" rx="4" fill="#27ae60"/>
+ <text class="filename" x="430" y="126" text-anchor="middle">b.parquet</text>
+ <text class="range" x="430" y="141" text-anchor="middle" fill="#fff">ts ∈
[100, 200]</text>
+
+ <rect x="350" y="157" width="160" height="36" rx="4" fill="#27ae60"/>
+ <text class="filename" x="430" y="172" text-anchor="middle">a.parquet</text>
+ <text class="range" x="430" y="187" text-anchor="middle" fill="#fff">ts ∈
[200, 300]</text>
+
+ <text class="verdict-good" x="430" y="220"
text-anchor="middle">validated_output_ordering() = Exact</text>
+ <text class="label" x="430" y="238" text-anchor="middle">→ Sort
removed</text>
+
+ <!-- Right column: number line -->
+ <text class="sub" x="640" y="55" text-anchor="middle">Range layout</text>
+ <line x1="560" y1="130" x2="800" y2="130" stroke="#888" stroke-width="1"/>
+ <line x1="560" y1="125" x2="560" y2="135" stroke="#888"/>
+ <line x1="640" y1="125" x2="640" y2="135" stroke="#888"/>
+ <line x1="720" y1="125" x2="720" y2="135" stroke="#888"/>
+ <line x1="800" y1="125" x2="800" y2="135" stroke="#888"/>
+ <text class="range" x="560" y="148" text-anchor="middle">0</text>
+ <text class="range" x="640" y="148" text-anchor="middle">100</text>
+ <text class="range" x="720" y="148" text-anchor="middle">200</text>
+ <text class="range" x="800" y="148" text-anchor="middle">300</text>
+
+ <!-- range bars -->
+ <rect x="560" y="80" width="80" height="14" rx="3" fill="#27ae60"/>
+ <text class="range" x="600" y="91" text-anchor="middle" fill="#fff">c</text>
+ <rect x="640" y="98" width="80" height="14" rx="3" fill="#27ae60"/>
+ <text class="range" x="680" y="109" text-anchor="middle" fill="#fff">b</text>
+ <rect x="720" y="116" width="80" height="14" rx="3" fill="#27ae60"/>
+ <text class="range" x="760" y="127" text-anchor="middle" fill="#fff">a</text>
+
+ <text class="label" x="680" y="170" text-anchor="middle">Non-overlapping →
ordering provable</text>
+
+ <!-- Bottom: SQL hint -->
+ <text x="410" y="290" text-anchor="middle" font-size="12" fill="#555"
font-family="'Courier New', monospace">SELECT * FROM events ORDER BY ts</text>
+</svg>
diff --git a/content/images/sort-pushdown/phase2-stats-overlap.svg
b/content/images/sort-pushdown/phase2-stats-overlap.svg
new file mode 100644
index 0000000..8580c21
--- /dev/null
+++ b/content/images/sort-pushdown/phase2-stats-overlap.svg
@@ -0,0 +1,89 @@
+<svg xmlns="http://www.w3.org/2000/svg"; viewBox="0 0 1000 440">
+ <style>
+ text { font-family: 'Segoe UI', Arial, sans-serif; }
+ .title { font-size: 20px; font-weight: 800; fill: #222; }
+ .panel-title { font-size: 18px; font-weight: 800; }
+ .label { font-size: 14px; fill: #555; }
+ .axis-num { font-size: 14px; fill: #333; font-family: 'Courier New',
monospace; }
+ .file-label { font-size: 15px; fill: #1d1d1f; font-family: 'Courier New',
monospace; font-weight: 700; }
+ .verdict-good { font-size: 18px; font-weight: 800; fill: #27ae60; }
+ .verdict-bad { font-size: 18px; font-weight: 800; fill: #c0392b; }
+ .verdict-sub { font-size: 14px; fill: #555; }
+ .axis { stroke: #888; stroke-width: 1.4; }
+ .tick { stroke: #888; stroke-width: 1.4; }
+ .gap { stroke: #27ae60; stroke-dasharray: 4,4; stroke-width: 1.4; }
+ .footer { font-size: 14px; fill: #555; font-style: italic; }
+ </style>
+
+ <text class="title" x="500" y="32" text-anchor="middle">Using min/max
statistics to prove non-overlap</text>
+
+ <!-- ============ LEFT: Non-overlapping (Exact) ============ -->
+ <rect x="30" y="55" width="460" height="370" rx="10" fill="#e8f5e9"
stroke="#27ae60" stroke-width="2"/>
+ <text class="panel-title" x="260" y="84" text-anchor="middle"
fill="#27ae60">Non-overlapping ranges</text>
+
+ <!-- file_c -->
+ <text class="file-label" x="135" y="112" text-anchor="middle">file_c
[0..100]</text>
+ <rect x="70" y="120" width="130" height="28" rx="4" fill="#27ae60"/>
+
+ <!-- file_b -->
+ <text class="file-label" x="265" y="172" text-anchor="middle">file_b
[100..200]</text>
+ <rect x="200" y="180" width="130" height="28" rx="4" fill="#27ae60"/>
+
+ <!-- file_a -->
+ <text class="file-label" x="395" y="232" text-anchor="middle">file_a
[200..300]</text>
+ <rect x="330" y="240" width="130" height="28" rx="4" fill="#27ae60"/>
+
+ <!-- axis -->
+ <line class="axis" x1="70" y1="300" x2="460" y2="300"/>
+ <line class="tick" x1="70" y1="294" x2="70" y2="306"/>
+ <line class="tick" x1="200" y1="294" x2="200" y2="306"/>
+ <line class="tick" x1="330" y1="294" x2="330" y2="306"/>
+ <line class="tick" x1="460" y1="294" x2="460" y2="306"/>
+ <text class="axis-num" x="70" y="322" text-anchor="middle">0</text>
+ <text class="axis-num" x="200" y="322" text-anchor="middle">100</text>
+ <text class="axis-num" x="330" y="322" text-anchor="middle">200</text>
+ <text class="axis-num" x="460" y="322" text-anchor="middle">300</text>
+ <text class="label" x="265" y="344" text-anchor="middle">min(ts) /
max(ts)</text>
+
+ <!-- gap markers -->
+ <line class="gap" x1="200" y1="112" x2="200" y2="298"/>
+ <line class="gap" x1="330" y1="172" x2="330" y2="298"/>
+
+ <text class="verdict-good" x="260" y="383" text-anchor="middle">Ordering:
Exact ✓</text>
+ <text class="verdict-sub" x="260" y="406" text-anchor="middle">Sort can be
removed</text>
+
+ <!-- ============ RIGHT: Overlapping (Inexact) ============ -->
+ <rect x="510" y="55" width="460" height="370" rx="10" fill="#fde8e8"
stroke="#c0392b" stroke-width="2"/>
+ <text class="panel-title" x="740" y="84" text-anchor="middle"
fill="#c0392b">Overlapping ranges</text>
+
+ <!-- file_x -->
+ <text class="file-label" x="667" y="112" text-anchor="middle">file_x
[0..180]</text>
+ <rect x="550" y="120" width="234" height="28" rx="4" fill="#c0392b"
opacity="0.85"/>
+
+ <!-- file_y -->
+ <text class="file-label" x="771" y="172" text-anchor="middle">file_y
[80..260]</text>
+ <rect x="654" y="180" width="234" height="28" rx="4" fill="#c0392b"
opacity="0.85"/>
+
+ <!-- file_z -->
+ <text class="file-label" x="836" y="232" text-anchor="middle">file_z
[140..300]</text>
+ <rect x="732" y="240" width="208" height="28" rx="4" fill="#c0392b"
opacity="0.85"/>
+
+ <!-- overlap shading -->
+ <rect x="654" y="120" width="130" height="28" fill="#c0392b" opacity="0.30"/>
+ <rect x="732" y="180" width="156" height="28" fill="#c0392b" opacity="0.30"/>
+
+ <!-- axis -->
+ <line class="axis" x1="550" y1="300" x2="940" y2="300"/>
+ <line class="tick" x1="550" y1="294" x2="550" y2="306"/>
+ <line class="tick" x1="680" y1="294" x2="680" y2="306"/>
+ <line class="tick" x1="810" y1="294" x2="810" y2="306"/>
+ <line class="tick" x1="940" y1="294" x2="940" y2="306"/>
+ <text class="axis-num" x="550" y="322" text-anchor="middle">0</text>
+ <text class="axis-num" x="680" y="322" text-anchor="middle">100</text>
+ <text class="axis-num" x="810" y="322" text-anchor="middle">200</text>
+ <text class="axis-num" x="940" y="322" text-anchor="middle">300</text>
+ <text class="label" x="745" y="344" text-anchor="middle">min(ts) /
max(ts)</text>
+
+ <text class="verdict-bad" x="740" y="383" text-anchor="middle">Ordering:
Inexact (overlap)</text>
+ <text class="verdict-sub" x="740" y="406" text-anchor="middle">Sort
stays</text>
+</svg>
diff --git a/content/images/sort-pushdown/plan-diff.svg
b/content/images/sort-pushdown/plan-diff.svg
new file mode 100644
index 0000000..8fdd8bc
--- /dev/null
+++ b/content/images/sort-pushdown/plan-diff.svg
@@ -0,0 +1,53 @@
+<svg xmlns="http://www.w3.org/2000/svg"; viewBox="0 0 820 360">
+ <style>
+ text { font-family: 'Segoe UI', Arial, sans-serif; }
+ .title { font-size: 14px; font-weight: 700; fill: #222; }
+ .panel { font-size: 13px; font-weight: 700; }
+ .node { font-size: 12px; font-weight: 600; fill: #fff; }
+ .detail { font-size: 10px; font-family: 'Courier New', monospace; fill:
#fff; }
+ .annot { font-size: 11px; fill: #c0392b; font-style: italic; }
+ .annot-g { font-size: 11px; fill: #27ae60; font-style: italic;
font-weight: 600; }
+ .arrow { fill: none; stroke: #555; stroke-width: 1.4; marker-end:
url(#arr2); }
+ </style>
+ <defs>
+ <marker id="arr2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6"
markerHeight="6" orient="auto">
+ <path d="M0,0 L10,5 L0,10 Z" fill="#555"/>
+ </marker>
+ </defs>
+
+ <text class="title" x="410" y="22" text-anchor="middle">EXPLAIN before /
after sort pushdown (single-partition)</text>
+
+ <!-- BEFORE -->
+ <rect x="20" y="40" width="370" height="300" rx="8" fill="#fff"
stroke="#c0392b" stroke-width="1.5"/>
+ <text class="panel" x="205" y="62" text-anchor="middle"
fill="#c0392b">Before — SortExec on top of scan</text>
+
+ <rect x="65" y="100" width="280" height="58" rx="4" fill="#c0392b"/>
+ <text class="node" x="205" y="122" text-anchor="middle">SortExec:
TopK(fetch=3)</text>
+ <text class="detail" x="205" y="140" text-anchor="middle">expr=[ts@0 ASC
NULLS LAST]</text>
+ <text class="detail" x="205" y="154"
text-anchor="middle">preserve_partitioning=[false]</text>
+
+ <line class="arrow" x1="205" y1="164" x2="205" y2="184"/>
+
+ <rect x="65" y="186" width="280" height="58" rx="4" fill="#5b8def"/>
+ <text class="node" x="205" y="208" text-anchor="middle">DataSourceExec</text>
+ <text class="detail" x="205" y="226" text-anchor="middle">file_groups={a, b,
c}</text>
+ <text class="detail" x="205" y="240"
text-anchor="middle">file_type=parquet</text>
+
+ <text class="annot" x="205" y="284" text-anchor="middle">blocking full sort
across the scan</text>
+ <text class="annot" x="205" y="302" text-anchor="middle">LIMIT applied after
sort</text>
+
+ <!-- AFTER -->
+ <rect x="430" y="40" width="370" height="300" rx="8" fill="#fff"
stroke="#27ae60" stroke-width="1.5"/>
+ <text class="panel" x="615" y="62" text-anchor="middle" fill="#27ae60">After
— SortExec eliminated</text>
+
+ <rect x="475" y="120" width="280" height="78" rx="4" fill="#27ae60"/>
+ <text class="node" x="615" y="142" text-anchor="middle">DataSourceExec</text>
+ <text class="detail" x="615" y="160" text-anchor="middle">file_groups={c, b,
a}</text>
+ <text class="detail" x="615" y="174" text-anchor="middle">limit=3</text>
+ <text class="detail" x="615" y="188"
text-anchor="middle">output_ordering=[ts ASC NULLS LAST]</text>
+
+ <text class="annot-g" x="615" y="232" text-anchor="middle">files reordered
by min/max stats</text>
+ <text class="annot-g" x="615" y="250" text-anchor="middle">LIMIT becomes a
static fetch on the source</text>
+ <text class="annot-g" x="615" y="268" text-anchor="middle">reader stops the
moment 3 rows are emitted</text>
+ <text class="annot-g" x="615" y="286" text-anchor="middle">no blocking sort,
no full scan</text>
+</svg>
diff --git a/content/images/sort-pushdown/pr21956-decision.svg
b/content/images/sort-pushdown/pr21956-decision.svg
new file mode 100644
index 0000000..f03aa85
--- /dev/null
+++ b/content/images/sort-pushdown/pr21956-decision.svg
@@ -0,0 +1,87 @@
+<svg xmlns="http://www.w3.org/2000/svg"; viewBox="0 0 1080 500">
+ <style>
+ text { font-family: 'Segoe UI', Arial, sans-serif; }
+ .title { font-size: 20px; font-weight: 800; fill: #222; }
+ .entry-label { font-size: 16px; font-weight: 800; fill: #fff; }
+ .entry-sub { font-size: 13px; fill: #d8e0e8; font-family: 'Courier New',
monospace; }
+ .check-label { font-size: 14px; fill: #1d1d1f; font-weight: 700; }
+ .check-sub { font-size: 13px; fill: #555; font-family: 'Courier New',
monospace; }
+ .term-label { font-size: 17px; font-weight: 800; fill: #fff; }
+ .term-sub { font-size: 13px; fill: #fff; font-family: 'Courier New',
monospace; }
+ .branch-yes { font-size: 14px; font-weight: 800; fill: #27ae60; }
+ .branch-no { font-size: 14px; font-weight: 800; fill: #c0392b; }
+ .effect { font-size: 13px; fill: #555; }
+ .arr { fill: none; stroke: #555; stroke-width: 1.8; marker-end: url(#arr);
}
+ .arr-yes { fill: none; stroke: #27ae60; stroke-width: 2; marker-end:
url(#arr-g); }
+ .arr-no { fill: none; stroke: #888; stroke-width: 1.8; marker-end:
url(#arr); }
+ </style>
+ <defs>
+ <marker id="arr" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6"
markerHeight="6" orient="auto">
+ <path d="M0,0 L10,5 L0,10 Z" fill="#555"/>
+ </marker>
+ <marker id="arr-g" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6"
markerHeight="6" orient="auto">
+ <path d="M0,0 L10,5 L0,10 Z" fill="#27ae60"/>
+ </marker>
+ </defs>
+
+ <text class="title" x="540" y="32" text-anchor="middle">try_pushdown_sort —
Exact / Inexact / Unsupported</text>
+
+ <!-- ============ Entry pill ============ -->
+ <rect x="40" y="60" width="280" height="60" rx="30" fill="#34495e"/>
+ <text class="entry-label" x="180" y="86" text-anchor="middle">PushdownSort
rule</text>
+ <text class="entry-sub" x="180" y="106"
text-anchor="middle">source.try_pushdown_sort(req)</text>
+
+ <!-- arrow entry → check1 -->
+ <line class="arr" x1="320" y1="90" x2="370" y2="90"/>
+
+ <!-- ============ Check 1 ============ -->
+ <rect x="370" y="50" width="320" height="80" rx="10" fill="#fff"
stroke="#5b8def" stroke-width="2"/>
+ <text class="check-label" x="530" y="80" text-anchor="middle">Natural
ordering satisfies request?</text>
+ <text class="check-sub" x="530" y="104"
text-anchor="middle">eq.ordering_satisfy(req)</text>
+
+ <!-- Branch yes → Exact (right) -->
+ <line class="arr-yes" x1="690" y1="90" x2="800" y2="90"/>
+ <text class="branch-yes" x="745" y="78" text-anchor="middle">yes</text>
+
+ <rect x="800" y="58" width="240" height="64" rx="10" fill="#27ae60"/>
+ <text class="term-label" x="920" y="84" text-anchor="middle">Exact</text>
+ <text class="term-sub" x="920" y="106" text-anchor="middle">drop
SortExec</text>
+
+ <!-- Branch no (down to check 2) -->
+ <line class="arr-no" x1="530" y1="130" x2="530" y2="200"/>
+ <text class="branch-no" x="544" y="170" text-anchor="start">no</text>
+
+ <!-- ============ Check 2 ============ -->
+ <rect x="370" y="200" width="320" height="100" rx="10" fill="#fff"
stroke="#5b8def" stroke-width="2"/>
+ <text class="check-label" x="530" y="232" text-anchor="middle">Pushdown
still applies?</text>
+ <text class="check-sub" x="530" y="256"
text-anchor="middle">column_in_file_schema</text>
+ <text class="check-sub" x="530" y="276" text-anchor="middle">OR
reversed_satisfies(req)</text>
+
+ <!-- Branch yes → Inexact (right) -->
+ <line class="arr-yes" x1="690" y1="250" x2="800" y2="250"/>
+ <text class="branch-yes" x="745" y="238" text-anchor="middle">yes</text>
+
+ <rect x="800" y="218" width="240" height="64" rx="10" fill="#e67e22"/>
+ <text class="term-label" x="920" y="244" text-anchor="middle">Inexact</text>
+ <text class="term-sub" x="920" y="266" text-anchor="middle">set
runtime-reorder flags</text>
+
+ <!-- Branch no (down to Unsupported) -->
+ <line class="arr-no" x1="530" y1="300" x2="530" y2="370"/>
+ <text class="branch-no" x="544" y="340" text-anchor="start">no</text>
+
+ <!-- ============ Unsupported terminal ============ -->
+ <rect x="370" y="370" width="320" height="64" rx="10" fill="#95a5a6"/>
+ <text class="term-label" x="530" y="396"
text-anchor="middle">Unsupported</text>
+ <text class="term-sub" x="530" y="418" text-anchor="middle">SortExec stays ·
full external sort</text>
+
+ <!-- ============ Effects legend (right side, bottom) ============ -->
+ <rect x="800" y="320" width="240" height="114" rx="8" fill="#fafafa"
stroke="#e3e3e3"/>
+ <text class="check-label" x="820" y="345">Outcomes</text>
+ <circle cx="826" cy="368" r="6" fill="#27ae60"/>
+ <text class="effect" x="840" y="372">Exact · static limit on source</text>
+ <circle cx="826" cy="392" r="6" fill="#e67e22"/>
+ <text class="effect" x="840" y="396">Inexact · TopK + RG pruner</text>
+ <circle cx="826" cy="416" r="6" fill="#95a5a6"/>
+ <text class="effect" x="840" y="420">Unsupported · no benefit</text>
+
+</svg>
diff --git a/content/images/sort-pushdown/pr21956-runtime-pipeline.svg
b/content/images/sort-pushdown/pr21956-runtime-pipeline.svg
new file mode 100644
index 0000000..1218af1
--- /dev/null
+++ b/content/images/sort-pushdown/pr21956-runtime-pipeline.svg
@@ -0,0 +1,71 @@
+<svg xmlns="http://www.w3.org/2000/svg"; viewBox="0 42 1180 578">
+ <style>
+ text { font-family: 'Segoe UI', Arial, sans-serif; }
+ .title { font-size: 22px; font-weight: 800; fill: #222; }
+ .source-title { font-size: 17px; font-weight: 800; fill: #34495e; }
+ .source-flag-name { font-size: 14px; fill: #1d1d1f; font-family: 'Courier
New', monospace; font-weight: 700; }
+ .source-flag-val { font-size: 13px; fill: #555; font-family: 'Courier
New', monospace; }
+ .step-circle { font-size: 22px; font-weight: 800; }
+ .step-title { font-size: 19px; font-weight: 800; fill: #fff; }
+ .step-scope { font-size: 13px; font-weight: 700; fill: #fff; opacity:
0.85; }
+ .step-body { font-size: 15px; fill: #fff; }
+ .step-code { font-size: 13px; font-family: 'Courier New', monospace; fill:
#fff; opacity: 0.85; }
+ .branch-label { font-size: 14px; fill: #c0392b; font-style: italic;
font-weight: 700; }
+ .footer-title { font-size: 17px; font-weight: 800; fill: #fff; }
+ .footer-sub { font-size: 14px; fill: #fff; opacity: 0.9; font-style:
italic; }
+ .arr { fill: none; stroke: #555; stroke-width: 2; marker-end: url(#arr); }
+ </style>
+ <defs>
+ <marker id="arr" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7"
markerHeight="7" orient="auto">
+ <path d="M0,0 L10,5 L0,10 Z" fill="#555"/>
+ </marker>
+ </defs>
+
+ <!-- ============ Source state ============ -->
+ <rect x="90" y="60" width="1000" height="60" rx="10" fill="#f6f7fa"
stroke="#34495e" stroke-width="2"/>
+ <text class="source-title" x="590" y="88" text-anchor="middle">ParquetSource
carries the inexact-pushdown decision</text>
+ <text class="source-flag-val" x="590" y="108" text-anchor="middle"
font-style="italic" font-family="Segoe UI" fill="#777">set by
try_pushdown_sort, read by the opener below</text>
+
+ <!-- arrow to step 1 -->
+ <line class="arr" x1="590" y1="124" x2="590" y2="190"/>
+
+ <!-- ============ Step 1: File-level reorder ============ -->
+ <rect x="90" y="195" width="1000" height="85" rx="10" fill="#5b8def"/>
+ <circle cx="135" cy="237" r="22" fill="#fff"/>
+ <text class="step-circle" x="135" y="245" text-anchor="middle"
fill="#5b8def">1</text>
+ <text class="step-title" x="180" y="220">File-level reorder</text>
+ <text class="step-scope" x="180" y="240">across the whole scan</text>
+ <text class="step-body" x="180" y="263">Reorder files by <tspan
font-family="Courier New" font-weight="700">min(col)</tspan> so the
most-promising file opens first</text>
+
+ <!-- arrow with label -->
+ <line class="arr" x1="590" y1="284" x2="590" y2="305"/>
+ <text class="branch-label" x="605" y="300" text-anchor="start">for each
opened file</text>
+
+ <!-- ============ Step 2: RG reorder ============ -->
+ <rect x="90" y="310" width="1000" height="85" rx="10" fill="#27ae60"/>
+ <circle cx="135" cy="352" r="22" fill="#fff"/>
+ <text class="step-circle" x="135" y="360" text-anchor="middle"
fill="#27ae60">2</text>
+ <text class="step-title" x="180" y="335">Row-group-level reorder</text>
+ <text class="step-scope" x="180" y="355">per opened file</text>
+ <text class="step-body" x="180" y="378">Sort row groups ASC by <tspan
font-family="Courier New" font-weight="700">min(col)</tspan> from the parquet
column statistics</text>
+
+ <!-- arrow with label -->
+ <line class="arr" x1="590" y1="399" x2="590" y2="420"/>
+ <text class="branch-label" x="605" y="415" text-anchor="start">if
reverse_row_groups</text>
+
+ <!-- ============ Step 3: Reverse iteration ============ -->
+ <rect x="90" y="425" width="1000" height="85" rx="10" fill="#e67e22"/>
+ <circle cx="135" cy="467" r="22" fill="#fff"/>
+ <text class="step-circle" x="135" y="475" text-anchor="middle"
fill="#e67e22">3</text>
+ <text class="step-title" x="180" y="450">Reverse iteration</text>
+ <text class="step-scope" x="180" y="470">DESC requests only</text>
+ <text class="step-body" x="180" y="493">Iterate row-group list in reverse →
<tspan font-family="Courier New" font-weight="700">"row groups DESC × rows
ASC"</tspan></text>
+
+ <!-- arrow to decoder -->
+ <line class="arr" x1="590" y1="514" x2="590" y2="535"/>
+
+ <!-- ============ Final: decoder ============ -->
+ <rect x="90" y="540" width="1000" height="70" rx="10" fill="#34495e"/>
+ <text class="footer-title" x="590" y="570" text-anchor="middle">Decoder
reads row groups in this order</text>
+ <text class="footer-sub" x="590" y="594" text-anchor="middle">approximate
ordering — the Sort / TopK above the source still enforces final ordering</text>
+</svg>
diff --git a/content/images/sort-pushdown/pruner_loop.png
b/content/images/sort-pushdown/pruner_loop.png
new file mode 100644
index 0000000..3852b0c
Binary files /dev/null and b/content/images/sort-pushdown/pruner_loop.png differ
diff --git a/content/images/sort-pushdown/pruning_stack.svg
b/content/images/sort-pushdown/pruning_stack.svg
new file mode 100644
index 0000000..f59decd
--- /dev/null
+++ b/content/images/sort-pushdown/pruning_stack.svg
@@ -0,0 +1,84 @@
+<svg xmlns="http://www.w3.org/2000/svg"; viewBox="0 0 1000 530">
+ <style>
+ text { font-family: 'Segoe UI', Arial, sans-serif; }
+ .title { font-size: 18px; font-weight: 700; fill: #2c3e7a; }
+ .subtitle{ font-size: 12px; font-style: italic; fill: #666; }
+ .hdr { font-size: 13px; font-weight: 700; }
+ .sub { font-size: 11px; font-style: italic; }
+ .code { font-size: 11px; font-family: 'Courier New', monospace; fill:
#333; }
+ .edge { font-size: 11px; font-style: italic; font-weight: 700; }
+ .spine { font-size: 11px; font-style: italic; font-weight: 700; }
+ .arrow { fill: none; stroke: #555; stroke-width: 1.5; marker-end:
url(#arr); }
+ </style>
+ <defs>
+ <marker id="arr" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6"
markerHeight="6" orient="auto">
+ <path d="M0,0 L10,5 L0,10 Z" fill="#555"/>
+ </marker>
+ </defs>
+
+ <text class="title" x="500" y="30" text-anchor="middle">Three-layer
pruning · file + row group + row</text>
+
+ <!-- TopK heap threshold -->
+ <rect x="555" y="66" width="300" height="46" rx="6" fill="#eaf2ff"
stroke="#5b8def" stroke-width="1.5"/>
+ <text class="hdr" x="705" y="87" text-anchor="middle" fill="#1f4fa3">TopK
heap threshold</text>
+ <text class="sub" x="705" y="104" text-anchor="middle" fill="#1f4fa3">pushes
DynamicFilter to all three layers</text>
+ <line class="arrow" x1="705" y1="112" x2="705" y2="149"/>
+
+ <!-- ===== File-level (purple) ===== -->
+ <rect x="470" y="150" width="470" height="58" rx="6" fill="#ede7f6"
stroke="#7e57c2" stroke-width="1.5"/>
+ <text class="hdr" x="705" y="174" text-anchor="middle"
fill="#4527a0">File-level dynamic prune</text>
+ <text class="sub" x="705" y="192" text-anchor="middle"
fill="#4527a0">file_pruner + EarlyStoppingStream</text>
+
+ <!-- branch: dead file -->
+ <text class="edge" x="405" y="172" text-anchor="middle"
fill="#2e7d32">skipped file</text>
+ <line class="arrow" x1="470" y1="179" x2="343" y2="179"/>
+ <rect x="40" y="150" width="300" height="58" rx="6" fill="#e8f5e9"
stroke="#2e7d32" stroke-width="1.5"/>
+ <text class="hdr" x="190" y="171" text-anchor="middle"
fill="#1b5e20">Skipped File ✓ done</text>
+ <text class="code" x="58" y="188">✗ no metadata load</text>
+ <text class="code" x="58" y="202">✗ no row-group decode</text>
+
+ <!-- spine: surviving file -->
+ <text class="spine" x="715" y="234" text-anchor="start"
fill="#4527a0">surviving file</text>
+ <line class="arrow" x1="705" y1="208" x2="705" y2="249"/>
+
+ <!-- ===== Row-group-level (red) ===== -->
+ <rect x="470" y="250" width="470" height="58" rx="6" fill="#fdecea"
stroke="#e05a4f" stroke-width="1.5"/>
+ <text class="hdr" x="705" y="274" text-anchor="middle"
fill="#b03a2e">Row-group prune (#22450)</text>
+ <text class="sub" x="705" y="292" text-anchor="middle" fill="#b03a2e">every
row-group boundary</text>
+
+ <!-- branch: pruned RG -->
+ <text class="edge" x="405" y="272" text-anchor="middle"
fill="#2e7d32">pruned RG</text>
+ <line class="arrow" x1="470" y1="279" x2="343" y2="279"/>
+ <rect x="40" y="250" width="300" height="58" rx="6" fill="#e8f5e9"
stroke="#2e7d32" stroke-width="1.5"/>
+ <text class="hdr" x="190" y="271" text-anchor="middle"
fill="#1b5e20">Pruned RG ✓ done</text>
+ <text class="code" x="58" y="288">✗ no byte fetch</text>
+ <text class="code" x="58" y="302">✗ no decode · ✗ no
RowFilter eval</text>
+
+ <!-- spine: surviving RG -->
+ <text class="spine" x="715" y="334" text-anchor="start"
fill="#b03a2e">surviving RG</text>
+ <line class="arrow" x1="705" y1="308" x2="705" y2="349"/>
+
+ <!-- ===== Row-level (orange) ===== -->
+ <rect x="470" y="350" width="470" height="58" rx="6" fill="#fff3e0"
stroke="#ef9a3d" stroke-width="1.5"/>
+ <text class="hdr" x="705" y="374" text-anchor="middle"
fill="#b9770e">Row-level RowFilter</text>
+ <text class="sub" x="705" y="392" text-anchor="middle"
fill="#b9770e">per-row · decode sort col + eval</text>
+
+ <!-- branch: reject row -->
+ <text class="edge" x="405" y="372" text-anchor="middle"
fill="#2e7d32">reject row</text>
+ <line class="arrow" x1="470" y1="379" x2="343" y2="379"/>
+ <rect x="40" y="350" width="300" height="76" rx="6" fill="#e8f5e9"
stroke="#2e7d32" stroke-width="1.5"/>
+ <text class="hdr" x="190" y="371" text-anchor="middle"
fill="#1b5e20">Reject row ✓ partial savings</text>
+ <text class="code" x="58" y="388">✗ skip other projection cols</text>
+ <text class="code" x="58" y="402">✗ skip their column IO</text>
+ <text class="sub" x="58" y="418" fill="#1b5e20">(sort col was decoded to
evaluate)</text>
+
+ <!-- spine: pass row -->
+ <text class="spine" x="715" y="434" text-anchor="start" fill="#b9770e">pass
row</text>
+ <line class="arrow" x1="705" y1="408" x2="705" y2="449"/>
+
+ <!-- ===== Pass row: full work (gray) ===== -->
+ <rect x="555" y="450" width="300" height="58" rx="6" fill="#eeeeee"
stroke="#9e9e9e" stroke-width="1.5"/>
+ <text class="hdr" x="705" y="471" text-anchor="middle" fill="#555">Pass row
· full work</text>
+ <text class="code" x="575" y="488">✓ decode every projection
col</text>
+ <text class="code" x="575" y="502">✓ materialize batch upward</text>
+</svg>
diff --git a/content/images/sort-pushdown/reverse-scan.svg
b/content/images/sort-pushdown/reverse-scan.svg
new file mode 100644
index 0000000..443a0a1
--- /dev/null
+++ b/content/images/sort-pushdown/reverse-scan.svg
@@ -0,0 +1,100 @@
+<svg xmlns="http://www.w3.org/2000/svg"; viewBox="0 0 820 380">
+ <style>
+ text { font-family: 'Segoe UI', Arial, sans-serif; }
+ .title { font-size: 14px; font-weight: 700; fill: #222; }
+ .panel-title { font-size: 13px; font-weight: 700; }
+ .label { font-size: 11px; fill: #555; }
+ .small { font-size: 10px; fill: #444; }
+ .badge-good { font-size: 12px; font-weight: 700; fill: #27ae60; }
+ .badge-bad { font-size: 12px; font-weight: 700; fill: #c0392b; }
+ .arrow { fill: none; stroke: #c0392b; stroke-width: 1.8; marker-end:
url(#arr3); }
+ .arrow-green { fill: none; stroke: #27ae60; stroke-width: 1.8; marker-end:
url(#arr3g); }
+ </style>
+ <defs>
+ <marker id="arr3" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6"
markerHeight="6" orient="auto">
+ <path d="M0,0 L10,5 L0,10 Z" fill="#c0392b"/>
+ </marker>
+ <marker id="arr3g" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6"
markerHeight="6" orient="auto">
+ <path d="M0,0 L10,5 L0,10 Z" fill="#27ae60"/>
+ </marker>
+ </defs>
+
+ <text class="title" x="410" y="22" text-anchor="middle">ORDER BY ts DESC
LIMIT 10 — row-group reverse vs page reverse</text>
+
+ <!-- row group reverse panel -->
+ <rect x="20" y="40" width="380" height="320" rx="8" fill="#fff8e1"
stroke="#e67e22" stroke-width="1.5"/>
+ <text class="panel-title" x="210" y="62" text-anchor="middle"
fill="#e67e22">Row-group reverse (today, merged)</text>
+
+ <!-- row group block with pages -->
+ <rect x="40" y="90" width="340" height="120" rx="6" fill="#fff"
stroke="#bbb"/>
+ <text class="small" x="50" y="106">RowGroup (last, ~128 MB)</text>
+
+ <!-- pages inside the row group -->
+ <g>
+ <rect x="50" y="115" width="38" height="80" fill="#bdc3c7"/>
+ <rect x="92" y="115" width="38" height="80" fill="#bdc3c7"/>
+ <rect x="134" y="115" width="38" height="80" fill="#bdc3c7"/>
+ <rect x="176" y="115" width="38" height="80" fill="#bdc3c7"/>
+ <rect x="218" y="115" width="38" height="80" fill="#bdc3c7"/>
+ <rect x="260" y="115" width="38" height="80" fill="#bdc3c7"/>
+ <rect x="302" y="115" width="38" height="80" fill="#bdc3c7"/>
+ <rect x="344" y="115" width="36" height="80" fill="#bdc3c7"/>
+ <text class="small" x="69" y="160" text-anchor="middle">P0</text>
+ <text class="small" x="111" y="160" text-anchor="middle">P1</text>
+ <text class="small" x="153" y="160" text-anchor="middle">P2</text>
+ <text class="small" x="195" y="160" text-anchor="middle">P3</text>
+ <text class="small" x="237" y="160" text-anchor="middle">P4</text>
+ <text class="small" x="279" y="160" text-anchor="middle">P5</text>
+ <text class="small" x="321" y="160" text-anchor="middle">P6</text>
+ <text class="small" x="362" y="160" text-anchor="middle">P7</text>
+ </g>
+
+ <text class="label" x="210" y="235" text-anchor="middle">Decode the entire
row group, reverse in memory, take 10.</text>
+ <text class="badge-bad" x="210" y="265" text-anchor="middle">Peak buffer:
~128 MB</text>
+ <text class="badge-bad" x="210" y="285" text-anchor="middle">Pages decoded:
8</text>
+ <text class="badge-bad" x="210" y="305"
text-anchor="middle">Time-to-first-N: ~29 µs</text>
+
+ <!-- arrows: all pages read -->
+ <line class="arrow" x1="362" y1="80" x2="362" y2="115"/>
+ <line class="arrow" x1="69" y1="80" x2="69" y2="115"/>
+ <line class="arrow" x1="111" y1="80" x2="111" y2="115"/>
+ <line class="arrow" x1="153" y1="80" x2="153" y2="115"/>
+ <line class="arrow" x1="195" y1="80" x2="195" y2="115"/>
+ <line class="arrow" x1="237" y1="80" x2="237" y2="115"/>
+ <line class="arrow" x1="279" y1="80" x2="279" y2="115"/>
+ <line class="arrow" x1="321" y1="80" x2="321" y2="115"/>
+
+ <!-- PAGE REVERSE panel -->
+ <rect x="420" y="40" width="380" height="320" rx="8" fill="#e8f5e9"
stroke="#27ae60" stroke-width="1.5"/>
+ <text class="panel-title" x="610" y="62" text-anchor="middle"
fill="#27ae60">Page reverse (upstream POC, arrow-rs #9937)</text>
+
+ <rect x="440" y="90" width="340" height="120" rx="6" fill="#fff"
stroke="#bbb"/>
+ <text class="small" x="450" y="106">RowGroup (last)</text>
+
+ <g>
+ <rect x="450" y="115" width="38" height="80" fill="#bdc3c7"/>
+ <rect x="492" y="115" width="38" height="80" fill="#bdc3c7"/>
+ <rect x="534" y="115" width="38" height="80" fill="#bdc3c7"/>
+ <rect x="576" y="115" width="38" height="80" fill="#bdc3c7"/>
+ <rect x="618" y="115" width="38" height="80" fill="#bdc3c7"/>
+ <rect x="660" y="115" width="38" height="80" fill="#bdc3c7"/>
+ <rect x="702" y="115" width="38" height="80" fill="#bdc3c7"/>
+ <rect x="744" y="115" width="36" height="80" fill="#27ae60"/>
+ <text class="small" x="469" y="160" text-anchor="middle">P0</text>
+ <text class="small" x="511" y="160" text-anchor="middle">P1</text>
+ <text class="small" x="553" y="160" text-anchor="middle">P2</text>
+ <text class="small" x="595" y="160" text-anchor="middle">P3</text>
+ <text class="small" x="637" y="160" text-anchor="middle">P4</text>
+ <text class="small" x="679" y="160" text-anchor="middle">P5</text>
+ <text class="small" x="721" y="160" text-anchor="middle">P6</text>
+ <text class="small" x="762" y="160" text-anchor="middle"
fill="#fff">P7</text>
+ </g>
+
+ <!-- only one green arrow on last page -->
+ <line class="arrow-green" x1="762" y1="80" x2="762" y2="115"/>
+
+ <text class="label" x="610" y="235" text-anchor="middle">Seek to last page
only via OffsetIndex, decode, reverse, return.</text>
+ <text class="badge-good" x="610" y="265" text-anchor="middle">Peak buffer:
~1 MB</text>
+ <text class="badge-good" x="610" y="285" text-anchor="middle">Pages decoded:
1</text>
+ <text class="badge-good" x="610" y="305"
text-anchor="middle">Time-to-first-N: ~565 ns (≈ 50× faster)</text>
+</svg>
diff --git a/content/images/sort-pushdown/rg_cascade.svg
b/content/images/sort-pushdown/rg_cascade.svg
new file mode 100644
index 0000000..dc13616
--- /dev/null
+++ b/content/images/sort-pushdown/rg_cascade.svg
@@ -0,0 +1,113 @@
+<svg xmlns="http://www.w3.org/2000/svg"; viewBox="0 0 1020 565">
+ <style>
+ text { font-family: 'Segoe UI', Arial, sans-serif; }
+ .title { font-size: 20px; font-weight: 700; fill: #2c3e7a; }
+ .subtitle { font-size: 13px; fill: #666; font-family: 'Courier New',
monospace; }
+ .callout { font-size: 13px; font-weight: 700; }
+ .axis { stroke: #999; stroke-width: 1.4; }
+ .axis-lbl { font-size: 13px; font-style: italic; fill: #666; }
+ .thresh { stroke: #c0392b; stroke-width: 2; }
+ .thresh-lbl{ font-size: 12px; font-weight: 700; fill: #c0392b; }
+ .rg { font-size: 11px; font-weight: 700; paint-order: stroke;
stroke: #fff; stroke-width: 3px; stroke-linejoin: round; }
+ .range { font-size: 11px; font-family: 'Courier New', monospace; fill:
#777; }
+ .x-mark { stroke: #c0392b; stroke-width: 2.4; opacity: 0.7; }
+ .lead { fill: none; stroke-width: 1.8; }
+ .sum-bold { font-size: 14px; font-weight: 700; fill: #2c3e7a; }
+ .sum-sub { font-size: 12px; font-style: italic; fill: #666; }
+ </style>
+ <defs>
+ <marker id="ar-o" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6"
markerHeight="6" orient="auto"><path d="M0,0 L10,5 L0,10 Z"
fill="#e67e22"/></marker>
+ <marker id="ar-r" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6"
markerHeight="6" orient="auto"><path d="M0,0 L10,5 L0,10 Z"
fill="#c0392b"/></marker>
+ <marker id="ar-g" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6"
markerHeight="6" orient="auto"><path d="M0,0 L10,5 L0,10 Z"
fill="#27ae60"/></marker>
+ </defs>
+
+ <text class="title" x="510" y="30" text-anchor="middle">Cascading prune
· how the heap prunes row groups</text>
+ <text class="subtitle" x="510" y="50" text-anchor="middle">ORDER BY x DESC
LIMIT 10 · one parquet file, 10 row groups</text>
+
+ <!-- ===== callouts ===== -->
+ <rect x="70" y="64" width="250" height="34" rx="6" fill="#fff3e0"
stroke="#e67e22" stroke-width="1.5"/>
+ <text class="callout" x="195" y="86" text-anchor="middle"
fill="#b9770e">① heap fills</text>
+ <rect x="350" y="64" width="290" height="34" rx="6" fill="#fdecea"
stroke="#c0392b" stroke-width="1.5"/>
+ <text class="callout" x="495" y="86" text-anchor="middle"
fill="#c0392b">② threshold climbs</text>
+ <rect x="670" y="64" width="320" height="34" rx="6" fill="#e8f5e9"
stroke="#27ae60" stroke-width="1.5"/>
+ <text class="callout" x="830" y="86" text-anchor="middle"
fill="#1b5e20">③ cascading prune at next row-group boundary</text>
+
+ <!-- leader lines -->
+ <path class="lead" stroke="#e67e22" marker-end="url(#ar-o)" d="M180,98
L135,146"/>
+ <path class="lead" stroke="#c0392b" marker-end="url(#ar-r)" d="M495,98
L610,192"/>
+ <path class="lead" stroke="#27ae60" marker-end="url(#ar-g)" d="M810,98
L600,301"/>
+
+ <!-- ===== axes ===== -->
+ <line class="axis" x1="70" y1="110" x2="70" y2="490"/>
+ <line class="axis" x1="70" y1="490" x2="1000" y2="490"/>
+ <text class="axis-lbl" x="30" y="300" text-anchor="middle"
transform="rotate(-90 30 300)">value range ↑</text>
+ <text class="axis-lbl" x="520" y="512" text-anchor="middle">scan time
→</text>
+
+ <!-- ===== threshold line ===== -->
+ <line class="thresh" x1="70" y1="196" x2="938" y2="196"/>
+ <text class="thresh-lbl" x="944" y="200">threshold</text>
+
+ <!-- ===== row groups: RG 9 (kept) then RG 8..0 (pruned) ===== -->
+ <!-- RG 9 kept -->
+ <rect x="90" y="150" width="80" height="42" rx="4" fill="#e8f5e9"
stroke="#27ae60" stroke-width="1.8"/>
+ <text class="rg" x="130" y="176" text-anchor="middle" fill="#1b5e20">Row
group 9</text>
+ <text class="range" x="130" y="207" text-anchor="middle">[90..100)</text>
+
+ <!-- RG 8 -->
+ <rect x="182" y="181" width="80" height="42" rx="4" fill="#f4f4f4"
stroke="#dddddd" stroke-width="1.5"/>
+ <line class="x-mark" x1="186" y1="185" x2="258" y2="219"/><line
class="x-mark" x1="258" y1="185" x2="186" y2="219"/>
+ <text class="rg" x="222" y="207" text-anchor="middle" fill="#666">Row group
8</text>
+ <text class="range" x="222" y="238" text-anchor="middle">[80..90)</text>
+
+ <!-- RG 7 -->
+ <rect x="274" y="212" width="80" height="42" rx="4" fill="#f4f4f4"
stroke="#dddddd" stroke-width="1.5"/>
+ <line class="x-mark" x1="278" y1="216" x2="350" y2="250"/><line
class="x-mark" x1="350" y1="216" x2="278" y2="250"/>
+ <text class="rg" x="314" y="238" text-anchor="middle" fill="#666">Row group
7</text>
+ <text class="range" x="314" y="269" text-anchor="middle">[70..80)</text>
+
+ <!-- RG 6 -->
+ <rect x="366" y="243" width="80" height="42" rx="4" fill="#f4f4f4"
stroke="#dddddd" stroke-width="1.5"/>
+ <line class="x-mark" x1="370" y1="247" x2="442" y2="281"/><line
class="x-mark" x1="442" y1="247" x2="370" y2="281"/>
+ <text class="rg" x="406" y="269" text-anchor="middle" fill="#666">Row group
6</text>
+ <text class="range" x="406" y="300" text-anchor="middle">[60..70)</text>
+
+ <!-- RG 5 -->
+ <rect x="458" y="274" width="80" height="42" rx="4" fill="#f4f4f4"
stroke="#dddddd" stroke-width="1.5"/>
+ <line class="x-mark" x1="462" y1="278" x2="534" y2="312"/><line
class="x-mark" x1="534" y1="278" x2="462" y2="312"/>
+ <text class="rg" x="498" y="300" text-anchor="middle" fill="#666">Row group
5</text>
+ <text class="range" x="498" y="331" text-anchor="middle">[50..60)</text>
+
+ <!-- RG 4 -->
+ <rect x="550" y="305" width="80" height="42" rx="4" fill="#f4f4f4"
stroke="#dddddd" stroke-width="1.5"/>
+ <line class="x-mark" x1="554" y1="309" x2="626" y2="343"/><line
class="x-mark" x1="626" y1="309" x2="554" y2="343"/>
+ <text class="rg" x="590" y="331" text-anchor="middle" fill="#666">Row group
4</text>
+ <text class="range" x="590" y="362" text-anchor="middle">[40..50)</text>
+
+ <!-- RG 3 -->
+ <rect x="642" y="336" width="80" height="42" rx="4" fill="#f4f4f4"
stroke="#dddddd" stroke-width="1.5"/>
+ <line class="x-mark" x1="646" y1="340" x2="718" y2="374"/><line
class="x-mark" x1="718" y1="340" x2="646" y2="374"/>
+ <text class="rg" x="682" y="362" text-anchor="middle" fill="#666">Row group
3</text>
+ <text class="range" x="682" y="393" text-anchor="middle">[30..40)</text>
+
+ <!-- RG 2 -->
+ <rect x="734" y="367" width="80" height="42" rx="4" fill="#f4f4f4"
stroke="#dddddd" stroke-width="1.5"/>
+ <line class="x-mark" x1="738" y1="371" x2="810" y2="405"/><line
class="x-mark" x1="810" y1="371" x2="738" y2="405"/>
+ <text class="rg" x="774" y="393" text-anchor="middle" fill="#666">Row group
2</text>
+ <text class="range" x="774" y="424" text-anchor="middle">[20..30)</text>
+
+ <!-- RG 1 -->
+ <rect x="826" y="398" width="80" height="42" rx="4" fill="#f4f4f4"
stroke="#dddddd" stroke-width="1.5"/>
+ <line class="x-mark" x1="830" y1="402" x2="902" y2="436"/><line
class="x-mark" x1="902" y1="402" x2="830" y2="436"/>
+ <text class="rg" x="866" y="424" text-anchor="middle" fill="#666">Row group
1</text>
+ <text class="range" x="866" y="455" text-anchor="middle">[10..20)</text>
+
+ <!-- RG 0 -->
+ <rect x="918" y="429" width="80" height="42" rx="4" fill="#f4f4f4"
stroke="#dddddd" stroke-width="1.5"/>
+ <line class="x-mark" x1="922" y1="433" x2="994" y2="467"/><line
class="x-mark" x1="994" y1="433" x2="922" y2="467"/>
+ <text class="rg" x="958" y="455" text-anchor="middle" fill="#666">Row group
0</text>
+ <text class="range" x="958" y="486" text-anchor="middle">[0..10)</text>
+
+ <!-- ===== summary ===== -->
+ <text class="sum-bold" x="510" y="536" text-anchor="middle">Row group 9
alone fills the heap of size 10, so the threshold rises into its range (≥
90).</text>
+ <text class="sum-sub" x="510" y="556" text-anchor="middle">At the next
row-group boundary, every remaining row group with max < 90 is pruned in one
pass — bytes never fetched.</text>
+</svg>
diff --git a/content/images/sort-pushdown/topk_tpch_bench.svg
b/content/images/sort-pushdown/topk_tpch_bench.svg
new file mode 100644
index 0000000..da65fd1
--- /dev/null
+++ b/content/images/sort-pushdown/topk_tpch_bench.svg
@@ -0,0 +1,121 @@
+<svg xmlns="http://www.w3.org/2000/svg"; viewBox="0 0 1000 480">
+ <style>
+ text { font-family: 'Segoe UI', Arial, sans-serif; }
+ .title { font-size: 15px; font-weight: 700; fill: #222; }
+ .axis { stroke: #888; stroke-width: 1; }
+ .grid { stroke: #eee; stroke-width: 1; }
+ .tick-lbl { font-size: 11px; fill: #555; }
+ .axis-lbl { font-size: 12px; fill: #555; }
+ .qlabel { font-size: 12px; font-weight: 600; fill: #333; }
+ .val { font-size: 9px; fill: #888; }
+ .speedup { font-size: 13px; font-weight: 700; fill: #27ae60; }
+ .legend { font-size: 12px; fill: #444; }
+ .summary { font-size: 12px; fill: #444; }
+ </style>
+
+ <text class="title" x="500" y="24" text-anchor="middle">topk_tpch (TPC-H
SF1, ORDER BY … LIMIT 100) — main vs runtime row-group pruning</text>
+
+ <!-- gridlines + y ticks (0..70 ms) -->
+ <line class="grid" x1="70" y1="360" x2="970" y2="360"/>
+ <line class="grid" x1="70" y1="310" x2="970" y2="310"/>
+ <line class="grid" x1="70" y1="260" x2="970" y2="260"/>
+ <line class="grid" x1="70" y1="210" x2="970" y2="210"/>
+ <line class="grid" x1="70" y1="160" x2="970" y2="160"/>
+ <line class="grid" x1="70" y1="110" x2="970" y2="110"/>
+ <line class="grid" x1="70" y1="60" x2="970" y2="60"/>
+ <line class="axis" x1="70" y1="60" x2="70" y2="410"/>
+ <line class="axis" x1="70" y1="410" x2="970" y2="410"/>
+ <text class="tick-lbl" x="60" y="414" text-anchor="end">0</text>
+ <text class="tick-lbl" x="60" y="364" text-anchor="end">10</text>
+ <text class="tick-lbl" x="60" y="314" text-anchor="end">20</text>
+ <text class="tick-lbl" x="60" y="264" text-anchor="end">30</text>
+ <text class="tick-lbl" x="60" y="214" text-anchor="end">40</text>
+ <text class="tick-lbl" x="60" y="164" text-anchor="end">50</text>
+ <text class="tick-lbl" x="60" y="114" text-anchor="end">60</text>
+ <text class="tick-lbl" x="60" y="64" text-anchor="end">70</text>
+ <text class="axis-lbl" x="24" y="235" text-anchor="middle"
transform="rotate(-90 24 235)">Median time (ms, lower is better)</text>
+
+ <!-- legend -->
+ <rect x="120" y="72" width="14" height="14" fill="#95a5a6"/><text
class="legend" x="140" y="83">main</text>
+ <rect x="230" y="72" width="14" height="14" fill="#d5d9dc"/><text
class="legend" x="250" y="83">this PR</text>
+ <rect x="360" y="72" width="14" height="14" fill="#27ae60"/><text
class="legend" x="380" y="83">this PR — ≥2× faster</text>
+
+ <!-- ===== Q1 (2.6 / 2.5) ===== -->
+ <rect x="81" y="397" width="28" height="13" fill="#95a5a6"/>
+ <rect x="113" y="398" width="28" height="12" fill="#d5d9dc"/>
+ <text class="val" x="95" y="393" text-anchor="middle">2.6</text>
+ <text class="qlabel" x="111" y="428" text-anchor="middle">Q1</text>
+
+ <!-- ===== Q2 (11.3 / 3.59) 3.15x ===== -->
+ <rect x="163" y="354" width="28" height="56" fill="#95a5a6"/>
+ <rect x="195" y="392" width="28" height="18" fill="#27ae60"/>
+ <text class="val" x="177" y="350" text-anchor="middle">11.3</text>
+ <rect x="165" y="312" width="56" height="24" rx="5" fill="#fff"
stroke="#27ae60" stroke-width="1.5"/>
+ <text class="speedup" x="193" y="329" text-anchor="middle">3.15×</text>
+ <text class="qlabel" x="193" y="428" text-anchor="middle">Q2</text>
+
+ <!-- ===== Q3 (32.1 / 31.9) ===== -->
+ <rect x="245" y="250" width="28" height="160" fill="#95a5a6"/>
+ <rect x="277" y="251" width="28" height="159" fill="#d5d9dc"/>
+ <text class="val" x="259" y="246" text-anchor="middle">32.1</text>
+ <text class="qlabel" x="275" y="428" text-anchor="middle">Q3</text>
+
+ <!-- ===== Q4 (12.2 / 3.23) 3.78x ===== -->
+ <rect x="326" y="349" width="28" height="61" fill="#95a5a6"/>
+ <rect x="358" y="394" width="28" height="16" fill="#27ae60"/>
+ <text class="val" x="340" y="345" text-anchor="middle">12.2</text>
+ <rect x="328" y="307" width="56" height="24" rx="5" fill="#fff"
stroke="#27ae60" stroke-width="1.5"/>
+ <text class="speedup" x="356" y="324" text-anchor="middle">3.78×</text>
+ <text class="qlabel" x="356" y="428" text-anchor="middle">Q4</text>
+
+ <!-- ===== Q5 (10.1 / 9.9) ===== -->
+ <rect x="408" y="360" width="28" height="50" fill="#95a5a6"/>
+ <rect x="440" y="361" width="28" height="49" fill="#d5d9dc"/>
+ <text class="val" x="422" y="356" text-anchor="middle">10.1</text>
+ <text class="qlabel" x="438" y="428" text-anchor="middle">Q5</text>
+
+ <!-- ===== Q6 (17.9 / 17.7) ===== -->
+ <rect x="490" y="321" width="28" height="89" fill="#95a5a6"/>
+ <rect x="522" y="322" width="28" height="88" fill="#d5d9dc"/>
+ <text class="val" x="504" y="317" text-anchor="middle">17.9</text>
+ <text class="qlabel" x="520" y="428" text-anchor="middle">Q6</text>
+
+ <!-- ===== Q7 (38.1 / 37.8) ===== -->
+ <rect x="572" y="220" width="28" height="190" fill="#95a5a6"/>
+ <rect x="604" y="221" width="28" height="189" fill="#d5d9dc"/>
+ <text class="val" x="586" y="216" text-anchor="middle">38.1</text>
+ <text class="qlabel" x="602" y="428" text-anchor="middle">Q7</text>
+
+ <!-- ===== Q8 (28.6 / 7.17) 3.99x ===== -->
+ <rect x="654" y="267" width="28" height="143" fill="#95a5a6"/>
+ <rect x="686" y="374" width="28" height="36" fill="#27ae60"/>
+ <text class="val" x="668" y="263" text-anchor="middle">28.6</text>
+ <rect x="656" y="225" width="56" height="24" rx="5" fill="#fff"
stroke="#27ae60" stroke-width="1.5"/>
+ <text class="speedup" x="684" y="242" text-anchor="middle">3.99×</text>
+ <text class="qlabel" x="684" y="428" text-anchor="middle">Q8</text>
+
+ <!-- ===== Q9 (36.9 / 8.50) 4.34x ===== -->
+ <rect x="735" y="226" width="28" height="184" fill="#95a5a6"/>
+ <rect x="767" y="368" width="28" height="42" fill="#27ae60"/>
+ <text class="val" x="749" y="222" text-anchor="middle">36.9</text>
+ <rect x="737" y="184" width="56" height="24" rx="5" fill="#fff"
stroke="#27ae60" stroke-width="1.5"/>
+ <text class="speedup" x="765" y="201" text-anchor="middle">4.34×</text>
+ <text class="qlabel" x="765" y="428" text-anchor="middle">Q9</text>
+
+ <!-- ===== Q10 (55.3 / 13.01) 4.25x ===== -->
+ <rect x="817" y="134" width="28" height="276" fill="#95a5a6"/>
+ <rect x="849" y="345" width="28" height="65" fill="#27ae60"/>
+ <text class="val" x="831" y="130" text-anchor="middle">55.3</text>
+ <rect x="819" y="92" width="56" height="24" rx="5" fill="#fff"
stroke="#27ae60" stroke-width="1.5"/>
+ <text class="speedup" x="847" y="109" text-anchor="middle">4.25×</text>
+ <text class="qlabel" x="847" y="428" text-anchor="middle">Q10</text>
+
+ <!-- ===== Q11 (3.7 / 3.6) ===== -->
+ <rect x="899" y="392" width="28" height="18" fill="#95a5a6"/>
+ <rect x="931" y="392" width="28" height="18" fill="#d5d9dc"/>
+ <text class="val" x="913" y="388" text-anchor="middle">3.7</text>
+ <text class="qlabel" x="929" y="428" text-anchor="middle">Q11</text>
+
+ <!-- summary -->
+ <text class="summary" x="500" y="458" text-anchor="middle">5 of 11 queries
3–4× faster · 0 regressions · total 248.8 → 139.1 ms (−44%)</text>
+</svg>
diff --git a/content/images/sort-pushdown/transition_anatomy.png
b/content/images/sort-pushdown/transition_anatomy.png
new file mode 100644
index 0000000..37a38b5
Binary files /dev/null and
b/content/images/sort-pushdown/transition_anatomy.png differ
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
