xudong963 commented on code in PR #803:
URL: https://github.com/apache/arrow-site/pull/803#discussion_r4266009340
##########
_posts/2026-10-12-analytic-file-formats-sans-io.md:
##########
@@ -0,0 +1,389 @@
+---
+layout: post
+title: "Implementing Analytic File Formats Sans I/O"
+description: "Why separating Parquet decoding from storage I/O makes analytic
file readers more composable, and how arrow-rs puts the design into practice"
+date: "2026-10-12 00:00:00"
+author: "Andrew Lamb, Xudong Wang, Neil Conway, and Daniël Heres"
+categories: [parquet, arrow]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+Opening a file, constructing a reader, and calling `next()` is a wonderfully
+simple interface. The reader handles fetching, buffering, and decoding, and
+many applications need nothing more. The boundary becomes harder to work with
+when the application already has a storage runtime, buffer manager, cache, or
+scheduler of its own.
+
+Consider a query engine scanning several Parquet files under a shared memory
+budget. It wants to serve cached ranges immediately, prioritize reads for one
+query, and overlap decoding with remote I/O. To coordinate that work, it needs
+to know both **what bytes the decoder needs** and **when decoding can
proceed**.
+A storage callback exposes some of this information, but the rest may remain
+inside the reader's control loop.
+
+A **Sans I/O** decoder makes that boundary explicit. It interprets bytes but
+never fetches them. When it cannot continue, it returns the byte ranges it
+needs; the application supplies buffers and decides when to resume decoding.
+Familiar blocking and asynchronous readers can be adapters around this core.
+
+> **The core idea:** the decoder determines its byte dependencies; the
+> application schedules the I/O that satisfies them.
+
+Sans I/O is an established pattern in [network protocol libraries][sans-io].
+This post explores its application to analytic file formats, using the
+[Apache Arrow Rust Parquet reader][arrow-rs-parquet]. The benefit is a reusable
+decoder that can fit different execution and storage policies. Performance
+still depends on the policy, workload, and implementation.
+
+## The Sans I/O contract
+
+The decoder performs no I/O, including through callbacks. Each time the
+application asks it to advance, it does one of three things:
+
+1. Produces decoded data.
+2. Reports that decoding is finished.
+3. Returns the byte ranges needed to make progress.
+
+For the third outcome, the application obtains the bytes from storage or a
+cache, supplies them to the decoder, and tries again. Errors propagate through
+the API as usual. In simplified Rust, with application-provided `fetch` and
+`consume` helpers, the interaction looks like this:
+
+```rust
+loop {
+ match decoder.try_decode()? {
+ DecodeResult::NeedsData(ranges) => {
+ // Fetch or reuse one buffer for each logical range, in order.
+ let buffers = io_policy.fetch(&ranges)?;
+ decoder.push_ranges(ranges, buffers)?;
+ }
+ DecodeResult::Data(batch) => consume(batch)?,
+ DecodeResult::Finished => break,
+ }
+}
+```
+
+The logical ranges describe the decoder's requirements, not necessarily the
+physical reads. A policy can merge nearby ranges into one storage request and
+return slices of the resulting buffer. It can also retain extra bytes in a
+cache for later requests. The decoder does not need to know which happened.
+
+<figure class="centered">
+ <img src="{{ site.baseurl }}/img/sans-io-analytic-formats/architecture.png"
+ alt="An integrated reader invokes a storage callback from its decoding
loop; a Sans I/O decoder returns byte requirements for the application to
schedule and supply"
+ class="img-fluid">
+ <figcaption>Figure 1: Moving byte requests out of the decoder's control loop
gives the application an explicit boundary for scheduling I/O and
decoding.</figcaption>
+</figure>
+
+This is a push-style storage API: the application asks the decoder to advance,
+then pushes bytes into it. "Push" describes the boundary between storage and
+decoding, not the execution model of the rest of the query engine.
+
+## A Parquet reader is a small query engine
+
+Streaming CSV and JSON parsers already use a similar pattern: accept a byte
+chunk, emit records or tokens, and ask for more input. Their next input is
+usually the next chunk in the stream. [Apache Parquet] requires a more complex
+conversation because its reads are non-sequential and depend on earlier data.
+
+<figure class="centered">
+ <img src="{{ site.baseurl
}}/img/sans-io-analytic-formats/parquet-state-machine.png"
+ alt="Parquet decoding alternates between I/O and CPU stages for the
file footer, metadata, optional pruning structures, and data pages"
+ width="520" class="img-fluid">
+ <figcaption>Figure 2: A simplified Parquet decoding state machine. Pink
states perform I/O, yellow states perform CPU work, and dashed states are
optional.</figcaption>
+</figure>
+
+The file's trailing footer tells the reader how much metadata to fetch. That
+metadata locates row groups and column chunks. Optional Bloom filters and page
+indexes can help eliminate row groups or pages before reading their data.
+Evaluating a predicate on one column can further reduce which pages are needed
+from another. The next byte requirements therefore depend on previously
+decoded information, as well as the query's projection and filters.
+
+A reusable decoder must preserve these dependencies while letting the host
+choose how to satisfy them. Query planning still supplies choices such as
+columns, row selections, and predicates; the decoder handles the format logic
+needed to execute that selection. Separating I/O does not require giving up
+filter pushdown or moving Parquet parsing into the application.
+
+## Why not just use a storage callback or async reader?
+
+A replaceable storage interface is already powerful. A `read_at(offset,
+length)` callback can serve cache hits, retry failures, or race reads against
+multiple replicas. An async implementation can queue requests in a shared
+scheduler and return futures for their results. These optimizations do not
+inherently require Sans I/O; the [Arrow Rust design discussion][async-callback]
+explicitly considered such an arrangement.
+
+The distinction is which decisions the public API exposes. A storage callback
+sees requests when the reader issues them. It does not, by itself, expose a
+separate operation to advance format processing, obtain ready-to-decode work,
+or discover requirements beyond the reader's current suspension point. A
+batched or asynchronous callback can offer substantial scheduling control, but
+that control still depends on the reader's buffering and execution contract.
+
+Async changes how execution waits. Sans I/O makes the decoder's byte
+dependencies and resumption points explicit, independently of any particular
+I/O interface. The host can keep pending requirements as ordinary data and
+advance each decoder when its scheduler is ready. Reproducing this boundary
+with callbacks is possible, but can require coordinating channels, futures,
+and polling around the reader's control loop.
+
+If a reader's existing callbacks and buffering policy fit the application,
+there is no need to replace them. Sans I/O is useful when the application needs
+the lower-level boundary as a supported library API.
+
+## Where the boundary helps in practice
+
+Two discussions that motivated the Arrow Rust work show why that boundary
+matters.
+
+In [the original design issue][io-motivation], Andrew described
+buffering entire files in memory or on local SSD because the reader did not
+expose enough information to coordinate more selective prefetching. The
+alternative was a series of smaller reads interleaved with decoding, a poor
+fit for high-latency remote storage. Whole-file buffering was a workable
+integration strategy, but it consumed resources and gave up some of the
+benefits of reading only selected columns and pages.
+
+A [separate discussion about incremental decoding][incremental-decoding]
+described the memory cost of buffering the pages for all selected columns in a
+row group. A later [production report][wide-column-report] highlighted the
+problem for columns containing very large values. A storage callback can
+change how the bytes arrive, but it cannot change a decoder's requirement to
+buffer that set of pages before making progress. This case also illustrates a
+limit of the abstraction: exposing byte requests alone does not make a decoder
+incremental at every page boundary.
+
+### Overlapping decoding and reads
+
+Arrow Rust provides a concrete example of a richer boundary through
+[`try_next_reader`]. Once the required data is available, this method returns a
+`ParquetRecordBatchReader` over buffered data. The application can then:
+
+1. Send that reader to a CPU worker to produce batches.
+2. Advance the push decoder to discover the next set of byte requirements.
+3. Schedule those reads while the worker decodes the preceding data.
+4. Limit the number of active readers and pending reads according to its
+ memory and CPU budgets.
+
+This gives the scheduler separate handles for batch decoding and further I/O.
+An ordinary storage callback cannot create that handoff on its own: the reader
+also needs to expose a suitable unit of decoding work. Our [example
+driver][prototype driver] demonstrates the handoff; a production scheduler
+would also bound the worker queue and preserve batch order where required.
+
+A decoder integrated into a database can implement the same overlap. The
+reason to expose it in a shared library is to let other systems use that
+format implementation with their own scheduling policy.
+
+## Putting the design into arrow-rs
+
+The APIs described here are available in `arrow-rs` `57.1.0`.
+[`ParquetPushDecoder`] preserves decoding state between calls to [`try_decode`]
+and returns `NeedsData(Vec<Range<u64>>)` when bytes are missing.
[`push_ranges`]
+accepts reference-counted `Bytes` buffers, allowing the caller to share
+resident input without copying it at the API boundary. Decompression and
+construction of Arrow arrays can still require allocations.
+
+The earlier loop starts after file metadata is available. Footer and metadata
+decoding use the same pattern through [`ParquetMetaDataPushDecoder`], so the
+application can control fetching from the first footer read onward.
+
+The existing asynchronous [`ParquetRecordBatchStream`] was [rewritten as an
+adapter][pr-8159] around the push decoder. It fetches `NeedsData` requests
+through `AsyncFileReader`, supplies the buffers, and exposes the familiar
+stream API. This consolidates decoding control flow while retaining a
+convenient entry point for applications that do not need custom orchestration.
+
+## What the decoder can tell a prefetcher
Review Comment:
Agree that a performance story needs a concrete demonstration. For this
API-focused post, I added a small worked example showing exactly how two
logical ranges become one physical read and two shared buffer slices.
For the follow-up, I propose building on DataFusion #24086: compare
demand-only reads with budgeted read-ahead on the same Parquet scan, then show
the I/O/decode timeline, time to first batch, total scan time, requests, bytes
fetched, and peak input buffering. We should include real object storage and a
local-disk baseline, and distinguish batch decoding from page release so the
results can be attributed to the right changes. That experiment has not been
run for this revision; the post now explicitly leaves performance evaluation to
the follow-up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
