sunchao commented on PR #11165: URL: https://github.com/apache/arrow-rs/pull/11165#issuecomment-5784906798
Thanks @adriangb, @alamb and @viirya for the review! It seems there are two independent API-design questions in this discussion, and separating them would help us settle the contract. ### 1. Should the returned ranges depend on already-available data? Should this API describe: - The ranges the scan may need, regardless of whether the decoder or an external cache already has those bytes; or - The additional ranges the decoder is expected to request, after accounting for its current buffers? ### 2. What information should accompany each range? Should the API return only byte ranges, or annotate them with information such as the row group, column, page kind, and rows covered? These two decisions are independent: we could return either plain or annotated ranges, with or without filtering based on buffer coverage. ## Proposed direction: a buffer-independent, annotated scan plan ### Keep planning separate from data availability For this planning API, I lean toward returning ranges independently of the decoder’s buffers and any external cache. For example: ``` The scan may need: A, B, C The decoder already has: A The external cache has: B The caller must fetch: C ``` The planner would return `A, B, C`. The caller would then decide how to satisfy those requirements: reuse decoder buffers, retrieve cached bytes, or issue storage requests. This keeps Parquet responsible for determining what the scan may need, while leaving availability, prefetching, request coalescing, and memory-budget decisions to the layers that own them. It also allows the same planning API to support different cache and I/O implementations without needing to understand each one. The caller still needs to track or query buffer/cache coverage; this design moves that responsibility rather than eliminating it. But I think that is a reasonable separation for an API intended to support caller-controlled I/O. ### Return useful annotations with the ranges I also support adding annotations that help callers interpret and schedule the ranges. For example, knowing that a range contains a dictionary page or serves a particular row span is more useful than knowing its offsets alone. We could return an annotated entry containing the byte range together with useful information about the column, row group, and page. Callers that only need byte offsets could simply extract the range: ```rust let ranges = decoder.scan_plan().map(|entry| entry.range); ``` In addition, we can potentially hide the detailed annotations behind an optional flag. This could be useful if generating those annotations requires additional work, allowing callers that only need byte ranges to avoid that cost. The choice would depend on how much work the annotations require. If the planner already has the information, a single annotated entry type may be sufficient. If some details require extra computation or metadata reads, making those details opt-in would give callers more control. ### Make the limits explicit The annotations should describe only what the planner can actually know. Without the necessary page index, we can return a column-chunk range rather than invent page boundaries or row spans. Likewise, when later reads depend on predicate results, the plan should identify those ranges as conditional rather than promise that every listed range will be read. Actual `NeedsData` requests would remain authoritative; the plan would support speculation, not require callers to fetch everything it lists. In short, I would favor: > A buffer-independent plan of potentially needed byte ranges, with useful annotations, while leaving cache lookup and fetch policy to the caller. This seems to preserve the useful distinction between planning ahead and satisfying actual decoder demand, while giving callers enough information to choose their own scheduling granularity. Let me know what you think! It'd be great if we can reach consensus on the API design first. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
