xudong963 commented on PR #803: URL: https://github.com/apache/arrow-site/pull/803#issuecomment-6118378677
I made some changes based on the reviews: 1. Explain Sans I/O earlier. Moved the API contract, decoding loop, and architecture diagram toward the beginning so readers understand the design before the detailed discussion. 2. Clarify the distinction from callbacks and async readers. Acknowledge that caching, retries, read racing, and shared scheduling can already work through callbacks. Explain the additional value of exposing byte dependencies, decoder resumption points, and independently schedulable decoding work. 3. Add concrete motivation. Include Andrew’s description of whole-file buffering at InfluxDB and a production report about memory pressure from wide columns. These illustrate integration challenges without claiming that Sans I/O automatically solves them. 4. Show a concrete scheduling example. Explain how `try_next_reader` lets an application overlap batch decoding with subsequent reads, including the need to bound concurrency and preserve output ordering. 5. Clarify prefetching limitations. Distinguish immediate NeedsData requirements from future read planning. Explain why selective, data-dependent prefetching may require additional format-aware APIs. 6. Improve positioning and scope. Replace the lengthy reader survey with a focused discussion of existing designs, including networking Sans I/O, streaming parsers, parquet2, Polars, and cuDF. Describe cross-format integration as an architectural direction rather than an existing universal interface. 7. Make the policy examples more useful. Add guidance on when exact-range reads, coalescing, read-ahead, and whole-file caching are appropriate. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
