ColinLeeo opened a new issue, #1138: URL: https://github.com/apache/tsfile/issues/1138
### Motivation TsFiles are increasingly read directly from object storage (OSS/S3), mostly through the Python binding: `TsFileReader` accepts any user-provided seekable file object (#930, #1055), and behind it every read goes through `PythonRandomAccessReadFile` into the C++ reader. On local disks the current read pattern is fine, but on object storage each read call is a network round trip with 10-100 ms latency, so the number of small reads dominates query time. #1055 covers *supporting* object storage; this issue covers making the read path fast on it. ### What the read path does today 1. **Every `read()` pays a seek/tell dance.** `PythonRandomAccessReadFile::read()` ([python/tsfile/python_random_access_read_file.cc:158](https://github.com/apache/tsfile/blob/develop/python/tsfile/python_random_access_read_file.cc#L158)) does `tell()`, `seek(offset)`, one or more `read()` calls, then `seek()` back to restore the caller position - four or more Python calls per underlying read, and for fsspec-backed sources the `read()` typically triggers its own ranged GET. 2. **Reads are small and page-scoped.** `ChunkReader::read_from_file_and_rewrap()` ([cpp/src/reader/chunk_reader.cc:258](https://github.com/apache/tsfile/blob/develop/cpp/src/reader/chunk_reader.cc#L258)) reads at least 4096 bytes per call (`DEFAULT_READ_SIZE = 4096`). The aligned reader has the same pattern ([cpp/src/reader/aligned_chunk_reader.cc:482](https://github.com/apache/tsfile/blob/develop/cpp/src/reader/aligned_chunk_reader.cc#L482)). A sequential chunk scan therefore issues on the order of one read per page, and pages larger than the 4 KiB window cost two. 3. **Page bodies that exceed the window are fetched twice.** When `remaining_size() < compressed_size_`, `decode_cur_page_data()` re-reads from the page start ([cpp/src/reader/chunk_reader.cc:320](https://github.com/apache/tsfile/blob/develop/cpp/src/reader/chunk_reader.cc#L320)), discarding the bytes already buffered from the previous 4 KiB read. On object storage this overlapping re-read is a wasted round trip per page. 4. **Nothing overlaps latency.** The only concurrency in the read path is aligned multi-column page decode when `parallel_read_enabled_` is set and the query is aligned with multiple columns; a scan over many chunks/devices issues its reads strictly serially. 5. **Metadata open costs two reads per file**, and the dataset layer repeats it per file; that is acceptable today but is the floor for the multi-file read cost (#1123 already made selective table-metadata reads possible). ### Proposed work - **Read-ahead / coalescing layer.** Add a configurable read-ahead window (e.g. 1-4 MiB, with `0` meaning the current behavior) so sequential page scans turn many small reads into a few large ones. Either a `PrefetchingRandomAccessReadFile` decorator over the existing interface ([cpp/src/file/random_access_read_file.h](https://github.com/apache/tsfile/blob/develop/cpp/src/file/random_access_read_file.h)) or a window in the chunk readers. This must be opt-in and must not change local-file behavior or memory cost when disabled. - **Consume buffered bytes instead of re-reading.** When a page body starts inside the current window, read only the missing suffix rather than re-fetching from the page start. - **Cheaper Python adapter.** Prefer an offset-based read (`read_at`/`pread`, or fsspec `cat_file(start, end)`) when the source exposes one, and skip the save/restore `seek` when it does; avoid the per-read `tell()`. - **Parallel prefetch across chunks/devices.** Overlap object-storage latency with decode by prefetching the next chunk(s) on the existing thread pool, gated by an explicit configuration option. This overlaps with the cross-device scan direction discussed on the mailing list. ### Acceptance criteria - A benchmark against an S3-compatible store (MinIO/moto or fsspec with a counting wrapper) showing both request count and wall time for: point query, one-device range scan, multi-device scan. - Sequential scan issues at least an order of magnitude fewer read requests than today with read-ahead enabled. - Local-file performance and memory unchanged with the feature off; C++ ASan/LSan clean. - No change to the file format, and Java/C++ cross-language behavior unaffected. ### Related - #1055 object storage support (feature), #930 Python seekable file objects - #1080 point queries rebuild index metadata on every call - #1115/#1114 SPRINTZ decode-side work (the other half of the latency equation) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
