ColinLeeo opened a new issue, #1138:
URL: https://github.com/apache/tsfile/issues/1138

   ### Motivation
   
   TsFiles are increasingly read directly from object storage (OSS/S3), mostly 
through the Python binding: `TsFileReader` accepts any user-provided seekable 
file object (#930, #1055), and behind it every read goes through 
`PythonRandomAccessReadFile` into the C++ reader. On local disks the current 
read pattern is fine, but on object storage each read call is a network round 
trip with 10-100 ms latency, so the number of small reads dominates query time. 
#1055 covers *supporting* object storage; this issue covers making the read 
path fast on it.
   
   ### What the read path does today
   
   1. **Every `read()` pays a seek/tell dance.** 
`PythonRandomAccessReadFile::read()` 
([python/tsfile/python_random_access_read_file.cc:158](https://github.com/apache/tsfile/blob/develop/python/tsfile/python_random_access_read_file.cc#L158))
 does `tell()`, `seek(offset)`, one or more `read()` calls, then `seek()` back 
to restore the caller position - four or more Python calls per underlying read, 
and for fsspec-backed sources the `read()` typically triggers its own ranged 
GET.
   
   2. **Reads are small and page-scoped.** 
`ChunkReader::read_from_file_and_rewrap()` 
([cpp/src/reader/chunk_reader.cc:258](https://github.com/apache/tsfile/blob/develop/cpp/src/reader/chunk_reader.cc#L258))
 reads at least 4096 bytes per call (`DEFAULT_READ_SIZE = 4096`). The aligned 
reader has the same pattern 
([cpp/src/reader/aligned_chunk_reader.cc:482](https://github.com/apache/tsfile/blob/develop/cpp/src/reader/aligned_chunk_reader.cc#L482)).
 A sequential chunk scan therefore issues on the order of one read per page, 
and pages larger than the 4 KiB window cost two.
   
   3. **Page bodies that exceed the window are fetched twice.** When 
`remaining_size() < compressed_size_`, `decode_cur_page_data()` re-reads from 
the page start 
([cpp/src/reader/chunk_reader.cc:320](https://github.com/apache/tsfile/blob/develop/cpp/src/reader/chunk_reader.cc#L320)),
 discarding the bytes already buffered from the previous 4 KiB read. On object 
storage this overlapping re-read is a wasted round trip per page.
   
   4. **Nothing overlaps latency.** The only concurrency in the read path is 
aligned multi-column page decode when `parallel_read_enabled_` is set and the 
query is aligned with multiple columns; a scan over many chunks/devices issues 
its reads strictly serially.
   
   5. **Metadata open costs two reads per file**, and the dataset layer repeats 
it per file; that is acceptable today but is the floor for the multi-file read 
cost (#1123 already made selective table-metadata reads possible).
   
   ### Proposed work
   
   - **Read-ahead / coalescing layer.** Add a configurable read-ahead window 
(e.g. 1-4 MiB, with `0` meaning the current behavior) so sequential page scans 
turn many small reads into a few large ones. Either a 
`PrefetchingRandomAccessReadFile` decorator over the existing interface 
([cpp/src/file/random_access_read_file.h](https://github.com/apache/tsfile/blob/develop/cpp/src/file/random_access_read_file.h))
 or a window in the chunk readers. This must be opt-in and must not change 
local-file behavior or memory cost when disabled.
   - **Consume buffered bytes instead of re-reading.** When a page body starts 
inside the current window, read only the missing suffix rather than re-fetching 
from the page start.
   - **Cheaper Python adapter.** Prefer an offset-based read 
(`read_at`/`pread`, or fsspec `cat_file(start, end)`) when the source exposes 
one, and skip the save/restore `seek` when it does; avoid the per-read `tell()`.
   - **Parallel prefetch across chunks/devices.** Overlap object-storage 
latency with decode by prefetching the next chunk(s) on the existing thread 
pool, gated by an explicit configuration option. This overlaps with the 
cross-device scan direction discussed on the mailing list.
   
   ### Acceptance criteria
   
   - A benchmark against an S3-compatible store (MinIO/moto or fsspec with a 
counting wrapper) showing both request count and wall time for: point query, 
one-device range scan, multi-device scan.
   - Sequential scan issues at least an order of magnitude fewer read requests 
than today with read-ahead enabled.
   - Local-file performance and memory unchanged with the feature off; C++ 
ASan/LSan clean.
   - No change to the file format, and Java/C++ cross-language behavior 
unaffected.
   
   ### Related
   
   - #1055 object storage support (feature), #930 Python seekable file objects
   - #1080 point queries rebuild index metadata on every call
   - #1115/#1114 SPRINTZ decode-side work (the other half of the latency 
equation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to