Hi all, Iād like to share a recently merged improvement to the TsFile C++ reader that adds memory-mapped I/O as an optional backend for local file reads.
PR: https://github.com/apache/tsfile/pull/920 Issue: https://github.com/apache/tsfile/issues/903 Previously, local reads in the C++ reader used the positioned-read path (pread on POSIX). This change introduces three configurable backends: PREAD: preserves the existing behavior and remains the default. AUTO: prefers memory mapping for supported local files and falls back to PREAD when mapping is unavailable. MMAP: explicitly requires memory mapping and returns an error if mapping cannot be used. The implementation supports POSIX mmap and Windows file mapping, and the configuration is also exposed through the C and Python APIs. I also ran benchmarks comparing PREAD and MMAP. For a DataFrame workload with many fine-grained metadata, chunk, and page reads, MMAP improved end-to-end throughput by approximately: 27ā30% with one query worker 11ā12% with four query workers MMAP is not faster for every workload. In particular, a newly created mapping can be slower for a one-time sequential scan because of first-touch page faults. It tends to be more useful when the same files are repeatedly accessed or when the workload contains many small or random reads. More details about the benchmark setup and results are available in the PR description. Feedback on the API, backend behavior, or performance results is very welcome, especially if you have workloads where this read path could be useful. Best, Leyang Lyyy [email protected]
