Hi all,

I’d like to share a recently merged improvement to the TsFile C++ reader that 
adds memory-mapped I/O as an optional backend for local file reads.

PR:
https://github.com/apache/tsfile/pull/920

Issue:
https://github.com/apache/tsfile/issues/903

Previously, local reads in the C++ reader used the positioned-read path 
(pread on POSIX). This change introduces three configurable backends:


PREAD: preserves the existing behavior and remains the default.



AUTO: prefers memory mapping for supported local files and falls back to PREAD 
when mapping is unavailable.



MMAP: explicitly requires memory mapping and returns an error if mapping cannot 
be used.


The implementation supports POSIX mmap and Windows file mapping, and the 
configuration is also exposed through the C and Python APIs.

I also ran benchmarks comparing PREAD and MMAP. For a DataFrame workload with 
many fine-grained metadata, chunk, and page reads, MMAP improved end-to-end 
throughput by approximately:


27–30% with one query worker



11–12% with four query workers


MMAP is not faster for every workload. In particular, a newly created mapping 
can be slower for a one-time sequential scan because of first-touch page 
faults. It tends to be more useful when the same files are repeatedly accessed 
or when the workload contains many small or random reads.

More details about the benchmark setup and results are available in the PR 
description.

Feedback on the API, backend behavior, or performance results is very welcome, 
especially if you have workloads where this read path could be useful.

Best,
Leyang




Lyyy
[email protected]

Reply via email to