ColinLeeo opened a new issue, #903:
URL: https://github.com/apache/tsfile/issues/903

   ## Motivation
   
   The C++ TsFile reader currently accesses local files through 
`storage::ReadFile`. On POSIX platforms, `ReadFile::read()` uses `pread` to 
read data at a specified offset. The C API and Python reader ultimately share 
this read path.
   
   This proposal introduces `mmap` as an optional local-file read backend while 
retaining the existing `pread` implementation.
   
   ## Proposed behavior
   
   Support the following modes:
   
   - `AUTO`: prefer `mmap` for supported local files and fall back to `pread` 
if mapping is unavailable or fails.
   - `MMAP`: explicitly use memory-mapped I/O and return a clear error if it 
cannot be used.
   - `PREAD`: preserve the current behavior.
   
   `AUTO` could be the default mode. The configuration should follow the 
existing C++ configuration style, for example:
   
   ```cpp
   common::set_file_read_backend(FileReadBackend::AUTO);
   ```
   
   `ReadFile` can remain the common entry point and dispatch reads to the 
selected backend, avoiding changes to the existing metadata, chunk, and page 
read paths. Equivalent configuration should also be exposed through the C and 
Python APIs.
   
   ## Potential benefits
   
   - Reduce system-call overhead for frequent, fine-grained random reads.
   - Efficiently locate metadata, chunks, and pages by file offset.
   - Provide a foundation for future read-only views over mapped regions to 
reduce intermediate copies.
   - On POSIX systems, close the original file descriptor after a successful 
mapping, reducing long-lived FD usage when many TsFiles are opened concurrently.
   
   Both `pread` and `mmap` use the OS page cache, so the actual performance 
impact must be validated with benchmarks.
   
   ## Considerations
   
   - `mmap` should only be used for supported local regular files; other file 
systems should keep their existing backends.
   - `AUTO` must safely fall back to `pread`.
   - Mappings and platform-specific resources must be released when the reader 
is closed.
   - Files must not be truncated while mapped, as this may cause errors such as 
`SIGBUS`.
   - Large files, virtual address space limits, empty files, and mapping 
failures must be handled correctly.
   - Windows may use an equivalent mapping implementation or fall back to the 
existing read path.
   - The initial implementation may preserve `ReadFile::read()` and copy data 
from the mapped region. A zero-copy view API can be considered separately.
   
   ## Acceptance criteria
   
   - The C++ reader can correctly read TsFiles through the `mmap` backend.
   - Users can select `AUTO`, `MMAP`, or `PREAD`.
   - C and Python APIs can configure or inherit the same backend selection.
   - The TsFile format and query results remain unchanged.
   - Mapping failures and unsupported environments are handled correctly.
   - All mappings, file descriptors, and related resources are released on 
close.
   - Benchmarks compare `mmap` and `pread` for sequential reads, random 
queries, metadata-intensive reads, and concurrent multi-file workloads.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to