ColinLeeo opened a new issue, #930:
URL: https://github.com/apache/tsfile/issues/930

   ## Motivation
   
   Apache TsFile has been integrated into Hugging Face datasets, supporting
   TsFile loading through:
   
   ```python
   load_dataset("tsfile", data_files=...)
   ```
   
   The required Python dependency has also been deployed in the Dataset Viewer
   environment.
   
   In Hugging Face streaming scenarios, datasets uses `xopen()` to obtain remote
   file objects managed by fsspec. For Hugging Face and storage services that
   support HTTP Range requests, these file objects provide random access to
   specific byte ranges.
   
   The Parquet Reader uses this capability to read the footer, metadata, and
   required data blocks from remote files. TsFile also uses footer metadata and
   offset-based reads, so it can follow the same remote random-access model.
   
   Related Hugging Face work:
   
   - https://github.com/huggingface/datasets/pull/8160
   - https://github.com/huggingface/dataset-viewer/pull/3358
   - https://github.com/huggingface/datasets/issues/8499
   
   ## Current limitation
   
   The current TsFile C++ read path is already offset-based. `ReadFile::read()`
   accepts an offset and length, and the Reader uses metadata near the end of 
the
   file to locate the required chunks and pages.
   
   However, the Python Reader ultimately passes its input to the C++ `ReadFile`
   implementation as a local filesystem path:
   
   ```text
   TsFileReader("hf://datasets/.../data.tsfile")
       -> Cython/C API passes a path string
       -> C++ ReadFile opens a local path
       -> the local file API cannot open the hf:// path
   ```
   
   In the Hugging Face TsFile builder, `open()` returns a remote file object
   created by `xopen()` in streaming mode. The builder then calls `read(6)` to
   read and validate the TsFile magic header. However, this remote file access
   capability is not carried into `TsFileReader`.
   
   The missing capability is therefore the connection between a remote
   random-access source and the existing offset-based TsFile read path.
   
   ## Proposal
   
   Add remote random-access reading support across the TsFile C++ Reader and
   Python bindings.
   
   The C++ read path should be able to consume a remote random-access source
   provided by a language binding while preserving the existing upper-level file
   parsing and query paths.
   
   The Python bindings should connect seekable file sources provided by fsspec 
or
   the calling environment to the C++ read path. Hugging Face and fsspec would
   continue to handle URLs, authentication, caching, retries, and HTTP Range
   requests.
   
   This proposal covers both the C++ I/O path and the Python integration, 
without
   prescribing a specific internal interface.
   
   The expected behavior is:
   
   1. Existing top-level Reader and query interfaces and semantics remain
      unchanged.
   2. The schema, metadata, chunk, page, and query layers do not depend on a
      specific I/O backend.
   3. Existing local TsFile reading behavior and performance are unaffected.
   4. Selective remote queries transfer only the required file ranges.
   5. Both the table model and tree model can use the same remote random-access
      capability.
   6. Local and remote reads produce consistent schemas and query results.
   
   ## Related local read backend work
   
   The community is already working on an mmap-based local read backend:
   
   - https://github.com/apache/tsfile/issues/903
   - https://github.com/apache/tsfile/pull/920
   
   Issue #903 and PR #920 focus on local file access methods and performance,
   while this proposal focuses on remote random-access sources provided by the
   Python environment.
   
   Both should preserve the existing upper-level read path. Their underlying I/O
   work should therefore be coordinated to avoid introducing duplicated or
   independent read paths.
   
   ## Expected outcome
   
   After this capability is available, the Hugging Face TsFile builder can use
   the remote file access context already provided by `xopen()`:
   
   ```text
   hf:// URL
       -> Hugging Face xopen
       -> fsspec seekable file source
       -> TsFile Python Reader
       -> existing offset-based TsFile read path
       -> HTTP Range requests
       -> Dataset Viewer output
   ```
   
   This would allow Dataset Viewer to read and preview remote TsFile datasets,
   and provide the technical foundation for officially registering
   `format:tsfile`, reindexing existing datasets, and enabling the TsFile format
   filter.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to