Hi community,

Apache TsFile has been integrated into Hugging Face datasets, supporting
TsFile loading through:

    load_dataset("tsfile", data_files=...)

The required Python dependency has also been deployed in the Dataset Viewer
environment.

In Hugging Face streaming scenarios, datasets uses xopen() to obtain remote
file objects managed by fsspec. For Hugging Face and storage services that
support HTTP Range requests, these file objects provide random access to
specific byte ranges. The Parquet Reader uses this capability to read the
footer, metadata, and required data blocks from remote files.

The TsFile read path is also offset-based. The Reader uses metadata stored
near the end of the file to locate and read the required chunks and pages.
However, the current Python Reader ultimately passes its input to the C++
ReadFile implementation as a local filesystem path. As a result, it cannot
directly use the remote file access capability provided by Hugging Face.

We would like to discuss adding remote random-access reading support to the
TsFile Python Reader. The Reader should be able to use a seekable remote file
source provided by the calling environment and read the required footer,
metadata, chunks, and pages on demand.

Hugging Face and fsspec would continue to handle URLs, authentication,
caching, retries, and HTTP Range requests. The TsFile Reader would continue
to reuse the existing file parsing and query paths.

We expect this capability to follow these principles:

1. The existing top-level Reader and query interfaces and semantics remain
   unchanged.
2. The schema, metadata, chunk, page, and query layers do not depend on a
   specific I/O backend.
3. Existing local TsFile reading behavior and performance are unaffected.
4. Selective remote queries transfer only the required file ranges.
5. Both the table model and tree model can use the same remote random-access
   capability.
6. Local and remote reads produce consistent schemas and query results.

The community is already working on an mmap-based local read backend:

- Issue #903:
  https://github.com/apache/tsfile/issues/903
- PR #920:
  https://github.com/apache/tsfile/pull/920

Issue #903 and PR #920 focus on local file access methods and performance,
while this proposal focuses on remote random-access sources provided by the
Python environment. Both should preserve the existing upper-level read path,
so their underlying I/O work should be coordinated to avoid introducing
duplicated or independent read paths.

Related Hugging Face work:

- TsFile datasets builder:
  https://github.com/huggingface/datasets/pull/8160
- Dataset Viewer TsFile dependency:
  https://github.com/huggingface/dataset-viewer/pull/3358
- TsFile format filter issue:
  https://github.com/huggingface/datasets/issues/8499

We would appreciate feedback from the community on the following questions:

1. Do we agree to add remote random-access reading support to the Python
   Reader?
2. Are there additional compatibility requirements for keeping the existing
   top-level interfaces and local read behavior unchanged?
3. Is there anything else that should be coordinated in advance with the
   local read backend work in issue #903 and PR #920?

Feedback and suggestions are welcome. Once we reach agreement on the
direction, we can follow up with an implementation proposal.

Thanks,
Colin

Reply via email to