ColinLeeo opened a new issue, #930:
URL: https://github.com/apache/tsfile/issues/930
## Motivation
Apache TsFile has been integrated into Hugging Face datasets, supporting
TsFile loading through:
```python
load_dataset("tsfile", data_files=...)
```
The required Python dependency has also been deployed in the Dataset Viewer
environment.
In Hugging Face streaming scenarios, datasets uses `xopen()` to obtain remote
file objects managed by fsspec. For Hugging Face and storage services that
support HTTP Range requests, these file objects provide random access to
specific byte ranges.
The Parquet Reader uses this capability to read the footer, metadata, and
required data blocks from remote files. TsFile also uses footer metadata and
offset-based reads, so it can follow the same remote random-access model.
Related Hugging Face work:
- https://github.com/huggingface/datasets/pull/8160
- https://github.com/huggingface/dataset-viewer/pull/3358
- https://github.com/huggingface/datasets/issues/8499
## Current limitation
The current TsFile C++ read path is already offset-based. `ReadFile::read()`
accepts an offset and length, and the Reader uses metadata near the end of
the
file to locate the required chunks and pages.
However, the Python Reader ultimately passes its input to the C++ `ReadFile`
implementation as a local filesystem path:
```text
TsFileReader("hf://datasets/.../data.tsfile")
-> Cython/C API passes a path string
-> C++ ReadFile opens a local path
-> the local file API cannot open the hf:// path
```
In the Hugging Face TsFile builder, `open()` returns a remote file object
created by `xopen()` in streaming mode. The builder then calls `read(6)` to
read and validate the TsFile magic header. However, this remote file access
capability is not carried into `TsFileReader`.
The missing capability is therefore the connection between a remote
random-access source and the existing offset-based TsFile read path.
## Proposal
Add remote random-access reading support across the TsFile C++ Reader and
Python bindings.
The C++ read path should be able to consume a remote random-access source
provided by a language binding while preserving the existing upper-level file
parsing and query paths.
The Python bindings should connect seekable file sources provided by fsspec
or
the calling environment to the C++ read path. Hugging Face and fsspec would
continue to handle URLs, authentication, caching, retries, and HTTP Range
requests.
This proposal covers both the C++ I/O path and the Python integration,
without
prescribing a specific internal interface.
The expected behavior is:
1. Existing top-level Reader and query interfaces and semantics remain
unchanged.
2. The schema, metadata, chunk, page, and query layers do not depend on a
specific I/O backend.
3. Existing local TsFile reading behavior and performance are unaffected.
4. Selective remote queries transfer only the required file ranges.
5. Both the table model and tree model can use the same remote random-access
capability.
6. Local and remote reads produce consistent schemas and query results.
## Related local read backend work
The community is already working on an mmap-based local read backend:
- https://github.com/apache/tsfile/issues/903
- https://github.com/apache/tsfile/pull/920
Issue #903 and PR #920 focus on local file access methods and performance,
while this proposal focuses on remote random-access sources provided by the
Python environment.
Both should preserve the existing upper-level read path. Their underlying I/O
work should therefore be coordinated to avoid introducing duplicated or
independent read paths.
## Expected outcome
After this capability is available, the Hugging Face TsFile builder can use
the remote file access context already provided by `xopen()`:
```text
hf:// URL
-> Hugging Face xopen
-> fsspec seekable file source
-> TsFile Python Reader
-> existing offset-based TsFile read path
-> HTTP Range requests
-> Dataset Viewer output
```
This would allow Dataset Viewer to read and preview remote TsFile datasets,
and provide the technical foundation for officially registering
`format:tsfile`, reindexing existing datasets, and enabling the TsFile format
filter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]