ColinLeeo opened a new issue, #1055:
URL: https://github.com/apache/tsfile/issues/1055

   ## Motivation
   
   Users want to write TsFiles directly to object storage (OSS/S3) and read 
them back. Reading has been solved for the Python binding through user-provided 
seekable file objects, and the C++ reader now accepts a pluggable byte source. 
Writing has no equivalent hook in C++/C/Python, and the Java object-storage 
path depends on classes that live outside this repository.
   
   ## Current state (verified)
   
   **Java**
   
   - The file-system abstraction already understands object storage: 
`FSType.OBJECT_STORAGE`, the `os://` prefix in `FSUtils`/`FSPath`, 
`OSFileInputFactory`, `OSFileOutputFactory`, `OSFSFactory`, and the hybrid 
factories.
   - Those factories load implementations reflectively from the config strings 
`objectStorageFile`, `objectStorageTsFileInput`, and 
`objectStorageTsFileOutput`, whose defaults are 
`org.apache.iotdb.os.fileSystem.OSFile`, `OSTsFileInput`, and `OSTsFileOutput`. 
Those classes are not in this repository, so the in-repo `os://` path is 
unverified and requires an external jar.
   - The write path goes through the `TsFileOutput` interface and 
`FSFactoryProducer.getFileOutputFactory()` (`TsFileIOWriter`, 
`RestorableTsFileIOWriter`, `ForceAppendTsFileWriter`); the read path goes 
through `TsFileInput`.
   
   **C++**
   
   - Read: `RandomAccessReadFile` is an abstract byte source, and 
`TsFileReader::open(std::unique_ptr<RandomAccessReadFile>)` accepts a 
caller-provided source. No object-storage implementation ships in the 
repository.
   - Write: `WriteFile` is a concrete, fd-based class (`create`, `write`, 
`get_position`, `sync`, `truncate`, `seek_to_end`, `close`) used directly by 
`TsFileIOWriter`, `RestorableTsFileIOWriter`, `TsFileWriter`, 
`TsFileTableWriter`, and the C API (`write_file_new(pathname, ...)`). 
`TsFileIOWriter` also calls `file_->get_fd()` as an "is open" check, which an 
interface would have to replace.
   
   **Python**
   
   - `TsFileReader(source)` accepts a path/bytes/PathLike or a seekable binary 
file object (`create_tsfile_reader_from_python_file`), so fsspec-style remote 
reads already work (see #930).
   - `TsFileWriter(pathname, ...)` accepts a path only; there is no write sink.
   
   ## Proposal
   
   1. **C++ write abstraction**: introduce a `RandomAccessWriteFile` (name TBD) 
interface mirroring `RandomAccessReadFile`, with `is_opened()`, `write(...)`, 
`get_position()`, `sync()`, `close()`, and — required by recovery/append — 
`truncate(size)` and `seek_to_end()`. Replace `get_fd()`-based checks with 
`is_opened()`.
   2. **In-memory implementation**: a `MemoryRandomAccessWriteFile` that 
accumulates bytes and can be handed back or uploaded. This is useful for tests 
and for "build in memory, then upload" workflows.
   3. **Object-storage implementation**: either an OSS/S3 backend in the 
repository, or a spooling sink that writes to a local temporary file and 
uploads on close (object stores are append-unfriendly). Define explicit 
behavior for `open_for_append`, recovery, and truncation on objects; likely 
these are unsupported and must return a clear error.
   4. **Wire the sink through** `TsFileWriter`, `TsFileTableWriter`, 
`RestorableTsFileIOWriter`, the C API, and the Python writer.
   5. **Java**: decide whether to ship an in-repo OSS implementation (with 
MinIO integration tests) or keep the external-class contract; either way, add 
an end-to-end test so the `os://` path is covered, and audit `TsFileOutput` for 
the operations a remote backend needs (positioning, truncation).
   6. **Tests**: unit tests with the in-memory sink; integration tests against 
a local MinIO/S3 mock (gated by environment variables or a Maven profile); 
cross-language read-back of files written to object storage.
   
   ## Acceptance criteria
   
   - C++/C/Python can write a TsFile to a non-local sink through a documented 
API.
   - In-memory sink tests prove the write path is backend-independent.
   - At least one object-storage integration test writes and reads back a file, 
or the limitation is documented and returns an explicit error.
   - Java object-storage read/write is either implemented and tested in-repo or 
explicitly documented as requiring external classes.
   - No change to local-file behavior or performance.
   
   ## Related
   
   - #930 — remote random-access reading for TsFile (read side, already closed)
   - #923 — appending to a normally closed TsFile in C++ and C APIs
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to