ColinLeeo opened a new issue, #1055: URL: https://github.com/apache/tsfile/issues/1055
## Motivation Users want to write TsFiles directly to object storage (OSS/S3) and read them back. Reading has been solved for the Python binding through user-provided seekable file objects, and the C++ reader now accepts a pluggable byte source. Writing has no equivalent hook in C++/C/Python, and the Java object-storage path depends on classes that live outside this repository. ## Current state (verified) **Java** - The file-system abstraction already understands object storage: `FSType.OBJECT_STORAGE`, the `os://` prefix in `FSUtils`/`FSPath`, `OSFileInputFactory`, `OSFileOutputFactory`, `OSFSFactory`, and the hybrid factories. - Those factories load implementations reflectively from the config strings `objectStorageFile`, `objectStorageTsFileInput`, and `objectStorageTsFileOutput`, whose defaults are `org.apache.iotdb.os.fileSystem.OSFile`, `OSTsFileInput`, and `OSTsFileOutput`. Those classes are not in this repository, so the in-repo `os://` path is unverified and requires an external jar. - The write path goes through the `TsFileOutput` interface and `FSFactoryProducer.getFileOutputFactory()` (`TsFileIOWriter`, `RestorableTsFileIOWriter`, `ForceAppendTsFileWriter`); the read path goes through `TsFileInput`. **C++** - Read: `RandomAccessReadFile` is an abstract byte source, and `TsFileReader::open(std::unique_ptr<RandomAccessReadFile>)` accepts a caller-provided source. No object-storage implementation ships in the repository. - Write: `WriteFile` is a concrete, fd-based class (`create`, `write`, `get_position`, `sync`, `truncate`, `seek_to_end`, `close`) used directly by `TsFileIOWriter`, `RestorableTsFileIOWriter`, `TsFileWriter`, `TsFileTableWriter`, and the C API (`write_file_new(pathname, ...)`). `TsFileIOWriter` also calls `file_->get_fd()` as an "is open" check, which an interface would have to replace. **Python** - `TsFileReader(source)` accepts a path/bytes/PathLike or a seekable binary file object (`create_tsfile_reader_from_python_file`), so fsspec-style remote reads already work (see #930). - `TsFileWriter(pathname, ...)` accepts a path only; there is no write sink. ## Proposal 1. **C++ write abstraction**: introduce a `RandomAccessWriteFile` (name TBD) interface mirroring `RandomAccessReadFile`, with `is_opened()`, `write(...)`, `get_position()`, `sync()`, `close()`, and — required by recovery/append — `truncate(size)` and `seek_to_end()`. Replace `get_fd()`-based checks with `is_opened()`. 2. **In-memory implementation**: a `MemoryRandomAccessWriteFile` that accumulates bytes and can be handed back or uploaded. This is useful for tests and for "build in memory, then upload" workflows. 3. **Object-storage implementation**: either an OSS/S3 backend in the repository, or a spooling sink that writes to a local temporary file and uploads on close (object stores are append-unfriendly). Define explicit behavior for `open_for_append`, recovery, and truncation on objects; likely these are unsupported and must return a clear error. 4. **Wire the sink through** `TsFileWriter`, `TsFileTableWriter`, `RestorableTsFileIOWriter`, the C API, and the Python writer. 5. **Java**: decide whether to ship an in-repo OSS implementation (with MinIO integration tests) or keep the external-class contract; either way, add an end-to-end test so the `os://` path is covered, and audit `TsFileOutput` for the operations a remote backend needs (positioning, truncation). 6. **Tests**: unit tests with the in-memory sink; integration tests against a local MinIO/S3 mock (gated by environment variables or a Maven profile); cross-language read-back of files written to object storage. ## Acceptance criteria - C++/C/Python can write a TsFile to a non-local sink through a documented API. - In-memory sink tests prove the write path is backend-independent. - At least one object-storage integration test writes and reads back a file, or the limitation is documented and returns an explicit error. - Java object-storage read/write is either implemented and tested in-repo or explicitly documented as requiring external classes. - No change to local-file behavior or performance. ## Related - #930 — remote random-access reading for TsFile (read side, already closed) - #923 — appending to a normally closed TsFile in C++ and C APIs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
