ColinLeeo opened a new issue, #923:
URL: https://github.com/apache/tsfile/issues/923
### Motivation
The Java implementation can reopen a normally closed (complete) TsFile for
appending through:
`RestorableTsFileIOWriter.getWriterForAppendingDataOnCompletedTsFile(File)`
This method detects a complete file, locates the start of `FileMetadata`
from the footer, truncates the separator marker and all tail metadata, recovers
the existing schema/chunk metadata, and then allows new data to be appended.
Closing the writer generates a new metadata section and tail magic.
The C++ implementation currently supports recovery and continued writing
only when the file is incomplete or its tail is damaged. When the tail magic is
valid, `RestorableTsFileIOWriter::self_check()` treats the file as complete,
sets `can_write_ = false`, and closes the write handle.
As a result, users of the C++ API and the C wrapper cannot append to a
TsFile that was closed normally, even though the equivalent workflow is
available in Java.
### Expected behavior
Provide a supported C++ API equivalent to the Java completed-file append
workflow, and expose it through the C API as well.
The workflow should:
1. Validate that the input is a compatible TsFile.
2. Detect whether the file is complete by checking the tail magic.
3. For a complete file, read the footer metadata size and locate the start
of `FileMetadata`.
4. Safely truncate the separator marker, metadata/index section, footer
size, and tail magic while retaining all completed chunk groups.
5. Recover the existing tree/table schema, alignment information, chunk
metadata, statistics, and per-device last timestamps.
6. Allow users to append data using the recovered schema.
7. Allow registration of supported new schemas after recovery, with clear
behavior for:
- a new device or measurement in the tree model;
- a new table;
- attempts to replace or evolve an existing table/aligned schema.
8. On close, write a new metadata section, footer, and tail magic so that
the resulting file is complete and readable.
9. Reject out-of-order timestamps relative to recovered data.
The operation should preserve the original file when validation or recovery
fails before truncation. The API documentation should make the
destructive/in-place nature of a successful append-open explicit.
### Current C++ behavior
For an incomplete or damaged-tail file, the following path already works:
`RestorableTsFileIOWriter::open(path, true)`
→ recover schema and chunk metadata
→ `TsFileWriter::init(&restorable_writer)`
→ append data
→ close and rebuild metadata
For a normally closed file, `open()` returns successfully but `can_write()`
is false, so it cannot be passed to `TsFileWriter::init()`.
### Suggested tests
- Append to a normally closed tree-model TsFile and verify old and new rows.
- Append to a normally closed table-model TsFile.
- Preserve recovered encoding/compression/data types and alignment.
- Register a new device and a new non-aligned measurement after recovery.
- Verify documented behavior for aligned-series and existing-table schema
evolution.
- Reject duplicate or incompatible schemas.
- Reject timestamps not newer than the recovered last timestamp.
- Handle empty, incomplete, incompatible, encrypted, and large-metadata
files.
- Verify that a no-op append followed by close produces a valid file without
losing data.
- Add coverage for the C wrapper API.
### Related implementation
- Java:
`RestorableTsFileIOWriter.getWriterForAppendingDataOnCompletedTsFile(File)`
- C++: `RestorableTsFileIOWriter::self_check()`
- C++ recovery integration: `TsFileWriter::init(RestorableTsFileIOWriter*)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]