Adarsh-Me commented on issue #923:
URL: https://github.com/apache/tsfile/issues/923#issuecomment-5793303724

   Design for append-on-complete in the C++/C APIs. Nothing here is written yet 
— I'd like a sanity check on the recovery strategy (§3) and the boundary-marker 
finding (§5) before code.
   
   ## 1. C++ surface
   
   One new method rather than an `open()` behaviour change:
   
   ```cpp
   // storage/RestorableTsFileIOWriter
   // Opens a TsFile for appending. A normally-closed file has its metadata tail
   // removed in place; an incomplete file is recovered as today. Returns E_OK 
and
   // can_write() == true, or an error with the file left byte-identical.
   int open_for_append(const std::string& file_path);
   
   bool is_append_on_complete() const;   // complete file, tail removed
   ```
   
   `open(path, truncate_corrupted)` must keep returning `can_write() == false` 
for a complete file — `test/file/restorable_tsfile_io_writer_test.cc:166` 
(`OpenCompleteFile`) pins `can_write()==false`, `has_crashed()==false`, 
`get_truncated_size()==TSFILE_CHECK_COMPLETE`, 
`get_tsfile_io_writer()==nullptr`. A separate entry point keeps every existing 
test untouched.
   
   `has_crashed()` should stay `false` on the append path. Today the recovery 
scan sets `crashed_ = true` unconditionally 
(`restorable_tsfile_io_writer.cc:837`), so that assignment needs to become 
mode-dependent; otherwise callers that use `has_crashed()` to decide whether 
data was lost will misread a clean append as a crash salvage.
   
   `get_truncated_size()` on the append path returns bytes removed (`file_size 
- new_size`), keeping the existing `-2`/`0` sentinels for the 
incompatible/complete cases.
   
   ## 2. C surface, and one data-loss trap
   
   ```c
   TsFileWriter tsfile_writer_append_new(const char* pathname, ERRNO* err_code);
   TsFileWriter tsfile_writer_append_new_with_memory_threshold(
       const char* pathname, uint64_t memory_threshold, ERRNO* err_code);
   ```
   
   These must not be built on `write_file_new()` / `TsFileWriter::open(path)`, 
because both truncate on open:
   
   - `src/cwrapper/tsfile_cwrapper.cc:177` — `int flags = O_RDWR | O_CREAT | 
O_TRUNC;`
   - `src/writer/tsfile_writer.cc:298` — `return open(file_path, O_RDWR | 
O_CREAT | O_TRUNC, 0666);`
   
   So a user who today tries to append through the existing C entry points 
loses the whole file before a writer object ever exists. The append entry 
points need their own non-truncating internal open (`O_RDWR|O_CREAT|O_BINARY`, 
no `O_TRUNC`, no `O_EXCL`), and the docs for `write_file_new()` should say 
plainly that it discards an existing file.
   
   Ownership: `TsFileWriter::init(RestorableTsFileIOWriter*)` sets 
`io_writer_owned_ = false` (`tsfile_writer.h:197`), so the C handle has to own 
the `RestorableTsFileIOWriter` and destroy it *after* the writer. The C++ side 
already has the right shapes — `TsFileTableWriter(RestorableTsFileIOWriter*, 
...)` at `src/writer/tsfile_table_writer.cc:27` and the tree equivalent at 
`src/writer/tsfile_tree_writer.cc:36` — so the C work is mostly a context 
struct plus lifetime handling. The C API exposes no `RestorableTsFileIOWriter` 
at all today, so this is the first recovery entry point in the C layer.
   
   ## 3. Recovery strategy: reuse the chunk scan, use the footer only to find 
the cut
   
   I checked the alternative — deserialize `FileMetadata` from the footer the 
way Java does — and I'd rather not. The scan already produces exactly what 
`TsFileWriter::init(RestorableTsFileIOWriter*)` consumes 
(`tsfile_writer.cc:152-231`): `ChunkGroupMeta`s with real device IDs, 
per-measurement `data_type_/encoding_/compression_type_`, recovered statistics, 
`aligned_devices_`, and `update_table_schema()` + `finalize_table_schemas()` 
for the table model. Footer-driven recovery means writing a second recovery 
path over index-node structures, and it would have to re-derive per-chunk 
statistics that the scan already decodes from the pages themselves. Reusing the 
scan also inherits ~12 existing recovery tests (`TableWriterRecoverAndWrite`, 
`AlignedTimeseriesRecoverAndWrite`, `MultiDeviceRecoverAndWriteWithTreeWriter`, 
`RecoveryAlignedSparseStatRespectsBitmap`, …) instead of starting from zero.
   
   Algorithm, all before any mutation:
   
   1. `fstat`; `size == 0` → existing empty-file path.
   2. Validate head magic + `VERSION_NUM_BYTE` (existing code, 
`restorable_tsfile_io_writer.cc:609-628`).
   3. Read tail magic. Absent → the file is already incomplete, fall straight 
into the existing recovery scan.
   4. Present → read `meta_size` as big-endian i32 at `size - 10`; require 
`meta_size > 0` and `meta_size + 11 <= size`; cut = `size - 10 - meta_size - 1`.
   5. Validate the marker byte at `cut` (§5).
   6. Bounded recovery scan over `[header_len, cut)`, which must consume 
exactly up to `cut`.
   7. Only now `truncate(cut)`, `seek_to_end()`, `can_write_ = true`.
   
   Step 6 is where this differs from the crash path on purpose. Crash recovery 
is best-effort salvage: it truncates at the last parseable offset, which is the 
right call for a half-written file. Here the file is known-good, so partial 
progress is a failure: any early stop returns an error and leaves the file 
alone. Without that rule, a file whose chunks the scanner can't parse gets 
truncated at step 7 down to ~7 bytes, i.e. append destroys a file it should 
have refused.
   
   Two things already work in our favour:
   
   - `OPERATION_INDEX_RANGE` is handled by the scan 
(`restorable_tsfile_io_writer.cc:721-737`), which matters because `end_file()` 
writes that segment *before* the separator (`tsfile_io_writer.cc:305-307`), so 
truncation leaves the previous one in place and each append/close cycle adds 
another. The 1+16 skip means they are consumed cleanly rather than tripping the 
unknown-marker branch at `:814`. I want an explicit test for repeated 
append→close→append→close cycles, since accumulation is only benign if it stays 
benign at 3+ cycles.
   - Out-of-order rejection (requirement 9) is already wired: 
`init(RestorableTsFileIOWriter*)` sets `enforce_recovered_last_time_order_` 
(`tsfile_writer.h:198-202`) and it is checked at ~12 write/flush sites. Nothing 
new to design.
   
   Existing-table semantics need no decision either: `register_table()` already 
returns `E_ALREADY_EXIST` when the name is present 
(`tsfile_writer.cc:261-272`), so "evolve an existing table or aligned schema" = 
rejected, and appending to a recovered table works without re-registering. New 
device/new non-aligned measurement/new table after recovery all go through the 
ordinary `register_*` paths. Preserving the *latest* codec per series when a 
series was written with different codecs across chunks is already handled at 
`tsfile_writer.cc:217-225`.
   
   ## 4. Encryption
   
   There is no encryption in `cpp/src` at all — no `EncryptParameter` type and 
no cipher path anywhere in the tree. The writer only records properties, and 
the spot where encryption would live is still a TODO: 
`src/file/tsfile_io_writer.h:76-78` reads `if 
(common::g_config_value_.encrypt_flag_) { // TODO: support encrypt` … 
`encrypt_level_ = "2";` So an encrypted file is an unreadable file for this 
library, and `open_for_append()` should fail with `E_TSFILE_CORRUPTED` (C: 
`RET_TSFILE_CORRUPTED`, 35) rather than truncate. The bounded-scan rule in step 
6 above is what actually guarantees that, since encrypted chunk bytes will not 
parse. I'd document encrypted input as unsupported rather than designing a 
rejection code for it.
   
   ## 5. Boundary-marker finding — needs a maintainer answer
   
   The byte at `cut` is `SEPARATOR_MARKER` (0x02) in both current writers — 
`tsfile_io_writer.cc:600-603` and `TsFileIOWriter.java:510-514` (Java records 
`metaOffset = out.getPosition()` *before* writing the separator, so Java's 
`truncate(fileMetadataPos - 1)` and our `cut` land on the same offset; I 
verified the two `loadMetadataSize`/`write_separator_marker` definitions agree 
at byte level).
   
   It is **not** 0x02 in the fixtures checked into this repo. I measured all 
five, big-endian i32 at `size-10`:
   
   | file | size | meta_size | `cut` | byte at `cut` |
   |---|---|---|---|---|
   | `cpp/examples/test_cpp.tsfile` | 447 | 242 | 194 | 0x03 |
   | `python/tests/resources/simple_table_t1.tsfile` | 2965 | 351 | 2603 | 0x03 
|
   | `python/tests/resources/simple_table_t2.tsfile` | 2677 | 351 | 2315 | 0x03 
|
   | `python/tests/resources/simple_tree.tsfile` | 842 | 174 | 657 | 0x03 |
   | `python/tests/resources/table_with_time_column.tsfile` | 644 | 259 | 374 | 
0x03 |
   
   All five also carry head magic `TsFile` + version byte `0x04` and a matching 
tail magic, so the arithmetic (`meta_size` covers the metadata body only, 
separator is 1 byte before it) holds; only the marker's identity differs. 
`MetaMarker.VERSION = 3` is `@Deprecated` in Java and nothing in 
`java/tsfile/src/main/java` writes it any more, so my guess is these fixtures 
predate the current tail, but I don't know their provenance and I'd rather not 
encode a guess. Two questions:
   
   1. Should `open_for_append()` accept 0x03 at the boundary as well, or reject 
anything that isn't 0x02? My default is reject-with-error, which is safe for 
files this library writes and conservative for everything else.
   2. Do we want a "compatible enough" gate on the version byte beyond the 
exact `VERSION_NUM_BYTE == 0x04` equality check, given `VERSION_NUMBER_V3 = 
0x03` exists in Java and v3 files may still be in users' hands?
   
   The related "large-metadata" test case: I would *not* copy the reader's 
`ASSERT(tsfile_meta_size > 0 && tsfile_meta_size <= (1ll << 20))` 
(`tsfile_io_reader.cc:555`). `ASSERT` compiles to `assert()` unless `NDEBUG` 
(`src/utils/util_define.h:139-142`), so on a malformed tail that aborts the 
process instead of returning an error code — bad for a function whose whole job 
is to validate untrusted file bytes — and the 1 MiB ceiling would also reject 
the legitimately-large-metadata file the issue asks us to cover. The append 
path uses returned error codes and a size bound expressed against the file 
length only.
   
   ## 6. Test plan
   
   C++ (`test/file/`, next to the existing recovery tests; each is the 
complete-file twin of an existing truncated-file test): tree append and verify 
old+new rows; table append; new device and new non-aligned measurement after 
recovery; aligned-series and existing-table registration rejected with the 
documented code; timestamps older than the recovered per-device `last_time_`; 
no-op append+close then a full read-back diff of every series' statistics — 
that one is the real guard, because the rewritten tail is regenerated from 
scanned chunk-level metadata, not copied from the original footer, so a silent 
loss there needs to be caught rather than assumed; repeated append/close cycles 
for the `OPERATION_INDEX_RANGE` accumulation; empty, header-only, bad-magic, 
bad-version, `meta_size` out of range, and separator-mismatch files all leaving 
the input byte-identical; plus the C wrapper round-trip, including that the 
append entry point does not truncate.
   
   Cross-implementation check I'd like to add: read a C++-appended file back 
with the Java reader, so the regenerated tail is proven acceptable to the 
reference implementation and not just to the code that wrote it.
   
   ## 7. What I have and have not verified
   
   Verified: the file-format arithmetic and every source claim above, by 
reading the code at the cited lines, plus the byte measurements in §5 run 
against the five checked-in fixtures.
   
   Not verified: I have not compiled or run anything. This host has no 
`cmake`/`ninja` and no configured MSVC/MinGW build of libtsfile, so no test in 
§6 has executed, and I am not claiming any behaviour I have not read from 
source. First PR will therefore be the truncation/validation step plus one 
round-trip test, run locally, before the C wrapper lands.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to