ColinLeeo opened a new issue, #1058:
URL: https://github.com/apache/tsfile/issues/1058

   ## Motivation
   
   Long-lived TsFile datasets need schema evolution: a table gains a column, a 
column is retired, or a column is renamed between writes. Today the table model 
has no supported API for any of these changes, and the file metadata format 
stores a single `TableSchema` per table, so the behavior after re-registering a 
schema is either undocumented or fails outright.
   
   ## Current state (verified)
   
   **Java**
   
   - `TsFileWriter.registerTableSchema` delegates to 
`Schema.registerTableSchema`, which simply replaces the map entry 
(`tableSchemaMap.put(...)`); there is no validation or evolution logic.
   - The tablet write path (`checkIsTableExistAndSetColumnCategoryList`) 
rejects any column missing from the registered `TableSchema` with 
`NoMeasurementException`, and rejects type changes with 
`ConflictDataTypeException`.
   - `TableSchema.update(ChunkGroupMetadata)` can append field columns 
discovered in flushed chunk metadata when `generateTableSchema` is enabled, and 
silently downgrades a conflicting column type to `STRING`. 
`generateTableSchema` defaults to off and is enabled by `ImportExecutor` and 
tests, so this is not a supported evolution path.
   - The file metadata stores one `TableSchema` per table (`TsFileIOWriter` 
writes `schema.getTableSchemaMap()` at close).
   - Readers resolve columns from that single file-level schema 
(`MetadataQuerierByFileImpl.tableSchemaMap`, `TableSchema.findColumnIndex`).
   - Aligned tree-model devices reject measurements registered after the first 
flush (`flushedMeasurementsInDeviceMap` → 
`error.write.tsfile_writer_new_measurement`).
   
   **C++**
   
   - `TsFileWriter::register_table_schema` returns `E_ALREADY_EXIST` when the 
table is already registered, so a table schema cannot be changed within a 
writer at all.
   - `do_check_and_prepare_tablet` returns `E_COLUMN_NOT_EXIST` for unknown 
columns and `E_TYPE_NOT_MATCH` for type changes.
   - Readers use the single file-level `TableSchema` 
(`file_metadata->get_table_schema`, `find_column_index`).
   - `TsFileSeriesScanIterator` already guards per-column chunk cursor counts 
against divergence in "files with schema evolution", so the read path has some 
tolerance for uneven column histories.
   
   **All languages**
   
   - There is no public add/drop/rename column API. The Python bindings expose 
`register_table` only, and the C API has no schema-evolution entry point.
   
   ## Proposal
   
   1. **Writer APIs** for the table model: add a column, drop a column, and 
rename a column. Decide whether the entry point is explicit methods 
(`alterTable(table, add/drop/rename ...)`) or an "apply an updated 
`TableSchema` with a diff" call. Define whether the tree model is in scope 
(measurement add/drop/rename per device) or explicitly out of scope for this 
issue.
   2. **Metadata model**: record schema changes so old and new chunk groups can 
both be interpreted:
      - keep a union schema at file level, plus a schema epoch/version or 
per-chunk-group schema reference;
      - represent drops as tombstones rather than deleting metadata;
      - represent renames as an alias mapping from the historical column name 
to the current name.
      Document the format implications and backward/forward compatibility rules 
(new readers on old files, old readers on evolved files).
   3. **Read semantics** to define and test:
      - after a drop, the column is absent from results (its data may remain on 
disk until compaction);
      - after a rename, data written under the old name is returned under the 
new name;
      - after an add, old chunk groups return null for the new column;
      - type changes are either rejected or explicit and documented — never 
silently coerced to `STRING`.
   4. **Name rules**: table-model column names are lowercased; renames must 
follow the same rule and must not collide with existing or tombstoned names 
(define the behavior when a name is reused with a different type).
   5. **Append/recovery**: `RestorableTsFileIOWriter` must recover the schema 
history, including drops and renames, when reopening a completed file (related: 
#923).
   6. **Language parity**: C++, C API, and Python writer/reader support 
equivalent operations, plus a way for readers to list the current columns and, 
if needed, the history.
   7. **Tests**: cross-language write/read; files written before and after each 
evolution step; queries touching old, new, and dropped columns; rename across 
multiple chunk groups; drop then re-add with a different type; append after 
evolution; and a compatibility check that files without evolution are 
byte-for-byte behavior-compatible with current readers.
   
   ## Open questions
   
   - Is a drop logical (hidden but recoverable) or physical (requires 
rewriting/compaction)? Logical is the only append-friendly option.
   - Are tag columns renamable? Tag columns participate in device-id encoding 
and the table-model id path, so this may need to be disallowed.
   - How should a 2.5.x reader behave when it encounters a file that uses 
schema evolution: fail with a clear error, or ignore the changes? This 
determines whether the change needs a format-version gate.
   - Do we need type narrowing/widening (for example INT32 → INT64), or is a 
type change always add-new + drop-old?
   
   ## Acceptance criteria
   
   - A documented, tested writer API to add, drop, and rename table-model 
columns (and a documented decision for the tree model).
   - Metadata and read semantics defined for mixed chunk groups, including 
nulls for added columns and alias resolution for renamed columns.
   - C++, Java, Python, and C API stay interoperable: a file evolved in one 
language reads back correctly in another.
   - Old files remain readable unchanged, and the behavior of older readers on 
evolved files is defined and documented.
   - Append/recovery preserves schema history.
   - Tests cover add/drop/rename, name reuse, type conflicts, and chunk-group 
boundaries in all supported languages.
   
   ## Related
   
   - #923 — appending to a normally closed TsFile in C++ and C APIs
   - #1057 — full Java/C++ compatibility matrix
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to