ColinLeeo opened a new issue, #1058:
URL: https://github.com/apache/tsfile/issues/1058
## Motivation
Long-lived TsFile datasets need schema evolution: a table gains a column, a
column is retired, or a column is renamed between writes. Today the table model
has no supported API for any of these changes, and the file metadata format
stores a single `TableSchema` per table, so the behavior after re-registering a
schema is either undocumented or fails outright.
## Current state (verified)
**Java**
- `TsFileWriter.registerTableSchema` delegates to
`Schema.registerTableSchema`, which simply replaces the map entry
(`tableSchemaMap.put(...)`); there is no validation or evolution logic.
- The tablet write path (`checkIsTableExistAndSetColumnCategoryList`)
rejects any column missing from the registered `TableSchema` with
`NoMeasurementException`, and rejects type changes with
`ConflictDataTypeException`.
- `TableSchema.update(ChunkGroupMetadata)` can append field columns
discovered in flushed chunk metadata when `generateTableSchema` is enabled, and
silently downgrades a conflicting column type to `STRING`.
`generateTableSchema` defaults to off and is enabled by `ImportExecutor` and
tests, so this is not a supported evolution path.
- The file metadata stores one `TableSchema` per table (`TsFileIOWriter`
writes `schema.getTableSchemaMap()` at close).
- Readers resolve columns from that single file-level schema
(`MetadataQuerierByFileImpl.tableSchemaMap`, `TableSchema.findColumnIndex`).
- Aligned tree-model devices reject measurements registered after the first
flush (`flushedMeasurementsInDeviceMap` →
`error.write.tsfile_writer_new_measurement`).
**C++**
- `TsFileWriter::register_table_schema` returns `E_ALREADY_EXIST` when the
table is already registered, so a table schema cannot be changed within a
writer at all.
- `do_check_and_prepare_tablet` returns `E_COLUMN_NOT_EXIST` for unknown
columns and `E_TYPE_NOT_MATCH` for type changes.
- Readers use the single file-level `TableSchema`
(`file_metadata->get_table_schema`, `find_column_index`).
- `TsFileSeriesScanIterator` already guards per-column chunk cursor counts
against divergence in "files with schema evolution", so the read path has some
tolerance for uneven column histories.
**All languages**
- There is no public add/drop/rename column API. The Python bindings expose
`register_table` only, and the C API has no schema-evolution entry point.
## Proposal
1. **Writer APIs** for the table model: add a column, drop a column, and
rename a column. Decide whether the entry point is explicit methods
(`alterTable(table, add/drop/rename ...)`) or an "apply an updated
`TableSchema` with a diff" call. Define whether the tree model is in scope
(measurement add/drop/rename per device) or explicitly out of scope for this
issue.
2. **Metadata model**: record schema changes so old and new chunk groups can
both be interpreted:
- keep a union schema at file level, plus a schema epoch/version or
per-chunk-group schema reference;
- represent drops as tombstones rather than deleting metadata;
- represent renames as an alias mapping from the historical column name
to the current name.
Document the format implications and backward/forward compatibility rules
(new readers on old files, old readers on evolved files).
3. **Read semantics** to define and test:
- after a drop, the column is absent from results (its data may remain on
disk until compaction);
- after a rename, data written under the old name is returned under the
new name;
- after an add, old chunk groups return null for the new column;
- type changes are either rejected or explicit and documented — never
silently coerced to `STRING`.
4. **Name rules**: table-model column names are lowercased; renames must
follow the same rule and must not collide with existing or tombstoned names
(define the behavior when a name is reused with a different type).
5. **Append/recovery**: `RestorableTsFileIOWriter` must recover the schema
history, including drops and renames, when reopening a completed file (related:
#923).
6. **Language parity**: C++, C API, and Python writer/reader support
equivalent operations, plus a way for readers to list the current columns and,
if needed, the history.
7. **Tests**: cross-language write/read; files written before and after each
evolution step; queries touching old, new, and dropped columns; rename across
multiple chunk groups; drop then re-add with a different type; append after
evolution; and a compatibility check that files without evolution are
byte-for-byte behavior-compatible with current readers.
## Open questions
- Is a drop logical (hidden but recoverable) or physical (requires
rewriting/compaction)? Logical is the only append-friendly option.
- Are tag columns renamable? Tag columns participate in device-id encoding
and the table-model id path, so this may need to be disallowed.
- How should a 2.5.x reader behave when it encounters a file that uses
schema evolution: fail with a clear error, or ignore the changes? This
determines whether the change needs a format-version gate.
- Do we need type narrowing/widening (for example INT32 → INT64), or is a
type change always add-new + drop-old?
## Acceptance criteria
- A documented, tested writer API to add, drop, and rename table-model
columns (and a documented decision for the tree model).
- Metadata and read semantics defined for mixed chunk groups, including
nulls for added columns and alias resolution for renamed columns.
- C++, Java, Python, and C API stay interoperable: a file evolved in one
language reads back correctly in another.
- Old files remain readable unchanged, and the behavior of older readers on
evolved files is defined and documented.
- Append/recovery preserves schema history.
- Tests cover add/drop/rename, name reuse, type conflicts, and chunk-group
boundaries in all supported languages.
## Related
- #923 — appending to a normally closed TsFile in C++ and C APIs
- #1057 — full Java/C++ compatibility matrix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]