JingsongLi commented on PR #9529:
URL: https://github.com/apache/paimon/pull/9529#issuecomment-5506179589
Following up on the version-model discussion, I compared this layout with
the metadata model used by LeRobot v3. I think a simpler persisted contract
would be to preserve the distinction that LeRobot already makes between Parquet
metadata and JSON metadata.
## Proposed model
1. Keep the LeRobot Parquet metadata as typed Paimon tables with the same
**logical** schema:
- `__tasks`: LeRobot task fields plus `version_id`
- `__episodes`: LeRobot episode fields plus `version_id`
Each published version is a complete metadata materialization in a new
`version_id` partition. It is not a delta.
2. Replace `__datasets` with a `__versions` manifest table. One row
represents one published LeRobot version and contains the JSON metadata and
publication information, for example:
```text
version_id
parent_version_id
version_sequence
status # PENDING / READY
published_at
info_json
stats_json
extra_metadata_json
frames_snapshot_id
episodes_schema_json
tasks_schema_json
source_revision
metadata_checksum
```
3. Keep frames snapshot-based. Frame rows should not carry `version_id`; the
exact frame state for a release is selected by `frames_snapshot_id` in the
version manifest. The snapshot should be pinned by a tag while that version is
retained.
This gives every component exactly one authoritative version selector:
```text
frames -> frames_snapshot_id
episodes -> version_id partition
tasks -> version_id partition
info/stats -> __versions.version_id
```
For a table group that contains only one logical dataset, partitioning by
`version_id` is enough. If companion tables can contain multiple datasets, use
`(dataset_id, version_id)` as the partition and include both fields in the
primary key:
```text
tasks PK: (dataset_id, version_id, task_index)
episodes PK:(dataset_id, version_id, episode_index)
```
## Update semantics
Suppose V2 adds one episode to V1:
```text
V1:
complete tasks metadata in partition V1
complete episodes metadata in partition V1
frames snapshot S1
V2:
complete tasks metadata in partition V2
complete episodes metadata in partition V2
frames snapshot S2 containing the old frames plus the new episode
```
Tasks and episodes are rewritten because they are relatively small metadata
and the full partition makes reading and deletion unambiguous. Frames are
appended and versioned by snapshot, so existing blob data is not copied for
every release.
## Publication protocol
```text
reserve version V as PENDING
write or append frames and obtain snapshot S
pin S with a tag
write the complete tasks partition V
write the complete episodes partition V
validate counts, references, schemas, and checksums
publish V as READY
optionally move dataset_heads.current_version_id to V
```
Readers must start from a `READY` version row. They must never infer a
release by independently opening the latest snapshots of the component tables.
`version_id` should remain an opaque release identifier. It should not be
used to determine the latest version; use a monotonic `version_sequence` or a
separate `dataset_heads` pointer.
## Schema compatibility
By "same schema" I mean the logical Arrow schema, not Parquet/Pandas
artifacts such as `__index_level_0__`. This should preserve fields such as
`tasks`, `data/chunk_index`, per-camera video offsets, and typed `stats/...`
fields instead of folding them into JSON.
There is one important constraint: Paimon schema is table-level, not
partition-level. Additive LeRobot changes can evolve the table schema, with old
partitions reading null for new fields. An incompatible type change for an
existing field must either:
- fail with a clear request to create a new table generation;
- use an explicitly defined canonical type; or
- fall back to lossless raw metadata storage.
The original Arrow schemas and fingerprints should therefore be recorded in
`__versions` and checked before writing a new partition.
I would also define a forward-compatibility rule for all metadata files:
- known `meta/*.parquet` files become typed companion tables;
- known `meta/*.json` files are stored in the version row;
- unknown files are preserved losslessly as `(version_id, path, bytes,
checksum)` rather than rejected or silently dropped.
This model is still a hybrid physical implementation, but it is not an
ambiguous version model: snapshot identity defines frame membership, partition
identity defines Parquet-metadata membership, and the `READY` manifest is the
single cross-table publication point. It also follows the native LeRobot v3
organization more closely than introducing a different normalized schema for
tasks and episodes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
