JingsongLi commented on PR #9529:
URL: https://github.com/apache/paimon/pull/9529#issuecomment-5496469621
I think the four-table split and the READY-last publication point are the
right direction. My remaining concern is that the current schema combines two
different version models without defining which one is authoritative for row
membership.
Today every component row carries `metadata_version`, while the dataset
manifest also records component snapshot IDs and creates tags. Consider two
imports:
```text
V1 / frames snapshot S1:
A(V1), B(V1)
V2 / frames snapshot S2:
A(V1), B(V1), A(V2), B(V2), C(V2)
```
Because the frame table is append-only, S2 is cumulative. Reading S2 or its
tag alone returns both V1 and V2. The actual V2 view still requires `dataset_id
= D AND metadata_version = V2`. This means `metadata_version`, rather than the
snapshot, currently defines which rows belong to a dataset version; the
snapshot/tag is only a retention and commit boundary.
That model is internally consistent only if every version is a complete
materialization. Adding C to an existing dataset must write A, B, and C again.
Writing only C under V2 would make the V2 filter return only the delta, unless
the reader implements parent-version replay and INSERT/UPDATE/DELETE overlay
semantics. The current implementation does not implement such a delta model.
I suggest choosing one explicit model before establishing this as the
persisted contract:
1. **Full row-version model**
- Logical keys include the version:
- frames: `(dataset_id, metadata_version, index)`
- episodes: `(dataset_id, metadata_version, episode_index)`
- tasks: `(dataset_id, metadata_version, task_index)`
- Every import is documented as a full materialization.
- Every reader must push down both `dataset_id` and `metadata_version`.
- Snapshot IDs/tags are documented as retention fences, not as sufficient
version selectors.
- The API must return `metadata_version`, and the manifest needs a
publication sequence or a separate head pointer because a UUID cannot define
the latest version.
- Version deletion/GC is required because expiring an old snapshot does
not remove old-version rows that remain live in the latest append snapshot.
2. **Snapshot-state model**
- Component rows represent current logical state and do not carry
`metadata_version`.
- Stable logical keys are independent of the release:
- frames: `(dataset_id, stable_frame_id)`
- episodes: `(dataset_id, stable_episode_id)`
- tasks: `(dataset_id, task_id)`
- A dataset version is one manifest row containing the exact frames,
episodes, and tasks snapshot IDs.
- Appending a new episode writes only the new rows. The new snapshots
logically contain old plus new state while reusing unchanged files.
- Updates and deletes touch only affected logical rows; old tagged
snapshots still reproduce the previous version.
- The cross-table UUID remains useful, but only as a `version_id` or
`publication_id` that resolves the manifest. The component snapshots define
data membership.
I prefer the snapshot-state model because it matches Paimon
snapshot/time-travel semantics and LeRobot recording behavior, which is
primarily append-by-episode. It also avoids `O(number of versions * full
dataset size)` storage for large BLOB-backed datasets.
A possible manifest is:
```text
dataset_versions
dataset_id
version_id
parent_version_id
status
frames_snapshot_id
episodes_snapshot_id
tasks_snapshot_id
published_at
```
The publication flow can then be:
```text
reserve (dataset_id, version_id) as PENDING
commit/update frames
commit/update episodes
commit/update tasks
pin the three snapshots
publish the manifest as READY
optionally move dataset_heads[dataset_id] to version_id
```
The public import result should contain at least `dataset_id`, `version_id`,
and the component snapshot IDs. This is also important for empty imports, where
the current API returns `None` even though a new metadata version was published.
The important point is not that UUIDs and snapshots cannot coexist. They
can, but they need distinct roles: the UUID should identify the cross-table
release, while exactly one mechanism must define which component rows belong to
that release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
