Hi Yuxia.
Thanks for sharing the FIP. The overall direction looks very promising to me.
Using RowId + RowPosIndex + LakeDv/LogDv to avoid full sort-merge during
primary-key union read makes sense, and the two-phase prepare/switch protocol
also looks like the right direction to reduce the stale-snapshot window.
I have several comments that I think are worth clarifying in the FIP before
moving forward:
RowPosIndex cleanup when files are removed externally
The design explains how LakeDv entries are cleaned when compaction replaces
files. However, I think we also need to clarify what happens when files are
removed by external operations, such as snapshot expiration, partition
expiration, rewrite, or data lifecycle management.
RowPosIndex is keyed by RowId and points to a physical file position. If the
underlying Paimon data file is removed outside the normal compaction path, the
RowPosIndex entry may become an orphan entry. Later updates/deletes may still
hit this stale RowPosIndex entry and generate LakeDv markers for a file that no
longer exists.
Could the FIP clarify whether such external file-removal operations are
forbidden for Fluss-managed Paimon tables, or whether we need an additional
reverse mapping from fileId to RowIds so RowPosIndex can be cleaned by removed
file?
Compatibility with FIP-27 and system column pollution
The design introduces a new __rowid column in Paimon files. Since FIP-27 is
trying to remove mandatory system columns from Fluss lake tables, I think the
FIP should clarify how __rowid fits into that direction.
In particular:
Is __rowid a hidden/internal column or a normal physical column?
Will external engines see it in SELECT *?
What happens if the user already has a column named __rowid?
How does schema evolution handle this internal column?
Without a clear answer, this may introduce schema pollution or compatibility
issues for downstream engines.
Merge engine compatibility should be narrowed or clarified
The FIP says all merge engines are compatible under FULL changelog mode. I
think this may be too optimistic.
For example, FIRST_ROW has special semantics and does not really support normal
UPDATE/DELETE behavior in Fluss today. But the DV design relies heavily on -U /
-D carrying the old RowId, so the old version can be located and masked. For
merge engines with special update/delete semantics, it is not obvious that this
assumption always holds.
I suggest narrowing the first version to the default DEDUPLICATE primary-key
table, or adding a detailed correctness explanation and test plan for each
supported merge engine.
RowPos SST file lifecycle and garbage collection
The FIP defines remote RowPos SST files under rowPos/{snapshotId}/..., which
are required for TabletServer prepare and recovery. However, the retention and
cleanup policy is not very clear yet.
If these files are deleted too early, TabletServer recovery may fail because it
cannot re-ingest RowPos SSTs. If they are never deleted, remote storage may
grow indefinitely.
Could the FIP define the retention rule for RowPos SSTs, and explain how it is
coordinated with:
DvRocksDB checkpoints,
current/latest DV-readable snapshot,
pending readable rounds,
Paimon snapshot expiration?
LakeDv returned to union read should be filtered by planned files
During union read, the Paimon scan already knows which data files will be read.
The FIP mentions that LakeDv entries may be filtered by planned file paths, but
I think this should be a required behavior instead of an optional optimization.
Otherwise, for delete-heavy tables, returning the whole bucket’s LakeDv may
introduce large network transfer, bitmap deserialization, and memory overhead.
The read cost may simply move from sort-merge to DV transfer.
I suggest making GetLakeDvSnapshot return only LakeDv entries for the files
planned by the current Paimon scan.
Overall, I think the FIP is heading in a good direction and the expected
performance improvement is very valuable. The main areas I would like to see
clarified are around lifecycle management, compatibility, and keeping the read
path lightweight.
Best,
Leonard
