Thanks for sharing the design. I have two concerns: 1. Possible Orphan entries in RowPosIndex
We use Amoro to periodically expire historical data by deleting Iceberg data files and committing a new snapshot. These deletions do not generate Fluss -D changelog records. During the next tiering commit, Fluss can detect previously unprocessed external Iceberg snapshots and derive the removed data files from their snapshot changes. However, RowPosIndex is keyed by RowId and cannot be cleaned using only the removed fileId. Since data expiration does not produce replacement files, there is also no new SST mapping to overwrite the old entries. As a result, orphan RowPosIndex entries may accumulate indefinitely. 2. FIP-27<https://cwiki.apache.org/confluence/spaces/FLUSS/pages/406619416/FIP-27+Remove+Mandatory+System+Columns+From+Fluss+Lake+Tables> proposes removing mandatory system columns to avoid schema pollution and downstream impact. However, this DV design requires __rowid and __bucket in Iceberg data files. Could the FIP clarify how schema compatibility with FIP-27 will be handled? Best regards, Liting 发件人: yuxia <[email protected]> 日期: 星期一, 2026年7月6日 17:14 收件人: dev <[email protected]> 主题: [DISCUSS] FIP-47: Introduce Deletion Vectors to accelerate Primary Key Table Union Read over Paimon Hi all, I would like to start a discussion for FIP-47: Introduce Deletion Vectors to accelerate Primary Key Table Union Read over Paimon. The main problem this FIP tries to solve is the read cost of primary key table union read over Paimon. Today, union read needs to deduplicate data across the Fluss hot layer and the Paimon historical layer. For primary key tables, this currently requires a full sort-merge on the read path, so the cost grows with the table size. FIP-47 proposes introducing deletion vectors to record which rows have been updated or deleted, so union read can skip stale rows through lightweight bitmap filtering while preserving exactly-once semantics. FIP link: https://cwiki.apache.org/confluence/spaces/FLUSS/pages/438009916/FIP-47+Introduce+Deletion+Vectors+to+accelerate+Primary+Key+Table+Union+Read+over+Paimon I also have a PoC branch here: https://github.com/luoyuxia/fluss/tree/paimon-dv-support I ran a simple benchmark to evaluate Flink batch union read performance over a Paimon table. The benchmark used a single bucket with 10 million rows. The result is: - Without deletion vector: 13.44s - With deletion vector: 1.39s This is about a 10x improvement in this benchmark. For people who are more familiar with Iceberg than Paimon, you can also refer to this design document: https://docs.google.com/document/d/10phRucA3LJA5z_PIeTTpR7I_o0CgZkKuffA8Ot-XBeU/edit?tab=t.0#heading=h.ylmcf7juvl7v That document discusses how deletion vectors can be integrated with Iceberg. The overall architecture and core idea are shared with the Paimon integration, without major conceptual differences. So if you are not very familiar with Paimon but know Iceberg well, the Iceberg design may help explain the motivation and mechanism. Looking forward to your feedback. Best regards, Yuxia
