Hi all, I would like to start a discussion for FIP-47: Introduce Deletion Vectors to accelerate Primary Key Table Union Read over Paimon.
The main problem this FIP tries to solve is the read cost of primary key table union read over Paimon. Today, union read needs to deduplicate data across the Fluss hot layer and the Paimon historical layer. For primary key tables, this currently requires a full sort-merge on the read path, so the cost grows with the table size. FIP-47 proposes introducing deletion vectors to record which rows have been updated or deleted, so union read can skip stale rows through lightweight bitmap filtering while preserving exactly-once semantics. FIP link: https://cwiki.apache.org/confluence/spaces/FLUSS/pages/438009916/FIP-47+Introduce+Deletion+Vectors+to+accelerate+Primary+Key+Table+Union+Read+over+Paimon I also have a PoC branch here: https://github.com/luoyuxia/fluss/tree/paimon-dv-support I ran a simple benchmark to evaluate Flink batch union read performance over a Paimon table. The benchmark used a single bucket with 10 million rows. The result is: - Without deletion vector: 13.44s - With deletion vector: 1.39s This is about a 10x improvement in this benchmark. For people who are more familiar with Iceberg than Paimon, you can also refer to this design document: https://docs.google.com/document/d/10phRucA3LJA5z_PIeTTpR7I_o0CgZkKuffA8Ot-XBeU/edit?tab=t.0#heading=h.ylmcf7juvl7v That document discusses how deletion vectors can be integrated with Iceberg. The overall architecture and core idea are shared with the Paimon integration, without major conceptual differences. So if you are not very familiar with Paimon but know Iceberg well, the Iceberg design may help explain the motivation and mechanism. Looking forward to your feedback. Best regards, Yuxia
