Shekharrajak opened a new issue, #2536: URL: https://github.com/apache/fluss/issues/2536
### Search before asking - [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar. ### Motivation ## Summary This proposal introduces Iceberg V3 table format support in Fluss, replacing V2 equality deletes with **deletion vectors** (Roaring bitmaps) for 100-500x storage reduction and 5-10x faster CDC queries. **Key V3 Features:** - Deletion Vectors (binary bitmaps in Puffin files) - New Data Types (Variant, Geometry, Geography, TimestampNano) - Default Values (schema evolution without backfill) - Row Lineage (`_row_id`, `_last_updated_sequence_number`) ## Why This Matters | Metric | V2 (Current) | V3 (Proposed) | |--------|--------------|---------------| | Delete storage | 100 bytes/delete | ~1 byte/delete | | Query with 10K deletes | 500ms | 50ms | | Schema evolution | Full rewrite | Instant (defaults) | ## Non-Goals (for initial implementation version) - Variant query pushdown - Geometry spatial indexing - Automatic V2→V3 migration - Cross-table transactions ### Solution ## Implementation Phases 1. Type System: Variant, Geometry, TimestampNano types 2. Deletion Vectors: DV writer, committer, merger 3. Row Lineage: _row_id assignment and tracking Testing & Docs along the way. We can have Github CI workflow setup to have integration testing with various system, query engine and environments. ## Rough Components: 1. V3DeltaTaskWriter - writes data files + builds deletion vectors - uses: DeletionVectorWriter - produces: IcebergWriteResultV3 2. DeletionVectorWriter - serializes Roaring bitmap to Puffin file - produces: DeletionVectorMetadata 3. IcebergWriteResultV3 - contains: dataFiles, deletionVectors, firstRowId, addedRows 4. DeletionVectorMetadata - contains: puffinFilePath, offset, length, referencedDataFile, cardinality 5. IcebergLakeCommitter - consumes: IcebergWriteResultV3 - does: atomic commit via RowDelta API 6. DeletionVectorMerger (background) - merges multiple DVs into one - triggers on: count > 5 OR cardinality > 10K ### Anything else? ## Request for Feedback - Architecture: Is async DV merge the right choice for Fluss workloads? - Performance: What benchmarks should we prioritize? - Compatibility: Any concerns with V2→V3 upgrade path? ### Willingness to contribute - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
