vladislav-sidorovich opened a new issue, #17146: URL: https://github.com/apache/iceberg/issues/17146
### Feature Request / Improvement Tracking issue for updating `delta-lake` module to support conversion from Delta Lake format to Iceberg v3 format. Initial update PR: https://github.com/apache/iceberg/pull/15407 ## Millestones: - [x] Converts all data types - [x] Converts table schema and partitions spec - [x] Support UPDATES and DELETS (Delta Lake Remove action) - [x] Support Delta VACUUM scenario - [x] Support DVs - [ ] Handle Schema evolution - [ ] Handle Edge cases for partitions and Generated columns - [ ] Support All Delta Lake actions - [ ] Support All Delta Lake features (column mapping, generated columns and so on) - [ ] Incremental Conversion (from/to a specific Delta Version) ## Detailed plan ### SCHEMA & METADATA - [ ] Map Delta (Spark) primitive types (IntegerType, StringType, TimestampNTZType) to Iceberg primitive types. - [ ] Map complex types (StructType, ArrayType, MapType) recursively. ### TABLE LAYOUT & PARTITIONING - [ ] Parse Delta's partitionColumns from the metadata action. - [ ] Analyze Delta's generatedColumns expressions (e.g., CAST(ts AS DATE), YEAR(ts)). - [ ] Map Delta generated expressions to Iceberg Transforms (day(ts), year(ts)). - [ ] Map explicit (non-generated) partition columns to Iceberg 'identity' transforms. - [ ] Read Delta's latest OPTIMIZE commit info to extract zOrderBy or Liquid Clustering configurations. - [ ] Translate Delta OPTIMIZE into an Iceberg SortOrder attached to the table metadata. ### DATA FILES & STATISTICS TRANSLATION - [ ] Handle Add/Remove Delta Actions - [ ] Extract minValues, maxValues, nullCount, and numRecords from Delta metadata (add action) - [ ] Convert Delta string stats to Iceberg's truncated byte buffers (handle 16-byte truncation and upper-bound increments) - [ ] Handle the NaN gap (leave Iceberg nan_value_counts empty). - [ ] Handle VACUUM operation for Delta Lake tables ### HANDLE ALL DELTA LAKE PROTOCOL ACTION - [ ] Metadata (Schema definition/evolution) action - [ ] Add action - [ ] Remove action - [ ] CDC action - [ ] Txn action - [ ] Protocol action - [ ] Commit info action - [ ] Domain metadata action - [ ] Sidecar action - [ ] Check point action - [ ] Reconciliation action ### ADVANCED FEATURES & MUTABILITY - [ ] Deletion Vectors Support - [ ] Table Features & Properties ### CONTINUOUS SYNC - [ ] Store the last synced Delta version in the Iceberg table properties (e.g., delta_sync_version=15). - [ ] Build a poller that reads Delta commits starting from delta_sync_version + 1. - [ ] Handle on-the-fly schema evolution (apply Iceberg UpdateSchema API if a new Delta commit contains altered Metadata). ### CLEAN-UP - [ ] Remove usage of Delta Kernel internal classes ### Query engine None ### Willingness to contribute - [x] I can contribute this improvement/feature independently - [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community - [ ] I cannot contribute this improvement/feature at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
