xylaaaaa opened a new issue, #69884: URL: https://github.com/apache/doris/issues/69884
## Background Doris can query Iceberg, Paimon and Hudi tables natively, but Delta Lake tables written by Spark / Databricks are not readable today. This issue tracks native read support for Delta Lake. ## Design summary - **FE**: a new `fe-connector-deltalake` plugin built on Delta Kernel Java 4.4.1 (snapshot loading, partition pruning, data skipping, split generation). Metadata sources: Hive Metastore and file system. Delta tables inside an existing Hive catalog are routed to the Delta connector through the sibling mechanism, the same way Hudi and Iceberg tables are. - **BE**: read through FileScannerV2 only. A thin `DeltaReader` on top of the existing V2 `ParquetReader` handles column mapping, deletion vectors and partition values. No JNI read path: every Delta reader feature in scope can be read natively, because Delta deletes and updates only produce new Parquet files plus row-index deletion vectors. - **Thrift**: two new optional structs (`TDeltaLakeFileDesc`, `TDeltaLakeDeletionVectorDesc`) and `TTableFormatFileDesc` field 16 `deltalake_params`. Field 16 was checked to be unused on master, branch-4.2, branch-4.1, branch-4.0 and branch-3.1. - **Unsupported reader features** (anything outside the native whitelist) fail at planning time with the feature name in the error message. An older BE that receives a `deltalake` range returns `NotSupported` instead of reading the file as plain Parquet, so rolling upgrades cannot return deleted rows. ## Scope of the first milestone (M1, delivered as one PR) | Step | Content | |------|---------| | M1.1 | Basic read: `deltalake` catalog (HMS / filesystem), schema from `_delta_log`, type mapping, partitioned and non-partitioned tables | | M1.2 | Partition pruning, data skipping, `count(*)` pushdown | | M1.3 | Column mapping (name / id), `timestampNtz`, type widening | | M1.4 | Deletion vectors (on-disk and inline) | | M1.5 | Classic / multi-part / V2 checkpoints, snapshot cache, streaming split generation for large tables | | M1.6 | Delta tables in Hive catalog, object storage coverage, docs | Out of scope for M1 (later milestones): time travel, Unity Catalog, AWS Glue, `variantType`, CBO statistics. ## Prerequisites - [x] Lock Delta Kernel version: 4.4.1 (`delta-kernel-api` + `delta-kernel-defaults`); `delta-spark_4.0_2.13` 4.4.1 for the Spark 4.0 test environment - [ ] Resolve dependency conflicts in the plugin (RoaringBitmap 0.9.25 pin, Parquet 1.18.0, exclude `hadoop-client-runtime` and Unity Catalog client) - [x] Verify the Kernel 4.4.1 APIs needed by the connector - [x] Verify the Delta DV byte layout against real files (golden tables decode with the existing Iceberg V3 DV decoder) - [x] Check thrift field ids on all release branches - [ ] Verify the V2 `ParquetReader` reads Spark + Delta written Parquet files correctly - [ ] Verify every type-widening combination allowed by the Delta protocol is converted correctly by the V2 column mapper ## Task checklist ### BE - [ ] Thrift definitions - [ ] DV decoding for the Delta format, inline DVs, cardinality check - [ ] `DeltaReader`: split preparation, column mapping, DV wiring, time types and type widening - [ ] Register in FileScannerV2 and always use V2 for Delta scans ### FE connector - [ ] Module, build and packaging - [ ] Catalog properties, Kernel engine and storage configuration - [ ] HMS and filesystem metadata sources - [ ] Type mapping, schema and partition columns, partition value normalization - [ ] Reader feature whitelist - [ ] Scan planning, statement scopes, refresh semantics ### Pruning and count pushdown - [ ] Predicate conversion to Kernel predicates - [ ] Partition pruning and data skipping - [ ] EXPLAIN / profile information - [ ] `count(*)` pushdown ### Column mapping, type widening, timestamp_ntz and DV (FE side) - [ ] Table-side and file-side schemas for column mapping - [ ] `timestampNtz`, type widening - [ ] DV descriptors ### Metadata formats, cache and large tables - [ ] Classic / multi-part / V2 checkpoints - [ ] Latest-version cache and snapshot cache - [ ] Streaming split generation ### Hive catalog integration - [ ] Detect Delta tables in HMS and route them to the Delta sibling connector ### Test environment and regression - [ ] Delta on the existing Spark 4.0 + MinIO docker environment - [ ] Delta golden tables (v4.4.1, Apache 2.0) as preinstalled data - [ ] p0 regression cases for each milestone step - [ ] p2 cases: object storages, large tables, performance comparison ### Docs - [ ] Delta Lake catalog documentation ## Open questions - Docker setup for Delta tables registered in HMS. - Whether a custom Kernel engine (caching log / checkpoint reads) is needed; to be decided by planning-time measurements on large tables. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
