xylaaaaa opened a new pull request, #68702:
URL: https://github.com/apache/doris/pull/68702
### What problem does this PR solve?
Issue Number: N/A
Related PR: #67470, #66782 (separate Iceberg/Azure work; not included as
feature changes here)
Problem Summary:
Doris currently exposes Databricks interoperability through the Iceberg
catalog path. This draft adds an experimental native Delta Lake catalog that
reads Delta metadata directly with Delta Kernel, rather than relying on an
Iceberg mirror.
The draft contains:
- A new `type=delta` connector with path and Unity Catalog adapters; Delta
Kernel 4.3.1, Unity Catalog client integration, snapshot/time-travel planning,
partition pruning, and an explicit supported-feature gate.
- Native Parquet scans, Delta deletion vectors and column mapping, plus
table-scoped AWS, Azure SAS and GCS credential handling. Shared Azure SAS/GCS
backend prerequisites are included because the Delta adapter emits those
credentials.
- Opt-in writes (`delta.write.enabled=true`): create, append, overwrite,
restricted copy-on-write DELETE/UPDATE/MERGE, TRUNCATE and Unity catalog
registration deletion. Catalog-managed snapshots/commits use the Unity/Kernel
path.
- Connector SPI 13.0 changes, snapshot-pinned COW routing and write
transactions, typed file-commit reports integrated with the current
final-report acknowledgment path, and focused FE/BE tests and Delta regression
suites.
This is a standalone, single-commit branch based on master `0c29961f45e`,
extracted from local development commit `1d957e6c8bb`. Course/dbt changes,
local configuration, independent Iceberg ADLS normalization and Iceberg Azure
dependency additions are excluded.
#### Draft status / validation boundary
This is **not ready to merge** and does not claim production Databricks
support.
Static checks passed on the source development checkout: SPI/Delta/FE-core
Checkstyle, connector import and metadata-funnel gates, BE header hygiene,
scoped clang-format 16 checks, and Java generation for the modified Thrift
definitions. The extracted branch passed code whitespace and scope/content
checks. Auto-generated regression .out files retain their original
tab-separated formatting (including empty fields and section separators); their
results were not edited by hand.
Full FE/BE compilation and FE/BE unit tests have **not been rerun on this
extracted branch**. Full compilation is being handled by the author. Prior
local fixture/SQL results are historical evidence only; they are not validation
of this draft.
A stale BE test enum reference found while extracting the branch is updated
to the current `ObjStorageProvider` name. The separate Azure auth source
formatting is normalized with clang-format 16. Neither change is a claim of
successful compilation.
Still required before merge:
- Standard FE/BE builds and relevant unit tests.
- Path and Unity SQL regression comparison with default FileScannerV2.
- Real Databricks workspace and cloud-storage E2E verification.
- Long-query credential renewal, feature/version compatibility and supported
DML semantics review. ABAC row filters/masks are rejected, not executed by this
implementation.
#### Outbound HTTP / privilege surface
Unity REST and object-storage requests are outbound HTTP/SSRF surfaces to
endpoints supplied through catalog properties or returned table/storage
metadata. Creating or changing the catalog configuration requires catalog
CREATE or ALTER privilege; scanning/writing requires the corresponding table
privileges and Unity/storage credentials. This connector introduces no
URL-based artifact download or dynamic code loading.
### Release note
Add an experimental native Delta Lake catalog with path and Unity Catalog
adapters. Databricks and multi-cloud production readiness remains unverified.
### Check List (For Author)
- Test:
- [x] Manual static validation (details and limitations above)
- [ ] Unit Test — rerun FE SPI/Delta/COW/write-report tests and relevant
BE tests on this branch
- [ ] Regression test — rerun native Delta path/Unity suites with the
default scanner
- Behavior changed:
- [ ] No.
- [x] Yes. Add an experimental external Delta catalog and shared storage
prerequisites.
- Does this need documentation?
- [ ] No.
- [x] Yes. A progress/validation record is included; user-facing website
documentation is still pending.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
