xylaaaaa opened a new pull request, #69904:
URL: https://github.com/apache/doris/pull/69904
### What problem does this PR solve?
Issue Number: close #69884
Problem Summary:
Add native read-only Delta Lake support through a Kernel 4.4.1 FE connector
and
FileScannerV2's native Parquet reader. The reader preserves the
transaction-log
snapshot, logical/physical column mapping and deletion-vector row positions.
Catalogs can discover tables through Hive Metastore or a filesystem
warehouse;
existing Hive catalogs can route Delta tables to the same native path.
**This is a draft for implementation review. Performance acceptance, the
remaining
storage matrix, the website documentation PR and final M1 acceptance are
open.**
Implemented scope:
- HMS/filesystem discovery, catalog properties, type/reader-feature
validation,
statement-pinned snapshots and catalog-owned filesystem clients.
- Native Parquet scans; none/name/id column mapping, nested schema evolution,
timestamps/NTZ and the tested type-widening combinations.
- Portable disk and inline deletion vectors, including file-global row
positions
across byte-range splits. Legacy native bitmap encoding is rejected.
- Partition/file skipping, native Parquet row-group/page pruning, and exact
unfiltered metadata COUNT when all required file statistics are present,
subtracting deletion-vector cardinality. Other cases scan normally.
- Classic/multipart/V2 JSON and Parquet checkpoints, sidecars and log
compaction.
Fixed latest-version TTL (hits do not extend it), user-scoped snapshot
caching,
REFRESH invalidation and lazy streaming split enumeration.
- EXPLAIN/profile diagnostics for snapshot version, files, partitions and
splits.
Streaming sources close before scheduling completion; close failures
propagate.
- Optional Spark/HMS/MinIO fixtures, retained test data, storage regression
and a
shared-file Hive/Delta benchmark. Normal restarts reuse initialized
fixtures.
Time travel, Unity/Glue catalogs, variant, Delta writes and CBO statistics
remain
outside M1. No additional log/checkpoint content cache is introduced: the
million-file measurements did not justify it for M1.
Compatibility: all new Thrift fields are optional. Before opening this draft,
`TTableFormatFileDesc` field 16 was rechecked against current master and
branch-4.2/4.1/4.0/3.1 (highest existing IDs: 15/13/12/10/9); no ID is
reused and
existing meanings are preserved. Native Delta queries require compatible
FE/BE
versions; older BE versions do not implement this reader.
Outbound HTTP / SSRF surface: object-store endpoints supplied through catalog
properties cause FE log/checkpoint reads and BE Parquet/DV reads through
existing
filesystem clients. Configuring endpoints requires catalog CREATE_PRIV or
ALTER_PRIV; SELECT_PRIV can trigger reads. This implementation reuses
fe-filesystem configuration and does not fetch/load runtime artifacts from
catalog-supplied URLs.
### Release note
Add native read-only Delta Lake catalogs backed by Hive Metastore or
filesystem
warehouses, plus native Delta reads through Hive catalogs. Support
mapped/evolved
schemas, portable deletion vectors, V2 checkpoints, pruning, metadata COUNT
and
streaming file planning.
### Check List (For Author)
- Test
- [x] Regression test: eight Delta p0 suites passed twice against the
optional
Docker environment; million-file planning p2 passed; MinIO and ordinary
HDFS 3.2.1 storage p2 passed using the same generated expected output.
- [x] Unit Test: 165 ASAN BE tests (including existing Iceberg/Paimon
readers),
75 Delta connector, 2 metastore, 25 FE core and 274 Hive-related tests
passed.
- [x] Manual test: full FE and ASAN BE builds on local macOS ARM64;
clang-format 16, header hygiene and clang-tidy passed. Thirteen
physical
Parquet fixtures matched PyArrow; semantic golden fixtures matched
Spark.
Metadata COUNT profiles verified zero data reads. Three million-file
source passes completed in a 512 MiB JVM (first split 22–60 ms,
enumeration
1.46–1.86 s, sampled heap peak about 200 MiB). This synthetic fixture
has
no physical data objects and measures planning/count only. Restart
checks
preserved all 1,227 MinIO objects by path, size and SHA-256.
- [ ] Linux CI and remaining acceptance gates below.
- Behavior changed:
- [x] Yes. Adds native Delta catalogs/Hive routing and scan diagnostics;
streaming source-close errors now fail scheduling instead of being
logged
after completion.
- Does this need documentation?
- [x] Yes. EN/ZH website patches are prepared locally. Documentation PR
and
full website build are pending.
#### Performance evidence and remaining gates
All 22 TPC-H and 103 TPC-DS queries completed cross-reader result comparison
and
three alternating paired repetitions on the same retained SF1 Parquet
objects.
This combines several official regression invocations, with original failures
retained; it is not an unmodified full-suite pass. Both paths use
FileScannerV2,
four pipeline tasks, disabled SQL/file caches, UTC, 8 GiB/query and 120
seconds.
| Local ASAN -O0 measurement | Hive sum of per-query medians | Delta sum |
Delta / Hive |
| --- | ---: | ---: | ---: |
| TPC-H, 22 queries | 134.565 s | 96.342 s | 0.716 |
| TPC-DS, 103 queries | 232.545 s | 293.214 s | 1.261 |
These are local relative observations, not RELEASE performance, concurrent
throughput or certified TPC results. TPC-DS uses the repository's SF100 query
parameters on SF1 data; 12 queries have empty results. Excluding them, the
TPC-DS ratio is 1.215.
Explicit controls: Q7 and Q85 use the same fixed join order on both readers
after
default Hive plans exceeded memory; Q15 uses an equivalent CTE; Q30 projects
the
current generator's review-date string column. Results use the framework's
typed
comparator (exact integer/decimal, floating tolerance 1e-8). Q39's original
exact
comparison differed by one ULP in one cell; the original difference is
retained.
Nine complete profile pairs show different join build sides, runtime-filter
placement and scanner parallelism. For example, Q72's Delta plan builds from
11.745M inventory rows versus Hive's 62,699-row filtered build side; Q43
builds
from 2.750M sales rows versus Hive's 366 date rows. Missing CBO statistics
can
disable join reordering. Q22 variability and small-table streaming startup
still
need isolation; **performance parity is not accepted yet**.
- [ ] Resolve performance gaps within the agreed M1 scope.
- [ ] Validate OSS, COS, OBS, Azure, GCS and Kerberos HDFS when environments
become
available. MinIO does not establish acceptance against live cloud services.
- [ ] Submit the EN/ZH documentation PR and run website validation.
- [ ] Finish M1 acceptance and reconcile the implementation/design checklist.
The local tasklist currently records 60/64 completed items. Environment
configs,
local toolchains, large generated datasets and diagnostic logs are excluded
from
this PR; reusable fixture generators and regression suites are included.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
