Gabriel39 opened a new pull request, #68998:
URL: https://github.com/apache/doris/pull/68998
### What problem does this PR solve?
External connector queries can fail during catalog initialization, file
listing, or partition pruning after dependency isolation. This fixes the
following paths:
- Keep DLF and Paimon OSS adapters with their plugin APIs, isolate the Hive
expression implementation required by DLF, and share the host Jindo SDK/native
runtime. Use the DLF client's actual constructor signature, and preserve
standard Hive storage-format class names in persisted metadata.
- Normalize numeric ranges for Hadoop file listing and recognize legacy OSS
authorities while preserving raw object keys and the configured endpoint.
- Resolve ambiguous escaped Hudi partition names through their storage
locations, keep TIMESTAMPTZ predicates for typed filtering, and pass the
session timezone when BE parses offset-free partition values.
- Continue MaxCompute schema enumeration when a listed schema disappears
during its lazy reload, while preserving other errors.
The QL isolation serves DLF's local expression-proxy initialization. Current
connector metadata operations use partition-name APIs; private Kryo expressions
are not treated as an external wire format.
### Release note
Fix external catalog initialization and file/partition discovery failures
affecting DLF, Paimon OSS, Hudi, Hadoop glob paths, legacy OSS locations, and
concurrent MaxCompute schema deletion.
### Check List (For Author)
- Test
- [x] Unit Test
- [x] Manual test (isolated packaged plugins with a shared Jindo runtime)
- Behavior changed:
- [x] Yes. Previously failing connector paths now preserve the expected
catalog, file, and partition semantics. Numeric glob expansion is bounded to
prevent excessive allocation.
- Does this need documentation?
- [x] No.
### Validation
- Connector module suites and packages: HMS 155 tests, Hive 491, Hudi 255,
MaxCompute 155 (1 skipped), Iceberg 1,527 (5 skipped), and Paimon 726 (1
skipped); zero failures or errors in the final runs.
- Filesystem compatibility suites: 153 tests passed, including real Hadoop
local-filesystem glob expansion and captured OSS requests.
- BE ASAN: all 29 `FileScannerV2Test` tests passed. Without the production
fix, the new timestamp-partition test fails and the other 28 pass.
- Regression tests cover isolated expression-proxy loading, offline real DLF
construction, OSS SPI discovery, canonical HMS storage-format metadata, schema
deletion races, escaped Hudi paths, typed timestamp residuals, and
explicit-offset precedence.
- Actual plugin ZIPs with a host Hive 3 QL jar: both offline DLF
constructors, OSS SPI discovery, adapter/interface ownership, canonical format
names, shared Jindo classes, and native initialization through both connector
loaders passed.
- FE Checkstyle, clang-format 16, changed-line clang-tidy, build hygiene,
shell syntax, and diff whitespace checks passed.
The remote external regression suites require external services and were not
rerun locally. Standalone Hive Kryo serialization against the host QL jar still
encounters a host `HiveConf` dependency gap; caller and vendor-bytecode checks
confirm the affected name-based metadata paths do not invoke it. This PR does
not claim to enable that separate path.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
