suxiaogang223 opened a new pull request, #70103:
URL: https://github.com/apache/doris/pull/70103
### What problem does this PR solve?
Issue Number: N/A
Related PR: #69343
Backport master commit `949a8061bfcabf51579f1892a5ffb18343aa42bf` to
branch-4.2. Upgrade the shared Paimon Java SDK from 1.4.2 to 2.1.0 for both FE
and the BE JNI connector.
The release branch predates the connector/plugin split on master. Adapt the
changes to its existing `fe-core` Paimon catalog, `paimon-connector` JNI
module, and `PaimonMergePlanner`; do not introduce the master connector SPI or
its module layout.
- Upgrade `hive-catalog-shade` to 4.1.0, which still embeds Hive 3.1.3 but
carries Paimon 2.1.0. The old shade bundle embeds Paimon 1.4.2 classes that
would otherwise shadow the upgraded SDK.
- Remove references and fixtures for the SDK's removed file-based privilege
APIs, retaining catalog query authorization and existing fallback-branch
coverage.
- Preserve NULL-safe MERGE UPDATE/DELETE with nullable primary keys.
MATCHED-only plans use INNER JOIN semantics; MERGE with INSERT uses a
target-presence marker before the outer join. Keep user expression binding
outside that private marker's scope.
- Preserve deferred `$file_indexes` query authorization on FE, including
each fallback branch. The existing 4.2 MVCC projection supplies the retained
source snapshot to both system-table payloads.
- Reject unsupported `scan.mode=latest-delta` OPTIONS instead of pinning it
as a full snapshot; explicit `@incr` ranges remain available.
- Mirror the existing Hadoop S3 configuration into catalog options for the
2.1 S3FileIO, while retaining explicit Paimon option precedence. Apply the
shared mapping to filesystem, HMS, JDBC, and DLF catalog construction.
- Adapt schema-manager and snapshot fixtures, cover default/checked integer
overflow and pre-epoch timestamp-to-date behavior, and enable the existing BE
`java.nio` access in JNI tests.
- Carry over the original nullable-key/SDK regression suites and
runner-generated expected outputs. New vector/full-text/ANN SQL features remain
outside this backport.
### Release note
Upgrade the Paimon Java SDK to 2.1.0. Fix NULL-safe MERGE matched
UPDATE/DELETE for nullable primary keys; preserve `$file_indexes` authorization
and reject unsupported `latest-delta` OPTIONS reads. Paimon 2.1 supports opt-in
checked integer aggregation and corrects pre-epoch timestamp-to-date casts. Its
legacy file-based privilege APIs have been removed; users of that upstream
authorization mechanism must migrate before upgrading.
FE and BE Java extensions must deploy the same SDK. Mixed 1.4.2/2.1.0
rolling-upgrade compatibility is not established.
### Check List (For Author)
- Unit tests: `EXTRA_FE_MODULES=paimon=be-java-extensions/paimon-connector
./run-fe-ut.sh --run
'Paimon*Test,AbstractPaimonPropertiesTest,DorisIOManagerTest,GlobalIndexAssignerTest'`
passed.
- FE: 480 scheduled, 478 passed, 2 pre-existing environment-gated catalog
tests skipped (`PaimonCatalogTest`, `PaimonDlfRestCatalogTest`).
- BE Java/JNI connector: 100 passed, no failures or skips.
- Includes SDK serialization with nullable keys, MERGE
target-presence/name collisions, deferred file-index authorization, S3
configuration precedence, cache governance and scan planning.
- FE and JNI Checkstyle: zero violations. `git -c
core.whitespace=-blank-at-eof diff --check` passed (retain the runner-generated
golden files' final blank separators).
- FE and JNI production/test Java compiled through the test runner against
Paimon 2.1.0 and hive-catalog-shade 4.1.0.
- Full `build.sh --fe` packaging was not completed: the local third-party
installation lacks the native Paimon/Lance archives, and that entry point
starts rebuilding the whole third-party set. No native C++ code is changed.
- Cluster regression and native BE runtime tests were not run locally.
Regression suites and golden outputs are carried from #69343; its master
validation totals are not claimed here.
- Behavior changed: Yes, Paimon 2.1.0 SDK behavior and the adaptations
described above.
- Does this need documentation: No separate documentation change;
compatibility changes are recorded in the release note.
### Self-review
- Goal and scope: backport the SDK upgrade and directly affected behavior to
the existing 4.2 layout. No master connector refactor or search feature is
included.
- Concurrency: no new threads, shared mutable state, or lock ordering.
Existing cache/transaction locks are unchanged.
- Lifecycle: no new resource owners or static initialization dependencies.
Tests close local catalog, writer, commit, and reader resources.
- Configuration: no new configuration key. Existing S3 settings now reach
the SDK's catalog-option reader; explicit Paimon settings retain precedence.
- Compatibility: the upstream legacy privilege API removal and same-SDK
deployment requirement are documented. Mixed-SDK rolling upgrades are
unverified.
- Protocol: no Thrift/Protobuf schema or field IDs are changed. FE and BE
dependency versions share one property.
- Parallel paths: FE catalog and BE JNI dependencies move together. Shared
S3 mapping covers all Hadoop-backed catalog constructors; fallback
authorization checks both branches.
- Conditions: the target marker distinguishes a real nullable-key row from
an outer join's null-extension. Unsupported delta mode fails explicitly.
- Tests: SDK serialization/nullable-key behavior, MERGE branch selection and
marker-name collisions, deferred authorization, S3 option precedence, and
existing Paimon reader/cache tests are included. Cluster regression files and
goldens are copied from the original PR, not hand-generated.
- Results: see validation above; original master test totals are not
represented as results for this backport.
- Observability: unsupported delta mode returns an actionable diagnostic;
existing scan error propagation is retained. No new metric is needed for this
adaptation.
- Persistence/transactions: no EditLog or recovery changes. Existing MERGE
cardinality and transaction atomicity logic remains in place; nullable-key
regression coverage includes duplicate-match rejection.
- Writes: only MERGE target-presence classification changes. Writer
commit/abort and storage ownership are unchanged.
- FE/BE variables: none added.
- Performance: no performance improvement is claimed; MERGE adds one
constant target projection only when INSERT branches require an outer join. S3
options are copied at catalog construction.
- Other concerns: full cluster/native regression validation and mixed-SDK
upgrades remain outside local verification.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
