suxiaogang223 opened a new pull request, #70395:
URL: https://github.com/apache/doris/pull/70395
### What problem does this PR solve?
Issue Number: None
Related PR: #70385 (backport of the Docker Paimon 2.1.0 fixture upgrade)
PR Type: Release/Dependency
Problem Summary:
Backport the final archive-based Docker Paimon upgrade from master to
branch-4.2. Spark 4.0.0 loads the Paimon 2.1.0 Spark 4.0 / Scala 2.13 connector
and S3 plugin from `iceberg_data_spark40_paimon210.zip`. Existing data
directories without these jars are refreshed before starting the stack, and the
catalog explicitly enables path-style access for MinIO.
### Details
- **Versions:** Docker Paimon jars move from 1.4.2 to 2.1.0. Spark 4.0.0 and
Iceberg 1.10.1 stay unchanged.
- **Upstream changes:** Exercise the Paimon 2.1.0 runtime in the Docker
Spark fixture, using the archive prepared for #70385.
- **Compatibility:** Preserve branch-4.2's separate Lance 0.4.0 download and
mount. Check only archive-provided Paimon jars when refreshing older data
directories. Branch-4.2's Doris Java dependency remains at 1.4.2; runtime
compatibility with the upgraded fixture has not been validated on this branch.
- **Rollback:** Revert this PR to restore the 1.4.2 fixture archive and
mounts.
- **Behavior change:** Docker startup uses the 2.1.0 archive and replaces
reused data directories that lack its Paimon jars.
### Testing
Locally passed `bash -n docker/thirdparties/run-thirdparties-docker.sh`,
`git diff --cached --check`, and Docker Compose configuration validation.
Verified that the prepared archive contains both mounted Paimon jars and that
branch-4.2's Lance download block is unchanged. No containers, Doris builds,
runtime regression suites, or CI test pipelines were started for this backport,
as requested.
The source master change previously passed all 14 Paimon fixture SQL files
and all 19 partition metadata comparisons using the prepared archive. Those
results are source-change evidence, not validation of the branch-4.2 runtime.
### Notes for Reviewers
Keep this PR in draft while pipeline testing is deferred. The commit
includes a CI skip directive; do not request pipeline execution until the
author resumes testing.
Required fixture object:
`regression/datalake/pipeline_data/iceberg_data_spark40_paimon210.zip` in the
regression buckets used by the pipeline. The archive was prepared alongside
#70385; its publication was not checked during this backport. Size: 278982666
bytes. SHA-256:
`d61a4b9ecacf3ef0682dd51b2be697452de0102cda5b8c7c91e58d45b3962f6f`.
Source commits: e2a3a1784dc1f59dd684357a85a26f318cf096d0 and
7d37a62c774ed10b9fc5cfe85db264ad7dd78fba. Applied together and adapted to
branch-4.2's startup script; no additional runtime preparation script is
introduced.
### Release note
None
### Check List (For Author)
- Test:
- [ ] Regression test
- [ ] Unit Test
- [x] Manual test: shell syntax, Compose configuration, archive jar
paths, and unchanged Lance download logic.
- Behavior changed:
- [ ] No.
- [x] Yes. The Docker Paimon fixture upgrades to 2.1.0 and older local
fixture directories are refreshed.
- Does this need documentation?
- [x] No.
- [ ] Yes.
- Pre-submission status: Not ready: pipeline testing is deferred at the
author's request; branch-4.2 runtime compatibility and published archive
availability remain unverified.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
### AI Disclosure
- Tool: Codex (GPT-6)
- Generated: Adapted backport, documentation and this description.
- Human review: Pending.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
