laskoviymishka opened a new pull request, #69031:
URL: https://github.com/apache/doris/pull/69031
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Upgrade the Iceberg Java SDK from 1.11.0 to 1.12.0. 1.12.0.
This change only upgrades the dependency and adapts Doris to it.
- Bump `iceberg.version` 1.11.0 -> 1.12.0, and `avro.version` 1.12.1 ->
1.12.2 to stay consistent with what iceberg 1.12.0 is built against (the pom
requires the two to move together).
- Bump the explicit caffeine pin in fe-connector-iceberg 2.9.3 -> 3.2.4.
Iceberg 1.12.0 moved to caffeine 3. The pin is a direct dependency, so it wins
Maven mediation; leaving it at 2.9.3 would package an older caffeine into the
iceberg plugin than iceberg-core is built against.
- Re-vendor `org.apache.iceberg.DeleteFileIndex` from 1.12.0 and re-apply
the Doris changes (public class/builder methods, schemasById comment,
equality-field-id sets). The vendored class shadows the SDK class in the same
package, and 1.12.0 changed it (an extra `specsById` constructor argument and
delete/data file partition-match validation), so keeping the 1.11.0 copy would
break the SDK's own callers at runtime. Since Doris passes `specsById` to the
builder, the cached scan planning path now also rejects delete files whose
partition spec or tuple does not match the data file, the same as upstream
iceberg.
- Pass `table.specs()` to `ManifestFiles.read` in `IcebergManifestCache`.
Iceberg 1.12.0 removed the deprecated `read(ManifestFile, FileIO)` overload,
which read the partition specs from the manifest file's own metadata (only
possible for Avro manifests). This matches the delete manifest path next to it
(`readDeleteManifest(..., table.specs())`) and `IcebergRemoveOrphanFilesAction`.
- Keep the AWS SDK at 2.29.52 and default iceberg-aws's HTTP client to
`urlconnection`. In 1.12.0, iceberg-aws's default `http-client.type` (`apache`)
builds an `Apache5HttpClient` from `software.amazon.awssdk:apache5-client`,
which first shipped in AWS SDK 2.32. Doris pins 2.29.52 on purpose (#58566:
newer SDKs add default S3 integrity checksums that break S3-compatible
services), so with the default every S3FileIO, Glue and STS client iceberg
builds failed with `NoClassDefFoundError:
software/amazon/awssdk/http/apache5/Apache5HttpClient$Builder`.
`IcebergCatalogFactory.buildBaseCatalogProperties`, the base of every catalog
flavor, now sets `http-client.type=urlconnection` unless the catalog sets it.
S3FileIO keeps the setting when it is serialized, so the BE iceberg metadata
scanner, which rebuilds the FileIO from it, uses the same client;
`url-connection-client` is added to that scanner, which only bundled
`apache-client` and `netty-nio-client`. An explicit `http-client.type=ap
ache` is rejected at CREATE/ALTER CATALOG instead of failing on the first S3
or Glue call.
`SchemaAwareDataTableScan` (the other vendored class) needs no change:
`DataTableScan`, `BaseTableScan`, `SnapshotScan` and `TableScanContext` are
identical between 1.11.0 and 1.12.0.
### Release note
Upgrade the Iceberg Java SDK to 1.12.0. Iceberg's AWS clients (S3FileIO,
Glue, STS) now use the URLConnection HTTP client by default, and the
`http-client.type=apache` catalog property is no longer accepted.
### Check List (For Author)
- Test: Unit Test
- Added
IcebergCatalogFactoryTest#buildBaseDefaultsIcebergAwsHttpClientToUrlConnection
(builds a real S3 client builder with the default) and
#buildBaseKeepsAnExplicitHttpClientType, and
IcebergCatalogPropertiesTest#createTimeRulesRejectTheApacheHttpClientType;
IcebergCatalogOptionsSnapshotTest snapshots now include the default.
- `build.sh --fe` passes. Ran every iceberg FE unit test with
run-fe-ut.sh (fe-connector-iceberg, fe-connector-metastore-iceberg,
be-java-extensions/iceberg-metadata-scanner): 1630 tests, 0 failures, 5 skipped.
- Iceberg regression suites not run.
- Behavior changed: Yes.
- Iceberg's AWS clients use the URLConnection HTTP client by default, so
`http-client.apache.*` properties no longer apply, and
`http-client.type=apache` is rejected at CREATE/ALTER CATALOG.
- Iceberg scans whose delete files have a partition spec or partition
tuple that does not match the data file now fail with a ValidationException
(iceberg 1.12.0 behavior) instead of applying the deletes.
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: related #68916
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
