loustler commented on PR #11648:
URL: https://github.com/apache/seatunnel/pull/11648#issuecomment-5187358491
Follow-up on the `hadoop-aliyun` lockstep bump: it needs two paired version
pins to move with it, and neither of them fails at build time. Both are in
`fdff4b65c`.
**`aliyun-sdk-oss` 3.4.1 → 3.13.2.** `hadoop-aliyun` 3.4.3 is compiled
against the version `hadoop-project` 3.4.3 manages, 3.13.2. SeaTunnel pinned
3.4.1 in three module-local properties, and that is the jar the distribution
ships. Resolving every `com/aliyun/oss` reference of `hadoop-aliyun` 3.4.3
against 3.4.1:
```
absent classes: com/aliyun/oss/model/ListObjectsV2Request
com/aliyun/oss/model/ListObjectsV2Result
<- AliyunOSSFileSystemStore, AliyunOSSFileSystemStore$2, OSSListRequest,
OSSListResult
signature changed (void -> VoidResult at 3.13.2):
OSSClient.listObjectsV2, .deleteObject, .abortMultipartUpload,
.setBucketAcl
```
| source | vs `aliyun-sdk-oss` | verdict |
|---|---|---|
| `hadoop-aliyun` 3.1.4 (`dev` today) | 3.4.1 | CLEAN |
| `hadoop-aliyun` 3.4.3 | 3.4.1 | **BROKEN** |
| `hadoop-aliyun` 3.4.3 | 3.13.2 | CLEAN |
`AliyunOSSFileSystemStore` is the core store class, so the middle row is a
`NoClassDefFoundError` on any OSS listing — the first thing an OSS read does.
**`org.jdom:jdom` 1.1 → `org.jdom:jdom2` 2.0.6.1.** `aliyun-sdk-oss` 3.13.2
references `org/jdom2/` at 23 sites and `org/jdom/` at none, and
`hadoop-project` 3.4.3 manages `jdom2` accordingly. Verified 3.13.2 resolves
clean against `jdom2` 2.0.6.1. The only external package it adds over 3.4.1 is
`com/aliyuncs/kms`, from `aliyun-java-sdk-core`, which is shipped at neither
version — so OSS with KMS/STS/RAM credentials was already unsupported and is
unchanged.
**Same check across the rest of the distribution.** `hadoop-aliyun` is the
only `org.apache.hadoop` artifact shipped as a bare jar; the rest of Hadoop
arrives inside the two self-contained shaded jars. The one other shipped
artifact linking to Hadoop across a version gap is `hive-exec` 3.1.3, built
against 3.1.0, and it comes back clean — **0 members definitely absent**, so
every Hadoop API method it calls on a shipped class resolves at 3.4.3. Its
unresolved Hadoop classes are `MiniDFSCluster`, `MiniMRCluster`,
`MiniYARNCluster`, `NameNode`, `FSNamesystem`, `DistCp`, `HadoopArchives`,
`registry/client/*` — HDFS-server, test-harness and tools packages the uber jar
has never carried (its pom diff against `dev` changes no bundled artifact), so
they were equally absent at 3.1.4.
**Coverage** — correcting one thing I said earlier: `connector-file-oss-e2e`
does exist, with `OssFileIT` and `OssFileWithMultipleTableIT`, and its SDK pin
is bumped here too so it resolves the same version the distribution ships. Both
tests are `@Disabled("Disabled because it needs user's personal oss account to
run this test")`, and MinIO cannot substitute because it speaks S3 rather than
the OSS protocol, so I have no way to run them. If you or another maintainer
can point `OssFileIT` at a real bucket, that would validate this end to end.
Gates on `fdff4b65c`: full reactor `install -DskipTests` **BUILD SUCCESS
(284/284)**; `tools/dependencies/checkLicense.sh` exit 0 (`aliyun-sdk-oss` and
`jdom2` are `provided` and the gate resolves `-DincludeScope=runtime`, so
neither appears in it — same as `jdom` before); root `spotless:check` exit 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
