geyanggang opened a new pull request, #14532:
URL: https://github.com/apache/gravitino/pull/14532
### What changes were proposed in this pull request?
This PR resolves the three CRITICAL scan findings in the server distribution
that need a change of release line: Avro 1.7.7 (shaded in
`hadoop-client-runtime-3.3.6.jar`), `ranger-plugins-common` 2.4.0, and
ZooKeeper 3.6.3. Hadoop stays at 3.3.6.
#### 1. Avro shaded in `hadoop-client-runtime` (`build.gradle.kts`)
Hadoop 3.3.6 shades Avro 1.7.7 under
`org/apache/hadoop/shaded/org/apache/avro`. A dependency constraint cannot
replace classes inside another jar, and the first Hadoop release that shades
Avro 1.11.4 is 3.4.2. Upgrading only the Hadoop client would mix 3.3.6 and
3.4.x in one class loader, because users drop bundles with `hadoop-aws:3.3.6`
and similar jars next to it. Upgrading all of Hadoop is a much larger change.
This PR adds a Gradle artifact transform, `StripShadedAvroTransform`:
- **Scope:** it is applied to `runtimeClasspath` and `testRuntimeClasspath`
of the five modules that ship `hadoop-client-runtime` in the distribution:
- `authorization-ranger`
- `catalog-fileset`
- `catalog-lakehouse-iceberg`
- `iceberg-rest-server`
- `iceberg-rest-server-experiment`
- **What it removes:** only from `hadoop-client-runtime-*.jar`, and only
when the shaded Avro is older than 1.11.4. It removes
`org/apache/hadoop/shaded/org/apache/avro/**` and
`META-INF/maven/org.apache.avro/**`. Every other entry is copied byte for byte,
the file name is unchanged, and all other artifacts pass through untouched.
- **Fail-safe:** the build fails if a jar reports an outdated Avro version
but has nothing to strip.
- **Retirement:** once Hadoop is upgraded to a release that shades Avro
1.11.4 or later, the transform becomes a no-op and can be removed.
- **Tests:** they run against the same stripped jar that the distribution
ships.
Inside Hadoop, the shaded Avro only backs Avro serialization:
`SequenceFile`/`MapFile` through `SerializationFactory`, MapReduce job history,
and `hadoop fs -text`. Gravitino uses none of these. The checks I made:
- The Gravitino sources do not use these APIs.
- No third-party jar in the five `libs` directories references them.
- The connector code in the five cloud bundles (S3A, ABFS, GCS, OSS, COS)
does not reference them. Users may put those bundles into
`catalogs/fileset/libs`.
- `Path`, `Text` and `DelegationKey` only carry Avro annotations, and the
JVM ignores missing annotation types.
- Iceberg uses its own unshaded Avro 1.12.x.
#### 2. ZooKeeper 3.6.3 to 3.8.8 (`hive-metastore2-libs`,
`hive-metastore3-libs`)
ZooKeeper only reaches the Hive metastore client class loaders transitively,
through `hadoop-common`, `hadoop-auth`, Curator and the Hive shims. Gravitino
has no ZooKeeper code. The classes that reference ZooKeeper in these
directories are all server-side:
- `ZKFailoverController` and `ActiveStandbyElector` (HA failover)
- `ZKDelegationTokenSecretManager` (delegation tokens)
- `ZKSignerSecretProvider` (HTTP authentication)
- `ZooKeeperTokenStore` (HMS token store)
Changes:
- `zookeeper` and `zookeeper-jute` are pinned to 3.8.8 in the existing
`constraints` blocks. 3.6 is end of life and 3.8.8 is the current stable
release.
- ZooKeeper 3.8 declares Logback as a compile dependency, so Logback is now
also excluded from `hadoop3.mapreduce.client.core` in `hive-metastore2-libs`.
That was the only path without the exclusion, and no Logback jar ships.
- Side effect: Netty in these directories moves from 4.1.63 to 4.1.138, and
commons-io from 2.8.0 to 2.17.0.
#### 3. Ranger 2.4.0 to 2.8.0 (`authorization-ranger`)
Bumping the version alone breaks the plugin. These are the changes needed to
make it work.
**Client reflection**
- `RangerClientExtension` looked up the private `RangerClient.callAPI(API,
Map, Object, GenericType)` by reflection. Ranger 2.8 changed that parameter to
`TypeReference`, so the constructor threw `NoSuchMethodException`.
- The handle was never used, so the lookup is removed. The two `callAPI`
overloads that are used have the same signatures in 2.4 and 2.8.
**JSON mapping of the user and group models**
- Ranger 2.8 serializes requests with `JsonUtils` (Jackson 2) and parses
responses with `JsonUtilsV2` (a plain Jackson 2 `ObjectMapper`). Ranger 2.4
used Gson and Jackson 1.
- `VXDataObject`, `VXUser`, `VXGroup`, `VXUserList`, `VXGroupList` and
`JsonDateSerializer` now use the equivalent Jackson 2 annotations.
- `ignoreUnknown` is added to the two list classes as well, so fields added
by newer Ranger Admin versions are tolerated.
**Immutable empty collections in Ranger model objects**
- In Ranger 2.8, initializing a `RangerBasePlugin` in the same JVM switches
the JVM-wide "null-safe supplier" to `v2`. In `v2`, empty model collections are
`Collections.emptyList()`/`emptyMap()`. The Spark authz plugin in the E2E tests
does this, and so does any deployment that embeds a Ranger plugin.
- In this mode, code that called `getPolicyItems().add(...)` or
`getResources().put(...)` threw `UnsupportedOperationException`. That surfaced
as 31 failures in `RangerHiveIT`.
- Those call sites now use the model mutators (`addPolicyItem`,
`addDenyPolicyItem`, `addAccess`, `addUser`, `addGroup`, `addRole`,
`setResource`).
- The remaining `removeIf` calls are safe in both modes.
**Existing bug: `RangerHelper.hasGravitinoManagedPolicyItem` modified the
policy**
- It appended the deny, row-filter and data-mask items to the policy's own
allow-item list before checking them. When a metalake or catalog is removed,
that modified policy was then sent back to Ranger.
- It now inspects the four lists without changing anything.
**Timestamp format sent to Ranger Admin**
- The Ranger 2.8 client writes `createTime`/`updateTime` as
`yyyyMMdd-HH:mm:ss.SSS-Z`. Ranger Admin 2.4 rejects that with HTTP 400.
Reproduced against a real Admin 2.4: update a policy fetched by id.
- Ranger Admin sets both fields itself on create and update, so
`RangerClientExtension.createPolicy`/`updatePolicy` now send the policy without
them. The caller's object is restored afterwards.
- I verified against Admin 2.4 that `createTime` is preserved and
`updateTime` advances.
**Dependencies**
Only the Ranger Admin REST client and the model classes are needed. Changes
in `authorization-ranger`:
| Change | Reason |
|---|---|
| Exclude `ranger-audit-core`, `ugsync-util`, `org.graalvm.js`,
`com.nimbusds`, `net.minidev`, `javax.servlet` | In-process plugin, audit,
user-sync and script-engine side, which Gravitino never runs |
| Exclude `org.apache.hadoop` | Ranger 2.8 wants Hadoop 3.4.2; keeps the
shared `hadoop-client-api`/`-runtime` 3.3.6 |
| Exclude Jackson 2.17 and commons-lang3 3.19 | Use the versions on the
server classpath, which the isolated class loader delegates to; Jackson is
`compileOnly` for the annotations |
| `commons-lang` 2 imports moved to `commons-lang3` | `commons-lang` 2.6 is
no longer pulled |
The same excludes are applied to the test-only `ranger-intg` dependencies in
`authorization-chain` and `integration-test-common`. Separately, `ugsync-util`
is added as `testRuntimeOnly` in `authorization-ranger` and
`authorization-chain`, because the Kyuubi Spark authz plugin in the E2E tests
runs the Ranger 2.8 policy engine.
**Resulting `authorizations/ranger/libs` changes**
- Removed: `ranger-intg-2.4.0`, `ranger-plugins-common-2.4.0`,
`jackson-core-asl`, `jackson-mapper-asl` and `jackson-jaxrs` 1.9.13 (Jackson
1), `commons-lang-2.6`, `jersey-bundle-1.19.3`.
- Added: `ranger-intg-2.8.0`, `ranger-plugins-common-2.8.0`,
`ranger-authz-api-2.8.0`, `jersey-bundle-1.19.4`.
**Behaviour that is unchanged**
Kerberos authentication to Ranger Admin behaves as before. `RangerClient`
needs `MiscUtil` for it. That class was in the excluded `ranger-plugins-audit`
in 2.4 and is in the excluded `ranger-audit-core` in 2.8, so it is not shipped
in either case.
**New tests**
- `TestRangerReferenceJson`: request JSON for users and groups is identical
to what Ranger 2.4 sent. Responses with unknown fields and dates parse through
the exact Ranger 2.8 code paths.
- `TestRangerClientExtension`: the user, group and policy APIs run over HTTP
against a stub Ranger Admin. Policy requests omit the server-managed timestamps
and leave the caller's object unchanged. A 404 maps to `NOT_FOUND`.
- `TestRangerImmutableModelDefaults`: policy building is parameterized over
both null-safe supplier modes. The test restores the exact previous JVM-wide
supplier afterwards. With one of the fixes reverted, it fails in `v2` mode.
### Why are the changes needed?
A scan of the server distribution built from `main` (`distribution/package`
and `distribution/package-all`) reports CRITICAL findings for Avro 1.7.7 shaded
in `hadoop-client-runtime-3.3.6.jar`, `ranger-plugins-common` 2.4.0 and
ZooKeeper 3.6.3. None of them can be fixed with a same-line patch bump. The
Netty `netty-handler` findings are handled separately in the companion
same-line update.
Fixes #14435
### Does this PR introduce _any_ user-facing change?
No API, property or configuration changes.
- **Distribution content:** the library changes listed above.
`hadoop-client-runtime-3.3.6.jar` in the five affected `libs` directories no
longer contains the shaded Avro, so its checksum differs from the Maven Central
artifact. The Maven artifacts and the `bundles/*` jars are unchanged.
- **Ranger:** the plugin now uses the Ranger 2.8 client. Tested against
Ranger Admin 2.4.0 (the CI image) and 2.8.0 (official image).
- **Hadoop:** the Hadoop APIs that depend on the shaded Avro
(`SequenceFile`, `MapFile`, Avro serialization) are not available in these
class loaders. Gravitino does not use them.
### How was this patch tested?
**Vulnerability scan**
`trivy rootfs --severity HIGH,CRITICAL distribution/package` before and
after:
| | CRITICAL | HIGH |
|---|---|---|
| Before | 12 findings (6 distinct) | 404 |
| After | 1 (`netty-handler` 4.1.118 in `catalogs/glue/libs`, tracked by the
companion ticket) | 382 |
No new findings.
**Static checks**
- `./gradlew build -x test` passes, including web, rat, spotless, errorprone
and javadoc.
- A bytecode linkage check (every method and field reference made by classes
reachable from the plugin) on the shipped classpath, server `libs` plus
`authorizations/ranger/libs`: 14245 references, 0 unresolved.
- The same check for the Hive metastore libs gives identical unresolved sets
with ZooKeeper 3.6.3 and 3.8.8.
- With and without the Avro transform, the resolved classpaths of all
affected configurations contain the same files.
**Unit tests**
`./gradlew test -PskipITs`: 15081 tests in 70 modules pass. The only
failure, `TestHTTPClient.testUnresolvableHostThrowsMeaningfulException`, also
fails on `main` in my environment, because the local resolver answers
`.invalid` hosts.
**Docker integration tests, embedded and deploy modes (deploy with
`-PjdbcBackend=mysql`), all passing**
| Module | Tests | Covers |
|---|---|---|
| `authorization-ranger` | 108 | Hive, HDFS, fileset, Iceberg, Paimon E2E |
| `authorization-chain` | 18 | |
| `catalog-hive` | 212 | Hive 2.3, Hive 3.1, Kerberos, S3, views |
| `catalog-fileset` | 184 | HDFS, Kerberos, impersonation, S3 |
| `catalog-lakehouse-iceberg` | 102 | Hive, Kerberos Hive, REST, JDBC |
| `iceberg-rest-server` | 457 | Hive, JDBC, Kerberos, token authorization |
**Ranger Admin compatibility**
A probe on the shipped classpath called every Ranger Admin API the plugin
uses against a real Ranger Admin 2.8.0 (official `apache/ranger` images): 21
checks covering services, users, groups, roles, grant/revoke, and policy
create/get/find/update/rename/delete. Against Admin 2.4.0, the user/group APIs
and policy create/get/update were verified directly, and the full plugin flow
is covered by the ITs above.
**Not covered**
Ranger Admin 2.5 to 2.7 were not tested directly. They share the same public
v2 REST API and the same Admin-side JSON provider as 2.4 and 2.8.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
