yandrey321 opened a new pull request, #11367:
URL: https://github.com/apache/ozone/pull/11367
## What changes were proposed in this pull request?
This patch upgrades Ozone from Jetty `9.4.58.v20250814` to Jetty `12.0.38`,
targeting the
**EE8** environment so that the `javax.servlet` namespace is retained.
### Why
Jetty 9.x is out of community support and 9.4.x will never support JDK 17.
Ozone server
components already require JDK 17 (HDDS-14439). Staying on 9.4 blocks
HDDS-8249 / HDDS-8250
(JDK 17 crash fixes) and HDDS-16121 (Jetty `VirtualThreadPool`).
Jetty 12's core no longer bundles a Servlet API. Servlet support moved into
per-environment
modules: `jetty-ee8-*` (`javax.servlet` 4.0), `jetty-ee9-*`
(`jakarta.servlet` 5) and
`jetty-ee10-*` (`jakarta.servlet` 6).
### Why EE8 rather than EE10
Ozone depends on Hadoop 3.4.3, which is still `javax.servlet` 3.1. Targeting
EE10 forces the
entire web stack to `jakarta.*`: a `javax`→`jakarta` filter/listener bridge,
forks of several
hadoop-common servlet classes, and moving Jersey, Weld, CDI, Guice and JAXB
to their jakarta
lines. An earlier EE10 attempt on this Jira came to **357 changed files**,
of which 265 were
pure `javax.`→`jakarta.` import churn.
EE8 reaches the same endpoint — one Jetty major and one `javax.servlet`
provider on the
classpath — in **48 files**. Jersey 2.48, Weld 3.x, Guice 6, CDI 2, JAXB
2.3.x and every
`web.xml` / `beans.xml` descriptor namespace are untouched.
### Dependency shape
| Item | From | To |
|---|---|---|
| `jetty.version` | `9.4.58.v20250814` | `12.0.38` |
| BOMs | `jetty-bom` | `jetty-bom` + `org.eclipse.jetty.ee8:jetty-ee8-bom` |
| `servlet-api.version` | `3.1.0` (`javax.servlet:javax.servlet-api`) |
`4.0.9` (`org.eclipse.jetty.toolchain:jetty-servlet-api`) |
| `jetty-servlet` / `-webapp` / `-proxy` | `org.eclipse.jetty` |
`org.eclipse.jetty.ee8:jetty-ee8-*` |
| `jetty-http` / `-server` / `-util` / `-client` | unchanged | unchanged
(Jetty 12 core) |
| Jersey, Weld, Guice, CDI, HK2, JAXB, `jakarta.*` | — | **unchanged** |
`org.eclipse.jetty.toolchain:jetty-servlet-api` becomes the single
`javax.servlet` provider and
`javax.servlet:javax.servlet-api` is dropped. Only the toolchain jar ships
the
`javax/servlet/catalog.xml` + XSD tree that `jetty-ee8-webapp`'s
`WebDescriptor.__nonValidatingStaticParser` — a **static** field — resolves
against; a missing
catalog entry throws rather than being tolerated, so keeping both jars risks
a fatal
class-initialization failure.
Both managed `hadoop-common` entries (jar and test-jar) now exclude
`javax.servlet:javax.servlet-api`
and `org.eclipse.jetty:*`. The Jetty exclusion is mandatory independently of
the EE choice:
`jetty-servlet` and `jetty-webapp` do not exist at 12.x, so `jetty-bom`
silently stops managing
them and Maven would otherwise fall back to Hadoop's 9.4.x.
### Notable behavioral changes
**Admin servlets.** hadoop-common's `org.apache.hadoop.conf.ConfServlet` and
`org.apache.hadoop.jmx.JMXJsonServlet` both route their admin-access check
through
`org.apache.hadoop.http.HttpServer2`, which references
`org.eclipse.jetty.servlet.ServletContextHandler`,
`org.eclipse.jetty.webapp.WebAppContext` and `FilterHolder`/`FilterMapping`
— all removed or moved
in Jetty 12. Those classes cannot load at all on a Jetty 12 classpath, so
`/jmx` and `/conf` would
answer 500 with a `NoClassDefFoundError`. This is unrelated to the EE
environment. `/conf` is
switched to Ozone's existing `HddsConfServlet`, and a new
`HddsJMXJsonServlet` covers `/jmx`.
**URI compliance.** Jetty 12 rejects ambiguous URIs (empty `//` segments,
`%25`, encoded path
separators, suspicious path characters) that 9.4 passed through. S3 object
keys and WebHDFS paths
need them. A new opt-in `Builder.allowAmbiguousUri(boolean)` installs an
`OZONE` `UriCompliance`
mode allowing exactly those four violations — and, unlike Jetty's `LEGACY`
mode, **not**
`%2e%2e` path traversal, UTF-16 `%u` encodings, truncated UTF-8 or userinfo.
S3 Gateway and
httpfs opt in; every other endpoint keeps Jetty's strict default.
**XML content type.** EE8's `Response.setContentType` appends `;charset=`
once `_encodingFrom`
has moved past `NOT_SET`, and `QuotingInputFilter` promotes it on every
request, whereas 9.4
reset it — so a bare `application/xml` would ship as
`application/xml;charset=utf-8`. A new
`S3ContentTypeFilter` resets the encoding to `NOT_SET` to preserve the S3
wire format.
**Reason phrases.** Jetty 12 discards custom HTTP reason phrases at the core
level, so
`S3SecretManagementEndpoint` moves its error code into a `text/plain` entity
body.
**TLS SNI.** Jetty 12 defaults `SecureRequestCustomizer#sniHostCheck` to
true, which breaks
Ozone's hostname-agnostic TLS endpoints; it is constructed with `false`.
**Base resources.** Core `ContextHandler.doStart()` now throws for a missing
base directory,
where 9.4 tolerated it. The log directory is created best-effort and `/logs`
is skipped rather
than failing startup if unavailable; `/static` is served only when its base
resource exists.
**Weld / CDI.** Weld 3.1.9's `JettyLegacyContainer` detects on
`org.eclipse.jetty.util.Decorator`,
which still exists in jetty-util 12, so it is selected — then
`LegacyWeldDecorator.process()`
touches the deleted `org.eclipse.jetty.servlet.ServletContextHandler` and
throws
`NoClassDefFoundError`. Because that is an `Error`, Weld's `catch
(Exception)` does not catch it
and S3 Gateway context startup fails. Fixed with Jetty's canonical
`jetty-ee8-cdi` module and a
`CdiDecoratingListener`, which makes Weld's probe match `JettyContainer`
first.
The listener is installed on each of the three S3 Gateway contexts that
register the Weld listener
in their descriptor, guarded by `isEnabled()`: `BaseHttpServer` builds no
`HttpServer2` at all for a
disabled server, and `ozone.s3g.sts.http.enabled` is `false` by default, so
an unguarded call would
NPE and kill the whole gateway at startup. `BaseHttpServer.isEnabled()` is
widened from private to
protected for that check.
**httpfs descriptors.** `<url-pattern>*</url-pattern>` is illegal per the
Servlet spec and is
corrected to `/*` (5 filter-mappings in each of the two descriptors).
**hadoop-kms removed from integration-test.** `MiniKMS` embeds an unshaded
Jetty 9 `Server` and
cannot coexist with a Jetty 12 runtime. `TestOzoneAtRestEncryption` and
`TestOzoneShellHA` now
pre-seed a `JavaKeyStoreProvider` jceks before cluster start instead. The
native secure-KMS path
is followed up in HDDS-16424.
### New dist artifacts
Beyond the expected EE8 set, the `jetty-ee8-cdi` choice pulls in
`jetty-ee8-annotations`,
`jetty-ee8-plus`, `jetty-plus`, `jetty-jndi` and — worth calling out
explicitly —
**`jakarta.transaction:jakarta.transaction-api`, a new third-party artifact
in the
distribution** (1.3.3, whose artifact name is literally "javax.transaction
API"; EPL-2.0 +
GPL2-w/-CPE, the same pairing as the `jakarta.annotation-api` already
shipped).
`jar-report.txt` and `bin/LICENSE.txt` are updated accordingly:
`javax.servlet-api.jar`,
`jetty-servlet.jar` and `jetty-webapp.jar` are removed;
`jetty-servlet-api.jar`, `jetty-ee.jar`,
`jetty-session.jar`, `jetty-jndi.jar`, `jetty-plus.jar`,
`jakarta.transaction-api.jar` and the
seven `jetty-ee8-*.jar` are added. `license.sh` reports no disallowed
licenses and needs no
`license.exceptions` entry.
### Known residuals (pre-existing, out of scope)
- `org.eclipse.jetty.websocket:websocket-{api,client,common}:9.4.57` remain
at **`test`** scope
under `ozone-integration-test`. Not on any runtime or dist classpath.
- `javax.servlet:servlet-api:2.5` reaches `ozone-filesystem-hadoop2` at
**`provided`** scope via
`hadoop-common:2.10.2`. This is the old Servlet 2.5 coordinate from the
Hadoop 2.x
compatibility module, absent from the distribution.
Generated-by: Claude Code (claude-opus-5)
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-8280
## How was this patch tested?
CI:
https://github.com/yandrey321/ozone/actions/runs/36659484942/job/109713286666
Unit tests:
- `hdds-server-framework`: `TestHttpServer2` (16), `TestHttpServer2SSL`
(10), `TestBaseHttpServer` (7),
`TestHttpServer2Metrics`, `TestHtmlQuoting`,
`TestPrometheusServletAuthorization` — 43 tests, green.
New coverage: URI-compliance admission at the wire level (including that
`%2e%2e` and `%u` stay
rejected under the relaxed mode), the base-resource/log-dir guards, and a
`testDefaultServletsAnswer` that drives `/jmx`, `/conf`, `/stacks` and
`/logLevel` over HTTP.
- `ozone-s3gateway`: `TestS3ContentTypeFilter` (7, new),
`TestS3GatewayHttpServers` (2, new),
`TestGateway`, `TestSecretGenerate`, `TestSecretRevoke` — 17 tests, green.
The new class asserts
that an enabled context carries Jetty's `org.eclipse.jetty.cdi` attribute
and that a disabled STS
server is constructed without one.
Integration tests:
- `TestOzoneManagerHttpServer`, `TestEDEKCacheLoader` (new) — green.
- `TestCpuMetrics`, `TestOzoneManagerRestInterface` — green.
- `ProxyServerIntegrationTest` — green.
- `TestS3SDK` (AWS SDK v1 + v2 against a live S3 Gateway) — 210 tests, green.
- `TestS3GatewayHealthCheck` — 3 tests, green; starts a real `Gateway`, so
it covers the CDI
install path end to end in-process.
- `TestOzoneAtRestEncryption` — 36 tests green.
- `TestOzoneShellHA` — green.
Build and checks: full reactor build and a wider `-Pdist` build (all 58
modules, shade + Recon +
docs enabled) with `dependency:analyze-only` **enabled** — no "Used
undeclared", "Unused declared"
or "Non-test scoped" findings. `checkstyle.sh` clean, `rat.sh` clean, no
`@author` tags.
`dependency.sh`'s jar-report comparison is empty against a clean `-Pdist`
build.
Classpath assertions over the reactor: every `org.eclipse.jetty*` coordinate
resolves to `12.0.38`
(`4.0.9` for `jetty-servlet-api`); no `ee9`/`ee10`;
`org.eclipse.jetty:jetty-servlet`,
`jetty-webapp` and `jetty-proxy` resolve to nothing.
Compose robot smoketests, against a dist built from this branch: `ozone`
(369 assertions, 18
suites) and `ozonesecure` (450 assertions) both pass with zero failures,
covering Recon, httpfs, the
S3 and secret suites and secure-mode SPNEGO.
`ozone-om-ha` ships `disabled-test.sh` upstream, so `ozone-ha` is run in its
place: 461 assertions
pass, 15 fail. All 15 trace to a single datanode container dying mid-run in
this local 13-container
environment, not to the upgrade — each failure is an
`AnnotatedNoRouteToHostException` against that
datanode's address, its container log stops abruptly four minutes before the
first failure with no
shutdown sequence or exception, and `ozone-ha` never stops a datanode
itself. The two multipart-upload
tests among them had already passed twice earlier in the same run under the
`OBJECT_STORE` and
`LEGACY` layouts. Re-running the affected `FILE_SYSTEM_OPTIMIZED` s3 block
against a fresh cluster
passes 149/149, including all 13 `Buckettagging` tests and both
multipart-upload tests.
The admin endpoints were also swept directly against a running cluster.
`/conf`, `/jmx`, `/prom`,
`/logLevel`, `/stacks` and `/logs/` answer 200 on OM, SCM, Recon and the S3
Gateway web admin
server, and on httpfs with `?user.name=`. `/static/` answers 403 because
directory listing stays
disabled, while a real asset under it (`/static/ozone.css`) answers 200. The
S3 Gateway returns
`Content-Type: application/xml` with no appended charset, confirming
`S3ContentTypeFilter` at the
wire level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
