[
https://issues.apache.org/jira/browse/HADOOP-19972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jose Luis López reassigned HADOOP-19972:
----------------------------------------
Hadoop Flags: Incompatible change
Target Version/s: 3.6.0
Assignee: Jose Luis López
> Upgrade to Jetty 12 ee8
> -----------------------
>
> Key: HADOOP-19972
> URL: https://issues.apache.org/jira/browse/HADOOP-19972
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build, common
> Reporter: Jose Luis López
> Assignee: Jose Luis López
> Priority: Major
> Labels: pull-request-available
>
> Move Hadoop's web servers from Jetty 9.4.58 to Jetty 12.1.14 on the ee8
> environment, keeping {{javax.servlet}}. This issue is the port only:
> everything that also works on Jetty 9.4 lands first, in the issues under
> *Depends on*.
> *Scope*:
> * hadoop-project/pom.xml: set jetty.version to 12.1.14. Replace the managed
> jetty-servlet, jetty-webapp, javax-websocket-server-impl and websocket-client
> with jetty-ee8-nested, jetty-ee8-servlet, jetty-ee8-webapp,
> jetty-ee8-websocket-jetty-server (excluding jetty-ee8-annotations, which
> brings the banned asm) and jetty-ee8-websocket-jetty-client. Drop
> jetty-servlet-tester, which nothing uses.
> * Rename the Jetty entries in the module poms: hadoop-auth, hadoop-common,
> hadoop-kms, hadoop-hdfs-httpfs, hadoop-yarn-client, hadoop-yarn-common,
> hadoop-yarn-services-api, the ApplicationHistoryServer, the NodeManager, the
> ResourceManager, the Router and timelineservice-hbase-tests. hadoop-sls also
> declares jetty-ee8-nested, and the NodeManager declares the WebSocket client
> for its tests.
> * hadoop-client-runtime and hadoop-client-minicluster: replace the Jetty 9.4
> coordinates in the exclusions and filters with the ee8 and
> jetty-websocket-core ones. The jetty-util exclusions of HADOOP-20014 stay.
> * LICENSE-binary and NOTICE-binary: the Jetty 12.1.14 artifacts in place of
> the 9.4 ones; drop javax.websocket-api.
> * HttpServer2: ee8 classes; Handler.Sequence in place of HandlerCollection;
> Server#setRequestLog in place of RequestLogHandler; a core handler for
> addHandlerAtFront; a /logs context that starts when hadoop.log.dir is
> missing; a connector that can reopen a closed listener; the two settings
> below.
> * HttpServer2Metrics: read Jetty 12's handle counters, convert nanoseconds to
> milliseconds, publish the five async metrics as 0.
> * Import changes to ee8 in the common servlets, McpHttpServer,
> ApplicationHistoryServer, ResourceManager, Router, ApiServerWebApp, SLSWebApp
> and the tests.
> * WebSocket: the client and NodeManager move to the ee8 WebSocket API.
> WebApps.Builder gets withServerConfigurer, so the NodeManager can install the
> WebSocket components before its context starts.
> * The YARN application catalog, only if it is still in the tree: exclude the
> ee8 artifacts from its four Hadoop dependencies, so its war keeps Jetty 9.4.
> * Tests: TestHttpServer, TestHttpServer2Metrics, TestSSLHttpServer,
> TestPseudoAuthenticator, TestCheckpoint and TestNMContainerWebSocket follow
> Jetty 12's behaviour.
> *Why*: Jetty 9.4 is end of life and its CVEs get no open-source fix
> (HADOOP-19910). Jetty 10 and 11 are end of life too, and Jetty 11 and later
> otherwise require {{jakarta.servlet}}. Jetty 12's ee8 environment still
> serves {{javax.servlet}}, so Hadoop reaches a supported Jetty without the
> namespace break. ee8 is a stage, not the destination: the port to Jetty 12's
> API happens here once, and HADOOP-19395 is then only the namespace change.
> *Incompatible changes*. The reason phrase alone makes this incompatible. This
> is the release note:
> # *No custom HTTP reason phrase.* The phrase is always the standard text for
> the status: "Unauthorized", not "Authentication required". Since
> HADOOP-20003, HADOOP-20004 and HADOOP-20005 the message is also in the
> response body, and this release's clients read it there. A client that reads
> only the phrase loses the message, and an older KMS client no longer
> re-authenticates on a 403 from the KMS, because it matches "Invalid
> signature" and "Anonymous requests are disallowed" against the phrase.
> # *Stricter URI parsing.* {{%2F}}, {{%2E}} segments and malformed encodings
> are refused with 400. {{hadoop.http.uri.compliance.violations}} controls what
> is allowed through; adding {{AMBIGUOUS_PATH_SEPARATOR}} accepts {{%2F}}
> again. The default keeps paths that Hadoop needs, and
> {{SUSPICIOUS_PATH_CHARACTERS}} in it also admits the encoded control
> characters {{%00}}, {{%09}}, {{%0A}}, {{%0D}} and {{%7F}}, as Jetty 9.4 did.
> # *TLS renegotiation is refused.* A client's TLS 1.2 renegotiation closes the
> connection. {{hadoop.http.ssl.renegotiation.allowed=true}} restores 9.4's
> behaviour.
> # *Metrics.* The five async HTTP metrics read 0, because Jetty 12 cannot
> count them. The {{dispatched*}} metrics now read Jetty 12's handle counters.
> All names, and the milliseconds, are unchanged.
> # *A response shorter than its Content-Length fails in the server.* The
> servlet's write fails instead of a truncated body reaching the client, so the
> error is reported on the other side: a short fsimage transfer now fails on
> the NameNode.
> # *Jetty's own output differs:* the error-page markup, {{charset=utf-8}} and
> {{Vary}} on static files, a {{Date}} header on error responses, the default
> acceptor count, and a context path without its trailing slash answering 301
> with a relative {{Location}} where 9.4 sent 302 with an absolute one.
> # *Internal API and dependencies.* {{InterfaceAudience.Private}} and
> {{LimitedPrivate}} classes that name Jetty types now name the ee8 ones:
> {{HttpServer2}} ({{addContext}}, {{defineFilter}}, {{getWebAppContext}},
> {{addDefaultApps}}), {{WebApps}} and {{ContainerShellWebSocket}}.
> {{addHandlerAtFront}} takes a core handler, so an ee8 context goes through
> {{getCoreContextHandler()}}. Code calling these must be rebuilt. Jetty's
> artifacts move to {{org.eclipse.jetty.ee8}}, so downstream exclusions need
> the new coordinates. Public {{javax.servlet}} extension points do not change;
> TestDownstreamServletCompatibility pins them.
> Everything else keeps 9.4's behaviour: {{/static}} lists no directory,
> {{/logs}} keeps its admin check when the log directory is missing,
> {{web.xml}} error pages work, a stopped listener can be reopened, the
> NodeManager's WebSocket upgrade works, and a {{Location}} set through
> {{sendRedirect}} stays absolute.
> *New settings*:
> ||Setting||Default||
> |{{hadoop.http.uri.compliance.violations}}|{{AMBIGUOUS_EMPTY_SEGMENT,AMBIGUOUS_PATH_ENCODING,SUSPICIOUS_PATH_CHARACTERS}}|
> |{{hadoop.http.ssl.renegotiation.allowed}}|{{false}}|
> *Downstream*:
> * Projects that use only the public API and {{javax.servlet}}: no rebuild, no
> namespace change.
> * Non-shaded artifacts: Jetty 12 and {{org.eclipse.jetty.ee8}} jars replace
> Jetty 9.4's on the classpath. A project pinning Jetty 9.4 next to Hadoop can
> no longer share one Jetty release with it.
> * hadoop-client-runtime and hadoop-client-minicluster: the relocated Jetty is
> 12. Relocated classes are not API.
> * The application catalog war keeps Jetty 9.4 through {{solr.jetty.version}}
> until HADOOP-19998, if HADOOP-20008 landed. If HADOOP-19998 landed first, the
> module is gone and no Jetty 9.4 artifact is left.
> *Depends on*: HADOOP-20012, HADOOP-20013, HADOOP-20014, HADOOP-20015,
> HADOOP-20016, HADOOP-19970, HADOOP-20003, HADOOP-20004, HADOOP-20005,
> HADOOP-20006, HADOOP-20007, and HADOOP-20008 or HADOOP-19998.
> *Not in scope*: {{jakarta.servlet}}, Jersey upgrades and ee9, ee10 or ee11:
> HADOOP-19395. Guice 6 (HADOOP-19999) and jsr311-api (HADOOP-20000) are not
> needed by this issue. HADOOP-19971 belongs to HADOOP-19395, and this issue
> does not depend on it. Requires Java 17, so it cannot go to branch-3.3 or
> branch-3.4.
> *CI*: touches hadoop-project, so a full build, including
> hadoop-client-check-invariants, hadoop-client-check-test-invariants and
> hadoop-client-integration-tests.
> *PR*: https://github.com/apache/hadoop/pull/8704 (draft, rebuilt on trunk and
> stacked on its prerequisites; review the last 4 commits)
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
