[
https://issues.apache.org/jira/browse/HADOOP-19972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jose Luis López updated HADOOP-19972:
-------------------------------------
Description:
Move Hadoop's web servers from Jetty 9.4.58 to Jetty 12.1.14 on the ee8
environment, keeping {{javax.servlet}}. This issue is the port only: everything
that also works on Jetty 9.4 lands first, in the issues under *Depends on*.
*Scope*:
* hadoop-project/pom.xml: set jetty.version to 12.1.14. Replace the managed
jetty-servlet, jetty-webapp, javax-websocket-server-impl and websocket-client
with jetty-ee8-nested, jetty-ee8-servlet, jetty-ee8-webapp,
jetty-ee8-websocket-jetty-server (excluding jetty-ee8-annotations, which brings
the banned asm) and jetty-ee8-websocket-jetty-client. Drop
jetty-servlet-tester, which nothing uses.
* Rename the Jetty entries in the module poms: hadoop-auth, hadoop-common,
hadoop-kms, hadoop-hdfs-httpfs, hadoop-yarn-client, hadoop-yarn-common,
hadoop-yarn-services-api, the ApplicationHistoryServer, the NodeManager, the
ResourceManager, the Router and timelineservice-hbase-tests. hadoop-sls also
declares jetty-ee8-nested, and the NodeManager declares the WebSocket client
for its tests.
* hadoop-client-runtime and hadoop-client-minicluster: replace the Jetty 9.4
coordinates in the exclusions and filters with the ee8 and jetty-websocket-core
ones. The jetty-util exclusions of HADOOP-20014 stay.
* LICENSE-binary and NOTICE-binary: the Jetty 12.1.14 artifacts in place of the
9.4 ones; drop javax.websocket-api.
* HttpServer2: ee8 classes; Handler.Sequence in place of HandlerCollection;
Server#setRequestLog in place of RequestLogHandler; a core handler for
addHandlerAtFront; a /logs context that starts when hadoop.log.dir is missing;
a connector that can reopen a closed listener; the two settings below.
* HttpServer2Metrics: read Jetty 12's handle counters, convert nanoseconds to
milliseconds, publish the five async metrics as 0.
* Import changes to ee8 in the common servlets, McpHttpServer,
ApplicationHistoryServer, ResourceManager, Router, ApiServerWebApp, SLSWebApp
and the tests.
* WebSocket: the client and NodeManager move to the ee8 WebSocket API.
WebApps.Builder gets withServerConfigurer, so the NodeManager can install the
WebSocket components before its context starts.
* The YARN application catalog, only if it is still in the tree: exclude the
ee8 artifacts from its four Hadoop dependencies, so its war keeps Jetty 9.4.
* Tests: TestHttpServer, TestHttpServer2Metrics, TestSSLHttpServer,
TestPseudoAuthenticator, TestCheckpoint and TestNMContainerWebSocket follow
Jetty 12's behaviour.
*Why*: Jetty 9.4 is end of life and its CVEs get no open-source fix
(HADOOP-19910). Jetty 10 and 11 are end of life too, and Jetty 11 and later
otherwise require {{jakarta.servlet}}. Jetty 12's ee8 environment still serves
{{javax.servlet}}, so Hadoop reaches a supported Jetty without the namespace
break. ee8 is a stage, not the destination: the port to Jetty 12's API happens
here once, and HADOOP-19395 is then only the namespace change.
*Incompatible changes*. The reason phrase alone makes this incompatible. This
is the release note:
# *No custom HTTP reason phrase.* The phrase is always the standard text for
the status: "Unauthorized", not "Authentication required". Since HADOOP-20003,
HADOOP-20004 and HADOOP-20005 the message is also in the response body, and
this release's clients read it there. A client that reads only the phrase loses
the message, and an older KMS client no longer re-authenticates on a 403 from
the KMS, because it matches "Invalid signature" and "Anonymous requests are
disallowed" against the phrase.
# *Stricter URI parsing.* {{%2F}}, {{%2E}} segments and malformed encodings are
refused with 400. {{hadoop.http.uri.compliance.violations}} controls what is
allowed through; adding {{AMBIGUOUS_PATH_SEPARATOR}} accepts {{%2F}} again. The
default keeps paths that Hadoop needs, and {{SUSPICIOUS_PATH_CHARACTERS}} in it
also admits the encoded control characters {{%00}}, {{%09}}, {{%0A}}, {{%0D}}
and {{%7F}}, as Jetty 9.4 did.
# *TLS renegotiation is refused.* A client's TLS 1.2 renegotiation closes the
connection. {{hadoop.http.ssl.renegotiation.allowed=true}} restores 9.4's
behaviour.
# *Metrics.* The five async HTTP metrics read 0, because Jetty 12 cannot count
them. The {{dispatched*}} metrics now read Jetty 12's handle counters. All
names, and the milliseconds, are unchanged.
# *A response shorter than its Content-Length fails in the server.* The
servlet's write fails instead of a truncated body reaching the client, so the
error is reported on the other side: a short fsimage transfer now fails on the
NameNode.
# *Jetty's own output differs:* the error-page markup, {{charset=utf-8}} and
{{Vary}} on static files, a {{Date}} header on error responses, the default
acceptor count, and a context path without its trailing slash answering 301
with a relative {{Location}} where 9.4 sent 302 with an absolute one.
# *Internal API and dependencies.* {{InterfaceAudience.Private}} and
{{LimitedPrivate}} classes that name Jetty types now name the ee8 ones:
{{HttpServer2}} ({{addContext}}, {{defineFilter}}, {{getWebAppContext}},
{{addDefaultApps}}), {{WebApps}} and {{ContainerShellWebSocket}}.
{{addHandlerAtFront}} takes a core handler, so an ee8 context goes through
{{getCoreContextHandler()}}. Code calling these must be rebuilt. Jetty's
artifacts move to {{org.eclipse.jetty.ee8}}, so downstream exclusions need the
new coordinates. Public {{javax.servlet}} extension points do not change;
TestDownstreamServletCompatibility pins them.
Everything else keeps 9.4's behaviour: {{/static}} lists no directory,
{{/logs}} keeps its admin check when the log directory is missing, {{web.xml}}
error pages work, a stopped listener can be reopened, the NodeManager's
WebSocket upgrade works, and a {{Location}} set through {{sendRedirect}} stays
absolute.
*New settings*:
||Setting||Default||
|{{hadoop.http.uri.compliance.violations}}|{{AMBIGUOUS_EMPTY_SEGMENT,AMBIGUOUS_PATH_ENCODING,SUSPICIOUS_PATH_CHARACTERS}}|
|{{hadoop.http.ssl.renegotiation.allowed}}|{{false}}|
*Downstream*:
* Projects that use only the public API and {{javax.servlet}}: no rebuild, no
namespace change.
* Non-shaded artifacts: Jetty 12 and {{org.eclipse.jetty.ee8}} jars replace
Jetty 9.4's on the classpath. A project pinning Jetty 9.4 next to Hadoop can no
longer share one Jetty release with it.
* hadoop-client-runtime and hadoop-client-minicluster: the relocated Jetty is
12. Relocated classes are not API.
* The application catalog war keeps Jetty 9.4 through {{solr.jetty.version}}
until HADOOP-19998, if HADOOP-20008 landed. If HADOOP-19998 landed first, the
module is gone and no Jetty 9.4 artifact is left.
*Depends on*: HADOOP-20012, HADOOP-20013, HADOOP-20014, HADOOP-20015,
HADOOP-20016, HADOOP-19970, HADOOP-20003, HADOOP-20004, HADOOP-20005,
HADOOP-20006, HADOOP-20007, and HADOOP-20008 or HADOOP-19998.
*Not in scope*: {{jakarta.servlet}}, Jersey upgrades and ee9, ee10 or ee11:
HADOOP-19395. Guice 6 (HADOOP-19999) and jsr311-api (HADOOP-20000) are not
needed by this issue. HADOOP-19971 belongs to HADOOP-19395, and this issue does
not depend on it. Requires Java 17, so it cannot go to branch-3.3 or branch-3.4.
*CI*: touches hadoop-project, so a full build, including
hadoop-client-check-invariants, hadoop-client-check-test-invariants and
hadoop-client-integration-tests.
*PR*: to be rebuilt on trunk; the draft is
https://github.com/apache/hadoop/pull/8704
was:
Hadoop's web server is Jetty 9.4, which is end of life and gets no security
fixes. Jetty 10 and later require the {{jakarta.servlet}} namespace, which
would break every project that embeds Hadoop's web stack. Jetty 12's ee8
environment still serves {{{}javax.servlet{}}}, so this moves to a supported
Jetty without that break.
Jetty 9.4.58 to 12.1.12 on ee8. No public signature changes, and downstream
projects need not rebuild. {*}Requires Java 17{*}, so it cannot go to
branch-3.3 or branch-3.4. The YARN application catalog webapp stays on Jetty
9.4.58, which Solr 8 needs, but we make it independent, so we can migrate it in
another independent migration action or retire the app.
h2. ee8 is a staging post
ee8 is a stage, not the destination. The point of landing it is that the port
to Jetty 12's API happens here and is not repeated: afterwards Hadoop is on a
supported Jetty, and what remains is the namespace change on its own — ee10 or
ee11 and {{{}jakarta.servlet{}}}, under HADOOP-19395 — instead of a Jetty
upgrade and an API break having to be taken together. ee8 does carry a
compatibility layer that adapts every request, so it is a place to stand for a
release or two rather than indefinitely; unlike Jetty 10 and 11 it has a
supported vendor behind it, which is what makes standing there reasonable.
h2. Incompatible changes
This is the release note. The reason phrase alone makes this incompatible.
# *No custom HTTP reason phrase is sent.* Refusals from the authentication,
KMS and CSRF filters and from image transfer carry their message in the
response body instead, PUT and DELETE included, and Hadoop's clients read it
there. An older client sees the standard phrase: "Unauthorized" rather than
"Authentication required".
# *Stricter URI parsing.* {{{}%2F{}}}, {{%2E}} segments and malformed
encodings are refused with 400. {{hadoop.http.uri.compliance.violations}}
controls what is allowed through.
# *TLS renegotiation is refused.* A client's TLS 1.2 renegotiation closes the
connection. {{hadoop.http.ssl.renegotiation.allowed=true}} restores 9.4's
behaviour.
# *The five async HTTP metrics read 0.* The names and the metric set are
unchanged.
# *Jetty's own output differs:* the error-page markup, charset and {{Vary}} on
static files, a {{Date}} header on error responses, the default acceptor count,
and a context path without its trailing slash redirecting 301 with a relative
{{Location}} where 9.4 sent 302 with an absolute one.
Everything else keeps 9.4's behaviour: {{/static}} lists no directory,
{{/logs}} keeps its admin check when the log directory is missing, {{web.xml}}
error pages work, the HTTP server metrics stay in milliseconds under their
existing names, a stopped listener can be reopened, the NodeManager's web
socket upgrade works, and a redirect's {{Location}} stays absolute.
h2. New settings
||Setting||Default||
|{{hadoop.http.uri.compliance.violations}}|{{AMBIGUOUS_EMPTY_SEGMENT,AMBIGUOUS_PATH_ENCODING,SUSPICIOUS_PATH_CHARACTERS}}|
|{{hadoop.http.ssl.renegotiation.allowed}}|{{false}}|
Adding {{AMBIGUOUS_PATH_SEPARATOR}} to the first accepts {{%2F}} again.
h2. Split out
To keep this issue to the Jetty port itself, the parts that also work on Jetty
9.4 land first, each on its own:
* HADOOP-20003: clients read a refusal's reason from the body
* HADOOP-20004: filters send the reason in the body for every HTTP method
* HADOOP-20005: refused fsimage transfers carry their reason in the body
* HADOOP-20006: {{JettyUtils#clearContentType}}
* HADOOP-20007: test for the downstream javax.servlet contract
* HADOOP-20008: the app catalog's own Jetty version
This issue is blocked by all six. The incompatible changes listed above still
apply; after the split, the only cause of the reason-phrase change is Jetty 12
itself. The app catalog stays on Jetty 9.4 through {{solr.jetty.version}} until
HADOOP-19998.
h2. Not in scope
No move to {{{}jakarta.servlet{}}}, no Jersey upgrade, and no ee9, ee10 or
ee11. HADOOP-19395 lands the namespace change, in a major release;
HADOOP-19912, the umbrella for this move to Jetty 12 ee8, keeps
{{javax.servlet}}. This only stops that decision from blocking the security fix.
> Upgrade to Jetty 12 ee8
> -----------------------
>
> Key: HADOOP-19972
> URL: https://issues.apache.org/jira/browse/HADOOP-19972
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build, common
> Reporter: Jose Luis López
> Priority: Major
> Labels: pull-request-available
>
> Move Hadoop's web servers from Jetty 9.4.58 to Jetty 12.1.14 on the ee8
> environment, keeping {{javax.servlet}}. This issue is the port only:
> everything that also works on Jetty 9.4 lands first, in the issues under
> *Depends on*.
> *Scope*:
> * hadoop-project/pom.xml: set jetty.version to 12.1.14. Replace the managed
> jetty-servlet, jetty-webapp, javax-websocket-server-impl and websocket-client
> with jetty-ee8-nested, jetty-ee8-servlet, jetty-ee8-webapp,
> jetty-ee8-websocket-jetty-server (excluding jetty-ee8-annotations, which
> brings the banned asm) and jetty-ee8-websocket-jetty-client. Drop
> jetty-servlet-tester, which nothing uses.
> * Rename the Jetty entries in the module poms: hadoop-auth, hadoop-common,
> hadoop-kms, hadoop-hdfs-httpfs, hadoop-yarn-client, hadoop-yarn-common,
> hadoop-yarn-services-api, the ApplicationHistoryServer, the NodeManager, the
> ResourceManager, the Router and timelineservice-hbase-tests. hadoop-sls also
> declares jetty-ee8-nested, and the NodeManager declares the WebSocket client
> for its tests.
> * hadoop-client-runtime and hadoop-client-minicluster: replace the Jetty 9.4
> coordinates in the exclusions and filters with the ee8 and
> jetty-websocket-core ones. The jetty-util exclusions of HADOOP-20014 stay.
> * LICENSE-binary and NOTICE-binary: the Jetty 12.1.14 artifacts in place of
> the 9.4 ones; drop javax.websocket-api.
> * HttpServer2: ee8 classes; Handler.Sequence in place of HandlerCollection;
> Server#setRequestLog in place of RequestLogHandler; a core handler for
> addHandlerAtFront; a /logs context that starts when hadoop.log.dir is
> missing; a connector that can reopen a closed listener; the two settings
> below.
> * HttpServer2Metrics: read Jetty 12's handle counters, convert nanoseconds to
> milliseconds, publish the five async metrics as 0.
> * Import changes to ee8 in the common servlets, McpHttpServer,
> ApplicationHistoryServer, ResourceManager, Router, ApiServerWebApp, SLSWebApp
> and the tests.
> * WebSocket: the client and NodeManager move to the ee8 WebSocket API.
> WebApps.Builder gets withServerConfigurer, so the NodeManager can install the
> WebSocket components before its context starts.
> * The YARN application catalog, only if it is still in the tree: exclude the
> ee8 artifacts from its four Hadoop dependencies, so its war keeps Jetty 9.4.
> * Tests: TestHttpServer, TestHttpServer2Metrics, TestSSLHttpServer,
> TestPseudoAuthenticator, TestCheckpoint and TestNMContainerWebSocket follow
> Jetty 12's behaviour.
> *Why*: Jetty 9.4 is end of life and its CVEs get no open-source fix
> (HADOOP-19910). Jetty 10 and 11 are end of life too, and Jetty 11 and later
> otherwise require {{jakarta.servlet}}. Jetty 12's ee8 environment still
> serves {{javax.servlet}}, so Hadoop reaches a supported Jetty without the
> namespace break. ee8 is a stage, not the destination: the port to Jetty 12's
> API happens here once, and HADOOP-19395 is then only the namespace change.
> *Incompatible changes*. The reason phrase alone makes this incompatible. This
> is the release note:
> # *No custom HTTP reason phrase.* The phrase is always the standard text for
> the status: "Unauthorized", not "Authentication required". Since
> HADOOP-20003, HADOOP-20004 and HADOOP-20005 the message is also in the
> response body, and this release's clients read it there. A client that reads
> only the phrase loses the message, and an older KMS client no longer
> re-authenticates on a 403 from the KMS, because it matches "Invalid
> signature" and "Anonymous requests are disallowed" against the phrase.
> # *Stricter URI parsing.* {{%2F}}, {{%2E}} segments and malformed encodings
> are refused with 400. {{hadoop.http.uri.compliance.violations}} controls what
> is allowed through; adding {{AMBIGUOUS_PATH_SEPARATOR}} accepts {{%2F}}
> again. The default keeps paths that Hadoop needs, and
> {{SUSPICIOUS_PATH_CHARACTERS}} in it also admits the encoded control
> characters {{%00}}, {{%09}}, {{%0A}}, {{%0D}} and {{%7F}}, as Jetty 9.4 did.
> # *TLS renegotiation is refused.* A client's TLS 1.2 renegotiation closes the
> connection. {{hadoop.http.ssl.renegotiation.allowed=true}} restores 9.4's
> behaviour.
> # *Metrics.* The five async HTTP metrics read 0, because Jetty 12 cannot
> count them. The {{dispatched*}} metrics now read Jetty 12's handle counters.
> All names, and the milliseconds, are unchanged.
> # *A response shorter than its Content-Length fails in the server.* The
> servlet's write fails instead of a truncated body reaching the client, so the
> error is reported on the other side: a short fsimage transfer now fails on
> the NameNode.
> # *Jetty's own output differs:* the error-page markup, {{charset=utf-8}} and
> {{Vary}} on static files, a {{Date}} header on error responses, the default
> acceptor count, and a context path without its trailing slash answering 301
> with a relative {{Location}} where 9.4 sent 302 with an absolute one.
> # *Internal API and dependencies.* {{InterfaceAudience.Private}} and
> {{LimitedPrivate}} classes that name Jetty types now name the ee8 ones:
> {{HttpServer2}} ({{addContext}}, {{defineFilter}}, {{getWebAppContext}},
> {{addDefaultApps}}), {{WebApps}} and {{ContainerShellWebSocket}}.
> {{addHandlerAtFront}} takes a core handler, so an ee8 context goes through
> {{getCoreContextHandler()}}. Code calling these must be rebuilt. Jetty's
> artifacts move to {{org.eclipse.jetty.ee8}}, so downstream exclusions need
> the new coordinates. Public {{javax.servlet}} extension points do not change;
> TestDownstreamServletCompatibility pins them.
> Everything else keeps 9.4's behaviour: {{/static}} lists no directory,
> {{/logs}} keeps its admin check when the log directory is missing,
> {{web.xml}} error pages work, a stopped listener can be reopened, the
> NodeManager's WebSocket upgrade works, and a {{Location}} set through
> {{sendRedirect}} stays absolute.
> *New settings*:
> ||Setting||Default||
> |{{hadoop.http.uri.compliance.violations}}|{{AMBIGUOUS_EMPTY_SEGMENT,AMBIGUOUS_PATH_ENCODING,SUSPICIOUS_PATH_CHARACTERS}}|
> |{{hadoop.http.ssl.renegotiation.allowed}}|{{false}}|
> *Downstream*:
> * Projects that use only the public API and {{javax.servlet}}: no rebuild, no
> namespace change.
> * Non-shaded artifacts: Jetty 12 and {{org.eclipse.jetty.ee8}} jars replace
> Jetty 9.4's on the classpath. A project pinning Jetty 9.4 next to Hadoop can
> no longer share one Jetty release with it.
> * hadoop-client-runtime and hadoop-client-minicluster: the relocated Jetty is
> 12. Relocated classes are not API.
> * The application catalog war keeps Jetty 9.4 through {{solr.jetty.version}}
> until HADOOP-19998, if HADOOP-20008 landed. If HADOOP-19998 landed first, the
> module is gone and no Jetty 9.4 artifact is left.
> *Depends on*: HADOOP-20012, HADOOP-20013, HADOOP-20014, HADOOP-20015,
> HADOOP-20016, HADOOP-19970, HADOOP-20003, HADOOP-20004, HADOOP-20005,
> HADOOP-20006, HADOOP-20007, and HADOOP-20008 or HADOOP-19998.
> *Not in scope*: {{jakarta.servlet}}, Jersey upgrades and ee9, ee10 or ee11:
> HADOOP-19395. Guice 6 (HADOOP-19999) and jsr311-api (HADOOP-20000) are not
> needed by this issue. HADOOP-19971 belongs to HADOOP-19395, and this issue
> does not depend on it. Requires Java 17, so it cannot go to branch-3.3 or
> branch-3.4.
> *CI*: touches hadoop-project, so a full build, including
> hadoop-client-check-invariants, hadoop-client-check-test-invariants and
> hadoop-client-integration-tests.
> *PR*: to be rebuilt on trunk; the draft is
> https://github.com/apache/hadoop/pull/8704
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
