[
https://issues.apache.org/jira/browse/HADOOP-19972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jose Luis López updated HADOOP-19972:
-------------------------------------
Description:
Hadoop's web server is Jetty 9.4, which is end of life and gets no security
fixes. Jetty 10 and later require the {{jakarta.servlet}} namespace, which
would break every project that embeds Hadoop's web stack. Jetty 12's ee8
environment still serves {{javax.servlet}}, so this moves to a supported Jetty
without that break.
Jetty 9.4.58 to 12.1.12 on ee8. No public signature changes, and downstream
projects need not rebuild. *Requires Java 17*, so it cannot go to branch-3.3 or
branch-3.4. The YARN application catalog webapp stays on Jetty 9.4.58, which
Solr 8 needs.
h2. ee8 is a staging post
ee8 is a stage, not the destination. The point of landing it is that the port
to Jetty 12's API happens here and is not repeated: afterwards Hadoop is on a
supported Jetty, and what remains is the namespace change on its own — ee10 and
{{jakarta.servlet}}, under HADOOP-19912 and HADOOP-19395 — instead of a Jetty
upgrade and an API break having to be taken together. ee8 does carry a
compatibility layer that adapts every request, so it is a place to stand for a
release or two rather than indefinitely; unlike Jetty 10 and 11 it has a
supported vendor behind it, which is what makes standing there reasonable.
h2. Incompatible changes
This is the release note. The reason phrase alone makes this incompatible.
# *No custom HTTP reason phrase is sent.* Refusals from the authentication, KMS
and CSRF filters and from image transfer carry their message in the response
body instead, PUT and DELETE included, and Hadoop's clients read it there. An
older client sees the standard phrase: "Unauthorized" rather than
"Authentication required".
# *Stricter URI parsing.* {{%2F}}, {{%2E}} segments and malformed encodings are
refused with 400. {{hadoop.http.uri.compliance.violations}} controls what is
allowed through.
# *TLS renegotiation is refused.* A client's TLS 1.2 renegotiation closes the
connection. {{hadoop.http.ssl.renegotiation.allowed=true}} restores 9.4's
behaviour.
# *The five async HTTP metrics read 0.* The names and the metric set are
unchanged.
# *Jetty's own output differs:* the error-page markup, charset and {{Vary}} on
static files, a {{Date}} header on error responses, the default acceptor count,
and a context path without its trailing slash redirecting 301 with a relative
{{Location}} where 9.4 sent 302 with an absolute one.
Everything else keeps 9.4's behaviour: {{/static}} lists no directory,
{{/logs}} keeps its admin check when the log directory is missing, {{web.xml}}
error pages work, the HTTP server metrics stay in milliseconds under their
existing names, a stopped listener can be reopened, the NodeManager's web
socket upgrade works, and a redirect's {{Location}} stays absolute.
h2. New settings
|| Setting || Default ||
| {{hadoop.http.uri.compliance.violations}} |
{{AMBIGUOUS_EMPTY_SEGMENT,AMBIGUOUS_PATH_ENCODING,SUSPICIOUS_PATH_CHARACTERS}} |
| {{hadoop.http.ssl.renegotiation.allowed}} | {{false}} |
Adding {{AMBIGUOUS_PATH_SEPARATOR}} to the first accepts {{%2F}} again.
h2. Depends on
HADOOP-19970 — this branch is stacked on it and it has to land first: without
it some modules hold two Jetty versions at once, which compiles and then fails
at run time.
*HADOOP-19971 is not a dependency.* It was re-scoped after this issue was filed
and now belongs with HADOOP-19395.
Needed for a green run rather than by the code: MAPREDUCE-7545, for
HADOOP-19970; and HADOOP-19979 with HADOOP-19987, which fix the one test
failing on this branch.
h2. Not in scope
No move to {{jakarta.servlet}}, no Jersey upgrade, and no ee9, ee10 or ee11.
HADOOP-19912 still lands the namespace change, in a major release. This only
stops that decision from blocking the security fix.
was:
WHY
===
Hadoop's web server is Jetty 9.4, which is end of life and no longer receives
security fixes. Getting off it is normally described as one large change,
because Jetty 10 and everything after it require the jakarta.servlet namespace.
That framing makes a security fix wait on an API break: the projects that embed
Hadoop's web stack - HBase, Hive, Spark, Ozone, Knox - would all have to move on
the same day.
Two facts sharpen the timing. Jetty 9.4 is unsupported now, and branch-3.4 and
branch-3.3 target Java 8 bytecode, so they can never take Jetty 12 - it needs
Java 17. For users, "off end-of-life Jetty" already means "on 3.5 or later".
Jetty 12 makes the two changes separable. It ships the servlet container as a
pluggable environment, and one of those environments, ee8, still serves
javax.servlet. This takes that route: a supported Jetty now, in an ordinary
minor release, with nothing for downstream projects to do.
WHAT CHANGES FOR ANYONE USING HADOOP
====================================
Hadoop runs on a supported web server. That is the whole of it.
No published method changes shape or disappears. The five public classes that
expose servlet types in their signatures expose exactly the same ones as before.
Projects that embed Hadoop's web stack need change nothing and need not rebuild.
No configuration setting changes.
Jetty 12 does behave differently from 9.4 in a few places, and those are handled
rather than passed on: a static file directory that would have started listing
its contents, a web socket upgrade that would have failed, and a listener that
could not be restarted once stopped. A deployment should see 9.4's behaviour
throughout.
WHAT CHANGES INSIDE
===================
Jetty moves from 9.4.58 to 12.0.37 on the ee8 environment. The servlet container
artifacts move under org.eclipse.jetty.ee8, and the servlet API now arrives as
org.eclipse.jetty.toolchain:jetty-servlet-api, which publishes the same
javax.servlet packages as the one it replaces.
Hadoop's HTTP server and its metrics, the YARN web application builder, the web
socket code and the remaining embedded servers - KMS, HttpFS, the scheduler
simulator and the YARN services API - are ported to the new API. The shaded
client artifacts and LICENSE-binary follow.
A new test states the contract this change is promising: the servlet types
Hadoop offers to the projects that embed it stay javax, and none of them may
become jakarta. It fails the day that stops being true.
Across the tree, 126 files still name javax.servlet and none names
jakarta.servlet - unchanged from today. 85 files changed in total.
WHAT THIS DOES NOT DO
=====================
It does not move Hadoop to jakarta.servlet, does not upgrade Jersey, and does
not touch the ee9, ee10 or ee11 environments.
ee8 is a staging post, not a destination. It preserves the namespace, but the
Jetty API port still had to happen, and ee8 carries a compatibility layer that
adapts every request. Unlike Jetty 10 and 11 it has a supported vendor behind
it, which is what makes it a reasonable place to stand for a release or two.
This is not an alternative to HADOOP-19912 and does not replace it.
HADOOP-19912 still lands the namespace change, in a major release, judged on its
own merits and with downstream projects warned. What this does is stop that
decision from being a precondition for fixing the security problem.
DEPENDS ON
==========
HADOOP-19970 and HADOOP-19971, in that order. The branch is stacked on both.
HADOOP-19970 matters functionally: without it, Jersey's test container drags a
Jetty 9 artifact onto around twenty test classpaths, and moving to Jetty 12
before that is fixed leaves those modules holding two Jetty versions at once -
which compiles and then fails at run time.
HOW IT WAS TESTED
=================
Beyond the unit suites, the areas where this kind of change goes wrong need
exercising against a running server rather than assumed: TLS configuration,
SPNEGO, WebHDFS, the YARN UI, KMS and HttpFS. HADOOP-19876 was an SSL
configuration regression on the current line and HADOOP-19848 a
NoSuchMethodError
from a patch-level bump, so that is where the risk sits.
> Upgrade to Jetty 12 without changing the servlet namespace
> ----------------------------------------------------------
>
> Key: HADOOP-19972
> URL: https://issues.apache.org/jira/browse/HADOOP-19972
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build, common
> Reporter: Jose Luis López
> Assignee: Jose Luis López
> Priority: Major
> Labels: pull-request-available
>
> Hadoop's web server is Jetty 9.4, which is end of life and gets no security
> fixes. Jetty 10 and later require the {{jakarta.servlet}} namespace, which
> would break every project that embeds Hadoop's web stack. Jetty 12's ee8
> environment still serves {{javax.servlet}}, so this moves to a supported
> Jetty without that break.
> Jetty 9.4.58 to 12.1.12 on ee8. No public signature changes, and downstream
> projects need not rebuild. *Requires Java 17*, so it cannot go to branch-3.3
> or branch-3.4. The YARN application catalog webapp stays on Jetty 9.4.58,
> which Solr 8 needs.
> h2. ee8 is a staging post
> ee8 is a stage, not the destination. The point of landing it is that the port
> to Jetty 12's API happens here and is not repeated: afterwards Hadoop is on a
> supported Jetty, and what remains is the namespace change on its own — ee10
> and {{jakarta.servlet}}, under HADOOP-19912 and HADOOP-19395 — instead of a
> Jetty upgrade and an API break having to be taken together. ee8 does carry a
> compatibility layer that adapts every request, so it is a place to stand for
> a release or two rather than indefinitely; unlike Jetty 10 and 11 it has a
> supported vendor behind it, which is what makes standing there reasonable.
> h2. Incompatible changes
> This is the release note. The reason phrase alone makes this incompatible.
> # *No custom HTTP reason phrase is sent.* Refusals from the authentication,
> KMS and CSRF filters and from image transfer carry their message in the
> response body instead, PUT and DELETE included, and Hadoop's clients read it
> there. An older client sees the standard phrase: "Unauthorized" rather than
> "Authentication required".
> # *Stricter URI parsing.* {{%2F}}, {{%2E}} segments and malformed encodings
> are refused with 400. {{hadoop.http.uri.compliance.violations}} controls what
> is allowed through.
> # *TLS renegotiation is refused.* A client's TLS 1.2 renegotiation closes the
> connection. {{hadoop.http.ssl.renegotiation.allowed=true}} restores 9.4's
> behaviour.
> # *The five async HTTP metrics read 0.* The names and the metric set are
> unchanged.
> # *Jetty's own output differs:* the error-page markup, charset and {{Vary}}
> on static files, a {{Date}} header on error responses, the default acceptor
> count, and a context path without its trailing slash redirecting 301 with a
> relative {{Location}} where 9.4 sent 302 with an absolute one.
> Everything else keeps 9.4's behaviour: {{/static}} lists no directory,
> {{/logs}} keeps its admin check when the log directory is missing,
> {{web.xml}} error pages work, the HTTP server metrics stay in milliseconds
> under their existing names, a stopped listener can be reopened, the
> NodeManager's web socket upgrade works, and a redirect's {{Location}} stays
> absolute.
> h2. New settings
> || Setting || Default ||
> | {{hadoop.http.uri.compliance.violations}} |
> {{AMBIGUOUS_EMPTY_SEGMENT,AMBIGUOUS_PATH_ENCODING,SUSPICIOUS_PATH_CHARACTERS}}
> |
> | {{hadoop.http.ssl.renegotiation.allowed}} | {{false}} |
> Adding {{AMBIGUOUS_PATH_SEPARATOR}} to the first accepts {{%2F}} again.
> h2. Depends on
> HADOOP-19970 — this branch is stacked on it and it has to land first: without
> it some modules hold two Jetty versions at once, which compiles and then
> fails at run time.
> *HADOOP-19971 is not a dependency.* It was re-scoped after this issue was
> filed and now belongs with HADOOP-19395.
> Needed for a green run rather than by the code: MAPREDUCE-7545, for
> HADOOP-19970; and HADOOP-19979 with HADOOP-19987, which fix the one test
> failing on this branch.
> h2. Not in scope
> No move to {{jakarta.servlet}}, no Jersey upgrade, and no ee9, ee10 or ee11.
> HADOOP-19912 still lands the namespace change, in a major release. This only
> stops that decision from blocking the security fix.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
