[
https://issues.apache.org/jira/browse/HADOOP-19912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18132846#comment-18132846
]
Jose Luis López commented on HADOOP-19912:
------------------------------------------
h3. Phase A is ready for review
Phase A is the first step of the Jetty 12 ee8 migration. It makes every module
resolve one Jetty release and one servlet API, declared in the modules that use
them. It changes dependencies only: no Jetty version change and no intended
change in behaviour. Besides poms and LICENSE-binary/NOTICE-binary, it touches
one test.
||Sub-task||Scope||Area||
|HADOOP-20012|The catalog-webapp and timelineservice-hbase tests stop pulling
third-party Jetty and the old servlet API (test scope).|YARN|
|HADOOP-20013|The duplicate javax.annotation-api 1.3.2 goes;
jakarta.annotation-api 1.3.5, already on the classpath, provides the same
javax.annotation classes.|build, LICENSE/NOTICE|
|HADOOP-20014|The Jersey tests run on the JDK HTTP server instead of Jetty
(Jersey 2 has no Jetty 12 container); hadoop-client-minicluster shades that
container instead.|build, shaded clients|
|HADOOP-20015|yarn-common, the ResourceManager, the ApplicationHistoryServer
and the Router declare the Jetty web artifacts they use (same versions and
scopes).|YARN|
|HADOOP-20016|hadoop-common, HttpFS, the shuffle handler and SLS declare
jetty-http, which they import. The declarations are optional (provided in SLS),
so they are not passed on to dependent projects.|common, HttpFS, MapReduce|
h3. How it was verified
A differential harness with several architectures (simple, ha, federated,
secure, other) builds trunk with and without the five patches, runs the same
checks on both, and fails on any difference that is not listed, with the JIRA
that causes it, in an allowlist:
* *Artifacts:* every file and jar entry of the distribution, the classes each
launcher's classpath sees and the jar each comes from, and every module's
resolved dependencies.
* *End to end:* daemons from each side's tarball in five topologies (simple,
secure with Kerberos and HTTPS, HA, federated, mixed), comparing every REST and
CLI response, the classes each daemon loads, and the errors in the logs.
* *Hadoop's own tests,* compared test by test: in every run the web and Jersey
tests of the changed modules plus the existing MiniKdc, HA and federation tests
(about 3,400); on demand, every test of the changed modules (13,739).
* *Downstream projects:* a small Maven project per published Hadoop artifact,
resolved against each side, alone and next to another Jetty version.
The CI builds include the native libraries.
h3. Results
All checks pass on the latest commit:
* *Artifacts:* 255 differences, all explained by the five JIRAs, and each
expected difference does show up.
* *End to end:* about 3,000 observations over the five topologies, no
regression. The few differences are pre-existing and also appear when trunk is
compared with itself.
* *Tests:* no regression, in the full run (13,739 tests) or in the regular
runs.
* *Downstream:* one intended change. javax.annotation-api 1.3.2 no longer
comes with hadoop-mapreduce-client-shuffle and hadoop-minicluster
(HADOOP-20013); jakarta.annotation-api still provides those classes.
h3. Pre-existing problems found on the way
These are the same with and without phase A:
* MAPREDUCE-7548: MapReduce tasks are rejected by the AM when
hadoop.security.authorization is enabled.
* YARN-12024: distributed shell cannot be submitted through the YARN Router
(patch available).
* YARN-12025: the YARN Router's REST forwarding cannot read JSON responses
(patch available).
* YARN-12030: the ApplicationHistoryServer picks its JSON writer anew at each
start, so its responses change shape between runs (patch available).
* HADOOP-20018: the ZKFC exits with a NullPointerException that hides the
BindException when its RPC port is in use.
* MAPREDUCE-7099: jobclient's "timeout in the fork" with no failing test.
TestMRIntermediateDataEncryption sometimes runs 30 minutes because an HDFS read
over loopback slows to about 10 KB/s on the 6.17 kernel of the GitHub runners.
Details in that issue.
h3. Proposal
I propose to open, review and evalutate the five pull requests against trunk,
one per sub-task, each based on trunk alone, and merge them lowest risk first.
# HADOOP-20012 (test scope only)
# HADOOP-20014 (test container, shaded minicluster)
# HADOOP-20013 (runtime classpath, LICENSE/NOTICE)
# HADOOP-20015 (Jetty web artifacts declared in YARN; same versions)
# HADOOP-20016 (jetty-http declared, optional; no downstream change)
The harness, with instructions to run it locally or in GitHub Actions:
[https://github.com/joseluisll/hadoop/tree/HADOOP-99999/dev-support/jetty12-phase-a]
> Upgrade to Jetty 12 on the ee8 environment keeping javax.servlet
> ----------------------------------------------------------------
>
> Key: HADOOP-19912
> URL: https://issues.apache.org/jira/browse/HADOOP-19912
> Project: Hadoop Common
> Issue Type: Task
> Reporter: PJ Fanning
> Assignee: Jose Luis López
> Priority: Major
> Labels: pull-request-available
> Attachments: report-maven.md
>
>
> The goal of this task is to move Hadoop's web servers off Jetty 9.4, which is
> end of life: its releases are now rare and sponsored, and its CVEs keep
> coming (HADOOP-19910, HADOOP-19915).
> h3. Approach
> Jetty 12 serves two servlet APIs side by side: the ee8 environment serves
> {{javax.servlet}}, while ee10 and ee11 serve {{jakarta.servlet}}. This issue
> moves Hadoop to Jetty 12 on ee8, so the servlet namespace, Jersey 2 and every
> public signature stay as they are.
> We will rollout changes to the modules preparing them for the migration to
> Jetty12-ee8 (handling of the HTTP reason code, auth filters, others).
> The move to the jakarta namespace (Jetty ee10, Jersey 3, JAXB 4) is a
> separate decision, tracked in HADOOP-19395, which starts from here.
> Jetty 12 needs Java 17, so this is for trunk. Release lines that stay on
> Jetty 9.4 are covered by HADOOP-19915.
> h3. Tasks
> * Resolve one Jetty release and one servlet API on every module classpath,
> without changing the Jetty version.
> * Drop jetty-util-ajax in favour of Jackson.
> * Upgrade Jetty 9.4 to 12 on the ee8 environment: {{HttpServer2}}, KMS,
> HttpFS, the YARN web apps, WebSocket, SLS, the services API, the shaded
> clients and LICENSE-binary. Keep trunk's behaviour wherever Jetty 12 allows,
> and list the rest for the release note.
> * Move the YARN application catalog off Jetty 9.4, the last Jetty 9.4 in the
> build, which it needs for Solr 8: retire the module, or upgrade it to Solr 10.
> * Remove the JAX-RS 1.1 API ({{jsr311-api}}) from
> {{hadoop-yarn-server-timelineservice-hbase-tests}}, where it duplicates
> JAX-RS 2.1 on the classpath.
> * Upgrade Guice to 6.X, as supports both javax and jakarta namespaces.
> h3. Behaviour changes
> The Jetty 12 upgrade is an incompatible change. Its release note covers what
> Jetty 12 changes for callers:
> * no custom HTTP reason phrase; refusal messages travel in the response body
> instead, and Hadoop's clients read them there;
> * stricter URI parsing, adjustable with
> {{hadoop.http.uri.compliance.violations}};
> * TLS 1.2 renegotiation off by default, adjustable with
> {{hadoop.http.ssl.renegotiation.allowed}};
> * the five async HTTP metrics read 0, because Jetty 12 cannot count them;
> * Jetty's own output: error-page markup, {{charset=utf-8}} and {{Vary}} on
> static files, a {{Date}} header on error responses, and the default acceptor
> count.
> h3. Done when
> * No Jetty 9.4 artifact is left on any module classpath of the trunk build.
> * Every daemon serves on Jetty 12, and the full test suite passes.
> * The release note lists every incompatible change.
> h3. Out of scope
> * The jakarta namespace, Jetty ee10/ee11, Jersey 3 and JAXB 4: HADOOP-19395.
> * Jetty 9.4 security updates on release lines that cannot move to Java 17:
> HADOOP-19915.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
