[
https://issues.apache.org/jira/browse/HADOOP-19978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jose Luis López updated HADOOP-19978:
-------------------------------------
Description:
*Why*
HADOOP-19964 reimplements TimedOutTestsListener as a JUnit Platform
TestExecutionListener, so a timed-out test prints a thread dump again. The
listener property that eight poms already carry binds only once Surefire
accepts platform listeners - SUREFIRE-1639, shipped in maven-surefire-plugin
3.6.0. On 3.5.3 the property is inert, so the listener stays dormant until
Hadoop moves to that version.
*What the upgrade needs*
Bumping 3.5.3 to 3.6.0 on its own turns 5 of 7 CI test jobs red: common, mr,
hdfs-rbf, hdfs-other, hdfs-slow. Controlled against another branch built the
same morning on the same runners with 3.5.3, which was 7/7 green. Every
failure is one fault - the forked JVM preferring IPv6, e.g.
TestNameNodeRpcServer.testNamenodeRpcBindAny expected <0.0.0.0> but was
<0:0:0:0:0:0:0:0>.
hadoop-project.xml sets java.net.preferIPv4Stack in surefire's
systemPropertyVariables. Those are applied by the forked booter once the JVM
is already running, never as -D at startup. 3.6.0 initialises the fork's
networking before applying them, so the flag is read too late; 3.5.3 applies
them before any networking class loads, which is why the same configuration
works there. No property-based fix can work.
*Changes*
# maven-surefire-plugin.version to 3.6.0.
# Move -Djava.net.preferIPv4Stack=true into maven-surefire-plugin.argLine,
where it is a real JVM startup argument. All 22 argLine overrides interpolate
that property and maven-failsafe-plugin shares it, so it propagates
everywhere. Drop hadoop-registry's duplicate copy of the same ineffective
systemPropertyVariable.
# Add hadoop-common's test-jar in test scope to hadoop-yarn-services-core and
hadoop-yarn-applications-catalog-webapp. Both inherit the listener property
from hadoop-yarn but resolve only hadoop-common's main jar, so Surefire
cannot load the listener class and silently drops it.
*Also in 3.6.0*
SUREFIRE-823 decouples -DskipTests from maven-failsafe-plugin: -DskipTests no
longer skips integration tests, only -DskipITs does.
hadoop-client-integration-tests relies on -DskipTests alone, so its ITs now
run under "mvn install -DskipTests". The cloud connectors gate on skipITs and
are unaffected. To be fixed here or in a follow-up.
The legacy surefire-junit3/junit4/junit47/testng providers are removed in
3.6.0. Hadoop has no JUnit 4 or TestNG tests, so this does not apply.
was:
h3. Why
HADOOP-19964 reimplements TimedOutTestsListener as a JUnit Platform
TestExecutionListener, so a timed-out test prints a thread dump again. The
{{listener}} provider property that eight poms already carry binds to it only
once Surefire accepts platform listeners there — SUREFIRE-1639, shipped in
maven-surefire-plugin 3.6.0. Until Hadoop moves to that version the listener
is dormant.
h3. What blocks the upgrade
Bumping {{maven-surefire-plugin.version}} from 3.5.3 to 3.6.0 turns 5 of 7 CI
test jobs red: common, mr, hdfs-rbf, hdfs-other, hdfs-slow. Controlled against
another branch built the same morning on the same runners with 3.5.3, which
was 7/7 green. Every failure is one fault — the forked JVM preferring IPv6:
{code:java}
TestNameNodeRpcServer.testNamenodeRpcBindAny expected: <0.0.0.0> but was:
<0:0:0:0:0:0:0:0>
TestHSAdminServer not a valid host:port authority: 0:0:0:0:0:0:0:0:33745
TestBlockRecovery2 not a valid host:port authority: 0:0:0:0:0:0:0:0:39677
TestDNS ArrayIndexOutOfBounds Index 3 out of bounds for length 1
TestWebHdfsTokens Connect localhost:44787: Connection refused
{code}
hadoop-project.xml sets {{java.net.preferIPv4Stack=true}} in surefire's
{{{}systemPropertyVariables{}}}. Those are applied by the forked booter, not as
{{-D}} at JVM startup — in 3.5.3 and 3.6.0 alike. Class-load traces
({{{}-Xlog:class+load{}}}) show SystemPropertyManager running at index 699 under
3.5.3 and 736 under 3.6.0; in that gap 3.6.0 loads {{sun.nio.ch.IOUtil}} and
{{{}sun.nio.ch.SocketChannelImpl{}}}, where 3.5.3 loads no networking classes at
all. 3.6.0 initialises the fork's networking before the property lands, so the
flag is read too late.
Reported upstream as apache/maven-surefire#3456.
h3. Changes
* Bump {{maven-surefire-plugin.version}} to 3.6.0.
* Move {{-Djava.net.preferIPv4Stack=true}} into
{{maven-surefire-plugin.argLine}} so it is a real JVM startup argument. All
22 {{<argLine>}} overrides in the tree already interpolate that property, and
maven-failsafe-plugin shares it, so it propagates everywhere. No
property-based fix can work: Surefire initialises networking before applying
any of them.
* Add hadoop-common's test-jar in test scope to hadoop-yarn-services-core and
hadoop-yarn-applications-catalog-webapp. Both inherit the {{listener}}
property from hadoop-yarn but depend only on the main jar, so with the
listener live Surefire cannot resolve the class and the fork dies before any
test runs. Verified: without the dependency the module reports "Tests run: 0"
and the fork terminates; with it, tests pass.
h3. Notes
Only {{java.net.preferIPv4Stack}} is affected. Checking the other
systemPropertyVariables against the same criterion — is anything initialised
before the properties are applied? — {{{}java.security.egd{}}},
{{{}java.security.krb5.conf{}}}, {{{}hadoop.log.dir{}}}, {{{}hadoop.tmp.dir{}}},
{{test.build.*}} and {{require.test.libhadoop}} are all read later and are
unaffected.
> Upgrade maven-surefire-plugin to 3.6
> ------------------------------------
>
> Key: HADOOP-19978
> URL: https://issues.apache.org/jira/browse/HADOOP-19978
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, test
> Reporter: Jose Luis López
> Priority: Major
> Labels: pull-request-available
>
> *Why*
> HADOOP-19964 reimplements TimedOutTestsListener as a JUnit Platform
> TestExecutionListener, so a timed-out test prints a thread dump again. The
> listener property that eight poms already carry binds only once Surefire
> accepts platform listeners - SUREFIRE-1639, shipped in maven-surefire-plugin
> 3.6.0. On 3.5.3 the property is inert, so the listener stays dormant until
> Hadoop moves to that version.
> *What the upgrade needs*
> Bumping 3.5.3 to 3.6.0 on its own turns 5 of 7 CI test jobs red: common, mr,
> hdfs-rbf, hdfs-other, hdfs-slow. Controlled against another branch built the
> same morning on the same runners with 3.5.3, which was 7/7 green. Every
> failure is one fault - the forked JVM preferring IPv6, e.g.
> TestNameNodeRpcServer.testNamenodeRpcBindAny expected <0.0.0.0> but was
> <0:0:0:0:0:0:0:0>.
> hadoop-project.xml sets java.net.preferIPv4Stack in surefire's
> systemPropertyVariables. Those are applied by the forked booter once the JVM
> is already running, never as -D at startup. 3.6.0 initialises the fork's
> networking before applying them, so the flag is read too late; 3.5.3 applies
> them before any networking class loads, which is why the same configuration
> works there. No property-based fix can work.
> *Changes*
> # maven-surefire-plugin.version to 3.6.0.
> # Move -Djava.net.preferIPv4Stack=true into maven-surefire-plugin.argLine,
> where it is a real JVM startup argument. All 22 argLine overrides interpolate
> that property and maven-failsafe-plugin shares it, so it propagates
> everywhere. Drop hadoop-registry's duplicate copy of the same ineffective
> systemPropertyVariable.
> # Add hadoop-common's test-jar in test scope to hadoop-yarn-services-core and
> hadoop-yarn-applications-catalog-webapp. Both inherit the listener property
> from hadoop-yarn but resolve only hadoop-common's main jar, so Surefire
> cannot load the listener class and silently drops it.
> *Also in 3.6.0*
> SUREFIRE-823 decouples -DskipTests from maven-failsafe-plugin: -DskipTests no
> longer skips integration tests, only -DskipITs does.
> hadoop-client-integration-tests relies on -DskipTests alone, so its ITs now
> run under "mvn install -DskipTests". The cloud connectors gate on skipITs and
> are unaffected. To be fixed here or in a follow-up.
> The legacy surefire-junit3/junit4/junit47/testng providers are removed in
> 3.6.0. Hadoop has no JUnit 4 or TestNG tests, so this does not apply.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
