[
https://issues.apache.org/jira/browse/HADOOP-13514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16236972#comment-16236972
]
Allen Wittenauer commented on HADOOP-13514:
-------------------------------------------
Latest version of surefire is 2.20.1, released in September.
I've been spending a few weeks looking at our usage on the ASF build machines.
We typically have anywhere from 8 to 40 zombie JVMs running around after the
hadoop-common, hadoop-hdfs, and hadoop-mapreduce allegedly complete. These do
nothing but eat resources. As a result, timeouts are pretty normal and problems
that are claimed to be "environmental" are mostly self-inflicted by badly
written tests that really only get any stress on the build machines.
I'm fairly convinced that HDFS-12711 can be prevented with SUREFIRE-773, which
was closed as a dupe of SUREFIRE-524. At this point, I think we absolutely
need to make this a priority to get into at least trunk and branch-2 ASAP.
[It's easy to say... "this wasn't a problem before!"... I'm not so convinced it
wasn't. I think we just didn't see it as often. But new tests are doing new
things that eat more resources and stay around longer which impacts more runs.
Additionally, running unit tests under Docker guaranteed that when the
container died, so did all of these stale JVMs. ]
> Upgrade maven surefire plugin to 2.19.1
> ---------------------------------------
>
> Key: HADOOP-13514
> URL: https://issues.apache.org/jira/browse/HADOOP-13514
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: 2.8.0
> Reporter: Ewan Higgs
> Assignee: Akira Ajisaka
> Priority: Major
> Attachments: HADOOP-13514-addendum.01.patch,
> HADOOP-13514-testing.001.patch, HADOOP-13514-testing.002.patch,
> HADOOP-13514-testing.003.patch, HADOOP-13514-testing.004.patch,
> HADOOP-13514.002.patch, HADOOP-13514.003.patch, HADOOP-13514.004.patch,
> HADOOP-13514.005.patch, surefire-2.19.patch
>
>
> A lot of people working on Hadoop don't want to run all the tests when they
> develop; only the bits they're working on. Surefire 2.19 introduced more
> useful test filters which let us run a subset of the tests that brings the
> build time down from 'come back tomorrow' to 'grab a coffee'.
> For instance, if I only care about the S3 adaptor, I might run:
> {code}
> mvn test -Dmaven.javadoc.skip=true -Pdist,native -Djava.awt.headless=true
> \"-Dtest=org.apache.hadoop.fs.*, org.apache.hadoop.hdfs.*,
> org.apache.hadoop.fs.s3a.*\"
> {code}
> We can work around this by specifying the surefire version on the command
> line but it would be better, imo, to just update the default surefire used.
> {code}
> mvn test -Dmaven.javadoc.skip=true -Pdist,native -Djava.awt.headless=true
> \"-Dtest=org.apache.hadoop.fs.*, org.apache.hadoop.hdfs.*,
> org.apache.hadoop.fs.s3a.*\" -Dmaven-surefire-plugin.version=2.19.1
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
