[jira] [Commented] (HADOOP-12261) Surefire needs to make sure the JVMs it fires up fit within the memory available
[
https://issues.apache.org/jira/browse/HADOOP-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15195663#comment-15195663
]
Hadoop QA commented on HADOOP-12261:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
46s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 8s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 9s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 12s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 9s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 9s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 6s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 6s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 7s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 7s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 6s
{color} | {color:green} hadoop-project in the patch passed with JDK v1.8.0_74.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 6s
{color} | {color:green} hadoop-project in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 4s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12793571/HADOOP-12261.002.patch
|
| JIRA Issue | HADOOP-12261 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml |
| uname | Linux 7ea4d2884af4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d457401 |
| Default Java | 1.7.0_95 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_74
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 |
| JDK v1.7.0_95 Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/8853/testReport/ |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
[jira] [Commented] (HADOOP-12261) Surefire needs to make sure the JVMs it fires up fit within the memory available
[
https://issues.apache.org/jira/browse/HADOOP-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089340#comment-15089340
]
Steve Loughran commented on HADOOP-12261:
-
Catching up on this constructive discussion showing open source communities at
work, can I point out that this is something you can actually tune on the
command line,
{code}
mvn clean test "-Dmaven-surefire-plugin.argLine=-Xmx6G"
{code}
It's not something we need to change just yet, unless its critical everywhere
—and if we do change it, anyone who does want to try and build & test on 32
bits can still change the option to a value they can handle.
On Java 8 you'd want to turn off the permgen value too, just to avoid being
told off by the JVM for setting it.
> Surefire needs to make sure the JVMs it fires up fit within the memory
> available
>
>
> Key: HADOOP-12261
> URL: https://issues.apache.org/jira/browse/HADOOP-12261
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: test
>Affects Versions: 2.7.1
>Reporter: Alan Burlison
>Assignee: Alan Burlison
> Attachments: HADOOP-12261.001.patch
>
>
> hadoop-project/pom.xml sets maven-surefire-plugin.argLine to include
> -Xmx4096m. Allocating that amount of memory requires a 64-bit JVM, but on
> platforms with both 32 and 64-bit JVMs surefire runs the 32 bit version by
> default and tests fail to start as a result. "-d64" should be added to the
> command-line arguments to ensure a 64-bit JVM is always used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12261) Surefire needs to make sure the JVMs it fires up fit within the memory available
[
https://issues.apache.org/jira/browse/HADOOP-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15022011#comment-15022011
]
Hadoop QA commented on HADOOP-12261:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
58s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 8s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 9s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 12s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 9s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 9s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 8s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 9s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 9s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 8s
{color} | {color:green} hadoop-project in the patch passed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 8s
{color} | {color:green} hadoop-project in the patch passed with JDK v1.7.0_85.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 32s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12773824/HADOOP-12261.001.patch
|
| JIRA Issue | HADOOP-12261 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml |
| uname | Linux 7f2547f23794 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 201f14e |
| JDK v1.7.0_85 Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/8139/testReport/ |
| modules | C: hadoop-project U: hadoop-project |
| Max memory used | 75MB |
| Powered by | Apache Yetus http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/8139/console |
This message was automatically
[jira] [Commented] (HADOOP-12261) Surefire needs to make sure the JVMs it fires up fit within the memory available
[ https://issues.apache.org/jira/browse/HADOOP-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704173#comment-14704173 ] Colin Patrick McCabe commented on HADOOP-12261: --- HDFS-8896 looks related to the higher-than-expected memory consumption of tests Surefire needs to make sure the JVMs it fires up fit within the memory available Key: HADOOP-12261 URL: https://issues.apache.org/jira/browse/HADOOP-12261 Project: Hadoop Common Issue Type: Sub-task Components: test Affects Versions: 2.7.1 Reporter: Alan Burlison Assignee: Alan Burlison hadoop-project/pom.xml sets maven-surefire-plugin.argLine to include -Xmx4096m. Allocating that amount of memory requires a 64-bit JVM, but on platforms with both 32 and 64-bit JVMs surefire runs the 32 bit version by default and tests fail to start as a result. -d64 should be added to the command-line arguments to ensure a 64-bit JVM is always used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12261) Surefire needs to make sure the JVMs it fires up fit within the memory available
[ https://issues.apache.org/jira/browse/HADOOP-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704177#comment-14704177 ] Colin Patrick McCabe commented on HADOOP-12261: --- Btw, [~alanbur]... I agree that the Hadoop test suite could be in better shape than it is. It is certainly frustrating. I think what you should do is create a blacklist of tests that you can't get to work, and work towards validating Solaris support on all the other tests. Over time, you can gradually take tests off the blacklist as we fix whatever is making them flaky or difficult to run in your environment. Surefire needs to make sure the JVMs it fires up fit within the memory available Key: HADOOP-12261 URL: https://issues.apache.org/jira/browse/HADOOP-12261 Project: Hadoop Common Issue Type: Sub-task Components: test Affects Versions: 2.7.1 Reporter: Alan Burlison Assignee: Alan Burlison hadoop-project/pom.xml sets maven-surefire-plugin.argLine to include -Xmx4096m. Allocating that amount of memory requires a 64-bit JVM, but on platforms with both 32 and 64-bit JVMs surefire runs the 32 bit version by default and tests fail to start as a result. -d64 should be added to the command-line arguments to ensure a 64-bit JVM is always used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12261) Surefire needs to make sure the JVMs it fires up fit within the memory available
[ https://issues.apache.org/jira/browse/HADOOP-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660182#comment-14660182 ] Alan Burlison commented on HADOOP-12261: I've tried setting -Xmx2048m and I got a GC time exceeded error, at 3Gb it seems to work but as I only get a small fraction of the way through the test suite due to other multiple failures I'm hesitant to say I have it fixed. I can't even get the Hadoop test suite to run successfully on a stripped-down VM with a vanilla network configuration. The parlous state of the Hadoop test suite is a major blocker, to the point where I'm beginning to question if I can realistically complete the addition of Solaris support - if I can't test Hadoop then I'll never know if the changes are actually correct or not. Surefire needs to make sure the JVMs it fires up fit within the memory available Key: HADOOP-12261 URL: https://issues.apache.org/jira/browse/HADOOP-12261 Project: Hadoop Common Issue Type: Sub-task Components: test Affects Versions: 2.7.1 Reporter: Alan Burlison Assignee: Alan Burlison hadoop-project/pom.xml sets maven-surefire-plugin.argLine to include -Xmx4096m. Allocating that amount of memory requires a 64-bit JVM, but on platforms with both 32 and 64-bit JVMs surefire runs the 32 bit version by default and tests fail to start as a result. -d64 should be added to the command-line arguments to ensure a 64-bit JVM is always used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12261) Surefire needs to make sure the JVMs it fires up fit within the memory available
[ https://issues.apache.org/jira/browse/HADOOP-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14644341#comment-14644341 ] Alan Burlison commented on HADOOP-12261: I don't think Docker will help much, it would introduce another complex dependency and further complicate the cross-platform support issues that Hadoop is already bedevilled by. Surefire needs to make sure the JVMs it fires up fit within the memory available Key: HADOOP-12261 URL: https://issues.apache.org/jira/browse/HADOOP-12261 Project: Hadoop Common Issue Type: Sub-task Components: test Affects Versions: 2.7.1 Reporter: Alan Burlison Assignee: Alan Burlison hadoop-project/pom.xml sets maven-surefire-plugin.argLine to include -Xmx4096m. Allocating that amount of memory requires a 64-bit JVM, but on platforms with both 32 and 64-bit JVMs surefire runs the 32 bit version by default and tests fail to start as a result. -d64 should be added to the command-line arguments to ensure a 64-bit JVM is always used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12261) Surefire needs to make sure the JVMs it fires up fit within the memory available
[ https://issues.apache.org/jira/browse/HADOOP-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14644335#comment-14644335 ] Alan Burlison commented on HADOOP-12261: The change is simply to edit hadoop-project/pom.xml and change the -Xmx4096m value, but I'm not convinced the tests will work with less memory - I've seen at least one OOM. Surefire needs to make sure the JVMs it fires up fit within the memory available Key: HADOOP-12261 URL: https://issues.apache.org/jira/browse/HADOOP-12261 Project: Hadoop Common Issue Type: Sub-task Components: test Affects Versions: 2.7.1 Reporter: Alan Burlison Assignee: Alan Burlison hadoop-project/pom.xml sets maven-surefire-plugin.argLine to include -Xmx4096m. Allocating that amount of memory requires a 64-bit JVM, but on platforms with both 32 and 64-bit JVMs surefire runs the 32 bit version by default and tests fail to start as a result. -d64 should be added to the command-line arguments to ensure a 64-bit JVM is always used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12261) Surefire needs to make sure the JVMs it fires up fit within the memory available
[ https://issues.apache.org/jira/browse/HADOOP-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14644726#comment-14644726 ] Allen Wittenauer commented on HADOOP-12261: --- Never mind the fact that we already have a working Dockerfile in the source tree... Surefire needs to make sure the JVMs it fires up fit within the memory available Key: HADOOP-12261 URL: https://issues.apache.org/jira/browse/HADOOP-12261 Project: Hadoop Common Issue Type: Sub-task Components: test Affects Versions: 2.7.1 Reporter: Alan Burlison Assignee: Alan Burlison hadoop-project/pom.xml sets maven-surefire-plugin.argLine to include -Xmx4096m. Allocating that amount of memory requires a 64-bit JVM, but on platforms with both 32 and 64-bit JVMs surefire runs the 32 bit version by default and tests fail to start as a result. -d64 should be added to the command-line arguments to ensure a 64-bit JVM is always used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12261) Surefire needs to make sure the JVMs it fires up fit within the memory available
[ https://issues.apache.org/jira/browse/HADOOP-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14644808#comment-14644808 ] Allen Wittenauer commented on HADOOP-12261: --- Oh, I'm definitely not saying that running the unit tests should require Docker. I'm just pointing out that we *already* have a Docker image that has all the things needed to run the unit tests. It doesn't help the predictability in any way/shape/form, other than making sure that the prereqs for running all of the unit tests (regardless of quality) are all there. Surefire needs to make sure the JVMs it fires up fit within the memory available Key: HADOOP-12261 URL: https://issues.apache.org/jira/browse/HADOOP-12261 Project: Hadoop Common Issue Type: Sub-task Components: test Affects Versions: 2.7.1 Reporter: Alan Burlison Assignee: Alan Burlison hadoop-project/pom.xml sets maven-surefire-plugin.argLine to include -Xmx4096m. Allocating that amount of memory requires a 64-bit JVM, but on platforms with both 32 and 64-bit JVMs surefire runs the 32 bit version by default and tests fail to start as a result. -d64 should be added to the command-line arguments to ensure a 64-bit JVM is always used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12261) Surefire needs to make sure the JVMs it fires up fit within the memory available
[ https://issues.apache.org/jira/browse/HADOOP-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14644752#comment-14644752 ] Allen Wittenauer commented on HADOOP-12261: --- That's exactly the model we're eventually going to move to for the Jenkins hosts, similar to a lot of the other Apache projects. The code is all there, just need to turn it on. But agreed: it doesn't help the fact that most of the Hadoop unit tests are broken. Surefire needs to make sure the JVMs it fires up fit within the memory available Key: HADOOP-12261 URL: https://issues.apache.org/jira/browse/HADOOP-12261 Project: Hadoop Common Issue Type: Sub-task Components: test Affects Versions: 2.7.1 Reporter: Alan Burlison Assignee: Alan Burlison hadoop-project/pom.xml sets maven-surefire-plugin.argLine to include -Xmx4096m. Allocating that amount of memory requires a 64-bit JVM, but on platforms with both 32 and 64-bit JVMs surefire runs the 32 bit version by default and tests fail to start as a result. -d64 should be added to the command-line arguments to ensure a 64-bit JVM is always used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12261) Surefire needs to make sure the JVMs it fires up fit within the memory available
[ https://issues.apache.org/jira/browse/HADOOP-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14644801#comment-14644801 ] Alan Burlison commented on HADOOP-12261: Using it on the Apache Jenkins infrastructure seem reasonable enough, making it a prerequisite for people trying to run the tests on their own machines, not. The biggest problem I have is the unpredictability of the tests - I get different results on every run, even on Linux, which makes it impossible to know if any particular failure is due to a change you've made or just part of the general random brokenness of the tests. Surefire needs to make sure the JVMs it fires up fit within the memory available Key: HADOOP-12261 URL: https://issues.apache.org/jira/browse/HADOOP-12261 Project: Hadoop Common Issue Type: Sub-task Components: test Affects Versions: 2.7.1 Reporter: Alan Burlison Assignee: Alan Burlison hadoop-project/pom.xml sets maven-surefire-plugin.argLine to include -Xmx4096m. Allocating that amount of memory requires a 64-bit JVM, but on platforms with both 32 and 64-bit JVMs surefire runs the 32 bit version by default and tests fail to start as a result. -d64 should be added to the command-line arguments to ensure a 64-bit JVM is always used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12261) Surefire needs to make sure the JVMs it fires up fit within the memory available
[ https://issues.apache.org/jira/browse/HADOOP-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14644742#comment-14644742 ] Alan Burlison commented on HADOOP-12261: Not the point. The point is you'd need to pull in docker and everything it uses. Surefire needs to make sure the JVMs it fires up fit within the memory available Key: HADOOP-12261 URL: https://issues.apache.org/jira/browse/HADOOP-12261 Project: Hadoop Common Issue Type: Sub-task Components: test Affects Versions: 2.7.1 Reporter: Alan Burlison Assignee: Alan Burlison hadoop-project/pom.xml sets maven-surefire-plugin.argLine to include -Xmx4096m. Allocating that amount of memory requires a 64-bit JVM, but on platforms with both 32 and 64-bit JVMs surefire runs the 32 bit version by default and tests fail to start as a result. -d64 should be added to the command-line arguments to ensure a 64-bit JVM is always used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12261) Surefire needs to make sure the JVMs it fires up fit within the memory available
[ https://issues.apache.org/jira/browse/HADOOP-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14644835#comment-14644835 ] Colin Patrick McCabe commented on HADOOP-12261: --- Oh, I wasn't suggesting making Docker a prerequisite for anything. I was just suggesting that using Docker would help ensure a repeatable test environment on Jenkins (and perhaps elsewhere). For example, in the past we had problems with unit tests creating directories with the executable bit not set. Maven clean does not remove those directories, so they could linger and cause all subsequent unit tests run on a particular slave to fail. We've also had problems with people using unsupported versions of Java (I forget the names, but there are some pretty esoteric JVMs out there, and they don't really work 100% with Hadoop.) Docker would help avoid both of these problems and give people running the tests at home a better experience. bq. The change is simply to edit hadoop-project/pom.xml and change the -Xmx4096m value, but I'm not convinced the tests will work with less memory - I've seen at least one OOM. Can you post the patch so that we can find out? :) Also, I can't +1 my own patch, so I really do need you (as opposed to me) to post it. It's a procedural thing... bq. The biggest problem I have is the unpredictability of the tests - I get different results on every run, even on Linux, which makes it impossible to know if any particular failure is due to a change you've made or just part of the general random brokenness of the tests. Yeah, it's frustrating that this happens... I have observed it as well. I wonder why local test runs are flaky, but Jenkins seems to be relatively predictable? Perhaps some tests can't run on smaller machines? Surefire needs to make sure the JVMs it fires up fit within the memory available Key: HADOOP-12261 URL: https://issues.apache.org/jira/browse/HADOOP-12261 Project: Hadoop Common Issue Type: Sub-task Components: test Affects Versions: 2.7.1 Reporter: Alan Burlison Assignee: Alan Burlison hadoop-project/pom.xml sets maven-surefire-plugin.argLine to include -Xmx4096m. Allocating that amount of memory requires a 64-bit JVM, but on platforms with both 32 and 64-bit JVMs surefire runs the 32 bit version by default and tests fail to start as a result. -d64 should be added to the command-line arguments to ensure a 64-bit JVM is always used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12261) Surefire needs to make sure the JVMs it fires up fit within the memory available
[ https://issues.apache.org/jira/browse/HADOOP-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14644920#comment-14644920 ] Alan Burlison commented on HADOOP-12261: .bq Can you post the patch so that we can find out? Also, I can't +1 my own patch, so I really do need you (as opposed to me) to post it. It's a procedural thing... I can but I have to take all patches through our internal legal approval process before I can post them. I'm loathe to do that unless I know it's an actual fix. .bq I have observed it as well. I wonder why local test runs are flaky, but Jenkins seems to be relatively predictable? Perhaps some tests can't run on smaller machines? What qualifies as smaller? And most of the errors I see aren't size-related, they are various network-related timeouts and size/count mismatches, random file creation issues and so forth. Surefire needs to make sure the JVMs it fires up fit within the memory available Key: HADOOP-12261 URL: https://issues.apache.org/jira/browse/HADOOP-12261 Project: Hadoop Common Issue Type: Sub-task Components: test Affects Versions: 2.7.1 Reporter: Alan Burlison Assignee: Alan Burlison hadoop-project/pom.xml sets maven-surefire-plugin.argLine to include -Xmx4096m. Allocating that amount of memory requires a 64-bit JVM, but on platforms with both 32 and 64-bit JVMs surefire runs the 32 bit version by default and tests fail to start as a result. -d64 should be added to the command-line arguments to ensure a 64-bit JVM is always used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12261) Surefire needs to make sure the JVMs it fires up fit within the memory available
[ https://issues.apache.org/jira/browse/HADOOP-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14642848#comment-14642848 ] Alan Burlison commented on HADOOP-12261: I've just reduced the heap down to 2Gb which should work in either case. In any case it's largely moot at the moment anyway as I've been unable to get *any* successful test runs at all, even on Linux. Seems to me that the non-repeatibility of the Hadoop test suite is a much larger concern. Surefire needs to make sure the JVMs it fires up fit within the memory available Key: HADOOP-12261 URL: https://issues.apache.org/jira/browse/HADOOP-12261 Project: Hadoop Common Issue Type: Sub-task Components: test Affects Versions: 2.7.1 Reporter: Alan Burlison Assignee: Alan Burlison hadoop-project/pom.xml sets maven-surefire-plugin.argLine to include -Xmx4096m. Allocating that amount of memory requires a 64-bit JVM, but on platforms with both 32 and 64-bit JVMs surefire runs the 32 bit version by default and tests fail to start as a result. -d64 should be added to the command-line arguments to ensure a 64-bit JVM is always used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12261) Surefire needs to make sure the JVMs it fires up fit within the memory available
[ https://issues.apache.org/jira/browse/HADOOP-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643159#comment-14643159 ] Colin Patrick McCabe commented on HADOOP-12261: --- Thanks, [~alanburlison]. Can you post the patch here? If it works on Jenkins, I suppose we can commit it, and that will get 32-bit working again. I agree that the difficulty of setting up a Hadoop test environment is a big concern. If you have any ideas for improving it, that would be great. Perhaps a docker image with a working setup would help? We also have a lot of tests with too-tight timeouts or big resource requirements that tend to fail on smaller build machines-- those should also be fixed. [~aw], I was just pointing out that someday we will want to have a discussion about the costs and benefits of 32-bit support. It looks like the proposed change is minor, so we can postpone that day. I don't think your issues with the PMC and the other folks on the project are relevant here and I would appreciate it if you could stay focused. Surefire needs to make sure the JVMs it fires up fit within the memory available Key: HADOOP-12261 URL: https://issues.apache.org/jira/browse/HADOOP-12261 Project: Hadoop Common Issue Type: Sub-task Components: test Affects Versions: 2.7.1 Reporter: Alan Burlison Assignee: Alan Burlison hadoop-project/pom.xml sets maven-surefire-plugin.argLine to include -Xmx4096m. Allocating that amount of memory requires a 64-bit JVM, but on platforms with both 32 and 64-bit JVMs surefire runs the 32 bit version by default and tests fail to start as a result. -d64 should be added to the command-line arguments to ensure a 64-bit JVM is always used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
