Fix is posted here: https://github.com/apache/mahout/pull/119
On Fri, 17 Apr 2015 at 12:04 Andrew Palumbo <[email protected]> wrote: > > > > -------- Forwarded Message -------- > Subject: Re: h2o failures on jenkins > Date: Fri, 17 Apr 2015 18:35:42 +0000 > From: Anand Avati <[email protected]> > To: Andrew Palumbo <[email protected]> > > > > I'm testing this patch: > > commit fafa597e0637a59249d754fb5a99d9746b1032f3 > Author: Anand Avati <[email protected] <mailto:[email protected]>> > Date: Fri Apr 17 11:34:45 2015 -0700 > > h2o-test: use unique session id strings > > Signed-off-by: Anand Avati <[email protected] <mailto: > [email protected]>> > > diff --git > a/h2o/src/test/scala/org/apache/mahout/h2obindings/test/DistributedH2OSuite.scala > b/h2o/src/test/scala/org/apache/mahout/h2obindings/test/Distribute > index 45c77a5..abb4289 100644 > --- > a/h2o/src/test/scala/org/apache/mahout/h2obindings/test/DistributedH2OSuite.scala > +++ > b/h2o/src/test/scala/org/apache/mahout/h2obindings/test/DistributedH2OSuite.scala > @@ -30,7 +30,7 @@ trait DistributedH2OSuite extends DistributedMahoutSuite > with LoggerConfiguratio > override protected def beforeEach() { > super.beforeEach() > > - mahoutCtx = mahoutH2OContext("mah2out") > + mahoutCtx = mahoutH2OContext("mah2out" + System.currentTimeMillis()) > } > > override protected def afterEach() { > > > On Fri, 17 Apr 2015 at 11:31 Anand Avati <[email protected] > <mailto:[email protected]>> wrote: > > That might be a quick fix. Just FYI - for a multi-node test > (whenever we start testing that), all nodes of a session should > share the exact string. > > On Fri, 17 Apr 2015 at 11:27 Andrew Palumbo <[email protected] > <mailto:[email protected]>> wrote: > > I suppose than an easy fix would be to name the session > something like "mah2out"+getcurrenttimemilliseconds () or > something like that. > > > Sent from my Verizon Wireless 4G LTE smartphone > > > -------- Original message -------- > From: Anand Avati > Date:04/17/2015 2:23 PM (GMT-05:00) > To: Andrew Palumbo > Subject: Re: h2o failures on jenkins > > It is quite possible that the two tests (based on hadoop2 and > hadoop1) are accidentally discovering each other when run in > parallel. I think giving each test a separate session id might > fix this problem. Currently the session id string "mah2out" is > hardcoded in the test/ code. If that can be somehow set to a > unique string at runtime (through env variable?), that might do > the trick. Thoughts on what's the best way we could try this? > > > On Fri, 17 Apr 2015 at 10:57 Andrew Palumbo <[email protected] > <mailto:[email protected]>> wrote: > > Hey Anand, > The h2o Module has been failing regularly in the Jenkins > build. It > seems to fail about every other build test. (i.e. it does > still pass). > Jenkins was recently moved to a new server around the same > time that we > started getting this. We also are running tests for 2 > profiles now, > hadoop 1 and hadoop 2, which made me think that its possible > that they > are discovering each other and running as a 2 node cloud > (just a theory)? > > I was thinking maybe we should have a different test context > for h2o > which does not call > > H2O.joinOthers(); > > Not sure if this is the answer though. Any thoughts? > > [32mDistributedDecompositionsSuite: [0m > 04-17 17:31:31.521 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #onsSuite INFO: ----- > H2O started ----- > 04-17 17:31:31.577 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #onsSuite INFO: Build > git branch: master > 04-17 17:31:31.577 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #onsSuite INFO: Build > git hash: f6be02b33d23e5be437bbd1b4055c0f1955e1083 > 04-17 17:31:31.577 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #onsSuite INFO: Build > git describe: RELEASE-0.1.24-47-gf6be02b > 04-17 17:31:31.577 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #onsSuite INFO: Build > project version: 0.1.25.99999 > 04-17 17:31:31.577 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #onsSuite INFO: Built > by: 'michal' > 04-17 17:31:31.577 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #onsSuite INFO: Built > on: '2015-02-05 10:44:16' > 04-17 17:31:31.578 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #onsSuite INFO: Java > availableProcessors: 16 > 04-17 17:31:31.581 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #onsSuite INFO: Java > heap totalMemory: 77.4 MB > 04-17 17:31:31.581 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #onsSuite INFO: Java > heap maxMemory: 910.3 MB > 04-17 17:31:31.581 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #onsSuite INFO: Java > version: Java 1.7.0_25 (from Oracle Corporation) > 04-17 17:31:31.581 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #onsSuite INFO: OS > version: Linux 3.13.0-36-lowlatency (i386) > 04-17 17:31:31.582 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #onsSuite INFO: > Possible IP Address: docker0 (docker0), > fe80:0:0:0:5484:7aff:fefe:9799%4 > 04-17 17:31:31.582 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #onsSuite INFO: > Possible IP Address: docker0 (docker0), 172.17.42.1 > 04-17 17:31:31.582 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #onsSuite INFO: > Possible IP Address: em1 (em1), > fe80:0:0:0:7ae3:b5ff:fe17:8984%2 > 04-17 17:31:31.582 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #onsSuite INFO: > Possible IP Address: em1 (em1), 67.195.81.190 > 04-17 17:31:31.582 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #onsSuite INFO: > Possible IP Address: lo (lo), 0:0:0:0:0:0:0:1%1 > 04-17 17:31:31.582 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #onsSuite INFO: > Possible IP Address: lo (lo), 127.0.0.1 > 04-17 17:31:31.582 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #onsSuite INFO: > Multiple local IPs detected: > 04-17 17:31:31.582 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #onsSuite INFO: > /172.17.42.1 <http://172.17.42.1> /67.195.81.190 > <http://67.195.81.190> > 04-17 17:31:31.582 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #onsSuite INFO: > Attempting to determine correct address... > 04-17 17:31:31.582 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #onsSuite INFO: Using > /67.195.81.190 <http://67.195.81.190> > 04-17 17:31:31.583 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #onsSuite INFO: > Internal communication uses port: 54322 > 04-17 17:31:31.583 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #onsSuite INFO: > Listening for HTTP and REST traffic > onhttp://67.195.81.190:54321/ > 04-17 <http://67.195.81.190:54321/04-17> 17:31:31.583 > 67.195.81.190:54321 <http://67.195.81.190:54321> 17293 > #onsSuite INFO: H2O cloud name: 'mah2out' on > /67.195.81.190:54321 <http://67.195.81.190:54321>, discovery > address /229.95.69.48:58719 <http://229.95.69.48:58719> > 04-17 17:31:31.583 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #onsSuite INFO: If you > have trouble connecting, try SSH tunneling from your local > machine (e.g., via port 55555): > 04-17 17:31:31.583 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #onsSuite INFO: 1. > Open a terminal and run 'ssh -L > 55555:localhost:[email protected] > <mailto:55555%3Alocalhost%[email protected]>' > 04-17 17:31:31.583 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #onsSuite INFO: 2. > Point your browser tohttp://localhost:55555 > 04-17 17:31:31.682 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #onsSuite INFO: Log > dir: '/tmp/h2o-jenkins/h2ologs' > 04-17 17:31:31.682 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #onsSuite INFO: Cur > dir:<https://builds.apache.org/job/Mahout-Quality/ws/h2o> > 04-17 17:31:31.813 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #onsSuite INFO: Cloud > of size 1 formed [/67.195.81.190:54321 > <http://67.195.81.190:54321>] > 04-17 17:31:31.993 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP > Receiver error on port > 54322java.lang.ArrayIndexOutOfBoundsException: -21 > 04-17 17:31:36.009 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP > Receiver error on port > 54322java.lang.ArrayIndexOutOfBoundsException: -17 > 04-17 17:31:37.016 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP > Receiver error on port > 54322java.lang.ArrayIndexOutOfBoundsException: -16 > 04-17 17:31:38.023 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP > Receiver error on port > 54322java.lang.ArrayIndexOutOfBoundsException: -15 > 04-17 17:31:41.038 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP > Receiver error on port > 54322java.lang.ArrayIndexOutOfBoundsException: -12 > 04-17 17:31:42.043 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP > Receiver error on port > 54322java.lang.ArrayIndexOutOfBoundsException: -11 > 04-17 17:31:43.048 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP > Receiver error on port > 54322java.lang.ArrayIndexOutOfBoundsException: -10 > 04-17 17:31:46.061 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP > Receiver error on port > 54322java.lang.ArrayIndexOutOfBoundsException: -7 > 04-17 17:31:47.067 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP > Receiver error on port > 54322java.lang.ArrayIndexOutOfBoundsException: -6 > 04-17 17:31:48.071 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP > Receiver error on port > 54322java.lang.ArrayIndexOutOfBoundsException: -5 > 04-17 17:31:51.081 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP > Receiver error on port > 54322java.lang.ArrayIndexOutOfBoundsException: -2 > 04-17 17:31:52.105 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP > Receiver error on port > 54322java.lang.ArrayIndexOutOfBoundsException: -1 > 04-17 17:31:56.118 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: Out of > Memory and no swap space left from /67.195.81.188:54321 > <http://67.195.81.188:54321> > 04-17 17:31:56.118 67.195.81.190:54321 > <http://67.195.81.190:54321> 17293 #UDP-Recv FATAL: Exiting. > [INFO] > [INFO] > > ------------------------------------------------------------------------ > [INFO] Skipping Apache Mahout > [INFO] This project has been banned from the build due to > previous failures. > [INFO] > > ------------------------------------------------------------------------ > [INFO] > [INFO] > > ------------------------------------------------------------------------ > [INFO] Skipping Mahout Build Tools > [INFO] This project has been banned from the build due to > previous failures. > [INFO] > > ------------------------------------------------------------------------ > [INFO] > > ------------------------------------------------------------------------ > [INFO] Reactor Summary: > [INFO] > [INFO] Mahout Build Tools ................................ > SUCCESS [1.989s] > [INFO] Apache Mahout ..................................... > SUCCESS [0.206s] > [INFO] Mahout Math ....................................... > SUCCESS [1:14.339s] > [INFO] Mahout HDFS ....................................... > SUCCESS [3.378s] > [INFO] Mahout Map-Reduce ................................. > SUCCESS [12:50.243s] > [INFO] Mahout Integration ................................ > SUCCESS [54.793s] > [INFO] Mahout Examples ................................... > SUCCESS [22.309s] > [INFO] Mahout Math Scala bindings ........................ > SUCCESS [1:36.708s] > [INFO] Mahout H2O backend ................................ > FAILURE [43.843s] > [INFO] Mahout Spark bindings ............................. > SKIPPED > [INFO] Mahout Spark bindings shell ....................... > SKIPPED > [INFO] Mahout Release Package ............................ > SKIPPED > [INFO] > > ------------------------------------------------------------------------ > [INFO] BUILD FAILURE > [INFO] > > ------------------------------------------------------------------------ > [INFO] Total time: 17:50.229s > [INFO] Finished at: Fri Apr 17 17:31:56 UTC 2015 > [INFO] Final Memory: 57M/438M > [INFO] > > ------------------------------------------------------------------------ > [ERROR] Failed to execute goal > org.scalatest:scalatest-maven-plugin:1.0:test (test) on > project mahout-h2o_2.10: There are test failures -> [Help 1] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run > Maven with the -e switch. > [ERROR] Re-run Maven using the -X switch to enable full > debug logging. > [ERROR] > [ERROR] For more information about the errors and possible > solutions, please read the following articles: > [ERROR] [Help > 1] > http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException > [ERROR] > [ERROR] After correcting the problems, you can resume the > build with the command > [ERROR] mvn <goals> -rf :mahout-h2o_2.10 > Build step 'Invoke top-level Maven targets' marked build as > failure > [PMD] Skipping publisher since build result is FAILURE > [TASKS] Skipping publisher since build result is FAILURE > > > > >
