Re: Maven out of memory error
Yes. That could be the cause. On Sun, Jan 18, 2015 at 11:47 AM, Sean Owen wrote: > Oh: are you running the tests with a different profile setting than > what the last assembly was built with? this particular test depends on > those matching. Not 100% sure that's the problem, but a good guess. > > On Sat, Jan 17, 2015 at 4:54 PM, Ted Yu wrote: > > The test passed here: > > > > > https://amplab.cs.berkeley.edu/jenkins/view/Spark/job/Spark-Master-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.4,label=centos/1215/consoleFull > > > > It passed locally with the following command: > > > > mvn -DHADOOP_PROFILE=hadoop-2.4 -Phadoop-2.4 -Pyarn -Phive test > > -Dtest=JavaAPISuite > > > > FYI > > > > On Sat, Jan 17, 2015 at 8:23 AM, Andrew Musselman > > wrote: > >> > >> Failing for me and another team member on the command line, for what > it's > >> worth. > >> > >> > On Jan 17, 2015, at 2:39 AM, Sean Owen wrote: > >> > > >> > Hm, this test hangs for me in IntelliJ. It could be a real problem, > >> > and a combination of a) just recently actually enabling Java tests, b) > >> > recent updates to the complicated Guava shading situation. > >> > > >> > The manifestation of the error usually suggests that something totally > >> > failed to start (because of, say, class incompatibility errors, etc.) > >> > Thus things hang and time out waiting for the dead component. It's > >> > sometimes hard to get answers from the embedded component that dies > >> > though. > >> > > >> > That said, it seems to pass on the command line. For example my recent > >> > Jenkins job shows it passes: > >> > > >> > > https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25682/consoleFull > >> > > >> > I'll try to uncover more later this weekend. Thoughts welcome though. > >> > > >> > On Fri, Jan 16, 2015 at 8:26 PM, Andrew Musselman > >> > wrote: > >> >> Thanks Ted, got farther along but now have a failing test; is this a > >> >> known > >> >> issue? > >> >> > >> >> --- > >> >> T E S T S > >> >> --- > >> >> Running org.apache.spark.JavaAPISuite > >> >> Tests run: 72, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: > >> >> 123.462 sec > >> >> <<< FAILURE! - in org.apache.spark.JavaAPISuite > >> >> testGuavaOptional(org.apache.spark.JavaAPISuite) Time elapsed: 106.5 > >> >> sec > >> >> <<< ERROR! > >> >> org.apache.spark.SparkException: Job aborted due to stage failure: > >> >> Master > >> >> removed our application: FAILED > >> >>at > >> >> > >> >> org.apache.spark.scheduler.DAGScheduler.org > $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1199) > >> >>at > >> >> > >> >> > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1188) > >> >>at > >> >> > >> >> > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1187) > >> >>at > >> >> > >> >> > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > >> >>at > >> >> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) > >> >>at > >> >> > >> >> > org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1187) > >> >>at > >> >> > >> >> > org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697) > >> >>at > >> >> > >> >> > org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697) > >> >>at scala.Option.foreach(Option.scala:236) > >> >>at > >> >> > >> >> > org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697) > >> >>at > >> >> > >> >> > org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1399) > >> >>at akka.actor.Actor$class.aroundReceive(Actor.scala:465) > >> >>at > >> >> > >> >> > org.apache.spark.scheduler.DAGSchedulerEventProcessActor.aroundReceive(DAGScheduler.scala:1360) > >> >>at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) > >> >>at akka.actor.ActorCell.invoke(ActorCell.scala:487) > >> >>at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) > >> >>at akka.dispatch.Mailbox.run(Mailbox.scala:220) > >> >>at > >> >> > >> >> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) > >> >>at > >> >> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > >> >>at > >> >> > >> >> > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > >> >>at > >> >> > >> >> > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > >> >>at > >> >> > >> >> > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > >> >> > >> >> Running org.apache.spark.JavaJdbcRDDSuite > >> >> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.846 > >>
Re: Maven out of memory error
Oh: are you running the tests with a different profile setting than what the last assembly was built with? this particular test depends on those matching. Not 100% sure that's the problem, but a good guess. On Sat, Jan 17, 2015 at 4:54 PM, Ted Yu wrote: > The test passed here: > > https://amplab.cs.berkeley.edu/jenkins/view/Spark/job/Spark-Master-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.4,label=centos/1215/consoleFull > > It passed locally with the following command: > > mvn -DHADOOP_PROFILE=hadoop-2.4 -Phadoop-2.4 -Pyarn -Phive test > -Dtest=JavaAPISuite > > FYI > > On Sat, Jan 17, 2015 at 8:23 AM, Andrew Musselman > wrote: >> >> Failing for me and another team member on the command line, for what it's >> worth. >> >> > On Jan 17, 2015, at 2:39 AM, Sean Owen wrote: >> > >> > Hm, this test hangs for me in IntelliJ. It could be a real problem, >> > and a combination of a) just recently actually enabling Java tests, b) >> > recent updates to the complicated Guava shading situation. >> > >> > The manifestation of the error usually suggests that something totally >> > failed to start (because of, say, class incompatibility errors, etc.) >> > Thus things hang and time out waiting for the dead component. It's >> > sometimes hard to get answers from the embedded component that dies >> > though. >> > >> > That said, it seems to pass on the command line. For example my recent >> > Jenkins job shows it passes: >> > >> > https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25682/consoleFull >> > >> > I'll try to uncover more later this weekend. Thoughts welcome though. >> > >> > On Fri, Jan 16, 2015 at 8:26 PM, Andrew Musselman >> > wrote: >> >> Thanks Ted, got farther along but now have a failing test; is this a >> >> known >> >> issue? >> >> >> >> --- >> >> T E S T S >> >> --- >> >> Running org.apache.spark.JavaAPISuite >> >> Tests run: 72, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: >> >> 123.462 sec >> >> <<< FAILURE! - in org.apache.spark.JavaAPISuite >> >> testGuavaOptional(org.apache.spark.JavaAPISuite) Time elapsed: 106.5 >> >> sec >> >> <<< ERROR! >> >> org.apache.spark.SparkException: Job aborted due to stage failure: >> >> Master >> >> removed our application: FAILED >> >>at >> >> >> >> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1199) >> >>at >> >> >> >> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1188) >> >>at >> >> >> >> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1187) >> >>at >> >> >> >> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) >> >>at >> >> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) >> >>at >> >> >> >> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1187) >> >>at >> >> >> >> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697) >> >>at >> >> >> >> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697) >> >>at scala.Option.foreach(Option.scala:236) >> >>at >> >> >> >> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697) >> >>at >> >> >> >> org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1399) >> >>at akka.actor.Actor$class.aroundReceive(Actor.scala:465) >> >>at >> >> >> >> org.apache.spark.scheduler.DAGSchedulerEventProcessActor.aroundReceive(DAGScheduler.scala:1360) >> >>at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) >> >>at akka.actor.ActorCell.invoke(ActorCell.scala:487) >> >>at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) >> >>at akka.dispatch.Mailbox.run(Mailbox.scala:220) >> >>at >> >> >> >> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) >> >>at >> >> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >> >>at >> >> >> >> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) >> >>at >> >> >> >> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) >> >>at >> >> >> >> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) >> >> >> >> Running org.apache.spark.JavaJdbcRDDSuite >> >> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.846 >> >> sec - >> >> in org.apache.spark.JavaJdbcRDDSuite >> >> >> >> Results : >> >> >> >> >> >> Tests in error: >> >> JavaAPISuite.testGuavaOptional » Spark Job aborted due to stage >> >> failure: >> >> Maste... >> >> >> >>> On Fri, Jan 16, 2015 at 12:06 PM, Ted Yu wrote: >> >>> >> >>> Can you try doing this before running mvn ? >> >>> >> >>> export
Re: Maven out of memory error
The test passed here: https://amplab.cs.berkeley.edu/jenkins/view/Spark/job/Spark-Master-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.4,label=centos/1215/consoleFull It passed locally with the following command: mvn -DHADOOP_PROFILE=hadoop-2.4 -Phadoop-2.4 -Pyarn -Phive test -Dtest=JavaAPISuite FYI On Sat, Jan 17, 2015 at 8:23 AM, Andrew Musselman < andrew.mussel...@gmail.com> wrote: > Failing for me and another team member on the command line, for what it's > worth. > > > On Jan 17, 2015, at 2:39 AM, Sean Owen wrote: > > > > Hm, this test hangs for me in IntelliJ. It could be a real problem, > > and a combination of a) just recently actually enabling Java tests, b) > > recent updates to the complicated Guava shading situation. > > > > The manifestation of the error usually suggests that something totally > > failed to start (because of, say, class incompatibility errors, etc.) > > Thus things hang and time out waiting for the dead component. It's > > sometimes hard to get answers from the embedded component that dies > > though. > > > > That said, it seems to pass on the command line. For example my recent > > Jenkins job shows it passes: > > > https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25682/consoleFull > > > > I'll try to uncover more later this weekend. Thoughts welcome though. > > > > On Fri, Jan 16, 2015 at 8:26 PM, Andrew Musselman > > wrote: > >> Thanks Ted, got farther along but now have a failing test; is this a > known > >> issue? > >> > >> --- > >> T E S T S > >> --- > >> Running org.apache.spark.JavaAPISuite > >> Tests run: 72, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: > 123.462 sec > >> <<< FAILURE! - in org.apache.spark.JavaAPISuite > >> testGuavaOptional(org.apache.spark.JavaAPISuite) Time elapsed: 106.5 > sec > >> <<< ERROR! > >> org.apache.spark.SparkException: Job aborted due to stage failure: > Master > >> removed our application: FAILED > >>at > >> org.apache.spark.scheduler.DAGScheduler.org > $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1199) > >>at > >> > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1188) > >>at > >> > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1187) > >>at > >> > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > >>at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) > >>at > >> > org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1187) > >>at > >> > org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697) > >>at > >> > org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697) > >>at scala.Option.foreach(Option.scala:236) > >>at > >> > org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697) > >>at > >> > org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1399) > >>at akka.actor.Actor$class.aroundReceive(Actor.scala:465) > >>at > >> > org.apache.spark.scheduler.DAGSchedulerEventProcessActor.aroundReceive(DAGScheduler.scala:1360) > >>at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) > >>at akka.actor.ActorCell.invoke(ActorCell.scala:487) > >>at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) > >>at akka.dispatch.Mailbox.run(Mailbox.scala:220) > >>at > >> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) > >>at > scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > >>at > >> > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > >>at > >> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > >>at > >> > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > >> > >> Running org.apache.spark.JavaJdbcRDDSuite > >> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.846 > sec - > >> in org.apache.spark.JavaJdbcRDDSuite > >> > >> Results : > >> > >> > >> Tests in error: > >> JavaAPISuite.testGuavaOptional » Spark Job aborted due to stage > failure: > >> Maste... > >> > >>> On Fri, Jan 16, 2015 at 12:06 PM, Ted Yu wrote: > >>> > >>> Can you try doing this before running mvn ? > >>> > >>> export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M > >>> -XX:ReservedCodeCacheSize=512m" > >>> > >>> What OS are you using ? > >>> > >>> Cheers > >>> > >>> On Fri, Jan 16, 2015 at 12:03 PM, Andrew Musselman > >>> wrote: > > Just got the latest from Github and tried running `mvn test`; is this > error common and do you have any advice on fixing it? > > Thanks! > > [INFO] --- scala-maven-plugin:3.2.0:compil
Re: Maven out of memory error
Failing for me and another team member on the command line, for what it's worth. > On Jan 17, 2015, at 2:39 AM, Sean Owen wrote: > > Hm, this test hangs for me in IntelliJ. It could be a real problem, > and a combination of a) just recently actually enabling Java tests, b) > recent updates to the complicated Guava shading situation. > > The manifestation of the error usually suggests that something totally > failed to start (because of, say, class incompatibility errors, etc.) > Thus things hang and time out waiting for the dead component. It's > sometimes hard to get answers from the embedded component that dies > though. > > That said, it seems to pass on the command line. For example my recent > Jenkins job shows it passes: > https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25682/consoleFull > > I'll try to uncover more later this weekend. Thoughts welcome though. > > On Fri, Jan 16, 2015 at 8:26 PM, Andrew Musselman > wrote: >> Thanks Ted, got farther along but now have a failing test; is this a known >> issue? >> >> --- >> T E S T S >> --- >> Running org.apache.spark.JavaAPISuite >> Tests run: 72, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 123.462 sec >> <<< FAILURE! - in org.apache.spark.JavaAPISuite >> testGuavaOptional(org.apache.spark.JavaAPISuite) Time elapsed: 106.5 sec >> <<< ERROR! >> org.apache.spark.SparkException: Job aborted due to stage failure: Master >> removed our application: FAILED >>at >> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1199) >>at >> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1188) >>at >> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1187) >>at >> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) >>at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) >>at >> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1187) >>at >> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697) >>at >> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697) >>at scala.Option.foreach(Option.scala:236) >>at >> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697) >>at >> org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1399) >>at akka.actor.Actor$class.aroundReceive(Actor.scala:465) >>at >> org.apache.spark.scheduler.DAGSchedulerEventProcessActor.aroundReceive(DAGScheduler.scala:1360) >>at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) >>at akka.actor.ActorCell.invoke(ActorCell.scala:487) >>at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) >>at akka.dispatch.Mailbox.run(Mailbox.scala:220) >>at >> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) >>at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >>at >> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) >>at >> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) >>at >> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) >> >> Running org.apache.spark.JavaJdbcRDDSuite >> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.846 sec - >> in org.apache.spark.JavaJdbcRDDSuite >> >> Results : >> >> >> Tests in error: >> JavaAPISuite.testGuavaOptional » Spark Job aborted due to stage failure: >> Maste... >> >>> On Fri, Jan 16, 2015 at 12:06 PM, Ted Yu wrote: >>> >>> Can you try doing this before running mvn ? >>> >>> export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M >>> -XX:ReservedCodeCacheSize=512m" >>> >>> What OS are you using ? >>> >>> Cheers >>> >>> On Fri, Jan 16, 2015 at 12:03 PM, Andrew Musselman >>> wrote: Just got the latest from Github and tried running `mvn test`; is this error common and do you have any advice on fixing it? Thanks! [INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile-first) @ spark-core_2.10 --- [WARNING] Zinc server is not available at port 3030 - reverting to normal incremental compile [INFO] Using incremental compilation [INFO] compiler plugin: BasicArtifact(org.scalamacros,paradise_2.10.4,2.0.1,null) [INFO] Compiling 400 Scala sources and 34 Java sources to /home/akm/spark/core/target/scala-2.10/classes... [WARNING] /home/akm/spark/core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala:22: imported `DataReadMethod' is permanently hidden by definition of object DataReadMethod in package
Re: Maven out of memory error
Hm, this test hangs for me in IntelliJ. It could be a real problem, and a combination of a) just recently actually enabling Java tests, b) recent updates to the complicated Guava shading situation. The manifestation of the error usually suggests that something totally failed to start (because of, say, class incompatibility errors, etc.) Thus things hang and time out waiting for the dead component. It's sometimes hard to get answers from the embedded component that dies though. That said, it seems to pass on the command line. For example my recent Jenkins job shows it passes: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25682/consoleFull I'll try to uncover more later this weekend. Thoughts welcome though. On Fri, Jan 16, 2015 at 8:26 PM, Andrew Musselman wrote: > Thanks Ted, got farther along but now have a failing test; is this a known > issue? > > --- > T E S T S > --- > Running org.apache.spark.JavaAPISuite > Tests run: 72, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 123.462 sec > <<< FAILURE! - in org.apache.spark.JavaAPISuite > testGuavaOptional(org.apache.spark.JavaAPISuite) Time elapsed: 106.5 sec > <<< ERROR! > org.apache.spark.SparkException: Job aborted due to stage failure: Master > removed our application: FAILED > at > org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1199) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1188) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1187) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) > at > org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1187) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697) > at scala.Option.foreach(Option.scala:236) > at > org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697) > at > org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1399) > at akka.actor.Actor$class.aroundReceive(Actor.scala:465) > at > org.apache.spark.scheduler.DAGSchedulerEventProcessActor.aroundReceive(DAGScheduler.scala:1360) > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) > at akka.actor.ActorCell.invoke(ActorCell.scala:487) > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) > at akka.dispatch.Mailbox.run(Mailbox.scala:220) > at > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) > at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > at > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > > Running org.apache.spark.JavaJdbcRDDSuite > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.846 sec - > in org.apache.spark.JavaJdbcRDDSuite > > Results : > > > Tests in error: > JavaAPISuite.testGuavaOptional » Spark Job aborted due to stage failure: > Maste... > > On Fri, Jan 16, 2015 at 12:06 PM, Ted Yu wrote: >> >> Can you try doing this before running mvn ? >> >> export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M >> -XX:ReservedCodeCacheSize=512m" >> >> What OS are you using ? >> >> Cheers >> >> On Fri, Jan 16, 2015 at 12:03 PM, Andrew Musselman >> wrote: >>> >>> Just got the latest from Github and tried running `mvn test`; is this >>> error common and do you have any advice on fixing it? >>> >>> Thanks! >>> >>> [INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile-first) @ >>> spark-core_2.10 --- >>> [WARNING] Zinc server is not available at port 3030 - reverting to normal >>> incremental compile >>> [INFO] Using incremental compilation >>> [INFO] compiler plugin: >>> BasicArtifact(org.scalamacros,paradise_2.10.4,2.0.1,null) >>> [INFO] Compiling 400 Scala sources and 34 Java sources to >>> /home/akm/spark/core/target/scala-2.10/classes... >>> [WARNING] >>> /home/akm/spark/core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala:22: >>> imported `DataReadMethod' is permanently hidden by definition of object >>> DataReadMethod in package executor >>> [WARNING] import org.apache.spark.executor.DataReadMethod >>> [WARNING] ^ >>> [WARNING] >>> /home/akm/spark/core/src/main/scala/org/apache/spark/TaskState.scala:41: >>> match may not be exhaustive. >>> It wou
Re: Maven out of memory error
I tried the following but still didn't see test output :-( diff --git a/pom.xml b/pom.xml index f4466e5..dae2ae8 100644 --- a/pom.xml +++ b/pom.xml @@ -1131,6 +1131,7 @@ true false +true On Fri, Jan 16, 2015 at 12:41 PM, Ted Yu wrote: > I got the same error: > > testGuavaOptional(org.apache.spark.JavaAPISuite) Time elapsed: 261.111 > sec <<< ERROR! > org.apache.spark.SparkException: Job aborted due to stage failure: Master > removed our application: FAILED > at org.apache.spark.scheduler.DAGScheduler.org > $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1199) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1188) > > Looking under ore/target/surefire-reports/ , I don't see test output. > Trying to figure out how test output can be generated. > > Cheers > > On Fri, Jan 16, 2015 at 12:26 PM, Andrew Musselman < > andrew.mussel...@gmail.com> wrote: > >> Thanks Ted, got farther along but now have a failing test; is this a >> known issue? >> >> --- >> T E S T S >> --- >> Running org.apache.spark.JavaAPISuite >> Tests run: 72, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 123.462 >> sec <<< FAILURE! - in org.apache.spark.JavaAPISuite >> testGuavaOptional(org.apache.spark.JavaAPISuite) Time elapsed: 106.5 >> sec <<< ERROR! >> org.apache.spark.SparkException: Job aborted due to stage failure: Master >> removed our application: FAILED >> at org.apache.spark.scheduler.DAGScheduler.org >> $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1199) >> at >> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1188) >> at >> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1187) >> at >> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) >> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) >> at >> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1187) >> at >> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697) >> at >> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697) >> at scala.Option.foreach(Option.scala:236) >> at >> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697) >> at >> org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1399) >> at akka.actor.Actor$class.aroundReceive(Actor.scala:465) >> at >> org.apache.spark.scheduler.DAGSchedulerEventProcessActor.aroundReceive(DAGScheduler.scala:1360) >> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) >> at akka.actor.ActorCell.invoke(ActorCell.scala:487) >> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) >> at akka.dispatch.Mailbox.run(Mailbox.scala:220) >> at >> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) >> at >> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >> at >> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) >> at >> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) >> at >> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) >> >> Running org.apache.spark.JavaJdbcRDDSuite >> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.846 sec >> - in org.apache.spark.JavaJdbcRDDSuite >> >> Results : >> >> >> Tests in error: >> JavaAPISuite.testGuavaOptional » Spark Job aborted due to stage >> failure: Maste... >> >> On Fri, Jan 16, 2015 at 12:06 PM, Ted Yu wrote: >> >>> Can you try doing this before running mvn ? >>> >>> export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M >>> -XX:ReservedCodeCacheSize=512m" >>> >>> What OS are you using ? >>> >>> Cheers >>> >>> On Fri, Jan 16, 2015 at 12:03 PM, Andrew Musselman < >>> andrew.mussel...@gmail.com> wrote: >>> Just got the latest from Github and tried running `mvn test`; is this error common and do you have any advice on fixing it? Thanks! [INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile-first) @ spark-core_2.10 --- [WARNING] Zinc server is not available at port 3030 - reverting to normal incremental compile [INFO] Using incremental compilation [INFO] compiler plugin: BasicArtifact(org.scalamacros,paradise_2.10.4,2.0.1,null) [INFO] Compiling 400 Scala sources and 34 Java sources to /home/akm/spark/core/target/scala-2.10/classes... [WARNING] /home/akm/spark/core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala:22: imported `D
Re: Maven out of memory error
I got the same error: testGuavaOptional(org.apache.spark.JavaAPISuite) Time elapsed: 261.111 sec <<< ERROR! org.apache.spark.SparkException: Job aborted due to stage failure: Master removed our application: FAILED at org.apache.spark.scheduler.DAGScheduler.org $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1199) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1188) Looking under ore/target/surefire-reports/ , I don't see test output. Trying to figure out how test output can be generated. Cheers On Fri, Jan 16, 2015 at 12:26 PM, Andrew Musselman < andrew.mussel...@gmail.com> wrote: > Thanks Ted, got farther along but now have a failing test; is this a known > issue? > > --- > T E S T S > --- > Running org.apache.spark.JavaAPISuite > Tests run: 72, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 123.462 > sec <<< FAILURE! - in org.apache.spark.JavaAPISuite > testGuavaOptional(org.apache.spark.JavaAPISuite) Time elapsed: 106.5 sec > <<< ERROR! > org.apache.spark.SparkException: Job aborted due to stage failure: Master > removed our application: FAILED > at org.apache.spark.scheduler.DAGScheduler.org > $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1199) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1188) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1187) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) > at > org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1187) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697) > at scala.Option.foreach(Option.scala:236) > at > org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697) > at > org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1399) > at akka.actor.Actor$class.aroundReceive(Actor.scala:465) > at > org.apache.spark.scheduler.DAGSchedulerEventProcessActor.aroundReceive(DAGScheduler.scala:1360) > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) > at akka.actor.ActorCell.invoke(ActorCell.scala:487) > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) > at akka.dispatch.Mailbox.run(Mailbox.scala:220) > at > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) > at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > at > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > > Running org.apache.spark.JavaJdbcRDDSuite > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.846 sec > - in org.apache.spark.JavaJdbcRDDSuite > > Results : > > > Tests in error: > JavaAPISuite.testGuavaOptional » Spark Job aborted due to stage failure: > Maste... > > On Fri, Jan 16, 2015 at 12:06 PM, Ted Yu wrote: > >> Can you try doing this before running mvn ? >> >> export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M >> -XX:ReservedCodeCacheSize=512m" >> >> What OS are you using ? >> >> Cheers >> >> On Fri, Jan 16, 2015 at 12:03 PM, Andrew Musselman < >> andrew.mussel...@gmail.com> wrote: >> >>> Just got the latest from Github and tried running `mvn test`; is this >>> error common and do you have any advice on fixing it? >>> >>> Thanks! >>> >>> [INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile-first) @ >>> spark-core_2.10 --- >>> [WARNING] Zinc server is not available at port 3030 - reverting to >>> normal incremental compile >>> [INFO] Using incremental compilation >>> [INFO] compiler plugin: >>> BasicArtifact(org.scalamacros,paradise_2.10.4,2.0.1,null) >>> [INFO] Compiling 400 Scala sources and 34 Java sources to >>> /home/akm/spark/core/target/scala-2.10/classes... >>> [WARNING] >>> /home/akm/spark/core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala:22: >>> imported `DataReadMethod' is permanently hidden by definition of object >>> DataReadMethod in package executor >>> [WARNING] import org.apache.spark.executor.DataReadMethod >>> [WARNING] ^ >>> [WARNING] >>> /home/akm/spark/core/src/main/scala/org/apache/spark/TaskState.scala:41: >>> match may not be exhaustive. >>> It would fail on the following input: TASK_ERROR >>> [WARNING] def fromMesos(mesosState: MesosTaskSt
Re: Maven out of memory error
Thanks Sean
On Fri, Jan 16, 2015 at 12:06 PM, Sean Owen wrote:
> Hey Andrew, you'll want to have a look at the Spark docs on building:
> http://spark.apache.org/docs/latest/building-spark.html
>
> It's the first thing covered there.
>
> The warnings are normal as you are probably building with newer Hadoop
> profiles and so old-Hadoop support code shows deprecation warnings on
> its use of old APIs.
>
> On Fri, Jan 16, 2015 at 8:03 PM, Andrew Musselman
> wrote:
> > Just got the latest from Github and tried running `mvn test`; is this
> error
> > common and do you have any advice on fixing it?
> >
> > Thanks!
> >
> > [INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile-first) @
> > spark-core_2.10 ---
> > [WARNING] Zinc server is not available at port 3030 - reverting to normal
> > incremental compile
> > [INFO] Using incremental compilation
> > [INFO] compiler plugin:
> > BasicArtifact(org.scalamacros,paradise_2.10.4,2.0.1,null)
> > [INFO] Compiling 400 Scala sources and 34 Java sources to
> > /home/akm/spark/core/target/scala-2.10/classes...
> > [WARNING]
> >
> /home/akm/spark/core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala:22:
> > imported `DataReadMethod' is permanently hidden by definition of object
> > DataReadMethod in package executor
> > [WARNING] import org.apache.spark.executor.DataReadMethod
> > [WARNING] ^
> > [WARNING]
> > /home/akm/spark/core/src/main/scala/org/apache/spark/TaskState.scala:41:
> > match may not be exhaustive.
> > It would fail on the following input: TASK_ERROR
> > [WARNING] def fromMesos(mesosState: MesosTaskState): TaskState =
> > mesosState match {
> > [WARNING] ^
> > [WARNING]
> >
> /home/akm/spark/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:89:
> > method isDirectory in class FileSystem is deprecated: see corresponding
> > Javadoc for more information.
> > [WARNING] if (!fileSystem.isDirectory(new Path(logBaseDir))) {
> > [WARNING] ^
> > [ERROR] PermGen space -> [Help 1]
> > [ERROR]
> > [ERROR] To see the full stack trace of the errors, re-run Maven with the
> -e
> > switch.
> > [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> > [ERROR]
> > [ERROR] For more information about the errors and possible solutions,
> please
> > read the following articles:
> > [ERROR] [Help 1]
> > http://cwiki.apache.org/confluence/display/MAVEN/OutOfMemoryError
> >
>
Re: Maven out of memory error
Thanks Ted, got farther along but now have a failing test; is this a known
issue?
---
T E S T S
---
Running org.apache.spark.JavaAPISuite
Tests run: 72, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 123.462
sec <<< FAILURE! - in org.apache.spark.JavaAPISuite
testGuavaOptional(org.apache.spark.JavaAPISuite) Time elapsed: 106.5 sec
<<< ERROR!
org.apache.spark.SparkException: Job aborted due to stage failure: Master
removed our application: FAILED
at org.apache.spark.scheduler.DAGScheduler.org
$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1199)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1188)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1187)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1187)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
at scala.Option.foreach(Option.scala:236)
at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1399)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessActor.aroundReceive(DAGScheduler.scala:1360)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Running org.apache.spark.JavaJdbcRDDSuite
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.846 sec -
in org.apache.spark.JavaJdbcRDDSuite
Results :
Tests in error:
JavaAPISuite.testGuavaOptional » Spark Job aborted due to stage failure:
Maste...
On Fri, Jan 16, 2015 at 12:06 PM, Ted Yu wrote:
> Can you try doing this before running mvn ?
>
> export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M
> -XX:ReservedCodeCacheSize=512m"
>
> What OS are you using ?
>
> Cheers
>
> On Fri, Jan 16, 2015 at 12:03 PM, Andrew Musselman <
> andrew.mussel...@gmail.com> wrote:
>
>> Just got the latest from Github and tried running `mvn test`; is this
>> error common and do you have any advice on fixing it?
>>
>> Thanks!
>>
>> [INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile-first) @
>> spark-core_2.10 ---
>> [WARNING] Zinc server is not available at port 3030 - reverting to normal
>> incremental compile
>> [INFO] Using incremental compilation
>> [INFO] compiler plugin:
>> BasicArtifact(org.scalamacros,paradise_2.10.4,2.0.1,null)
>> [INFO] Compiling 400 Scala sources and 34 Java sources to
>> /home/akm/spark/core/target/scala-2.10/classes...
>> [WARNING]
>> /home/akm/spark/core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala:22:
>> imported `DataReadMethod' is permanently hidden by definition of object
>> DataReadMethod in package executor
>> [WARNING] import org.apache.spark.executor.DataReadMethod
>> [WARNING] ^
>> [WARNING]
>> /home/akm/spark/core/src/main/scala/org/apache/spark/TaskState.scala:41:
>> match may not be exhaustive.
>> It would fail on the following input: TASK_ERROR
>> [WARNING] def fromMesos(mesosState: MesosTaskState): TaskState =
>> mesosState match {
>> [WARNING] ^
>> [WARNING]
>> /home/akm/spark/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:89:
>> method isDirectory in class FileSystem is deprecated: see corresponding
>> Javadoc for more information.
>> [WARNING] if (!fileSystem.isDirectory(new Path(logBaseDir))) {
>> [WARNING] ^
>> [ERROR] PermGen space -> [Help 1]
>> [ERROR]
>> [ERROR] To see the full stack trace of the errors, re-run Maven with the
>> -e switch.
>> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
>> [ERROR]
>> [ERROR] For more information about the errors and possible solutions,
>> please read the following articles:
>> [ERROR] [Help 1]
>> http://cwiki.apache.org/confluence/dis
Re: Maven out of memory error
Can you try doing this before running mvn ?
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M
-XX:ReservedCodeCacheSize=512m"
What OS are you using ?
Cheers
On Fri, Jan 16, 2015 at 12:03 PM, Andrew Musselman <
andrew.mussel...@gmail.com> wrote:
> Just got the latest from Github and tried running `mvn test`; is this
> error common and do you have any advice on fixing it?
>
> Thanks!
>
> [INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile-first) @
> spark-core_2.10 ---
> [WARNING] Zinc server is not available at port 3030 - reverting to normal
> incremental compile
> [INFO] Using incremental compilation
> [INFO] compiler plugin:
> BasicArtifact(org.scalamacros,paradise_2.10.4,2.0.1,null)
> [INFO] Compiling 400 Scala sources and 34 Java sources to
> /home/akm/spark/core/target/scala-2.10/classes...
> [WARNING]
> /home/akm/spark/core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala:22:
> imported `DataReadMethod' is permanently hidden by definition of object
> DataReadMethod in package executor
> [WARNING] import org.apache.spark.executor.DataReadMethod
> [WARNING] ^
> [WARNING]
> /home/akm/spark/core/src/main/scala/org/apache/spark/TaskState.scala:41:
> match may not be exhaustive.
> It would fail on the following input: TASK_ERROR
> [WARNING] def fromMesos(mesosState: MesosTaskState): TaskState =
> mesosState match {
> [WARNING] ^
> [WARNING]
> /home/akm/spark/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:89:
> method isDirectory in class FileSystem is deprecated: see corresponding
> Javadoc for more information.
> [WARNING] if (!fileSystem.isDirectory(new Path(logBaseDir))) {
> [WARNING] ^
> [ERROR] PermGen space -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the
> -e switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions,
> please read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/OutOfMemoryError
>
>
Re: Maven out of memory error
Hey Andrew, you'll want to have a look at the Spark docs on building:
http://spark.apache.org/docs/latest/building-spark.html
It's the first thing covered there.
The warnings are normal as you are probably building with newer Hadoop
profiles and so old-Hadoop support code shows deprecation warnings on
its use of old APIs.
On Fri, Jan 16, 2015 at 8:03 PM, Andrew Musselman
wrote:
> Just got the latest from Github and tried running `mvn test`; is this error
> common and do you have any advice on fixing it?
>
> Thanks!
>
> [INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile-first) @
> spark-core_2.10 ---
> [WARNING] Zinc server is not available at port 3030 - reverting to normal
> incremental compile
> [INFO] Using incremental compilation
> [INFO] compiler plugin:
> BasicArtifact(org.scalamacros,paradise_2.10.4,2.0.1,null)
> [INFO] Compiling 400 Scala sources and 34 Java sources to
> /home/akm/spark/core/target/scala-2.10/classes...
> [WARNING]
> /home/akm/spark/core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala:22:
> imported `DataReadMethod' is permanently hidden by definition of object
> DataReadMethod in package executor
> [WARNING] import org.apache.spark.executor.DataReadMethod
> [WARNING] ^
> [WARNING]
> /home/akm/spark/core/src/main/scala/org/apache/spark/TaskState.scala:41:
> match may not be exhaustive.
> It would fail on the following input: TASK_ERROR
> [WARNING] def fromMesos(mesosState: MesosTaskState): TaskState =
> mesosState match {
> [WARNING] ^
> [WARNING]
> /home/akm/spark/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:89:
> method isDirectory in class FileSystem is deprecated: see corresponding
> Javadoc for more information.
> [WARNING] if (!fileSystem.isDirectory(new Path(logBaseDir))) {
> [WARNING] ^
> [ERROR] PermGen space -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/OutOfMemoryError
>
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
Maven out of memory error
Just got the latest from Github and tried running `mvn test`; is this error
common and do you have any advice on fixing it?
Thanks!
[INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile-first) @
spark-core_2.10 ---
[WARNING] Zinc server is not available at port 3030 - reverting to normal
incremental compile
[INFO] Using incremental compilation
[INFO] compiler plugin:
BasicArtifact(org.scalamacros,paradise_2.10.4,2.0.1,null)
[INFO] Compiling 400 Scala sources and 34 Java sources to
/home/akm/spark/core/target/scala-2.10/classes...
[WARNING]
/home/akm/spark/core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala:22:
imported `DataReadMethod' is permanently hidden by definition of object
DataReadMethod in package executor
[WARNING] import org.apache.spark.executor.DataReadMethod
[WARNING] ^
[WARNING]
/home/akm/spark/core/src/main/scala/org/apache/spark/TaskState.scala:41:
match may not be exhaustive.
It would fail on the following input: TASK_ERROR
[WARNING] def fromMesos(mesosState: MesosTaskState): TaskState =
mesosState match {
[WARNING] ^
[WARNING]
/home/akm/spark/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:89:
method isDirectory in class FileSystem is deprecated: see corresponding
Javadoc for more information.
[WARNING] if (!fileSystem.isDirectory(new Path(logBaseDir))) {
[WARNING] ^
[ERROR] PermGen space -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions,
please read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/OutOfMemoryError
