Re: Pls assist: Spark 2.0 build failure on Ubuntu 16.06
Hi Sean thanks. I managed to build Spark 2 (it was actually 2.1, not 2.0...i m sourcing it from here (git clone git://github.com/apache/spark.git)) Now, i managed to build it but i had to - use Java 1.7 along with MAVEN_OPTS (using java1.8 send the whole process into insufficient memory for JVM while compiling spark-core) - use Scala 2.10, installed on my host (when i tried with Scala 2.11, i got same results as with java 1.8) . kr marco On Sat, Oct 1, 2016 at 9:29 PM, Sean Owenwrote: > "Compile failed via zinc server" > > Try shutting down zinc. Something's funny about your compile server. > It's not required anyway. > > On Sat, Oct 1, 2016 at 3:24 PM, Marco Mistroni > wrote: > > Hi guys > > sorry to annoy you on this but i am getting nowhere. So far i have > tried to > > build spark 2.0 on my local laptop with no success so i blamed my > > laptop poor performance > > So today i fired off an EC2 Ubuntu 16.06 Instance and installed the > > following (i copy paste commands here) > > > > ubuntu@ip-172-31-40-104:~/spark$ history > > sudo apt-get install -y python-software-properties > > sudo apt-get install -y software-properties-common > > sudo add-apt-repository -y ppa:webupd8team/java > > sudo apt-get install -y oracle-java8-installer > > sudo apt-get install -y git > > git clone git://github.com/apache/spark.git > > cd spark > > > > Then launched the following commands: > > First thi sone, with only yarn, as i dont need hadoop > > > > ./build/mvn -Pyarn -DskipTests clean package > > > > This failed. after kicking off same command with -X i got this > > > > ERROR] Failed to execute goal > > net.alchim31.maven:scala-maven-plugin:3.2.2:compile > (scala-compile-first) on > > project spark-core_2.11: Execution scala-compile-first of goal > > net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed. > CompileFailed -> > > [Help 1] > > org.apache.maven.lifecycle.LifecycleExecutionException: Failed to > execute > > goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile > > (scala-compile-first) on project spark-core_2.11: Execution > > scala-compile-first of goal > > net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed. > > at > > org.apache.maven.lifecycle.internal.MojoExecutor.execute( > MojoExecutor.java:212) > > at > > org.apache.maven.lifecycle.internal.MojoExecutor.execute( > MojoExecutor.java:153) > > at > > org.apache.maven.lifecycle.internal.MojoExecutor.execute( > MojoExecutor.java:145) > > at > > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject( > LifecycleModuleBuilder.java:116) > > at > > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject( > LifecycleModuleBuilder.java:80) > > at > > org.apache.maven.lifecycle.internal.builder.singlethreaded. > SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) > > at > > org.apache.maven.lifecycle.internal.LifecycleStarter. > execute(LifecycleStarter.java:128) > > at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java: > 307) > > at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java: > 193) > > at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106) > > at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863) > > at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288) > > at org.apache.maven.cli.MavenCli.main(MavenCli.java:199) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > sun.reflect.NativeMethodAccessorImpl.invoke( > NativeMethodAccessorImpl.java:62) > > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at > > org.codehaus.plexus.classworlds.launcher.Launcher. > launchEnhanced(Launcher.java:289) > > at > > org.codehaus.plexus.classworlds.launcher.Launcher. > launch(Launcher.java:229) > > at > > org.codehaus.plexus.classworlds.launcher.Launcher. > mainWithExitCode(Launcher.java:415) > > at > > org.codehaus.plexus.classworlds.launcher.Launcher. > main(Launcher.java:356) > > Caused by: org.apache.maven.plugin.PluginExecutionException: Execution > > scala-compile-first of goal > > net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed. > > at > > org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo( > DefaultBuildPluginManager.java:145) > > at > > org.apache.maven.lifecycle.internal.MojoExecutor.execute( > MojoExecutor.java:207) > > ... 20 more > > Caused by: Compile failed via zinc server > > at > > sbt_inc.SbtIncrementalCompiler.zincCompile(SbtIncrementalCompiler.java: > 136) > > at > > sbt_inc.SbtIncrementalCompiler.compile(SbtIncrementalCompiler.java:86) > > at > > scala_maven.ScalaCompilerSupport.incrementalCompile( >
Re: Pls assist: Spark 2.0 build failure on Ubuntu 16.06
"Compile failed via zinc server" Try shutting down zinc. Something's funny about your compile server. It's not required anyway. On Sat, Oct 1, 2016 at 3:24 PM, Marco Mistroniwrote: > Hi guys > sorry to annoy you on this but i am getting nowhere. So far i have tried to > build spark 2.0 on my local laptop with no success so i blamed my > laptop poor performance > So today i fired off an EC2 Ubuntu 16.06 Instance and installed the > following (i copy paste commands here) > > ubuntu@ip-172-31-40-104:~/spark$ history > sudo apt-get install -y python-software-properties > sudo apt-get install -y software-properties-common > sudo add-apt-repository -y ppa:webupd8team/java > sudo apt-get install -y oracle-java8-installer > sudo apt-get install -y git > git clone git://github.com/apache/spark.git > cd spark > > Then launched the following commands: > First thi sone, with only yarn, as i dont need hadoop > > ./build/mvn -Pyarn -DskipTests clean package > > This failed. after kicking off same command with -X i got this > > ERROR] Failed to execute goal > net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first) on > project spark-core_2.11: Execution scala-compile-first of goal > net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed. CompileFailed -> > [Help 1] > org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute > goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile > (scala-compile-first) on project spark-core_2.11: Execution > scala-compile-first of goal > net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed. > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116) > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80) > at > org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) > at > org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128) > at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307) > at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193) > at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106) > at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863) > at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288) > at org.apache.maven.cli.MavenCli.main(MavenCli.java:199) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) > at > org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) > at > org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) > at > org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) > Caused by: org.apache.maven.plugin.PluginExecutionException: Execution > scala-compile-first of goal > net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed. > at > org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:145) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207) > ... 20 more > Caused by: Compile failed via zinc server > at > sbt_inc.SbtIncrementalCompiler.zincCompile(SbtIncrementalCompiler.java:136) > at > sbt_inc.SbtIncrementalCompiler.compile(SbtIncrementalCompiler.java:86) > at > scala_maven.ScalaCompilerSupport.incrementalCompile(ScalaCompilerSupport.java:303) > at > scala_maven.ScalaCompilerSupport.compile(ScalaCompilerSupport.java:119) > at > scala_maven.ScalaCompilerSupport.doExecute(ScalaCompilerSupport.java:99) > at scala_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java:482) > at > org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) > ... 21 more > > > > Then i tried to use hadoop even though i dont need hadoop for my code > > ./build/mvn -Pyarn -Phadoop-2.4 -DskipTests clean package > > This failed again, at exactly the same point, with the same error > > Then i thought maybe i used an old version of hadoop, so i tried to use 2.7 > > ./build/mvn -Pyarn -Phadoop-2.7 -DskipTests clean
Re: Pls assist: Spark 2.0 build failure on Ubuntu 16.06
Hi guys sorry to annoy you on this but i am getting nowhere. So far i have tried to build spark 2.0 on my local laptop with no success so i blamed my laptop poor performance So today i fired off an EC2 Ubuntu 16.06 Instance and installed the following (i copy paste commands here) ubuntu@ip-172-31-40-104:~/spark$ history sudo apt-get install -y python-software-properties sudo apt-get install -y software-properties-common sudo add-apt-repository -y ppa:webupd8team/java sudo apt-get install -y oracle-java8-installer sudo apt-get install -y git git clone git://github.com/apache/spark.git cd spark Then launched the following commands: First thi sone, with only yarn, as i dont need hadoop ./build/mvn -Pyarn -DskipTests clean package This failed. after kicking off same command with -X i got this ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first) on project spark-core_2.11: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed. CompileFailed -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first) on project spark-core_2.11: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed. at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288) at org.apache.maven.cli.MavenCli.main(MavenCli.java:199) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) Caused by: org.apache.maven.plugin.PluginExecutionException: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed. at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:145) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207) ... 20 more Caused by: Compile failed via zinc server at sbt_inc.SbtIncrementalCompiler.zincCompile(SbtIncrementalCompiler.java:136) at sbt_inc.SbtIncrementalCompiler.compile(SbtIncrementalCompiler.java:86) at scala_maven.ScalaCompilerSupport.incrementalCompile(ScalaCompilerSupport.java:303) at scala_maven.ScalaCompilerSupport.compile(ScalaCompilerSupport.java:119) at scala_maven.ScalaCompilerSupport.doExecute(ScalaCompilerSupport.java:99) at scala_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java:482) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) ... 21 more Then i tried to use hadoop even though i dont need hadoop for my code ./build/mvn -Pyarn -Phadoop-2.4 -DskipTests clean package This failed again, at exactly the same point, with the same error Then i thought maybe i used an old version of hadoop, so i tried to use 2.7 ./build/mvn -Pyarn -Phadoop-2.7 -DskipTests clean package Nada. same error. I have tried sbt, no luck Could anyone suggest me a mvn proflie to use to build spark 2.0? i am starting to suspect (before arguing that there's something wrong with spark git) that i am using somehow wrong parameters. or perhaps i should install scala 2.11 before i install spark? or Maven ? kr marco On Fri, Sep 30, 2016 at 8:23 PM, Marco Mistroni
Re: Pls assist: Spark 2.0 build failure on Ubuntu 16.06
Hi all this problem is still bothering me. Here's my setup - Ubuntu 16.06 - Java 8 - Spark 2.0 - have launched following command: ./build/mvn -X -Pyarn -Phadoop-2.7 -DskipTests clean package and i am gettign this exception: org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) on project spark-mllib_2.10: Execution scala-test-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile failed. at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288) at org.apache.maven.cli.MavenCli.main(MavenCli.java:199) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) Caused by: org.apache.maven.plugin.PluginExecutionException: Execution scala-test-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile failed. at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:145) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207) ... 20 more Caused by: Compile failed via zinc server at sbt_inc.SbtIncrementalCompiler.zincCompile(SbtIncrementalCompiler.java:136) at sbt_inc.SbtIncrementalCompiler.compile(SbtIncrementalCompiler.java:86) at scala_maven.ScalaCompilerSupport.incrementalCompile(ScalaCompilerSupport.java:303) at scala_maven.ScalaCompilerSupport.compile(ScalaCompilerSupport.java:119) at scala_maven.ScalaCompilerSupport.doExecute(ScalaCompilerSupport.java:99) at scala_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java:482) at scala_maven.ScalaTestCompileMojo.execute(ScalaTestCompileMojo.java:48) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) anyone found a similar error? kr On Sat, Sep 3, 2016 at 2:54 PM, Marco Mistroniwrote: > hi all > > i am getting failures when building spark 2.0 on Ubuntu 16.06 > Here's details of what i have installed on the ubuntu host > - java 8 > - scala 2.11 > - git > > When i launch the command > > ./build/mvn -Pyarn -Phadoop-2.7 -DskipTests clean package > > everything compiles sort of fine and at the end i get this exception > > INFO] BUILD FAILURE > [INFO] > > [INFO] Total time: 01:05 h > [INFO] Finished at: 2016-09-03T13:25:27+00:00 > [INFO] Final Memory: 57M/208M > [INFO] > > [ERROR] Failed to execute goal > net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile > (scala-test-compile-first) on project spark-streaming_2.11: Execution > scala-test-compile-first of goal > net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile > failed. CompileFailed -> [Help 1] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the > -e switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, > please read the following articles: > [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ > PluginExecutionException > [ERROR] > [ERROR] After correcting the problems, you can resume the build
Re: Pls assist: Spark 2.0 build failure on Ubuntu 16.06
Please run with -X and post logs here. We can get exact error from it. On Sat, Sep 3, 2016 at 7:24 PM, Marco Mistroniwrote: > hi all > > i am getting failures when building spark 2.0 on Ubuntu 16.06 > Here's details of what i have installed on the ubuntu host > - java 8 > - scala 2.11 > - git > > When i launch the command > > ./build/mvn -Pyarn -Phadoop-2.7 -DskipTests clean package > > everything compiles sort of fine and at the end i get this exception > > INFO] BUILD FAILURE > [INFO] > > [INFO] Total time: 01:05 h > [INFO] Finished at: 2016-09-03T13:25:27+00:00 > [INFO] Final Memory: 57M/208M > [INFO] > > [ERROR] Failed to execute goal > net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile > (scala-test-compile-first) on project spark-streaming_2.11: Execution > scala-test-compile-first of goal > net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile > failed. CompileFailed -> [Help 1] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the > -e switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, > please read the following articles: > [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ > PluginExecutionException > [ERROR] > [ERROR] After correcting the problems, you can resume the build with the > command > [ERROR] mvn -rf :spark-streaming_2.11 > > could anyone help? > > kr >
Pls assist: Spark 2.0 build failure on Ubuntu 16.06
hi all i am getting failures when building spark 2.0 on Ubuntu 16.06 Here's details of what i have installed on the ubuntu host - java 8 - scala 2.11 - git When i launch the command ./build/mvn -Pyarn -Phadoop-2.7 -DskipTests clean package everything compiles sort of fine and at the end i get this exception INFO] BUILD FAILURE [INFO] [INFO] Total time: 01:05 h [INFO] Finished at: 2016-09-03T13:25:27+00:00 [INFO] Final Memory: 57M/208M [INFO] [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) on project spark-streaming_2.11: Execution scala-test-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile failed. CompileFailed -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :spark-streaming_2.11 could anyone help? kr