Hi Sean thanks. I managed to build Spark 2 (it was actually 2.1, not 2.0...i m sourcing it from here (git clone git://github.com/apache/spark.git))
Now, i managed to build it but i had to - use Java 1.7 along with MAVEN_OPTS (using java1.8 send the whole process into insufficient memory for JVM while compiling spark-core....) - use Scala 2.10, installed on my host (when i tried with Scala 2.11, i got same results as with java 1.8) . kr marco On Sat, Oct 1, 2016 at 9:29 PM, Sean Owen <so...@cloudera.com> wrote: > "Compile failed via zinc server" > > Try shutting down zinc. Something's funny about your compile server. > It's not required anyway. > > On Sat, Oct 1, 2016 at 3:24 PM, Marco Mistroni <mmistr...@gmail.com> > wrote: > > Hi guys > > sorry to annoy you on this but i am getting nowhere. So far i have > tried to > > build spark 2.0 on my local laptop with no success.... so i blamed my > > laptop poor performance > > So today i fired off an EC2 Ubuntu 16.06 Instance and installed the > > following (i copy paste commands here) > > > > ubuntu@ip-172-31-40-104:~/spark$ history > > sudo apt-get install -y python-software-properties > > sudo apt-get install -y software-properties-common > > sudo add-apt-repository -y ppa:webupd8team/java > > sudo apt-get install -y oracle-java8-installer > > sudo apt-get install -y git > > git clone git://github.com/apache/spark.git > > cd spark > > > > Then launched the following commands: > > First thi sone, with only yarn, as i dont need hadoop > > > > ./build/mvn -Pyarn -DskipTests clean package > > > > This failed. after kicking off same command with -X i got this > > > > ERROR] Failed to execute goal > > net.alchim31.maven:scala-maven-plugin:3.2.2:compile > (scala-compile-first) on > > project spark-core_2.11: Execution scala-compile-first of goal > > net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed. > CompileFailed -> > > [Help 1] > > org.apache.maven.lifecycle.LifecycleExecutionException: Failed to > execute > > goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile > > (scala-compile-first) on project spark-core_2.11: Execution > > scala-compile-first of goal > > net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed. > > at > > org.apache.maven.lifecycle.internal.MojoExecutor.execute( > MojoExecutor.java:212) > > at > > org.apache.maven.lifecycle.internal.MojoExecutor.execute( > MojoExecutor.java:153) > > at > > org.apache.maven.lifecycle.internal.MojoExecutor.execute( > MojoExecutor.java:145) > > at > > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject( > LifecycleModuleBuilder.java:116) > > at > > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject( > LifecycleModuleBuilder.java:80) > > at > > org.apache.maven.lifecycle.internal.builder.singlethreaded. > SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) > > at > > org.apache.maven.lifecycle.internal.LifecycleStarter. > execute(LifecycleStarter.java:128) > > at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java: > 307) > > at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java: > 193) > > at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106) > > at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863) > > at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288) > > at org.apache.maven.cli.MavenCli.main(MavenCli.java:199) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > sun.reflect.NativeMethodAccessorImpl.invoke( > NativeMethodAccessorImpl.java:62) > > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at > > org.codehaus.plexus.classworlds.launcher.Launcher. > launchEnhanced(Launcher.java:289) > > at > > org.codehaus.plexus.classworlds.launcher.Launcher. > launch(Launcher.java:229) > > at > > org.codehaus.plexus.classworlds.launcher.Launcher. > mainWithExitCode(Launcher.java:415) > > at > > org.codehaus.plexus.classworlds.launcher.Launcher. > main(Launcher.java:356) > > Caused by: org.apache.maven.plugin.PluginExecutionException: Execution > > scala-compile-first of goal > > net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed. > > at > > org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo( > DefaultBuildPluginManager.java:145) > > at > > org.apache.maven.lifecycle.internal.MojoExecutor.execute( > MojoExecutor.java:207) > > ... 20 more > > Caused by: Compile failed via zinc server > > at > > sbt_inc.SbtIncrementalCompiler.zincCompile(SbtIncrementalCompiler.java: > 136) > > at > > sbt_inc.SbtIncrementalCompiler.compile(SbtIncrementalCompiler.java:86) > > at > > scala_maven.ScalaCompilerSupport.incrementalCompile( > ScalaCompilerSupport.java:303) > > at > > scala_maven.ScalaCompilerSupport.compile(ScalaCompilerSupport.java:119) > > at > > scala_maven.ScalaCompilerSupport.doExecute(ScalaCompilerSupport.java:99) > > at scala_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java: > 482) > > at > > org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo( > DefaultBuildPluginManager.java:134) > > ... 21 more > > > > > > > > Then i tried to use hadoop even though i dont need hadoop for my code > > > > ./build/mvn -Pyarn -Phadoop-2.4 -DskipTests clean package > > > > This failed again, at exactly the same point, with the same error > > > > Then i thought maybe i used an old version of hadoop, so i tried to use > 2.7 > > > > ./build/mvn -Pyarn -Phadoop-2.7 -DskipTests clean package > > > > Nada. same error. > > > > > > I have tried sbt, no luck > > > > Could anyone suggest me a mvn proflie to use to build spark 2.0? i am > > starting to suspect (before arguing that there's something wrong with > spark > > git) that i am using somehow wrong parameters..... or perhaps i should > > install scala 2.11 before i install spark? or Maven ? > > > > kr > > marco > > > > > > On Fri, Sep 30, 2016 at 8:23 PM, Marco Mistroni <mmistr...@gmail.com> > wrote: > >> > >> Hi all > >> this problem is still bothering me. > >> Here's my setup > >> - Ubuntu 16.06 > >> - Java 8 > >> - Spark 2.0 > >> - have launched following command: ./build/mvn -X -Pyarn -Phadoop-2.7 > >> -DskipTests clean package > >> and i am gettign this exception: > >> > >> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to > execute > >> goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile > >> (scala-test-compile-first) on project spark-mllib_2.10: Execution > >> scala-test-compile-first of goal > >> net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile failed. > >> > >> at > >> org.apache.maven.lifecycle.internal.MojoExecutor.execute( > MojoExecutor.java:212) > >> at > >> org.apache.maven.lifecycle.internal.MojoExecutor.execute( > MojoExecutor.java:153) > >> at > >> org.apache.maven.lifecycle.internal.MojoExecutor.execute( > MojoExecutor.java:145) > >> at > >> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder. > buildProject(LifecycleModuleBuilder.java:116) > >> at > >> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder. > buildProject(LifecycleModuleBuilder.java:80) > >> at > >> org.apache.maven.lifecycle.internal.builder.singlethreaded. > SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) > >> at > >> org.apache.maven.lifecycle.internal.LifecycleStarter. > execute(LifecycleStarter.java:128) > >> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java: > 307) > >> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java: > 193) > >> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106) > >> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863) > >> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288) > >> at org.apache.maven.cli.MavenCli.main(MavenCli.java:199) > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> at > >> sun.reflect.NativeMethodAccessorImpl.invoke( > NativeMethodAccessorImpl.java:62) > >> at > >> sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > >> at java.lang.reflect.Method.invoke(Method.java:498) > >> at > >> org.codehaus.plexus.classworlds.launcher.Launcher. > launchEnhanced(Launcher.java:289) > >> at > >> org.codehaus.plexus.classworlds.launcher.Launcher. > launch(Launcher.java:229) > >> at > >> org.codehaus.plexus.classworlds.launcher.Launcher. > mainWithExitCode(Launcher.java:415) > >> at > >> org.codehaus.plexus.classworlds.launcher.Launcher. > main(Launcher.java:356) > >> Caused by: org.apache.maven.plugin.PluginExecutionException: Execution > >> scala-test-compile-first of goal > >> net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile failed. > >> at > >> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo( > DefaultBuildPluginManager.java:145) > >> at > >> org.apache.maven.lifecycle.internal.MojoExecutor.execute( > MojoExecutor.java:207) > >> ... 20 more > >> Caused by: Compile failed via zinc server > >> at > >> sbt_inc.SbtIncrementalCompiler.zincCompile(SbtIncrementalCompiler.java: > 136) > >> at > >> sbt_inc.SbtIncrementalCompiler.compile(SbtIncrementalCompiler.java:86) > >> at > >> scala_maven.ScalaCompilerSupport.incrementalCompile( > ScalaCompilerSupport.java:303) > >> at > >> scala_maven.ScalaCompilerSupport.compile(ScalaCompilerSupport.java:119) > >> at > >> scala_maven.ScalaCompilerSupport.doExecute( > ScalaCompilerSupport.java:99) > >> at scala_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java: > 482) > >> at > >> scala_maven.ScalaTestCompileMojo.execute(ScalaTestCompileMojo.java:48) > >> at > >> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo( > DefaultBuildPluginManager.java:134) > >> > >> anyone found a similar error? > >> > >> kr >
