Ted, I switched to this: ./make-distribution.sh --name spark-latest --tgz -Dhadoop.version=2.6.0 -Phadoop-2.6 -Phive -Phive-thriftserver -Pyarn -DskipTests clean package -U
Same error. No .gz file. Here's the bottom output log: + rm -rf /home/emperor/javaprojects/spark/dist + mkdir -p /home/emperor/javaprojects/spark/dist/lib + echo 'Spark [WARNING] See http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin (git revision 3689beb) built for Hadoop [WARNING] See http://docs.codehaus.org/display/MAVENUSER/Shade+Pl + echo 'Build flags: -Dhadoop.version=2.6.0' -Phadoop-2.6 -Phive -Phive-thriftserver -Pyarn -DskipTests clean package -U + cp /home/emperor/javaprojects/spark/assembly/target/scala-2.10/spark-assembly-1.6.0-SNAPSHOT-hadoop2.6.0.jar /home/emperor/javaprojects/spark/dist/lib/ + cp /home/emperor/javaprojects/spark/examples/target/scala-2.10/spark-examples-1.6.0-SNAPSHOT-hadoop2.6.0.jar /home/emperor/javaprojects/spark/dist/lib/ + cp /home/emperor/javaprojects/spark/network/yarn/target/scala-2.10/spark-1.6.0-SNAPSHOT-yarn-shuffle.jar /home/emperor/javaprojects/spark/dist/lib/ + mkdir -p /home/emperor/javaprojects/spark/dist/examples/src/main + cp -r /home/emperor/javaprojects/spark/examples/src/main /home/emperor/javaprojects/spark/dist/examples/src/ + '[' 1 == 1 ']' + cp /home/emperor/javaprojects/spark/lib_managed/jars/datanucleus-api-jdo-3.2.6.jar /home/emperor/javaprojects/spark/lib_managed/jars/datanucleus-core-3.2.10.jar /home/emperor/javaprojects ed/jars/datanucleus-rdbms-3.2.9.jar /home/emperor/javaprojects/spark/dist/lib/ + cp /home/emperor/javaprojects/spark/LICENSE /home/emperor/javaprojects/spark/dist + cp -r /home/emperor/javaprojects/spark/licenses /home/emperor/javaprojects/spark/dist + cp /home/emperor/javaprojects/spark/NOTICE /home/emperor/javaprojects/spark/dist + '[' -e /home/emperor/javaprojects/spark/CHANGES.txt ']' + cp -r /home/emperor/javaprojects/spark/data /home/emperor/javaprojects/spark/dist + mkdir /home/emperor/javaprojects/spark/dist/conf + cp /home/emperor/javaprojects/spark/conf/docker.properties.template /home/emperor/javaprojects/spark/conf/fairscheduler.xml.template /home/emperor/javaprojects/spark/conf/log4j.properties emperor/javaprojects/spark/conf/metrics.properties.template /home/emperor/javaprojects/spark/conf/slaves.template /home/emperor/javaprojects/spark/conf/spark-defaults.conf.template /home/em ts/spark/conf/spark-env.sh.template /home/emperor/javaprojects/spark/dist/conf + cp /home/emperor/javaprojects/spark/README.md /home/emperor/javaprojects/spark/dist + cp -r /home/emperor/javaprojects/spark/bin /home/emperor/javaprojects/spark/dist + cp -r /home/emperor/javaprojects/spark/python /home/emperor/javaprojects/spark/dist + cp -r /home/emperor/javaprojects/spark/sbin /home/emperor/javaprojects/spark/dist + cp -r /home/emperor/javaprojects/spark/ec2 /home/emperor/javaprojects/spark/dist + '[' -d /home/emperor/javaprojects/spark/R/lib/SparkR ']' + '[' false == true ']' + '[' true == true ']' + TARDIR_NAME='spark-[WARNING] See http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest' + TARDIR='/home/emperor/javaprojects/spark/spark-[WARNING] See http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest' + rm -rf '/home/emperor/javaprojects/spark/spark-[WARNING] See http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest' + cp -r /home/emperor/javaprojects/spark/dist '/home/emperor/javaprojects/spark/spark-[WARNING] See http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest' cp: cannot create directory `/home/emperor/javaprojects/spark/spark-[WARNING] See http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest': No such file or directory On Tue, Oct 27, 2015 at 2:14 PM, Ted Yu <yuzhih...@gmail.com> wrote: > Can you try the same command shown in the pull request ? > > Thanks > > On Oct 27, 2015, at 12:40 AM, Kayode Odeyemi <drey...@gmail.com> wrote: > > Thank you. > > But I'm getting same warnings and it's still preventing the archive from > being generated. > > I've ran this on both OSX Lion and Ubuntu 12. Same error. No .gz file > > On Mon, Oct 26, 2015 at 9:10 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >> Looks like '-Pyarn' was missing in your command. >> >> On Mon, Oct 26, 2015 at 12:06 PM, Kayode Odeyemi <drey...@gmail.com> >> wrote: >> >>> I used this command which is synonymous to what you have: >>> >>> ./make-distribution.sh --name spark-latest --tgz --mvn mvn >>> -Dhadoop.version=2.6.0 -Phadoop-2.6 -Phive -Phive-thriftserver -DskipTests >>> clean package -U >>> >>> But I still see WARNINGS like this in the output and no .gz file created: >>> >>> cp: /usr/local/spark-latest/spark-[WARNING] See >>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/.part-r-00005.gz.parquet.crc: >>> No such file or directory >>> cp: /usr/local/spark-latest/spark-[WARNING] See >>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/part-r-00005.gz.parquet: >>> No such file or directory >>> cp: /usr/local/spark-latest/spark-[WARNING] See >>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9: >>> No such file or directory >>> cp: >>> /usr/local/spark-latest/dist/python/test_support/sql/parquet_partitioned/year=2015/month=9: >>> unable to copy extended attributes to >>> /usr/local/spark-latest/spark-[WARNING] See >>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9: >>> No such file or directory >>> cp: /usr/local/spark-latest/spark-[WARNING] See >>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1: >>> No such file or directory >>> cp: >>> /usr/local/spark-latest/dist/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1: >>> unable to copy extended attributes to >>> /usr/local/spark-latest/spark-[WARNING] See >>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1: >>> No such file or directory >>> cp: /usr/local/spark-latest/spark-[WARNING] See >>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1/.part-r-00007.gz.parquet.crc: >>> No such file or directory >>> >>> On Mon, Oct 26, 2015 at 8:58 PM, Ted Yu <yuzhih...@gmail.com> wrote: >>> >>>> If you use the command shown in: >>>> https://github.com/apache/spark/pull/9281 >>>> >>>> You should have got the following: >>>> >>>> >>>> ./dist/python/test_support/sql/parquet_partitioned/year=2014/month=9/day=1/part-r-00008.gz.parquet >>>> >>>> ./dist/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1/part-r-00007.gz.parquet >>>> >>>> ./dist/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/part-r-00004.gz.parquet >>>> >>>> ./dist/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/part-r-00002.gz.parquet >>>> >>>> ./dist/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/part-r-00005.gz.parquet >>>> >>>> On Mon, Oct 26, 2015 at 11:47 AM, Kayode Odeyemi <drey...@gmail.com> >>>> wrote: >>>> >>>>> I see a lot of stuffs like this after the a successful maven build: >>>>> >>>>> cp: /usr/local/spark-latest/spark-[WARNING] See >>>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2014/month=9/day=1/ >>>>> part-r-00008.gz.parquet: No such file or directory >>>>> >>>>> Seems it fails when it tries to package the build as an archive. >>>>> >>>>> I'm using the latest code on github master. >>>>> >>>>> Any ideas please? >>>>> >>>>> On Mon, Oct 26, 2015 at 6:20 PM, Yana Kadiyska < >>>>> yana.kadiy...@gmail.com> wrote: >>>>> >>>>>> In 1.4 ./make_distribution produces a .tgz file in the root >>>>>> directory (same directory that make_distribution is in) >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Oct 26, 2015 at 8:46 AM, Kayode Odeyemi <drey...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> The ./make_distribution task completed. However, I can't seem to >>>>>>> locate the >>>>>>> .tar.gz file. >>>>>>> >>>>>>> Where does Spark save this? or should I just work with the dist >>>>>>> directory? >>>>>>> >>>>>>> On Fri, Oct 23, 2015 at 4:23 PM, Kayode Odeyemi <drey...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> I saw this when I tested manually (without ./make-distribution) >>>>>>>> >>>>>>>> Detected Maven Version: 3.2.2 is not in the allowed range 3.3.3. >>>>>>>> >>>>>>>> So I simply upgraded maven to 3.3.3. >>>>>>>> >>>>>>>> Resolved. Thanks >>>>>>>> >>>>>>>> On Fri, Oct 23, 2015 at 3:17 PM, Sean Owen <so...@cloudera.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> This doesn't show the actual error output from Maven. I have a >>>>>>>>> strong >>>>>>>>> guess that you haven't set MAVEN_OPTS to increase the memory Maven >>>>>>>>> can >>>>>>>>> use. >>>>>>>>> >>>>>>>>> On Fri, Oct 23, 2015 at 6:14 AM, Kayode Odeyemi <drey...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> > Hi, >>>>>>>>> > >>>>>>>>> > I can't seem to get a successful maven build. Please see command >>>>>>>>> output >>>>>>>>> > below: >>>>>>>>> > >>>>>>>>> > bash-3.2$ ./make-distribution.sh --name spark-latest --tgz --mvn >>>>>>>>> mvn >>>>>>>>> > -Dhadoop.version=2.7.0 -Phadoop-2.7 -Phive -Phive-thriftserver >>>>>>>>> -DskipTests >>>>>>>>> > clean package >>>>>>>>> > +++ dirname ./make-distribution.sh >>>>>>>>> > ++ cd . >>>>>>>>> > ++ pwd >>>>>>>>> > + SPARK_HOME=/usr/local/spark-latest >>>>>>>>> > + DISTDIR=/usr/local/spark-latest/dist >>>>>>>>> > + SPARK_TACHYON=false >>>>>>>>> > + TACHYON_VERSION=0.7.1 >>>>>>>>> > + TACHYON_TGZ=tachyon-0.7.1-bin.tar.gz >>>>>>>>> > + >>>>>>>>> > TACHYON_URL= >>>>>>>>> https://github.com/amplab/tachyon/releases/download/v0.7.1/tachyon-0.7.1-bin.tar.gz >>>>>>>>> > + MAKE_TGZ=false >>>>>>>>> > + NAME=none >>>>>>>>> > + MVN=/usr/local/spark-latest/build/mvn >>>>>>>>> > + (( 12 )) >>>>>>>>> > + case $1 in >>>>>>>>> > + NAME=spark-latest >>>>>>>>> > + shift >>>>>>>>> > + shift >>>>>>>>> > + (( 10 )) >>>>>>>>> > + case $1 in >>>>>>>>> > + MAKE_TGZ=true >>>>>>>>> > + shift >>>>>>>>> > + (( 9 )) >>>>>>>>> > + case $1 in >>>>>>>>> > + MVN=mvn >>>>>>>>> > + shift >>>>>>>>> > + shift >>>>>>>>> > + (( 7 )) >>>>>>>>> > + case $1 in >>>>>>>>> > + break >>>>>>>>> > + '[' -z >>>>>>>>> /Library/Java/JavaVirtualMachines/jdk1.8.0_20.jdk/Contents/Home ']' >>>>>>>>> > + '[' -z >>>>>>>>> /Library/Java/JavaVirtualMachines/jdk1.8.0_20.jdk/Contents/Home ']' >>>>>>>>> > ++ command -v git >>>>>>>>> > + '[' /usr/bin/git ']' >>>>>>>>> > ++ git rev-parse --short HEAD >>>>>>>>> > + GITREV=487d409 >>>>>>>>> > + '[' '!' -z 487d409 ']' >>>>>>>>> > + GITREVSTRING=' (git revision 487d409)' >>>>>>>>> > + unset GITREV >>>>>>>>> > ++ command -v mvn >>>>>>>>> > + '[' '!' /usr/bin/mvn ']' >>>>>>>>> > ++ mvn help:evaluate -Dexpression=project.version >>>>>>>>> -Dhadoop.version=2.7.0 >>>>>>>>> > -Phadoop-2.7 -Phive -Phive-thriftserver -DskipTests clean package >>>>>>>>> > ++ grep -v INFO >>>>>>>>> > ++ tail -n 1 >>>>>>>>> > + VERSION='[ERROR] [Help 1] >>>>>>>>> > >>>>>>>>> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException >>>>>>>>> ' >>>>>>>>> > >>>>>>>>> > Same output error with JDK 7 >>>>>>>>> > >>>>>>>>> > Appreciate your help. >>>>>>>>> > >>>>>>>>> > >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>> >>> >> >
