I used this command which is synonymous to what you have: ./make-distribution.sh --name spark-latest --tgz --mvn mvn -Dhadoop.version=2.6.0 -Phadoop-2.6 -Phive -Phive-thriftserver -DskipTests clean package -U
But I still see WARNINGS like this in the output and no .gz file created: cp: /usr/local/spark-latest/spark-[WARNING] See http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/.part-r-00005.gz.parquet.crc: No such file or directory cp: /usr/local/spark-latest/spark-[WARNING] See http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/part-r-00005.gz.parquet: No such file or directory cp: /usr/local/spark-latest/spark-[WARNING] See http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9: No such file or directory cp: /usr/local/spark-latest/dist/python/test_support/sql/parquet_partitioned/year=2015/month=9: unable to copy extended attributes to /usr/local/spark-latest/spark-[WARNING] See http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9: No such file or directory cp: /usr/local/spark-latest/spark-[WARNING] See http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1: No such file or directory cp: /usr/local/spark-latest/dist/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1: unable to copy extended attributes to /usr/local/spark-latest/spark-[WARNING] See http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1: No such file or directory cp: /usr/local/spark-latest/spark-[WARNING] See http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1/.part-r-00007.gz.parquet.crc: No such file or directory On Mon, Oct 26, 2015 at 8:58 PM, Ted Yu <yuzhih...@gmail.com> wrote: > If you use the command shown in: > https://github.com/apache/spark/pull/9281 > > You should have got the following: > > > ./dist/python/test_support/sql/parquet_partitioned/year=2014/month=9/day=1/part-r-00008.gz.parquet > > ./dist/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1/part-r-00007.gz.parquet > > ./dist/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/part-r-00004.gz.parquet > > ./dist/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/part-r-00002.gz.parquet > > ./dist/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/part-r-00005.gz.parquet > > On Mon, Oct 26, 2015 at 11:47 AM, Kayode Odeyemi <drey...@gmail.com> > wrote: > >> I see a lot of stuffs like this after the a successful maven build: >> >> cp: /usr/local/spark-latest/spark-[WARNING] See >> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2014/month=9/day=1/ >> part-r-00008.gz.parquet: No such file or directory >> >> Seems it fails when it tries to package the build as an archive. >> >> I'm using the latest code on github master. >> >> Any ideas please? >> >> On Mon, Oct 26, 2015 at 6:20 PM, Yana Kadiyska <yana.kadiy...@gmail.com> >> wrote: >> >>> In 1.4 ./make_distribution produces a .tgz file in the root directory >>> (same directory that make_distribution is in) >>> >>> >>> >>> On Mon, Oct 26, 2015 at 8:46 AM, Kayode Odeyemi <drey...@gmail.com> >>> wrote: >>> >>>> Hi, >>>> >>>> The ./make_distribution task completed. However, I can't seem to locate >>>> the >>>> .tar.gz file. >>>> >>>> Where does Spark save this? or should I just work with the dist >>>> directory? >>>> >>>> On Fri, Oct 23, 2015 at 4:23 PM, Kayode Odeyemi <drey...@gmail.com> >>>> wrote: >>>> >>>>> I saw this when I tested manually (without ./make-distribution) >>>>> >>>>> Detected Maven Version: 3.2.2 is not in the allowed range 3.3.3. >>>>> >>>>> So I simply upgraded maven to 3.3.3. >>>>> >>>>> Resolved. Thanks >>>>> >>>>> On Fri, Oct 23, 2015 at 3:17 PM, Sean Owen <so...@cloudera.com> wrote: >>>>> >>>>>> This doesn't show the actual error output from Maven. I have a strong >>>>>> guess that you haven't set MAVEN_OPTS to increase the memory Maven can >>>>>> use. >>>>>> >>>>>> On Fri, Oct 23, 2015 at 6:14 AM, Kayode Odeyemi <drey...@gmail.com> >>>>>> wrote: >>>>>> > Hi, >>>>>> > >>>>>> > I can't seem to get a successful maven build. Please see command >>>>>> output >>>>>> > below: >>>>>> > >>>>>> > bash-3.2$ ./make-distribution.sh --name spark-latest --tgz --mvn mvn >>>>>> > -Dhadoop.version=2.7.0 -Phadoop-2.7 -Phive -Phive-thriftserver >>>>>> -DskipTests >>>>>> > clean package >>>>>> > +++ dirname ./make-distribution.sh >>>>>> > ++ cd . >>>>>> > ++ pwd >>>>>> > + SPARK_HOME=/usr/local/spark-latest >>>>>> > + DISTDIR=/usr/local/spark-latest/dist >>>>>> > + SPARK_TACHYON=false >>>>>> > + TACHYON_VERSION=0.7.1 >>>>>> > + TACHYON_TGZ=tachyon-0.7.1-bin.tar.gz >>>>>> > + >>>>>> > TACHYON_URL= >>>>>> https://github.com/amplab/tachyon/releases/download/v0.7.1/tachyon-0.7.1-bin.tar.gz >>>>>> > + MAKE_TGZ=false >>>>>> > + NAME=none >>>>>> > + MVN=/usr/local/spark-latest/build/mvn >>>>>> > + (( 12 )) >>>>>> > + case $1 in >>>>>> > + NAME=spark-latest >>>>>> > + shift >>>>>> > + shift >>>>>> > + (( 10 )) >>>>>> > + case $1 in >>>>>> > + MAKE_TGZ=true >>>>>> > + shift >>>>>> > + (( 9 )) >>>>>> > + case $1 in >>>>>> > + MVN=mvn >>>>>> > + shift >>>>>> > + shift >>>>>> > + (( 7 )) >>>>>> > + case $1 in >>>>>> > + break >>>>>> > + '[' -z >>>>>> /Library/Java/JavaVirtualMachines/jdk1.8.0_20.jdk/Contents/Home ']' >>>>>> > + '[' -z >>>>>> /Library/Java/JavaVirtualMachines/jdk1.8.0_20.jdk/Contents/Home ']' >>>>>> > ++ command -v git >>>>>> > + '[' /usr/bin/git ']' >>>>>> > ++ git rev-parse --short HEAD >>>>>> > + GITREV=487d409 >>>>>> > + '[' '!' -z 487d409 ']' >>>>>> > + GITREVSTRING=' (git revision 487d409)' >>>>>> > + unset GITREV >>>>>> > ++ command -v mvn >>>>>> > + '[' '!' /usr/bin/mvn ']' >>>>>> > ++ mvn help:evaluate -Dexpression=project.version >>>>>> -Dhadoop.version=2.7.0 >>>>>> > -Phadoop-2.7 -Phive -Phive-thriftserver -DskipTests clean package >>>>>> > ++ grep -v INFO >>>>>> > ++ tail -n 1 >>>>>> > + VERSION='[ERROR] [Help 1] >>>>>> > >>>>>> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException >>>>>> ' >>>>>> > >>>>>> > Same output error with JDK 7 >>>>>> > >>>>>> > Appreciate your help. >>>>>> > >>>>>> > >>>>>> >>>>> >>>>> >>>>> >>>> >>> >> >> >
