Looks like '-Pyarn' was missing in your command. On Mon, Oct 26, 2015 at 12:06 PM, Kayode Odeyemi <drey...@gmail.com> wrote:
> I used this command which is synonymous to what you have: > > ./make-distribution.sh --name spark-latest --tgz --mvn mvn > -Dhadoop.version=2.6.0 -Phadoop-2.6 -Phive -Phive-thriftserver -DskipTests > clean package -U > > But I still see WARNINGS like this in the output and no .gz file created: > > cp: /usr/local/spark-latest/spark-[WARNING] See > http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/.part-r-00005.gz.parquet.crc: > No such file or directory > cp: /usr/local/spark-latest/spark-[WARNING] See > http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/part-r-00005.gz.parquet: > No such file or directory > cp: /usr/local/spark-latest/spark-[WARNING] See > http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9: > No such file or directory > cp: > /usr/local/spark-latest/dist/python/test_support/sql/parquet_partitioned/year=2015/month=9: > unable to copy extended attributes to > /usr/local/spark-latest/spark-[WARNING] See > http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9: > No such file or directory > cp: /usr/local/spark-latest/spark-[WARNING] See > http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1: > No such file or directory > cp: > /usr/local/spark-latest/dist/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1: > unable to copy extended attributes to > /usr/local/spark-latest/spark-[WARNING] See > http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1: > No such file or directory > cp: /usr/local/spark-latest/spark-[WARNING] See > http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1/.part-r-00007.gz.parquet.crc: > No such file or directory > > On Mon, Oct 26, 2015 at 8:58 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >> If you use the command shown in: >> https://github.com/apache/spark/pull/9281 >> >> You should have got the following: >> >> >> ./dist/python/test_support/sql/parquet_partitioned/year=2014/month=9/day=1/part-r-00008.gz.parquet >> >> ./dist/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1/part-r-00007.gz.parquet >> >> ./dist/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/part-r-00004.gz.parquet >> >> ./dist/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/part-r-00002.gz.parquet >> >> ./dist/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/part-r-00005.gz.parquet >> >> On Mon, Oct 26, 2015 at 11:47 AM, Kayode Odeyemi <drey...@gmail.com> >> wrote: >> >>> I see a lot of stuffs like this after the a successful maven build: >>> >>> cp: /usr/local/spark-latest/spark-[WARNING] See >>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2014/month=9/day=1/ >>> part-r-00008.gz.parquet: No such file or directory >>> >>> Seems it fails when it tries to package the build as an archive. >>> >>> I'm using the latest code on github master. >>> >>> Any ideas please? >>> >>> On Mon, Oct 26, 2015 at 6:20 PM, Yana Kadiyska <yana.kadiy...@gmail.com> >>> wrote: >>> >>>> In 1.4 ./make_distribution produces a .tgz file in the root directory >>>> (same directory that make_distribution is in) >>>> >>>> >>>> >>>> On Mon, Oct 26, 2015 at 8:46 AM, Kayode Odeyemi <drey...@gmail.com> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> The ./make_distribution task completed. However, I can't seem to >>>>> locate the >>>>> .tar.gz file. >>>>> >>>>> Where does Spark save this? or should I just work with the dist >>>>> directory? >>>>> >>>>> On Fri, Oct 23, 2015 at 4:23 PM, Kayode Odeyemi <drey...@gmail.com> >>>>> wrote: >>>>> >>>>>> I saw this when I tested manually (without ./make-distribution) >>>>>> >>>>>> Detected Maven Version: 3.2.2 is not in the allowed range 3.3.3. >>>>>> >>>>>> So I simply upgraded maven to 3.3.3. >>>>>> >>>>>> Resolved. Thanks >>>>>> >>>>>> On Fri, Oct 23, 2015 at 3:17 PM, Sean Owen <so...@cloudera.com> >>>>>> wrote: >>>>>> >>>>>>> This doesn't show the actual error output from Maven. I have a strong >>>>>>> guess that you haven't set MAVEN_OPTS to increase the memory Maven >>>>>>> can >>>>>>> use. >>>>>>> >>>>>>> On Fri, Oct 23, 2015 at 6:14 AM, Kayode Odeyemi <drey...@gmail.com> >>>>>>> wrote: >>>>>>> > Hi, >>>>>>> > >>>>>>> > I can't seem to get a successful maven build. Please see command >>>>>>> output >>>>>>> > below: >>>>>>> > >>>>>>> > bash-3.2$ ./make-distribution.sh --name spark-latest --tgz --mvn >>>>>>> mvn >>>>>>> > -Dhadoop.version=2.7.0 -Phadoop-2.7 -Phive -Phive-thriftserver >>>>>>> -DskipTests >>>>>>> > clean package >>>>>>> > +++ dirname ./make-distribution.sh >>>>>>> > ++ cd . >>>>>>> > ++ pwd >>>>>>> > + SPARK_HOME=/usr/local/spark-latest >>>>>>> > + DISTDIR=/usr/local/spark-latest/dist >>>>>>> > + SPARK_TACHYON=false >>>>>>> > + TACHYON_VERSION=0.7.1 >>>>>>> > + TACHYON_TGZ=tachyon-0.7.1-bin.tar.gz >>>>>>> > + >>>>>>> > TACHYON_URL= >>>>>>> https://github.com/amplab/tachyon/releases/download/v0.7.1/tachyon-0.7.1-bin.tar.gz >>>>>>> > + MAKE_TGZ=false >>>>>>> > + NAME=none >>>>>>> > + MVN=/usr/local/spark-latest/build/mvn >>>>>>> > + (( 12 )) >>>>>>> > + case $1 in >>>>>>> > + NAME=spark-latest >>>>>>> > + shift >>>>>>> > + shift >>>>>>> > + (( 10 )) >>>>>>> > + case $1 in >>>>>>> > + MAKE_TGZ=true >>>>>>> > + shift >>>>>>> > + (( 9 )) >>>>>>> > + case $1 in >>>>>>> > + MVN=mvn >>>>>>> > + shift >>>>>>> > + shift >>>>>>> > + (( 7 )) >>>>>>> > + case $1 in >>>>>>> > + break >>>>>>> > + '[' -z >>>>>>> /Library/Java/JavaVirtualMachines/jdk1.8.0_20.jdk/Contents/Home ']' >>>>>>> > + '[' -z >>>>>>> /Library/Java/JavaVirtualMachines/jdk1.8.0_20.jdk/Contents/Home ']' >>>>>>> > ++ command -v git >>>>>>> > + '[' /usr/bin/git ']' >>>>>>> > ++ git rev-parse --short HEAD >>>>>>> > + GITREV=487d409 >>>>>>> > + '[' '!' -z 487d409 ']' >>>>>>> > + GITREVSTRING=' (git revision 487d409)' >>>>>>> > + unset GITREV >>>>>>> > ++ command -v mvn >>>>>>> > + '[' '!' /usr/bin/mvn ']' >>>>>>> > ++ mvn help:evaluate -Dexpression=project.version >>>>>>> -Dhadoop.version=2.7.0 >>>>>>> > -Phadoop-2.7 -Phive -Phive-thriftserver -DskipTests clean package >>>>>>> > ++ grep -v INFO >>>>>>> > ++ tail -n 1 >>>>>>> > + VERSION='[ERROR] [Help 1] >>>>>>> > >>>>>>> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException >>>>>>> ' >>>>>>> > >>>>>>> > Same output error with JDK 7 >>>>>>> > >>>>>>> > Appreciate your help. >>>>>>> > >>>>>>> > >>>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >>> >> >
