[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16017087#comment-16017087 ] Hive QA commented on HIVE-14735: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12855485/HIVE-14735.5.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5344/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5344/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5344/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2017-05-19 09:01:48.936 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-5344/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2017-05-19 09:01:48.938 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 3be1eed HIVE-16672: Parquet vectorization doesn't work for tables with partition info (Colin Ma, reviewed by Ferdinand Xu) + git clean -f -d Removing ql/src/gen/vectorization/UDAFTemplates/VectorUDAFAvgDecimal.txt Removing ql/src/gen/vectorization/UDAFTemplates/VectorUDAFAvgDecimalMerge.txt Removing ql/src/gen/vectorization/UDAFTemplates/VectorUDAFAvgMerge.txt Removing ql/src/gen/vectorization/UDAFTemplates/VectorUDAFAvgTimestamp.txt Removing ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFSumTimestamp.java + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 3be1eed HIVE-16672: Parquet vectorization doesn't work for tables with partition info (Colin Ma, reviewed by Ferdinand Xu) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2017-05-19 09:01:50.111 + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: itests/thirdparty/.gitignore: already exists in working directory error: itests/thirdparty/pom.xml: already exists in working directory The patch does not appear to apply with p0, p1, or p2 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12855485 - PreCommit-HIVE-Build > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta > Attachments: HIVE-14735.1.patch, HIVE-14735.1.patch, > HIVE-14735.1.patch, HIVE-14735.1.patch, HIVE-14735.2.patch, > HIVE-14735.3.patch, HIVE-14735.4.patch, HIVE-14735.4.patch, HIVE-14735.5.patch > > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957570#comment-15957570 ] Vaibhav Gumashta commented on HIVE-14735: - [~mmccline] You can disable the {{download-spark}} execution in various poms. > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich > Attachments: HIVE-14735.1.patch, HIVE-14735.1.patch, > HIVE-14735.1.patch, HIVE-14735.1.patch, HIVE-14735.2.patch, > HIVE-14735.3.patch, HIVE-14735.4.patch, HIVE-14735.4.patch, HIVE-14735.5.patch > > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957471#comment-15957471 ] Matt McCline commented on HIVE-14735: - Is there someway I can avoid downloading the file each time? It is 135 Mb plus and is right now burning up my broadband HotSpot Gb limit. Thanks. > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich > Attachments: HIVE-14735.1.patch, HIVE-14735.1.patch, > HIVE-14735.1.patch, HIVE-14735.1.patch, HIVE-14735.2.patch, > HIVE-14735.3.patch, HIVE-14735.4.patch, HIVE-14735.4.patch, HIVE-14735.5.patch > > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932639#comment-15932639 ] Zoltan Haindrich commented on HIVE-14735: - [~spena] I've asked the spark developers about this: http://apache-spark-developers-list.1001551.n3.nabble.com/spark-without-hive-assembly-for-hive-build-development-purposes-td21188.html I didn't got back a clear answer to my question...beyond a "why do we use that" and a reference to HIVE-15302. what should we do now? > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich > Attachments: HIVE-14735.1.patch, HIVE-14735.1.patch, > HIVE-14735.1.patch, HIVE-14735.1.patch, HIVE-14735.2.patch, > HIVE-14735.3.patch, HIVE-14735.4.patch, HIVE-14735.4.patch, HIVE-14735.5.patch > > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926524#comment-15926524 ] Sergio Peña commented on HIVE-14735: [~kgyrtkirk] Would you mind asking the spark list if we can publish this jar on its side? > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich > Attachments: HIVE-14735.1.patch, HIVE-14735.1.patch, > HIVE-14735.1.patch, HIVE-14735.1.patch, HIVE-14735.2.patch, > HIVE-14735.3.patch, HIVE-14735.4.patch, HIVE-14735.4.patch, HIVE-14735.5.patch > > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895056#comment-15895056 ] Ashutosh Chauhan commented on HIVE-14735: - I am not sure whether publishing an artifact of another project is a good idea. Ideally, spark project itself should publish these artifacts. At the very least we shall ask on spark list of our intention for this and see what feedback we get. > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich > Attachments: HIVE-14735.1.patch, HIVE-14735.1.patch, > HIVE-14735.1.patch, HIVE-14735.1.patch, HIVE-14735.2.patch, > HIVE-14735.3.patch, HIVE-14735.4.patch, HIVE-14735.4.patch, HIVE-14735.5.patch > > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894803#comment-15894803 ] Zoltan Haindrich commented on HIVE-14735: - [~spena] the current patch may need some cleanup - but it should be ok overall :) the ptest server is unable to build it because the patch removes thirdparty from the .gitignore - and the previous version of this patch is still in there... About moving the build dependency to {{repository.apache.org}}, the current artifact name is: {{group: org.apache.hive.aux, artifactId:spark-without-hive}} - i'm open to change this to anything else...since this artifact is somewhere in the middle between hive and spark projects...I'm not sure what to choose for groupId - ideally it would be an artifact published during spark releasing...but since 2.0.0 is already out - we missed that train already :) > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich > Attachments: HIVE-14735.1.patch, HIVE-14735.1.patch, > HIVE-14735.1.patch, HIVE-14735.1.patch, HIVE-14735.2.patch, > HIVE-14735.3.patch, HIVE-14735.4.patch, HIVE-14735.4.patch, HIVE-14735.5.patch > > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894728#comment-15894728 ] Sergio Peña commented on HIVE-14735: Thanks [~kgyrtkirk]. I'm gonna take a look at the patch. Having the patch on the apache repository sounds like a good idea. Let's ask [~ashutoshc]. Do you think we can publish the spark-without-hive tarball on {{repository.apache.org}} for hive building purposes? > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich > Attachments: HIVE-14735.1.patch, HIVE-14735.1.patch, > HIVE-14735.1.patch, HIVE-14735.1.patch, HIVE-14735.2.patch, > HIVE-14735.3.patch, HIVE-14735.4.patch, HIVE-14735.4.patch, HIVE-14735.5.patch > > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891352#comment-15891352 ] Hive QA commented on HIVE-14735: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12855485/HIVE-14735.5.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3879/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3879/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3879/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2017-03-02 00:14:01.641 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-3879/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2017-03-02 00:14:01.643 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at ba8de30 HIVE-14459: TestBeeLineDriver - migration and re-enable (Peter Vary via Zoltan Haindrich reviewed by Vihang Karajgaonkar) + git clean -f -d + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at ba8de30 HIVE-14459: TestBeeLineDriver - migration and re-enable (Peter Vary via Zoltan Haindrich reviewed by Vihang Karajgaonkar) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2017-03-02 00:14:02.776 + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: itests/thirdparty/.gitignore: already exists in working directory error: itests/thirdparty/pom.xml: already exists in working directory The patch does not appear to apply with p0, p1, or p2 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12855485 - PreCommit-HIVE-Build > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich > Attachments: HIVE-14735.1.patch, HIVE-14735.1.patch, > HIVE-14735.1.patch, HIVE-14735.1.patch, HIVE-14735.2.patch, > HIVE-14735.3.patch, HIVE-14735.4.patch, HIVE-14735.4.patch, HIVE-14735.5.patch > > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891263#comment-15891263 ] Zoltan Haindrich commented on HIVE-14735: - it looks like the build node has a non-clean copy...this patch adds {{itests/thirdparty/pom.xml}} - I submit it again..it may have been fixed [~spena] I think it would be much better to server this artifact from {{repository.apache.org}} - it seems pretty straightforward to upload it to there - would it be okay to try? :) > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich > Attachments: HIVE-14735.1.patch, HIVE-14735.1.patch, > HIVE-14735.1.patch, HIVE-14735.1.patch, HIVE-14735.2.patch, > HIVE-14735.3.patch, HIVE-14735.4.patch, HIVE-14735.4.patch > > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887322#comment-15887322 ] Hive QA commented on HIVE-14735: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12855000/HIVE-14735.4.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3828/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3828/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3828/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2017-02-28 05:44:31.918 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-3828/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2017-02-28 05:44:31.921 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 2869eca HIVE-15958: LLAP: IPC connections are not being reused for umbilical protocol (Prasanth Jayachandran reviewed by Siddharth Seth) + git clean -f -d + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 2869eca HIVE-15958: LLAP: IPC connections are not being reused for umbilical protocol (Prasanth Jayachandran reviewed by Siddharth Seth) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2017-02-28 05:44:32.822 + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: itests/thirdparty/.gitignore: already exists in working directory error: itests/thirdparty/pom.xml: already exists in working directory The patch does not appear to apply with p0, p1, or p2 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12855000 - PreCommit-HIVE-Build > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich > Attachments: HIVE-14735.1.patch, HIVE-14735.1.patch, > HIVE-14735.1.patch, HIVE-14735.1.patch, HIVE-14735.2.patch, > HIVE-14735.3.patch, HIVE-14735.4.patch, HIVE-14735.4.patch > > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879889#comment-15879889 ] Hive QA commented on HIVE-14735: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12854079/HIVE-14735.4.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3713/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3713/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3713/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2017-02-23 05:27:14.653 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-3713/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2017-02-23 05:27:14.655 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 759766e HIVE-15955: make explain formatted to include opId and etc (Pengcheng Xiong, reviewed by Ashutosh Chauhan) + git clean -f -d + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 759766e HIVE-15955: make explain formatted to include opId and etc (Pengcheng Xiong, reviewed by Ashutosh Chauhan) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2017-02-23 05:27:15.852 + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: itests/thirdparty/.gitignore: already exists in working directory error: itests/thirdparty/pom.xml: already exists in working directory The patch does not appear to apply with p0, p1, or p2 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12854079 - PreCommit-HIVE-Build > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich > Attachments: HIVE-14735.1.patch, HIVE-14735.1.patch, > HIVE-14735.1.patch, HIVE-14735.1.patch, HIVE-14735.2.patch, > HIVE-14735.3.patch, HIVE-14735.4.patch > > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15798344#comment-15798344 ] Zoltan Haindrich commented on HIVE-14735: - thanks for the info and link to HIVE-14240 [~spena], I wasn't aware that! deploying the repo files "as" a maven repository only needs the files organized in a directory tree...so it should be no problem serving them from s3 (I guess). I'll write a shell-script then...because in this case - this shouldnt stay for "long" - and will be replaced with official spark artifacts later - I will evaluate this option in the mean time... > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich > Attachments: HIVE-14735.1.patch, HIVE-14735.1.patch, > HIVE-14735.1.patch, HIVE-14735.1.patch, HIVE-14735.2.patch, HIVE-14735.3.patch > > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15761504#comment-15761504 ] Sergio Peña commented on HIVE-14735: We're using Amazon S3 to store the .tgz binary for now. That's why I was wondering if we could have a way get the files to upload them manually (to a non-mvn server). I don't know if the Spark team will agree on publishing the 'spark-without-hive' as this only benefits us. Anyway, the S3 storage is temporary until we solve the spark-without-hive dependency mentioned on HIVE-14240. Either maven or shell script is good. > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich > Attachments: HIVE-14735.1.patch, HIVE-14735.1.patch, > HIVE-14735.1.patch, HIVE-14735.1.patch, HIVE-14735.2.patch, HIVE-14735.3.patch > > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751612#comment-15751612 ] Zoltan Haindrich commented on HIVE-14735: - Hello [~spena], thank you for taking a look! :) * skipSparkAssemblyDeploy - there is a single leftover setting of this variable to true - sorry for it: it was part of the previous patch version; i'll remove it...because now its not neccessary as the thirdparty project does the unpacking - it will skip even downloading/unpacking if the tests are being skipped * in its current form the publish doesnt work; because it tries to use my own private server - in its current form gradle can upload the artifacts using ssh access to any host - to make it work with another server; both of the rxd.hu references should be changed. gradle / etc topic: * the simplest would be to move this gradle project outside the project...into a custom repo; and place pointers in the readme file to it. * if the spark project would be willing to publish the 'spark-without-hive' artifact as a zip into the central maven repo - that would make this whole gradle/etc thing unneccessary ; but in this case they would need to publish this new artifact for spark-2.0.0 - because hive currently uses that version - this has other "+" sides to. as it doesnt need an extra repository declaration. * I will look into alternatives...possibly using maven...or some shell scripts to achieve the same results as with gradle... [~spena] which one of the above would you prefer ? > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich > Attachments: HIVE-14735.1.patch, HIVE-14735.1.patch, > HIVE-14735.1.patch, HIVE-14735.1.patch, HIVE-14735.2.patch, HIVE-14735.3.patch > > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15745569#comment-15745569 ] Sergio Peña commented on HIVE-14735: - is skipSparkTests or skipSparkAssemblyDeploy? - can we use maven instead of gradle? I just want to avoid using another build tool that contributors will require to learn to do maintenance. - can you add on the README how to publish files manually? The current repo is not a maven repo, so the publish function won't work. > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich > Attachments: HIVE-14735.1.patch, HIVE-14735.1.patch, > HIVE-14735.1.patch, HIVE-14735.1.patch, HIVE-14735.2.patch, HIVE-14735.3.patch > > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15742633#comment-15742633 ] Hive QA commented on HIVE-14735: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12842623/HIVE-14735.3.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10795 tests executed *Failed tests:* {noformat} TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=108) [groupby_grouping_id2.q,input17.q,bucketmapjoin12.q,ppd_gby_join.q,auto_join10.q,ptf_rcfile.q,vectorized_rcfile_columnar.q,vector_elt.q,ppd_join5.q,ppd_join.q,join_filters_overlap.q,join_cond_pushdown_1.q,timestamp_3.q,load_dyn_part6.q,stats_noscan_2.q] TestVectorizedColumnReaderBase - did not produce a TEST-*.xml file (likely timed out) (batchId=250) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a] (batchId=134) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2] (batchId=134) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_based_fetch_decision] (batchId=150) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2542/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2542/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2542/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 10 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12842623 - PreCommit-HIVE-Build > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich > Attachments: HIVE-14735.1.patch, HIVE-14735.1.patch, > HIVE-14735.1.patch, HIVE-14735.1.patch, HIVE-14735.2.patch, HIVE-14735.3.patch > > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15736556#comment-15736556 ] Zoltan Haindrich commented on HIVE-14735: - Thank you for the command [~stakiar], i've added it to the patch. I've uploaded #3: I hope I didn't break anything ...the ptest execution will shed light on this. [~spena] i've addressed much of your comments (however I still use fixed version for the maven plugins - i've forgot fix that) and also...i've missed your previous question about "where the downloaded file is": it's inside the local maven repository. i've changed the following: * added a project to repack the spark artifact under dev-support, with a readme describing the procedure * {{itests/thirparty}} is now a module - this way these maven "tricks" are isolated, other modules rely on that thirdparty have already finished - this also enabled to support even multiple spark versions - which may come handy for people who switch between branches which pull different spark version * it now only unpacks the spark assembly to only 1 place [~spena] what do you think about the new changes? > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich > Attachments: HIVE-14735.1.patch, HIVE-14735.1.patch, > HIVE-14735.1.patch, HIVE-14735.1.patch, HIVE-14735.2.patch, HIVE-14735.3.patch > > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733799#comment-15733799 ] Ferdinand Xu commented on HIVE-14735: - Yes, please use the command as [~stakiar] said. > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich > Attachments: HIVE-14735.1.patch, HIVE-14735.1.patch, > HIVE-14735.1.patch, HIVE-14735.1.patch, HIVE-14735.2.patch > > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732729#comment-15732729 ] Sahil Takiar commented on HIVE-14735: - You have to git clone the Spark repo and run the command: {code} sh ./dev/make-distribution.sh --name hadoop2-without-hive --tgz -Phadoop-2.7 -Pyarn -Pparquet-provided -Dhadoop.version=2.7.3 {code} That will built the build Spark distribution, which will contain all the JAR files. > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich > Attachments: HIVE-14735.1.patch, HIVE-14735.1.patch, > HIVE-14735.1.patch, HIVE-14735.1.patch, HIVE-14735.2.patch > > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732711#comment-15732711 ] Sergio Peña commented on HIVE-14735: [~stakiar] [~Ferd] Do you have instructions on how to build the spark-without-hive assembly jar? Zoltan is working on a better way to download the jars, and he would like to know how the jar is built. > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich > Attachments: HIVE-14735.1.patch, HIVE-14735.1.patch, > HIVE-14735.1.patch, HIVE-14735.1.patch, HIVE-14735.2.patch > > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732552#comment-15732552 ] Zoltan Haindrich commented on HIVE-14735: - [~vgumashta] i'm sorry to hear that...there might be some issues because the patch is in prototype stage. [~spena] I've left some of the existing code as is for now - to aid reviewing the key parts which will change after this - i'll clean that up for the final version. I've published my "prototype" jar publishing mechanism on github: https://github.com/kgyrtkirk/hive-14735 - but it's currently just "repackages" the original artifact into a new form...it would make things easier if I would know more about how the current artifact is being built - I think it would be great to even add the spark-without-hive artifact builder to the hive repository > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich > Attachments: HIVE-14735.1.patch, HIVE-14735.1.patch, > HIVE-14735.1.patch, HIVE-14735.1.patch, HIVE-14735.2.patch > > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726831#comment-15726831 ] Sergio Peña commented on HIVE-14735: [~kgyrtkirk] This prototype works pretty good when downloading the file. I like it. Let's continue working on this to use it as replacement of the shell script. A few comments: * Could you use a variable for the plugin version? You can find them on the root pom.xml org.apache.maven.plugins maven-dependency-plugin 2.10 * is the 'target/download.sh' needed now? We can keep only the log4j2.properties. The log cannot be packed into the artifact because it might be edited by other Hive developers. * How do you generate the new artifacts? We should write a README for this. * Where is the file downloaded? I see this {{${project.build.directory}/spark}}, but I don't find it. Can you use the original thirdparty directory instead? > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich > Attachments: HIVE-14735.1.patch, HIVE-14735.1.patch, > HIVE-14735.1.patch, HIVE-14735.1.patch, HIVE-14735.2.patch > > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15720770#comment-15720770 ] Vaibhav Gumashta commented on HIVE-14735: - [~kgyrtkirk] The build fails for me with patch v2. > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich > Attachments: HIVE-14735.1.patch, HIVE-14735.1.patch, > HIVE-14735.1.patch, HIVE-14735.1.patch, HIVE-14735.2.patch > > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15713790#comment-15713790 ] Vaibhav Gumashta commented on HIVE-14735: - [~kgyrtkirk] I'm + 1 on this. [~spena] let us know what do you think? > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich > Attachments: HIVE-14735.1.patch, HIVE-14735.1.patch, > HIVE-14735.1.patch, HIVE-14735.1.patch, HIVE-14735.2.patch > > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15644916#comment-15644916 ] Sergio Peña commented on HIVE-14735: Thanks [~kgyrtkirk]. The patch looks good, but I need to dig a little more, and test it. But it looks promising. I'll try to review it this week. > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich > Attachments: HIVE-14735.1.patch, HIVE-14735.1.patch, > HIVE-14735.1.patch, HIVE-14735.1.patch, HIVE-14735.2.patch > > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15639454#comment-15639454 ] Zoltan Haindrich commented on HIVE-14735: - in #2 i've changed to use https instead of http - it's still a proof of concept patch. [~spena]: It seems to me that this method works how do you like it? [~sseth]: I think there were some problems with the ptest executors... in an earlier build I tried to use http protocol...and maven attempted to use localhost:3128 as the http proxy - but there were some connection refused problems...the logs are wiped since that...so i can't give a link... > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich > Attachments: HIVE-14735.1.patch, HIVE-14735.1.patch, > HIVE-14735.1.patch, HIVE-14735.1.patch, HIVE-14735.2.patch > > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15638310#comment-15638310 ] Hive QA commented on HIVE-14735: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12837309/HIVE-14735.2.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10628 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype] (batchId=148) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid] (batchId=150) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=145) org.apache.hive.spark.client.TestSparkClient.testJobSubmission (batchId=272) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1979/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1979/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-1979/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12837309 - PreCommit-HIVE-Build > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich > Attachments: HIVE-14735.1.patch, HIVE-14735.1.patch, > HIVE-14735.1.patch, HIVE-14735.1.patch, HIVE-14735.2.patch > > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634146#comment-15634146 ] Hive QA commented on HIVE-14735: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12836920/HIVE-14735.1.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1953/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1953/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-1953/ Messages: {noformat} This message was trimmed, see log for full details [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/Iterable.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/2.7.2/hadoop-common-2.7.2.jar(org/apache/hadoop/io/Writable.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/String.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/aggregate/jetty-all-server/7.6.0.v20120127/jetty-all-server-7.6.0.v20120127.jar(org/eclipse/jetty/http/HttpStatus.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/HashMap.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/com/sun/jersey/jersey-core/1.14/jersey-core-1.14.jar(javax/ws/rs/core/MediaType.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/com/sun/jersey/jersey-core/1.14/jersey-core-1.14.jar(javax/ws/rs/core/Response.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar(org/codehaus/jackson/map/ObjectMapper.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/Exception.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/Throwable.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/Serializable.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/Enum.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/Comparable.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/com/sun/jersey/jersey-server/1.14/jersey-server-1.14.jar(com/sun/jersey/api/core/PackagesResourceConfig.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/com/sun/jersey/jersey-servlet/1.14/jersey-servlet-1.14.jar(com/sun/jersey/spi/container/servlet/ServletContainer.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/FileInputStream.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/ql/target/hive-exec-2.2.0-SNAPSHOT.jar(org/apache/commons/lang3/StringUtils.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/ql/target/hive-exec-2.2.0-SNAPSHOT.jar(org/apache/commons/lang3/ArrayUtils.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/common/target/hive-common-2.2.0-SNAPSHOT.jar(org/apache/hadoop/hive/common/classification/InterfaceStability.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-hdfs/2.7.2/hadoop-hdfs-2.7.2.jar(org/apache/hadoop/hdfs/web/AuthFilter.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-2.2.0-SNAPSHOT.jar(org/apache/hadoop/hive/shims/Utils.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/2.7.2/hadoop-common-2.7.2.jar(org/apache/hadoop/security/UserGroupInformation.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-auth/2.7.2/hadoop-auth-2.7.2.jar(org/apache/hadoop/security/authentication/client/PseudoAuthenticator.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-auth/2.7.2/hadoop-auth-2.7.2.jar(org/apache/hadoop/security/authentication/server/PseudoAuthenticationHandler.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/2.7.2/hadoop-common-2.7.2.jar(org/apache/hadoop/util/GenericOptionsParser.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/aggregate/jetty-all-server/7.6.0.v20120127/jetty-all-server-7.6.0.v20120127.jar(org/eclipse/jetty/rewrite/handler/RedirectPatternRule.class)]] [loading
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633701#comment-15633701 ] Hive QA commented on HIVE-14735: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12836895/HIVE-14735.1.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1949/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1949/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-1949/ Messages: {noformat} This message was trimmed, see log for full details [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/Iterable.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/2.7.2/hadoop-common-2.7.2.jar(org/apache/hadoop/io/Writable.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/String.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/aggregate/jetty-all-server/7.6.0.v20120127/jetty-all-server-7.6.0.v20120127.jar(org/eclipse/jetty/http/HttpStatus.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/HashMap.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/com/sun/jersey/jersey-core/1.14/jersey-core-1.14.jar(javax/ws/rs/core/MediaType.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/com/sun/jersey/jersey-core/1.14/jersey-core-1.14.jar(javax/ws/rs/core/Response.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar(org/codehaus/jackson/map/ObjectMapper.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/Exception.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/Throwable.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/Serializable.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/Enum.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/Comparable.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/com/sun/jersey/jersey-server/1.14/jersey-server-1.14.jar(com/sun/jersey/api/core/PackagesResourceConfig.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/com/sun/jersey/jersey-servlet/1.14/jersey-servlet-1.14.jar(com/sun/jersey/spi/container/servlet/ServletContainer.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/FileInputStream.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/ql/target/hive-exec-2.2.0-SNAPSHOT.jar(org/apache/commons/lang3/StringUtils.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/ql/target/hive-exec-2.2.0-SNAPSHOT.jar(org/apache/commons/lang3/ArrayUtils.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/common/target/hive-common-2.2.0-SNAPSHOT.jar(org/apache/hadoop/hive/common/classification/InterfaceStability.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-hdfs/2.7.2/hadoop-hdfs-2.7.2.jar(org/apache/hadoop/hdfs/web/AuthFilter.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-2.2.0-SNAPSHOT.jar(org/apache/hadoop/hive/shims/Utils.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/2.7.2/hadoop-common-2.7.2.jar(org/apache/hadoop/security/UserGroupInformation.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-auth/2.7.2/hadoop-auth-2.7.2.jar(org/apache/hadoop/security/authentication/client/PseudoAuthenticator.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-auth/2.7.2/hadoop-auth-2.7.2.jar(org/apache/hadoop/security/authentication/server/PseudoAuthenticationHandler.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/2.7.2/hadoop-common-2.7.2.jar(org/apache/hadoop/util/GenericOptionsParser.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/aggregate/jetty-all-server/7.6.0.v20120127/jetty-all-server-7.6.0.v20120127.jar(org/eclipse/jetty/rewrite/handler/RedirectPatternRule.class)]] [loading
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15615854#comment-15615854 ] Sergio Peña commented on HIVE-14735: The TAR file is uploaded manually to a public server where we the community have read-only access. This is not very good (as the community can't help too much), but that was the only way we found to do it. I don't know how Maven would work with this file. If there is a way to keep it published in Maven, and tell maven to download it and cache it, then it would be a lot of help. This file is only used by Hive, and it is a spark assembly built without hive libraries, so I don't think it should be inside the spark project. Btw, these are not JARS, but an assembly package from Spark used to execute a minispark or something. How would we deploy this on Maven? > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15614347#comment-15614347 ] Zoltan Haindrich commented on HIVE-14735: - I think it would be better (and possibly more portable) to load this custom artifact by utilizing maven to download it from a maven repository.. This would also enable maven to cache this file in the local maven repo...which is not affected by {{git clean -dfx}}, and may reduce unneccessary network load. Any problems with this? - i can't put this this spark artifact into maven central (at least not under org.apache.spark) - someone would be intrested in putting it there - the best would be inside the spark project? ...or it doesn't matter where it comes from as long as it works ;) hmm..i've just started wonderinghow those cloudfront artifacts are deployed? > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536491#comment-15536491 ] Vaibhav Gumashta commented on HIVE-14735: - Still getting the same hash: {code} vgumashta:hive vgumashta$ md5 -r ./itests/thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz b3889e892b3d290a5ef95b0047295f80 ./itests/thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz vgumashta:hive vgumashta$ md5 ./itests/thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz MD5 (./itests/thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz) = b3889e892b3d290a5ef95b0047295f80 {code} Are you on current master? > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536423#comment-15536423 ] Sergio Peña commented on HIVE-14735: That hash looks different. I read that to get a similar output as md5sum in osx you run {{md5 -r file}}. Can you try it? https://www.garron.me/en/bits/how-to-md5sum-mac-os-x.html > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536408#comment-15536408 ] Vaibhav Gumashta commented on HIVE-14735: - OSX has an equivalent md5 command: {code} MD5 (./itests/thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz) = b3889e892b3d290a5ef95b0047295f80 {code} > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536175#comment-15536175 ] Sergio Peña commented on HIVE-14735: This is how md5sum works on linux: {noformat} $ md5sum spark-1.6.0-bin-hadoop2-without-hive.tgz 296c808fe75a09518226fc149752bf3f spark-1.6.0-bin-hadoop2-without-hive.tgz {noformat} What is the behavior in OSX? I don't have access to it. If we can get the same MD5 sum with it, then I can add that to the code to use md5sum or md5. > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15535044#comment-15535044 ] Vaibhav Gumashta commented on HIVE-14735: - Looks like OSX may not have md5sum installed by default. Should we use md5 on OSX? > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534076#comment-15534076 ] Matt McCline commented on HIVE-14735: - Oh, on my Mac laptop and usually current master. > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534060#comment-15534060 ] Sergio Peña commented on HIVE-14735: are you running linux or mac? which linux distro if so? > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533893#comment-15533893 ] Matt McCline commented on HIVE-14735: - {code} ... [exec] arget/spark [exec] + [[ ! -f ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz ]] [exec] + local md5File=spark-1.6.0-bin-hadoop2-without-hive.tgz.md5sum [exec] + curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz.md5sum http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz.md5sum [exec] + cd ./../thirdparty [exec] + md5sum -c spark-1.6.0-bin-hadoop2-without-hive.tgz.md5sum [exec] ../target/download.sh: line 18: md5sum: command not found [exec] + curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz [exec] + cd - [exec] + tar -zxf ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz -C ./target [exec] /Users/mmccline/VecDetail/itests/qtest-spark [exec] + mv ./target/spark-1.6.0-bin-hadoop2-without-hive ./target/spark [exec] + cp -f ./target/../../..//data/conf/spark/log4j2.properties ./target/spark/conf/ {code} After the "./target/download.sh: line 18: md5sum: command not found" line, the download of "+ curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz " takes a very long time and happens everytime. I tried downloading a version of md5sum and that seems to make it worse -- the build went off and hung. > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15532978#comment-15532978 ] Sergio Peña commented on HIVE-14735: We added a .md5sum spark file to detect if a file must be downloaded again in the next build. This saves time if you already have an exact copy of the spark assembly. Where is the issue happening? On our Jenkins build? > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531584#comment-15531584 ] Rui Li commented on HIVE-14735: --- I think that means you have to install md5sum. Maybe we should document md5sum is required for the build. [~spena], any ideas? > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529319#comment-15529319 ] Matt McCline commented on HIVE-14735: - It is preceeded by ../target/download.sh: line 18: md5sum: command not found > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529318#comment-15529318 ] Matt McCline commented on HIVE-14735: - It is preceeded by ../target/download.sh: line 18: md5sum: command not found > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)