[
https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15616460#comment-15616460
]
Zoltan Haindrich edited comment on HIVE-14735 at 10/28/16 8:57 PM:
-------------------------------------------------------------------
[~spena] it's good to know these things...i've tryed this out - because in
gradle this would be easy...i assumed maven can do it too...well, it seems it
does! but it needs quite a bunch of xml to do something like this ;)
i've experimented with it...and it looks like it works - i would like to submit
a test ptest run to check that everything is all right - but since my own
server serves this spark related maven repo, I don't really want it to go in ;)
i've published a preliminary "conceptional" repackaging tool here:
https://github.com/kgyrtkirk/hive-14735
[~spena], can you take a look at it, and see if it could be a viable
alternative for the current artifact delivery method (or not)...I've tryed it
out locally...in the readme i've sketched my steps how I tried it out - hope it
helps evaluating it!
I think this will eventually work...download/unpack/etc is done by maven
plugins which should be highly portable.
notes:
* {{mvn clean}} clears the unpacked things - which is good
* unpacking a new version doesn't remove the old files, just pastes the new
tree on top of it...but in case someone changes distinct branches I think he
will use {{mvn clean}} or a harder {{git clean -dfx}} - so it should be ok
* there is a log4j2 properties filewhich gets copied into this unpacked
directory...it can be included in the artifact...or keep it like this?
was (Author: kgyrtkirk):
[~spena] it's good to know these things...i've tryed this out - because in
gradle this would be easy...i assumed maven can do it too...well, it seems it
does! but it needs quite a bunch of xml to do something like this ;)
i've experimented with it...and it looks like it works - i would like to submit
a test ptest run to check that everything is all right - but since my own
server serves this spark related maven repo, I don't really want it to go in ;)
i've published a preliminary "conceptional" repackaging tool here:
https://github.com/kgyrtkirk/hive-14735
[~spena], you might want to try it out locally, i've sketched a readme in that
git repo - hope it helps evaluating this maven option.
I think this will eventually work...download/unpack/etc is done by maven
plugins which should be highly portable.
notes:
* {{mvn clean}} clears the unpacked things - which is good
* unpacking a new version doesn't remove the old files, just pastes the new
tree on top of it...but in case someone changes distinct branches I think he
will use {{mvn clean}} or a harder {{git clean -dfx}} - so it should be ok
* there is a log4j2 properties filewhich gets copied into this unpacked
directory...it can be included in the artifact...or keep it like this?
> Build Infra: Spark artifacts download takes a long time
> -------------------------------------------------------
>
> Key: HIVE-14735
> URL: https://issues.apache.org/jira/browse/HIVE-14735
> Project: Hive
> Issue Type: Bug
> Components: Build Infrastructure
> Reporter: Vaibhav Gumashta
> Assignee: Zoltan Haindrich
> Attachments: HIVE-14735.1.patch
>
>
> In particular this command:
> {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz
> http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
