[jira] [Commented] (HIVE-9664) Hive "add jar" command should be able to download and add jars from a repository
[
https://issues.apache.org/jira/browse/HIVE-9664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823584#comment-15823584
]
anishek commented on HIVE-9664:
---
So is hhtp scheme not supported ?
{code}
add [FILE|JAR|ARCHIVE] *
{code}
> Hive "add jar" command should be able to download and add jars from a
> repository
>
>
> Key: HIVE-9664
> URL: https://issues.apache.org/jira/browse/HIVE-9664
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 0.14.0
>Reporter: Anant Nag
>Assignee: Anant Nag
> Labels: hive, patch
> Fix For: 1.2.0
>
> Attachments: HIVE-9664.4.patch, HIVE-9664.5.patch, HIVE-9664.patch,
> HIVE-9664.patch, HIVE-9664.patch
>
>
> Currently Hive's "add jar" command takes a local path to the dependency jar.
> This clutters the local file-system as users may forget to remove this jar
> later
> It would be nice if Hive supported a Gradle like notation to download the jar
> from a repository.
> Example: add jar org:module:version
>
> It should also be backward compatible and should take jar from the local
> file-system as well.
> RB: https://reviews.apache.org/r/31628/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9664) Hive add jar command should be able to download and add jars from a repository
[ https://issues.apache.org/jira/browse/HIVE-9664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573744#comment-14573744 ] Anthony Hsu commented on HIVE-9664: --- Thanks. Looks good. Hive add jar command should be able to download and add jars from a repository Key: HIVE-9664 URL: https://issues.apache.org/jira/browse/HIVE-9664 Project: Hive Issue Type: Improvement Affects Versions: 0.14.0 Reporter: Anant Nag Assignee: Anant Nag Labels: TODOC1.2, hive, patch Fix For: 1.2.0 Attachments: HIVE-9664.4.patch, HIVE-9664.5.patch, HIVE-9664.patch, HIVE-9664.patch, HIVE-9664.patch Currently Hive's add jar command takes a local path to the dependency jar. This clutters the local file-system as users may forget to remove this jar later It would be nice if Hive supported a Gradle like notation to download the jar from a repository. Example: add jar org:module:version It should also be backward compatible and should take jar from the local file-system as well. RB: https://reviews.apache.org/r/31628/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9664) Hive add jar command should be able to download and add jars from a repository
[ https://issues.apache.org/jira/browse/HIVE-9664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572528#comment-14572528 ] Anant Nag commented on HIVE-9664: - Yes, this can be done. I'll update the wiki to make it more clear. Hive add jar command should be able to download and add jars from a repository Key: HIVE-9664 URL: https://issues.apache.org/jira/browse/HIVE-9664 Project: Hive Issue Type: Improvement Affects Versions: 0.14.0 Reporter: Anant Nag Assignee: Anant Nag Labels: TODOC1.2, hive, patch Fix For: 1.2.0 Attachments: HIVE-9664.4.patch, HIVE-9664.5.patch, HIVE-9664.patch, HIVE-9664.patch, HIVE-9664.patch Currently Hive's add jar command takes a local path to the dependency jar. This clutters the local file-system as users may forget to remove this jar later It would be nice if Hive supported a Gradle like notation to download the jar from a repository. Example: add jar org:module:version It should also be backward compatible and should take jar from the local file-system as well. RB: https://reviews.apache.org/r/31628/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9664) Hive add jar command should be able to download and add jars from a repository
[
https://issues.apache.org/jira/browse/HIVE-9664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14571603#comment-14571603
]
Anthony Hsu commented on HIVE-9664:
---
Looks good. One question: Can you mix Ivy and other URIs in the same ADD JAR
command? Something like:
{code}
ADD JARS ivy://... hdfs://...;
{code}
Hive add jar command should be able to download and add jars from a
repository
Key: HIVE-9664
URL: https://issues.apache.org/jira/browse/HIVE-9664
Project: Hive
Issue Type: Improvement
Affects Versions: 0.14.0
Reporter: Anant Nag
Assignee: Anant Nag
Labels: TODOC1.2, hive, patch
Fix For: 1.2.0
Attachments: HIVE-9664.4.patch, HIVE-9664.5.patch, HIVE-9664.patch,
HIVE-9664.patch, HIVE-9664.patch
Currently Hive's add jar command takes a local path to the dependency jar.
This clutters the local file-system as users may forget to remove this jar
later
It would be nice if Hive supported a Gradle like notation to download the jar
from a repository.
Example: add jar org:module:version
It should also be backward compatible and should take jar from the local
file-system as well.
RB: https://reviews.apache.org/r/31628/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9664) Hive add jar command should be able to download and add jars from a repository
[ https://issues.apache.org/jira/browse/HIVE-9664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14570996#comment-14570996 ] Anant Nag commented on HIVE-9664: - I've updated the documentation. Please have a look and let me know if something is incorrect or requires improvement. Hive add jar command should be able to download and add jars from a repository Key: HIVE-9664 URL: https://issues.apache.org/jira/browse/HIVE-9664 Project: Hive Issue Type: Improvement Affects Versions: 0.14.0 Reporter: Anant Nag Assignee: Anant Nag Labels: TODOC1.2, hive, patch Fix For: 1.2.0 Attachments: HIVE-9664.4.patch, HIVE-9664.5.patch, HIVE-9664.patch, HIVE-9664.patch, HIVE-9664.patch Currently Hive's add jar command takes a local path to the dependency jar. This clutters the local file-system as users may forget to remove this jar later It would be nice if Hive supported a Gradle like notation to download the jar from a repository. Example: add jar org:module:version It should also be backward compatible and should take jar from the local file-system as well. RB: https://reviews.apache.org/r/31628/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9664) Hive add jar command should be able to download and add jars from a repository
[ https://issues.apache.org/jira/browse/HIVE-9664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568021#comment-14568021 ] Anthony Hsu commented on HIVE-9664: --- Hey [~nntnag17], Could you please update the [Hive CLI documentation|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Cli] with information on how to use this feature you added? [~leftylev] suggested some places to update in [this comment|https://issues.apache.org/jira/browse/HIVE-9664?focusedCommentId=1439page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-1439]. Hive add jar command should be able to download and add jars from a repository Key: HIVE-9664 URL: https://issues.apache.org/jira/browse/HIVE-9664 Project: Hive Issue Type: Improvement Affects Versions: 0.14.0 Reporter: Anant Nag Assignee: Anant Nag Labels: TODOC1.2, hive, patch Fix For: 1.2.0 Attachments: HIVE-9664.4.patch, HIVE-9664.5.patch, HIVE-9664.patch, HIVE-9664.patch, HIVE-9664.patch Currently Hive's add jar command takes a local path to the dependency jar. This clutters the local file-system as users may forget to remove this jar later It would be nice if Hive supported a Gradle like notation to download the jar from a repository. Example: add jar org:module:version It should also be backward compatible and should take jar from the local file-system as well. RB: https://reviews.apache.org/r/31628/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9664) Hive add jar command should be able to download and add jars from a repository
[
https://issues.apache.org/jira/browse/HIVE-9664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14511760#comment-14511760
]
Sushanth Sowmyan commented on HIVE-9664:
One more note - this jira also makes changes in itests/pom.xml that are not
portable:
{code}
diff --git a/itests/pom.xml b/itests/pom.xml
index
fd4bb41fd8f74cdf261b791d1944dbca4b04f927..6f6cf742c41a11647589692ac4f266f467be2812
100644
--- a/itests/pom.xml
+++ b/itests/pom.xml
@@ -93,6 +93,9 @@
mkdir -p $DOWNLOAD_DIR
download
http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-${spark.version}-bin-hadoop2-without-hive.tgz;
spark
cp -f $HIVE_ROOT/data/conf/spark/log4j.properties
$BASE_DIR/spark/conf/
+ sed '/package /d'
${basedir}/${hive.path.to.root}/contrib/src/java/org/apache/hadoop/hive/contrib/udf/example/UDFExampleAdd.java
/tmp/UDFExampleAdd.java
+ javac -cp
${settings.localRepository}/org/apache/hive/hive-exec/${project.version}/hive-exec-${project.version}.jar
/tmp/UDFExampleAdd.java -d /tmp
+ jar -cf /tmp/udfexampleadd-1.0.jar -C /tmp
UDFExampleAdd.class
/echo
/target
/configuration
{code}
sed is not something that's available in windows, and instead of compiling
UDFExampleAdd.java in here, it might be better to create another quick jar
build in the contrib package, and depend on it from the qtests to show that
we're able to appropriately add jars. (Again, moving this to HIVE-10251 as well)
Hive add jar command should be able to download and add jars from a
repository
Key: HIVE-9664
URL: https://issues.apache.org/jira/browse/HIVE-9664
Project: Hive
Issue Type: Improvement
Affects Versions: 0.14.0
Reporter: Anant Nag
Assignee: Anant Nag
Labels: TODOC1.2, hive, patch
Fix For: 1.2.0
Attachments: HIVE-9664.4.patch, HIVE-9664.5.patch, HIVE-9664.patch,
HIVE-9664.patch, HIVE-9664.patch
Currently Hive's add jar command takes a local path to the dependency jar.
This clutters the local file-system as users may forget to remove this jar
later
It would be nice if Hive supported a Gradle like notation to download the jar
from a repository.
Example: add jar org:module:version
It should also be backward compatible and should take jar from the local
file-system as well.
RB: https://reviews.apache.org/r/31628/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9664) Hive add jar command should be able to download and add jars from a repository
[
https://issues.apache.org/jira/browse/HIVE-9664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484482#comment-14484482
]
Sushanth Sowmyan commented on HIVE-9664:
Hi [~cwsteinbach]/[~nntnag17]/[~erwaman], this patch seems to have broken
CliSessionState instantiation with a NPE:
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.session.DependencyResolver.init(DependencyResolver.java:61)
at org.apache.hadoop.hive.ql.session.SessionState.init(SessionState.java:343)
at org.apache.hadoop.hive.ql.session.SessionState.init(SessionState.java:334)
at org.apache.hadoop.hive.cli.CliSessionState.init(CliSessionState.java:60)
{noformat}
This happens because of the following bit:
{noformat}
// If HIVE_HOME is not defined or file is not found in HIVE_HOME/conf then
load default ivysettings.xml from class loader
if (ivysettingsPath == null || !(new File(ivysettingsPath).exists())) {
ivysettingsPath =
ClassLoader.getSystemResource(ivysettings.xml).getFile();
_console.printInfo(ivysettings.xml file not found in HIVE_HOME or
HIVE_CONF_DIR, + ivysettingsPath + will be used);
}
{noformat}
This makes it so that an attempt to instantiate CliSessionState without an
ivysettings.xml file will cause hive to fail with an NPE. Hive should not have
a hard dependency on a ivysettings,xml being present, and this feature should
gracefully fail in that case instead.
Could someone please fix this please?
Hive add jar command should be able to download and add jars from a
repository
Key: HIVE-9664
URL: https://issues.apache.org/jira/browse/HIVE-9664
Project: Hive
Issue Type: Improvement
Affects Versions: 0.14.0
Reporter: Anant Nag
Assignee: Anant Nag
Labels: TODOC1.2, hive, patch
Fix For: 1.2.0
Attachments: HIVE-9664.4.patch, HIVE-9664.5.patch, HIVE-9664.patch,
HIVE-9664.patch, HIVE-9664.patch
Currently Hive's add jar command takes a local path to the dependency jar.
This clutters the local file-system as users may forget to remove this jar
later
It would be nice if Hive supported a Gradle like notation to download the jar
from a repository.
Example: add jar org:module:version
It should also be backward compatible and should take jar from the local
file-system as well.
RB: https://reviews.apache.org/r/31628/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9664) Hive add jar command should be able to download and add jars from a repository
[ https://issues.apache.org/jira/browse/HIVE-9664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484486#comment-14484486 ] Sushanth Sowmyan commented on HIVE-9664: I've opened HIVE-10251 for the same. Hive add jar command should be able to download and add jars from a repository Key: HIVE-9664 URL: https://issues.apache.org/jira/browse/HIVE-9664 Project: Hive Issue Type: Improvement Affects Versions: 0.14.0 Reporter: Anant Nag Assignee: Anant Nag Labels: TODOC1.2, hive, patch Fix For: 1.2.0 Attachments: HIVE-9664.4.patch, HIVE-9664.5.patch, HIVE-9664.patch, HIVE-9664.patch, HIVE-9664.patch Currently Hive's add jar command takes a local path to the dependency jar. This clutters the local file-system as users may forget to remove this jar later It would be nice if Hive supported a Gradle like notation to download the jar from a repository. Example: add jar org:module:version It should also be backward compatible and should take jar from the local file-system as well. RB: https://reviews.apache.org/r/31628/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9664) Hive add jar command should be able to download and add jars from a repository
[ https://issues.apache.org/jira/browse/HIVE-9664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1439#comment-1439 ] Lefty Leverenz commented on HIVE-9664: -- Doc note: The ADD FILE | JAR | ARCHIVE commands are documented in several places, so this information needs to be added to all of them or perhaps just to Hive Resources in the CLI doc with links from the others. * [Commands | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Commands] * [HiveServer2 Clients -- Beeline Hive Commands | https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-BeelineHiveCommands] * [CLI -- Hive Interactive Shell Commands | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Cli#LanguageManualCli-HiveInteractiveShellCommands] * [CLI -- Hive Resources | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Cli#LanguageManualCli-HiveResources] Hive add jar command should be able to download and add jars from a repository Key: HIVE-9664 URL: https://issues.apache.org/jira/browse/HIVE-9664 Project: Hive Issue Type: Improvement Affects Versions: 0.14.0 Reporter: Anant Nag Assignee: Anant Nag Labels: TODOC1.2, hive, patch Fix For: 1.2.0 Attachments: HIVE-9664.4.patch, HIVE-9664.5.patch, HIVE-9664.patch, HIVE-9664.patch, HIVE-9664.patch Currently Hive's add jar command takes a local path to the dependency jar. This clutters the local file-system as users may forget to remove this jar later It would be nice if Hive supported a Gradle like notation to download the jar from a repository. Example: add jar org:module:version It should also be backward compatible and should take jar from the local file-system as well. RB: https://reviews.apache.org/r/31628/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9664) Hive add jar command should be able to download and add jars from a repository
[
https://issues.apache.org/jira/browse/HIVE-9664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380792#comment-14380792
]
Hive QA commented on HIVE-9664:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12707187/HIVE-9664.5.patch
{color:green}SUCCESS:{color} +1 8344 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3153/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3153/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3153/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12707187 - PreCommit-HIVE-TRUNK-Build
Hive add jar command should be able to download and add jars from a
repository
Key: HIVE-9664
URL: https://issues.apache.org/jira/browse/HIVE-9664
Project: Hive
Issue Type: Improvement
Affects Versions: 0.14.0
Reporter: Anant Nag
Assignee: Anant Nag
Labels: hive, patch
Attachments: HIVE-9664.4.patch, HIVE-9664.5.patch, HIVE-9664.patch,
HIVE-9664.patch, HIVE-9664.patch
Currently Hive's add jar command takes a local path to the dependency jar.
This clutters the local file-system as users may forget to remove this jar
later
It would be nice if Hive supported a Gradle like notation to download the jar
from a repository.
Example: add jar org:module:version
It should also be backward compatible and should take jar from the local
file-system as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9664) Hive add jar command should be able to download and add jars from a repository
[ https://issues.apache.org/jira/browse/HIVE-9664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380928#comment-14380928 ] Anthony Hsu commented on HIVE-9664: --- +1 (though I don't have Hive commit access) Hive add jar command should be able to download and add jars from a repository Key: HIVE-9664 URL: https://issues.apache.org/jira/browse/HIVE-9664 Project: Hive Issue Type: Improvement Affects Versions: 0.14.0 Reporter: Anant Nag Assignee: Anant Nag Labels: hive, patch Attachments: HIVE-9664.4.patch, HIVE-9664.5.patch, HIVE-9664.patch, HIVE-9664.patch, HIVE-9664.patch Currently Hive's add jar command takes a local path to the dependency jar. This clutters the local file-system as users may forget to remove this jar later It would be nice if Hive supported a Gradle like notation to download the jar from a repository. Example: add jar org:module:version It should also be backward compatible and should take jar from the local file-system as well. RB: https://reviews.apache.org/r/31628/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9664) Hive add jar command should be able to download and add jars from a repository
[
https://issues.apache.org/jira/browse/HIVE-9664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375780#comment-14375780
]
Hive QA commented on HIVE-9664:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12706501/HIVE-9664.4.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 7825 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_artifactoryDownload
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3116/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3116/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3116/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12706501 - PreCommit-HIVE-TRUNK-Build
Hive add jar command should be able to download and add jars from a
repository
Key: HIVE-9664
URL: https://issues.apache.org/jira/browse/HIVE-9664
Project: Hive
Issue Type: Improvement
Affects Versions: 0.14.0
Reporter: Anant Nag
Assignee: Anant Nag
Labels: hive, patch
Attachments: HIVE-9664.4.patch, HIVE-9664.patch, HIVE-9664.patch,
HIVE-9664.patch
Currently Hive's add jar command takes a local path to the dependency jar.
This clutters the local file-system as users may forget to remove this jar
later
It would be nice if Hive supported a Gradle like notation to download the jar
from a repository.
Example: add jar org:module:version
It should also be backward compatible and should take jar from the local
file-system as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9664) Hive add jar command should be able to download and add jars from a repository
[
https://issues.apache.org/jira/browse/HIVE-9664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14376566#comment-14376566
]
Anthony Hsu commented on HIVE-9664:
---
Seems like the {{artifactoryDownload.q}} test is failing. Also, small
suggestion: I think a name like {{ivyDownload.q}} is more appropriate, since
the repository may not be Artifactory.
Hive add jar command should be able to download and add jars from a
repository
Key: HIVE-9664
URL: https://issues.apache.org/jira/browse/HIVE-9664
Project: Hive
Issue Type: Improvement
Affects Versions: 0.14.0
Reporter: Anant Nag
Assignee: Anant Nag
Labels: hive, patch
Attachments: HIVE-9664.4.patch, HIVE-9664.patch, HIVE-9664.patch,
HIVE-9664.patch
Currently Hive's add jar command takes a local path to the dependency jar.
This clutters the local file-system as users may forget to remove this jar
later
It would be nice if Hive supported a Gradle like notation to download the jar
from a repository.
Example: add jar org:module:version
It should also be backward compatible and should take jar from the local
file-system as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9664) Hive add jar command should be able to download and add jars from a repository
[
https://issues.apache.org/jira/browse/HIVE-9664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375242#comment-14375242
]
Anthony Hsu commented on HIVE-9664:
---
Hey [~nntnag17],
Looks like the build is failing with your latest patch (see the Hive QA comment
above):
{code}
[exec] cp: cannot stat
`/data/hive-ptest/working/apache-svn-trunk-source/itests/../data/files/udfexampleadd-1.0.jar':
No such file or directory
{code}
Can you please fix this and upload a new patch? Also, could you name your
patches {{HIVE-9664.N.patch}}, where {{N}} is the revision number. This is
suggested
[here|https://cwiki.apache.org/confluence/display/Hive/HowToContribute#HowToContribute-UpdatingaPatch].
Hive add jar command should be able to download and add jars from a
repository
Key: HIVE-9664
URL: https://issues.apache.org/jira/browse/HIVE-9664
Project: Hive
Issue Type: Improvement
Affects Versions: 0.14.0
Reporter: Anant Nag
Assignee: Anant Nag
Labels: hive, patch
Attachments: HIVE-9664.patch, HIVE-9664.patch, HIVE-9664.patch
Currently Hive's add jar command takes a local path to the dependency jar.
This clutters the local file-system as users may forget to remove this jar
later
It would be nice if Hive supported a Gradle like notation to download the jar
from a repository.
Example: add jar org:module:version
It should also be backward compatible and should take jar from the local
file-system as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9664) Hive add jar command should be able to download and add jars from a repository
[
https://issues.apache.org/jira/browse/HIVE-9664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365100#comment-14365100
]
Hive QA commented on HIVE-9664:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12705028/HIVE-9664.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3055/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3055/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3055/
Messages:
{noformat}
This message was trimmed, see log for full details
[INFO] Installing
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/target/hive-it-custom-serde-1.2.0-SNAPSHOT.jar
to
/data/hive-ptest/working/maven/org/apache/hive/hive-it-custom-serde/1.2.0-SNAPSHOT/hive-it-custom-serde-1.2.0-SNAPSHOT.jar
[INFO] Installing
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/pom.xml to
/data/hive-ptest/working/maven/org/apache/hive/hive-it-custom-serde/1.2.0-SNAPSHOT/hive-it-custom-serde-1.2.0-SNAPSHOT.pom
[INFO]
[INFO]
[INFO] Building Hive Integration - HCatalog Unit Tests 1.2.0-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-hcatalog-it-unit
---
[INFO] Deleting
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit (includes
= [datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @
hive-hcatalog-it-unit ---
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (download-spark) @ hive-hcatalog-it-unit
---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @
hive-hcatalog-it-unit ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
hive-hcatalog-it-unit ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/src/main/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @
hive-hcatalog-it-unit ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @
hive-hcatalog-it-unit ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
hive-hcatalog-it-unit ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/src/test/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @
hive-hcatalog-it-unit ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/target/tmp
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/target/warehouse
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/target/tmp/conf
[copy] Copying 11 files to
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
hive-hcatalog-it-unit ---
[INFO] Compiling 8 source files to
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/target/test-classes
[WARNING]
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/ManyMiniCluster.java:
Some input files use or override a deprecated API.
[WARNING]
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/ManyMiniCluster.java:
Recompile with -Xlint:deprecation for details.
[INFO]
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @
hive-hcatalog-it-unit ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-hcatalog-it-unit ---
[INFO] Building jar:
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/target/hive-hcatalog-it-unit-1.2.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
hive-hcatalog-it-unit ---
[INFO]
[INFO] --- maven-jar-plugin:2.2:test-jar (default) @ hive-hcatalog-it-unit ---
[INFO] Building jar:
[jira] [Commented] (HIVE-9664) Hive add jar command should be able to download and add jars from a repository
[
https://issues.apache.org/jira/browse/HIVE-9664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338279#comment-14338279
]
Anant Nag commented on HIVE-9664:
-
[~appodictic], [~leftylev]: Thank you for your input. I'm making use of the
grapes internally to download resources from the artifactory.
Following are the changes that will be made to the commands:
Add resource command:
Adding resources from hdfs and local filesystem works as before. Resources can
be added from the artifactory using the following:
Syntax:
{code}
add jar ivy://org:module:version?query_string jar
ivy://org:module:version?query_string*
{code}
group - Which module group the module comes from. Translates directly to a
Maven groupId or an Ivy Organization.
module - The name of the module to load. Translated directly to a Maven
artifactId or an Ivy artifact.
version - The version of the module to use. Any version or * (for latest) or an
Ivy Range '[2.2.1,)' meaning 2.2.1 or any greater version) can be used.
Various parameters can be passed in the url to configure how and which jars are
added to the artifactory. The parameters are in the form of key value pairs
where key is the name of the paramter and the value is the value of the
parameter. These key-value pairs should be separated by .
example:
{code}
add jar ivy://org:module:version?key=valuekey1=value1key2=value2;
{code}
The different parameters that can be passed are:
1) exclude: Takes a comma separated value of the form (org:module)
Sometimes you will want to exclude transitive dependencies as you might be
already using a slightly different but compatible version of some artifact.
exclude can be defined in the query string. It takes a comma(,) separated
values of the form org:module, all these dependencies won't be downloaded from
the artifactory.
Usage:
{code}
add jar ivy://org:module:version?exclude=org1:module1,org2:module2,org3:module3;
{code}
Example:
{code}
add jar ivy://org.apache.pig:pig:0.10.0?exclude=org.apache.hadoop:avro;
{code}
2) transitive: Takes values true or false. Defaults to true.
When transitive = true, all the transitive dependencies are downloaded and
added to the classpath.
Usage:
{code}
add jar ivy://org:module:version?transitive=trueexclude=org1:module1
add jar ivy://org.module:version?transitive=false;
{code}
example:
{code}
add jar
ivy://org.apache.pig:pig:0.10.0?exclude=org.apache.hadoop:avrotransitive=true;
add jar ivy://org.apache.pig:pig:0.10.0?transitive=false
{code}
3) ext: The extension of the file to add. jar by default.
example:
{code}
add jar ivy://org.apache.pig:pig:0.10.0?ext=jar
add jar
ivy://com.linkedin.informed-bridge-jira:informed-bridge-jira:0.0.3?classifier=docs-jsonext=tar.gz;
{code}
4) classifier: The maven classifier to resolve by
example:
{code}
add jar
ivy://com.linkedin.informed-bridge-jira:informed-bridge-jira:0.0.3?classifier=docs-jsonext=tar.gz;
{code}
Delete resource command:
Delete resource works as before for the resources added from hdfs and local
file system.
Resources added from the artifactory can be deleted using the following command.
{code}
delete jar ivy://org:module:version
{code}
The delete jar command will delete all the transitive dependencies of the jar
which was added using the same org:module:version.
If two jars share a set of transitive dependencies and one of the jars is
deleted using the above syntax, then all the transitive dependencies will be
deleted for the jar except the ones which are shared.
Example:
{code}
add jar ivy://org.apache.pig:pig:0.10.0
add jar ivy://org.apache.pig:pig:0.11.1.15
delete jar ivy://org.apache.pig:pig:0.10.0
{code}
if A is the set containing the transitive dependencies of pig-0.10.0 and B is
the set containing the transitive dependencies of pig-0.11.1.15, then after
executing the above commands, A-(A intersection B) will be deleted.
Now on executing
{code}
delete jar ivy://org.apache.pig:pig:0.11.1.15
{code}
all the remaining dependencies will be deleted.
List resource: works as before.
Hive add jar command should be able to download and add jars from a
repository
Key: HIVE-9664
URL: https://issues.apache.org/jira/browse/HIVE-9664
Project: Hive
Issue Type: Improvement
Reporter: Anant Nag
Labels: hive
Currently Hive's add jar command takes a local path to the dependency jar.
This clutters the local file-system as users may forget to remove this jar
later
It would be nice if Hive supported a Gradle like notation to download the jar
from a repository.
Example: add jar org:module:version
It should also be backward compatible and should take jar from the local
file-system as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
