[jira] [Commented] (HIVE-4278) HCat needs to get current Hive jars instead of pulling them from maven repo
[ https://issues.apache.org/jira/browse/HIVE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635700#comment-13635700 ] Hudson commented on HIVE-4278: -- Integrated in Hive-trunk-hadoop2 #165 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/165/]) HIVE-4278 : HCat needs to get current Hive jars instead of pulling them from maven repo (Sushanth Sowmyan via Ashutosh Chauhan) (Revision 1469348) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1469348 Files : * /hive/trunk/beeline/ivy.xml * /hive/trunk/build-common.xml * /hive/trunk/build.properties * /hive/trunk/cli/ivy.xml * /hive/trunk/hcatalog/build-support/ant/deploy.xml * /hive/trunk/hcatalog/build.properties * /hive/trunk/hcatalog/core/pom.xml * /hive/trunk/hcatalog/hcatalog-pig-adapter/pom.xml * /hive/trunk/hcatalog/pom.xml * /hive/trunk/hcatalog/server-extensions/pom.xml * /hive/trunk/hcatalog/storage-handlers/hbase/pom.xml * /hive/trunk/hcatalog/webhcat/java-client/pom.xml * /hive/trunk/hcatalog/webhcat/svr/pom.xml * /hive/trunk/hwi/ivy.xml * /hive/trunk/ql/build.xml * /hive/trunk/ql/ivy.xml HCat needs to get current Hive jars instead of pulling them from maven repo --- Key: HIVE-4278 URL: https://issues.apache.org/jira/browse/HIVE-4278 Project: Hive Issue Type: Sub-task Components: Build Infrastructure, HCatalog Affects Versions: 0.11.0 Reporter: Alan Gates Assignee: Sushanth Sowmyan Priority: Blocker Fix For: 0.11.0 Attachments: HIVE-4278.approach2.patch, HIVE-4278.approach2.patch.2.for.branch.11, HIVE-4278.approach2.patch.2.for.branch.12, HIVE-4278.approach2.patch.3.for.branch.12, HIVE-4278.D10257.1.patch, HIVE-4278.D9981.1.patch The HCatalog build is currently pulling Hive jars from the maven repo instead of using the ones built as part of the current build. Now that it is part of Hive it should use the jars being built instead of pulling them from maven. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4278) HCat needs to get current Hive jars instead of pulling them from maven repo
[ https://issues.apache.org/jira/browse/HIVE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13636103#comment-13636103 ] Hudson commented on HIVE-4278: -- Integrated in Hive-trunk-h0.21 #2070 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2070/]) HIVE-4278 : HCat needs to get current Hive jars instead of pulling them from maven repo (Sushanth Sowmyan via Ashutosh Chauhan) (Revision 1469348) Result = ABORTED hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1469348 Files : * /hive/trunk/beeline/ivy.xml * /hive/trunk/build-common.xml * /hive/trunk/build.properties * /hive/trunk/cli/ivy.xml * /hive/trunk/hcatalog/build-support/ant/deploy.xml * /hive/trunk/hcatalog/build.properties * /hive/trunk/hcatalog/core/pom.xml * /hive/trunk/hcatalog/hcatalog-pig-adapter/pom.xml * /hive/trunk/hcatalog/pom.xml * /hive/trunk/hcatalog/server-extensions/pom.xml * /hive/trunk/hcatalog/storage-handlers/hbase/pom.xml * /hive/trunk/hcatalog/webhcat/java-client/pom.xml * /hive/trunk/hcatalog/webhcat/svr/pom.xml * /hive/trunk/hwi/ivy.xml * /hive/trunk/ql/build.xml * /hive/trunk/ql/ivy.xml HCat needs to get current Hive jars instead of pulling them from maven repo --- Key: HIVE-4278 URL: https://issues.apache.org/jira/browse/HIVE-4278 Project: Hive Issue Type: Sub-task Components: Build Infrastructure, HCatalog Affects Versions: 0.11.0 Reporter: Alan Gates Assignee: Sushanth Sowmyan Priority: Blocker Fix For: 0.11.0 Attachments: HIVE-4278.approach2.patch, HIVE-4278.approach2.patch.2.for.branch.11, HIVE-4278.approach2.patch.2.for.branch.12, HIVE-4278.approach2.patch.3.for.branch.12, HIVE-4278.D10257.1.patch, HIVE-4278.D9981.1.patch The HCatalog build is currently pulling Hive jars from the maven repo instead of using the ones built as part of the current build. Now that it is part of Hive it should use the jars being built instead of pulling them from maven. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4278) HCat needs to get current Hive jars instead of pulling them from maven repo
[ https://issues.apache.org/jira/browse/HIVE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13634090#comment-13634090 ] Ashutosh Chauhan commented on HIVE-4278: Above I found for trunk (.12) patch. 0.11 patch worked fine. HCat needs to get current Hive jars instead of pulling them from maven repo --- Key: HIVE-4278 URL: https://issues.apache.org/jira/browse/HIVE-4278 Project: Hive Issue Type: Sub-task Components: Build Infrastructure, HCatalog Affects Versions: 0.11.0 Reporter: Alan Gates Assignee: Sushanth Sowmyan Priority: Blocker Fix For: 0.11.0 Attachments: HIVE-4278.approach2.patch, HIVE-4278.approach2.patch.2.for.branch.11, HIVE-4278.approach2.patch.2.for.branch.12, HIVE-4278.D10257.1.patch, HIVE-4278.D9981.1.patch The HCatalog build is currently pulling Hive jars from the maven repo instead of using the ones built as part of the current build. Now that it is part of Hive it should use the jars being built instead of pulling them from maven. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4278) HCat needs to get current Hive jars instead of pulling them from maven repo
[ https://issues.apache.org/jira/browse/HIVE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13633261#comment-13633261 ] Alan Gates commented on HIVE-4278: -- +1 to doing this as a stop-gap to unblock 0.11 until we reach consensus on how to converge the build the tools between hcat and hive. HCat needs to get current Hive jars instead of pulling them from maven repo --- Key: HIVE-4278 URL: https://issues.apache.org/jira/browse/HIVE-4278 Project: Hive Issue Type: Sub-task Components: Build Infrastructure, HCatalog Affects Versions: 0.11.0 Reporter: Alan Gates Assignee: Travis Crawford Priority: Blocker Fix For: 0.11.0 Attachments: HIVE-4278.approach2.patch, HIVE-4278.D10257.1.patch, HIVE-4278.D9981.1.patch The HCatalog build is currently pulling Hive jars from the maven repo instead of using the ones built as part of the current build. Now that it is part of Hive it should use the jars being built instead of pulling them from maven. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4278) HCat needs to get current Hive jars instead of pulling them from maven repo
[ https://issues.apache.org/jira/browse/HIVE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13633323#comment-13633323 ] Phabricator commented on HIVE-4278: --- travis has accepted the revision HIVE-4278 [jira] HCat needs to get current Hive jars instead of pulling them from maven repo. +1 INLINE COMMENTS hcatalog/build-support/ant/deploy.xml:43 I like this approach - good idea. It duplicates publishing, but is a less invasive change. This provides a path forward until the dependency management converges. REVISION DETAIL https://reviews.facebook.net/D10257 BRANCH HIVE-4278 ARCANIST PROJECT hive To: JIRA, cwsteinbach, travis, ashutoshc, khorgath HCat needs to get current Hive jars instead of pulling them from maven repo --- Key: HIVE-4278 URL: https://issues.apache.org/jira/browse/HIVE-4278 Project: Hive Issue Type: Sub-task Components: Build Infrastructure, HCatalog Affects Versions: 0.11.0 Reporter: Alan Gates Assignee: Travis Crawford Priority: Blocker Fix For: 0.11.0 Attachments: HIVE-4278.approach2.patch, HIVE-4278.D10257.1.patch, HIVE-4278.D9981.1.patch The HCatalog build is currently pulling Hive jars from the maven repo instead of using the ones built as part of the current build. Now that it is part of Hive it should use the jars being built instead of pulling them from maven. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4278) HCat needs to get current Hive jars instead of pulling them from maven repo
[ https://issues.apache.org/jira/browse/HIVE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13633368#comment-13633368 ] Phabricator commented on HIVE-4278: --- ashutoshc has requested changes to the revision HIVE-4278 [jira] HCat needs to get current Hive jars instead of pulling them from maven repo. Patch is not applying cleanly. Need to rebase the patch. INLINE COMMENTS hcatalog/build-support/ant/deploy.xml:82 Will making ant .. call be better here? REVISION DETAIL https://reviews.facebook.net/D10257 BRANCH HIVE-4278 ARCANIST PROJECT hive To: JIRA, cwsteinbach, travis, ashutoshc, khorgath HCat needs to get current Hive jars instead of pulling them from maven repo --- Key: HIVE-4278 URL: https://issues.apache.org/jira/browse/HIVE-4278 Project: Hive Issue Type: Sub-task Components: Build Infrastructure, HCatalog Affects Versions: 0.11.0 Reporter: Alan Gates Assignee: Sushanth Sowmyan Priority: Blocker Fix For: 0.11.0 Attachments: HIVE-4278.approach2.patch, HIVE-4278.D10257.1.patch, HIVE-4278.D9981.1.patch The HCatalog build is currently pulling Hive jars from the maven repo instead of using the ones built as part of the current build. Now that it is part of Hive it should use the jars being built instead of pulling them from maven. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4278) HCat needs to get current Hive jars instead of pulling them from maven repo
[ https://issues.apache.org/jira/browse/HIVE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13633452#comment-13633452 ] Phabricator commented on HIVE-4278: --- khorgath has commented on the revision HIVE-4278 [jira] HCat needs to get current Hive jars instead of pulling them from maven repo. INLINE COMMENTS hcatalog/build-support/ant/deploy.xml:82 Agreed, we should be using the ant task, changing that and uploading rebased patch on the jira. REVISION DETAIL https://reviews.facebook.net/D10257 BRANCH HIVE-4278 ARCANIST PROJECT hive To: JIRA, cwsteinbach, travis, ashutoshc, khorgath HCat needs to get current Hive jars instead of pulling them from maven repo --- Key: HIVE-4278 URL: https://issues.apache.org/jira/browse/HIVE-4278 Project: Hive Issue Type: Sub-task Components: Build Infrastructure, HCatalog Affects Versions: 0.11.0 Reporter: Alan Gates Assignee: Sushanth Sowmyan Priority: Blocker Fix For: 0.11.0 Attachments: HIVE-4278.approach2.patch, HIVE-4278.D10257.1.patch, HIVE-4278.D9981.1.patch The HCatalog build is currently pulling Hive jars from the maven repo instead of using the ones built as part of the current build. Now that it is part of Hive it should use the jars being built instead of pulling them from maven. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4278) HCat needs to get current Hive jars instead of pulling them from maven repo
[ https://issues.apache.org/jira/browse/HIVE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13633736#comment-13633736 ] Ashutosh Chauhan commented on HIVE-4278: +1 will commit if tests pass. HCat needs to get current Hive jars instead of pulling them from maven repo --- Key: HIVE-4278 URL: https://issues.apache.org/jira/browse/HIVE-4278 Project: Hive Issue Type: Sub-task Components: Build Infrastructure, HCatalog Affects Versions: 0.11.0 Reporter: Alan Gates Assignee: Sushanth Sowmyan Priority: Blocker Fix For: 0.11.0 Attachments: HIVE-4278.approach2.patch, HIVE-4278.approach2.patch.2.for.branch.11, HIVE-4278.approach2.patch.2.for.branch.12, HIVE-4278.D10257.1.patch, HIVE-4278.D9981.1.patch The HCatalog build is currently pulling Hive jars from the maven repo instead of using the ones built as part of the current build. Now that it is part of Hive it should use the jars being built instead of pulling them from maven. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4278) HCat needs to get current Hive jars instead of pulling them from maven repo
[ https://issues.apache.org/jira/browse/HIVE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13632204#comment-13632204 ] Sushanth Sowmyan commented on HIVE-4278: [~cwsteinbach], [~traviscrawford] : Any thoughts on this approach in the time being? HCat needs to get current Hive jars instead of pulling them from maven repo --- Key: HIVE-4278 URL: https://issues.apache.org/jira/browse/HIVE-4278 Project: Hive Issue Type: Sub-task Components: Build Infrastructure, HCatalog Affects Versions: 0.11.0 Reporter: Alan Gates Assignee: Travis Crawford Priority: Blocker Fix For: 0.11.0 Attachments: HIVE-4278.approach2.patch, HIVE-4278.D9981.1.patch The HCatalog build is currently pulling Hive jars from the maven repo instead of using the ones built as part of the current build. Now that it is part of Hive it should use the jars being built instead of pulling them from maven. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4278) HCat needs to get current Hive jars instead of pulling them from maven repo
[ https://issues.apache.org/jira/browse/HIVE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13632213#comment-13632213 ] Carl Steinbach commented on HIVE-4278: -- [~sushanth]: I'm fine with this approach for the time being. HCat needs to get current Hive jars instead of pulling them from maven repo --- Key: HIVE-4278 URL: https://issues.apache.org/jira/browse/HIVE-4278 Project: Hive Issue Type: Sub-task Components: Build Infrastructure, HCatalog Affects Versions: 0.11.0 Reporter: Alan Gates Assignee: Travis Crawford Priority: Blocker Fix For: 0.11.0 Attachments: HIVE-4278.approach2.patch, HIVE-4278.D9981.1.patch The HCatalog build is currently pulling Hive jars from the maven repo instead of using the ones built as part of the current build. Now that it is part of Hive it should use the jars being built instead of pulling them from maven. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4278) HCat needs to get current Hive jars instead of pulling them from maven repo
[ https://issues.apache.org/jira/browse/HIVE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13632314#comment-13632314 ] Phabricator commented on HIVE-4278: --- khorgath has added reviewers to the revision HIVE-4278 [jira] HCat needs to get current Hive jars instead of pulling them from maven repo. Added Reviewers: cwsteinbach, travis, ashutoshc REVISION DETAIL https://reviews.facebook.net/D10257 To: JIRA, cwsteinbach, travis, ashutoshc, khorgath HCat needs to get current Hive jars instead of pulling them from maven repo --- Key: HIVE-4278 URL: https://issues.apache.org/jira/browse/HIVE-4278 Project: Hive Issue Type: Sub-task Components: Build Infrastructure, HCatalog Affects Versions: 0.11.0 Reporter: Alan Gates Assignee: Travis Crawford Priority: Blocker Fix For: 0.11.0 Attachments: HIVE-4278.approach2.patch, HIVE-4278.D10257.1.patch, HIVE-4278.D9981.1.patch The HCatalog build is currently pulling Hive jars from the maven repo instead of using the ones built as part of the current build. Now that it is part of Hive it should use the jars being built instead of pulling them from maven. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4278) HCat needs to get current Hive jars instead of pulling them from maven repo
[ https://issues.apache.org/jira/browse/HIVE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13623961#comment-13623961 ] Carl Steinbach commented on HIVE-4278: -- I'm planning to look at this patch more closely tonight. In the meantime I have a couple questions: bq. I looked into this a bit and recommend we publish intermediate artifacts to the local /.m2/repository during the Hive build (currently /.ivy/local) Why do this? We're still using Ivy, so shouldn't we use the default Ivy local directory or respect the value of ivy.default.ivy.user.dir if the user has set it? bq. Both Ivy and maven support a changing revision and this should be pretty straightforward. If both Ivy and Maven support this, and if we're currently using Ivy, why not stick with Ivy instead of introducing another build system into the mix? HCat needs to get current Hive jars instead of pulling them from maven repo --- Key: HIVE-4278 URL: https://issues.apache.org/jira/browse/HIVE-4278 Project: Hive Issue Type: Sub-task Components: Build Infrastructure, HCatalog Affects Versions: 0.11.0 Reporter: Alan Gates Assignee: Travis Crawford Priority: Blocker Fix For: 0.11.0 Attachments: HIVE-4278.D9981.1.patch The HCatalog build is currently pulling Hive jars from the maven repo instead of using the ones built as part of the current build. Now that it is part of Hive it should use the jars being built instead of pulling them from maven. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4278) HCat needs to get current Hive jars instead of pulling them from maven repo
[ https://issues.apache.org/jira/browse/HIVE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13623982#comment-13623982 ] Travis Crawford commented on HIVE-4278: --- Thanks for the high-level feedback. Quick summary of the Hive/HCatalog build. Both Hive HCatalog use ant as their build system. A difference is Hive uses Ivy to resolve dependencies, while HCatalog uses maven (through its maven-ant-tasks integration) to resolve dependencies. HCatalog does *not* use maven for its build - just resolving dependencies. With the summary in mind, the problem we're trying to solve here is during a Hive build, make the currently built jars available to HCatalog. Hive already has support for fetching artifacts from Maven repos, and with a relatively minor change we can publish Hive jars to the local maven repo, which makes the artifacts available to both dependency resolution tools. Overall I view this as a pretty minor change. That said, I believe our long-term goal is integrating HCatalog into Hive as a first-class citizen, which I assume means merging the build scripts using a single tool for dependency resolution. Once we determine ivy/maven-ant-tasks as the tool we want both project to use I think we should open a jira to track the transition. Additionally, I'd love to make some improvements to the Hive build, through the lens of simplification (I think there are some quick wins) and merging the hive/hcat ant targets. That's the run-down of what's going on here. This unblocks other HCat integration work, and once we have a build system integration plan we can do that as a separate issue. HCat needs to get current Hive jars instead of pulling them from maven repo --- Key: HIVE-4278 URL: https://issues.apache.org/jira/browse/HIVE-4278 Project: Hive Issue Type: Sub-task Components: Build Infrastructure, HCatalog Affects Versions: 0.11.0 Reporter: Alan Gates Assignee: Travis Crawford Priority: Blocker Fix For: 0.11.0 Attachments: HIVE-4278.D9981.1.patch The HCatalog build is currently pulling Hive jars from the maven repo instead of using the ones built as part of the current build. Now that it is part of Hive it should use the jars being built instead of pulling them from maven. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4278) HCat needs to get current Hive jars instead of pulling them from maven repo
[
https://issues.apache.org/jira/browse/HIVE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13624119#comment-13624119
]
Sushanth Sowmyan commented on HIVE-4278:
Travis, the patch you provided is working for me for a base ant run, and it
seems to be picking up the locally built hive artifacts, so that's a good first
step.
It still however is failing for me if I try to do something like this:
{noformat}
ant -Dversion=0.11.0-sush -Dhive.version=0.11.0-sush
{noformat}
The failures are all from it complaining about not finding
org.apache.hive#hive-builtins;0.11.0-sush!hive-builtins.jar
(This is what's been driving me crazy - I've been trying to exclude builtins
all over, trying to figure out which dependency is still bringing it in, but
that's been eluding me)
Another couple of notes:
a) In hcatalog/pom.xml :
Ideally, we don't want to specify a different version here if possible, thus
linking hive.version, hcatalog.version and version as specified as the project
version to the root hive build version.
b) ant maven-build currently fails because make-pom is not defined on hcatalog,
and hcatalog is in iterate.hive.all, which is specified for make-pom in the
maven-build target definition. We should also potentially check other targets
using the iterate variables to see that hcat's inclusion doesn't break them.
I'll update if I find any more.
(Note: I'm doing a ant clean ant very-clean rm -rf ~/.m2 ~/.ivy2 between
each run)
HCat needs to get current Hive jars instead of pulling them from maven repo
---
Key: HIVE-4278
URL: https://issues.apache.org/jira/browse/HIVE-4278
Project: Hive
Issue Type: Sub-task
Components: Build Infrastructure, HCatalog
Affects Versions: 0.11.0
Reporter: Alan Gates
Assignee: Travis Crawford
Priority: Blocker
Fix For: 0.11.0
Attachments: HIVE-4278.D9981.1.patch
The HCatalog build is currently pulling Hive jars from the maven repo instead
of using the ones built as part of the current build. Now that it is part of
Hive it should use the jars being built instead of pulling them from maven.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4278) HCat needs to get current Hive jars instead of pulling them from maven repo
[ https://issues.apache.org/jira/browse/HIVE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13624153#comment-13624153 ] Carl Steinbach commented on HIVE-4278: -- bq. Hive already has support for fetching artifacts from Maven repos, and with a relatively minor change we can publish Hive jars to the local maven repo, which makes the artifacts available to both dependency resolution tools. This sounds like a good change to make, but I'm having a hard time understanding why we would want to use maven-ant-tasks to do this when Ivy already provides the same functionality. bq. Once we determine ivy/maven-ant-tasks as the tool we want both project to use I think we should open a jira to track the transition. What are the advantages of using maven-ant-tasks in place of Ivy? HCat needs to get current Hive jars instead of pulling them from maven repo --- Key: HIVE-4278 URL: https://issues.apache.org/jira/browse/HIVE-4278 Project: Hive Issue Type: Sub-task Components: Build Infrastructure, HCatalog Affects Versions: 0.11.0 Reporter: Alan Gates Assignee: Travis Crawford Priority: Blocker Fix For: 0.11.0 Attachments: HIVE-4278.D9981.1.patch The HCatalog build is currently pulling Hive jars from the maven repo instead of using the ones built as part of the current build. Now that it is part of Hive it should use the jars being built instead of pulling them from maven. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4278) HCat needs to get current Hive jars instead of pulling them from maven repo
[ https://issues.apache.org/jira/browse/HIVE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13624224#comment-13624224 ] Travis Crawford commented on HIVE-4278: --- Hey [~cwsteinbach] - Hive actually already uses maven-ant-tasks for publishing to maven repos, which is why we use it for publishing to the local repo. Looking at the [maven-publish-artifact|http://svn.apache.org/repos/asf/hive/trunk/build.xml] task we see the maven-ant-tasks used, with options to publish local or a remote repo (this section could actually be simplified a bit - see [hcatalog/build-support/ant/deploy.xml|http://svn.apache.org/repos/asf/hive/trunk/hcatalog/build-support/ant/deploy.xml]). I'm happy to use Ivy to use [ivy:publish|http://ant.apache.org/ivy/history/2.2.0/use/publish.html] when installing into the local maven cache, for since we already need maven-ant-tasks for deploying remotely and they work well, I decided to use them when installing locally too. When interacting with maven repos I think the maven-ant-tasks are the way to go, even if Ivy is used for dependency resolution. I'll file a followup jira about being consistent with which tool we use to manage dependencies, but I think that's separate from this issue. HCat needs to get current Hive jars instead of pulling them from maven repo --- Key: HIVE-4278 URL: https://issues.apache.org/jira/browse/HIVE-4278 Project: Hive Issue Type: Sub-task Components: Build Infrastructure, HCatalog Affects Versions: 0.11.0 Reporter: Alan Gates Assignee: Travis Crawford Priority: Blocker Fix For: 0.11.0 Attachments: HIVE-4278.D9981.1.patch The HCatalog build is currently pulling Hive jars from the maven repo instead of using the ones built as part of the current build. Now that it is part of Hive it should use the jars being built instead of pulling them from maven. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4278) HCat needs to get current Hive jars instead of pulling them from maven repo
[ https://issues.apache.org/jira/browse/HIVE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13624245#comment-13624245 ] Carl Steinbach commented on HIVE-4278: -- bq. When interacting with maven repos I think the maven-ant-tasks are the way to go, even if Ivy is used for dependency resolution. Why? What makes the maven-ant-tasks a better option for interacting with Maven repositories? bq. I'll file a followup jira about being consistent with which tool we use to manage dependencies, but I think that's separate from this issue. I think in the meantime we should avoid introducing more inconsistencies. Hive uses Ivy for 90% of it's dependency resolution needs, and I suspect that the other 10% where we use maven-ant-tasks were introduced by a developer who was more familiar with Maven than Ivy. Please use Ivy instead of maven-ant-tasks. HCat needs to get current Hive jars instead of pulling them from maven repo --- Key: HIVE-4278 URL: https://issues.apache.org/jira/browse/HIVE-4278 Project: Hive Issue Type: Sub-task Components: Build Infrastructure, HCatalog Affects Versions: 0.11.0 Reporter: Alan Gates Assignee: Travis Crawford Priority: Blocker Fix For: 0.11.0 Attachments: HIVE-4278.D9981.1.patch The HCatalog build is currently pulling Hive jars from the maven repo instead of using the ones built as part of the current build. Now that it is part of Hive it should use the jars being built instead of pulling them from maven. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4278) HCat needs to get current Hive jars instead of pulling them from maven repo
[ https://issues.apache.org/jira/browse/HIVE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13624254#comment-13624254 ] Travis Crawford commented on HIVE-4278: --- Re: why maven-ant-tasks are used at all, I could be mistaken but I don't think you can publish to a remote maven repo with ivy. This is likely why Hive already uses maven-ant-tasks. The reason I prefer maven-ant-tasks to ivy when interacting with local maven repos is because you don't need to special-case things. Before redoing the patch, I'd like to reach consensus on what approach to take. Am I hearing we should completely migrate HCatalog to ivy for its dependencies now rather than making a small change to where Hive publishes its internal dependencies? HCat needs to get current Hive jars instead of pulling them from maven repo --- Key: HIVE-4278 URL: https://issues.apache.org/jira/browse/HIVE-4278 Project: Hive Issue Type: Sub-task Components: Build Infrastructure, HCatalog Affects Versions: 0.11.0 Reporter: Alan Gates Assignee: Travis Crawford Priority: Blocker Fix For: 0.11.0 Attachments: HIVE-4278.D9981.1.patch The HCatalog build is currently pulling Hive jars from the maven repo instead of using the ones built as part of the current build. Now that it is part of Hive it should use the jars being built instead of pulling them from maven. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4278) HCat needs to get current Hive jars instead of pulling them from maven repo
[ https://issues.apache.org/jira/browse/HIVE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13624270#comment-13624270 ] Carl Steinbach commented on HIVE-4278: -- The ivy:publish task allows you publish to either a local or remote repository. Here's a page on the Apache website that describes how to publish release artifacts to Apache's Nexus Pro repository using Ant+Ivy: http://www.apache.org/dev/publishing-maven-artifacts.html#ant bq. The reason I prefer maven-ant-tasks to ivy when interacting with local maven repos is because you don't need to special-case things. What do you need to special-case? bq. Am I hearing we should completely migrate HCatalog to ivy for its dependencies now rather than making a small change to where Hive publishes its internal dependencies? The small change that's been proposed is a hack that won't make any sense to someone who encounters it six months from now. Hive uses Ivy, so why should it publish artifacts to the local Maven repository? HCat needs to get current Hive jars instead of pulling them from maven repo --- Key: HIVE-4278 URL: https://issues.apache.org/jira/browse/HIVE-4278 Project: Hive Issue Type: Sub-task Components: Build Infrastructure, HCatalog Affects Versions: 0.11.0 Reporter: Alan Gates Assignee: Travis Crawford Priority: Blocker Fix For: 0.11.0 Attachments: HIVE-4278.D9981.1.patch The HCatalog build is currently pulling Hive jars from the maven repo instead of using the ones built as part of the current build. Now that it is part of Hive it should use the jars being built instead of pulling them from maven. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4278) HCat needs to get current Hive jars instead of pulling them from maven repo
[
https://issues.apache.org/jira/browse/HIVE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622726#comment-13622726
]
Travis Crawford commented on HIVE-4278:
---
Status update:
I've made good progress, but am running into an issue with {{builtins}}. In
[builtins/build.xml|http://svn.apache.org/repos/asf/hive/trunk/builtins/build.xml]
building/installing the jar is deferred to the packaging phase, which means
HCatalog cannot depend on this artifact during its build.
Looking at the {{builtins}} jar in more details, there's very little in there.
Can someone clue me into why we have such a small jar, and why we need to defer
jar'ing until later in the build? Looking through the build files I'm not
understanding the reason for this – some context would be appreciated.
{code}
hoth:~ travis$ jar -tvf ~/Downloads/hive-builtins-0.10.0.jar
0 Tue Dec 18 16:38:50 PST 2012 META-INF/
242 Tue Dec 18 16:38:48 PST 2012 META-INF/MANIFEST.MF
0 Tue Dec 18 15:20:02 PST 2012 org/
0 Tue Dec 18 15:20:02 PST 2012 org/apache/
0 Tue Dec 18 15:20:02 PST 2012 org/apache/hive/
0 Tue Dec 18 15:20:02 PST 2012 org/apache/hive/builtins/
152 Tue Dec 18 15:20:02 PST 2012 org/apache/hive/builtins/BuiltinUtils.class
2734 Tue Dec 18 15:20:02 PST 2012
org/apache/hive/builtins/UDAFUnionMap$Evaluator.class
581 Tue Dec 18 15:20:02 PST 2012
org/apache/hive/builtins/UDAFUnionMap$State.class
1077 Tue Dec 18 15:20:02 PST 2012 org/apache/hive/builtins/UDAFUnionMap.class
108 Tue Dec 18 16:38:48 PST 2012 META-INF/class-info.xml
80 Tue Dec 18 16:38:50 PST 2012 META-INF/class-registration.sql
hoth:~ travis$
{code}
HCat needs to get current Hive jars instead of pulling them from maven repo
---
Key: HIVE-4278
URL: https://issues.apache.org/jira/browse/HIVE-4278
Project: Hive
Issue Type: Sub-task
Components: Build Infrastructure, HCatalog
Affects Versions: 0.11.0
Reporter: Alan Gates
Priority: Blocker
Fix For: 0.11.0
The HCatalog build is currently pulling Hive jars from the maven repo instead
of using the ones built as part of the current build. Now that it is part of
Hive it should use the jars being built instead of pulling them from maven.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4278) HCat needs to get current Hive jars instead of pulling them from maven repo
[ https://issues.apache.org/jira/browse/HIVE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622822#comment-13622822 ] Sushanth Sowmyan commented on HIVE-4278: Travis, I was experimenting too, and I don't see why we need to depend on builtins from hcatalog. I think our dependency is from back when we used to package hive along with hcatalog, and can be removed. That way, it won't matter to hcatalog when the builtins jar is built. HCat needs to get current Hive jars instead of pulling them from maven repo --- Key: HIVE-4278 URL: https://issues.apache.org/jira/browse/HIVE-4278 Project: Hive Issue Type: Sub-task Components: Build Infrastructure, HCatalog Affects Versions: 0.11.0 Reporter: Alan Gates Priority: Blocker Fix For: 0.11.0 The HCatalog build is currently pulling Hive jars from the maven repo instead of using the ones built as part of the current build. Now that it is part of Hive it should use the jars being built instead of pulling them from maven. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4278) HCat needs to get current Hive jars instead of pulling them from maven repo
[ https://issues.apache.org/jira/browse/HIVE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622827#comment-13622827 ] Sushanth Sowmyan commented on HIVE-4278: I mean, we have HCATALOG-187, but even there, the error seems to be a runtime error for the hcat commandline, from a org.apache.hadoop.hive.ql.session.SessionState.init, not a compile/package-time problem. HCat needs to get current Hive jars instead of pulling them from maven repo --- Key: HIVE-4278 URL: https://issues.apache.org/jira/browse/HIVE-4278 Project: Hive Issue Type: Sub-task Components: Build Infrastructure, HCatalog Affects Versions: 0.11.0 Reporter: Alan Gates Priority: Blocker Fix For: 0.11.0 The HCatalog build is currently pulling Hive jars from the maven repo instead of using the ones built as part of the current build. Now that it is part of Hive it should use the jars being built instead of pulling them from maven. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4278) HCat needs to get current Hive jars instead of pulling them from maven repo
[
https://issues.apache.org/jira/browse/HIVE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622832#comment-13622832
]
Travis Crawford commented on HIVE-4278:
---
Hey [~sushanth] - the issue is transitive dependencies.
[hcatalog-core-0.5.0-incubating.pom|http://search.maven.org/remotecontent?filepath=org/apache/hcatalog/hcatalog-core/0.5.0-incubating/hcatalog-core-0.5.0-incubating.pom]
depends on
[hive-exec-0.10.0.pom|http://search.maven.org/remotecontent?filepath=org/apache/hive/hive-exec/0.10.0/hive-exec-0.10.0.pom],
which depends on
[hive-builtins-0.10.0.pom|http://search.maven.org/remotecontent?filepath=org/apache/hive/hive-builtins/0.10.0/hive-builtins-0.10.0.pom]
We could exclude {{hive-builtins}} from {{hive-exec}} - it would be a runtime
issue for hive, but I don't think HCatalog would mind.
HCat needs to get current Hive jars instead of pulling them from maven repo
---
Key: HIVE-4278
URL: https://issues.apache.org/jira/browse/HIVE-4278
Project: Hive
Issue Type: Sub-task
Components: Build Infrastructure, HCatalog
Affects Versions: 0.11.0
Reporter: Alan Gates
Priority: Blocker
Fix For: 0.11.0
The HCatalog build is currently pulling Hive jars from the maven repo instead
of using the ones built as part of the current build. Now that it is part of
Hive it should use the jars being built instead of pulling them from maven.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4278) HCat needs to get current Hive jars instead of pulling them from maven repo
[ https://issues.apache.org/jira/browse/HIVE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622836#comment-13622836 ] Travis Crawford commented on HIVE-4278: --- Just seeing your previous comment; I'll try excluding and see if that helps. Awesome you're taking a look at this too! My branch is https://github.com/traviscrawford/hive/tree/HIVE-4278_publish_local if you want to coolaborate on the working branch. HCat needs to get current Hive jars instead of pulling them from maven repo --- Key: HIVE-4278 URL: https://issues.apache.org/jira/browse/HIVE-4278 Project: Hive Issue Type: Sub-task Components: Build Infrastructure, HCatalog Affects Versions: 0.11.0 Reporter: Alan Gates Priority: Blocker Fix For: 0.11.0 The HCatalog build is currently pulling Hive jars from the maven repo instead of using the ones built as part of the current build. Now that it is part of Hive it should use the jars being built instead of pulling them from maven. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4278) HCat needs to get current Hive jars instead of pulling them from maven repo
[ https://issues.apache.org/jira/browse/HIVE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622843#comment-13622843 ] Sushanth Sowmyan commented on HIVE-4278: Cool, sounds good, checking out your branch. HCat needs to get current Hive jars instead of pulling them from maven repo --- Key: HIVE-4278 URL: https://issues.apache.org/jira/browse/HIVE-4278 Project: Hive Issue Type: Sub-task Components: Build Infrastructure, HCatalog Affects Versions: 0.11.0 Reporter: Alan Gates Priority: Blocker Fix For: 0.11.0 The HCatalog build is currently pulling Hive jars from the maven repo instead of using the ones built as part of the current build. Now that it is part of Hive it should use the jars being built instead of pulling them from maven. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4278) HCat needs to get current Hive jars instead of pulling them from maven repo
[
https://issues.apache.org/jira/browse/HIVE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621524#comment-13621524
]
Travis Crawford commented on HIVE-4278:
---
I looked into this a bit and recommend we publish intermediate artifacts to the
local {{~/.m2/repository}} during the Hive build (currently {{~/.ivy/local}}),
and configure all submodules to use the local maven repo first when resolving
artifacts. Both Ivy and maven support a changing revision and this should be
pretty straightforward.
Details about Ivy support of changing revisions:
http://ant.apache.org/ivy/history/latest-milestone/concept.html#change
Long-term, I think we should consider using a single resolver for all
dependencies. When incubating, I migrated HCatalog from ivy to the maven ant
tasks and have been very happy with them. If folks are comfortable with this
I'd be interested in migrating Hive entirely to the maven ant tasks. However,
if folks want to stick with Ivy I think we should switch HCatalog back to Ivy
too for consistency.
HCat needs to get current Hive jars instead of pulling them from maven repo
---
Key: HIVE-4278
URL: https://issues.apache.org/jira/browse/HIVE-4278
Project: Hive
Issue Type: Sub-task
Components: Build Infrastructure, HCatalog
Affects Versions: 0.11.0
Reporter: Alan Gates
Priority: Blocker
Fix For: 0.11.0
The HCatalog build is currently pulling Hive jars from the maven repo instead
of using the ones built as part of the current build. Now that it is part of
Hive it should use the jars being built instead of pulling them from maven.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4278) HCat needs to get current Hive jars instead of pulling them from maven repo
[
https://issues.apache.org/jira/browse/HIVE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621544#comment-13621544
]
Travis Crawford commented on HIVE-4278:
---
Here are more details about the proposal - this is excerpted from the HCatalog
build.
In
[hcatalog/build-support/ant/build-common.xml|http://svn.apache.org/repos/asf/hive/trunk/hcatalog/build-support/ant/build-common.xml]
we see how HCatalog creates a jar. In addition to creating the jar we
[install:http://maven.apache.org/ant-tasks/examples/install-deploy.html] into
the local maven repo.
{code}
target name=jar depends=mvn-dependencies,compile,compile-resource
echo message=${ant.project.name}/
jar jarfile=${build.dir}/${jar.name} basedir=${build.classes}
metainf dir=${path.to.basedir}
includes=**/LICENSE.txt,**/NOTICE.txt/
/jar
artifact:install
file=${build.dir}/${ant.project.name}-${hcatalog.version}.jar
artifact:pom file=${pom.file}/
artifact:localRepository path=${mvn.local.repo}/
/artifact:install
/target
{code}
In Hive's existing
[build-common.xml|http://svn.apache.org/repos/asf/hive/trunk/build-common.xml]
we see how things currently work. Notice how we publish to the local Ivy repo.
{code}
target name=jar depends=compile
echo message=Project: ${ant.project.name} /
jar
jarfile=${build.dir}/hive-${ant.project.name}-${version}.jar
basedir=${build.classes}
manifest
!-- Not putting these in their own manifest section, since that inserts
a new-line, which breaks the reading of the attributes. --
attribute name=Implementation-Title value=Hive/
attribute name=Implementation-Version value=${version}/
attribute name=Implementation-Vendor value=Apache/
/manifest
metainf dir=${hive.root} includes=LICENSE,NOTICE/
/jar
ivy:publish settingsRef=${ant.project.name}.ivy.settings
resolver=local pubrevision=${version} overwrite=true
artifactspattern=${build.dir}/${ivy.publish.pattern}/
/target
{code}
We see they're quite similar, and we should definitely look to merge the build
files so its maintainable and feels less bolted on over time. This would be a
pretty targeted change, simply changing where we publish SNAPSHOT jars, and a
resolver change to always check the local maven repo for changing artifacts.
Its worth noting the Hive build already uses maven ant tasks when publishing
artifacts to a remote maven repo. See {{maven-publish-artifact}} in
[build.xml|http://svn.apache.org/repos/asf/hive/trunk/build.xml] for details.
There would likely be some build file refactoring, but overall not too major a
change.
HCat needs to get current Hive jars instead of pulling them from maven repo
---
Key: HIVE-4278
URL: https://issues.apache.org/jira/browse/HIVE-4278
Project: Hive
Issue Type: Sub-task
Components: Build Infrastructure, HCatalog
Affects Versions: 0.11.0
Reporter: Alan Gates
Priority: Blocker
Fix For: 0.11.0
The HCatalog build is currently pulling Hive jars from the maven repo instead
of using the ones built as part of the current build. Now that it is part of
Hive it should use the jars being built instead of pulling them from maven.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
