[jira] [Commented] (HIVE-4305) Use a single system for dependency resolution
[ https://issues.apache.org/jira/browse/HIVE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13670568#comment-13670568 ] Carl Steinbach commented on HIVE-4305: -- [~alangates]: last I heard you were planning to arrange a meeting between you, me, and Travis to go over the details of integrating HCatalog into Hive's existing build infrastructure. Is this still going to happen? I also agree that getting this resolved is a priority -- the amount of time required to build Hive doubled with the addition of HCatalog. Use a single system for dependency resolution - Key: HIVE-4305 URL: https://issues.apache.org/jira/browse/HIVE-4305 Project: Hive Issue Type: Improvement Components: Build Infrastructure, HCatalog Reporter: Travis Crawford Assignee: Carl Steinbach Attachments: HIVE-4305.1.wip.patch Both Hive and HCatalog use ant as their build tool. However, Hive uses ivy for dependency resolution while HCatalog uses maven-ant-tasks. With the project merge we should converge on a single tool for dependency resolution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4305) Use a single system for dependency resolution
[ https://issues.apache.org/jira/browse/HIVE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13669942#comment-13669942 ] Alan Gates commented on HIVE-4305: -- All, We need to make some forward progress on this. Not knowing what we're doing next with the build system is delaying other JIRAs such as fixing the nightly build (the equivalent of HIVE-4387 which needs to be done on trunk) and HIVE-4460. What do we need to do to decide here? I have no opinion in the matter, but I'd prefer not to dismantle all of Travis' hard work in the HCat build files without his agreement. Use a single system for dependency resolution - Key: HIVE-4305 URL: https://issues.apache.org/jira/browse/HIVE-4305 Project: Hive Issue Type: Improvement Components: Build Infrastructure, HCatalog Reporter: Travis Crawford Assignee: Carl Steinbach Attachments: HIVE-4305.1.wip.patch Both Hive and HCatalog use ant as their build tool. However, Hive uses ivy for dependency resolution while HCatalog uses maven-ant-tasks. With the project merge we should converge on a single tool for dependency resolution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4305) Use a single system for dependency resolution
[
https://issues.apache.org/jira/browse/HIVE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13641516#comment-13641516
]
Carl Steinbach commented on HIVE-4305:
--
{quote}
Both ivy and maven-ant-tasks are able to both publish to a local maven cache
and use the same codepath to publish to a maven repo as well. Ignoring
maven-ant-tasks for now, is there a need to have ivy publish it to ivy-cache
for the build, and have a separate task to publish to a maven repo? Couldn't we
streamline this to have ivy just publish and pull from the local maven cache?
{quote}
We use a separate task to publish to a remote maven repository because this
operation requires the user to supply credentials. I also think it makes sense
for the jar target to publish to the local ivy repository by default since
this allows us to leverage Ivy configuration defaults, and as a result I think
it's probably less fragile than the alternative option. However, we can also
supply an alternative ivysettings file that points to the local Maven
repository, thus giving users the option of using this repository if they want
to.
{quote}
Here, the properties seem to be read and parsed after the version is set, so
it's usable inside this pom.xml. However, for the child pom.xml files, inside
hcatalog-pig-adaptor, for example, we have to refer to the parent pom.xml, but
at the time we encounter this pom.xml, we either need to specify its version,
and then say that its parent is the same version, or we have to skip specifying
its version, and specify the parent's version before it can load the parent
pom.xml. What this means is that either way, I wind up explicitly having a line
in there with 0.12.0-SNAPSHOT.
{quote}
Ivy allows you to define properties in the ivysettings file, and also gives you
option of sourcing other properties files. Properties which are defined in the
ivysettings file can be referenced in any ivy.xml file. In a minute I will
attach a patch which demonstrates this in action: Hive's ivysettings file
sources build.properties and libraries.properties, and the ivy.xml files in
turn references ${version} instead of using a hardcoded version value.
Use a single system for dependency resolution
-
Key: HIVE-4305
URL: https://issues.apache.org/jira/browse/HIVE-4305
Project: Hive
Issue Type: Improvement
Components: Build Infrastructure, HCatalog
Reporter: Travis Crawford
Assignee: Carl Steinbach
Both Hive and HCatalog use ant as their build tool. However, Hive uses ivy
for dependency resolution while HCatalog uses maven-ant-tasks. With the
project merge we should converge on a single tool for dependency resolution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4305) Use a single system for dependency resolution
[ https://issues.apache.org/jira/browse/HIVE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13641531#comment-13641531 ] Carl Steinbach commented on HIVE-4305: -- Changes in HIVE-4305.1.wip.patch: * 'ant' with no arguments prints out available targets. * Consolidate third-party dependencies in parent ivy.xml file. * source build.properties and libraries.properties in ivysettings.xml * Fix offline mode. * Determine submodule build order using ivy:buildlist task. Use a single system for dependency resolution - Key: HIVE-4305 URL: https://issues.apache.org/jira/browse/HIVE-4305 Project: Hive Issue Type: Improvement Components: Build Infrastructure, HCatalog Reporter: Travis Crawford Assignee: Carl Steinbach Attachments: HIVE-4305.1.wip.patch Both Hive and HCatalog use ant as their build tool. However, Hive uses ivy for dependency resolution while HCatalog uses maven-ant-tasks. With the project merge we should converge on a single tool for dependency resolution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4305) Use a single system for dependency resolution
[ https://issues.apache.org/jira/browse/HIVE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13641539#comment-13641539 ] Carl Steinbach commented on HIVE-4305: -- Someone claimed earlier that IDE support is better for Maven. I don't know if that's true or not, but I wanted to point out that there's a list of IDE plugins for Ivy on the Ivy website: http://ant.apache.org/ivy/links.html The list includes three Intellij plugins. Use a single system for dependency resolution - Key: HIVE-4305 URL: https://issues.apache.org/jira/browse/HIVE-4305 Project: Hive Issue Type: Improvement Components: Build Infrastructure, HCatalog Reporter: Travis Crawford Assignee: Carl Steinbach Attachments: HIVE-4305.1.wip.patch Both Hive and HCatalog use ant as their build tool. However, Hive uses ivy for dependency resolution while HCatalog uses maven-ant-tasks. With the project merge we should converge on a single tool for dependency resolution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4305) Use a single system for dependency resolution
[
https://issues.apache.org/jira/browse/HIVE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639706#comment-13639706
]
Sushanth Sowmyan commented on HIVE-4305:
A lot of comparisons on this thread have been comparing a pure maven approach
with an ant-based approach, and to be honest, there are good sides and bad for
both. A pure maven project that is done well is simpler for developers and
better/simpler as a build system. And it is worth it for most development
projects to try to spend the time trying to fix their build systems. And yes,
hive's build is a complex enough beast to want to simplify it. But that is a
huge undertaking, with no promise that it'll be a successful transition -
projects that start as Maven projects have an easier time getting there than
projects that don't and then undergo mavenization.
Ant+Ivy, for the most part, works currently. If someone does care about
mavenization enough to work on a patch and contribute it, we can compare the
approaches. Without that, we're arguing our individual bad experiences with
spaghetti ant builds and inflexible maven builds, and feeding those experiences
into unproductive vitriol(which, btw, also chases away other people who want to
contribute to the discussion). Let's try to cool down a bit and look at
implementable changes for now.
On the question of standardizing between ant or maven as the primary build
system, I'm going to suggest we go with ant for now.
--
Given ant for building, there are still multiple build combinations potentially
at play, and it is those that I hoped we could discuss in this thread :
Ant for building, in conjucntion with:
1) ivy for publishing to local ivy cache
2) ivy for publishing to local maven cache
3) maven-ant-tasks for publishing to local maven cache
using
a) ivy for dependency resolution retrieving
b) maven-ant-tasks for dependency resolution retrieving
I) The publishing scenario:
Among the systems I've described above, HCatalog was using 3-b Hive was using
1-a along with a bit of 3 as a separate bit for publishing to repositories.
When we were building hcatalog outside of hive and dependent on it, we always
had to build hive, then do a maven publish, and then use it from hcat. When
hcat was merged in with hive, this became a problem because hcat's build was
integrated into the middle of hive's build, before we got an opportunity to
publish to the local maven cache, which we have now temporarily patched in a
hacky manner.
In my experience, ivy is more flexible than maven-ant-tasks in terms of
dependency resolution (I'll get to that in the second section), so Ivy can
fetch from a maven-published cache/repo, but maven-ant-tasks has issues
fetching from an ivy cache. In terms of how third-party projects can consume
hive and/or hcatalog, publishing to a maven repo is the way to go to be
permissible and flexible.
Both ivy and maven-ant-tasks are able to both publish to a local maven cache
and use the same codepath to publish to a maven repo as well. Ignoring
maven-ant-tasks for now, is there a need to have ivy publish it to ivy-cache
for the build, and have a separate task to publish to a maven repo? Couldn't we
streamline this to have ivy just publish and pull from the local maven cache?
This is not an invasive change to hive, and it makes it easier for other
projects to depend on and work with hive, and it streamlines hive's build as
well, by not making it be a special case to publish maven artifacts at publish
time. Is there a good technical reason to avoid this?
I'm okay with using ivy to publish and thus streamline, but I would prefer to
publish and retrieve from local maven-cache in doing so.
--
II) The dependency resolution scenario.
The two tools at hand here are maven-ant-tasks and ivy. At HCatalog, we used to
use ivy, and then we moved to maven-ant-tasks in an attempt to eventually
mavenize, and thus use a single pom.xml which would be a transition point to
eventual mavenization, but we hadn't got there yet at the time we merged with
hive. At this point in time, I'm leaning towards using ivy, and changing
HCatalog back to using ivy.
The real problems that we've faced with maven-ant-tasks, however, is with
transitive dependencies and variable definitions. I might simply not know how
to resolve this, so if you can tell me how to resolve these issues, we might be
able to fix these.
Problem#1 : variable definitions. Currently, hcatalog has a primary pom.xml,
with all its subcomponents defining that pom.xml as their parent. Now, the
problem is that they have to explicitly mention which version of the parent is
their specific parent. So:
In our primary pom.xml, we have:
{noformat}
groupIdorg.apache.hcatalog/groupId
artifactIdhcatalog/artifactId
version0.12.0-SNAPSHOT/version
properties
...
[jira] [Commented] (HIVE-4305) Use a single system for dependency resolution
[ https://issues.apache.org/jira/browse/HIVE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637633#comment-13637633 ] Carl Steinbach commented on HIVE-4305: -- bq. It contains native executables. So does Hive. Please look around. bq. It contains native libraries. So does Hive. Please look around. bq. It contains jni libraries. I'm not sure what makes this different from the previous two cases, but will admit that you've got me there. bq. Moving to Maven would be making it better in the opinion of the majority of the development community. So you have already conducted a poll? How large was your sample size and what are the margins of error? Can you please publish the questions that were used? Thanks. bq. Certainly, I believe it is possible to make things worse with Maven. How do we know that isn't going to happen here? bq. I'm not a fan of how the Hadoop mavenization was done... In your opinion what did they do wrong? It's starting to sound like there's more than a little room for error when transitioning a project over to Maven. Maybe the Maven community would benefit from having a best practices document that highlighted these potential pitfalls. bq. ... and I deeply regret not taking the time to make it better as it went in. It's not too late to fix it, right? You're a committer on that project, which along with your extensive Maven knowledge puts you in an excellent position to lead this effort. bq. ... but it was still better than the ant + ivy + maven ant tasks that we had ... I think part of the problem with the original build was that you were using three components (ant, ivy, maven ant tasks), when you only needed to use two (ant, ivy). Our build, especially post-HCatalog merge, suffers from this same problem. This ticket was originally filed to track the task of removing either Ivy or maven ant asks, but was quickly hijacked by people intent on replacing everything with Maven. Owen, would it be OK with you if we return to the original focus of this ticket and resolve it before again considering your proposal? This will also give you time to strengthen your argument by fixing the bugs and design defects in Hadoop's Maven build. bq. If it hadn't been, it would have been rejected. I hope you're joking. bq. That said, in my experience most projects are better off with Maven builds than ant + ivy + maven ant tasks. What characteristics define a project that falls into the latter category, or did you mean to say *all* instead of *most*? Use a single system for dependency resolution - Key: HIVE-4305 URL: https://issues.apache.org/jira/browse/HIVE-4305 Project: Hive Issue Type: Improvement Components: Build Infrastructure, HCatalog Reporter: Travis Crawford Assignee: Carl Steinbach Both Hive and HCatalog use ant as their build tool. However, Hive uses ivy for dependency resolution while HCatalog uses maven-ant-tasks. With the project merge we should converge on a single tool for dependency resolution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4305) Use a single system for dependency resolution
[ https://issues.apache.org/jira/browse/HIVE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637159#comment-13637159 ] Carl Steinbach commented on HIVE-4305: -- bq. or compare it to Hadoop, which has a LOT more complexity in their build Owen, please give some concrete examples of things that make Hadoop's build more complex than Hive's. Also, assuming for a moment that this claim is valid, do you think this added level of complexity excuses the fact that even the most basic things are broken? What benefits does the project get from using Maven if simple things like 'mvn compile' and 'mvn eclipse:eclipse' don't work? And more to the point, are there any situations where you think that Maven is not the right tool for the job, and if so, what are they? I think I would find this proposal a lot more credible if someone demonstrated that it's possible to fix these issues. Here are the tickets in case anyone is interested in taking up that challenge: https://issues.apache.org/jira/browse/HADOOP-9383 https://issues.apache.org/jira/browse/HADOOP-9489 I am particularly interested in seeing whether or not someone can fix a problem like HADOOP-9383 without resorting to git bisect or related methods. bq. I propose that we have Travis make a Maven build file for the combined Hive and HCat systems. Then we can debate the value and issues in the particular patch and how to move the project forward. Travis has every right to do spend time doing this if he wants to, and I'm eager to see the results, but I also hope we all agree this doesn't mean we have decided to switch to Maven. This patch will be judged using the same criteria that we apply to every other patch. If it's a net positive for the project it goes in, otherwise it stays out. Personally I think it would be more pragmatic to spend time improving the build that we currently have, but this is a volunteer effort where everyone makes their own decisions about how they spend their time. bq. The current state is painful with extremely long builds. This is a qualitative description of a quantitave value. Why be vague? How long do you think the build should take to complete. What do you think is reasonable. Please give me a number, and explain what that includes (i.e. jars only, jars + javadoc, jars + javadoc + tarballs, etc). bq. We need to move forward and enable the project to evolve quickly so that Hive can compete with its many comercial competitors. I have yet to encounter a single Hive user who thinks switching to Maven is critical for the success of the project. I also think that general code quality issues have a much bigger impact on developer productivity than anything related to the build system, but that's just my opinion. Use a single system for dependency resolution - Key: HIVE-4305 URL: https://issues.apache.org/jira/browse/HIVE-4305 Project: Hive Issue Type: Improvement Components: Build Infrastructure, HCatalog Reporter: Travis Crawford Assignee: Carl Steinbach Both Hive and HCatalog use ant as their build tool. However, Hive uses ivy for dependency resolution while HCatalog uses maven-ant-tasks. With the project merge we should converge on a single tool for dependency resolution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4305) Use a single system for dependency resolution
[
https://issues.apache.org/jira/browse/HIVE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637405#comment-13637405
]
Owen O'Malley commented on HIVE-4305:
-
{quote}
Owen, please give some concrete examples of things that make Hadoop's build
more complex than Hive's.
{quote}
* It contains native executables.
* It contains native libraries.
* It contains jni libraries.
{quote}
I think it would be more pragmatic to spend time improving the build that we
currently have
{quote}
Moving to Maven would be making it better in the opinion of the majority of the
development community. The current Hive build is a complex mess and Ivy and
maven ant tasks is really hard to debug.
Certainly, I believe it is possible to make things worse with Maven. I'm not a
fan of how the Hadoop mavenization was done and I deeply regret not taking the
time to make it better as it went in, but it was still better than the ant +
ivy + maven ant tasks that we had. If it hadn't been, it would have been
rejected. That said, in my experience most projects are better off with Maven
builds than ant + ivy + maven ant tasks.
Use a single system for dependency resolution
-
Key: HIVE-4305
URL: https://issues.apache.org/jira/browse/HIVE-4305
Project: Hive
Issue Type: Improvement
Components: Build Infrastructure, HCatalog
Reporter: Travis Crawford
Assignee: Carl Steinbach
Both Hive and HCatalog use ant as their build tool. However, Hive uses ivy
for dependency resolution while HCatalog uses maven-ant-tasks. With the
project merge we should converge on a single tool for dependency resolution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4305) Use a single system for dependency resolution
[ https://issues.apache.org/jira/browse/HIVE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13636534#comment-13636534 ] Owen O'Malley commented on HIVE-4305: - Carl, Rather than debate it theoretically or compare it to Hadoop, which has a *LOT* more complexity in their build, I propose that we have Travis make a Maven build file for the combined Hive and HCat systems. Then we can debate the value and issues in the particular patch and how to move the project forward. The current state is painful with extremely long builds. We need to move forward and enable the project to evolve quickly so that Hive can compete with its many comercial competitors. Use a single system for dependency resolution - Key: HIVE-4305 URL: https://issues.apache.org/jira/browse/HIVE-4305 Project: Hive Issue Type: Improvement Components: Build Infrastructure, HCatalog Reporter: Travis Crawford Assignee: Carl Steinbach Both Hive and HCatalog use ant as their build tool. However, Hive uses ivy for dependency resolution while HCatalog uses maven-ant-tasks. With the project merge we should converge on a single tool for dependency resolution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4305) Use a single system for dependency resolution
[ https://issues.apache.org/jira/browse/HIVE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635734#comment-13635734 ] Carl Steinbach commented on HIVE-4305: -- bq. Does ivy have a completely offline mode? That is what I am most interested in and haven't been able to find it. For example, ivy.cache.ttl.default=eternal doesn't stop the downloading. [~brocknoland] I have good news and bad news. The good news is that Ivy supports completely offline builds via the resolver's useCacheOnly property. I updated my patch for HIVE-4200 with these changes and verified that offline builds work with the network cable pulled out. The bad news is that the HCatalog build is still doing its own thing and doesn't respect the offline flag, so to make this work I had to remove hcatalog from the submodule lists in build.properties. I plan to fix this over the weekend by switching hcatalog over to Ivy. Use a single system for dependency resolution - Key: HIVE-4305 URL: https://issues.apache.org/jira/browse/HIVE-4305 Project: Hive Issue Type: Improvement Components: Build Infrastructure, HCatalog Reporter: Travis Crawford Both Hive and HCatalog use ant as their build tool. However, Hive uses ivy for dependency resolution while HCatalog uses maven-ant-tasks. With the project merge we should converge on a single tool for dependency resolution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4305) Use a single system for dependency resolution
[ https://issues.apache.org/jira/browse/HIVE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635831#comment-13635831 ] Carl Steinbach commented on HIVE-4305: -- Owen, I think you just won the debate! In Ant you have to type -Doffline=true. I tried figuring out how many extra characters that is but kept losing count. Use a single system for dependency resolution - Key: HIVE-4305 URL: https://issues.apache.org/jira/browse/HIVE-4305 Project: Hive Issue Type: Improvement Components: Build Infrastructure, HCatalog Reporter: Travis Crawford Assignee: Carl Steinbach Both Hive and HCatalog use ant as their build tool. However, Hive uses ivy for dependency resolution while HCatalog uses maven-ant-tasks. With the project merge we should converge on a single tool for dependency resolution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4305) Use a single system for dependency resolution
[ https://issues.apache.org/jira/browse/HIVE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13634524#comment-13634524 ] Carl Steinbach commented on HIVE-4305: -- bq. Carl Steinbach, you had asked for a list of benefits, are you convinced by the arguments here? No, I remain unconvinced. So far the points that have been made in favor of switching to Maven are either factually inaccurate (e.g. Ivy doesn't support X when it actually does), or are personal opinions masquerading as statements of fact (e.g. Owen's five points). People need to give concrete reasons (backed with examples) that demonstrate the superiority of Maven over Ivy. I'm happy to reciprocate with counterexamples. Use a single system for dependency resolution - Key: HIVE-4305 URL: https://issues.apache.org/jira/browse/HIVE-4305 Project: Hive Issue Type: Improvement Components: Build Infrastructure, HCatalog Reporter: Travis Crawford Both Hive and HCatalog use ant as their build tool. However, Hive uses ivy for dependency resolution while HCatalog uses maven-ant-tasks. With the project merge we should converge on a single tool for dependency resolution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4305) Use a single system for dependency resolution
[ https://issues.apache.org/jira/browse/HIVE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13634593#comment-13634593 ] Brock Noland commented on HIVE-4305: Does ivy have a completely offline mode? That is what I am most interested in and haven't been able to find it. For example, ivy.cache.ttl.default=eternal doesn't stop the downloading. Use a single system for dependency resolution - Key: HIVE-4305 URL: https://issues.apache.org/jira/browse/HIVE-4305 Project: Hive Issue Type: Improvement Components: Build Infrastructure, HCatalog Reporter: Travis Crawford Both Hive and HCatalog use ant as their build tool. However, Hive uses ivy for dependency resolution while HCatalog uses maven-ant-tasks. With the project merge we should converge on a single tool for dependency resolution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4305) Use a single system for dependency resolution
[ https://issues.apache.org/jira/browse/HIVE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13634624#comment-13634624 ] Owen O'Malley commented on HIVE-4305: - Carl, I fully acknowledge that ant vs maven is a religious discussion. However, to back up my five points: * IDE support is much better From http://www.jetbrains.com/idea/features/ant_maven.html : Maven integration reads the files and builds the modules and dependencies between them. Ant integration executes ant targets. This is similar to eclipse too. For Maven projects, you don't need to maintain a set of helper files that set up the project in the IDE. They can build it automatically. Even with our eclipse helper scripts, users give up on building Hive in an IDE. * Offline support is much better Try turning off the internet and build Hive. It is relatively difficult. Maven will just work if you have the required jars in your cache. * You can download a Maven project and build it without reading the build file. This is obviously true from the fundamentals of each system. Ant provides a wide open playing field and you can build tar in one project and package in another. There are no rules. In Maven, I know what package will build. * Publishing to Maven central is much easier. Ivy can't publish to Maven central, so you end up use ant's maven tasks to publish. This requires that you have two different descriptions of the projects dependencies one for ivy and one for ant's maven tasks. Furthermore, based on my experience as the release manager for Hadoop, ant's maven tasks are much more error-prone. Futhermore, they don't support features like storing your password encrypted. * Profiles work much better in Maven. Ok, this one is debatable. In my opinion, Maven profiles are cleaner and better designed. Finally, I fully support Brock's point: * Maven is used by the other Hadoop ecosystem projects. Hadoop in particular was using ant, ivy, and maven ant tasks for a long time and traded them in for Maven. There is significant value in using similar tools. Use a single system for dependency resolution - Key: HIVE-4305 URL: https://issues.apache.org/jira/browse/HIVE-4305 Project: Hive Issue Type: Improvement Components: Build Infrastructure, HCatalog Reporter: Travis Crawford Both Hive and HCatalog use ant as their build tool. However, Hive uses ivy for dependency resolution while HCatalog uses maven-ant-tasks. With the project merge we should converge on a single tool for dependency resolution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4305) Use a single system for dependency resolution
[ https://issues.apache.org/jira/browse/HIVE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13632015#comment-13632015 ] Gunther Hagleitner commented on HIVE-4305: -- I'm looking at this for HIVE-4200. From reading through this is seems there's more support for maven and there are a bunch of reasons listed for it. [~cwsteinbach], you had asked for a list of benefits, are you convinced by the arguments here? Use a single system for dependency resolution - Key: HIVE-4305 URL: https://issues.apache.org/jira/browse/HIVE-4305 Project: Hive Issue Type: Improvement Components: Build Infrastructure, HCatalog Reporter: Travis Crawford Both Hive and HCatalog use ant as their build tool. However, Hive uses ivy for dependency resolution while HCatalog uses maven-ant-tasks. With the project merge we should converge on a single tool for dependency resolution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4305) Use a single system for dependency resolution
[ https://issues.apache.org/jira/browse/HIVE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13625421#comment-13625421 ] Brock Noland commented on HIVE-4305: Hi Carl, Absolutely. I don't want to push the hive community towards a tool they don't like, however the community seems to be sitting on the fence. 1. Since many users/developers work on other projects in the ecosystem, there is no learning curve for hive. For example, I use maven on Hadoop, Crunch, MRUnit, and Flume among others. I have never used ivy and therefore I have learn two tools to do the same thing. Given the widespread use of maven in the Hadoop ecosystem, it's fair to say I am not the only person who has had to learn ivy for Hive. 2. Since dependency resolution will be done by other projects on my machine, using maven shares that cache. This is a non-trivial amount of build time. 3. Maven has an offline mode which can be enabled on the command line. Ivy allows you to set ttl to eternal but this does not disable downloads entirely. I believe there is an offline mode, but as I understand it requires modification of the build configuration. Since I am no ivy expert, see #1, I got this information from this page: http://stackoverflow.com/questions/6971517/how-do-i-tell-ivy4r-to-stay-offline. The big benefit from my perspective is #1. Cross pollination among projects is easier if learning a new dependency resolution tool is not required. Therefore, if all else is equal, maven is a good choice. Use a single system for dependency resolution - Key: HIVE-4305 URL: https://issues.apache.org/jira/browse/HIVE-4305 Project: Hive Issue Type: Improvement Components: Build Infrastructure, HCatalog Reporter: Travis Crawford Both Hive and HCatalog use ant as their build tool. However, Hive uses ivy for dependency resolution while HCatalog uses maven-ant-tasks. With the project merge we should converge on a single tool for dependency resolution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4305) Use a single system for dependency resolution
[ https://issues.apache.org/jira/browse/HIVE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13625693#comment-13625693 ] Travis Crawford commented on HIVE-4305: --- Unless there are objections, I think we should convert HCatalog back to ivy for managing its dependencies. Reason being Hive already uses it and we're integrating HCatalog into Hive, so unless there's a strong reason not to we should do things the hive way. I'd very much like to avoid any path that involves major changes to stuff that already works. maven-ant-tasks vs. ivy are basically the same. I happen to prefer the former but they're basically equivalent. Thoughts? Use a single system for dependency resolution - Key: HIVE-4305 URL: https://issues.apache.org/jira/browse/HIVE-4305 Project: Hive Issue Type: Improvement Components: Build Infrastructure, HCatalog Reporter: Travis Crawford Both Hive and HCatalog use ant as their build tool. However, Hive uses ivy for dependency resolution while HCatalog uses maven-ant-tasks. With the project merge we should converge on a single tool for dependency resolution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4305) Use a single system for dependency resolution
[ https://issues.apache.org/jira/browse/HIVE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13625854#comment-13625854 ] Owen O'Malley commented on HIVE-4305: - I think we should push forward to Maven. Ivy is *really* painful. Use a single system for dependency resolution - Key: HIVE-4305 URL: https://issues.apache.org/jira/browse/HIVE-4305 Project: Hive Issue Type: Improvement Components: Build Infrastructure, HCatalog Reporter: Travis Crawford Both Hive and HCatalog use ant as their build tool. However, Hive uses ivy for dependency resolution while HCatalog uses maven-ant-tasks. With the project merge we should converge on a single tool for dependency resolution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4305) Use a single system for dependency resolution
[ https://issues.apache.org/jira/browse/HIVE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13625860#comment-13625860 ] Owen O'Malley commented on HIVE-4305: - The advantages of Maven are: * IDE support is much better * Offline support is much better * You can download a Maven project and build it without reading the build file. * Publishing to Maven central is much easier. * Profiles work much better in Maven. Use a single system for dependency resolution - Key: HIVE-4305 URL: https://issues.apache.org/jira/browse/HIVE-4305 Project: Hive Issue Type: Improvement Components: Build Infrastructure, HCatalog Reporter: Travis Crawford Both Hive and HCatalog use ant as their build tool. However, Hive uses ivy for dependency resolution while HCatalog uses maven-ant-tasks. With the project merge we should converge on a single tool for dependency resolution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4305) Use a single system for dependency resolution
[ https://issues.apache.org/jira/browse/HIVE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13625870#comment-13625870 ] Brock Noland commented on HIVE-4305: I agree with Owen. I'd prefer a move towards maven for all of those reasons. Use a single system for dependency resolution - Key: HIVE-4305 URL: https://issues.apache.org/jira/browse/HIVE-4305 Project: Hive Issue Type: Improvement Components: Build Infrastructure, HCatalog Reporter: Travis Crawford Both Hive and HCatalog use ant as their build tool. However, Hive uses ivy for dependency resolution while HCatalog uses maven-ant-tasks. With the project merge we should converge on a single tool for dependency resolution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4305) Use a single system for dependency resolution
[ https://issues.apache.org/jira/browse/HIVE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13625082#comment-13625082 ] Carl Steinbach commented on HIVE-4305: -- [~brocknoland] Would you mind listing the benefits? Use a single system for dependency resolution - Key: HIVE-4305 URL: https://issues.apache.org/jira/browse/HIVE-4305 Project: Hive Issue Type: Improvement Components: Build Infrastructure, HCatalog Reporter: Travis Crawford Both Hive and HCatalog use ant as their build tool. However, Hive uses ivy for dependency resolution while HCatalog uses maven-ant-tasks. With the project merge we should converge on a single tool for dependency resolution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4305) Use a single system for dependency resolution
[ https://issues.apache.org/jira/browse/HIVE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13624242#comment-13624242 ] Gunther Hagleitner commented on HIVE-4305: -- Sorry wrong ticket. Use a single system for dependency resolution - Key: HIVE-4305 URL: https://issues.apache.org/jira/browse/HIVE-4305 Project: Hive Issue Type: Improvement Components: Build Infrastructure, HCatalog Reporter: Travis Crawford Both Hive and HCatalog use ant as their build tool. However, Hive uses ivy for dependency resolution while HCatalog uses maven-ant-tasks. With the project merge we should converge on a single tool for dependency resolution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4305) Use a single system for dependency resolution
[ https://issues.apache.org/jira/browse/HIVE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13624247#comment-13624247 ] Carl Steinbach commented on HIVE-4305: -- What are the advantages of maven-ant-tasks over Ivy? Use a single system for dependency resolution - Key: HIVE-4305 URL: https://issues.apache.org/jira/browse/HIVE-4305 Project: Hive Issue Type: Improvement Components: Build Infrastructure, HCatalog Reporter: Travis Crawford Both Hive and HCatalog use ant as their build tool. However, Hive uses ivy for dependency resolution while HCatalog uses maven-ant-tasks. With the project merge we should converge on a single tool for dependency resolution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4305) Use a single system for dependency resolution
[ https://issues.apache.org/jira/browse/HIVE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13624258#comment-13624258 ] Travis Crawford commented on HIVE-4305: --- The main advantage is you write a standard POM, which can use the Apache parent pom to setup repositories, license, etc. You also can specify repositories that will be inherited by consumers, which we need for the datanucleus and sqlline dependencies. By using maven-ant-tasks the pom you publish in the central repo can be better than the one generated by ivy. The feeling I'm getting though is Hive folks like what's already there and would prefer HCatalog adopt Ivy. I'm fine with that. I'd just like to make sure everyone is on the same page before doing a bunch of migration work. Use a single system for dependency resolution - Key: HIVE-4305 URL: https://issues.apache.org/jira/browse/HIVE-4305 Project: Hive Issue Type: Improvement Components: Build Infrastructure, HCatalog Reporter: Travis Crawford Both Hive and HCatalog use ant as their build tool. However, Hive uses ivy for dependency resolution while HCatalog uses maven-ant-tasks. With the project merge we should converge on a single tool for dependency resolution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4305) Use a single system for dependency resolution
[ https://issues.apache.org/jira/browse/HIVE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13624278#comment-13624278 ] Carl Steinbach commented on HIVE-4305: -- bq. The main advantage is you write a standard POM, which can use the Apache parent pom to setup repositories, license, etc. Ivy's makepom task allows you specify a template POM file that defines the structure of the generated POM. Couldn't we put the parent pom, repository setup, license, etc, etc in this template? More info here: http://ant.apache.org/ivy/history/trunk/use/makepom.html bq. You also can specify repositories that will be inherited by consumers, which we need for the datanucleus and sqlline dependencies. It looks like this is probably covered by the template approach. Please correct me if I'm wrong. I also wanted to add that since HS2 got committed Hive no longer depends on sqlline. We should remove this dependency from jdbc/ivy.xml bq. The feeling I'm getting though is Hive folks like what's already there and would prefer HCatalog adopt Ivy. It's not that I like Ivy, it's just that I don't think any of the alternatives will be any better. Use a single system for dependency resolution - Key: HIVE-4305 URL: https://issues.apache.org/jira/browse/HIVE-4305 Project: Hive Issue Type: Improvement Components: Build Infrastructure, HCatalog Reporter: Travis Crawford Both Hive and HCatalog use ant as their build tool. However, Hive uses ivy for dependency resolution while HCatalog uses maven-ant-tasks. With the project merge we should converge on a single tool for dependency resolution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4305) Use a single system for dependency resolution
[ https://issues.apache.org/jira/browse/HIVE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13624298#comment-13624298 ] Brock Noland commented on HIVE-4305: It seems that most of the hadoop ecosystem uses Maven. All else being equal, isn't there some benefit in being standard with the related projects? Use a single system for dependency resolution - Key: HIVE-4305 URL: https://issues.apache.org/jira/browse/HIVE-4305 Project: Hive Issue Type: Improvement Components: Build Infrastructure, HCatalog Reporter: Travis Crawford Both Hive and HCatalog use ant as their build tool. However, Hive uses ivy for dependency resolution while HCatalog uses maven-ant-tasks. With the project merge we should converge on a single tool for dependency resolution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
