[
https://issues.apache.org/jira/browse/HIVE-98?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashish Thusoo updated HIVE-98:
------------------------------
Attachment: patch-98_3.txt
Added the following things:
1. ivy dependencies on hadoop so we do not have to package hadoop jars, bin and
conf files along with hive. With Ivy, the hadoop version can be downloaded
automatically. In order to compile with say 0.18.0 you have to specify
ant -Dhadoop.version="0.18.0"
Currently the mirror used for downloads is
http://archive.apache.org/dist
This can also be configured using -Dhadoop.mirror="url"
2. Added ant based preprocessing so we can exclude code which is incompatible
with a certain version of hadoop. The only file that does this right now is
HiveInputFormat.java which excludes validateInput method while compiling with
0.19.*. The exclusion macros are defined in ql/build.xml (we may later move
this to build-common, if we see cases of conditional compilation in other parts
of the hive code - for now it was convenient for me to reuse gen-java
directories in ql/build.xml in order to store the preprocessed code).
3. Fixed certain ordering sensitive tests so that they are not dependent on the
different behavior of reducer merge algorithm when the reduce key is random.
4. Another minor change is in HiveConf.java, where in the hadoop conf directory
is also picked relative to HADOOP_HOME
Open issues:
With 0.17 and 0.18 the following tests still fail (input16_cc.q, input16.q and
input3.q). I will open a separate JIRA to address those as those are related to
how we discover user defined serdes and user defined functions from aux.jars.
Also once those are fixed, we can add another target in ant (test-long) so that
a transaction can be regressed with all versions of hadoop - and this can be
run in hudson to validate submitted patches.
> Dependency management with hadoop core using either maven or ivy
> ----------------------------------------------------------------
>
> Key: HIVE-98
> URL: https://issues.apache.org/jira/browse/HIVE-98
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Build Infrastructure
> Reporter: Ashish Thusoo
> Assignee: Ashish Thusoo
> Attachments: patch-98.txt, patch-98_2.txt, patch-98_3.txt
>
>
> We need to move from pre packaging jars to managing external dependencies
> with hadoop core (and later other packaged jars) using maven or ivy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
