GitHub user zhzhan opened a pull request:
https://github.com/apache/spark/pull/2241
Spark 2706
Given that a lot of users are trying to use hive 0.13 in spark, and the
incompatibility between hive-0.12 and hive-0.13 on the API level I want to
propose following approach, which has no or minimum impact on existing
hive-0.12 support, but be able to jumpstart the development of hive-0.13 and
future version support.
Approach: Introduce âhive-versionâ property, and manipulate pom.xml
files to support different hive version at compiling time through shim layer,
e.g., hive-0.12.0 and hive-0.13.1. More specifically,
1. For each different hive version, there is a very light layer of shim
code to handle API differences, sitting in sql/hive/version, e.g.,
sql/hive/v0.12 or sql/hive/v0.13
2. Add a new profile hive-default active by default, which picks up all
existing configuration and hive-0.12.0 shim (v0.12) if no hive.version is
specified.
3. If user specifies different version (currently only 0.13.1 by
-Dhive.version = 0.13.1), hive-0.13 profile will be activated, which pick up
hive-0.13.1 specific configuration, mainly the hive jars and hive-0.13.1 shim
(v0.13).
4. With this approach, nothing is changed with current hive-0.12 support.
No change by default: sbt/sbt -Phive
For example: sbt/sbt -Phive -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0
assembly
To enable hive-0.13: sbt/sbt -Dhive.version=0.13.1
For example: sbt/sbt -Dhive.version=0.13.1 -Pyarn -Phadoop-2.4
-Dhadoop.version=2.4.0 assembly
Note that in hive-0.13, hive-thriftserver is not enabled, which should be
fixed by other Jira, and we donât need -Phive with -Dhive.version in building
(probably we should use -Phive -Dhive.version=xxx instead after thrift server
is also supported in hive-0.13).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhzhan/spark spark-2706
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2241.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2241
----
commit 7d5fce2f1dad4a4f42c21b82e8feabf3dbe50903
Author: Zhan Zhang <[email protected]>
Date: 2014-08-08T17:47:18Z
test
commit 42585ecfce299d0bda54bd169daba944263e59d3
Author: Zhan Zhang <[email protected]>
Date: 2014-08-08T17:47:18Z
test
commit 70ffd9398a5d1d5196ec027402bfc8990c07fef3
Author: Zhan Zhang <[email protected]>
Date: 2014-08-08T17:57:06Z
revert
commit fe0f379bade257bb69b610328ce042d8850c26bc
Author: Zhan Zhang <[email protected]>
Date: 2014-08-08T18:01:12Z
Merge branch 'master' of https://github.com/zhzhan/spark
commit 70964fe6e3ee952d3b8301cc17bad83804fe492f
Author: Zhan Zhang <[email protected]>
Date: 2014-08-08T18:02:01Z
revert
commit dbedff3e6fbb1c6558ce3047cf60edbc43ffa02a
Author: Zhan Zhang <[email protected]>
Date: 2014-08-29T07:14:35Z
Merge remote-tracking branch 'upstream/master'
commit ba14f283d8a68cc90ee828eee5dcc9a95e6dd5a4
Author: Zhan Zhang <[email protected]>
Date: 2014-08-08T17:47:18Z
test
commit f6a8a40baa588539727d2cd450f6b638756b9b29
Author: Zhan Zhang <[email protected]>
Date: 2014-08-08T17:57:06Z
revert
commit cb53a2cc4b25878c9ce1e8629f13739d0e26b2bf
Author: Zhan Zhang <[email protected]>
Date: 2014-08-30T18:01:37Z
Merge branch 'master' of https://github.com/apache/spark
commit 789ea21617178ea039b306070a69fcc9c807a053
Author: Zhan Zhang <[email protected]>
Date: 2014-09-02T20:31:23Z
Merge branch 'master' of https://github.com/apache/spark
commit f896b2a5df7f914ba81140e0becdeb0c92c7eafe
Author: Zhan Zhang <[email protected]>
Date: 2014-09-02T20:31:58Z
Merge branch 'master' into spark-2706
commit 921e914083929aeccf95c7c6876fe182ccc9ee74
Author: Zhan Zhang <[email protected]>
Date: 2014-09-02T23:07:30Z
Merge branch 'master' of https://github.com/apache/spark
commit 87ebf3b2ec3fdf81c2188c0be46452d7621508e4
Author: Zhan Zhang <[email protected]>
Date: 2014-09-02T23:08:17Z
Merge branch 'master' into spark-2706
commit 94b4fdc8600cc6d5d4fe338e293be74081a0e2d4
Author: Zhan Zhang <[email protected]>
Date: 2014-09-02T23:28:33Z
Spark-2706: hive-0.13.1 support on spark
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
