Hey Cheng,
Right now we aren't using stable API's to communicate with the Hive
Metastore. We didn't want to drop support for Hive 0.12 so right now
we are using a shim layer to support compiling for 0.12 and 0.13. This
is very costly to maintain.
If Hive has a stable meta-data API for talking to a Metastore, we
should use that (is HCatalog sufficient for this purpose?). Ideally we
would be able to talk to multiple versions of the Hive metastore and
we can keep a single internal version of Hive for our use of Serde's,
etc.
I've created SPARK-4114 for this:
https://issues.apache.org/jira/browse/SPARK-4114
This is a very important issue for Spark SQL, so I'd welcome comments
on that JIRA from anyone who is familiar with Hive/HCatalog internals.
- Patrick
On Mon, Oct 27, 2014 at 9:54 PM, Cheng, Hao wrote:
> Hi, all
>
>I have some PRs blocked by hive upgrading (e.g.
> https://github.com/apache/spark/pull/2570), the problem is some internal
> hive method signature changed, it's hard to make the compatible in code
> level (sql/hive) when switching back/forth the Hive versions.
>
>
>
> I guess the motivation of the upgrading is to support the Metastore with
> different Hive versions. So, how about just keep the metastore related hive
> jars upgrading or utilize the HCatalog directly? And of course we can either
> leaving hive-exec.jar hive-cli.jar etc as 0.12 or upgrade to 0.13.1, but not
> support them both.
>
>
>
> Sorry if I missed some discussion of Hive upgrading.
>
>
>
> Cheng Hao
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
