Hey Cheng, Right now we aren't using stable API's to communicate with the Hive Metastore. We didn't want to drop support for Hive 0.12 so right now we are using a shim layer to support compiling for 0.12 and 0.13. This is very costly to maintain.
If Hive has a stable meta-data API for talking to a Metastore, we should use that (is HCatalog sufficient for this purpose?). Ideally we would be able to talk to multiple versions of the Hive metastore and we can keep a single internal version of Hive for our use of Serde's, etc. I've created SPARK-4114 for this: https://issues.apache.org/jira/browse/SPARK-4114 This is a very important issue for Spark SQL, so I'd welcome comments on that JIRA from anyone who is familiar with Hive/HCatalog internals. - Patrick On Mon, Oct 27, 2014 at 9:54 PM, Cheng, Hao <hao.ch...@intel.com> wrote: > Hi, all > > I have some PRs blocked by hive upgrading (e.g. > https://github.com/apache/spark/pull/2570), the problem is some internal > hive method signature changed, it's hard to make the compatible in code > level (sql/hive) when switching back/forth the Hive versions. > > > > I guess the motivation of the upgrading is to support the Metastore with > different Hive versions. So, how about just keep the metastore related hive > jars upgrading or utilize the HCatalog directly? And of course we can either > leaving hive-exec.jar hive-cli.jar etc as 0.12 or upgrade to 0.13.1, but not > support them both. > > > > Sorry if I missed some discussion of Hive upgrading. > > > > Cheng Hao --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
