I have preliminary patch against spark1.0.2, which is attached to spark-2706.
Now I am working on supporting both hive-0.12 and hive-0.13.1 with
non-intrusive way (not breaking any existing hive-0.12 when introduce
supporting new version). I will attach a proposal to solve multi-version
support
Thanks for working on this! Its unclear at the moment exactly how we are
going to handle this, since the end goal is to be compatible with as many
versions of Hive as possible. That said, I think it would be great to open
a PR in this case. Even if we don't merge it, thats a good way to get it
see to upgrade to Hive 0.13.1+ from Hive 0.12.0.
There may be others out there that I'm not aware of, but I do see it coming.
my 2 cents.
From: mich...@databricks.com
Date: Mon, 25 Aug 2014 13:08:42 -0700
Subject: Re: Working Formula for Hive 0.13?
To: wangf...@huawei.com
CC: dev
I have worked for a branch update the hive version to hive-0.13(by
org.apache.hive)---https://github.com/scwf/spark/tree/hive-0.13
I am wondering whether it's ok to make a PR now because hive-0.13 version is
not compatible with hive-0.12 and here i used org.apache.hive.
On 2014/7/29 8:22,
The API change seems not major. I have locally change it and compiled, but
not test yet. The major problem is still how to solve the hive-exec jar
dependency. I am willing to help on this issue. Is it better stick to the
same way as hive-0.12 until hive-exec is cleaned enough to switch back?
--
I can compile with no error, but my patch also includes other stuff.
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/Working-Formula-for-Hive-0-13-tp7551p7775.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Here is the patch. Please ignore the pom.xml related change, which just for
compiling purpose. I need to further work on this one based on Wandou's
previous work.
--
View this message in context:
Sorry, forget to upload files. I have never posted before :) hive.diff
http://apache-spark-developers-list.1001551.n3.nabble.com/file/n/hive.diff
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/Working-Formula-for-Hive-0-13-tp7551p.html
Could you make a PR as described here:
https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark
On Fri, Aug 8, 2014 at 1:57 PM, Zhan Zhang zhaz...@gmail.com wrote:
Sorry, forget to upload files. I have never posted before :) hive.diff
Attached the diff the PR SPARK-2706. I am currently working on this problem.
If somebody are also working on this, we can share the load.
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/Working-Formula-for-Hive-0-13-tp7551p7782.html
Sent from the
I found 0.13.1 artifacts in maven:
http://search.maven.org/#artifactdetails%7Corg.apache.hive%7Chive-metastore%7C0.13.1%7Cjar
However, Spark uses groupId of org.spark-project.hive, not org.apache.hive
Can someone tell me how it is supposed to work ?
Cheers
On Mon, Jul 28, 2014 at 7:44 AM,
Yes, it is published. As of previous versions, at least, hive-exec
included all of its dependencies *in its artifact*, making it unusable
as-is because it contained copies of dependencies that clash with
versions present in other artifacts, and can't be managed with Maven
mechanisms.
I am not
Talked with Owen offline. He confirmed that as of 0.13, hive-exec is still
uber jar.
Right now I am facing the following error building against Hive 0.13.1 :
[ERROR] Failed to execute goal on project spark-hive_2.10: Could not
resolve dependencies for project
It would be great if the hive team can fix that issue. If not, we'll
have to continue forking our own version of Hive to change the way it
publishes artifacts.
- Patrick
On Mon, Jul 28, 2014 at 9:34 AM, Ted Yu yuzhih...@gmail.com wrote:
Talked with Owen offline. He confirmed that as of 0.13,
Owen helped me find this:
https://issues.apache.org/jira/browse/HIVE-7423
I guess this means that for Hive 0.14, Spark should be able to directly
pull in hive-exec-core.jar
Cheers
On Mon, Jul 28, 2014 at 9:55 AM, Patrick Wendell pwend...@gmail.com wrote:
It would be great if the hive team
Where and how is that fork being maintained? I'm not seeing an obviously
correct branch or tag in the main asf hive repo github mirror.
On Mon, Jul 28, 2014 at 9:55 AM, Patrick Wendell pwend...@gmail.com wrote:
It would be great if the hive team can fix that issue. If not, we'll
have to
Yeah so we need a model for this (Mark - do you have any ideas?). I
did this in a personal github repo. I just did it quickly because
dependency issues were blocking the 1.0 release:
https://github.com/pwendell/hive/tree/branch-0.12-shaded-protobuf
I think what we want is to have a semi official
AFAIK, according a recent talk, Hulu team in China has built Spark SQL
against Hive 0.13 (or 0.13.1?) successfully. Basically they also
re-packaged Hive 0.13 as what the Spark team did. The slides of the talk
hasn't been released yet though.
On Tue, Jul 29, 2014 at 1:01 AM, Ted Yu
I've heard from Cloudera that there were hive internal changes between
0.12 and 0.13 that required code re-writing. Over time it might be
possible for us to integrate with hive using API's that are more
stable (this is the domain of Michael/Cheng/Yin more than me!). It
would be interesting to see
Getting and maintaining our own branch in the main asf hive repo is a
non-starter or isn't workable?
On Mon, Jul 28, 2014 at 10:17 AM, Patrick Wendell pwend...@gmail.com
wrote:
Yeah so we need a model for this (Mark - do you have any ideas?). I
did this in a personal github repo. I just did
Exactly, forgot to mention Hulu team also made changes to cope with those
incompatibility issues, but they said that’s relatively easy once the
re-packaging work is done.
On Tue, Jul 29, 2014 at 1:20 AM, Patrick Wendell pwend...@gmail.com wrote:
I've heard from Cloudera that there were hive
After manually copying hive 0.13.1 jars to local maven repo, I got the
following errors when building spark-hive_2.10 module :
[ERROR]
/homes/xx/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala:182:
type mismatch;
found : String
required: Array[String]
[ERROR]
I was looking for a class where reflection-related code should reside.
I found this but don't think it is the proper class for bridging
differences between hive 0.12 and 0.13.1:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
Cheers
On Mon, Jul 28, 2014 at 3:41
A few things:
- When we upgrade to Hive 0.13.0, Patrick will likely republish the
hive-exec jar just as we did for 0.12.0
- Since we have to tie into some pretty low level APIs it is unsurprising
that the code doesn't just compile out of the box against 0.13.0
- ScalaReflection is for
The larger goal is to get a clean compile test in the environment I have
to use. As near as I can tell, tests fail in parquet because parquet was
only added in Hive 0.13. There could well be issues in later meta-stores,
but one thing at a time...
- SteveN
On 7/28/14, 17:22, Michael
bq. Either way its unclear to if there is any reason to use reflection to
support multiple versions, instead of just upgrading to Hive 0.13.0
Which Spark release would this Hive upgrade take place ?
I agree it is cleaner to upgrade Hive dependency vs. introducing reflection.
Cheers
On Mon, Jul
26 matches
Mail list logo
