hi, all:
Recently, we have encountered a problem while using spark sql to write orc
table, which is related to https://issues.apache.org/jira/browse/HIVE-10790.
In order to fix this problem we decided to patched the PR to the hive
branch which spark1.5 rely on.
We pull the hive branch(
https://github.com/pwendell/hive/tree/release-1.2.1-spark) and compile it
with cmd: mvn clean package -Phadoop-2,dist -DskipTests, and then upload to
the nexus without any problem.
<https://github.com/pwendell/hive/tree/release-1.2.1-spark>
But when we compile spark with hive (group: org.spark-project.hive,
version: 1.2.1.spark) using cmd: ./make-distribution.sh --tgz -Phive
-Phive-thriftserver -Psparkr -Pyarn -Dhadoop.version=2.4.1
-Dprotobuf.version=2.5.0 -DskipTests
we get this error msg:
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
spark-hive_2.10 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 3 resources
[INFO][INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @
spark-hive_2.10 ---
[INFO] Using zinc server for incremental compilation
[info] Compiling 27 Scala sources and 1 Java source to
/home/sankuai/zenglinxi/spark/sql/hive/target/scala-2.10/classes...
[warn] Class org.apache.hadoop.hive.shims.HadoopShims not found -
continuing with a stub.
[error]
/home/sankuai/zenglinxi/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala:35:
object shims is not a member of package org.apache.hadoop.hive
[error] import org.apache.hadoop.hive.shims.{HadoopShims, ShimLoader}
[error] ^
[error]
/home/sankuai/zenglinxi/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala:114:
not found: value ShimLoader
[error] val loadedShimsClassName =
ShimLoader.getHadoopShims.getClass.getCanonicalName
[error] ^
[error]
/home/sankuai/zenglinxi/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala:123:
not found: type ShimLoader
[error] val shimsField =
classOf[ShimLoader].getDeclaredField("hadoopShims")
[error] ^
[error]
/home/sankuai/zenglinxi/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala:127:
not found: type HadoopShims
[error] val shims =
classOf[HadoopShims].cast(shimsClass.newInstance())
[error] ^
[warn] Class org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge not found
- continuing with a stub.
[warn] Class org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge not found
- continuing with a stub.
[warn] Class org.apache.hadoop.hive.shims.HadoopShims not found -
continuing with a stub.
[warn] four warnings found
[error] four errors found
[error] Compile failed at 2016-5-13 16:34:44 [4.348s]
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [
3.105 s]
[INFO] Spark Project Launcher ............................. SUCCESS [
8.360 s]
[INFO] Spark Project Networking ........................... SUCCESS [
8.491 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [
5.110 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [
6.854 s]
[INFO] Spark Project Core ................................. SUCCESS [02:33
min]
[INFO] Spark Project Bagel ................................ SUCCESS [
5.183 s]
[INFO] Spark Project GraphX ............................... SUCCESS [
15.744 s]
[INFO] Spark Project Streaming ............................ SUCCESS [
39.070 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [
57.416 s]
[INFO] Spark Project SQL .................................. SUCCESS [01:11
min]
[INFO] Spark Project ML Library ........................... SUCCESS [01:28
min]
[INFO] Spark Project Tools ................................ SUCCESS [
2.539 s]
[INFO] Spark Project Hive ................................. FAILURE [
13.273 s]
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project YARN ................................. SKIPPED
This error information does throw doubt on
https://issues.apache.org/jira/browse/HIVE-10790.
Any one meet the same question with me?
