I am not sure this is going to resolve INSERT OVEERWRITE into ORC table issue. Can you go to hive and do
show create table custom.rank_less_orc_none and send the output. Is that table defined as transactional? Other alternative is to use Spark to insert into a normal text table and do insert from the text table into ORC using HiveContext or doing it purely in Hive Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com On 15 May 2016 at 04:01, linxi zeng <linxizeng0...@gmail.com> wrote: > hi, all: > Recently, we have encountered a problem while using spark sql to write orc > table, which is related to > https://issues.apache.org/jira/browse/HIVE-10790. > In order to fix this problem we decided to patched the PR to the hive > branch which spark1.5 rely on. > We pull the hive branch( > https://github.com/pwendell/hive/tree/release-1.2.1-spark) and compile it > with cmd: mvn clean package -Phadoop-2,dist -DskipTests, and then upload to > the nexus without any problem. > <https://github.com/pwendell/hive/tree/release-1.2.1-spark> > But when we compile spark with hive (group: org.spark-project.hive, > version: 1.2.1.spark) using cmd: ./make-distribution.sh --tgz -Phive > -Phive-thriftserver -Psparkr -Pyarn -Dhadoop.version=2.4.1 > -Dprotobuf.version=2.5.0 -DskipTests > we get this error msg: > > [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ > spark-hive_2.10 --- > [INFO] Using 'UTF-8' encoding to copy filtered resources. > [INFO] Copying 1 resource > [INFO] Copying 3 resources > [INFO][INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @ > spark-hive_2.10 --- > [INFO] Using zinc server for incremental compilation > [info] Compiling 27 Scala sources and 1 Java source to > /home/sankuai/zenglinxi/spark/sql/hive/target/scala-2.10/classes... > [warn] Class org.apache.hadoop.hive.shims.HadoopShims not found - > continuing with a stub. > [error] > /home/sankuai/zenglinxi/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala:35: > object shims is not a member of package org.apache.hadoop.hive > [error] import org.apache.hadoop.hive.shims.{HadoopShims, ShimLoader} > [error] ^ > [error] > /home/sankuai/zenglinxi/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala:114: > not found: value ShimLoader > [error] val loadedShimsClassName = > ShimLoader.getHadoopShims.getClass.getCanonicalName > [error] ^ > [error] > /home/sankuai/zenglinxi/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala:123: > not found: type ShimLoader > [error] val shimsField = > classOf[ShimLoader].getDeclaredField("hadoopShims") > [error] ^ > [error] > /home/sankuai/zenglinxi/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala:127: > not found: type HadoopShims > [error] val shims = > classOf[HadoopShims].cast(shimsClass.newInstance()) > [error] ^ > [warn] Class org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge not > found - continuing with a stub. > [warn] Class org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge not > found - continuing with a stub. > [warn] Class org.apache.hadoop.hive.shims.HadoopShims not found - > continuing with a stub. > [warn] four warnings found > [error] four errors found > [error] Compile failed at 2016-5-13 16:34:44 [4.348s] > [INFO] > ------------------------------------------------------------------------ > [INFO] Reactor Summary: > [INFO] > [INFO] Spark Project Parent POM ........................... SUCCESS [ > 3.105 s] > [INFO] Spark Project Launcher ............................. SUCCESS [ > 8.360 s] > [INFO] Spark Project Networking ........................... SUCCESS [ > 8.491 s] > [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ > 5.110 s] > [INFO] Spark Project Unsafe ............................... SUCCESS [ > 6.854 s] > [INFO] Spark Project Core ................................. SUCCESS [02:33 > min] > [INFO] Spark Project Bagel ................................ SUCCESS [ > 5.183 s] > [INFO] Spark Project GraphX ............................... SUCCESS [ > 15.744 s] > [INFO] Spark Project Streaming ............................ SUCCESS [ > 39.070 s] > [INFO] Spark Project Catalyst ............................. SUCCESS [ > 57.416 s] > [INFO] Spark Project SQL .................................. SUCCESS [01:11 > min] > [INFO] Spark Project ML Library ........................... SUCCESS [01:28 > min] > [INFO] Spark Project Tools ................................ SUCCESS [ > 2.539 s] > [INFO] Spark Project Hive ................................. FAILURE [ > 13.273 s] > [INFO] Spark Project REPL ................................. SKIPPED > [INFO] Spark Project YARN ................................. SKIPPED > > This error information does throw doubt on > https://issues.apache.org/jira/browse/HIVE-10790. > Any one meet the same question with me? >
