Hive on Spark engine only works with Spark 1.3.1.

Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 29 November 2016 at 07:56, Furcy Pin <furcy....@flaminem.com> wrote:

> ClassNotFoundException generally means that jars are missing from your
> class path.
>
> You probably need to link the spark jar to $HIVE_HOME/lib
> https://cwiki.apache.org/confluence/display/Hive/Hive+
> on+Spark%3A+Getting+Started#HiveonSpark:GettingStarted-ConfiguringHive
>
> On Tue, Nov 29, 2016 at 2:03 AM, Joaquin Alzola <joaquin.alz...@lebara.com
> > wrote:
>
>> Hi Guys
>>
>>
>>
>> No matter what I do that when I execute “select count(*) from employee” I
>> get the following output on the logs:
>>
>> It is quiet funny because if I put hive.execution.engine=mr the output is
>> correct. If I put hive.execution.engine=spark then I get the bellow errors.
>>
>> If I do the search directly through spark-shell it work great.
>>
>> +-------+
>>
>> |    _c0|
>>
>> +-------+
>>
>> |1005635|
>>
>> +-------+
>>
>> So there has to be a problem from hive to spark.
>>
>>
>>
>> Seems as the RPC(??) connection is not setup …. Can somebody guide me on
>> what to look for.
>>
>> spark.master=spark://172.16.173.31:7077
>>
>> hive.execution.engine=spark
>>
>> spark.executor.extraClassPath    /mnt/spark/lib/spark-1.6.2-yar
>> n-shuffle.jar:/mnt/hive/lib/hive-exec-2.0.1.jar
>>
>>
>>
>> Hive2.0.1à Spark 1.6.2 –> Hadoop – 2.6.5 à Scala 2.10
>>
>>
>>
>> 2016-11-29T00:35:11,099 WARN  [RPC-Handler-2]: rpc.RpcDispatcher
>> (RpcDispatcher.java:handleError(142)) - Received error
>> message:io.netty.handler.codec.DecoderException:
>> java.lang.NoClassDefFoundError: org/apache/hive/spark/client/Job
>>
>>         at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteT
>> oMessageDecoder.java:358)
>>
>>         at io.netty.handler.codec.ByteToMessageDecoder.channelRead(Byte
>> ToMessageDecoder.java:230)
>>
>>         at io.netty.handler.codec.ByteToMessageCodec.channelRead(ByteTo
>> MessageCodec.java:103)
>>
>>         at io.netty.channel.AbstractChannelHandlerContext.invokeChannel
>> Read(AbstractChannelHandlerContext.java:308)
>>
>>         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRe
>> ad(AbstractChannelHandlerContext.java:294)
>>
>>         at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(Ch
>> annelInboundHandlerAdapter.java:86)
>>
>>         at io.netty.channel.AbstractChannelHandlerContext.invokeChannel
>> Read(AbstractChannelHandlerContext.java:308)
>>
>>         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRe
>> ad(AbstractChannelHandlerContext.java:294)
>>
>>         at io.netty.channel.DefaultChannelPipeline.fireChannelRead(Defa
>> ultChannelPipeline.java:846)
>>
>>         at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.
>> read(AbstractNioByteChannel.java:131)
>>
>>         at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEven
>> tLoop.java:511)
>>
>>         at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimiz
>> ed(NioEventLoop.java:468)
>>
>>         at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEve
>> ntLoop.java:382)
>>
>>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
>>
>>         at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(
>> SingleThreadEventExecutor.java:111)
>>
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> Caused by: java.lang.NoClassDefFoundError: org/apache/hive/spark/client/J
>> ob
>>
>>         at java.lang.ClassLoader.defineClass1(Native Method)
>>
>>         at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>>
>>         at java.security.SecureClassLoader.defineClass(SecureClassLoade
>> r.java:142)
>>
>>         at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>>
>>         at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>>
>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>>
>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>>
>>         at java.security.AccessController.doPrivileged(Native Method)
>>
>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>>
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>
>>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>>
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
>>
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>
>>         at java.lang.Class.forName0(Native Method)
>>
>>         at java.lang.Class.forName(Class.java:348)
>>
>>         at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassR
>> esolver.readName(DefaultClassResolver.java:154)
>>
>>         at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassR
>> esolver.readClass(DefaultClassResolver.java:133)
>>
>>         at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(
>> Kryo.java:670)
>>
>>         at org.apache.hive.com.esotericsoftware.kryo.serializers.
>> ObjectField.read(ObjectField.java:118)
>>
>>         at org.apache.hive.com.esotericsoftware.kryo.serializers.
>> FieldSerializer.read(FieldSerializer.java:551)
>>
>>         at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndO
>> bject(Kryo.java:790)
>>
>>         at org.apache.hive.spark.client.rpc.KryoMessageCodec.decode(Kry
>> oMessageCodec.java:97)
>>
>>         at io.netty.handler.codec.ByteToMessageCodec$1.decode(ByteToMes
>> sageCodec.java:42)
>>
>>         at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteT
>> oMessageDecoder.java:327)
>>
>>         ... 15 more
>>
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.hive.spark.client.Job
>>
>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>>
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>
>>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>>
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>
>>         ... 39 more
>> This email is confidential and may be subject to privilege. If you are
>> not the intended recipient, please do not copy or disclose its content but
>> contact the sender immediately upon receipt.
>>
>
>

Reply via email to