[
https://issues.apache.org/jira/browse/HIVE-7747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14103706#comment-14103706
]
Venki Korukanti commented on HIVE-7747:
---------------------------------------
Test failure here is related to the change. Failure is complicated. It turns
out that output of {{HiveConf(srcHiveConf, SessionState.class)}} is not same as
srcHiveConf in terms of (property, value) pairs. Executed as part of
constructor, the {{HiveConf.initialize}} method applies system properties on
top of copied properties from srcHiveConf. So from the moment srcHiveConf is
created to the moment of cloning HiveConf if there are any System properties
set, cloned HiveConf inherits those properties. In the test case ({{MiniHS2}})
scratchdir property is modified in System properties (See
[here|https://github.com/apache/hive/blob/trunk/itests/hive-unit/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java#L184]),
but the default scratchdir value is {{$\{test.tmp.dir\}/scratchdir}} from
hive-site.xml. Scrathdir set in {{MiniHS2}} is never used before, but with this
change HS2 started using it. Scratchdir created in {{MiniHS2}} (See
[here|https://github.com/apache/hive/blob/trunk/itests/hive-unit/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java#L183])
doesn't have 777 permissions, so whenever we have user impersonation there are
issues (thats where the test is failing). Before this change, scratchdir is
always {{$\{test.tmp.dir\}/scratchdir}} which is created in HS2 with 777
permissions (See
[here|https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L3458]),
so there were no issues with the impersonation.
I think it is better to fix this in SparkClient by fetching the jar directly
than through HiveConf, to avoid unexpected issues.
> Submitting a query to Spark from HiveServer2 fails
> --------------------------------------------------
>
> Key: HIVE-7747
> URL: https://issues.apache.org/jira/browse/HIVE-7747
> Project: Hive
> Issue Type: Bug
> Components: Spark
> Affects Versions: 0.13.1
> Reporter: Venki Korukanti
> Assignee: Venki Korukanti
> Attachments: HIVE-7747.1.patch
>
>
> {{spark.serializer}} is set to
> {{org.apache.spark.serializer.KryoSerializer}}. Same configuration works fine
> from Hive CLI.
> Spark tasks fails with following error:
> {code}
> Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most
> recent failure: Lost task 0.3 in stage 1.0 (TID 9, 192.168.168.216):
> java.lang.IllegalStateException: unread block data
>
> java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2421)
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1382)
>
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
>
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:84)
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:181)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> java.lang.Thread.run(Thread.java:744)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
