[
https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15328738#comment-15328738
]
Sergey Shelukhin commented on HIVE-13930:
-----------------------------------------
Actually nm the nm, the real error for Spark tests is
{noformat}
2016-06-13T17:22:47,157 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(616)) - 16/06/13 17:22:47 INFO RemoteDriver: Failed
to run job 27c59fd3-9610-4ae4-b126-f548a64dfdf3
2016-06-13T17:22:47,157 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(616)) - java.lang.NoClassDefFoundError: Could not
initialize class org.apache.hadoop.hive.ql.io.AcidUtils
2016-06-13T17:22:47,157 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(616)) - at
org.apache.hadoop.hive.ql.plan.TableScanDesc.<init>(TableScanDesc.java:129)
2016-06-13T17:22:47,157 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(616)) - at
org.apache.hadoop.hive.ql.plan.TableScanDesc.<init>(TableScanDesc.java:122)
2016-06-13T17:22:47,157 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(616)) - at
org.apache.hadoop.hive.ql.plan.TableScanDesc.<init>(TableScanDesc.java:113)
2016-06-13T17:22:47,157 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(616)) - at
org.apache.hadoop.hive.ql.plan.TableScanDescConstructorAccess.newInstance(Unknown
Source)
2016-06-13T17:22:47,157 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(616)) - at
org.apache.hive.com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy$1.newInstance(Kryo.java:1237)
{noformat}
It looks like AcidUtils fails to initialize due to static init error from
shims. When logged, the shims init has the same error - KeyProvider class is
not found.
{noformat}
2016-06-13T17:45:01,037 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(616)) - java.lang.NoClassDefFoundError:
org/apache/hadoop/crypto/key/KeyProvider
2016-06-13T17:45:01,037 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(616)) - at java.lang.Class.forName0(Native
Method)
2016-06-13T17:45:01,037 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(616)) - at
java.lang.Class.forName(Class.java:264)
2016-06-13T17:45:01,037 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(616)) - at
org.apache.hadoop.hive.shims.ShimLoader.createShim(ShimLoader.java:143)
2016-06-13T17:45:01,037 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(616)) - at
org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:138)
2016-06-13T17:45:01,037 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(616)) - at
org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:95)
2016-06-13T17:45:01,037 INFO [stderr-redir-1[]]: client.SparkClientImpl
(SparkClientImpl.java:run(616)) - at
org.apache.hadoop.hive.ql.io.AcidUtils.<clinit>(AcidUtils.java:119)
{noformat}
As far as I see, the packaging for KeyProvider didn't change between Hadoop 2.6
and 2.7.2.
I wonder if it's caused by the fact that HoS tests use prepackaged spark that
includes Hadoop 2.4. KeyProvider class didn't exist in that version of Hadoop,
so it might be causing the issue. Not sure why it worked on 2.6
[~xuefuz] can you please comment? As I have mentioned before this dependency
scheme is really problematic, but whereas previously it was just inconvenient,
in this case it's likely causing a real issue where we would be blocked from
upgrading Hadoop unless we disable Spark mini-cluster tests, or there's some
quick way to the the tgz repackaged with a newer hadoop version.
Or otherwise, I wonder if there's a different error here that isn't in hive.log
> upgrade Hive to latest Hadoop version
> -------------------------------------
>
> Key: HIVE-13930
> URL: https://issues.apache.org/jira/browse/HIVE-13930
> Project: Hive
> Issue Type: Bug
> Reporter: Sergey Shelukhin
> Assignee: Sergey Shelukhin
> Attachments: HIVE-13930.01.patch, HIVE-13930.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
