[
https://issues.apache.org/jira/browse/SPARK-20059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15942976#comment-15942976
]
Saisai Shao edited comment on SPARK-20059 at 3/27/17 10:02 AM:
---------------------------------------------------------------
[~sowen], it probably is not the same issue.
In SPARK-20019, jars are added through SQL command in to Hive's SessionState,
the code path is different.
And in SPARK-11421, AFAIK, the target is to add jars to the current
classloader in the runtime.
Here the issue is during start, jars specified through {{--jars}} should be
added to Spark's child classloader in yarn cluster mode.
They're probably all classloader related issues, but I think they're targeted
to the different area, and touched the different code path.
was (Author: jerryshao):
[~sowen], it probably is not the same issue.
In SPARK-20019, jars are added through SQL command in to Hive's SessionState,
the code path is different.
And in SPARK-11421, AFAIK, the target is to add jars to the current
classloader in the runtime.
Here the issue is during start, jars specified through {{--jars}} should be
added to Spark's child classloader in yarn cluster mode.
They're all probably all classloader related issues, but I think they're
targeted to the different area, and touched the different code path.
> HbaseCredentialProvider uses wrong classloader
> ----------------------------------------------
>
> Key: SPARK-20059
> URL: https://issues.apache.org/jira/browse/SPARK-20059
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 2.1.0, 2.2.0
> Reporter: Saisai Shao
>
> {{HBaseCredentialProvider}} uses system classloader instead of child
> classloader, which will make HBase jars specified with {{--jars}} fail to
> work, so here we should use the right class loader.
> Besides in yarn cluster mode jars specified with {{--jars}} is not added into
> client's class path, which will make it fail to load HBase jars and issue
> tokens in our scenario. Also some customized credential provider cannot be
> registered into client.
> So here I will fix this two issues.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
