[
https://issues.apache.org/jira/browse/SPARK-11818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15011168#comment-15011168
]
Jungtaek Lim edited comment on SPARK-11818 at 11/18/15 3:18 PM:
----------------------------------------------------------------
Yes, assembly jar has HBase libs, and unzipped assembly jar has
hbase-default.xml.
I attached assembly jar by --jars options, and run some codes which initializes
HBase configuration from "Executor side".
Executor found "spark.repl.class.uri" config, and replace current thread's
context class loader to ExecutorClassLoader, which has URLClassLoader to parent
class loader.
Assembly jar is loaded to parent class loader via
Executor.updateDependencies(), and ExecutorClassLoader can see classes from
parent class loader so there're no issue to load classes.
But HBaseConfiguration.addHbaseResources() tries to load hbase-default.xml from
classloader which seems not succeed.
was (Author: kabhwan):
Yes, assembly jar has HBase libs, and unzipped assembly jar has
hbase-default.xml.
I attached assembly jar by --jars options, and run some codes which initializes
HBase configuration from "Executor".
Executor found "spark.repl.class.uri" config, and replace current thread's
context class loader to ExecutorClassLoader, which has URLClassLoader to parent
class loader.
Assembly jar is loaded to parent class loader via
Executor.updateDependencies(), and ExecutorClassLoader can see classes from
parent class loader so there're no issue to load classes.
But HBaseConfiguration.addHbaseResources() tries to load hbase-default.xml from
classloader which seems not succeed.
> REPL ExecutorClassLoader cannot see any resources from parent class loader
> --------------------------------------------------------------------------
>
> Key: SPARK-11818
> URL: https://issues.apache.org/jira/browse/SPARK-11818
> Project: Spark
> Issue Type: Bug
> Components: Spark Shell
> Environment: CentOS 6, spark 1.4.1-hadoop2.4, mesos 0.22.1
> Reporter: Jungtaek Lim
>
> This issue starts from finding root reason from strange problem from
> spark-shell (and zeppelin) which is not a problem for spark-submit.
> https://mail-archives.apache.org/mod_mbox/spark-user/201511.mbox/%3CCAF5108jMXyOjiGmCgr%3Ds%2BNvTMcyKWMBVM1GsrH7Pz4xUj48LfA%40mail.gmail.com%3E
> After some hours (over days) digging into the detail, I found that
> ExecutorClassLoader cannot see any resource files which can be seen from
> parent class loader.
> ExecutorClassLoader itself doesn't need to lookup resource files cause REPL
> doesn't generate these, but it should delegate lookup to parent class loader.
> I'll provide the pull request which includes tests which fails on master soon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
