Thank you so much, Marcelo!
It WORKS!
2015-05-21 2:05 GMT+08:00 Marcelo Vanzin :
> Hello,
>
> Sorry for the delay. The issue you're running into is because most HBase
> classes are in the system class path, while jars added with "--jars" are
> only visible to the application class loader created
Hello,
Sorry for the delay. The issue you're running into is because most HBase
classes are in the system class path, while jars added with "--jars" are
only visible to the application class loader created by Spark. So classes
in the system class path cannot see them.
You can work around this by
Thanks, Marcelo!
Below is the full log,
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/cloudera/p
Thanks, Wilfred.
The problem is, the jar "/opt/cloudera/parcels/CD
H/lib/hbase/lib/htrace-core-3.1.0-incubating.jar"
is on every node in the cluster, since we installed CDH 5.4.
thus no matter we run on client or cluster, the driver has access to the
jar.
What's more, the driver does not depend
thanks, Wilfred.
In our program, the "htrace-core-3.1.0-incubating.jar" dependency is only
required in the executor, not in the driver.
while in both "yarn-client" and "yarn-cluster", the executor runs in
cluster.
and it's clearly in "yarn-cluster" mode, the jar IS in
"spark.yarn.secondary.jars",
I look into the "Environment" in both modes.
yarn-client:
spark.jars
local:/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar,file:/home/xxx/my-app.jar
yarn-cluster:
spark.yarn.secondary.jars
local:/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar
I won
Hadoop version: CDH 5.4.
We need to connect to HBase, thus need extra
"/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar"
dependency.
It works in yarn-client mode:
"spark-submit --class xxx.xxx.MyApp --master yarn-client --num-executors 10
--executor-memory 10g --jars
/opt/
