For Crail I put all jars in the directory including all dependencies. If you
build it from source these will be in assembly/...
Regards,
Jonas
On Thu, 20 Jun 2019 13:16:12 +0000
David Crespi <[email protected]> wrote:
I started off doing that way, and I think I was hitting another java
error with something like
a “number out of bounds” error, as in too many files perhaps. Do
you only put specific files
into the jar directory? I’ll go back and play around with it again
today and report back.
Regards,
David
________________________________
From: Jonas Pfefferle <[email protected]>
Sent: Thursday, June 20, 2019 12:17:36 AM
To: [email protected]; David Crespi
Subject: Re: Question on wildcard used with spark-default.conf
Hi David,
the wildcard works for me without problem. However I do not use the
environment variable. Can you try putting the actual Crail path
instead of
the env variable.
Jonas
On Wed, 19 Jun 2019 22:33:09 +0000
David Crespi <[email protected]> wrote:
A question on what is documented on the crail website for
configuring to use
the HDFS adapter. The website shows the use of a wildcard (in red):
spark.driver.extraClassPath
$CRAIL_HOME/jars/*:$CRAIL_HOME/jars/crail-spark-1.0.jar:.
spark.executor.extraClassPath
$CRAIL_HOME/jars/*:$CRAIL_HOME/jars/crail-spark-1.0.jar:.
But this didn’t work for me. I had to put all the specific jars
onto this path in order for this to work,
such as the lines below. Is there a trick to using the wildcard?
spark.driver.extraClassPath
/crail/jars/crail-client-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-hdfs-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-namenode-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-darpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-spark-1.0.jar:/crail/jars/crail-storage-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT-tests.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-rdma-1.2-incubating-SNAPSHOT.jar:/crail/jars/darpc-1.8.jar:/crail/jars/disni-2.0.jar:/crail/jars/htrace-core-3.1.0-incubating.jar:/crail/jars/narpc-1.4.jar:/jNVMf/target/*:/dinsi/target/*
spark.executor.extraClassPath
/crail/jars/crail-client-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-hdfs-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-namenode-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-darpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-spark-1.0.jar:/crail/jars/crail-storage-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT-tests.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-rdma-1.2-incubating-SNAPSHOT.jar:/crail/jars/darpc-1.8.jar:/crail/jars/disni-2.0.jar:/crail/jars/htrace-core-3.1.0-incubating.jar:/crail/jars/narpc-1.4.jar:/jNVMf/target/*:/dinsi/target/*
This is the error I get with using the wildcard:
Exception in thread "main" java.lang.RuntimeException:
java.lang.ClassNotFoundException: Class
org.apache.crail.hdfs.CrailHadoopFileSystem not found
And if I add that one to the path, then I get the next one it
doesn’t find.
Regards,
David
