Vladimir,
There are 2 things that I'm experiencing so far:
1. I have added the following code to spark-env.sh in my CDH installation
IGNITE_HOME=/etc/ignite-1.5.0
IGNITE_LIBS="${IGNITE_HOME}/libs/*"
for file in ${IGNITE_HOME}/libs/*
do
if [ -d ${file} ] && [ "${file}" != "${IGNITE_HOME}"/libs/optional ];
then
IGNITE_LIBS=${IGNITE_LIBS}:${file}/*
fi
done
but Ignite jars are still not recognized by Spark after restarting Spark as
well as the entire CDH
. My location of CDH is the default one:
/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/etc/spark/conf.dist
So I decided to compile my code into a jar-with-dependencies which led me to
the second issue:
2. Looks like the jars were discovered by spark-submit, however, now I'm
getting the following exception:
Exception in thread "main" java.lang.NoClassDefFoundError:
javax/cache/configuration/MutableConfiguration
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.marshallerSystemCache(IgnitionEx.java:2098)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.initializeDefaultCacheConfiguration(IgnitionEx.java:1914)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.initializeConfiguration(IgnitionEx.java:1899)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1573)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1547)
at
org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1003)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:534)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:515)
at org.apache.ignite.Ignition.start(Ignition.java:322)
at
org.apache.ignite.spark.IgniteContext.ignite(IgniteContext.scala:153)
at
org.apache.ignite.spark.IgniteContext.<init>(IgniteContext.scala:62)
at
training.PnLMergerVectorIgnite$.main(PnLMergerVectorIgnite.scala:50)
at training.PnLMergerVectorIgnite.main(PnLMergerVectorIgnite.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at
org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException:
javax.cache.configuration.MutableConfiguration
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
I know that the class javax.cache.configuration.MutableConfiguration belongs
to cache-api-1.0.0.jar and I see it present in my jar-with-dependencies
Thank you for looking at this.
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Ignite-Installation-with-Spark-under-CDH-tp4457p4507.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
