Re: Ignite Installation with Spark under CDH

Hi Michael, Ok, so it looks like the process didn't have enough heap. Thank you for your inputs about CDH configuration. We will improve our documentation based on this. Vladimir On Thu, Apr 28, 2016 at 5:15 PM, mdolgonos wrote: > Vladimir, > > I fixed this by

Re: Ignite Installation with Spark under CDH

Hi Michael, Did you have a chance to analyze heap dump to understand what caused OOME? As per *IgniteConfiguration*, it is made non-serialziable intentionally, because we do not expect it to be passed over the wire. Could you please provide a stack trace where you see it is being serialized?

Re: Ignite Installation with Spark under CDH

Vladimir, Update - I think I solved the ClassNotFound exception. It looks like the Ignite installation document for Spark and CDH is outdated and doesn't contain complete information on integrating Ignite with Spark running in a 'Yarn' (cluster) mode on CDH which I have. This is what I have done

Re: Ignite Installation with Spark under CDH

Vladimir, Unless I'm missing something there is no console to echo it to since in CDH all services start by Cloudera Manager. The only console I have is the one I execute spark-submit in but it doesn't have any Spark nor CDH related properties. Thanks -- View this message in context:

Re: Ignite Installation with Spark under CDH

Hi Michael, I meant to echo to the console SPARK_CLASSPATH variable which you created in *spark-env.sh* following recommendations from Ignite docs. We need to ensure that JCache jar is included into it. BTW, do you see Ignite jars in " java.class.path" property? Vladimir. On Wed, Apr 27, 2016

Re: Ignite Installation with Spark under CDH

Vladimir, I don't see this property. I put the following code at the very beginning of my main() method: println(" Props=" + System.getProperties()) The only classpath related property I see is java.class.path which doesn't contain cache-api-1.0.0.jar despite my changing spark-env.sh as

Re: Ignite Installation with Spark under CDH

Hi Michael, Could you please print resulting value of the SPARK_CLASSPATH variable? Vladimir. On Tue, Apr 26, 2016 at 8:55 PM, mdolgonos wrote: > Vladimir, > I verified that the cache jar is in the Cloudera jars directory. All the > cache packages are also included

Re: Ignite Installation with Spark under CDH

Vladimir, I verified that the cache jar is in the Cloudera jars directory. All the cache packages are also included in the deployment jar-with-dependencies as I used org.apache.ignite ignite-spark

Re: Ignite Installation with Spark under CDH

Hi, This is pretty hard to say what is the root cause, especially in complex deployments like CDH. Most probably you JAR is packaged incorrectly because your application is able to load Ignite classes, but cannot load jcache API. Could you try to simply put cache-api-1.0.0.jar to all places and

Re: Ignite Installation with Spark under CDH

Vladimir, There are 2 things that I'm experiencing so far: 1. I have added the following code to spark-env.sh in my CDH installation IGNITE_HOME=/etc/ignite-1.5.0 IGNITE_LIBS="${IGNITE_HOME}/libs/*" for file in ${IGNITE_HOME}/libs/* do if [ -d ${file} ] && [ "${file}" !=

Re: Ignite Installation with Spark under CDH

Hi, Could you please clarify the exact problem? Do you see any exception? Vladimir. On Fri, Apr 22, 2016 at 4:57 PM, mdolgonos wrote: > I'm trying to install and integrate Ignite with Spark under CDH by > following > the recommendation at >