Sure! I'm trying to set up spark and ignite under CDH for shared deployment
and run this example https://github.com/knoldus/spark-ignite
What I have done:
1. Download Ignite binaries and set IGNITE_HOME
2. Added library dependencies (ignite-core, ignite-spark and ignite-spring)
and built it using sbt assembly
3. Added JARs to Spark classpath (spark-env.sh)
4. Started Ignite nodes using ./bin/ignite.sh (except master)
4. Submitted Spark job (spark-submit --master yarn --deploy-mode
cluster.....etc)
It successfully creates 1024 partitions and creates cache but when
retrieving RDD the results are not consistent for example:
//producer
...
val data= Array (1,2,3,4,5,6,7,8,9,10)
sharedRDD.savePairs(sc.parallelize(data, 10).map(i=> (i, 1)))
//consumer
...
val sharedRDD = ic.fromCache[Integer, Integer]("partitioned")
println("The count is:::::::::::: "+sharedRDD.count())
The count at times is 4, other times 10, etc...I don't know if it is some
configuration setting that I'm missing in Cloudera or if there are some
locks needed when reading and writing to cache. Any suggestions would be
appreciated. Thanks!
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Running-Spark-app-in-cluster-tp10073p10079.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
