Can you pleas tell me what ports should be opened on driver or yarn cluster
machine? this is our production env, I suspect that something need be
configured on firewall ports.
1)
I verified all these steps on my local cluster(3 spark machine[1 master and
2 workers], one yarn cluster machine), everything is OK except cache
counter.
cache counter issues:
val igniteContext = new IgniteContext[Integer,Integer](sc, () => new
IgniteConfiguration(),false)
val sharedRDD = igniteContext.fromCache("sharedBaselineCacheRDD")
val initalRDD = sc.parallelize(1 to 100000,10).map(i => (new Integer(i),
new Integer(i)))
println("initalRDD.couner=/. " + initalRDD.count() +"\tpartition=> " +
initalRDD.partitions.size)
sharedRDD.savePairs(initalRDD, true)
println("=====>totalcounter" + sharedRDD.count + "\t paris => " +
sharedRDD.partitions.size)
println("=====>" + sharedRDD.filter(_._2 > 50000).count)
*final result is 40000 NOT 50000, this is happened also on embed integration
with spark.*
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tp6910p7051.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
