My detailed issues as below this link, please refer to this link
http://apache-ignite-users.70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tt6910.html
My question:
1) What ports should be opened on driver machine or yarn machine?(There
issues were happned on our production env)
2) But in my local env(3 spark cluster[1 master, 2 workers] and 1 yarn
cluster) and there is no issues as i manually closed my firewall, then fixed
this issue.
I faced another issues as below
========================
1)
val igniteContext = new IgniteContext[Integer,Integer](sc, () => new
IgniteConfiguration())
val sharedRDD = igniteContext.fromCache("sharedBaselineCacheRDD")
val initalRDD = sc.parallelize(1 to 100000,10).map(i => (new Integer(i),
new Integer(i)))
println("initalRDD.couner=/. " + initalRDD.count() +"\tpartition=> " +
initalRDD.partitions.size)
sharedRDD.savePairs(initalRDD, true)
println("=====>totalcounter" + sharedRDD.count + "\t paris => " +
sharedRDD.partitions.size)
println("=====>" + sharedRDD.filter(_._2 > 50000).count)
2) submit a jar to yarn cluster as below command
/u01/spark-1.6.0-hive/bin/spark-submit --driver-memory 4G --class
com.ValidSparkCache --master yarn --executor-cores 5 --executor-memory 1000m
--num-executors 10 --conf spark.rdd.compress=false --conf
spark.shuffle.compress=false --conf spark.broadcast.compress=false
/u01/xxx/sparkignite.jar -quesize 10 -batchSize 5000 -writethread 30
-runningSeconds 20
3) final result should be 50000, but actullay result is 40000,3000(ramdom)
final result is 40000 NOT 50000, this is happened also on embed integration
with spark.
--
View this message in context:
http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tp10686.html
Sent from the Apache Ignite Developers mailing list archive at Nabble.com.
