Thanks for your reply Val.

We tried that, now it stopped spinning up several servers, all of those
became clients now (I still dont know the difference). But still just for a
single row rdd it takes several minutes to retrieve it. 

It still spins up 100s of executor even though i specify 1 executor 1 core.
When digged deeper, i think its creating way too many tasks, more than 1000,
so its making existing executor to backlog and making spark to spin up new
executors to process. 

If this is the expected behaviour, then I'm not sure Ignite SharedRDD is
tested enough and is functional.

Just for reference, i will create a bug as well on this behaviour and adding
code snippet of my consumer (Its slightly tweaked version of the SharedRDD
example in github).

val ccfg = new CacheConfiguration[Int, AssetWTag]()
ccfg.setName(args(0).toString)
ccfg.setIndexedTypes(classOf[Int], classOf[AssetWTag])
//ccfg.setCacheMode(CacheMode.REPLICATED)
//ccfg.setAtomicityMode(CacheAtomicityMode.ATOMIC)
//ccfg.setBackups(0)
//ccfg.setAffinity(new RendezvousAffinityFunction(false, 1));
igniteContext.ignite().addCacheConfiguration(ccfg)

// Retrieve sharedRDD back from the Cache.
val transformedValues: IgniteRDD[Int, SomeEntity] =
igniteContext.fromCache(ccfg)

println(">>> Transforming values stored in Ignite Shared RDD...")

// Filter out pairs which square roots are less than 100 and
// take the first five elements from the transformed IgniteRDD and print
them.
transformedValues.take(5).foreach(println) // THIS RETURNS 1 ROW AFTER
SPINNING UP 100 EXECUTORS, AROUND 1.5 mins

println(">>> Executing SQL query over Ignite Shared RDD...")

// Execute a SQL query over the Ignite Shared RDD.
val df = transformedValues.sql("select * from SomeEntity")

// Show ten rows from the result set.
df.show(10)  // THIS ALWAYS RETURNS EMPTY
vkulichenko wrote
> Looks like executors start additional server nodes, while they should be
> in client mode instead when standalone mode is used. I would recommend to
> explicitly provide clientMode=true property in the config provided to
> IgniteContext.
> 
> I also created a ticket for this:
> https://issues.apache.org/jira/browse/IGNITE-5981
> 
> -Val





--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Ignite-behaving-strange-with-Spark-SharedRDD-in-AWS-EMR-Yarn-Client-Mode-tp16011p16059.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Reply via email to