Hi Stan, 'executor die, that leads ignite cluster rebalance', I haven't expected this before. Our solution is strange, it's a bit like borrowing spark job scheduling API to deploy ignite and other platforms, so, how to persuade them work together is the major consideration, below is my plan, will let you know the actual PDT performance.
1. Spark ExecutorAllocationManager.scala <https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala> there're settings could help with the static Allocations like spark.dynamicAllocation.enabled=false and spark.shuffle.service.enabled=false. 2. As to the data allocation, the only anti-shuffle solution that I could find out is to build a global partition algorithm(for both of Spark RDD and Ignite Affinity collocation), let them do the computing locally and forcedly. 3. In case of executor crash(rebalance), we could lower the ignite cache size using TTL policy, and eventually ensure the realtime balance and overall performance. Marco Stanislav Lukyanov wrote > Well… I mean, if you really want to… :) > > The issue is that spark may start or stop its executors as it sees fit – > or, at least, > not taking Ignite’s lifecycle into account. > If an executor dies, the Ignite node dies. It doesn’t always lead to data > loss, > but it does lead to topology changes and rebalance which are the most > costly operations for Ignite. > So each time spark will try to start or stop an executor to make your > computations more > efficient, it will actually lead to Ignite doing a lot of extra work. > > You can try using the embedded mode, but keep in mind possible performance > drops > when executors are requested or removed, possible data loss (complete or > partial) > and the fact that this mode is eventually going away. I’d advise to go for > the standalone deployment. > > Stan -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
