1.a. So basically Ignite cannot help much here. Would wrapping the save
around an IgniteTransaction help? When spark node crashes I can rollback the
transaction so that the data in the data grid is consistent. This also means
I should be using IgniteTransaction api for all the operations performed on
the data grid. Is my understanding right?
1.b. Ok. Assuming I have backups configured, I do not need a transaction to
be wrapped around to keep the data consistent in the data grid.
2. Right, I understand that the data is not loaded in the spark memory when
igniteContext.fromCache("partitioned") call is made. Assume the data flow is
as below
a. val employeesRDD = igniteContext.fromCache("partitioned")
b. //work on the rdd by modifying the salaries of all employees in
California. This step will generate a new rdd named employeesModifiedRDD
c. employeesModifiedRDD.savePairs() //save the rdd back to the data grid
Assuming the employees cache is partitioned by region. In 2.c. can ignite
determine that only the partition that holds the employee records of
California region are updated, hence I should execute the update only on
that server node that holds those records and not blindly update all records
in the cache.
Thanks for your inputs.
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Consistency-Guarantees-Smart-Updates-with-Spark-Integration-tp10091p10112.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
