Hello Ray, It's hard to say that the issue you mentioned is the cause of your problem. To determine it, it will be very good if you get thread dumps on next such network glitch both from server and client nodes (using jstack e.g.). I'm not aware of Ignite Spark DataFrames implementation features, but in general, in case of any node join/left Ignite cluster triggers process named PME (Partitions Map Exchange). During this process, all write operations to the cluster are frozen until PME end. In your case, PME has performed a long time due to communication issues between server nodes (server nodes were unable to send an acknowledgment to each other to continue PME, while Discovery worked well which is a bit strange). This can cause why you didn't see updates. In case of client connection exception, a client node should try to reconnect to another server node and complete his data write futures with an exception, so Spark Executor which uses client node to stream data to Ignite should catch this network exception and perform re-connect and retry of data batch writing.
For more details about Spark DataFrames implementation in Ignite, you may ask Nikolay Izhikov (I attached his email as the recipient in this letter). 2018-07-26 5:50 GMT+03:00 Ray <ray...@cisco.com>: > Hello Pavel, > > Here's the log for for node ids = [429edc2b-eb14-414f-a978-9bfe35443c8c, > 6783732c-9a13-466f-800a-ad4c8d9be3bf]. > 6783732c-9a13-466f-800a-ad4c8d9be3bf.zip > <http://apache-ignite-users.70518.x6.nabble.com/file/ > t1346/6783732c-9a13-466f-800a-ad4c8d9be3bf.zip> > 429edc2b-eb14-414f-a978-9bfe35443c8c.zip > <http://apache-ignite-users.70518.x6.nabble.com/file/ > t1346/429edc2b-eb14-414f-a978-9bfe35443c8c.zip> > I examined the logs and looks like there's a network issue here because > there's a lot of "java.net.NoRouteToHostException: No route to host" > exception. > > I did a little research and found this ticket may be the cause. > https://issues.apache.org/jira/browse/IGNITE-8739 > > Will the client(spark executor in this case) retry data insert if I apply > this patch when the network glitch is resolved? > > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
