RE: Node failed to startup due to deadlock

lexey From: Alexey Popov Sent: Monday, November 13, 2017 2:12 PM To: user@ignite.apache.org Subject: RE: Node failed to startup due to deadlock Hi Naresh, I still don't have a clear understanding of your case. Very probably you just need a Cache Store with Read-Through enabled. Please have

RE: Node failed to startup due to deadlock

Hi Naresh, I still don't have a clear understanding of your case. Very probably you just need a Cache Store with Read-Through enabled. Please have a look at [1] and Cache Store examples. As for code provided - you can have a workaround here until

RE: Node failed to startup due to deadlock

Hi Alexey, Please find the attached sample code to reproduce the deadlock (App should be started first, then wait till cache is locked, then start the App2. App2 will fail to start Ignite) App.java App2.java

RE: Node failed to startup due to deadlock

sday, November 9, 2017 9:54 AM To: user@ignite.apache.org Subject: RE: Node failed to startup due to deadlock Hi Alexey, Thank you for pointing out the problem with caches causing deadlocks. I fixed the classpath pointing to 1.9, but the issue still exists. Actually we could reproduce dead

RE: Node failed to startup due to deadlock

Hi Alexey, Thank you for pointing out the problem with caches causing deadlocks. I fixed the classpath pointing to 1.9, but the issue still exists. Actually we could reproduce deadlock with a scenario when two nodes trying to comeup during same time, and the first node holds lock on a cache.

RE: Node failed to startup due to deadlock

gnite ver. 2.3.0-SNAPSHOT#19691231-sha1:DEV INFO: IGNITE_HOME=C:\development\software\apache-ignite-fabric-1.9.0-bin Please update IGNITE_HOME to a correct path. Thank you, Alexey From: naresh.goty Sent: Monday, November 6, 2017 5:36 AM To: user@ignite.apache.org Subject: RE: Node failed to st

RE: Node failed to startup due to deadlock

Hi Alexey, We are still seeing the deadlocks for the scenario i have specified earlier. We tried the below two changes, but still seeing the deadlocks. Can you please provide some pointers about the issue based on the logs and threaddumps attached. 1) As Rajeev mentioned, we tried offloading

RE: Node failed to startup due to deadlock

Hi Alexey, Even after commenting all the code in the event handlers (and offloading to application threadpools), the problem persists. Your description about this related to topology changes, leads me to suspect this is the same defect as: https://issues.apache.org/jira/browse/IGNITE-6380# I have

RE: Node failed to startup due to deadlock

: Topology snapshot [ver=4, servers=2, clients=0, CPUs=8, heap=5.5GB] Thank you, Alexey From: rajivgandhi Sent: Thursday, November 2, 2017 7:43 PM To: user@ignite.apache.org Subject: Re: Node failed to startup due to deadlock Can you please also help us understand, how you diagnized the symptom

Re: Node failed to startup due to deadlock

Thanks Alexey. Do you have specific information on which operation? Like I said, the only in thread ignite operation we do is ignite Message. How are you seeing issues with cache operations? Are you saying reads are also a problem cause we do reads in ignite thread. Writes are in our own thread.

Re: Node failed to startup due to deadlock

Hi Rajeev, All Ignite callbacks are processed in (=called from) internal Ignite thread pools. The generic rule here is to avoid direct Ignite calls from such callbacks. So, it is better to do all your things in separate threads (use ExecutorService for instance). You should pass all work to your

Re: Node failed to startup due to deadlock

Hi Alexey, Actually i have shared threaddumps from both nodes earlier. I have re-created the scenario, and attached the log and threaddumps from the Node1. Note: Multiple Thread dumps are taken in different times for Node 1 when Node2 was down and started again. Regards, Naresh

Re: Node failed to startup due to deadlock

Hi Alexey, We do have event listeners to handle data loss events. We do 3 things related to ignite (amongst other unrelated things) in the listener (btw, an instance of local listener if running on each node, these listeners have remote filters as well): 1. Read an Ignite cache 2. Publish a

Re: Node failed to startup due to deadlock

Naresh, Rajeev I've looked through the logs. Node2 is stuck because it is waiting for Node1 (id=96eea214-a2e3-4e0a-b8c6-7fdf3f29727c) partition map exchange response. So it looks like the real deadlock is at Node1 (id=96eea214-a2e3-4e0a-b8c6-7fdf3f29727c) Can you also send node1 logs and thread

Re: Node failed to startup due to deadlock

Thanks Alexey. Please find the attached logs and configuration on Node 2. Regards, Naresh Node2.log Node2_configuration.txt -- Sent

Re: Node failed to startup due to deadlock

Hi Alexey, Is there an associated defect in Jira? Is there an isolated & verified patch that can be backported to 2.0? thanks, Rajeev -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Node failed to startup due to deadlock

Hi Naresh, Can you share your log from Node 2 and configuration file? The issue is related to Cache PartitionExchange procedure during node startup. Very probably this issue is already fixed on master branch. Thank you in advance, Alexey -- Sent from: