Re: System.exit() not exiting cleanly (locked on IgnitionEx$IgniteNamedInstance)

Thanks for your response. We are disabling shmem this weekend in production as per your advice. Will let you know how that helps next week. The issue started happening after 3 months of smooth running, so we were curious to understand how it became a problem suddenly. -- View this message in

Re: System.exit() not exiting cleanly (locked on IgnitionEx$IgniteNamedInstance)

Binti, please see the answers below > Denis, is it ok to delete this file while the cluster is up? or this folder? > servers and clietns might be connected to it. Delete the whole folder after the cluster is down. > is System.exit() or Ignite.stop(grid, false) not a good solution? Should we >

Re: System.exit() not exiting cleanly (locked on IgnitionEx$IgniteNamedInstance)

Denis, is it ok to delete this file while the cluster is up? or this folder? servers and clietns might be connected to it. I posted another question for the community trying to understand the use of the work directory. Could you please respond to that? This lock.file has not been updated since

Re: System.exit() not exiting cleanly (locked on IgnitionEx$IgniteNamedInstance)

Barrett, The data streamer is designed for caches preloading. There are no any serious issues related to it that prevents from using it in production. If you have one please share more details. — Denis > On Jun 27, 2016, at 1:46 AM, Barrett Strausser wrote: > >

Re: System.exit() not exiting cleanly (locked on IgnitionEx$IgniteNamedInstance)

Binti, As I see from the thread dumps the node is not stopped because the following shmem related Thread is trying to acquire a lock preventing the node from finalization "ipc-shmem-gc-#24%DataGridServer-Production%" prio=10 tid=0x7f2124cc4000 nid=0x12cc05 sleeping[0x7f20fd435000]

Re: System.exit() not exiting cleanly (locked on IgnitionEx$IgniteNamedInstance)

bearrito, in 1.6 we saw streamer hung on flush() or close() and would never return. We do not see that issue in 1.5.0.final. Igniters, could you please look at the earlier attached log files and thread dumps to help with the original issue. Thanks, Binti -- View this message in context:

Re: System.exit() not exiting cleanly (locked on IgnitionEx$IgniteNamedInstance)

Off-topic to this thread but I'm curious about the issues the OP (Binti) mentioned with DataStreamer introducint instability. I'm planning a mid-size deployment that will rely heavily on custom data streamer and JMS streamer. Binti, would you mind emailing me directly or posting to the group? -b

Re: System.exit() not exiting cleanly (locked on IgnitionEx$IgniteNamedInstance)

Attached is the zipped RGP.zip log file (RGP.log) from the same client node that hangs. I see the shmem thread gets into a GC iteration at the same time. Do you think that might be causing it not to shut down? Thanks, Binti

Re: System.exit() not exiting cleanly (locked on IgnitionEx$IgniteNamedInstance)

We are seeing multiple issues with 1.6.0 with dataStreamer and when server nodes are killed, the grid becomes unstable and unresponsive. So at this point we cannot really use 1.6 here is the thread dump for the thread you asked for and the ones I thought were relevant. I will try to get all

Re: System.exit() not exiting cleanly (locked on IgnitionEx$IgniteNamedInstance)

Please provide full thread dumps from all the nodes at the time it happens again. In particular I’m interested in what is going with the thread named "ipc-shmem-gc”. In addition to the thread dumps attache the logs from all the nodes for investigation. As a workaround you can disable shmem as

Re: System.exit() not exiting cleanly (locked on IgnitionEx$IgniteNamedInstance)

JDK 1.7, 16 node cluster, each JVM 10 GB heap and 4 GB off heap. 4 nodes run on 4 linux boxes, each box has 64 GB memory. the heap is under-utilized when this happens. so doubt GC will cause it. multiple clients connect to it. clients cannot exit properly either with System.exit() or

Re: System.exit() not exiting cleanly (locked on IgnitionEx$IgniteNamedInstance)

Coul you tell more detail how reproduce issue. What OS and java version used. How many memory has OS and how many used. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/System-exit-not-exiting-cleanly-locked-on-IgnitionEx-IgniteNamedInstance-tp5814p5870.html

Re: System.exit() not exiting cleanly (locked on IgnitionEx$IgniteNamedInstance)

Hello bintisepaha, Try disable shared memory in config file TcpCommunicationSpi/setSharedMemoryPort(-1) Maybe Runtime.getRuntime().halt(int) can terminate the program. -- View this message in context:

Re: System.exit() not exiting cleanly (locked on IgnitionEx$IgniteNamedInstance)

I also see this when the process is stuck at system.exit, many SIGINT calls. Name: SIGINT handler State: BLOCKED on java.lang.Class@4debc642 owned by: main Total blocked: 1 Total waited: 0 Stack trace: java.lang.Shutdown.exit(Shutdown.java:212) java.lang.Terminator$1.handle(Terminator.java:52)

System.exit() not exiting cleanly (locked on IgnitionEx$IgniteNamedInstance)

Hi Igniters, We are on 1.5.0.final and have been on it for a while. But we started seeing this issue in production recently. We want a JVM to call System.exit() when its done its work, but we see the below locking. This affects us badly, we cannot kill the processes on linux with a Ctrl+C either.