Is it possible this is somehow causing the issue of the node stopping? On Thu, 24 Oct 2019 at 11:24, Ilya Kasnacheev <[email protected]> wrote:
> Hello! > > This likely means that you have reachability problems in your cluster, > such as, xxx.xxx.xxx.68 can connect to xxx.xxx.xxx.82 (on range > 47100-47200) but not the other way around. > > Regards, > -- > Ilya Kasnacheev > > > пн, 21 окт. 2019 г. в 19:36, John Smith <[email protected]>: > >> I also see this printing every few seconds on my client application... >> org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi Accepted >> incoming communication connection [locAddr=/xxx.xxx.xxx.68:47101, >> rmtAddr=/xxx.xxx.xxx.82:49816 >> >> On Mon, 21 Oct 2019 at 12:04, John Smith <[email protected]> wrote: >> >>> Hi, thanks. I already made sure that each ignite VM runs on a separate >>> host within our cloud. I'm not doing any of that migration stuff. >>> >>> Also recently disabled metrics igniteConfig.setMetricsLogFrequency(0); >>> Just to make sure it doesn't get too chatty. But i doubt this would affect >>> it... >>> >>> Should I maybe set the node timeout a bit higher then 30 seconds maybe >>> put it to 60 seconds? I remember people suggesting this but not sure... >>> >>> On Fri, 18 Oct 2019 at 06:09, Denis Mekhanikov <[email protected]> >>> wrote: >>> >>>> The following documentation page has some useful points on deployment >>>> in a virtualised environment: >>>> https://apacheignite.readme.io/docs/vmware-deployment >>>> >>>> Denis >>>> On 17 Oct 2019, 17:41 +0300, John Smith <[email protected]>, >>>> wrote: >>>> >>>> Ok I have metribeat running on the VM hopefully I will see something... >>>> >>>> On Thu, 17 Oct 2019 at 05:09, Denis Mekhanikov <[email protected]> >>>> wrote: >>>> >>>>> There are no long pauses in the GC logs, so it must be the whole VM >>>>> pause. >>>>> >>>>> Denis >>>>> On 16 Oct 2019, 23:07 +0300, John Smith <[email protected]>, >>>>> wrote: >>>>> >>>>> Sorry here is the gc logs for all 3 machines: >>>>> https://www.dropbox.com/s/chbbxigahd4v9di/gc-logs.zip?dl=0 >>>>> >>>>> On Wed, 16 Oct 2019 at 15:49, John Smith <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi, so it happened again here is my latest gc.log stats: >>>>>> https://gceasy.io/diamondgc-report.jsp?oTxnId_value=a215d573-d1cf-4d53-acf1-9001432bb28e >>>>>> >>>>>> Everything seems ok to me. I also have Elasticsearch Metricbeat >>>>>> running, the CPU usage looked normal at the time. >>>>>> >>>>>> On Thu, 10 Oct 2019 at 13:05, Denis Mekhanikov <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Unfortunately, I don’t. >>>>>>> You can ask the VM vendor or the cloud provider (if you use any) for >>>>>>> a proper tooling or logs. >>>>>>> Make sure, that there is no such step in the VM’s lifecycle that >>>>>>> makes it freeze for a minute. >>>>>>> Also make sure that the physical CPU is not overutilized and no VMs >>>>>>> that run on it are starving. >>>>>>> >>>>>>> Denis >>>>>>> On 10 Oct 2019, 19:03 +0300, John Smith <[email protected]>, >>>>>>> wrote: >>>>>>> >>>>>>> Do you know of any good tools I can use to check the VM? >>>>>>> >>>>>>> On Thu, 10 Oct 2019 at 11:38, Denis Mekhanikov < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> > Hi Dennis, so are you saying I should enable GC logs + the safe >>>>>>>> point logs as well? >>>>>>>> >>>>>>>> Having safepoint statistics in your GC logs may be useful, so I >>>>>>>> recommend enabling them for troubleshooting purposes. >>>>>>>> Check the lifecycle of your virtual machines. There is a high >>>>>>>> chance that the whole machine is frozen, not just the Ignite node. >>>>>>>> >>>>>>>> Denis >>>>>>>> On 10 Oct 2019, 18:25 +0300, John Smith <[email protected]>, >>>>>>>> wrote: >>>>>>>> >>>>>>>> Hi Dennis, so are you saying I should enable GC logs + the safe >>>>>>>> point logs as well? >>>>>>>> >>>>>>>> On Thu, 10 Oct 2019 at 11:22, John Smith <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> You are correct, it is running in a VM. >>>>>>>>> >>>>>>>>> On Thu, 10 Oct 2019 at 10:11, Denis Mekhanikov < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi! >>>>>>>>>> >>>>>>>>>> There are the following messages in the logs: >>>>>>>>>> >>>>>>>>>> [22:26:21,816][WARNING][jvm-pause-detector-worker][IgniteKernal%xxxxxx] >>>>>>>>>> Possible too long JVM pause: *55705 milliseconds*. >>>>>>>>>> ... >>>>>>>>>> [22:26:21,847][SEVERE][ttl-cleanup-worker-#48%xxxxxx%][G] Blocked >>>>>>>>>> system-critical thread has been detected. This can lead to >>>>>>>>>> cluster-wide >>>>>>>>>> undefined behaviour [threadName=partition-exchanger, blockedFor= >>>>>>>>>> *57s*] >>>>>>>>>> >>>>>>>>>> Looks like the JVM was paused for almost a minute. It doesn’t >>>>>>>>>> seem to be caused by a garbage collection, since there is no >>>>>>>>>> evidence of GC >>>>>>>>>> pressure in the GC log. Usually such big pauses happen in virtualised >>>>>>>>>> environments when backups are captured from machines or they just >>>>>>>>>> don’t >>>>>>>>>> have enough CPU time. >>>>>>>>>> >>>>>>>>>> Looking at safepoint statistics may also reveal some interesting >>>>>>>>>> details. You can learn about safepoints here: >>>>>>>>>> https://blog.gceasy.io/2016/12/22/total-time-for-which-application-threads-were-stopped/ >>>>>>>>>> >>>>>>>>>> Denis >>>>>>>>>> On 9 Oct 2019, 23:14 +0300, John Smith <[email protected]>, >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> So the error sais to set clientFailureDetectionTimeout=30000 >>>>>>>>>> >>>>>>>>>> 1- Do I put a higher value than 30000? >>>>>>>>>> 2- Do I do it on the client or the server nodes or all nodes? >>>>>>>>>> 3- Also if a client is misbehaving why shutoff the server node? >>>>>>>>>> >>>>>>>>>> On Thu, 3 Oct 2019 at 21:02, John Smith <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> But if it's the client node that's failing why is the server >>>>>>>>>>> node stopping? I'm pretty sure we do verry simple put and get >>>>>>>>>>> operations. >>>>>>>>>>> All the client nodes are started as client=true >>>>>>>>>>> >>>>>>>>>>> On Thu., Oct. 3, 2019, 4:18 p.m. Denis Magda, <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi John, >>>>>>>>>>>> >>>>>>>>>>>> I don't see any GC pressure or STW pauses either. If not GC >>>>>>>>>>>> then it might have been caused by a network glitch or some >>>>>>>>>>>> long-running >>>>>>>>>>>> operation started by the app. These logs statement >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [22:26:21,827][WARNING][tcp-disco-client-message-worker-#10%xxxxxx%][TcpDiscoverySpi] >>>>>>>>>>>> Client node considered as unreachable and will be dropped from >>>>>>>>>>>> cluster, >>>>>>>>>>>> because no metrics update messages received in interval: >>>>>>>>>>>> TcpDiscoverySpi.clientFailureDetectionTimeout() ms. It may be >>>>>>>>>>>> caused by >>>>>>>>>>>> network problems or long GC pause on client node, try to increase >>>>>>>>>>>> this >>>>>>>>>>>> parameter. [nodeId=b07182d0-bf70-4318-9fe3-d7d5228bd6ef, >>>>>>>>>>>> clientFailureDetectionTimeout=30000] >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [22:26:21,839][WARNING][tcp-disco-client-message-worker-#12%xxxxxx%][TcpDiscoverySpi] >>>>>>>>>>>> Client node considered as unreachable and will be dropped from >>>>>>>>>>>> cluster, >>>>>>>>>>>> because no metrics update messages received in interval: >>>>>>>>>>>> TcpDiscoverySpi.clientFailureDetectionTimeout() ms. It may be >>>>>>>>>>>> caused by >>>>>>>>>>>> network problems or long GC pause on client node, try to increase >>>>>>>>>>>> this >>>>>>>>>>>> parameter. [nodeId=302cff60-b88d-40da-9e12-b955e6bf973d, >>>>>>>>>>>> clientFailureDetectionTimeout=30000] >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [22:26:21,847][SEVERE][ttl-cleanup-worker-#48%xxxxxx%][G] >>>>>>>>>>>> Blocked system-critical thread has been detected. This can lead to >>>>>>>>>>>> cluster-wide undefined behaviour [threadName=partition-exchanger, >>>>>>>>>>>> blockedFor=57s] >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 22:26:21,954][SEVERE][ttl-cleanup-worker-#48%xxxxxx%][] >>>>>>>>>>>> Critical system error detected. Will be handled accordingly to >>>>>>>>>>>> configured >>>>>>>>>>>> handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, >>>>>>>>>>>> timeout=0, >>>>>>>>>>>> super=AbstractFailureHandler >>>>>>>>>>>> [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], >>>>>>>>>>>> failureCtx=FailureContext >>>>>>>>>>>> [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: >>>>>>>>>>>> GridWorker >>>>>>>>>>>> [name=partition-exchanger, igniteInstanceName=xxxxxx, >>>>>>>>>>>> finished=false, >>>>>>>>>>>> heartbeatTs=1568931981805]]] >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> - >>>>>>>>>>>> Denis >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Oct 3, 2019 at 11:50 AM John Smith < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> So I have been monitoring my node and the same one seems to >>>>>>>>>>>>> stop once in a while. >>>>>>>>>>>>> >>>>>>>>>>>>> https://www.dropbox.com/s/7n5qfsl5uyi1obt/ignite-logs.zip?dl=0 >>>>>>>>>>>>> >>>>>>>>>>>>> I have attached the GC logs and the ignite logs. From what I >>>>>>>>>>>>> see from gc.logs I don't see big pauses. I could be wrong. >>>>>>>>>>>>> >>>>>>>>>>>>> The machine is 16GB and I have the configs here: >>>>>>>>>>>>> https://www.dropbox.com/s/hkv38s3vce5a4sk/ignite-config.xml?dl=0 >>>>>>>>>>>>> >>>>>>>>>>>>> Here are the JVM settings... >>>>>>>>>>>>> >>>>>>>>>>>>> if [ -z "$JVM_OPTS" ] ; then >>>>>>>>>>>>> JVM_OPTS="-Xms2g -Xmx2g -server -XX:MaxMetaspaceSize=256m" >>>>>>>>>>>>> fi >>>>>>>>>>>>> >>>>>>>>>>>>> JVM_OPTS="$JVM_OPTS -XX:+UseG1GC -verbose:gc >>>>>>>>>>>>> -XX:+PrintGCDetails -Xloggc:/var/log/apache-ignite/gc.log" >>>>>>>>>>>>> >>>>>>>>>>>>> JVM_OPTS="${JVM_OPTS} -Xss16m" >>>>>>>>>>>>> >>>>>>>>>>>>
