Hello! Can you collect thread dumps from all nodes in the cluster, share those with us?
Regards, -- Ilya Kasnacheev чт, 30 мая 2019 г. в 00:31, Jay Fernandez <[email protected]>: > This did stop the error from being logged. However, when I start the > loadCaches program, nothing is logged and it seems to just hang. The > ignite logs show that a client connected but nothing after that. In > addition, the web console heap size monitoring jumps up right away and then > stops monitoring immediately after. > > On Tue, May 28, 2019 at 9:42 AM Jay Fernandez <[email protected]> > wrote: > >> Thanks for the reply Denis. Is the correct way to disable the checker? >> >> <property name="systemWorkerBlockedTimeout" value="#{-1}"/> >> >> On Fri, May 24, 2019 at 5:59 PM Denis Magda <[email protected]> wrote: >> >>> Hi Jay, >>> >>> Could you please try to disable the "crtical workers checker"? >>> >>> https://apacheignite.readme.io/docs/critical-failures-handling#section-critical-workers-health-check >>> >>> It will be disabled by default in Ignite 2.7.5 since requires more >>> automation and tuning. >>> >>> Let us know if it doesn't work. >>> >>> - >>> Denis >>> >>> >>> On Fri, May 24, 2019 at 9:57 AM jay.fernandez <[email protected]> >>> wrote: >>> >>>> Hello, very new to Ignite and excited about using the application. I >>>> have >>>> installed one Apache Ignite 2.7 node on a GCP VM. I have the web agent >>>> running locally and I am using Gridgain's Web Console. I am getting an >>>> error trying to run the LoadCaches java application that the Gridgain >>>> Web >>>> Console generated based on my MySQL database. >>>> >>>> Logs from Ignite Server: >>>> >>>> May 24 16:54:50 gdw-mysql57 service.sh[26542]: [16:54:50] Ignite node >>>> started OK (id=1b7f4add) >>>> May 24 16:54:50 gdw-mysql57 service.sh[26542]: [16:54:50] Topology >>>> snapshot >>>> [ver=1, locNode=1b7f4add, servers=1, clients=0, state=ACTIVE, CPUs=2, >>>> offheap=1.5GB, heap=1.0GB] >>>> May 24 16:55:03 gdw-mysql57 service.sh[26542]: [16:55:03] Topology >>>> snapshot >>>> [ver=2, locNode=1b7f4add, servers=1, clients=1, state=ACTIVE, CPUs=10, >>>> offheap=1.5GB, heap=8.1GB] >>>> >>>> >>>> Error from the Java project below, any help would be appreciated. >>>> >>>> May 24, 2019 12:53:02 PM java.util.logging.LogManager$RootLogger log >>>> WARNING: Failed to resolve default logging config file: >>>> config/java.util.logging.properties >>>> [12:53:02] __________ ________________ >>>> [12:53:02] / _/ ___/ |/ / _/_ __/ __/ >>>> [12:53:02] _/ // (7 7 // / / / / _/ >>>> [12:53:02] /___/\___/_/|_/___/ /_/ /___/ >>>> [12:53:02] >>>> [12:53:02] ver. 2.7.0#20181130-sha1:256ae401 >>>> [12:53:02] 2018 Copyright(C) Apache Software Foundation >>>> [12:53:02] >>>> [12:53:02] Ignite documentation: http://ignite.apache.org >>>> [12:53:02] >>>> [12:53:02] Quiet mode. >>>> [12:53:02] ^-- Logging by 'JavaLogger [quiet=true, config=null]' >>>> [12:53:02] ^-- To see **FULL** console log here add >>>> -DIGNITE_QUIET=false >>>> or "-v" to ignite.{sh|bat} >>>> [12:53:02] >>>> [12:53:02] OS: Windows 10 10.0 amd64 >>>> [12:53:02] VM information: Java(TM) SE Runtime Environment 1.8.0_201-b09 >>>> Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.201-b09 >>>> [12:53:02] Please set system property '-Djava.net.preferIPv4Stack=true' >>>> to >>>> avoid possible problems in mixed environments. >>>> [12:53:02] Initial heap size is 510MB (should be no less than 512MB, use >>>> -Xms512m -Xmx512m). >>>> [12:53:02] Configured plugins: >>>> [12:53:02] ^-- None >>>> [12:53:02] >>>> [12:53:02] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler >>>> [tryStop=false, timeout=0, super=AbstractFailureHandler >>>> [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]]] >>>> [12:53:03] Message queue limit is set to 0 which may lead to potential >>>> OOMEs >>>> when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to >>>> message queues growth on sender and receiver sides. >>>> [12:53:03] Security status [authentication=off, tls/ssl=off] >>>> [12:53:03] REST protocols do not start on client node. To start the >>>> protocols on client node set '-DIGNITE_REST_START_ON_CLIENT=true' system >>>> property. >>>> log4j:WARN No appenders could be found for logger >>>> (org.springframework.beans.factory.support.DefaultListableBeanFactory). >>>> log4j:WARN Please initialize the log4j system properly. >>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig >>>> for >>>> more info. >>>> May 24, 2019 12:53:18 PM org.apache.ignite.logger.java.JavaLogger error >>>> SEVERE: Blocked system-critical thread has been detected. This can lead >>>> to >>>> cluster-wide undefined behaviour [threadName=partition-exchanger, >>>> blockedFor=12s] >>>> May 24, 2019 12:53:18 PM java.util.logging.LogManager$RootLogger log >>>> SEVERE: Critical system error detected. Will be handled accordingly to >>>> configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, >>>> timeout=0, super=AbstractFailureHandler >>>> [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], >>>> failureCtx=FailureContext >>>> [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker >>>> [name=partition-exchanger, igniteInstanceName=ImportedCluster, >>>> finished=false, heartbeatTs=1558716785615]]] >>>> class org.apache.ignite.IgniteException: GridWorker >>>> [name=partition-exchanger, igniteInstanceName=ImportedCluster, >>>> finished=false, heartbeatTs=1558716785615] >>>> at >>>> >>>> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831) >>>> at >>>> >>>> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826) >>>> at >>>> >>>> org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233) >>>> at >>>> >>>> org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297) >>>> at >>>> >>>> org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:221) >>>> at >>>> >>>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) >>>> at java.lang.Thread.run(Thread.java:748) >>>> >>>> >>>> >>>> -- >>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >>>> >>>
