Hi,
we have a null pointer exception in one of our servers. No major activity was
happening when the server crashed
Please check the logs and see if there is any workaround we can use.
We are in a production environment.
regards
mahesh
[05:42:42,194][SEVERE][sys-stripe-19-#20][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.lang.NullPointerException]]
java.lang.NullPointerException
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:1064)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:953)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:909)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$200(IgniteTxHandler.java:123)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:217)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:215)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
at org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:745)
[05:42:42,216][INFO][grid-timeout-worker-#43][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=76a9edc3, uptime=14:26:42.572]
^-- H/N/C [hosts=13, nodes=13, CPUs=50]
^-- CPU [cur=4.37%, avg=2.77%, GC=0%]
^-- PageMemory [pages=7420567]
^-- Heap [used=19580MB, free=60.16%, comm=49152MB]
^-- Off-heap [used=29326MB, free=55.46%, comm=65736MB]
^-- sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
^-- default region [used=29326MB, free=55.25%, comm=65536MB]
^-- metastoreMemPlc region [used=0MB, free=99.75%, comm=0MB]
^-- TxLog region [used=0MB, free=100%, comm=100MB]
^-- Ignite persistence [used=42982MB]
^-- sysMemPlc region [used=0MB]
^-- default region [used=42981MB]
^-- metastoreMemPlc region [used=0MB]
^-- TxLog region [used=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=32, qSize=0]
[05:42:42,460][WARNING][sys-stripe-19-#20][CacheDiagnosticManager] Page locks dump:
Thread=[name=checkpoint-runner-#125, id=195], state=WAITING
Locked pages = []
Locked pages log: name=checkpoint-runner-#125 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=checkpoint-runner-#126, id=196], state=WAITING
Locked pages = []
Locked pages log: name=checkpoint-runner-#126 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=checkpoint-runner-#127, id=197], state=WAITING
Locked pages = []
Locked pages log: name=checkpoint-runner-#127 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=checkpoint-runner-#128, id=198], state=WAITING
Locked pages = []
Locked pages log: name=checkpoint-runner-#128 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=client-connector-#513, id=680], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#513 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=client-connector-#515, id=682], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#515 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=client-connector-#517, id=684], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#517 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=client-connector-#518, id=685], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#518 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=client-connector-#519, id=686], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#519 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=client-connector-#520, id=687], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#520 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=client-connector-#521, id=688], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#521 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=client-connector-#523, id=690], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#523 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=client-connector-#524, id=691], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#524 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=client-connector-#525, id=692], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#525 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=client-connector-#526, id=693], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#526 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=client-connector-#612, id=790], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#612 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=client-connector-#614, id=792], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#614 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=client-connector-#616, id=794], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#616 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=client-connector-#617, id=795], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#617 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=client-connector-#618, id=796], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#618 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=db-checkpoint-thread-#124, id=194], state=TIMED_WAITING
Locked pages = []
Locked pages log: name=db-checkpoint-thread-#124 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=dms-writer-thread-#91, id=161], state=WAITING
Locked pages = []
Locked pages log: name=dms-writer-thread-#91 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=exchange-worker-#86, id=155], state=TIMED_WAITING
Locked pages = []
Locked pages log: name=exchange-worker-#86 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=main, id=1], state=WAITING
Locked pages = []
Locked pages log: name=main time=(1596606162196, 2020-08-05 05:42:42.196)
Thread=[name=rebalance-striped-#10911, id=11594], state=TIMED_WAITING
Locked pages = []
Locked pages log: name=rebalance-striped-#10911 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-0-#1, id=20], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-0-#1 time=(1596606162196, 2020-08-05 05:42:42.196)
Thread=[name=sys-stripe-1-#2, id=21], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-1-#2 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-10-#11, id=30], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-10-#11 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-11-#12, id=31], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-11-#12 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-12-#13, id=32], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-12-#13 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-13-#14, id=33], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-13-#14 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-14-#15, id=34], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-14-#15 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-15-#16, id=35], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-15-#16 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-16-#17, id=36], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-16-#17 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-17-#18, id=37], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-17-#18 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-18-#19, id=38], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-18-#19 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-19-#20, id=39], state=RUNNABLE
Locked pages = []
Locked pages log: name=sys-stripe-19-#20 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-2-#3, id=22], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-2-#3 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-20-#21, id=40], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-20-#21 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-21-#22, id=41], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-21-#22 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-22-#23, id=42], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-22-#23 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-23-#24, id=43], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-23-#24 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-24-#25, id=44], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-24-#25 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-25-#26, id=45], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-25-#26 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-26-#27, id=46], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-26-#27 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-27-#28, id=47], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-27-#28 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-28-#29, id=48], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-28-#29 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-29-#30, id=49], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-29-#30 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-3-#4, id=23], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-3-#4 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-30-#31, id=50], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-30-#31 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-31-#32, id=51], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-31-#32 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-4-#5, id=24], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-4-#5 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-5-#6, id=25], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-5-#6 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-6-#7, id=26], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-6-#7 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-7-#8, id=27], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-7-#8 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-8-#9, id=28], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-8-#9 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=sys-stripe-9-#10, id=29], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-9-#10 time=(1596606162197, 2020-08-05 05:42:42.197)
Thread=[name=ttl-cleanup-worker-#69, id=120], state=TIMED_WAITING
Locked pages = []
Locked pages log: name=ttl-cleanup-worker-#69 time=(1596606162197, 2020-08-05 05:42:42.197)
[05:42:42,461][SEVERE][sys-stripe-19-#20][] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.lang.NullPointerException]]
