Hi,
I'm unable to download the logs provide by you. Get '404' error all the
time.
Please upload them to the other storage and in addition attach configs
that are used by your servers.
--
Denis
On 12/15/2015 10:16 AM, sunyson wrote:
Hi, I have download 1.5.0-b1 and make it happened again. For I want to
use Ignite in our production environment, I want to make sure whether
it's caused by not correct configuration. " " The follow is the
exception logs from node 10.1.XX.XX which left the grid.
XX_[hadoop-s1]_logs.txt
<http://apache-ignite-users.70518.x6.nabble.com/file/n2219/XX_%5Bhadoop-s1%5D_logs.txt>
The follow is the logs from the other 2 nodes. gridlogs.rar
<http://apache-ignite-users.70518.x6.nabble.com/file/n2219/gridlogs.rar>
2015-12-14T16:25:34+0000 DEBUG
[grid-time-coordinator-#62%SCHEDULE_NEW%]
org.apache.ignite.internal.processors.clock.GridClockSyncProcessor -
Collected time sync results: {9e22645c-52ce-404f-bd42-b82e98a1b593=0}
2015-12-14T16:25:34+0000 DEBUG [ignite-#16%pub-SCHEDULE_NEW%]
org.apache.ignite.spi.failover.always.AlwaysFailoverSpi - Received
failed job result: GridJobResultImpl [job=C2 [],
sib=GridJobSiblingImpl
[sesId=47ec4de9151-0d8281ae-7bfd-4a19-bf6f-35128bef7c8c,
jobId=57ec4de9151-148fc364-78cd-475f-81bd-b3188a47f5f6,
nodeId=148fc364-78cd-475f-81bd-b3188a47f5f6, isJobDone=false],
jobCtx=GridJobContextImpl
[jobId=57ec4de9151-148fc364-78cd-475f-81bd-b3188a47f5f6,
timeoutObj=null, attrs={}], node=TcpDiscoveryNode
[id=148fc364-78cd-475f-81bd-b3188a47f5f6, addrs=[0:0:0:0:0:0:0:1%1,
10.1.XX.XX, 127.0.0.1], sockAddrs=[hadoop-s1/10.1.XX.XX:47500,
/0:0:0:0:0:0:0:1%1:47500, /10.1.XX.XX:47500, /127.0.0.1:47500],
discPort=47500, order=1, intOrder=1, lastExchangeTime=1450068071263,
loc=false, ver=1.5.0#20151201-sha1:062d440c, isClient=false], ex=class
org.apache.ignite.cluster.ClusterTopologyException: Node has left
grid: 148fc364-78cd-475f-81bd-b3188a47f5f6, hasRes=true,
isCancelled=false, isOccupied=true] 2015-12-14T16:25:34+0000 WARN
[ignite-#16%pub-SCHEDULE_NEW%]
org.apache.ignite.spi.failover.always.AlwaysFailoverSpi - Job failover
failed because number of maximum failover attempts for affinity call
is exceeded [failedJob=C2 [], maxFailoverAttempts=1]
2015-12-14T16:25:34+0000 DEBUG [ignite-#16%pub-SCHEDULE_NEW%]
org.apache.ignite.internal.processors.task.GridTaskWorker - Failed to
failover a job to another node (failover SPI returned null) [job=C2
[], node=TcpDiscoveryNode [id=148fc364-78cd-475f-81bd-b3188a47f5f6,
addrs=[0:0:0:0:0:0:0:1%1, 10.1.XX.XX, 127.0.0.1],
sockAddrs=[hadoop-s1/10.1.XX.XX:47500, /0:0:0:0:0:0:0:1%1:47500,
/10.1.XX.XX:47500, /127.0.0.1:47500], discPort=47500, order=1,
intOrder=1, lastExchangeTime=1450068071263, loc=false,
ver=1.5.0#20151201-sha1:062d440c, isClient=false]]
2015-12-14T16:25:34+0000 DEBUG [ignite-#16%pub-SCHEDULE_NEW%]
org.apache.ignite.internal.processors.task.GridTaskWorker - Creating
fake response because node left grid [job=C2 [],
nodeId=148fc364-78cd-475f-81bd-b3188a47f5f6] 2015-12-14T16:25:34+0000
DEBUG [ignite-#16%pub-SCHEDULE_NEW%]
org.apache.ignite.internal.processors.task.GridTaskWorker - Simulating
fake response from left node [res=GridJobExecuteResponse
[nodeId=148fc364-78cd-475f-81bd-b3188a47f5f6,
sesId=eaec4de9151-0d8281ae-7bfd-4a19-bf6f-35128bef7c8c,
jobId=faec4de9151-148fc364-78cd-475f-81bd-b3188a47f5f6, gridEx=null,
isCancelled=false], nodeId=148fc364-78cd-475f-81bd-b3188a47f5f6]
2015-12-14T16:25:34+0000 DEBUG [ignite-#16%pub-SCHEDULE_NEW%]
org.apache.ignite.internal.processors.task.GridTaskWorker - Obtained
job result policy [policy=FAILOVER, ses=GridTaskSessionImpl
[taskName=com.ukelink.demo.schedule.service.ScheduleCompute$2,
dep=LocalDeployment [super=GridDeployment [ts=1450068069718,
depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@35e5ebbf,
clsLdrId=65dcace9151-0d8281ae-7bfd-4a19-bf6f-35128bef7c8c, userVer=0,
loc=true, sampleClsName=java.lang.String, pendingUndeploy=false,
undeployed=false, usage=0]],
taskClsName=com.ukelink.demo.schedule.service.ScheduleCompute$2,
sesId=eaec4de9151-0d8281ae-7bfd-4a19-bf6f-35128bef7c8c,
startTime=1450110329992, endTime=9223372036854775807,
taskNodeId=0d8281ae-7bfd-4a19-bf6f-35128bef7c8c,
clsLdr=sun.misc.Launcher$AppClassLoader@35e5ebbf, closed=false,
cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false,
subjId=0d8281ae-7bfd-4a19-bf6f-35128bef7c8c, mapFut=IgniteFuture
[orig=GridFutureAdapter [resFlag=2, res=null, startTime=1450110329992,
endTime=1450110329992, ignoreInterrupts=false, lsnr=null,
state=DONE]]]] 2015-12-14T16:25:34+0000 DEBUG
[ignite-#16%pub-SCHEDULE_NEW%]
org.apache.ignite.spi.failover.always.AlwaysFailoverSpi - Received
failed job result: GridJobResultImpl [job=C2 [],
sib=GridJobSiblingImpl
[sesId=eaec4de9151-0d8281ae-7bfd-4a19-bf6f-35128bef7c8c,
jobId=faec4de9151-148fc364-78cd-475f-81bd-b3188a47f5f6,
nodeId=148fc364-78cd-475f-81bd-b3188a47f5f6, isJobDone=false],
jobCtx=GridJobContextImpl
[jobId=faec4de9151-148fc364-78cd-475f-81bd-b3188a47f5f6,
timeoutObj=null, attrs={}], node=TcpDiscoveryNode
[id=148fc364-78cd-475f-81bd-b3188a47f5f6, addrs=[0:0:0:0:0:0:0:1%1,
10.1.XX.XX, 127.0.0.1], sockAddrs=[hadoop-s1/10.1.XX.XX:47500,
/0:0:0:0:0:0:0:1%1:47500, /10.1.XX.XX:47500, /127.0.0.1:47500],
discPort=47500, order=1, intOrder=1, lastExchangeTime=1450068071263,
loc=false, ver=1.5.0#20151201-sha1:062d440c, isClient=false], ex=class
org.apache.ignite.cluster.ClusterTopologyException: Node has left
grid: 148fc364-78cd-475f-81bd-b3188a47f5f6, hasRes=true,
isCancelled=false, isOccupied=true] 2015-12-14T16:25:34+0000 DEBUG
[ignite-#1%pub-SCHEDULE_NEW%]
org.apache.ignite.internal.processors.closure.GridClosureProcessor -
Grid runnable started: closure-proc-worker 2015-12-14T16:25:34+0000
WARN [ignite-#16%pub-SCHEDULE_NEW%]
org.apache.ignite.spi.failover.always.AlwaysFailoverSpi - Job failover
failed because number of maximum failover attempts for affinity call
is exceeded [failedJob=C2 [], maxFailoverAttempts=1]
2015-12-14T16:25:34+0000 DEBUG [ignite-#1%pub-SCHEDULE_NEW%]
org.apache.ignite.internal.processors.closure.GridClosureProcessor -
Grid runnable finished normally: closure-proc-worker
2015-12-14T16:25:34+0000 DEBUG [ignite-#16%pub-SCHEDULE_NEW%]
org.apache.ignite.internal.processors.task.GridTaskWorker - Failed to
failover a job to another node (failover SPI returned null) [job=C2
[], node=TcpDiscoveryNode [id=148fc364-78cd-475f-81bd-b3188a47f5f6,
addrs=[0:0:0:0:0:0:0:1%1, 10.1.XX.XX, 127.0.0.1],
sockAddrs=[hadoop-s1/10.1.XX.XX:47500, /0:0:0:0:0:0:0:1%1:47500,
/10.1.XX.XX:47500, /127.0.0.1:47500], discPort=47500, order=1,
intOrder=1, lastExchangeTime=1450068071263, loc=false,
ver=1.5.0#20151201-sha1:062d440c, isClient=false]]
2015-12-14T16:25:34+0000 DEBUG [ignite-#16%pub-SCHEDULE_NEW%]
org.apache.ignite.internal.processors.closure.GridClosureProcessor -
Grid runnable finished normally: closure-proc-worker
2015-12-14T16:25:34+0000 DEBUG [tcp-disco-msg-worker-#2%SCHEDULE_NEW]
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi - Message has been
sent to next node [msg=TcpDiscoveryDiscardMessage
[msgId=80fc4de9151-0d8281ae-7bfd-4a19-bf6f-35128bef7c8c,
customMsgDiscard=true, super=TcpDiscoveryAbstractMessage
[sndNodeId=null, id=a1fc4de9151-0d8281ae-7bfd-4a19-bf6f-35128bef7c8c,
verifierNodeId=0d8281ae-7bfd-4a19-bf6f-35128bef7c8c, topVer=0,
pendingIdx=0, failedNodes=null, isClient=false]],
next=9e22645c-52ce-404f-bd42-b82e98a1b593, res=1]
2015-12-14T16:25:34+0000 DEBUG [tcp-disco-msg-worker-#2%SCHEDULE_NEW]
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi - Processing
message [cls=TcpDiscoveryDiscardMessage,
id=01fc4de9151-0d8281ae-7bfd-4a19-bf6f-35128bef7c8c]
2015-12-14T16:25:34+0000 DEBUG [tcp-disco-msg-worker-#2%SCHEDULE_NEW]
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi - Processing
message [cls=TcpDiscoveryDiscardMessage,
id=31fc4de9151-0d8281ae-7bfd-4a19-bf6f-35128bef7c8c]
2015-12-14T16:25:34+0000 DEBUG [tcp-disco-sock-reader-#6%SCHEDULE_NEW]
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi - Message has been
received: TcpDiscoveryDiscardMessage
[msgId=80fc4de9151-0d8281ae-7bfd-4a19-bf6f-35128bef7c8c,
customMsgDiscard=true, super=TcpDiscoveryAbstractMessage
[sndNodeId=9e22645c-52ce-404f-bd42-b82e98a1b593,
id=a1fc4de9151-0d8281ae-7bfd-4a19-bf6f-35128bef7c8c,
verifierNodeId=0d8281ae-7bfd-4a19-bf6f-35128bef7c8c, topVer=0,
pendingIdx=0, failedNodes=null, isClient=false]]
2015-12-14T16:25:34+0000 DEBUG [tcp-disco-msg-worker-#2%SCHEDULE_NEW]
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi - Processing
message [cls=TcpDiscoveryDiscardMessage,
id=a1fc4de9151-0d8281ae-7bfd-4a19-bf6f-35128bef7c8c]
2015-12-14T16:25:34+0000 DEBUG [tcp-disco-sock-reader-#6%SCHEDULE_NEW]
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi - Message has been
added to queue: TcpDiscoveryDiscardMessage
[msgId=80fc4de9151-0d8281ae-7bfd-4a19-bf6f-35128bef7c8c,
customMsgDiscard=true, super=TcpDiscoveryAbstractMessage
[sndNodeId=9e22645c-52ce-404f-bd42-b82e98a1b593,
id=a1fc4de9151-0d8281ae-7bfd-4a19-bf6f-35128bef7c8c,
verifierNodeId=0d8281ae-7bfd-4a19-bf6f-35128bef7c8c, topVer=0,
pendingIdx=0, failedNodes=null, isClient=false]]
2015-12-14T16:25:34+0000 DEBUG [disco-event-worker-#46%SCHEDULE_NEW%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - Forcing
NIO client close since node has left
[nodeId=148fc364-78cd-475f-81bd-b3188a47f5f6,
client=GridTcpNioCommunicationClient [ses=GridSelectorNioSessionImpl
[selectorIdx=0, queueSize=0, writeBuf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], recovery=GridNioRecoveryDescriptor
[acked=412368, resendCnt=0, rcvCnt=304545, reserved=true,
lastAck=304544, nodeLeft=false, node=TcpDiscoveryNode
[id=148fc364-78cd-475f-81bd-b3188a47f5f6, addrs=[0:0:0:0:0:0:0:1%1,
10.1.XX.XX, 127.0.0.1], sockAddrs=[hadoop-s1/10.1.XX.XX:47500,
/0:0:0:0:0:0:0:1%1:47500, /10.1.XX.XX:47500, /127.0.0.1:47500],
discPort=47500, order=1, intOrder=1, lastExchangeTime=1450068070384,
loc=false, ver=1.5.0#20151201-sha1:062d440c, isClient=false],
connected=true, connectCnt=0, queueLimit=5120],
super=GridNioSessionImpl [locAddr=/10.1.XX.ZZ:47100,
rmtAddr=/10.1.XX.XX:59033, createTime=1450068070495, closeTime=0,
bytesSent=497671887, bytesRcvd=503364967, sndSchedTime=1450110333152,
lastSndTime=1450110333152, lastRcvTime=1450110322496,
readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=org.apache.ignite.internal.util.nio.GridDirectParser@20fa7c47,
directMode=true], GridConnectionBytesVerifyFilter], accepted=true]],
super=GridAbstractCommunicationClient [lastUsed=1450068070505,
reserves=0]]] 2015-12-14T16:25:34+0000 DEBUG
[disco-event-worker-#46%SCHEDULE_NEW%]
org.apache.ignite.internal.managers.communication.GridIoManager -
Removed message set due to node leaving grid:
GridCommunicationMessageSet
[nodeId=148fc364-78cd-475f-81bd-b3188a47f5f6, endTime=1450110336341,
timeoutId=f8dcace9151-c1a9e05a-67c3-4ee0-8cea-7fb5cf78a5a8, topic=T6
[topic=TOPIC_CACHE, id1=83e8ca36-2305-3266-8e65-1463be879baa, id2=0],
plc=6, msgs=[], reserved=false, timeout=10000, skipOnTimeout=false,
lastTs=1450068072063] 2015-12-14T16:25:34+0000 DEBUG
[disco-event-worker-#46%SCHEDULE_NEW%]
org.apache.ignite.internal.managers.communication.GridIoManager -
Removed messages from discovery startup delay list (sender node left
topology): null 2015-12-14T16:25:34+0000 DEBUG
[disco-event-worker-#46%SCHEDULE_NEW%]
org.apache.ignite.internal.processors.cache.GridCacheMvccManager -
Processing node left [nodeId=148fc364-78cd-475f-81bd-b3188a47f5f6]
2015-12-14T16:25:34+0000 DEBUG [disco-event-worker-#46%SCHEDULE_NEW%]
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicUpdateFuture
- Processing node leave event [fut=GridDhtAtomicUpdateFuture
[futVer=GridCacheVersion [topVer=61547991, nodeOrderDrId=2,
globalTime=1450110331961, order=1450069365377],
writeVer=GridCacheVersion [topVer=61547991, nodeOrderDrId=2,
globalTime=1450110331961, order=1450069365376],
forceTransformBackups=false,
mappings={148fc364-78cd-475f-81bd-b3188a47f5f6=GridDhtAtomicUpdateRequest
[nodeId=148fc364-78cd-475f-81bd-b3188a47f5f6, futVer=GridCacheVersion
[topVer=61547991, nodeOrderDrId=2, globalTime=1450110331961,
order=1450069365377], writeVer=GridCacheVersion [topVer=61547991,
nodeOrderDrId=2, globalTime=1450110331961, order=1450069365376],
topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0],
keys=[KeyCacheObjectImpl [val=AffinityKey [key=119522,
affKey=Groupid8], hasValBytes=true]], vals=[CacheObjectImpl
[val=ID:119522 VsimID:867743 FlowByte:1014838 Amount:67.0
BuyTime:2015-12-14T16:25:31.962Z Guid:Groupid8 Remark:Just a
test-3370460822845607988!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!,
hasValBytes=true]], prevVals=null, ttls=null,
conflictExpireTimes=null, nearTtls=null, nearExpireTimes=null,
syncMode=PRIMARY_SYNC, nearKeys=null, nearVals=null,
forceTransformBackups=false,
subjId=0d8281ae-7bfd-4a19-bf6f-35128bef7c8c, taskNameHash=0,
updateCntrs=GridLongList [idx=1, arr=[152308]], onRes=false,
keepBinary=false, super=GridCacheMessage [msgId=397580, depInfo=null,
err=null, skipPrepare=false, cacheId=-1699403857,
cacheId=-1699403857]]}, updateReq=GridNearAtomicUpdateRequest
[nodeId=0d8281ae-7bfd-4a19-bf6f-35128bef7c8c, futVer=GridCacheVersion
[topVer=61547991, nodeOrderDrId=2, globalTime=1450110331961,
order=1450069365375], fastMap=false, updateVer=null,
topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0],
topLocked=false, syncMode=PRIMARY_SYNC, op=UPDATE,
keys=[UserKeyCacheObjectImpl [val=AffinityKey [key=119522,
affKey=Groupid8], hasValBytes=false]], conflictTtls=null,
conflictExpireTimes=null, retval=false, expiryPlc=null,
hasPrimary=true, subjId=0d8281ae-7bfd-4a19-bf6f-35128bef7c8c,
taskNameHash=0, skipStore=false, clientReq=false, keepBinary=false,
filter=[CacheEntryPredicateContainsValue [val=UserCacheObjectImpl
[val=ID:119522 VsimID:867743 FlowByte:1014838 Amount:67.0
BuyTime:2015-12-14T15:57:21.865Z Guid:Groupid8 Remark:Just a
test-3370460822845607988!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!,
hasValBytes=false]]], parent=GridCacheMessage [msgId=-1, depInfo=null,
err=null, skipPrepare=false, cacheId=-1699403857,
cacheId=-1699403857]], updateRes=GridNearAtomicUpdateResponse
[nodeId=0d8281ae-7bfd-4a19-bf6f-35128bef7c8c, futVer=GridCacheVersion
[topVer=61547991, nodeOrderDrId=2, globalTime=1450110331961,
order=1450069365375], err=null, ret=GridCacheReturn [v=null,
cacheObj=null, success=true, invokeRes=false, loc=true, cacheId=0],
failedKeys=null, remapKeys=null, nearVals=null, nearVer=null,
nearTtls=null, nearExpireTimes=null, super=parent],
waitForExchange=true, resCnt=0],
nodeId=148fc364-78cd-475f-81bd-b3188a47f5f6]
------------------------------------------------------------------------
View this message in context: Re: Grid stopped with
IgniteCheckedException: Failed to deserialize object with given class
loader: sun.misc.Launcher$AppClassLoader
<http://apache-ignite-users.70518.x6.nabble.com/Grid-stopped-with-IgniteCheckedException-Failed-to-deserialize-object-with-given-class-loader-sun-mir-tp2123p2219.html>
Sent from the Apache Ignite Users mailing list archive
<http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com.
