[
https://issues.apache.org/jira/browse/IGNITE-5759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109422#comment-16109422
]
Dmitriy Pavlov commented on IGNITE-5759:
----------------------------------------
[~agoncharuk], latest test failure from TC with
- additional logging code
- and test stopping in case of failure
was with failed to wait part map update, see
https://issues.apache.org/jira/secure/attachment/12879880/threadDumpFromLogs.log
{noformat}
Caused by: class org.apache.ignite.IgniteException: Timeout of waiting for
topology map update
[igniteInstanceName=distributed.GridCachePartitionEvictionDuringReadThroughSelfTest4,
cache=config, cacheId=-1354792126, topVer=AffinityTopologyVersion [topVer=5,
minorTopVer=1], p=16, readVer=AffinityTopologyVersion [topVer=5,
minorTopVer=1], locNode=TcpDiscoveryNode
[id=e790c9e4-49b3-4b78-8ab3-585d8ce00004, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:47504], discPort=47504, order=5, intOrder=5,
lastExchangeTime=1501604069038, loc=true, ver=2.2.0#19700101-sha1:00000000,
isClient=false]]
at
org.apache.ignite.testframework.junits.common.GridCommonAbstractTest.awaitPartitionMapExchange(GridCommonAbstractTest.java:700)
at
org.apache.ignite.testframework.junits.common.GridCommonAbstractTest.awaitPartitionMapExchange(GridCommonAbstractTest.java:537)
at
org.apache.ignite.testframework.junits.common.GridCommonAbstractTest.awaitPartitionMapExchange(GridCommonAbstractTest.java:521)
at
org.apache.ignite.internal.processors.cache.distributed.GridCachePartitionEvictionDuringReadThroughSelfTest.access$300(GridCachePartitionEvictionDuringReadThroughSelfTest.java:47)
at
org.apache.ignite.internal.processors.cache.distributed.GridCachePartitionEvictionDuringReadThroughSelfTest$2.call(GridCachePartitionEvictionDuringReadThroughSelfTest.java:122)
at
org.apache.ignite.internal.processors.cache.distributed.GridCachePartitionEvictionDuringReadThroughSelfTest$2.call(GridCachePartitionEvictionDuringReadThroughSelfTest.java:117)
at
org.apache.ignite.testframework.GridTestUtils$7.run(GridTestUtils.java:914)
at
org.apache.ignite.testframework.GridTestUtils$9.call(GridTestUtils.java:1210)
at
org.apache.ignite.testframework.GridTestThread.run(GridTestThread.java:86)
{noformat}
All exchange workers were waiting for next exchange future in queue:
LinkedBlockingDeque.poll() and it seems exchange was finished.
> IgniteCache5 suite timed out by
> GridCachePartitionEvictionDuringReadThroughSelfTest.testPartitionRent
> -----------------------------------------------------------------------------------------------------
>
> Key: IGNITE-5759
> URL: https://issues.apache.org/jira/browse/IGNITE-5759
> Project: Ignite
> Issue Type: Bug
> Reporter: Dmitriy Pavlov
> Assignee: Dmitriy Pavlov
> Priority: Critical
> Labels: MakeTeamcityGreenAgain, test-fail
> Fix For: 2.2
>
> Attachments: threadDumpFromLogs.log
>
>
> http://ci.ignite.apache.org/viewLog.html?buildId=727951&buildTypeId=Ignite20Tests_IgniteCache5
> There is no 'Test has been timed out' message in logs.
> Last 'Starting test:' message was
> GridCachePartitionEvictionDuringReadThroughSelfTest#testPartitionRent
> Latest exception from working test was as follows;
> {noformat}
> [23:19:11]W: [org.apache.ignite:ignite-core] [2017-07-14
> 20:19:11,392][ERROR][tcp-comm-worker-#8980%distributed.GridCachePartitionEvictionDuringReadThroughSelfTest4%][TcpCommunicationSpi]
> TcpCommunicationSpi failed to establish connection to node, node will be
> dropped from cluster [rmtNode=TcpDiscoveryNode
> [id=a93fce57-6b2d-4947-8c23-8a677b900003, addrs=[127.0.0.1],
> sockAddrs=[/127.0.0.1:47503], discPort=47503, order=4, intOrder=4,
> lastExchangeTime=1500063443391, loc=false, ver=2.1.0#19700101-sha1:00000000,
> isClient=false]]
> [23:19:11]W: [org.apache.ignite:ignite-core] class
> org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node
> still alive?). Make sure that each ComputeTask and cache Transaction has a
> timeout set in order to prevent parties from waiting forever in case of
> network issues [nodeId=a93fce57-6b2d-4947-8c23-8a677b900003,
> addrs=[/127.0.0.1:45273]]
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3173)
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2757)
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2649)
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.access$5900(TcpCommunicationSpi.java:245)
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.processDisconnect(TcpCommunicationSpi.java:4065)
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:3891)
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
> [23:19:11]W: [org.apache.ignite:ignite-core] Suppressed:
> class org.apache.ignite.IgniteCheckedException: Failed to connect to address
> [addr=/127.0.0.1:45273, err=Connection refused]
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3178)
> [23:19:11]W: [org.apache.ignite:ignite-core] ... 6
> more
> [23:19:11]W: [org.apache.ignite:ignite-core] Caused by:
> java.net.ConnectException: Connection refused
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:117)
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3024)
> [23:19:11]W: [org.apache.ignite:ignite-core] ... 6
> more
> {noformat}
> and then
> {noformat}
> [23:19:11]W: [org.apache.ignite:ignite-core] [2017-07-14
> 20:19:11,895][WARN ][main][root] Interrupting threads started so far: 5
> [23:19:11] : [Step 4/5] [2017-07-14 20:19:11,895][INFO ][main][root] >>>
> Stopping test class: GridCachePartitionEvictionDuringReadThroughSelfTest <<<
> [23:19:11]W: [org.apache.ignite:ignite-core] [20:19:11] (err)
> Failed to execute compound future reducer: GridCompoundFuture
> [rdc=LongSumReducer [sum=0], initFlag=1, lsnrCalls=1, done=false,
> cancelled=false, err=null, futs=[true, true]]class
> org.apache.ignite.IgniteCheckedException: null
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7246)
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.util.future.GridFutureAdapter.resolve(GridFutureAdapter.java:258)
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:170)
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:139)
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:92)
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:45)
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:382)
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:346)
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:334)
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:494)
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:473)
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:461)
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.testframework.GridTestUtils$7.run(GridTestUtils.java:919)
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.testframework.GridTestUtils$9.call(GridTestUtils.java:1210)
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.testframework.GridTestThread.run(GridTestThread.java:86)
> [23:19:11]W: [org.apache.ignite:ignite-core] Caused by:
> java.lang.InterruptedException
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> java.lang.Object.wait(Native Method)
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> java.lang.Thread.join(Thread.java:1281)
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> java.lang.Thread.join(Thread.java:1355)
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.testframework.GridTestUtils.runMultiThreaded(GridTestUtils.java:806)
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.testframework.GridTestUtils$4.call(GridTestUtils.java:746)
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.testframework.GridTestUtils$4.call(GridTestUtils.java:744)
> [23:19:11]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.testframework.GridTestUtils$7.run(GridTestUtils.java:914)
> [23:19:11]W: [org.apache.ignite:ignite-core] ... 2 more
> {noformat}
> Then at 0200+ test was considered timed out
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
