[
https://issues.apache.org/jira/browse/IGNITE-16562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yury Gerzhedovich updated IGNITE-16562:
---------------------------------------
Priority: Blocker (was: Major)
> ItSortAggregateTest.initTestData is flaky
> -----------------------------------------
>
> Key: IGNITE-16562
> URL: https://issues.apache.org/jira/browse/IGNITE-16562
> Project: Ignite
> Issue Type: Bug
> Reporter: Mirza Aliev
> Priority: Blocker
> Labels: ignite-3
> Attachments: _Integration_Tests_Module_Runner_1655.log
>
>
> ItSortAggregateTest#initTestData could not be completed because of spamming
> {noformat}
> 2022-02-15 12:34:59:608 +0300
> [ERROR][%ItSortAggregateTest_null_2%Raft-Group-Client-8][RaftGroupServiceImpl]
> Failed to refresh a leader
> [groupId=a6328a6b-7da8-4a8f-a1b5-a27c75c6007b_part_9]
> java.util.concurrent.CompletionException:
> java.util.concurrent.TimeoutException
> at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
> at
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
> at
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
> at
> org.apache.ignite.raft.jraft.rpc.impl.RaftGroupServiceImpl.sendWithRetry(RaftGroupServiceImpl.java:502)
> at
> org.apache.ignite.raft.jraft.rpc.impl.RaftGroupServiceImpl$1.lambda$accept$1(RaftGroupServiceImpl.java:544)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.util.concurrent.TimeoutException
> {noformat}
> Log contains suspicions warnings from scalecube on a very beginning
> {noformat}
> [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15
> 12:34:46:180 +0300 [WARNING][sc-cluster-3345-150][MetadataStore]
> [default:ItSortAggregateTest_null_1:[email protected]:3345][03483828-9532-4700-8fce-538486364fdf-1644917650003]
> Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause:
> java.util.concurrent.TimeoutException: Did not observe any item or terminal
> signal within 1000ms in 'source(MonoDefer)' (and no fallback has been
> configured)
> [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15
> 12:34:46:180 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol]
> [default:ItSortAggregateTest_null_1:[email protected]:3345][updateMembership][SYNC]
> Skipping to add/update member: {m:
> default:ItSortAggregateTest_null_2:[email protected]:3346,
> s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause:
> java.util.concurrent.TimeoutException: Did not observe any item or terminal
> signal within 1000ms in 'source(MonoDefer)' (and no fallback has been
> configured))
> [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15
> 12:34:46:180 +0300 [WARNING][sc-cluster-3345-150][MetadataStore]
> [default:ItSortAggregateTest_null_1:[email protected]:3345][03483828-9532-4700-8fce-538486364fdf-1644917650004]
> Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause:
> java.util.concurrent.TimeoutException: Did not observe any item or terminal
> signal within 1000ms in 'source(MonoDefer)' (and no fallback has been
> configured)
> [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15
> 12:34:46:180 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol]
> [default:ItSortAggregateTest_null_1:[email protected]:3345][updateMembership][SYNC]
> Skipping to add/update member: {m:
> default:ItSortAggregateTest_null_2:[email protected]:3346,
> s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause:
> java.util.concurrent.TimeoutException: Did not observe any item or terminal
> signal within 1000ms in 'source(MonoDefer)' (and no fallback has been
> configured))
> [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15
> 12:34:46:181 +0300 [WARNING][sc-cluster-3345-150][MetadataStore]
> [default:ItSortAggregateTest_null_1:[email protected]:3345][03483828-9532-4700-8fce-538486364fdf-1644917650005]
> Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause:
> java.util.concurrent.TimeoutException: Did not observe any item or terminal
> signal within 1000ms in 'source(MonoDefer)' (and no fallback has been
> configured)
> [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15
> 12:34:46:181 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol]
> [default:ItSortAggregateTest_null_1:[email protected]:3345][updateMembership][SYNC]
> Skipping to add/update member: {m:
> default:ItSortAggregateTest_null_2:[email protected]:3346,
> s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause:
> java.util.concurrent.TimeoutException: Did not observe any item or terminal
> signal within 1000ms in 'source(MonoDefer)' (and no fallback has been
> configured))
> [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15
> 12:34:46:181 +0300 [WARNING][sc-cluster-3345-150][MetadataStore]
> [default:ItSortAggregateTest_null_1:[email protected]:3345][03483828-9532-4700-8fce-538486364fdf-1644917650006]
> Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause:
> java.util.concurrent.TimeoutException: Did not observe any item or terminal
> signal within 1000ms in 'source(MonoDefer)' (and no fallback has been
> configured)
> [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15
> 12:34:46:181 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol]
> [default:ItSortAggregateTest_null_1:[email protected]:3345][updateMembership][SYNC]
> Skipping to add/update member: {m:
> default:ItSortAggregateTest_null_2:[email protected]:3346,
> s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause:
> java.util.concurrent.TimeoutException: Did not observe any item or terminal
> signal within 1000ms in 'source(MonoDefer)' (and no fallback has been
> configured))
> [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15
> 12:34:46:181 +0300 [WARNING][sc-cluster-3345-150][MetadataStore]
> [default:ItSortAggregateTest_null_1:[email protected]:3345][03483828-9532-4700-8fce-538486364fdf-1644917650007]
> Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause:
> java.util.concurrent.TimeoutException: Did not observe any item or terminal
> signal within 1000ms in 'source(MonoDefer)' (and no fallback has been
> configured)
> [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15
> 12:34:46:181 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol]
> [default:ItSortAggregateTest_null_1:[email protected]:3345][updateMembership][SYNC]
> Skipping to add/update member: {m:
> default:ItSortAggregateTest_null_2:[email protected]:3346,
> s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause:
> java.util.concurrent.TimeoutException: Did not observe any item or terminal
> signal within 1000ms in 'source(MonoDefer)' (and no fallback has been
> configured))
> [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15
> 12:34:46:181 +0300 [WARNING][sc-cluster-3345-150][MetadataStore]
> [default:ItSortAggregateTest_null_1:[email protected]:3345][03483828-9532-4700-8fce-538486364fdf-1644917650008]
> Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause:
> java.util.concurrent.TimeoutException: Did not observe any item or terminal
> signal within 1000ms in 'source(MonoDefer)' (and no fallback has been
> configured)
> [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15
> 12:34:46:181 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol]
> [default:ItSortAggregateTest_null_1:[email protected]:3345][updateMembership][SYNC]
> Skipping to add/update member: {m:
> default:ItSortAggregateTest_null_2:[email protected]:3346,
> s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause:
> java.util.concurrent.TimeoutException: Did not observe any item or terminal
> signal within 1000ms in 'source(MonoDefer)' (and no fallback has been
> configured))
> [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15
> 12:34:46:182 +0300 [WARNING][sc-cluster-3345-150][MetadataStore]
> [default:ItSortAggregateTest_null_1:[email protected]:3345][03483828-9532-4700-8fce-538486364fdf-1644917650009]
> Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause:
> java.util.concurrent.TimeoutException: Did not observe any item or terminal
> signal within 1000ms in 'source(MonoDefer)' (and no fallback has been
> configured)
> [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15
> 12:34:46:182 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol]
> [default:ItSortAggregateTest_null_1:[email protected]:3345][updateMembership][SYNC]
> Skipping to add/update member: {m:
> default:ItSortAggregateTest_null_2:[email protected]:3346,
> s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause:
> java.util.concurrent.TimeoutException: Did not observe any item or terminal
> signal within 1000ms in 'source(MonoDefer)' (and no fallback has been
> configured))
> [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15
> 12:34:46:182 +0300 [WARNING][sc-cluster-3345-150][MetadataStore]
> [default:ItSortAggregateTest_null_1:[email protected]:3345][03483828-9532-4700-8fce-538486364fdf-1644917650010]
> Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause:
> java.util.concurrent.TimeoutException: Did not observe any item or terminal
> signal within 1000ms in 'source(MonoDefer)' (and no fallback has been
> configured)
> [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15
> 12:34:46:182 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol]
> [default:ItSortAggregateTest_null_1:[email protected]:3345][updateMembership][SYNC]
> Skipping to add/update member: {m:
> default:ItSortAggregateTest_null_2:[email protected]:3346,
> s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause:
> java.util.concurrent.TimeoutException: Did not observe any item or terminal
> signal within 1000ms in 'source(MonoDefer)' (and no fallback has been
> configured))
> [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15
> 12:34:46:182 +0300 [WARNING][sc-cluster-3345-150][MetadataStore]
> [default:ItSortAggregateTest_null_1:[email protected]:3345][03483828-9532-4700-8fce-538486364fdf-1644917650011]
> Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause:
> java.util.concurrent.TimeoutException: Did not observe any item or terminal
> signal within 1000ms in 'source(MonoDefer)' (and no fallback has been
> configured)
> [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15
> 12:34:46:182 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol]
> [default:ItSortAggregateTest_null_1:[email protected]:3345][updateMembership][SYNC]
> Skipping to add/update member: {m:
> default:ItSortAggregateTest_null_2:[email protected]:3346,
> s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause:
> java.util.concurrent.TimeoutException: Did not observe any item or terminal
> signal within 1000ms in 'source(MonoDefer)' (and no fallback has been
> configured))
> [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15
> 12:34:46:182 +0300 [WARNING][sc-cluster-3345-150][MetadataStore]
> [default:ItSortAggregateTest_null_1:[email protected]:3345][03483828-9532-4700-8fce-538486364fdf-1644917650012]
> Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause:
> java.util.concurrent.TimeoutException: Did not observe any item or terminal
> signal within 1000ms in 'source(MonoDefer)' (and no fallback has been
> configured)
> [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15
> 12:34:46:182 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol]
> [default:ItSortAggregateTest_null_1:[email protected]:3345][updateMembership][SYNC]
> Skipping to add/update member: {m:
> default:ItSortAggregateTest_null_2:[email protected]:3346,
> s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause:
> java.util.concurrent.TimeoutException: Did not observe any item or terminal
> signal within 1000ms in 'source(MonoDefer)' (and no fallback has been
> configured))
> [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15
> 12:34:46:183 +0300 [WARNING][sc-cluster-3345-150][MetadataStore]
> [default:ItSortAggregateTest_null_1:[email protected]:3345][03483828-9532-4700-8fce-538486364fdf-1644917650013]
> Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause:
> java.util.concurrent.TimeoutException: Did not observe any item or terminal
> signal within 1000ms in 'source(MonoDefer)' (and no fallback has been
> configured)
> [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15
> 12:34:46:183 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol]
> [default:ItSortAggregateTest_null_1:[email protected]:3345][updateMembership][SYNC]
> Skipping to add/update member: {m:
> default:ItSortAggregateTest_null_2:[email protected]:3346,
> s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause:
> java.util.concurrent.TimeoutException: Did not observe any item or terminal
> signal within 1000ms in 'source(MonoDefer)' (and no fallback has been
> configured))
> [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15
> 12:34:46:183 +0300 [WARNING][sc-cluster-3345-150][MetadataStore]
> [default:ItSortAggregateTest_null_1:[email protected]:3345][03483828-9532-4700-8fce-538486364fdf-1644917650014]
> Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause:
> java.util.concurrent.TimeoutException: Did not observe any item or terminal
> signal within 1000ms in 'source(MonoDefer)' (and no fallback has been
> configured)
> [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15
> 12:34:46:183 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol]
> [default:ItSortAggregateTest_null_1:[email protected]:3345][updateMembership][SYNC]
> Skipping to add/update member: {m:
> default:ItSortAggregateTest_null_2:[email protected]:3346,
> s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause:
> java.util.concurrent.TimeoutException: Did not observe any item or terminal
> signal within 1000ms in 'source(MonoDefer)' (and no fallback has been
> configured))
> [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15
> 12:34:46:183 +0300 [WARNING][sc-cluster-3345-150][MetadataStore]
> [default:ItSortAggregateTest_null_1:[email protected]:3345][03483828-9532-4700-8fce-538486364fdf-1644917650015]
> Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause:
> java.util.concurrent.TimeoutException: Did not observe any item or terminal
> signal within 1000ms in 'source(MonoDefer)' (and no fallback has been
> configured)
> [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15
> 12:34:46:183 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol]
> [default:ItSortAggregateTest_null_1:[email protected]:3345][updateMembership][SYNC]
> Skipping to add/update member: {m:
> default:ItSortAggregateTest_null_2:[email protected]:3346,
> s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause:
> java.util.concurrent.TimeoutException: Did not observe any item or terminal
> signal within 1000ms in 'source(MonoDefer)' (and no fallback has been
> configured))
> [12:34:57]W: [
> {noformat}
> Seems that this might be the reason why some groups cannot elect leaders as
> far as we get {{Failed to refresh a leader}} for some partitions groups, but
> we cannot get leader elected message like {{Node
> <a6328a6b-7da8-4a8f-a1b5-a27c75c6007b_part_4/172.17.0.6:3344> become leader
> of group}} for some partitions
> Note that you have to set replica to 1 for all tables that you create as far
> as there are some limitations on unstable topologies where leader could be
> reelected.
> https://ci.ignite.apache.org/buildConfiguration/ignite3_Test_RunAllTests/6417876?logFilter=debug
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
