[ https://issues.apache.org/jira/browse/IGNITE-16562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yury Gerzhedovich resolved IGNITE-16562. ---------------------------------------- Resolution: Cannot Reproduce > ItSortAggregateTest.initTestData is flaky > ----------------------------------------- > > Key: IGNITE-16562 > URL: https://issues.apache.org/jira/browse/IGNITE-16562 > Project: Ignite > Issue Type: Bug > Reporter: Mirza Aliev > Assignee: Semyon Danilov > Priority: Blocker > Labels: ignite-3 > Attachments: _Integration_Tests_Module_Runner_1655.log > > > ItSortAggregateTest#initTestData could not be completed because of spamming > {noformat} > 2022-02-15 12:34:59:608 +0300 > [ERROR][%ItSortAggregateTest_null_2%Raft-Group-Client-8][RaftGroupServiceImpl] > Failed to refresh a leader > [groupId=a6328a6b-7da8-4a8f-a1b5-a27c75c6007b_part_9] > java.util.concurrent.CompletionException: > java.util.concurrent.TimeoutException > at > java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) > at > java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346) > at > java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632) > at > java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) > at > java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) > at > org.apache.ignite.raft.jraft.rpc.impl.RaftGroupServiceImpl.sendWithRetry(RaftGroupServiceImpl.java:502) > at > org.apache.ignite.raft.jraft.rpc.impl.RaftGroupServiceImpl$1.lambda$accept$1(RaftGroupServiceImpl.java:544) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:834) > Caused by: java.util.concurrent.TimeoutException > {noformat} > Log contains suspicious warnings from scalecube on a very beginning > {noformat} > [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 > 12:34:46:180 +0300 [WARNING][sc-cluster-3345-150][MetadataStore] > [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][03483828-9532-4700-8fce-538486364fdf-1644917650003] > Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause: > java.util.concurrent.TimeoutException: Did not observe any item or terminal > signal within 1000ms in 'source(MonoDefer)' (and no fallback has been > configured) > [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 > 12:34:46:180 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol] > [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][updateMembership][SYNC] > Skipping to add/update member: {m: > default:ItSortAggregateTest_null_2:40801246-c98f-4db1-8acd-bcda982bd343@172.17.0.6:3346, > s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: > java.util.concurrent.TimeoutException: Did not observe any item or terminal > signal within 1000ms in 'source(MonoDefer)' (and no fallback has been > configured)) > [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 > 12:34:46:180 +0300 [WARNING][sc-cluster-3345-150][MetadataStore] > [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][03483828-9532-4700-8fce-538486364fdf-1644917650004] > Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause: > java.util.concurrent.TimeoutException: Did not observe any item or terminal > signal within 1000ms in 'source(MonoDefer)' (and no fallback has been > configured) > [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 > 12:34:46:180 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol] > [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][updateMembership][SYNC] > Skipping to add/update member: {m: > default:ItSortAggregateTest_null_2:40801246-c98f-4db1-8acd-bcda982bd343@172.17.0.6:3346, > s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: > java.util.concurrent.TimeoutException: Did not observe any item or terminal > signal within 1000ms in 'source(MonoDefer)' (and no fallback has been > configured)) > [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 > 12:34:46:181 +0300 [WARNING][sc-cluster-3345-150][MetadataStore] > [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][03483828-9532-4700-8fce-538486364fdf-1644917650005] > Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause: > java.util.concurrent.TimeoutException: Did not observe any item or terminal > signal within 1000ms in 'source(MonoDefer)' (and no fallback has been > configured) > [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 > 12:34:46:181 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol] > [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][updateMembership][SYNC] > Skipping to add/update member: {m: > default:ItSortAggregateTest_null_2:40801246-c98f-4db1-8acd-bcda982bd343@172.17.0.6:3346, > s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: > java.util.concurrent.TimeoutException: Did not observe any item or terminal > signal within 1000ms in 'source(MonoDefer)' (and no fallback has been > configured)) > [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 > 12:34:46:181 +0300 [WARNING][sc-cluster-3345-150][MetadataStore] > [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][03483828-9532-4700-8fce-538486364fdf-1644917650006] > Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause: > java.util.concurrent.TimeoutException: Did not observe any item or terminal > signal within 1000ms in 'source(MonoDefer)' (and no fallback has been > configured) > [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 > 12:34:46:181 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol] > [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][updateMembership][SYNC] > Skipping to add/update member: {m: > default:ItSortAggregateTest_null_2:40801246-c98f-4db1-8acd-bcda982bd343@172.17.0.6:3346, > s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: > java.util.concurrent.TimeoutException: Did not observe any item or terminal > signal within 1000ms in 'source(MonoDefer)' (and no fallback has been > configured)) > [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 > 12:34:46:181 +0300 [WARNING][sc-cluster-3345-150][MetadataStore] > [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][03483828-9532-4700-8fce-538486364fdf-1644917650007] > Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause: > java.util.concurrent.TimeoutException: Did not observe any item or terminal > signal within 1000ms in 'source(MonoDefer)' (and no fallback has been > configured) > [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 > 12:34:46:181 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol] > [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][updateMembership][SYNC] > Skipping to add/update member: {m: > default:ItSortAggregateTest_null_2:40801246-c98f-4db1-8acd-bcda982bd343@172.17.0.6:3346, > s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: > java.util.concurrent.TimeoutException: Did not observe any item or terminal > signal within 1000ms in 'source(MonoDefer)' (and no fallback has been > configured)) > [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 > 12:34:46:181 +0300 [WARNING][sc-cluster-3345-150][MetadataStore] > [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][03483828-9532-4700-8fce-538486364fdf-1644917650008] > Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause: > java.util.concurrent.TimeoutException: Did not observe any item or terminal > signal within 1000ms in 'source(MonoDefer)' (and no fallback has been > configured) > [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 > 12:34:46:181 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol] > [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][updateMembership][SYNC] > Skipping to add/update member: {m: > default:ItSortAggregateTest_null_2:40801246-c98f-4db1-8acd-bcda982bd343@172.17.0.6:3346, > s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: > java.util.concurrent.TimeoutException: Did not observe any item or terminal > signal within 1000ms in 'source(MonoDefer)' (and no fallback has been > configured)) > [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 > 12:34:46:182 +0300 [WARNING][sc-cluster-3345-150][MetadataStore] > [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][03483828-9532-4700-8fce-538486364fdf-1644917650009] > Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause: > java.util.concurrent.TimeoutException: Did not observe any item or terminal > signal within 1000ms in 'source(MonoDefer)' (and no fallback has been > configured) > [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 > 12:34:46:182 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol] > [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][updateMembership][SYNC] > Skipping to add/update member: {m: > default:ItSortAggregateTest_null_2:40801246-c98f-4db1-8acd-bcda982bd343@172.17.0.6:3346, > s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: > java.util.concurrent.TimeoutException: Did not observe any item or terminal > signal within 1000ms in 'source(MonoDefer)' (and no fallback has been > configured)) > [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 > 12:34:46:182 +0300 [WARNING][sc-cluster-3345-150][MetadataStore] > [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][03483828-9532-4700-8fce-538486364fdf-1644917650010] > Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause: > java.util.concurrent.TimeoutException: Did not observe any item or terminal > signal within 1000ms in 'source(MonoDefer)' (and no fallback has been > configured) > [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 > 12:34:46:182 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol] > [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][updateMembership][SYNC] > Skipping to add/update member: {m: > default:ItSortAggregateTest_null_2:40801246-c98f-4db1-8acd-bcda982bd343@172.17.0.6:3346, > s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: > java.util.concurrent.TimeoutException: Did not observe any item or terminal > signal within 1000ms in 'source(MonoDefer)' (and no fallback has been > configured)) > [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 > 12:34:46:182 +0300 [WARNING][sc-cluster-3345-150][MetadataStore] > [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][03483828-9532-4700-8fce-538486364fdf-1644917650011] > Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause: > java.util.concurrent.TimeoutException: Did not observe any item or terminal > signal within 1000ms in 'source(MonoDefer)' (and no fallback has been > configured) > [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 > 12:34:46:182 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol] > [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][updateMembership][SYNC] > Skipping to add/update member: {m: > default:ItSortAggregateTest_null_2:40801246-c98f-4db1-8acd-bcda982bd343@172.17.0.6:3346, > s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: > java.util.concurrent.TimeoutException: Did not observe any item or terminal > signal within 1000ms in 'source(MonoDefer)' (and no fallback has been > configured)) > [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 > 12:34:46:182 +0300 [WARNING][sc-cluster-3345-150][MetadataStore] > [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][03483828-9532-4700-8fce-538486364fdf-1644917650012] > Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause: > java.util.concurrent.TimeoutException: Did not observe any item or terminal > signal within 1000ms in 'source(MonoDefer)' (and no fallback has been > configured) > [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 > 12:34:46:182 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol] > [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][updateMembership][SYNC] > Skipping to add/update member: {m: > default:ItSortAggregateTest_null_2:40801246-c98f-4db1-8acd-bcda982bd343@172.17.0.6:3346, > s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: > java.util.concurrent.TimeoutException: Did not observe any item or terminal > signal within 1000ms in 'source(MonoDefer)' (and no fallback has been > configured)) > [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 > 12:34:46:183 +0300 [WARNING][sc-cluster-3345-150][MetadataStore] > [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][03483828-9532-4700-8fce-538486364fdf-1644917650013] > Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause: > java.util.concurrent.TimeoutException: Did not observe any item or terminal > signal within 1000ms in 'source(MonoDefer)' (and no fallback has been > configured) > [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 > 12:34:46:183 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol] > [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][updateMembership][SYNC] > Skipping to add/update member: {m: > default:ItSortAggregateTest_null_2:40801246-c98f-4db1-8acd-bcda982bd343@172.17.0.6:3346, > s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: > java.util.concurrent.TimeoutException: Did not observe any item or terminal > signal within 1000ms in 'source(MonoDefer)' (and no fallback has been > configured)) > [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 > 12:34:46:183 +0300 [WARNING][sc-cluster-3345-150][MetadataStore] > [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][03483828-9532-4700-8fce-538486364fdf-1644917650014] > Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause: > java.util.concurrent.TimeoutException: Did not observe any item or terminal > signal within 1000ms in 'source(MonoDefer)' (and no fallback has been > configured) > [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 > 12:34:46:183 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol] > [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][updateMembership][SYNC] > Skipping to add/update member: {m: > default:ItSortAggregateTest_null_2:40801246-c98f-4db1-8acd-bcda982bd343@172.17.0.6:3346, > s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: > java.util.concurrent.TimeoutException: Did not observe any item or terminal > signal within 1000ms in 'source(MonoDefer)' (and no fallback has been > configured)) > [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 > 12:34:46:183 +0300 [WARNING][sc-cluster-3345-150][MetadataStore] > [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][03483828-9532-4700-8fce-538486364fdf-1644917650015] > Timeout getting GetMetadataResp from 172.17.0.6:3346 within 1000 ms, cause: > java.util.concurrent.TimeoutException: Did not observe any item or terminal > signal within 1000ms in 'source(MonoDefer)' (and no fallback has been > configured) > [12:34:57]W: [org.apache.ignite:ignite-runner] 2022-02-15 > 12:34:46:183 +0300 [WARNING][sc-cluster-3345-150][MembershipProtocol] > [default:ItSortAggregateTest_null_1:03483828-9532-4700-8fce-538486364fdf@172.17.0.6:3345][updateMembership][SYNC] > Skipping to add/update member: {m: > default:ItSortAggregateTest_null_2:40801246-c98f-4db1-8acd-bcda982bd343@172.17.0.6:3346, > s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: > java.util.concurrent.TimeoutException: Did not observe any item or terminal > signal within 1000ms in 'source(MonoDefer)' (and no fallback has been > configured)) > [12:34:57]W: [ > {noformat} > Seems that this might be the reason why some groups cannot elect leaders as > far as we get {{Failed to refresh a leader}} for some partitions groups, but > we cannot get leader elected message like {{Node > <a6328a6b-7da8-4a8f-a1b5-a27c75c6007b_part_4/172.17.0.6:3344> become leader > of group}} for some partitions > Note that you have to set replica to 1 for all tables that you create as far > as there are some limitations on unstable topologies where leader could be > reelected. > https://ci.ignite.apache.org/buildConfiguration/ignite3_Test_RunAllTests/6417876?logFilter=debug -- This message was sent by Atlassian Jira (v8.20.10#820010)