Igor created IGNITE-22553:
-----------------------------
Summary: Internal Server Error after node replacement in 3 node
cluster
Key: IGNITE-22553
URL: https://issues.apache.org/jira/browse/IGNITE-22553
Project: Ignite
Issue Type: Bug
Components: general, persistence, rest
Affects Versions: 3.0.0-beta2
Environment: 3 nodes cluster (1 CMG node)
Reporter: Igor
Attachments: Server_logs.zip
{color:#de350b}*This issue is replaced by
https://issues.apache.org/jira/browse/IGNITE-22517 from time to time.*{color}
*Steps to reproduce:*
# Create 3 nodes cluster with 1 CMG node (node_0 - CMG,node_1,node_2).
# Create zone with replication equals to amount of nodes (3).
# Create 10 tables inside the zone.
# Insert 100 rows in every table.
# Await all tables*partitions*nodes local state is "HEALTHY"
# Await all tables*partitions*nodes global state is "AVAILABLE"
# Kill non CMG node with kill -9. (kill node_1)
# Create new node and attach it to cluster instead of the killed one (update
lookup configs of alive nodes to allow them find). (node_1_replacement)
# Using REST API await *physical* topology has 3 alive nodes.
# Using REST API await *logical* topology has 3 alive nodes.
# Await all tables*partitions*nodes local state is "HEALTHY" (by connecting to
REST of node_2).
*Expected:*
All partitions become "HEALTHY".
*Actual:*
The exception from REST client:
{code:java}
org.gridgain.ai3tests.core.generated.restapi.invoker.ApiException: Message:
Internal Server ErrorHTTP response code: 500HTTP response body:
{"title":"Internal Server
Error","status":500,"code":"IGN-RECOVERY-3","type":null,"detail":"org.apache.ignite.internal.network.UnresolvableConsistentIdException:
IGN-NETWORK-1 TraceId:1e07b3f1-9714-4cb6-ba12-eed3296fb821 Recipient
consistent ID cannot be resolved:
ClusterFailover3NodesTest_cluster_1","node":null,"traceId":"1e07b3f1-9714-4cb6-ba12-eed3296fb821","invalidParams":null}HTTP
response headers: {connection=[keep-alive], content-length=[384],
content-type=[application/json+problem], date=[Thu, 20 Jun 2024 15:43:07 GMT]}
at
app//org.gridgain.ai3tests.core.generated.restapi.invoker.ApiClient.handleResponse(ApiClient.java:1131)
at
app//org.gridgain.ai3tests.core.generated.restapi.invoker.ApiClient.execute(ApiClient.java:1044)
at
app//org.gridgain.ai3tests.core.generated.restapi.api.RecoveryApi.getLocalPartitionStatesWithHttpInfo(RecoveryApi.java:335)
at
app//org.gridgain.ai3tests.core.generated.restapi.api.RecoveryApi.getLocalPartitionStates(RecoveryApi.java:312)
at
app//org.gridgain.ai3tests.tests.failover.ClusterFailoverTestBase.getLocalPartitionsStatesDistinct(ClusterFailoverTestBase.java:501)
at
app//org.gridgain.ai3tests.tests.failover.ClusterFailoverTestBase.lambda$awaitLocalState$15(ClusterFailoverTestBase.java:489)
at
app//org.gridgain.ai3tests.core.utils.RetryUtils.retryOnAllowedException(RetryUtils.java:61)
at
app//org.gridgain.ai3tests.tests.failover.ClusterFailoverTestBase.awaitLocalState(ClusterFailoverTestBase.java:485)
at
app//org.gridgain.ai3tests.tests.failover.ClusterFailoverTestBase.awaitAllPartitionsHealthyAndAvailable(ClusterFailoverTestBase.java:461)
at
app//org.gridgain.ai3tests.tests.failover.ClusterFailover3NodesTest.killNodeAndReplaceWithNewEmptyOne(ClusterFailover3NodesTest.java:165)
at [email protected]/java.lang.reflect.Method.invoke(Method.java:568) at
[email protected]/java.util.concurrent.FutureTask.run(FutureTask.java:264) at
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at [email protected]/java.lang.Thread.run(Thread.java:833) {code}
Server logs are in attachment.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
