[jira] [Updated] (IGNITE-28855) Client node permanently loses discovery events delivered right after the reconnect response
[ https://issues.apache.org/jira/browse/IGNITE-28855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Chugunov updated IGNITE-28855: - Affects Version/s: 2.18 > Client node permanently loses discovery events delivered right after the > reconnect response > --- > > Key: IGNITE-28855 > URL: https://issues.apache.org/jira/browse/IGNITE-28855 > Project: Ignite > Issue Type: Task >Affects Versions: 2.18 >Reporter: Anton Vinogradov >Assignee: Sergey Chugunov >Priority: Major > Fix For: 2.19 > > Time Spent: 20m > Remaining Estimate: 0h > > Regression of IGNITE-26111 (TcpDiscoverySpi uses MessageSerializer). > Before IGNITE-26111, ClientImpl.SocketStream owned a single > BufferedInputStream shared by every reader of the discovery socket. After the > refactoring, Reconnector and SocketReader each create their own > TcpDiscoveryIoSession, each wrapping the raw socket into a new > BufferedInputStream. > When the router server writes the reconnect response and a subsequent > discovery message back-to-back (a single TCP segment -- typical during a > topology storm, when a NodeLeftMessage is enqueued to the client's message > worker right behind the reconnect response), the Reconnector's session reads > ahead and buffers the bytes of the following message. The Reconnector stops > reading at the reconnect response, the socket is handed over to SocketReader, > which attaches a fresh session to the raw socket -- the buffered bytes are > silently discarded and the message is lost forever. > Consequences on the client: > * a discovery event (e.g. NODE_LEFT) is permanently lost; > * if any later topology message arrives, the client fails with a critical > error: > java.lang.AssertionError: lastVer=7, newVer=9 ... at > ClientImpl.updateTopologyHistory(ClientImpl.java:932) > (seen in IgniteCacheGroupsPartitionLossPolicySelfTest: > https://ci2.ignite.apache.org/viewLog.html?buildId=9169034); > * if the lost message was the last one, the client silently stays on a stale > topology (DiscoCache / PME desync). > Fix: create TcpDiscoveryIoSession once per socket (in sendJoinRequest, the > same session that has read the handshake response) and carry it inside > SocketStream; Reconnector and SocketReader reuse it instead of creating their > own. This restores the pre-IGNITE-26111 invariant: a single buffered reader > per socket. > Reproducer: TcpDiscoveryClientTopologyGapTest (5 servers + 4 clients, > sequential graceful stop of 3 servers including the clients' router and the > coordinator). Without the fix a client gets stuck one topology version short > within ~150 iterations; with the fix 3x300 iterations pass clean. > TcpClientDiscoverySpiSelfTest (52 tests), > TcpDiscoveryPendingMessageDeliveryTest and > TcpClientDiscoverySpiCoordinatorChangeTest pass as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-28855) Client node permanently loses discovery events delivered right after the reconnect response
[ https://issues.apache.org/jira/browse/IGNITE-28855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Chugunov updated IGNITE-28855: - Ignite Flags: (was: Docs Required,Release Notes Required) > Client node permanently loses discovery events delivered right after the > reconnect response > --- > > Key: IGNITE-28855 > URL: https://issues.apache.org/jira/browse/IGNITE-28855 > Project: Ignite > Issue Type: Task >Reporter: Anton Vinogradov >Assignee: Sergey Chugunov >Priority: Major > Fix For: 2.19 > > Time Spent: 20m > Remaining Estimate: 0h > > Regression of IGNITE-26111 (TcpDiscoverySpi uses MessageSerializer). > Before IGNITE-26111, ClientImpl.SocketStream owned a single > BufferedInputStream shared by every reader of the discovery socket. After the > refactoring, Reconnector and SocketReader each create their own > TcpDiscoveryIoSession, each wrapping the raw socket into a new > BufferedInputStream. > When the router server writes the reconnect response and a subsequent > discovery message back-to-back (a single TCP segment -- typical during a > topology storm, when a NodeLeftMessage is enqueued to the client's message > worker right behind the reconnect response), the Reconnector's session reads > ahead and buffers the bytes of the following message. The Reconnector stops > reading at the reconnect response, the socket is handed over to SocketReader, > which attaches a fresh session to the raw socket -- the buffered bytes are > silently discarded and the message is lost forever. > Consequences on the client: > * a discovery event (e.g. NODE_LEFT) is permanently lost; > * if any later topology message arrives, the client fails with a critical > error: > java.lang.AssertionError: lastVer=7, newVer=9 ... at > ClientImpl.updateTopologyHistory(ClientImpl.java:932) > (seen in IgniteCacheGroupsPartitionLossPolicySelfTest: > https://ci2.ignite.apache.org/viewLog.html?buildId=9169034); > * if the lost message was the last one, the client silently stays on a stale > topology (DiscoCache / PME desync). > Fix: create TcpDiscoveryIoSession once per socket (in sendJoinRequest, the > same session that has read the handshake response) and carry it inside > SocketStream; Reconnector and SocketReader reuse it instead of creating their > own. This restores the pre-IGNITE-26111 invariant: a single buffered reader > per socket. > Reproducer: TcpDiscoveryClientTopologyGapTest (5 servers + 4 clients, > sequential graceful stop of 3 servers including the clients' router and the > coordinator). Without the fix a client gets stuck one topology version short > within ~150 iterations; with the fix 3x300 iterations pass clean. > TcpClientDiscoverySpiSelfTest (52 tests), > TcpDiscoveryPendingMessageDeliveryTest and > TcpClientDiscoverySpiCoordinatorChangeTest pass as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-28855) Client node permanently loses discovery events delivered right after the reconnect response
[ https://issues.apache.org/jira/browse/IGNITE-28855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Chugunov updated IGNITE-28855: - Fix Version/s: 2.19 > Client node permanently loses discovery events delivered right after the > reconnect response > --- > > Key: IGNITE-28855 > URL: https://issues.apache.org/jira/browse/IGNITE-28855 > Project: Ignite > Issue Type: Task >Reporter: Anton Vinogradov >Assignee: Sergey Chugunov >Priority: Major > Fix For: 2.19 > > Time Spent: 20m > Remaining Estimate: 0h > > Regression of IGNITE-26111 (TcpDiscoverySpi uses MessageSerializer). > Before IGNITE-26111, ClientImpl.SocketStream owned a single > BufferedInputStream shared by every reader of the discovery socket. After the > refactoring, Reconnector and SocketReader each create their own > TcpDiscoveryIoSession, each wrapping the raw socket into a new > BufferedInputStream. > When the router server writes the reconnect response and a subsequent > discovery message back-to-back (a single TCP segment -- typical during a > topology storm, when a NodeLeftMessage is enqueued to the client's message > worker right behind the reconnect response), the Reconnector's session reads > ahead and buffers the bytes of the following message. The Reconnector stops > reading at the reconnect response, the socket is handed over to SocketReader, > which attaches a fresh session to the raw socket -- the buffered bytes are > silently discarded and the message is lost forever. > Consequences on the client: > * a discovery event (e.g. NODE_LEFT) is permanently lost; > * if any later topology message arrives, the client fails with a critical > error: > java.lang.AssertionError: lastVer=7, newVer=9 ... at > ClientImpl.updateTopologyHistory(ClientImpl.java:932) > (seen in IgniteCacheGroupsPartitionLossPolicySelfTest: > https://ci2.ignite.apache.org/viewLog.html?buildId=9169034); > * if the lost message was the last one, the client silently stays on a stale > topology (DiscoCache / PME desync). > Fix: create TcpDiscoveryIoSession once per socket (in sendJoinRequest, the > same session that has read the handshake response) and carry it inside > SocketStream; Reconnector and SocketReader reuse it instead of creating their > own. This restores the pre-IGNITE-26111 invariant: a single buffered reader > per socket. > Reproducer: TcpDiscoveryClientTopologyGapTest (5 servers + 4 clients, > sequential graceful stop of 3 servers including the clients' router and the > coordinator). Without the fix a client gets stuck one topology version short > within ~150 iterations; with the fix 3x300 iterations pass clean. > TcpClientDiscoverySpiSelfTest (52 tests), > TcpDiscoveryPendingMessageDeliveryTest and > TcpClientDiscoverySpiCoordinatorChangeTest pass as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
