yashmayya commented on code in PR #19875:
URL: https://github.com/apache/pinot/pull/19875#discussion_r4364612135


##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/service/dispatch/QueryDispatcher.java:
##########
@@ -896,16 +899,31 @@ DispatchClient 
getOrCreateDispatchClient(QueryServerInstance queryServerInstance
             _dispatchMaxInboundMessageSizeBytes));
   }
 
-  /// Reset the connection backoff for a server. When the GRPC channel enters 
a TRANSIENT_FAILURE state from
-  /// connection failures, it will fast fail requests and reconnect with 
exponential backoff. This method
-  /// resets the backoff so servers that have recovered can be reconnected to 
immediately.
-  public void resetClientConnectionBackoff(ServerInstance serverInstance) {
-    String hostname = serverInstance.getHostname();
-    int port = serverInstance.getQueryServicePort();
-    DispatchClient dispatchClient = 
_dispatchClientMap.get(toHostnamePortKey(hostname, port));
+  /// Called when a server becomes routable, which it does when it starts and 
when it is enabled again after it was
+  /// disabled or excluded. The server may have restarted at a new address, so:
+  ///
+  /// - the channel to its address resolves the hostname again before the next 
request, and connects without
+  ///   reconnect backoff. A channel resolves only when a connection fails or 
is lost, and the last time was likely
+  ///   when the server went away, while DNS could still have the old IP 
address. Reconnecting to that address could
+  ///   reach another server that took it over.
+  /// - the channel to its previous address, if the server moved to another 
port, is shut down, since nothing uses it
+  ///   again.
+  public void onServerEnabled(ServerInstance serverInstance) {
+    String hostnamePortKey = toHostnamePortKey(serverInstance.getHostname(), 
serverInstance.getQueryServicePort());
+    String previousKey = 
_dispatchAddressByInstanceId.put(serverInstance.getInstanceId(), 
hostnamePortKey);
+    if (previousKey != null && !previousKey.equals(hostnamePortKey)
+        && !_dispatchAddressByInstanceId.containsValue(previousKey)) {

Review Comment:
   Good point, added `testServerEnabledKeepsChannelToAddressTakenOver` in 
c7c51f37b4: A is at addr1, B takes over addr1, then A moves to addr2, and B's 
client stays open. The test fails without the `containsValue` guard.
   
   I kept the `containsValue` scan. It runs only when a server comes back at a 
new address, and it reads one entry per server.
   



##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/mailbox/channel/ChannelManager.java:
##########
@@ -150,18 +156,63 @@ private ManagedChannel watchState(ManagedChannel channel, 
String hostname, int p
     return channel;
   }
 
-  /// Resets the connection backoff for the channel to the given server if the 
channel is in
-  /// TRANSIENT_FAILURE state. Returns true if a reset was performed, false 
otherwise.
+  /// Makes the channel to the given address resolve the hostname again and 
open a new connection before its next
+  /// stream, dropping the connection it has, and any reconnect backoff. 
Streams in flight finish on the old
+  /// connection. Returns false if there is no channel to the address, or if 
it was refreshed less than a second ago.
+  ///
+  /// A channel resolves the hostname only when it starts, and again after a 
connection fails or is lost. When the
+  /// server at that address goes away, gRPC resolves once, right then, and 
keeps that answer until the next failure.
+  /// If DNS still had the old IP address at that moment (a terminating pod 
stays in DNS for a while), the channel
+  /// later reconnects to the old address, and when another server took that 
address over and listens on the same
+  /// port, the connection succeeds and is never questioned again. Call this 
when the address can be stale, for example
+  /// when the server at the address has restarted.
   ///
-  /// @return true if the channel was in TRANSIENT_FAILURE and backoff was 
reset
-  public boolean resetConnectBackoff(String hostname, int port) {
-    ManagedChannel channel = _channelMap.get(Pair.of(hostname, port));
-    if (channel != null && channel.getState(false) == 
ConnectivityState.TRANSIENT_FAILURE) {
-      LOGGER.info("Resetting mailbox channel backoff for server: {}:{}", 
hostname, port);
-      channel.resetConnectBackoff();
-      return true;
+  /// Moving the channel to IDLE, rather than replacing it, keeps the channel 
object that concurrent senders may
+  /// already hold valid. The refreshes are limited to one a second, because 
each refresh drops the connection that the
+  /// previous one opened.
+  public boolean refreshChannel(String hostname, int port) {
+    Pair<String, Integer> key = Pair.of(hostname, port);
+    ManagedChannel channel = _channelMap.get(key);
+    if (channel == null) {
+      return false;
+    }
+    long nowNs = System.nanoTime();
+    AtomicLong lastRefreshTimeNs =
+        _lastRefreshTimeNs.computeIfAbsent(key, k -> new AtomicLong(nowNs - 
MIN_REFRESH_INTERVAL_NS));
+    long lastNs = lastRefreshTimeNs.get();
+    if (nowNs - lastNs < MIN_REFRESH_INTERVAL_NS || 
!lastRefreshTimeNs.compareAndSet(lastNs, nowNs)) {
+      return false;
+    }
+    channel.enterIdle();

Review Comment:
   I checked this, and in grpc-java 1.81 `enterIdle()` does not wait for 
streams in flight:
   
   - `ManagedChannelImpl.enterIdleMode()` shuts down the name resolver and the 
load balancer at once, creates a new resolver, and moves the channel to IDLE.
   - The old subchannel shuts down gracefully about 5 s later 
(`SUBCHANNEL_SHUTDOWN_DELAY_SECONDS`), so streams in flight finish on the old 
connection.
   - The next stream exits IDLE with the new resolver.
   
   The part that waits until no calls are active is the idle timeout, not 
`enterIdle()`.
   
   I also tried it with a stream open. The channel was IDLE 1 ms after 
`refreshChannel`, the open stream still delivered, and the next stream arrived 
on a new connection. So I kept the wording. The Javadoc already says that 
streams in flight finish on the old connection.
   



##########
pinot-server/src/main/java/org/apache/pinot/server/starter/helix/ServerMailboxChannelRefreshHandler.java:
##########
@@ -145,24 +167,51 @@ private static boolean 
isShutdownInProgress(InstanceConfig config) {
         
config.getRecord().getStringField(CommonConstants.Helix.IS_SHUTDOWN_IN_PROGRESS,
 "false"));
   }
 
+  /// Records the server's current mailbox address, and retires the channel to 
its previous one if it moved.
+  private void updateMailboxAddress(String instanceId, InstanceConfig config) {
+    Pair<String, Integer> address = getMailboxAddress(instanceId, config);
+    Pair<String, Integer> previousAddress =
+        address != null ? _mailboxAddresses.put(instanceId, address) : 
_mailboxAddresses.remove(instanceId);
+    if (previousAddress != null && !previousAddress.equals(address)) {
+      retireChannel(instanceId, previousAddress);
+    }
+  }
 
-  private void resetBackoffForServer(String instanceId, InstanceConfig config) 
{
+  private void retireChannel(String instanceId, Pair<String, Integer> address) 
{
+    // Another server may have taken over the address
+    if (_mailboxAddresses.containsValue(address)) {

Review Comment:
   Agreed, I'll keep it as is. It runs only when a server moves or leaves, and 
it reads one entry per server.
   



##########
pinot-server/src/main/java/org/apache/pinot/server/starter/helix/ServerMailboxChannelRefreshHandler.java:
##########
@@ -95,18 +106,28 @@ public synchronized void 
onInstanceConfigChange(List<InstanceConfig> instanceCon
   /// Full cluster scan: used on INIT (first callback) and when instances are 
added/removed.
   private void handleFullScan() {
     Set<String> currentShuttingDown = new HashSet<>();
+    Set<String> currentServers = new HashSet<>();
     for (String instance : _helixAdmin.getInstancesInCluster(_clusterName)) {
       if (!InstanceTypeUtils.isServer(instance) || 
instance.equals(_selfInstanceId)) {
         continue;
       }
+      // Before reading the config: a server whose config cannot be read now 
has not left the cluster
+      currentServers.add(instance);
       InstanceConfig config = getInstanceConfig(instance);
       if (config == null) {
         continue;
       }
+      updateMailboxAddress(instance, config);

Review Comment:
   Good catch, fixed in c7c51f37b4. The full scan now records the new address 
of every server first, and only then retires the old addresses. So the takeover 
check sees the result of the whole scan.
   
   `testFullScanKeepsChannelToAddressTakenOverInTheSameScan` reads the server 
that moves away first. It failed before the change.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to