[kafka] branch trunk updated: KAFKA-8115: Reduce flakiness in Trogdor JsonRestServer shutdown (#12830)

[ijuma]

This is an automated email from the ASF dual-hosted git repository.

ijuma pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/kafka.git


The following commit(s) were added to refs/heads/trunk by this push:
     new fe375dce547 KAFKA-8115: Reduce flakiness in Trogdor JsonRestServer 
shutdown (#12830)
fe375dce547 is described below

commit fe375dce54717f0699e9ab860c4a33669d017f22
Author: Greg Harris <greg.har...@aiven.io>
AuthorDate: Sat Apr 15 12:21:56 2023 -0700

    KAFKA-8115: Reduce flakiness in Trogdor JsonRestServer shutdown (#12830)
    
    The GRACEFUL_SHUTDOWN_TIMEOUT_MS for the Trogdor JsonRestServer is 100ms.
    In heavily loaded CI environments, this timeout can be exceeded. When this 
happens,
    it causes the jettyServer.stop() and jettyServer.destroy() calls to throw 
exceptions, which
    prevents shutdownExecutor.shutdown() from running. This has the effect of 
causing the JsonRestServer::waitForShutdown method to block for 1 day, which 
exceeds the 120s
    timeout on the CoordinatorTest (and any other test relying on 
MiniTrogdorCluster).
    
    This change makes it such that the graceful shutdown timeout is less likely 
to be exceeded,
    and when it is, the timeout does not cause the waitForShutdown method to 
block for much
    longer than the graceful shutdown timeout.
    
    Reviewers: Ismael Juma <ism...@juma.me.uk>
---
 .../java/org/apache/kafka/trogdor/rest/JsonRestServer.java     | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git 
a/trogdor/src/main/java/org/apache/kafka/trogdor/rest/JsonRestServer.java 
b/trogdor/src/main/java/org/apache/kafka/trogdor/rest/JsonRestServer.java
index e5388f8a6f9..cc3bbc0cc6d 100644
--- a/trogdor/src/main/java/org/apache/kafka/trogdor/rest/JsonRestServer.java
+++ b/trogdor/src/main/java/org/apache/kafka/trogdor/rest/JsonRestServer.java
@@ -56,7 +56,7 @@ import java.util.concurrent.TimeUnit;
 public class JsonRestServer {
     private static final Logger log = 
LoggerFactory.getLogger(JsonRestServer.class);
 
-    private static final long GRACEFUL_SHUTDOWN_TIMEOUT_MS = 100;
+    private static final long GRACEFUL_SHUTDOWN_TIMEOUT_MS = 10 * 1000;
 
     private final ScheduledExecutorService shutdownExecutor;
 
@@ -142,9 +142,13 @@ public class JsonRestServer {
                 } catch (Exception e) {
                     log.error("Unable to stop REST server", e);
                 } finally {
-                    jettyServer.destroy();
+                    try {
+                        jettyServer.destroy();
+                    } catch (Exception e) {
+                        log.error("Unable to destroy REST server", e);
+                    }
+                    shutdownExecutor.shutdown();
                 }
-                shutdownExecutor.shutdown();
                 return null;
             });
         }