[geode] branch support/1.12 updated: GEODE-8131: reader thread blocked attempting to issue an alert (#5132)

[bschuchardt]

This is an automated email from the ASF dual-hosted git repository.

bschuchardt pushed a commit to branch support/1.12
in repository https://gitbox.apache.org/repos/asf/geode.git


The following commit(s) were added to refs/heads/support/1.12 by this push:
     new dd500e8  GEODE-8131: reader thread blocked attempting to issue an 
alert (#5132)
dd500e8 is described below

commit dd500e8509150f1c753e777bac6ce3ec24f7adce
Author: Bruce Schuchardt <bschucha...@pivotal.io>
AuthorDate: Thu May 21 14:19:37 2020 -0700

    GEODE-8131: reader thread blocked attempting to issue an alert (#5132)
    
    * GEODE-8131: reader thread blocked attempting to issue an alert
    
    This PR does not solve the problem of the alert system causing a P2P
    message reader to block but instead ensures that the reader thread will
    write deserialization information to the log before issuing a fatal
    alert.  If the alert level is set to "info" this won't help, but no-one
    would set the alert level to that level.
    
    Along the way I removed some duplicate strings.
    
    * removing duplicate code & logging toString of exceptions at fatal-level
    
    (cherry picked from commit 358fd7067cc56b1ceb8c3d7c271c3a5254d7ee93)
---
 .../org/apache/geode/internal/tcp/Connection.java    | 20 +++++++++++++++++---
 1 file changed, 17 insertions(+), 3 deletions(-)

diff --git 
a/geode-core/src/main/java/org/apache/geode/internal/tcp/Connection.java 
b/geode-core/src/main/java/org/apache/geode/internal/tcp/Connection.java
index d266688..a25e40c 100644
--- a/geode-core/src/main/java/org/apache/geode/internal/tcp/Connection.java
+++ b/geode-core/src/main/java/org/apache/geode/internal/tcp/Connection.java
@@ -2977,13 +2977,13 @@ public class Connection implements Runnable {
         failureMsg = "ClassNotFound deserializing message";
         failureEx = ex;
         rpId = md.getRPid();
-        logger.fatal("ClassNotFound deserializing message: {}", ex.toString());
+        logAtInfoAndFatal(failureMsg, failureEx);
       } catch (IOException ex) {
         owner.getConduit().getStats().decMessagesBeingReceived(md.size());
         failureMsg = "IOException deserializing message";
         failureEx = ex;
         rpId = md.getRPid();
-        logger.fatal("IOException deserializing message", failureEx);
+        logAtInfoAndFatal(failureMsg, failureEx);
       } catch (InterruptedException ex) {
         interrupted = true;
         owner.getConduit().getCancelCriterion().checkCancelInProgress(ex);
@@ -3004,7 +3004,7 @@ public class Connection implements Runnable {
         failureMsg = "Unexpected failure deserializing message";
         failureEx = ex;
         rpId = md.getRPid();
-        logger.fatal("Unexpected failure deserializing message", failureEx);
+        logAtInfoAndFatal(failureMsg, failureEx);
       } finally {
         msgLength = md.size();
         releaseMsgDestreamer(messageId, md);
@@ -3045,6 +3045,20 @@ public class Connection implements Runnable {
     }
   }
 
+  /**
+   * For exceptions that we absolutely must see in the log files, use this 
method
+   * to log the problem first at "info" level and then at "fatal" level. We do 
this
+   * in case the "fatal" level log entry generates an alert that gets blocked 
in
+   * transmitting the data to an alert listener like the JMX Manager
+   */
+  private void logAtInfoAndFatal(String failureMsg, Throwable failureEx) {
+    // log at info level first in case fatal-level alert notification becomes 
blocked
+    logger.info(failureMsg, failureEx);
+    // log at fatal-level with toString() on the exception since this will 
generate an
+    // alert
+    logger.fatal(failureMsg, failureEx.toString());
+  }
+
   void readHandshakeForSender(DataInputStream dis, ByteBuffer peerDataBuffer) {
     try {
       int replyCode = dis.readUnsignedByte();