Andrey Yarovoy created HDDS-17868:
-------------------------------------

             Summary: XceiverClientShortCircuit logs every create, start, close 
and idle timeout at INFO, with a stack trace on idle timeout
                 Key: HDDS-17868
                 URL: https://issues.apache.org/jira/browse/HDDS-17868
             Project: Apache Ozone
          Issue Type: Improvement
          Components: Ozone Client
            Reporter: Andrey Yarovoy
            Assignee: Andrey Yarovoy


h3. Problem

{\{XceiverClientShortCircuit}} logs its normal lifecycle at INFO:

{noformat}
:129 LOG.info("{} is created for pipeline {}", ..., pipeline)
:146 LOG.info("{} is started", prefix)
:160 LOG.info("{} is closed for {} with {} requests sent and {} responses 
received",
domainSocket.toString(), dn, requestSent, responseReceived)
:545 LOG.info("{} receiveResponseTask is closed after send {} requests and 
received {} responses, due to {}",
domainSocket.toString(), requestSent, responseReceived, e.getClass().getName(), 
e)
{noformat}

Short-circuit clients are created and closed often. \{{XceiverClientManager}} 
keys clients per pipeline, user and short-circuit flag, and evicts them after 
\{{scm.container.client.idle.threshold}} (10s). Every Ratis pipeline that 
contains the local datanode gets its own short-circuit client. A busy client 
such as an HBase RegionServer therefore writes INFO lines continuously in 
normal operation.

Line 545 also passes the exception as the last argument, so every normal idle 
close (\{{SocketTimeoutException}}, \{{EOFException}}, 
\{{ClosedChannelException}}) logs a full stack trace at INFO.

{\{domainSocket.toString()}} is evaluated eagerly at lines 160 and 545 even 
when INFO is disabled.

h3. Evidence

async-profiler wall profile of an HBase RegionServer (Ratis, short-circuit read 
on, YCSB E). The Ozone build predates HDDS-16812, so evicted clients were 
closed on the RPC handler thread:
* \{{XceiverClientShortCircuit.close}} took 9.0% of busy handler time;
* inside it, 60.5% is \{{DomainSocket.toString}} (\{{String.format}}) for the 
INFO line at :160, and 39.2% is \{{DomainSocket.close}}.

HDDS-16812 moves the close off the caller thread, so the latency effect is gone 
on master. The log volume and the eager formatting remain.

h3. Proposed fix

* Lines 129, 146 and 160: log at DEBUG.
* Line 160: pass \{{domainSocket}} itself instead of 
\{{domainSocket.toString()}}, so the string is built only when the line is 
logged.
* Line 545: for the expected idle-close exceptions 
(\{{SocketTimeoutException}}, \{{EOFException}}, \{{ClosedChannelException}}), 
log at DEBUG with the exception class name only. Unexpected exceptions keep 
their stack trace at WARN, as the \{{catch (Throwable e)}} branch below it does.

h3. Compatibility

Log output only. Operators who relied on these INFO lines to watch 
short-circuit client churn can enable DEBUG for 
\{{org.apache.hadoop.hdds.scm.XceiverClientShortCircuit}}.

h3. Testing

No functional change. Existing short-circuit tests 
(\{{TestXceiverClientManagerSC}}, \{{TestLocalChunkInputStream}}) cover the 
code paths.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to