[
https://issues.apache.org/jira/browse/HDFS-17332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xing Lin updated HDFS-17332:
----------------------------
Description: In DFSInputStream#actualGetFromOneDataNode(), it would send
the exception stacktrace to the dfsClient.LOG whenever we fail on a DN.
However, in most cases, the read request will be served successfully by reading
from the next available DN. The existence of exception stacktrace in the log
has caused multiple hadoop users at Linkedin to consider this WARN message as
the RC/fatal error for their jobs. We would like to improve the log message
and avoid sending the stacktrace to dfsClient.LOG when a read succeeds. The
stackTrace when reading reach DN is sent to the log only when we really need to
fail a read request (when chooseDataNode()/refetchLocations() throws a
BlockMissingException).
> DFSInputStream: avoid logging stacktrace until when we really need to fail a
> read request with a MissingBlockException
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-17332
> URL: https://issues.apache.org/jira/browse/HDFS-17332
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
> Environment: In DFSInputStream#actualGetFromOneDataNode(), it would
> send the exception stacktrace to the dfsClient.LOG whenever we fail on a DN.
> However, in most cases, the read request will be served successfully by
> reading from the next available DN. The existence of exception stacktrace in
> the log has caused multiple hadoop users at Linkedin to consider this WARN
> message as the RC/fatal error for their jobs. We would like to improve the
> log message and avoid sending the stacktrace to dfsClient.LOG when a read
> succeeds. The stackTrace when reading reach DN is sent to the log only when
> we really need to fail a read request (when
> chooseDataNode()/refetchLocations() throws a BlockMissingException).
> Reporter: Xing Lin
> Priority: Minor
>
> In DFSInputStream#actualGetFromOneDataNode(), it would send the exception
> stacktrace to the dfsClient.LOG whenever we fail on a DN. However, in most
> cases, the read request will be served successfully by reading from the next
> available DN. The existence of exception stacktrace in the log has caused
> multiple hadoop users at Linkedin to consider this WARN message as the
> RC/fatal error for their jobs. We would like to improve the log message and
> avoid sending the stacktrace to dfsClient.LOG when a read succeeds. The
> stackTrace when reading reach DN is sent to the log only when we really need
> to fail a read request (when chooseDataNode()/refetchLocations() throws a
> BlockMissingException).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
