[jira] [Updated] (HDFS-9178) Slow datanode I/O can cause a wrong node to be marked bad
[ https://issues.apache.org/jira/browse/HDFS-9178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjun Zhang updated HDFS-9178: Description: When non-leaf datanode in a pipeline is slow on or stuck at disk I/O, the downstream node can timeout on reading packet since even the heartbeat packets will not be relayed down. The packet read timeout is set in {{DataXceiver#run()}}: {code} peer.setReadTimeout(dnConf.socketTimeout); {code} When the downstream node times out and closes the connection to the upstream, the upstream node's {{PacketResponder}} gets {{EOFException}} and it sends an ack upstream with the downstream node status set to {{ERROR}}. This caused the client to exclude the downstream node, even though the upstream node was the one got stuck. The connection to downstream has longer timeout, so the downstream will always timeout first. The downstream timeout is set in {{writeBlock()}} {code} int timeoutValue = dnConf.socketTimeout + (HdfsConstants.READ_TIMEOUT_EXTENSION * targets.length); int writeTimeout = dnConf.socketWriteTimeout + (HdfsConstants.WRITE_TIMEOUT_EXTENSION * targets.length); NetUtils.connect(mirrorSock, mirrorTarget, timeoutValue); OutputStream unbufMirrorOut = NetUtils.getOutputStream(mirrorSock, writeTimeout); {code} was: When non-leaf datanode in a pipeline is slow on or stuck at disk I/O, the downstream node can timeout on reading packet since even the heartbeat packets will not be relayed down. The packet read timeout is set in {{DataXceiver#run()}}: {code} peer.setReadTimeout(dnConf.socketTimeout); {code} When the downstream node times out and closes the connection to the upstream, the upstream node's {{PacketResponder}} gets {{EOFException}} and it sends an ack upstream with the downstream node status set to {{ERROR}}. This caused the client to exclude the downstream node, even thought the upstream node was the one got stuck. The connection to downstream has longer timeout, so the downstream will always timeout first. The downstream timeout is set in {{writeBlock()}} {code} int timeoutValue = dnConf.socketTimeout + (HdfsConstants.READ_TIMEOUT_EXTENSION * targets.length); int writeTimeout = dnConf.socketWriteTimeout + (HdfsConstants.WRITE_TIMEOUT_EXTENSION * targets.length); NetUtils.connect(mirrorSock, mirrorTarget, timeoutValue); OutputStream unbufMirrorOut = NetUtils.getOutputStream(mirrorSock, writeTimeout); {code} > Slow datanode I/O can cause a wrong node to be marked bad > - > > Key: HDFS-9178 > URL: https://issues.apache.org/jira/browse/HDFS-9178 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > Fix For: 2.8.0, 2.7.2, 2.6.4, 3.0.0-alpha1 > > Attachments: 002-HDFS-9178.branch-2.6.patch, > HDFS-9178.branch-2.6.patch, HDFS-9178.patch > > > When non-leaf datanode in a pipeline is slow on or stuck at disk I/O, the > downstream node can timeout on reading packet since even the heartbeat > packets will not be relayed down. > The packet read timeout is set in {{DataXceiver#run()}}: > {code} > peer.setReadTimeout(dnConf.socketTimeout); > {code} > When the downstream node times out and closes the connection to the upstream, > the upstream node's {{PacketResponder}} gets {{EOFException}} and it sends an > ack upstream with the downstream node status set to {{ERROR}}. This caused > the client to exclude the downstream node, even though the upstream node was > the one got stuck. > The connection to downstream has longer timeout, so the downstream will > always timeout first. The downstream timeout is set in {{writeBlock()}} > {code} > int timeoutValue = dnConf.socketTimeout + > (HdfsConstants.READ_TIMEOUT_EXTENSION * targets.length); > int writeTimeout = dnConf.socketWriteTimeout + > (HdfsConstants.WRITE_TIMEOUT_EXTENSION * targets.length); > NetUtils.connect(mirrorSock, mirrorTarget, timeoutValue); > OutputStream unbufMirrorOut = NetUtils.getOutputStream(mirrorSock, > writeTimeout); > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9178) Slow datanode I/O can cause a wrong node to be marked bad
[ https://issues.apache.org/jira/browse/HDFS-9178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-9178: - Fix Version/s: 2.8.0 > Slow datanode I/O can cause a wrong node to be marked bad > - > > Key: HDFS-9178 > URL: https://issues.apache.org/jira/browse/HDFS-9178 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > Fix For: 2.8.0, 2.7.2, 2.6.4, 3.0.0-alpha1 > > Attachments: 002-HDFS-9178.branch-2.6.patch, > HDFS-9178.branch-2.6.patch, HDFS-9178.patch > > > When non-leaf datanode in a pipeline is slow on or stuck at disk I/O, the > downstream node can timeout on reading packet since even the heartbeat > packets will not be relayed down. > The packet read timeout is set in {{DataXceiver#run()}}: > {code} > peer.setReadTimeout(dnConf.socketTimeout); > {code} > When the downstream node times out and closes the connection to the upstream, > the upstream node's {{PacketResponder}} gets {{EOFException}} and it sends an > ack upstream with the downstream node status set to {{ERROR}}. This caused > the client to exclude the downstream node, even thought the upstream node was > the one got stuck. > The connection to downstream has longer timeout, so the downstream will > always timeout first. The downstream timeout is set in {{writeBlock()}} > {code} > int timeoutValue = dnConf.socketTimeout + > (HdfsConstants.READ_TIMEOUT_EXTENSION * targets.length); > int writeTimeout = dnConf.socketWriteTimeout + > (HdfsConstants.WRITE_TIMEOUT_EXTENSION * targets.length); > NetUtils.connect(mirrorSock, mirrorTarget, timeoutValue); > OutputStream unbufMirrorOut = NetUtils.getOutputStream(mirrorSock, > writeTimeout); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9178) Slow datanode I/O can cause a wrong node to be marked bad
[ https://issues.apache.org/jira/browse/HDFS-9178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-9178: - Fix Version/s: 2.6.4 > Slow datanode I/O can cause a wrong node to be marked bad > - > > Key: HDFS-9178 > URL: https://issues.apache.org/jira/browse/HDFS-9178 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > Fix For: 2.7.2, 2.6.4 > > Attachments: 002-HDFS-9178.branch-2.6.patch, > HDFS-9178.branch-2.6.patch, HDFS-9178.patch > > > When non-leaf datanode in a pipeline is slow on or stuck at disk I/O, the > downstream node can timeout on reading packet since even the heartbeat > packets will not be relayed down. > The packet read timeout is set in {{DataXceiver#run()}}: > {code} > peer.setReadTimeout(dnConf.socketTimeout); > {code} > When the downstream node times out and closes the connection to the upstream, > the upstream node's {{PacketResponder}} gets {{EOFException}} and it sends an > ack upstream with the downstream node status set to {{ERROR}}. This caused > the client to exclude the downstream node, even thought the upstream node was > the one got stuck. > The connection to downstream has longer timeout, so the downstream will > always timeout first. The downstream timeout is set in {{writeBlock()}} > {code} > int timeoutValue = dnConf.socketTimeout + > (HdfsConstants.READ_TIMEOUT_EXTENSION * targets.length); > int writeTimeout = dnConf.socketWriteTimeout + > (HdfsConstants.WRITE_TIMEOUT_EXTENSION * targets.length); > NetUtils.connect(mirrorSock, mirrorTarget, timeoutValue); > OutputStream unbufMirrorOut = NetUtils.getOutputStream(mirrorSock, > writeTimeout); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9178) Slow datanode I/O can cause a wrong node to be marked bad
[ https://issues.apache.org/jira/browse/HDFS-9178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-9178: - Attachment: 002-HDFS-9178.branch-2.6.patch The patch for branch-2.6 is stale. Update 002 patch that sync with latest 2.6 branch. [~kihwal], would you help to review it? Thanks! > Slow datanode I/O can cause a wrong node to be marked bad > - > > Key: HDFS-9178 > URL: https://issues.apache.org/jira/browse/HDFS-9178 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > Fix For: 2.7.2 > > Attachments: 002-HDFS-9178.branch-2.6.patch, > HDFS-9178.branch-2.6.patch, HDFS-9178.patch > > > When non-leaf datanode in a pipeline is slow on or stuck at disk I/O, the > downstream node can timeout on reading packet since even the heartbeat > packets will not be relayed down. > The packet read timeout is set in {{DataXceiver#run()}}: > {code} > peer.setReadTimeout(dnConf.socketTimeout); > {code} > When the downstream node times out and closes the connection to the upstream, > the upstream node's {{PacketResponder}} gets {{EOFException}} and it sends an > ack upstream with the downstream node status set to {{ERROR}}. This caused > the client to exclude the downstream node, even thought the upstream node was > the one got stuck. > The connection to downstream has longer timeout, so the downstream will > always timeout first. The downstream timeout is set in {{writeBlock()}} > {code} > int timeoutValue = dnConf.socketTimeout + > (HdfsConstants.READ_TIMEOUT_EXTENSION * targets.length); > int writeTimeout = dnConf.socketWriteTimeout + > (HdfsConstants.WRITE_TIMEOUT_EXTENSION * targets.length); > NetUtils.connect(mirrorSock, mirrorTarget, timeoutValue); > OutputStream unbufMirrorOut = NetUtils.getOutputStream(mirrorSock, > writeTimeout); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9178) Slow datanode I/O can cause a wrong node to be marked bad
[ https://issues.apache.org/jira/browse/HDFS-9178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated HDFS-9178: -- Fix Version/s: (was: 3.0.0) > Slow datanode I/O can cause a wrong node to be marked bad > - > > Key: HDFS-9178 > URL: https://issues.apache.org/jira/browse/HDFS-9178 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > Fix For: 2.7.2 > > Attachments: HDFS-9178.branch-2.6.patch, HDFS-9178.patch > > > When non-leaf datanode in a pipeline is slow on or stuck at disk I/O, the > downstream node can timeout on reading packet since even the heartbeat > packets will not be relayed down. > The packet read timeout is set in {{DataXceiver#run()}}: > {code} > peer.setReadTimeout(dnConf.socketTimeout); > {code} > When the downstream node times out and closes the connection to the upstream, > the upstream node's {{PacketResponder}} gets {{EOFException}} and it sends an > ack upstream with the downstream node status set to {{ERROR}}. This caused > the client to exclude the downstream node, even thought the upstream node was > the one got stuck. > The connection to downstream has longer timeout, so the downstream will > always timeout first. The downstream timeout is set in {{writeBlock()}} > {code} > int timeoutValue = dnConf.socketTimeout + > (HdfsConstants.READ_TIMEOUT_EXTENSION * targets.length); > int writeTimeout = dnConf.socketWriteTimeout + > (HdfsConstants.WRITE_TIMEOUT_EXTENSION * targets.length); > NetUtils.connect(mirrorSock, mirrorTarget, timeoutValue); > OutputStream unbufMirrorOut = NetUtils.getOutputStream(mirrorSock, > writeTimeout); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9178) Slow datanode I/O can cause a wrong node to be marked bad
[ https://issues.apache.org/jira/browse/HDFS-9178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-9178: - Target Version/s: 2.7.2, 2.6.4 (was: 2.7.2) > Slow datanode I/O can cause a wrong node to be marked bad > - > > Key: HDFS-9178 > URL: https://issues.apache.org/jira/browse/HDFS-9178 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > Fix For: 3.0.0, 2.7.2 > > Attachments: HDFS-9178.branch-2.6.patch, HDFS-9178.patch > > > When non-leaf datanode in a pipeline is slow on or stuck at disk I/O, the > downstream node can timeout on reading packet since even the heartbeat > packets will not be relayed down. > The packet read timeout is set in {{DataXceiver#run()}}: > {code} > peer.setReadTimeout(dnConf.socketTimeout); > {code} > When the downstream node times out and closes the connection to the upstream, > the upstream node's {{PacketResponder}} gets {{EOFException}} and it sends an > ack upstream with the downstream node status set to {{ERROR}}. This caused > the client to exclude the downstream node, even thought the upstream node was > the one got stuck. > The connection to downstream has longer timeout, so the downstream will > always timeout first. The downstream timeout is set in {{writeBlock()}} > {code} > int timeoutValue = dnConf.socketTimeout + > (HdfsConstants.READ_TIMEOUT_EXTENSION * targets.length); > int writeTimeout = dnConf.socketWriteTimeout + > (HdfsConstants.WRITE_TIMEOUT_EXTENSION * targets.length); > NetUtils.connect(mirrorSock, mirrorTarget, timeoutValue); > OutputStream unbufMirrorOut = NetUtils.getOutputStream(mirrorSock, > writeTimeout); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9178) Slow datanode I/O can cause a wrong node to be marked bad
[ https://issues.apache.org/jira/browse/HDFS-9178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-9178: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.7.2 3.0.0 Status: Resolved (was: Patch Available) > Slow datanode I/O can cause a wrong node to be marked bad > - > > Key: HDFS-9178 > URL: https://issues.apache.org/jira/browse/HDFS-9178 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > Fix For: 3.0.0, 2.7.2 > > Attachments: HDFS-9178.patch > > > When non-leaf datanode in a pipeline is slow on or stuck at disk I/O, the > downstream node can timeout on reading packet since even the heartbeat > packets will not be relayed down. > The packet read timeout is set in {{DataXceiver#run()}}: > {code} > peer.setReadTimeout(dnConf.socketTimeout); > {code} > When the downstream node times out and closes the connection to the upstream, > the upstream node's {{PacketResponder}} gets {{EOFException}} and it sends an > ack upstream with the downstream node status set to {{ERROR}}. This caused > the client to exclude the downstream node, even thought the upstream node was > the one got stuck. > The connection to downstream has longer timeout, so the downstream will > always timeout first. The downstream timeout is set in {{writeBlock()}} > {code} > int timeoutValue = dnConf.socketTimeout + > (HdfsConstants.READ_TIMEOUT_EXTENSION * targets.length); > int writeTimeout = dnConf.socketWriteTimeout + > (HdfsConstants.WRITE_TIMEOUT_EXTENSION * targets.length); > NetUtils.connect(mirrorSock, mirrorTarget, timeoutValue); > OutputStream unbufMirrorOut = NetUtils.getOutputStream(mirrorSock, > writeTimeout); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9178) Slow datanode I/O can cause a wrong node to be marked bad
[ https://issues.apache.org/jira/browse/HDFS-9178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-9178: - Attachment: HDFS-9178.branch-2.6.patch Attaching a patch for branch-2.6, in case someone wants it. > Slow datanode I/O can cause a wrong node to be marked bad > - > > Key: HDFS-9178 > URL: https://issues.apache.org/jira/browse/HDFS-9178 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > Fix For: 3.0.0, 2.7.2 > > Attachments: HDFS-9178.branch-2.6.patch, HDFS-9178.patch > > > When non-leaf datanode in a pipeline is slow on or stuck at disk I/O, the > downstream node can timeout on reading packet since even the heartbeat > packets will not be relayed down. > The packet read timeout is set in {{DataXceiver#run()}}: > {code} > peer.setReadTimeout(dnConf.socketTimeout); > {code} > When the downstream node times out and closes the connection to the upstream, > the upstream node's {{PacketResponder}} gets {{EOFException}} and it sends an > ack upstream with the downstream node status set to {{ERROR}}. This caused > the client to exclude the downstream node, even thought the upstream node was > the one got stuck. > The connection to downstream has longer timeout, so the downstream will > always timeout first. The downstream timeout is set in {{writeBlock()}} > {code} > int timeoutValue = dnConf.socketTimeout + > (HdfsConstants.READ_TIMEOUT_EXTENSION * targets.length); > int writeTimeout = dnConf.socketWriteTimeout + > (HdfsConstants.WRITE_TIMEOUT_EXTENSION * targets.length); > NetUtils.connect(mirrorSock, mirrorTarget, timeoutValue); > OutputStream unbufMirrorOut = NetUtils.getOutputStream(mirrorSock, > writeTimeout); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9178) Slow datanode I/O can cause a wrong node to be marked bad
[ https://issues.apache.org/jira/browse/HDFS-9178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-9178: - Attachment: HDFS-9178.patch The patch implements the proposed fix. > Slow datanode I/O can cause a wrong node to be marked bad > - > > Key: HDFS-9178 > URL: https://issues.apache.org/jira/browse/HDFS-9178 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > Attachments: HDFS-9178.patch > > > When non-leaf datanode in a pipeline is slow on or stuck at disk I/O, the > downstream node can timeout on reading packet since even the heartbeat > packets will not be relayed down. > The packet read timeout is set in {{DataXceiver#run()}}: > {code} > peer.setReadTimeout(dnConf.socketTimeout); > {code} > When the downstream node times out and closes the connection to the upstream, > the upstream node's {{PacketResponder}} gets {{EOFException}} and it sends an > ack upstream with the downstream node status set to {{ERROR}}. This caused > the client to exclude the downstream node, even thought the upstream node was > the one got stuck. > The connection to downstream has longer timeout, so the downstream will > always timeout first. The downstream timeout is set in {{writeBlock()}} > {code} > int timeoutValue = dnConf.socketTimeout + > (HdfsConstants.READ_TIMEOUT_EXTENSION * targets.length); > int writeTimeout = dnConf.socketWriteTimeout + > (HdfsConstants.WRITE_TIMEOUT_EXTENSION * targets.length); > NetUtils.connect(mirrorSock, mirrorTarget, timeoutValue); > OutputStream unbufMirrorOut = NetUtils.getOutputStream(mirrorSock, > writeTimeout); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9178) Slow datanode I/O can cause a wrong node to be marked bad
[ https://issues.apache.org/jira/browse/HDFS-9178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-9178: - Status: Patch Available (was: Open) > Slow datanode I/O can cause a wrong node to be marked bad > - > > Key: HDFS-9178 > URL: https://issues.apache.org/jira/browse/HDFS-9178 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > Attachments: HDFS-9178.patch > > > When non-leaf datanode in a pipeline is slow on or stuck at disk I/O, the > downstream node can timeout on reading packet since even the heartbeat > packets will not be relayed down. > The packet read timeout is set in {{DataXceiver#run()}}: > {code} > peer.setReadTimeout(dnConf.socketTimeout); > {code} > When the downstream node times out and closes the connection to the upstream, > the upstream node's {{PacketResponder}} gets {{EOFException}} and it sends an > ack upstream with the downstream node status set to {{ERROR}}. This caused > the client to exclude the downstream node, even thought the upstream node was > the one got stuck. > The connection to downstream has longer timeout, so the downstream will > always timeout first. The downstream timeout is set in {{writeBlock()}} > {code} > int timeoutValue = dnConf.socketTimeout + > (HdfsConstants.READ_TIMEOUT_EXTENSION * targets.length); > int writeTimeout = dnConf.socketWriteTimeout + > (HdfsConstants.WRITE_TIMEOUT_EXTENSION * targets.length); > NetUtils.connect(mirrorSock, mirrorTarget, timeoutValue); > OutputStream unbufMirrorOut = NetUtils.getOutputStream(mirrorSock, > writeTimeout); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)