[jira] [Updated] (HADOOP-12430) Fix HDFS client gets errors trying to to connect to IPv6 DataNode
[ https://issues.apache.org/jira/browse/HADOOP-12430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HADOOP-12430: -- Fix Version/s: HADOOP-11890 > Fix HDFS client gets errors trying to to connect to IPv6 DataNode > - > > Key: HADOOP-12430 > URL: https://issues.apache.org/jira/browse/HADOOP-12430 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.6.0 >Reporter: Nate Edel >Assignee: Nate Edel >Priority: Major > Labels: BB2015-05-TBR, ipv6 > Fix For: HADOOP-11890, HADOOP-17800 > > Attachments: HDFS-8078-HADOOP-17800.001.patch, > HDFS-8078-HADOOP-17800.002.patch, HDFS-8078.10.patch, HDFS-8078.11.patch, > HDFS-8078.12.patch, HDFS-8078.13.patch, HDFS-8078.14.patch, > HDFS-8078.15.patch, HDFS-8078.9.patch, dummy.patch > > > 1st exception, on put: > 15/03/23 18:43:18 WARN hdfs.DFSClient: DataStreamer Exception > java.lang.IllegalArgumentException: Does not contain a valid host:port > authority: 2401:db00:1010:70ba:face:0:8:0:50010 > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:212) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153) > at > org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1607) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588) > Appears to actually stem from code in DataNodeID which assumes it's safe to > append together (ipaddr + ":" + port) -- which is OK for IPv4 and not OK for > IPv6. NetUtils.createSocketAddr( ) assembles a Java URI object, which > requires the format proto://[2401:db00:1010:70ba:face:0:8:0]:50010 > Currently using InetAddress.getByName() to validate IPv6 (guava > InetAddresses.forString has been flaky) but could also use our own parsing. > (From logging this, it seems like a low-enough frequency call that the extra > object creation shouldn't be problematic, and for me the slight risk of > passing in bad input that is not actually an IPv4 or IPv6 address and thus > calling an external DNS lookup is outweighed by getting the address > normalized and avoiding rewriting parsing.) > Alternatively, sun.net.util.IPAddressUtil.isIPv6LiteralAddress() > --- > 2nd exception (on datanode) > 15/04/13 13:18:07 ERROR datanode.DataNode: > dev1903.prn1.facebook.com:50010:DataXceiver error processing unknown > operation src: /2401:db00:20:7013:face:0:7:0:54152 dst: > /2401:db00:11:d010:face:0:2f:0:50010 > java.io.EOFException > at java.io.DataInputStream.readShort(DataInputStream.java:315) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:226) > at java.lang.Thread.run(Thread.java:745) > Which also comes as client error "-get: 2401 is not an IP string literal." > This one has existing parsing logic which needs to shift to the last colon > rather than the first. Should also be a tiny bit faster by using lastIndexOf > rather than split. Could alternatively use the techniques above. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12430) Fix HDFS client gets errors trying to to connect to IPv6 DataNode
[ https://issues.apache.org/jira/browse/HADOOP-12430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HADOOP-12430: -- Fix Version/s: HADOOP-17800 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) [~hemanthboyina] thanks confirmation and contribution .. Committed to HADOOP-17800 branch. > Fix HDFS client gets errors trying to to connect to IPv6 DataNode > - > > Key: HADOOP-12430 > URL: https://issues.apache.org/jira/browse/HADOOP-12430 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.6.0 >Reporter: Nate Edel >Assignee: Nate Edel >Priority: Major > Labels: BB2015-05-TBR, ipv6 > Fix For: HADOOP-17800 > > Attachments: HDFS-8078-HADOOP-17800.001.patch, > HDFS-8078-HADOOP-17800.002.patch, HDFS-8078.10.patch, HDFS-8078.11.patch, > HDFS-8078.12.patch, HDFS-8078.13.patch, HDFS-8078.14.patch, > HDFS-8078.15.patch, HDFS-8078.9.patch, dummy.patch > > > 1st exception, on put: > 15/03/23 18:43:18 WARN hdfs.DFSClient: DataStreamer Exception > java.lang.IllegalArgumentException: Does not contain a valid host:port > authority: 2401:db00:1010:70ba:face:0:8:0:50010 > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:212) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153) > at > org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1607) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588) > Appears to actually stem from code in DataNodeID which assumes it's safe to > append together (ipaddr + ":" + port) -- which is OK for IPv4 and not OK for > IPv6. NetUtils.createSocketAddr( ) assembles a Java URI object, which > requires the format proto://[2401:db00:1010:70ba:face:0:8:0]:50010 > Currently using InetAddress.getByName() to validate IPv6 (guava > InetAddresses.forString has been flaky) but could also use our own parsing. > (From logging this, it seems like a low-enough frequency call that the extra > object creation shouldn't be problematic, and for me the slight risk of > passing in bad input that is not actually an IPv4 or IPv6 address and thus > calling an external DNS lookup is outweighed by getting the address > normalized and avoiding rewriting parsing.) > Alternatively, sun.net.util.IPAddressUtil.isIPv6LiteralAddress() > --- > 2nd exception (on datanode) > 15/04/13 13:18:07 ERROR datanode.DataNode: > dev1903.prn1.facebook.com:50010:DataXceiver error processing unknown > operation src: /2401:db00:20:7013:face:0:7:0:54152 dst: > /2401:db00:11:d010:face:0:2f:0:50010 > java.io.EOFException > at java.io.DataInputStream.readShort(DataInputStream.java:315) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:226) > at java.lang.Thread.run(Thread.java:745) > Which also comes as client error "-get: 2401 is not an IP string literal." > This one has existing parsing logic which needs to shift to the last colon > rather than the first. Should also be a tiny bit faster by using lastIndexOf > rather than split. Could alternatively use the techniques above. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12430) Fix HDFS client gets errors trying to to connect to IPv6 DataNode
[ https://issues.apache.org/jira/browse/HADOOP-12430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Boyina updated HADOOP-12430: Attachment: HDFS-8078-HADOOP-17800.002.patch > Fix HDFS client gets errors trying to to connect to IPv6 DataNode > - > > Key: HADOOP-12430 > URL: https://issues.apache.org/jira/browse/HADOOP-12430 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.6.0 >Reporter: Nate Edel >Assignee: Nate Edel >Priority: Major > Labels: BB2015-05-TBR, ipv6 > Attachments: HDFS-8078-HADOOP-17800.001.patch, > HDFS-8078-HADOOP-17800.002.patch, HDFS-8078.10.patch, HDFS-8078.11.patch, > HDFS-8078.12.patch, HDFS-8078.13.patch, HDFS-8078.14.patch, > HDFS-8078.15.patch, HDFS-8078.9.patch, dummy.patch > > > 1st exception, on put: > 15/03/23 18:43:18 WARN hdfs.DFSClient: DataStreamer Exception > java.lang.IllegalArgumentException: Does not contain a valid host:port > authority: 2401:db00:1010:70ba:face:0:8:0:50010 > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:212) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153) > at > org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1607) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588) > Appears to actually stem from code in DataNodeID which assumes it's safe to > append together (ipaddr + ":" + port) -- which is OK for IPv4 and not OK for > IPv6. NetUtils.createSocketAddr( ) assembles a Java URI object, which > requires the format proto://[2401:db00:1010:70ba:face:0:8:0]:50010 > Currently using InetAddress.getByName() to validate IPv6 (guava > InetAddresses.forString has been flaky) but could also use our own parsing. > (From logging this, it seems like a low-enough frequency call that the extra > object creation shouldn't be problematic, and for me the slight risk of > passing in bad input that is not actually an IPv4 or IPv6 address and thus > calling an external DNS lookup is outweighed by getting the address > normalized and avoiding rewriting parsing.) > Alternatively, sun.net.util.IPAddressUtil.isIPv6LiteralAddress() > --- > 2nd exception (on datanode) > 15/04/13 13:18:07 ERROR datanode.DataNode: > dev1903.prn1.facebook.com:50010:DataXceiver error processing unknown > operation src: /2401:db00:20:7013:face:0:7:0:54152 dst: > /2401:db00:11:d010:face:0:2f:0:50010 > java.io.EOFException > at java.io.DataInputStream.readShort(DataInputStream.java:315) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:226) > at java.lang.Thread.run(Thread.java:745) > Which also comes as client error "-get: 2401 is not an IP string literal." > This one has existing parsing logic which needs to shift to the last colon > rather than the first. Should also be a tiny bit faster by using lastIndexOf > rather than split. Could alternatively use the techniques above. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12430) Fix HDFS client gets errors trying to to connect to IPv6 DataNode
[ https://issues.apache.org/jira/browse/HADOOP-12430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Boyina updated HADOOP-12430: Attachment: HDFS-8078-HADOOP-17800.001.patch Status: Patch Available (was: Reopened) > Fix HDFS client gets errors trying to to connect to IPv6 DataNode > - > > Key: HADOOP-12430 > URL: https://issues.apache.org/jira/browse/HADOOP-12430 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.6.0 >Reporter: Nate Edel >Assignee: Nate Edel >Priority: Major > Labels: BB2015-05-TBR, ipv6 > Attachments: HDFS-8078-HADOOP-17800.001.patch, HDFS-8078.10.patch, > HDFS-8078.11.patch, HDFS-8078.12.patch, HDFS-8078.13.patch, > HDFS-8078.14.patch, HDFS-8078.15.patch, HDFS-8078.9.patch, dummy.patch > > > 1st exception, on put: > 15/03/23 18:43:18 WARN hdfs.DFSClient: DataStreamer Exception > java.lang.IllegalArgumentException: Does not contain a valid host:port > authority: 2401:db00:1010:70ba:face:0:8:0:50010 > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:212) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153) > at > org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1607) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588) > Appears to actually stem from code in DataNodeID which assumes it's safe to > append together (ipaddr + ":" + port) -- which is OK for IPv4 and not OK for > IPv6. NetUtils.createSocketAddr( ) assembles a Java URI object, which > requires the format proto://[2401:db00:1010:70ba:face:0:8:0]:50010 > Currently using InetAddress.getByName() to validate IPv6 (guava > InetAddresses.forString has been flaky) but could also use our own parsing. > (From logging this, it seems like a low-enough frequency call that the extra > object creation shouldn't be problematic, and for me the slight risk of > passing in bad input that is not actually an IPv4 or IPv6 address and thus > calling an external DNS lookup is outweighed by getting the address > normalized and avoiding rewriting parsing.) > Alternatively, sun.net.util.IPAddressUtil.isIPv6LiteralAddress() > --- > 2nd exception (on datanode) > 15/04/13 13:18:07 ERROR datanode.DataNode: > dev1903.prn1.facebook.com:50010:DataXceiver error processing unknown > operation src: /2401:db00:20:7013:face:0:7:0:54152 dst: > /2401:db00:11:d010:face:0:2f:0:50010 > java.io.EOFException > at java.io.DataInputStream.readShort(DataInputStream.java:315) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:226) > at java.lang.Thread.run(Thread.java:745) > Which also comes as client error "-get: 2401 is not an IP string literal." > This one has existing parsing logic which needs to shift to the last colon > rather than the first. Should also be a tiny bit faster by using lastIndexOf > rather than split. Could alternatively use the techniques above. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12430) Fix HDFS client gets errors trying to to connect to IPv6 DataNode
[ https://issues.apache.org/jira/browse/HADOOP-12430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12430: --- Summary: Fix HDFS client gets errors trying to to connect to IPv6 DataNode (was: HDFS client gets errors trying to to connect to IPv6 DataNode) > Fix HDFS client gets errors trying to to connect to IPv6 DataNode > - > > Key: HADOOP-12430 > URL: https://issues.apache.org/jira/browse/HADOOP-12430 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.6.0 >Reporter: Nate Edel >Assignee: Nate Edel > Labels: BB2015-05-TBR, ipv6 > Attachments: HDFS-8078.10.patch, HDFS-8078.11.patch, > HDFS-8078.12.patch, HDFS-8078.13.patch, HDFS-8078.14.patch, > HDFS-8078.15.patch, HDFS-8078.9.patch, dummy.patch > > > 1st exception, on put: > 15/03/23 18:43:18 WARN hdfs.DFSClient: DataStreamer Exception > java.lang.IllegalArgumentException: Does not contain a valid host:port > authority: 2401:db00:1010:70ba:face:0:8:0:50010 > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:212) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153) > at > org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1607) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588) > Appears to actually stem from code in DataNodeID which assumes it's safe to > append together (ipaddr + ":" + port) -- which is OK for IPv4 and not OK for > IPv6. NetUtils.createSocketAddr( ) assembles a Java URI object, which > requires the format proto://[2401:db00:1010:70ba:face:0:8:0]:50010 > Currently using InetAddress.getByName() to validate IPv6 (guava > InetAddresses.forString has been flaky) but could also use our own parsing. > (From logging this, it seems like a low-enough frequency call that the extra > object creation shouldn't be problematic, and for me the slight risk of > passing in bad input that is not actually an IPv4 or IPv6 address and thus > calling an external DNS lookup is outweighed by getting the address > normalized and avoiding rewriting parsing.) > Alternatively, sun.net.util.IPAddressUtil.isIPv6LiteralAddress() > --- > 2nd exception (on datanode) > 15/04/13 13:18:07 ERROR datanode.DataNode: > dev1903.prn1.facebook.com:50010:DataXceiver error processing unknown > operation src: /2401:db00:20:7013:face:0:7:0:54152 dst: > /2401:db00:11:d010:face:0:2f:0:50010 > java.io.EOFException > at java.io.DataInputStream.readShort(DataInputStream.java:315) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:226) > at java.lang.Thread.run(Thread.java:745) > Which also comes as client error "-get: 2401 is not an IP string literal." > This one has existing parsing logic which needs to shift to the last colon > rather than the first. Should also be a tiny bit faster by using lastIndexOf > rather than split. Could alternatively use the techniques above. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
