[
https://issues.apache.org/jira/browse/HDFS-8078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14559707#comment-14559707
]
Colin Patrick McCabe commented on HDFS-8078:
--------------------------------------------
Thanks, [~nkedel]. I understand the frustration at the fact that Hadoop / HDFS
doesn't yet support ipv6. It seems like something that we should be supporting
in 2015. However, I guess most cluster operators have simply found that they
can use ipv4 link-local addresses privately, and so the push was never there.
It sounds like this is starting to change.
I think a feature branch would be nice because then:
* You could get non-committers to +1 your patches on the feature branch (move
faster)
* We could design a more coherent test plan for this functionality (how will we
know if a new change breaks ipv6 functionality? Right now we have no idea
because Jenkins explicitly disables ipv6.)
* We could think a little more globally about what needs to be changed. Should
we start passing around InetSocketAddress objects inside DatanodeID instead of
(or in addition to) host:port strings, for example?
Do you have a patch set internally that you could post? I especially think
unit tests would be helpful in keeping us honest here.
> HDFS client gets errors trying to to connect to IPv6 DataNode
> -------------------------------------------------------------
>
> Key: HDFS-8078
> URL: https://issues.apache.org/jira/browse/HDFS-8078
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
> Affects Versions: 2.6.0
> Reporter: Nate Edel
> Assignee: Nate Edel
> Labels: BB2015-05-TBR, ipv6
> Attachments: HDFS-8078.9.patch
>
>
> 1st exception, on put:
> 15/03/23 18:43:18 WARN hdfs.DFSClient: DataStreamer Exception
> java.lang.IllegalArgumentException: Does not contain a valid host:port
> authority: 2401:db00:1010:70ba:face:0:8:0:50010
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:212)
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164)
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1607)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588)
> Appears to actually stem from code in DataNodeID which assumes it's safe to
> append together (ipaddr + ":" + port) -- which is OK for IPv4 and not OK for
> IPv6. NetUtils.createSocketAddr( ) assembles a Java URI object, which
> requires the format proto://[2401:db00:1010:70ba:face:0:8:0]:50010
> Currently using InetAddress.getByName() to validate IPv6 (guava
> InetAddresses.forString has been flaky) but could also use our own parsing.
> (From logging this, it seems like a low-enough frequency call that the extra
> object creation shouldn't be problematic, and for me the slight risk of
> passing in bad input that is not actually an IPv4 or IPv6 address and thus
> calling an external DNS lookup is outweighed by getting the address
> normalized and avoiding rewriting parsing.)
> Alternatively, sun.net.util.IPAddressUtil.isIPv6LiteralAddress()
> -------
> 2nd exception (on datanode)
> 15/04/13 13:18:07 ERROR datanode.DataNode:
> dev1903.prn1.facebook.com:50010:DataXceiver error processing unknown
> operation src: /2401:db00:20:7013:face:0:7:0:54152 dst:
> /2401:db00:11:d010:face:0:2f:0:50010
> java.io.EOFException
> at java.io.DataInputStream.readShort(DataInputStream.java:315)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:226)
> at java.lang.Thread.run(Thread.java:745)
> Which also comes as client error "-get: 2401 is not an IP string literal."
> This one has existing parsing logic which needs to shift to the last colon
> rather than the first. Should also be a tiny bit faster by using lastIndexOf
> rather than split. Could alternatively use the techniques above.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
