[
https://issues.apache.org/jira/browse/HDFS-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136305#comment-13136305
]
Daryn Sharp commented on HDFS-2450:
-----------------------------------
# "can you get thatScheme before making if ..."
#* Gladly! That was one of the aforementioned things I was "itching" to do,
but removed before posting the patch to minimize the change.
# "All checks seem to be using defaultPort"
#* No, the default port is only used if there is not a port in the uri
# "Why remove?"
#* Default uri is sufficient because {{FileSystem.get(defaultUri,
conf).getUri()}} equals {{defaultUri}}. Also note about instantiating the
default fs:
#*# It's relatively expensive, and horrifically bad if the fs cache is disabled
to create a new fs
#*# If the default fs is down, the thread blocks while rpc retries occur
#*# The thrown {{RuntimeException}} is likely blow up the client, as opposed to
the {{IllegalArgumentException}} the caller expects.
# "If thisAuthority and thatAuthority is null, the new mehod throw
IllegalArgumentException"
#* No, the == on the authorities will catch this case.
# "If thisAuthority and thatAuthority are not null, you end up parsing the URI
for thatAuthority twice."
#* Yes, once before canonicalization, once after canonicalization.
# "DistributedFileSystem#makeQualified remove makes me concerned"
#* The only reason it existed is because {{NameNode.getUri()}} was stripping
off the default port. This alone would cause {{p = new
Path("hdfs://host:default-port/file"); fs = p.getFileSystem(conf)}} to be
incompatible because the path has a port, but the fs stripped it off. So an
override of {{checkPath}} and {{makeQualified}} were added to strip the default
port from paths too. All of it is completely unnecessary, especially with
canonicalization.
Tests:
# full test suite
# manual testing with FsShell "ls":
#* scheme://short
#* scheme://short:port
#* scheme://short.partial
#* scheme://short.partial:port
#* scheme://fqdn
#* scheme://fqdn:port
#* scheme://ip
#* scheme://ip:port
# adding more unit tests to automate stressing {{checkPath}} via {{DFS}}, both
positive and negative cases
> Only complete hostname is supported to access data via hdfs://
> --------------------------------------------------------------
>
> Key: HDFS-2450
> URL: https://issues.apache.org/jira/browse/HDFS-2450
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.205.0
> Reporter: Rajit Saha
> Assignee: Daryn Sharp
> Attachments: HDFS-2450-1.patch, HDFS-2450-2.patch, HDFS-2450.patch,
> IP vs. Hostname.pdf
>
>
> If my complete hostname is host1.abc.xyz.com, only complete hostname must be
> used to access data via hdfs://
> I am running following in .20.205 Client to get data from .20.205 NN (host1)
> $hadoop dfs -copyFromLocal /etc/passwd hdfs://host1/tmp
> copyFromLocal: Wrong FS: hdfs://host1/tmp, expected: hdfs://host1.abc.xyz.com
> Usage: java FsShell [-copyFromLocal <localsrc> ... <dst>]
> $hadoop dfs -copyFromLocal /etc/passwd hdfs://host1.abc/tmp/
> copyFromLocal: Wrong FS: hdfs://host1.blue/tmp/1, expected:
> hdfs://host1.abc.xyz.com
> Usage: java FsShell [-copyFromLocal <localsrc> ... <dst>]
> $hadoop dfs -copyFromLocal /etc/passwd hftp://host1.abc.xyz/tmp/
> copyFromLocal: Wrong FS: hdfs://host1.blue/tmp/1, expected:
> hdfs://host1.abc.xyz.com
> Usage: java FsShell [-copyFromLocal <localsrc> ... <dst>]
> Only following is supported
> $hadoop dfs -copyFromLocal /etc/passwd hdfs://host1.abc.xyz.com/tmp/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
