[
https://issues.apache.org/jira/browse/BEAM-8399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17032745#comment-17032745
]
Udi Meiri commented on BEAM-8399:
---------------------------------
#10223 is out for review. It supports the correct URL format, but doesn't use
the namenodehost portion.
Future changes could include:
- Using namenodehost portion of URL instead of --hdfs_{host,port,user}.
- Connecting to more than one HDFS host.
> Python HDFS implementation should support filenames of the format
> "hdfs://namenodehost/parent/child"
> ----------------------------------------------------------------------------------------------------
>
> Key: BEAM-8399
> URL: https://issues.apache.org/jira/browse/BEAM-8399
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
> Reporter: Chamikara Madhusanka Jayalath
> Assignee: Udi Meiri
> Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> "hdfs://namenodehost/parent/child" and "/parent/child" seems to be the
> correct filename formats for HDFS based on [1] but we currently support
> format "hdfs://parent/child".
> To not break existing users, we have to either (1) somehow support both
> versions by default (based on [2] seems like HDFS does not allow colons in
> file path so this might be possible) (2) make
> "hdfs://namenodehost/parent/child" optional for now and change it to default
> after few versions.
> We should also make sure that Beam Java and Python HDFS file-system
> implementations are consistent in this regard.
>
> [1][https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html]
> [2] https://issues.apache.org/jira/browse/HDFS-13
>
> cc: [~udim]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
