[
https://issues.apache.org/jira/browse/YARN-6725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Angus Davis updated YARN-6725:
------------------------------
Attachment: YARN-6725.001.patch
> NodeManager cannot localize files when URL contains opaque authority
> --------------------------------------------------------------------
>
> Key: YARN-6725
> URL: https://issues.apache.org/jira/browse/YARN-6725
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
> Affects Versions: 2.8.1
> Reporter: Angus Davis
> Priority: Minor
> Attachments: YARN-6725.001.patch
>
>
> When presented with a URI or URL that has a valid structure, but where the
> URI / URL authority cannot be interpreted as a valid hostname, the java
> URI#getHost method will return null. In the YARN api records the URL message
> is decomposed into hostname, username, port, etc, but it does not provide a
> means to transport an opaque authority. This becomes problematic when using
> object stores or other file systems that use something other than a hostname
> as an authority.
> In our particular case, Google Cloud Storage allows buckets to be named with
> underscores which is causing the NodeManagers to attempt to localize
> resources of the form 'gs:///path/to/object' when they should be attempting
> to localize 'gs://bucket/path/to/object'. Components that are written in
> terms of o.a.h.fs.Path instances seem to handle this case properly, as do
> most Path utility methods (toUri, etc).
> It seems that by transporting the authority along with the host, user info,
> etc, a YARN api URL can properly represent o.a.h.fs.Path instances of this
> particular form.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
