[
https://issues.apache.org/jira/browse/YARN-8690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jialei weng updated YARN-8690:
------------------------------
Description:
With YARN-1953 change, in yarn 2.9.1, we can not use path, like
hdfs://hostname/path, to local resource allocation as it will be resolved to
hdfs://hostname{color:#ff0000}:0{color}/path. We have to add the port 443 in
path, like hdfs://hostname{color:#ff0000}:443{color}/path, to make it work. It
isn't a consistent change. If we can make it consistent without customer's
change? [~leftnoteasy]
|Handle resource location path in 2.7|Handle resource location logic in 2.9|
| public static Path getPathFromYarnURL(URL url) throws URISyntaxException {
String scheme = url.getScheme() == null ? "" : url.getScheme();
String authority = "";
if (url.getHost() != null) {
authority = url.getHost();
if (url.getUserInfo() != null) {
authority = url.getUserInfo() + "@" + authority;
}
{color:#d04437} if (url.getPort() > 0) {{color}{color:#d04437}
authority += ":" + url.getPort();{color}{color:#d04437} }{color} }
return new Path(
(new URI(scheme, authority, url.getFile(), null, null)).normalize());
}| public Path toPath() throws URISyntaxException \{ return new
Path(new URI(getScheme(), getUserInfo(), getHost(), getPort(), getFile(),
null, null)); }|
was:
With YARN-1953 change, in yarn 2.9.1, we can not use path, like
hdfs://hostname/path, to local resource allocation as it will be resolved to
hdfs://hostname{color:#ff0000}:0{color}/path. We have to add the port 443 in
path, like hdfs://hostname{color:#ff0000}:443{color}/path, to make it work. It
isn't a consistent change. If we can make it consistent without customer's
change? [~leftnoteasy]
|Handle resource location path in 2.7|Handle resource location logic in 2.9|
| public static Path getPathFromYarnURL(URL url) throws URISyntaxException {
String scheme = url.getScheme() == null ? "" : url.getScheme();
String authority = "";
if (url.getHost() != null) {
authority = url.getHost();
if (url.getUserInfo() != null) {
authority = url.getUserInfo() + "@" + authority;
}
{color:#d04437} if (url.getPort() > 0) {{color}
{color:#d04437} authority += ":" + url.getPort();{color}
{color:#d04437} }{color}
}
return new Path(
(new URI(scheme, authority, url.getFile(), null, null)).normalize());
}| public Path toPath() throws URISyntaxException {
return new Path(new URI(getScheme(), getUserInfo(),
getHost(), getPort(), getFile(), null, null));
}|
> Currently path not consistent in LocalResourceRequest to yarn 2.7
> -----------------------------------------------------------------
>
> Key: YARN-8690
> URL: https://issues.apache.org/jira/browse/YARN-8690
> Project: Hadoop YARN
> Issue Type: Bug
> Affects Versions: 2.9.1
> Reporter: jialei weng
> Assignee: Wangda Tan
> Priority: Major
>
> With YARN-1953 change, in yarn 2.9.1, we can not use path, like
> hdfs://hostname/path, to local resource allocation as it will be resolved to
> hdfs://hostname{color:#ff0000}:0{color}/path. We have to add the port 443 in
> path, like hdfs://hostname{color:#ff0000}:443{color}/path, to make it work.
> It isn't a consistent change. If we can make it consistent without
> customer's change? [~leftnoteasy]
> |Handle resource location path in 2.7|Handle resource location logic in 2.9|
> | public static Path getPathFromYarnURL(URL url) throws URISyntaxException {
> String scheme = url.getScheme() == null ? "" : url.getScheme();
>
> String authority = "";
> if (url.getHost() != null) {
> authority = url.getHost();
> if (url.getUserInfo() != null) {
> authority = url.getUserInfo() + "@" + authority;
> }
> {color:#d04437} if (url.getPort() > 0) {{color}{color:#d04437}
> authority += ":" + url.getPort();{color}{color:#d04437} }{color} }
>
> return new Path(
> (new URI(scheme, authority, url.getFile(), null, null)).normalize());
> }| public Path toPath() throws URISyntaxException \{ return new
> Path(new URI(getScheme(), getUserInfo(), getHost(), getPort(),
> getFile(), null, null)); }|
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
