[jira] [Commented] (YARN-1946) need Public interface for WebAppUtils.getProxyHostAndPort
[ https://issues.apache.org/jira/browse/YARN-1946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17802787#comment-17802787 ] Shilun Fan commented on YARN-1946: -- Bulk update: moved all 3.4.0 non-blocker issues, please move back if it is a blocker. Retarget 3.5.0. > need Public interface for WebAppUtils.getProxyHostAndPort > - > > Key: YARN-1946 > URL: https://issues.apache.org/jira/browse/YARN-1946 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, webapp >Affects Versions: 2.4.0 >Reporter: Thomas Graves >Priority: Major > > ApplicationMasters are supposed to go through the ResourceManager web app > proxy if they have web UI's so they are properly secured. There is currently > no public interface for Application Masters to conveniently get the proxy > host and port. There is a function in WebAppUtils, but that class is > private. > We should provide this as a utility since any properly written AM will need > to do this. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1946) need Public interface for WebAppUtils.getProxyHostAndPort
[ https://issues.apache.org/jira/browse/YARN-1946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525242#comment-14525242 ] Steve Loughran commented on YARN-1946: -- This has got more complex with the HA proxy stuff: there's 1 (host,port) pair and the caller needs to handle failover. its still something that could be made public for anyone wanting to do their own AmIPFilter; they'd just need to add failover logic themselves need Public interface for WebAppUtils.getProxyHostAndPort - Key: YARN-1946 URL: https://issues.apache.org/jira/browse/YARN-1946 Project: Hadoop YARN Issue Type: Sub-task Components: api, webapp Affects Versions: 2.4.0 Reporter: Thomas Graves Priority: Critical ApplicationMasters are supposed to go through the ResourceManager web app proxy if they have web UI's so they are properly secured. There is currently no public interface for Application Masters to conveniently get the proxy host and port. There is a function in WebAppUtils, but that class is private. We should provide this as a utility since any properly written AM will need to do this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1946) need Public interface for WebAppUtils.getProxyHostAndPort
[ https://issues.apache.org/jira/browse/YARN-1946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524054#comment-14524054 ] Xuan Gong commented on YARN-1946: - [~tgraves] Do we still need this ? need Public interface for WebAppUtils.getProxyHostAndPort - Key: YARN-1946 URL: https://issues.apache.org/jira/browse/YARN-1946 Project: Hadoop YARN Issue Type: Sub-task Components: api, webapp Affects Versions: 2.4.0 Reporter: Thomas Graves Priority: Critical ApplicationMasters are supposed to go through the ResourceManager web app proxy if they have web UI's so they are properly secured. There is currently no public interface for Application Masters to conveniently get the proxy host and port. There is a function in WebAppUtils, but that class is private. We should provide this as a utility since any properly written AM will need to do this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1946) need Public interface for WebAppUtils.getProxyHostAndPort
[ https://issues.apache.org/jira/browse/YARN-1946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13973901#comment-13973901 ] Steve Loughran commented on YARN-1946: -- Thomas -we proxy the GUI, but for the REST API we don't. The proxy doesn't forward any operation than GET, and there's no guarantee clients will handle 307 redirects with the same HTTP verb anyway. If/when the proxy supports more operations, we can try with it. the filter just says ws/* - no proxy else: proxy and ws/ doesn't do GUI need Public interface for WebAppUtils.getProxyHostAndPort - Key: YARN-1946 URL: https://issues.apache.org/jira/browse/YARN-1946 Project: Hadoop YARN Issue Type: Sub-task Components: api, webapp Affects Versions: 2.4.0 Reporter: Thomas Graves Priority: Critical ApplicationMasters are supposed to go through the ResourceManager web app proxy if they have web UI's so they are properly secured. There is currently no public interface for Application Masters to conveniently get the proxy host and port. There is a function in WebAppUtils, but that class is private. We should provide this as a utility since any properly written AM will need to do this. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1946) need Public interface for WebAppUtils.getProxyHostAndPort
[
https://issues.apache.org/jira/browse/YARN-1946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13972887#comment-13972887
]
Steve Loughran commented on YARN-1946:
--
We've replaced the normal AmFilter with one that doesn't proxy under ws/*,
looking into is code
{{WebAppUitls.getProxyHostAndPort(Configuration conf)}} does the work; it just
extracts it from {{conf.get(YarnConfiguration.PROXY_ADDRESS)}}, which is coming
out of yarn-site.xml
..and looking at that -it is public on branch-2.4.0, so I'd close this -after
you verify that the 2.4 release works for you as it does for us.
That method will work for YARN apps sharing the classpath, but doesn't do
anything for apps in other languages -the proxy does need to be passed down,
ideally during AM registration
need Public interface for WebAppUtils.getProxyHostAndPort
-
Key: YARN-1946
URL: https://issues.apache.org/jira/browse/YARN-1946
Project: Hadoop YARN
Issue Type: Sub-task
Components: api, webapp
Affects Versions: 2.4.0
Reporter: Thomas Graves
Priority: Critical
ApplicationMasters are supposed to go through the ResourceManager web app
proxy if they have web UI's so they are properly secured. There is currently
no public interface for Application Masters to conveniently get the proxy
host and port. There is a function in WebAppUtils, but that class is
private.
We should provide this as a utility since any properly written AM will need
to do this.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1946) need Public interface for WebAppUtils.getProxyHostAndPort
[
https://issues.apache.org/jira/browse/YARN-1946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13972959#comment-13972959
]
Thomas Graves commented on YARN-1946:
-
{quote}
We've replaced the normal AmFilter with one that doesn't proxy under ws/*,
looking into is code
{quote}
sorry I don't understand what you are saying here? You are saying your
application has? If it doesn't proxy it doesn't really follow yarn security
rules?
You are correct that the YarnConfiguration is public, but the
getProxyHostAndPort and now getProxyHostsAndPortsForAmFilter handle other
things for you. https, RM HA, etc...
need Public interface for WebAppUtils.getProxyHostAndPort
-
Key: YARN-1946
URL: https://issues.apache.org/jira/browse/YARN-1946
Project: Hadoop YARN
Issue Type: Sub-task
Components: api, webapp
Affects Versions: 2.4.0
Reporter: Thomas Graves
Priority: Critical
ApplicationMasters are supposed to go through the ResourceManager web app
proxy if they have web UI's so they are properly secured. There is currently
no public interface for Application Masters to conveniently get the proxy
host and port. There is a function in WebAppUtils, but that class is
private.
We should provide this as a utility since any properly written AM will need
to do this.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1946) need Public interface for WebAppUtils.getProxyHostAndPort
[
https://issues.apache.org/jira/browse/YARN-1946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13970589#comment-13970589
]
Steve Loughran commented on YARN-1946:
--
It's retrievable via
{{System.getenv(ApplicationConstants.APPLICATION_WEB_PROXY_BASE_ENV);}}
This is cross-language and does work. Do we need another way to make it public?
need Public interface for WebAppUtils.getProxyHostAndPort
-
Key: YARN-1946
URL: https://issues.apache.org/jira/browse/YARN-1946
Project: Hadoop YARN
Issue Type: Bug
Components: api, webapp
Affects Versions: 2.4.0
Reporter: Thomas Graves
Priority: Critical
ApplicationMasters are supposed to go through the ResourceManager web app
proxy if they have web UI's so they are properly secured. There is currently
no public interface for Application Masters to conveniently get the proxy
host and port. There is a function in WebAppUtils, but that class is
private.
We should provide this as a utility since any properly written AM will need
to do this.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1946) need Public interface for WebAppUtils.getProxyHostAndPort
[ https://issues.apache.org/jira/browse/YARN-1946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13971421#comment-13971421 ] Thomas Graves commented on YARN-1946: - Thanks for the suggestion Steve. That is only the proxy base /proxy/... Which I am actually also using: val proxy = WebAppUtils.getProxyHostAndPort(conf) val uriBase = http://; + proxy + System.getenv(ApplicationConstants.APPLICATION_WEB_PROXY_BASE_ENV) Do you know if there is a way to more easily get the proxy host and port? If you are an application that doesn't use the Hadoop HttpServer for webapps then in order to use the web app proxy you have to use install AmIpFilter directly and provide the PROXY_HOST and PROXY_URI_BASE. If you are using the HttpServer then you can use the AmFilterInitializer which figures out the host and port for you. Thus why I need the host and port. Note that I would basically copy this code into the application trying to do this but if I need to copy it more then likely other applications would also so perhaps we should make Public utility functions for it. Looking some more I could also try to use the AmFilterInitializer by creating a class with the interface for FilterContainer and then call initFilter directly. This however would specialize the code whereas using the AmIpFilter directly fits in with installing any normal java servlet filter. I would prefer not to do the specialized code since an application may run on multiple framework - yarn being on of them. Also note back in hadoop 0.23 this was easier to get (publicly available) as it was in YarnConfiguration.getProxyHostAndPort it actually also looks like my code is out of date as it was updated recently to handle HA: YARN-1811 need Public interface for WebAppUtils.getProxyHostAndPort - Key: YARN-1946 URL: https://issues.apache.org/jira/browse/YARN-1946 Project: Hadoop YARN Issue Type: Bug Components: api, webapp Affects Versions: 2.4.0 Reporter: Thomas Graves Priority: Critical ApplicationMasters are supposed to go through the ResourceManager web app proxy if they have web UI's so they are properly secured. There is currently no public interface for Application Masters to conveniently get the proxy host and port. There is a function in WebAppUtils, but that class is private. We should provide this as a utility since any properly written AM will need to do this. -- This message was sent by Atlassian JIRA (v6.2#6252)
