[
https://issues.apache.org/jira/browse/SPARK-23548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dieter resolved SPARK-23548.
----------------------------
Resolution: Not A Problem
resolved. was name resolution issue
> Redirect loop from Resourcemanager to Spark Webui
> -------------------------------------------------
>
> Key: SPARK-23548
> URL: https://issues.apache.org/jira/browse/SPARK-23548
> Project: Spark
> Issue Type: Bug
> Components: Web UI, YARN
> Affects Versions: 2.3.0
> Environment: *SW*
> hadoop-3.0.0
> Spark 2.3.0 (git revision a0d7949896) built for Hadoop 2.7.3
>
> *Nodes*
> jupyter
> proxyserver
> historyserver
> resourcemanager
> nodemanager ([http://9d581cfb391f:8042|http://9d581cfb391f:8042/])
>
> *Config of sparkcontext*
> spark.app.id=application_1519904229414_0011
> spark.app.name=Spark shell
> spark.driver.appUIAddress=http://jupyter:4040
> spark.driver.extraJavaOptions=-Dlog4j.configuration=file:/tmp/driver_log4j.properties
> spark.driver.host=jupyter
> spark.driver.port=35813
> spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/tmp/driver_log4j.properties
> spark.executor.id=driver
> spark.home=/usr/local/spark
> spark.jars=
> spark.logConf=true
> spark.master=yarn
> spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_HOSTS=proxyserver
> spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_URI_BASES=[http://proxyserver:8089/proxy/application_1519904229414_0011]
>
>
> *yarn-site.xml*
> <configuration>
> <!-- Site specific YARN configuration properties -->
> <property>
> <name>yarn.web-proxy.address</name>
> <value>proxyserver:8089</value>
> </property>
> <property>
> <name>yarn.resourcemanager.fs.state-store.uri</name>
> <value>/rmstate</value>
> </property>
> <property>
> <name>yarn.resourcemanager.webapp.address</name>
> <value>resourcemanager:8088</value>
> </property>
> <property>
> <name>yarn.timeline-service.generic-application-history.enabled</name>
> <value>true</value>
> </property>
> <property>
> <name>yarn.resourcemanager.recovery.enabled</name>
> <value>true</value>
> </property>
> <property>
> <name>yarn.nodemanager.bind-host</name>
> <value>0.0.0.0</value>
> </property>
> <property>
> <name>yarn.timeline-service.bind-host</name>
> <value>0.0.0.0</value>
> </property>
> <property>
> <name>yarn.log-aggregation-enable</name>
> <value>true</value>
> </property>
> <property>
> <name>yarn.timeline-service.enabled</name>
> <value>true</value>
> </property>
> <property>
> <name>yarn.resourcemanager.store.class</name>
>
> <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore</value>
> </property>
> <property>
> <name>yarn.resourcemanager.system-metrics-publisher.enabled</name>
> <value>true</value>
> </property>
> <property>
> <name>yarn.nodemanager.remote-app-log-dir</name>
> <value>/app-logs</value>
> </property>
> <property>
> <name>yarn.nodemanager.vmem-pmem-ratio</name>
> <value>4.1</value>
> </property>
> <property>
> <name>yarn.resourcemanager.resource-tracker.address</name>
> <value>resourcemanager:8031</value>
> </property>
> <property>
> <name>yarn.resourcemanager.hostname</name>
> <value>resourcemanager</value>
> </property>
> <property>
> <name>yarn.timeline-service.hostname</name>
> <value>historyserver</value>
> </property>
> <property>
> <name>yarn.log.server.url</name>
> <value>http://historyserver:8188/applicationhistory/logs/</value>
> </property>
> <property>
> <name>yarn.resourcemanager.bind-host</name>
> <value>0.0.0.0</value>
> </property>
> <property>
> <name>yarn.resourcemanager.scheduler.address</name>
> <value>resourcemanager:8030</value>
> </property>
> <property>
> <name>yarn.resourcemanager.address</name>
> <value>resourcemanager:8032</value>
> </property>
> </configuration>
> Reporter: Dieter
> Priority: Major
>
> Access from Resourcemanager to Webui in a distributed environment results in
> a redirect loop. It seems webui is not recognizing request is coming from
> resourcemanager proxy.
>
> *Situation*
> Spark application running on jupyter
>
> *link in resourcemanger:8088 (behind Trackingui->Applicationmaster)*
> [http://proxyserver:8089/proxy/application_1519904229414_0011/]
>
> *Response of webui on jupyter:4040 (redirects to proxyserver)*
> {{<html>}}
> {{ <head>}}
> {{ <title> Moved </title> </head> <body> <h1> Moved
> </h1> <div> Content has moved <a
> href="http://proxyserver:8089/proxy/application_1519904229414_0011/";>here</a>
> </div> }}
> {{ </body>}}
> {{</html>}}{{ }}
>
> *log output of proxyserver (results in redirect)*
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,905 DEBUG strategy.ExecuteProduceConsume: EPC
> Prod/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@5c4d391a producing
>
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,908 DEBUG protocol.RequestAuthCache: Auth cache
> not set in the context
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,908 DEBUG http.wire: >> "[\r][\n]"
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,908 DEBUG http.headers: >> GET / HTTP/1.1
>
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,908 DEBUG http.headers: >> Accept:
> text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
>
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,908 DEBUG http.headers: >> Host: jupyter:4040
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,911 DEBUG http.headers: << Server:
> Jetty(9.3.z-SNAPSHOT)
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,911 DEBUG client.DefaultRedirectStrategy:
> Redirect requested to location
> 'http://proxyserver:8089/proxy/application_1519904229414_0001/'
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,911 DEBUG http.wire: << " Moved[\n]"
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,911 DEBUG http.wire: << " Content has moved
> [\n]"
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,911 DEBUG http.wire: << " <a
> href="http://proxyserver:8089/proxy/application_1519904229414_0001/";>here</a>[\n]"
>
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,911 DEBUG conn.BasicClientConnectionManager:
> Connection can be kept alive indefinitely
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,911 DEBUG conn.DefaultClientConnectionOperator:
> Connecting to proxyserver:8089
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,912 DEBUG io.ManagedSelector: Queued change
> org.eclipse.jetty.io.ManagedSelector$Accept@5498de79 on
> org.eclipse.jetty.io.ManagedSelector@558bdf1f id=0 keys=63 selected=0
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,912 DEBUG io.ManagedSelector: Selector loop
> woken up from select, 0/63 selected
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,913 DEBUG strategy.ExecuteProduceConsume: EPC
> Pend/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@5c4d391a dispatch
>
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,913 DEBUG thread.QueuedThreadPool: queue EPC
> Pend/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@5c4d391a
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,913 DEBUG io.SelectChannelEndPoint:
> changeInterests p=false 0->1 for
> SelectChannelEndPoint@77d3aac7\{/10.0.1.17:48699<->8089,Open,in,out,FI,-,0/10000,HttpConnection@7d134d7f}{io=0/1,kio=0,kro=0}
>
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,914 DEBUG conn.DefaultClientConnection: Sending
> request: GET /proxy/application_1519904229414_0001/ HTTP/1.1
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,914 DEBUG http.wire: >> "Accept-Encoding: gzip,
> deflate[\r][\n]"
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,914 DEBUG http.wire: >> "Host:
> proxyserver:8089[\r][\n]"
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,914 DEBUG http.headers: >> GET
> /proxy/application_1519904229414_0001/ HTTP/1.1
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,915 DEBUG io.FillInterest:
> FillInterest@301a7eba\{true,AC.ReadCB@7d134d7f{HttpConnection@7d134d7f[SelectChannelEndPoint@77d3aac7{/10.0.1.17:48699<->8089,Open,in,out,FI,-,2/10000,HttpConnection@7d134d7f}{io=1/0,kio=1,kro=1}][p=HttpParser\{s=START,0
> of
> 0},g=HttpGenerator@17fc5588\{s=START},c=HttpChannelOverHttp@54440a5\{r=0,c=false,a=IDLE,uri=null}]}}
> fillable
> AC.ReadCB@7d134d7f\{HttpConnection@7d134d7f[SelectChannelEndPoint@77d3aac7{/10.0.1.17:48699<->8089,Open,in,out,FI,-,2/10000,HttpConnection@7d134d7f}{io=1/0,kio=1,kro=1}][p=HttpParser\{s=START,0
> of
> 0},g=HttpGenerator@17fc5588\{s=START},c=HttpChannelOverHttp@54440a5\{r=0,c=false,a=IDLE,uri=null}]}
>
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,915 DEBUG io.ChannelEndPoint: filled 426
> SelectChannelEndPoint@77d3aac7\{/10.0.1.17:48699<->8089,Open,in,out,-,-,2/10000,HttpConnection@7d134d7f}{io=1/0,kio=1,kro=1}
>
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,915 DEBUG server.HttpConnection:
> HttpConnection@7d134d7f[SelectChannelEndPoint@77d3aac7\{/10.0.1.17:48699<->8089,Open,in,out,-,-,0/10000,HttpConnection@7d134d7f}{io=1/0,kio=1,kro=1}][p=HttpParser\{s=START,0
> of
> 0},g=HttpGenerator@17fc5588\{s=START},c=HttpChannelOverHttp@54440a5\{r=0,c=false,a=IDLE,uri=null}]
> filled 426 HeapByteBuffer@5b3c5d02[p=0,l=426,c=8192,r=426]=\{<<<GET
> /proxy/applic... Keep-Alive\r\n\r\n>>>Accept-Language:
> ...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,915 DEBUG http.HttpParser: SPACE2 -->
> REQUEST_VERSION
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,915 DEBUG http.HttpParser: HEADER_IN_VALUE -->
> HEADER
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,915 DEBUG http.HttpParser: HEADER_IN_VALUE -->
> HEADER
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,915 DEBUG http.HttpParser: HEADER -->
> HEADER_VALUE
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,915 DEBUG server.HttpConnection:
> HttpConnection@7d134d7f[SelectChannelEndPoint@77d3aac7\{/10.0.1.17:48699<->8089,Open,in,out,-,-,0/10000,HttpConnection@7d134d7f}{io=1/0,kio=1,kro=1}][p=HttpParser\{s=END,0
> of
> 0},g=HttpGenerator@17fc5588\{s=START},c=HttpChannelOverHttp@54440a5\{r=1,c=false,a=IDLE,uri=//proxyserver:8089/proxy/application_1519904229414_0001/}]
> parsed true HttpParser\{s=END,0 of 0}
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,916 DEBUG server.session: session=null
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,916 DEBUG servlet.ServletHandler: call filter
> static_user_filter
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,918 DEBUG io.ManagedSelector: Selector loop
> waiting on select
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,929 DEBUG webproxy.WebAppProxyServlet: local
> InetAddress for proxy host: proxyserver/10.0.1.17
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,930 DEBUG conn.BasicClientConnectionManager: Get
> connection for route proxyserver/10.0.1.17->{}->http://jupyter:4040
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,933 DEBUG protocol.RequestTargetAuthentication:
> Target auth state: UNCHALLENGED
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,934 DEBUG conn.DefaultClientConnection: Sending
> request: GET / HTTP/1.1
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,934 DEBUG http.wire: >> "Accept:
> text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8[\r][\n]"
>
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,934 DEBUG http.wire: >> "User-Agent: Mozilla/5.0
> (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)
> Chrome/64.0.3282.167 Safari/537.36[\r][\n]"
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,934 DEBUG http.wire: >> "Host:
> jupyter:4040[\r][\n]"
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,934 DEBUG http.wire: >> "Connection:
> Keep-Alive[\r][\n]"
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,934 DEBUG http.headers: >> Connection:
> Keep-Alive
> March 1st 2018, 12:38:50.000
> /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver
> stderr | 2018-03-01 11:38:49,937 DEBUG http.headers: << Location:
> http://proxyserver:8089/proxy/application_1519904229414_0001/
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
