Dieter created SPARK-23548:
------------------------------
Summary: Redirect loop from Resourcemanager to Spark Webui
Key: SPARK-23548
URL: https://issues.apache.org/jira/browse/SPARK-23548
Project: Spark
Issue Type: Bug
Components: Web UI, YARN
Affects Versions: 2.3.0
Environment: *SW*
hadoop-3.0.0
Spark 2.3.0 (git revision a0d7949896) built for Hadoop 2.7.3
*Nodes*
jupyter
proxyserver
historyserver
resourcemanager
nodemanager ([http://9d581cfb391f:8042|http://9d581cfb391f:8042/])
*Config of sparkcontext*
spark.app.id=application_1519904229414_0011
spark.app.name=Spark shell
spark.driver.appUIAddress=http://jupyter:4040
spark.driver.extraJavaOptions=-Dlog4j.configuration=file:/tmp/driver_log4j.properties
spark.driver.host=jupyter
spark.driver.port=35813
spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/tmp/driver_log4j.properties
spark.executor.id=driver
spark.home=/usr/local/spark
spark.jars=
spark.logConf=true
spark.master=yarn
spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_HOSTS=proxyserver
spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_URI_BASES=[http://proxyserver:8089/proxy/application_1519904229414_0011]
*yarn-site.xml*
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.web-proxy.address</name>
<value>proxyserver:8089</value>
</property>
<property>
<name>yarn.resourcemanager.fs.state-store.uri</name>
<value>/rmstate</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>resourcemanager:8088</value>
</property>
<property>
<name>yarn.timeline-service.generic-application-history.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.recovery.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.nodemanager.bind-host</name>
<value>0.0.0.0</value>
</property>
<property>
<name>yarn.timeline-service.bind-host</name>
<value>0.0.0.0</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.timeline-service.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.store.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore</value>
</property>
<property>
<name>yarn.resourcemanager.system-metrics-publisher.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/app-logs</value>
</property>
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>4.1</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>resourcemanager:8031</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>resourcemanager</value>
</property>
<property>
<name>yarn.timeline-service.hostname</name>
<value>historyserver</value>
</property>
<property>
<name>yarn.log.server.url</name>
<value>http://historyserver:8188/applicationhistory/logs/</value>
</property>
<property>
<name>yarn.resourcemanager.bind-host</name>
<value>0.0.0.0</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>resourcemanager:8030</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>resourcemanager:8032</value>
</property>
</configuration>
Reporter: Dieter
Access from Resourcemanager to Webui in a distributed environment results in a
redirect loop. It seems webui is not recognizing request is coming from
resourcemanager proxy.
*Situation*
Spark application running on jupyter
*link in resourcemanger:8088 (behind Trackingui->Applicationmaster)*
[http://proxyserver:8089/proxy/application_1519904229414_0011/]
*Response of webui on jupyter:4040 (redirects to proxyserver)*
{{<html>}}
{{ <head>}}
{{ <title> Moved </title> </head> <body> <h1> Moved
</h1> <div> Content has moved <a
href="http://proxyserver:8089/proxy/application_1519904229414_0011/";>here</a>
</div> }}
{{ </body>}}
{{</html>}}{{ }}
*log output of proxyserver (results in redirect)*
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,905 DEBUG strategy.ExecuteProduceConsume: EPC
Prod/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@5c4d391a producing
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,908 DEBUG protocol.RequestAuthCache: Auth cache not set
in the context
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,908 DEBUG http.wire: >> "[\r][\n]"
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,908 DEBUG http.headers: >> GET / HTTP/1.1
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,908 DEBUG http.headers: >> Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,908 DEBUG http.headers: >> Host: jupyter:4040
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,911 DEBUG http.headers: << Server: Jetty(9.3.z-SNAPSHOT)
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,911 DEBUG client.DefaultRedirectStrategy: Redirect
requested to location
'http://proxyserver:8089/proxy/application_1519904229414_0001/'
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,911 DEBUG http.wire: << " Moved[\n]"
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,911 DEBUG http.wire: << " Content has moved [\n]"
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,911 DEBUG http.wire: << " <a
href="http://proxyserver:8089/proxy/application_1519904229414_0001/";>here</a>[\n]"
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,911 DEBUG conn.BasicClientConnectionManager: Connection
can be kept alive indefinitely
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,911 DEBUG conn.DefaultClientConnectionOperator:
Connecting to proxyserver:8089
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,912 DEBUG io.ManagedSelector: Queued change
org.eclipse.jetty.io.ManagedSelector$Accept@5498de79 on
org.eclipse.jetty.io.ManagedSelector@558bdf1f id=0 keys=63 selected=0
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,912 DEBUG io.ManagedSelector: Selector loop woken up from
select, 0/63 selected
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,913 DEBUG strategy.ExecuteProduceConsume: EPC
Pend/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@5c4d391a dispatch
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,913 DEBUG thread.QueuedThreadPool: queue EPC
Pend/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@5c4d391a
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,913 DEBUG io.SelectChannelEndPoint: changeInterests
p=false 0->1 for
SelectChannelEndPoint@77d3aac7\{/10.0.1.17:48699<->8089,Open,in,out,FI,-,0/10000,HttpConnection@7d134d7f}{io=0/1,kio=0,kro=0}
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,914 DEBUG conn.DefaultClientConnection: Sending request:
GET /proxy/application_1519904229414_0001/ HTTP/1.1
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,914 DEBUG http.wire: >> "Accept-Encoding: gzip,
deflate[\r][\n]"
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,914 DEBUG http.wire: >> "Host: proxyserver:8089[\r][\n]"
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,914 DEBUG http.headers: >> GET
/proxy/application_1519904229414_0001/ HTTP/1.1
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,915 DEBUG io.FillInterest:
FillInterest@301a7eba\{true,AC.ReadCB@7d134d7f{HttpConnection@7d134d7f[SelectChannelEndPoint@77d3aac7{/10.0.1.17:48699<->8089,Open,in,out,FI,-,2/10000,HttpConnection@7d134d7f}{io=1/0,kio=1,kro=1}][p=HttpParser\{s=START,0
of
0},g=HttpGenerator@17fc5588\{s=START},c=HttpChannelOverHttp@54440a5\{r=0,c=false,a=IDLE,uri=null}]}}
fillable
AC.ReadCB@7d134d7f\{HttpConnection@7d134d7f[SelectChannelEndPoint@77d3aac7{/10.0.1.17:48699<->8089,Open,in,out,FI,-,2/10000,HttpConnection@7d134d7f}{io=1/0,kio=1,kro=1}][p=HttpParser\{s=START,0
of
0},g=HttpGenerator@17fc5588\{s=START},c=HttpChannelOverHttp@54440a5\{r=0,c=false,a=IDLE,uri=null}]}
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,915 DEBUG io.ChannelEndPoint: filled 426
SelectChannelEndPoint@77d3aac7\{/10.0.1.17:48699<->8089,Open,in,out,-,-,2/10000,HttpConnection@7d134d7f}{io=1/0,kio=1,kro=1}
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,915 DEBUG server.HttpConnection:
HttpConnection@7d134d7f[SelectChannelEndPoint@77d3aac7\{/10.0.1.17:48699<->8089,Open,in,out,-,-,0/10000,HttpConnection@7d134d7f}{io=1/0,kio=1,kro=1}][p=HttpParser\{s=START,0
of
0},g=HttpGenerator@17fc5588\{s=START},c=HttpChannelOverHttp@54440a5\{r=0,c=false,a=IDLE,uri=null}]
filled 426 HeapByteBuffer@5b3c5d02[p=0,l=426,c=8192,r=426]=\{<<<GET
/proxy/applic... Keep-Alive\r\n\r\n>>>Accept-Language:
...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,915 DEBUG http.HttpParser: SPACE2 --> REQUEST_VERSION
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,915 DEBUG http.HttpParser: HEADER_IN_VALUE --> HEADER
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,915 DEBUG http.HttpParser: HEADER_IN_VALUE --> HEADER
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,915 DEBUG http.HttpParser: HEADER --> HEADER_VALUE
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,915 DEBUG server.HttpConnection:
HttpConnection@7d134d7f[SelectChannelEndPoint@77d3aac7\{/10.0.1.17:48699<->8089,Open,in,out,-,-,0/10000,HttpConnection@7d134d7f}{io=1/0,kio=1,kro=1}][p=HttpParser\{s=END,0
of
0},g=HttpGenerator@17fc5588\{s=START},c=HttpChannelOverHttp@54440a5\{r=1,c=false,a=IDLE,uri=//proxyserver:8089/proxy/application_1519904229414_0001/}]
parsed true HttpParser\{s=END,0 of 0}
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,916 DEBUG server.session: session=null
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,916 DEBUG servlet.ServletHandler: call filter
static_user_filter
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,918 DEBUG io.ManagedSelector: Selector loop waiting on
select
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,929 DEBUG webproxy.WebAppProxyServlet: local InetAddress
for proxy host: proxyserver/10.0.1.17
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,930 DEBUG conn.BasicClientConnectionManager: Get
connection for route proxyserver/10.0.1.17->{}->http://jupyter:4040
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,933 DEBUG protocol.RequestTargetAuthentication: Target
auth state: UNCHALLENGED
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,934 DEBUG conn.DefaultClientConnection: Sending request:
GET / HTTP/1.1
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,934 DEBUG http.wire: >> "Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8[\r][\n]"
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,934 DEBUG http.wire: >> "User-Agent: Mozilla/5.0 (Windows
NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/64.0.3282.167 Safari/537.36[\r][\n]"
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,934 DEBUG http.wire: >> "Host: jupyter:4040[\r][\n]"
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,934 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]"
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,934 DEBUG http.headers: >> Connection: Keep-Alive
March 1st 2018, 12:38:50.000
/hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr
| 2018-03-01 11:38:49,937 DEBUG http.headers: << Location:
http://proxyserver:8089/proxy/application_1519904229414_0001/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
