Dean Chen created SPARK-6868:
--------------------------------
Summary: Container link broken on Spark UI Executors page when
YARN is set to HTTPS_ONLY
Key: SPARK-6868
URL: https://issues.apache.org/jira/browse/SPARK-6868
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.3.0, 1.2.1, 1.2.0, 1.1.1, 1.1.0
Reporter: Dean Chen
The stdout and stderr log links on the executor page will use the http://
prefix even if the node manager does not support http and only https via
setting yarn.http.policy=HTTPS_ONLY.
Unfortunately the unencrypted http link in that case does not return a 404 but
a binary file containing random binary chars. This causes a lot of confusion
for the end user since it seems like the log file exists and is just filled
with garbage. (see attached screenshot)
The fix is to prefix container log links with https:// instead of http:// if
yarn.http.policy=HTTPS_ONLY. YARN's job page has this exact logic as seen here:
https://github.com/apache/hadoop/blob/e1109fb65608a668cd53dc324dadc6f63a74eeb9/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JobBlock.java#L108
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
