Tarik Beganović created SPARK-62086:
---------------------------------------
Summary: Support a custom driver log URL in the History Server
Key: SPARK-62086
URL: https://issues.apache.org/jira/browse/SPARK-62086
Project: Spark
Issue Type: Improvement
Components: Spark Core, Web UI
Affects Versions: 4.4.0
Reporter: Tarik Beganović
On the History Server's Executors page, the driver row has no log link when the
driver recorded no log URLs in the event log. This is the case on Kubernetes,
and for standalone and YARN in client mode.
{{spark.history.custom.executor.log.url}} can't fill this gap. It rebuilds
links from the attributes recorded in the event log, and a driver without log
URLs has no attributes either: {{AppStatusListener}} only records driver
attributes together with driver log URLs. So the driver row stays empty, even
when an external log service keeps the driver's log.
*Proposal*
Add History Server configs for the driver row, as the driver counterpart of
{{spark.history.custom.executor.log.url}}:
* {{spark.history.custom.driver.log.url}}: a URL pattern for the driver row. It
uses the driver attributes recorded in the event log. If the driver recorded no
log URLs, {{APP_ID}} is taken from the application info. If a pattern variable
can't be resolved, the recorded driver log URLs are kept, as for executors.
* {{spark.history.custom.driver.log.url.applyIncompleteApplication}}: whether
to apply the pattern to incomplete applications. Defaults to the value of
{{spark.history.custom.executor.log.url.applyIncompleteApplication}}.
When {{spark.history.custom.driver.log.url}} is not set, the behavior is
unchanged.
Example:
{noformat}
spark.history.custom.driver.log.url=https://log-server/log?appId={{APP_ID}}&execId=driver
{noformat}
*Relation to SPARK-47573*
SPARK-47573 lets a Kubernetes driver record its own log URL
({{spark.ui.custom.driver.log.url}}), which also covers the live UI. This
change is complementary: it covers event logs without driver log URLs,
including logs written before that change or by Spark versions without it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]