Tarik Beganović created SPARK-62086:
---------------------------------------

             Summary:  Support a custom driver log URL in the History Server
                 Key: SPARK-62086
                 URL: https://issues.apache.org/jira/browse/SPARK-62086
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core, Web UI
    Affects Versions: 4.4.0
            Reporter: Tarik Beganović


On the History Server's Executors page, the driver row has no log link when the 
driver recorded no log URLs in the event log. This is the case on Kubernetes, 
and for standalone and YARN in client mode.

{{spark.history.custom.executor.log.url}} can't fill this gap. It rebuilds 
links from the attributes recorded in the event log, and a driver without log 
URLs has no attributes either: {{AppStatusListener}} only records driver 
attributes together with driver log URLs. So the driver row stays empty, even 
when an external log service keeps the driver's log.

*Proposal*

Add History Server configs for the driver row, as the driver counterpart of 
{{spark.history.custom.executor.log.url}}:
* {{spark.history.custom.driver.log.url}}: a URL pattern for the driver row. It 
uses the driver attributes recorded in the event log. If the driver recorded no 
log URLs, {{APP_ID}} is taken from the application info. If a pattern variable 
can't be resolved, the recorded driver log URLs are kept, as for executors.
* {{spark.history.custom.driver.log.url.applyIncompleteApplication}}: whether 
to apply the pattern to incomplete applications. Defaults to the value of 
{{spark.history.custom.executor.log.url.applyIncompleteApplication}}.

When {{spark.history.custom.driver.log.url}} is not set, the behavior is 
unchanged.

Example:
{noformat}
spark.history.custom.driver.log.url=https://log-server/log?appId={{APP_ID}}&execId=driver
{noformat}

*Relation to SPARK-47573*

SPARK-47573 lets a Kubernetes driver record its own log URL 
({{spark.ui.custom.driver.log.url}}), which also covers the live UI. This 
change is complementary: it covers event logs without driver log URLs, 
including logs written before that change or by Spark versions without it.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to