[ https://issues.apache.org/jira/browse/SPARK-33185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mridul Muralidharan resolved SPARK-33185. ----------------------------------------- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30096 [https://github.com/apache/spark/pull/30096] > YARN: Print direct links to driver logs alongside application report in > cluster mode > ------------------------------------------------------------------------------------ > > Key: SPARK-33185 > URL: https://issues.apache.org/jira/browse/SPARK-33185 > Project: Spark > Issue Type: Improvement > Components: YARN > Affects Versions: 3.0.1 > Reporter: Erik Krogen > Assignee: Erik Krogen > Priority: Major > Fix For: 3.1.0 > > > Currently when run in {{cluster}} mode on YARN, the Spark {{yarn.Client}} > will print out the application report into the logs, to be easily viewed by > users. For example: > {code} > INFO yarn.Client: > client token: Token { kind: YARN_CLIENT_TOKEN, service: } > diagnostics: N/A > ApplicationMaster host: X.X.X.X > ApplicationMaster RPC port: 0 > queue: default > start time: 1602782566027 > final status: UNDEFINED > tracking URL: http://hostname:8888/proxy/application_<id>/ > user: xkrogen > {code} > Typically, the tracking URL can be used to find the logs of the > ApplicationMaster/driver while the application is running. Later, the Spark > History Server can be used to track this information down, using the > stdout/stderr links on the Executors page. > However, in the situation when the driver crashed _before_ writing out a > history file, the SHS may not be aware of this application, and thus does not > contain links to the driver logs. When this situation arises, it can be > difficult for users to debug further, since they can't easily find their > driver logs. > It is possible to reach the logs by using the {{yarn logs}} commands, but the > average Spark user isn't aware of this and shouldn't have to be. > I propose adding, alongside the application report, some additional lines > like: > {code} > Driver Logs (stdout): > http://hostname:8042/node/containerlogs/container_<id>/xkrogen/stdout?start=-4096 > Driver Logs (stderr): > http://hostname:8042/node/containerlogs/container_<id>/xkrogen/stderr?start=-4096 > {code} > With this information available, users can quickly jump to their driver logs, > even if it crashed before the SHS became aware of the application. This has > the additional benefit of providing a quick way to access driver logs, which > often contain useful information, in a single click (instead of navigating > through the Spark UI). -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org