xkrogen commented on a change in pull request #30096:
URL: https://github.com/apache/spark/pull/30096#discussion_r517627254
##########
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
##########
@@ -1080,9 +1085,9 @@ private[spark] class Client(
// If DEBUG is enabled, log report details every iteration
// Otherwise, log them every time the application changes state
if (log.isDebugEnabled) {
- logDebug(formatReportDetails(report))
+ logDebug(formatReportDetails(report,
getDriverLogsLink(report.getApplicationId)))
} else if (lastState != state) {
- logInfo(formatReportDetails(report))
+ logInfo(formatReportDetails(report,
getDriverLogsLink(report.getApplicationId)))
Review comment:
Sure, added a configuration for this.
You can't necessarily just do it a single time, since with different
application attempts you can get a different driver link. Given that there is
already logic in place to only print the report when the status changes, this
seems like a pretty low cost. We already poll the RM API to get an
`ApplicationReport` once per second (by default). By comparison, this new logic
will cause the `appattempts` API to get hit just a few times over the entire
lifetime of the application (`ACCEPTED`, `RUNNING`, `FINISHED`). So the
relative cost is very low vs. existing logic (up to hundreds or thousands of
calls to get application report vs. 3-ish calls to get driver log links).
This assumes that you don't have DEBUG logging enabled. One thing we can
consider is only updating the driver logs link if `lastState != state`,
regardless of the log level.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
