tgravescs commented on a change in pull request #30096:
URL: https://github.com/apache/spark/pull/30096#discussion_r517636953
##########
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
##########
@@ -1577,6 +1618,19 @@ private object Client extends Logging {
writer.flush()
out.closeEntry()
}
+
+ private[yarn] def parseAppAttemptsJsonResponse(jsonString: String):
Option[(String, String)] = {
+ val objectMapper = new ObjectMapper()
+ // If JSON response is malformed somewhere along the way, MissingNode will
be returned,
+ // which allows for safe continuation of chaining. The `elements()` call
will be empty,
+ // and None will get returned.
+ objectMapper.readTree(jsonString)
+ .path("appAttempts").path("appAttempt")
+ .elements().asScala.toList.takeRight(1).headOption
+ .map(_.path("logsLink").asText(""))
+ .filterNot(_ == "")
+ .map(baseUrl => (s"$baseUrl/stdout?start=-4096",
s"$baseUrl/stderr?start=-4096"))
Review comment:
I just remembered there is some dealing with container logs in
YarnContainerInfoHelper. while its different since not going thru rest api it
might be nice to at least have constants for the "/stdout?start=-4096" and
"stderr?start=-4096" so not hardcoded in multiple places
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
