GitHub user rvesse opened a pull request:
https://github.com/apache/spark/pull/22215
[SPARK-25222][K8S][WIP] Improve container status logging
## What changes were proposed in this pull request?
Currently when running Spark on Kubernetes a logger is run by the client
that watches the K8S API for events related to the Driver pod and logs them.
However for the container status aspect of the logging this simply dumps the
raw object which is not human readable e.g.
This is despite the fact that the logging class in question actually has
methods to pretty print this information but only invokes these at the end of a
job.
This PR improves the logging to always use the pretty printing methods,
additionally modifying them to include further useful information provided by
the K8S API.
A similar issue also exists when tasks are lost that will be addressed by
further commits to this PR
- [x] Improved `LoggingPodStatusWatcher`
- [ ] Improved container status on task failure
## How was this patch tested?
Built and launched jobs with the updated Spark client and observed the new
human readable output:
Suggested reviewers: @liyinan926 @mccheah
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rvesse/spark SPARK-25222
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22215.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22215
----
commit ebcbf05b212937aad21742bf14e592ffc1b14383
Author: Rob Vesse <rvesse@...>
Date: 2018-08-24T09:32:14Z
[SPARK-25222][K8S] Improve container status logging
Actually log human readable container status information rather than
dumping the raw status object returned by the K8S API
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
