[jira] [Assigned] (SPARK-25222) Spark on Kubernetes Pod Watcher dumps raw container status
[
https://issues.apache.org/jira/browse/SPARK-25222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcelo Vanzin reassigned SPARK-25222:
--
Assignee: Rob Vesse
> Spark on Kubernetes Pod Watcher dumps raw container status
> --
>
> Key: SPARK-25222
> URL: https://issues.apache.org/jira/browse/SPARK-25222
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
>Affects Versions: 2.3.0, 2.3.1
>Reporter: Rob Vesse
>Assignee: Rob Vesse
>Priority: Minor
> Fix For: 3.0.0
>
>
> Spark on Kubernetes provides logging of the pod/container status as a monitor
> of the job progress. However the logger just dumps the raw container status
> object leading to fairly unreadable output like so:
> {noformat}
> 18/08/24 09:03:27 INFO LoggingPodStatusWatcherImpl: State changed, new state:
>pod name: spark-groupby-1535101393784-driver
>namespace: default
>labels: spark-app-selector -> spark-47f7248122b9444b8d5fd3701028a1e8,
> spark-role -> driver
>pod uid: 88de6467-a77c-11e8-b9da-a4bf0128b75b
>creation time: 2018-08-24T09:03:14Z
>service account name: spark
>volumes: spark-local-dir-1, spark-conf-volume, spark-token-kjxkv
>node name: tab-cmp4
>start time: 2018-08-24T09:03:14Z
>container images: rvesse/spark:latest
>phase: Running
>status:
> [ContainerStatus(containerID=docker://23ae58571f59505e837dca40455d0347fb90e9b88e2a2b145a38e2919fceb447,
> image=rvesse/spark:latest,
> imageID=docker-pullable://rvesse/spark@sha256:92abf0b718743d0f5a26068fc94ec42233db0493c55a8570dc8c851c62a4bc0a,
> lastState=ContainerState(running=null, terminated=null, waiting=null,
> additionalProperties={}), name=spark-kubernetes-driver, ready=true,
> restartCount=0,
> state=ContainerState(running=ContainerStateRunning(startedAt=Time(time=2018-08-24T09:03:26Z,
> additionalProperties={}), additionalProperties={}), terminated=null,
> waiting=null, additionalProperties={}), additionalProperties={})]
> {noformat}
> The {{LoggingPodStatusWatcher}} actually already includes code to nicely
> format this information but only invokes it at the end of the job:
> {noformat}
> 18/08/24 09:04:07 INFO LoggingPodStatusWatcherImpl: Container final statuses:
> Container name: spark-kubernetes-driver
> Container image: rvesse/spark:latest
> Container state: Terminated
> Exit code: 0
> {noformat}
> It would be nice if we continually used the nice formatting throughout the
> logging.
> We already have patched this on our internal fork and will upstream a fix
> shortly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-25222) Spark on Kubernetes Pod Watcher dumps raw container status
[
https://issues.apache.org/jira/browse/SPARK-25222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-25222:
Assignee: Apache Spark
> Spark on Kubernetes Pod Watcher dumps raw container status
> --
>
> Key: SPARK-25222
> URL: https://issues.apache.org/jira/browse/SPARK-25222
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
>Affects Versions: 2.3.0, 2.3.1
>Reporter: Rob Vesse
>Assignee: Apache Spark
>Priority: Minor
>
> Spark on Kubernetes provides logging of the pod/container status as a monitor
> of the job progress. However the logger just dumps the raw container status
> object leading to fairly unreadable output like so:
> {noformat}
> 18/08/24 09:03:27 INFO LoggingPodStatusWatcherImpl: State changed, new state:
>pod name: spark-groupby-1535101393784-driver
>namespace: default
>labels: spark-app-selector -> spark-47f7248122b9444b8d5fd3701028a1e8,
> spark-role -> driver
>pod uid: 88de6467-a77c-11e8-b9da-a4bf0128b75b
>creation time: 2018-08-24T09:03:14Z
>service account name: spark
>volumes: spark-local-dir-1, spark-conf-volume, spark-token-kjxkv
>node name: tab-cmp4
>start time: 2018-08-24T09:03:14Z
>container images: rvesse/spark:latest
>phase: Running
>status:
> [ContainerStatus(containerID=docker://23ae58571f59505e837dca40455d0347fb90e9b88e2a2b145a38e2919fceb447,
> image=rvesse/spark:latest,
> imageID=docker-pullable://rvesse/spark@sha256:92abf0b718743d0f5a26068fc94ec42233db0493c55a8570dc8c851c62a4bc0a,
> lastState=ContainerState(running=null, terminated=null, waiting=null,
> additionalProperties={}), name=spark-kubernetes-driver, ready=true,
> restartCount=0,
> state=ContainerState(running=ContainerStateRunning(startedAt=Time(time=2018-08-24T09:03:26Z,
> additionalProperties={}), additionalProperties={}), terminated=null,
> waiting=null, additionalProperties={}), additionalProperties={})]
> {noformat}
> The {{LoggingPodStatusWatcher}} actually already includes code to nicely
> format this information but only invokes it at the end of the job:
> {noformat}
> 18/08/24 09:04:07 INFO LoggingPodStatusWatcherImpl: Container final statuses:
> Container name: spark-kubernetes-driver
> Container image: rvesse/spark:latest
> Container state: Terminated
> Exit code: 0
> {noformat}
> It would be nice if we continually used the nice formatting throughout the
> logging.
> We already have patched this on our internal fork and will upstream a fix
> shortly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-25222) Spark on Kubernetes Pod Watcher dumps raw container status
[
https://issues.apache.org/jira/browse/SPARK-25222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-25222:
Assignee: (was: Apache Spark)
> Spark on Kubernetes Pod Watcher dumps raw container status
> --
>
> Key: SPARK-25222
> URL: https://issues.apache.org/jira/browse/SPARK-25222
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
>Affects Versions: 2.3.0, 2.3.1
>Reporter: Rob Vesse
>Priority: Minor
>
> Spark on Kubernetes provides logging of the pod/container status as a monitor
> of the job progress. However the logger just dumps the raw container status
> object leading to fairly unreadable output like so:
> {noformat}
> 18/08/24 09:03:27 INFO LoggingPodStatusWatcherImpl: State changed, new state:
>pod name: spark-groupby-1535101393784-driver
>namespace: default
>labels: spark-app-selector -> spark-47f7248122b9444b8d5fd3701028a1e8,
> spark-role -> driver
>pod uid: 88de6467-a77c-11e8-b9da-a4bf0128b75b
>creation time: 2018-08-24T09:03:14Z
>service account name: spark
>volumes: spark-local-dir-1, spark-conf-volume, spark-token-kjxkv
>node name: tab-cmp4
>start time: 2018-08-24T09:03:14Z
>container images: rvesse/spark:latest
>phase: Running
>status:
> [ContainerStatus(containerID=docker://23ae58571f59505e837dca40455d0347fb90e9b88e2a2b145a38e2919fceb447,
> image=rvesse/spark:latest,
> imageID=docker-pullable://rvesse/spark@sha256:92abf0b718743d0f5a26068fc94ec42233db0493c55a8570dc8c851c62a4bc0a,
> lastState=ContainerState(running=null, terminated=null, waiting=null,
> additionalProperties={}), name=spark-kubernetes-driver, ready=true,
> restartCount=0,
> state=ContainerState(running=ContainerStateRunning(startedAt=Time(time=2018-08-24T09:03:26Z,
> additionalProperties={}), additionalProperties={}), terminated=null,
> waiting=null, additionalProperties={}), additionalProperties={})]
> {noformat}
> The {{LoggingPodStatusWatcher}} actually already includes code to nicely
> format this information but only invokes it at the end of the job:
> {noformat}
> 18/08/24 09:04:07 INFO LoggingPodStatusWatcherImpl: Container final statuses:
> Container name: spark-kubernetes-driver
> Container image: rvesse/spark:latest
> Container state: Terminated
> Exit code: 0
> {noformat}
> It would be nice if we continually used the nice formatting throughout the
> logging.
> We already have patched this on our internal fork and will upstream a fix
> shortly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
