Veera created SPARK-30357:
-----------------------------
Summary: SparkContext: Invoking stop() from shutdown hook
Key: SPARK-30357
URL: https://issues.apache.org/jira/browse/SPARK-30357
Project: Spark
Issue Type: Bug
Components: Deploy, Kubernetes, MLlib, PySpark
Affects Versions: 2.4.4
Environment: pivotal kubernetes container service PKS having 3master
nodes and 3 worker nodes
OS : ubuntu for k8-s cluster
Reporter: Veera
I'm getting below error while running spark-submit job in kubernetes , i didn't
get any specific error
19/12/26 07:38:18 INFO SparkContext: Invoking stop() from shutdown hook
19/12/26 07:38:18 INFO SparkUI: Stopped Spark web UI at
http://spark-ml-test-1577345808987-driver-svc.spark-jobs.svc:4040
19/12/26 07:38:18 INFO KubernetesClusterSchedulerBackend: Shutting down all
executors
19/12/26 07:38:18 INFO
KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each
executor to shut down
19/12/26 07:38:18 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has
been closed (this is expected if the application is shutting down.)
19/12/26 07:38:18 INFO MapOutputTrackerMasterEndpoint:
MapOutputTrackerMasterEndpoint stopped!
19/12/26 07:38:18 INFO MemoryStore: MemoryStore cleared
19/12/26 07:38:18 INFO BlockManager: BlockManager stopped
19/12/26 07:38:18 INFO BlockManagerMaster: BlockManagerMaster stopped
19/12/26 07:38:18 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
OutputCommitCoordinator stopped!
19/12/26 07:38:18 INFO SparkContext: Successfully stopped SparkContext
19/12/26 07:38:18 INFO ShutdownHookManager: Shutdown hook called
19/12/26 07:38:18 INFO ShutdownHookManager: Deleting directory
/var/data/spark-ec98f5d3-022d-4eae-87aa-c23a96c77555/spark-b757b85f-3709-4215-8208-95db840a294a/pyspark-665955c9-0dc6-49ee-9b57-0392be430f10
19/12/26 07:38:18 INFO ShutdownHookManager: Deleting directory
/tmp/spark-fa047fa8-b573-44aa-ab65-e235fb0d5b09
19/12/26 07:38:18 INFO ShutdownHookManager: Deleting directory
/var/data/spark-ec98f5d3-022d-4eae-87aa-c23a96c77555/spark-b757b85f-3709-4215-8208-95db840a294a
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
