Spark on K8s , some applications ended ungracefully

[Pralabh Kumar]

Hi Spark Team

Some of my spark applications on K8s ended with the below error . These
applications though completed successfully (as per the event log
SparkListenerApplicationEnd event at the end)
stil have even files with .inprogress. This causes the application to be
shown as inprogress in SHS.

Spark v : 3.0.1



22/03/31 08:33:34 WARN ShutdownHookManager: ShutdownHook '$anon$2' timeout,
java.util.concurrent.TimeoutException

java.util.concurrent.TimeoutException

                at java.util.concurrent.FutureTask.get(FutureTask.java:205)

                at
org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:68)

22/03/31 08:33:34 WARN SparkContext: Ignoring Exception while stopping
SparkContext from shutdown hook

java.lang.InterruptedException

                at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)

                at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)

                at
java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1475)

                at
org.apache.spark.util.ThreadUtils$.shutdown(ThreadUtils.scala:348)





Please let me know if there is a solution for it ..

Regards

Pralabh Kumar