Srimantha Rajapaksa created SPARK-33490:
-------------------------------------------
Summary: calling system.exit(-1) from a future does not result in
the driver reflecting the fact that spark app failed.
Key: SPARK-33490
URL: https://issues.apache.org/jira/browse/SPARK-33490
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.4.0
Reporter: Srimantha Rajapaksa
Hi Guys,
when calling system. exit(-1) from within a code block defined for a future
observed that the vm does not exit immediately as you would expect a java app
to do. in fact it seems to exit the future thread and the diver program carries
on as normal. the status of the app when it exits is also misleading. It says
FINISHED instead of FAILED. I have attached code sample see below) that
replicate this issue. is this a spark bug or a "feature" :) any thoughts?
Note: I have been able to work around this by not calling system .exit(-1) from
within a future but this was definitely a gotcha that I thought worthwhile
raising..
{code:java}
// code placeholder
package com.spark.example
import com.typesafe.config.Config
import com.typesafe.scalalogging.Logger
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.SparkSession
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration.DurationInt
import scala.concurrent.{Await, Future}
import scala.util.{Failure, Success, Try}
object SystemExitTestApp extends App {
val appName = "NumberCountSparkApp"
val logger = Logger(getClass)
val sparkMaster = "spark://adl-matspm01:7077"
val ss = SparkSession.builder()
.master(sparkMaster)
.appName(appName).getOrCreate()
logger.info(s"Starting test. thread id ${Thread.currentThread().getId}")
val data = Seq(1, 2, 3, 4)
val rdd: RDD[Int] = ss.sparkContext.parallelize(data)
val future = Future {
logger.info(s"inside future thread id ${Thread.currentThread().getId}")
Try {
throw new Exception("My dummy exception")
rdd.count()
} match {
case Success(_) => {
(true, 1)
}
case Failure(exception) => {
logger.info(s"inside future. Exception ${exception} thread id
${Thread.currentThread().getId}")
(false, 0)
}
}
}
var result: Boolean = true
future onComplete {
case Success(x) => {
logger.info(s"success ${x}")
if (!x._1) {
logger.info(s"exiting in the success block of on complete. thread id
${Thread.currentThread().getId}")
result = x._1
ss.close()
sys.exit(-1)
} else {
logger.info(s"not exiting in the success block of on complete. thread
id ${Thread.currentThread().getId}")
result = x._1
}
}
case Failure(exception) => {
logger.info(s"exiting in the failure block of on complete as exception
occurred ${exception}. thread id ${Thread.currentThread().getId}")
result = false
sys.exit(-1)
}
}
Await.result(future, 3 hours)
logger.info(s"should not see this really if we do lets use the result
outside. thread id ${Thread.currentThread().getId}")
if (!result) {
logger.info(s" inside main block should not see this really. thread id
${Thread.currentThread().getId}")
//sys.exit(-1)
}
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
