This is an automated email from the ASF dual-hosted git repository.
chengpan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new 54687e0af [KYUUBI #6206] Update comments for job group id of Spark
broadcast jobs
54687e0af is described below
commit 54687e0af71f9029d88ecdbb5fb100f19506d183
Author: Cheng Pan <[email protected]>
AuthorDate: Mon Mar 25 17:15:38 2024 +0800
[KYUUBI #6206] Update comments for job group id of Spark broadcast jobs
# :mag: Description
Prior SPARK-43952 (3.5.0), broadcast jobs uses a different group id, so we
can cancel those broadcast jobs. See more details in SPARK-20774 (3.0.0)
## Types of changes :bookmark:
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
## Test Plan ๐งช
Only touch comment, review is sufficient.
---
# Checklist ๐
- [x] This patch was not authored or co-authored using [Generative
Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6206 from pan3793/spark-cancel.
Closes #6206
6a4304d35 [Cheng Pan] fix
a242efab7 [Cheng Pan] Update comments for job group id of Spark broadcast
jobs
Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
---
.../org/apache/kyuubi/engine/spark/operation/SparkOperation.scala | 2 ++
.../main/scala/org/apache/spark/kyuubi/SQLOperationListener.scala | 6 ++----
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git
a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/SparkOperation.scala
b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/SparkOperation.scala
index ec9db84e4..d6d68da0c 100644
---
a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/SparkOperation.scala
+++
b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/SparkOperation.scala
@@ -184,6 +184,8 @@ abstract class SparkOperation(session: Session)
// We should use Throwable instead of Exception since
`java.lang.NoClassDefFoundError`
// could be thrown.
case e: Throwable =>
+ // Prior SPARK-43952 (3.5.0), broadcast jobs uses a different group id,
so we can not
+ // cancel those broadcast jobs. See more details in SPARK-20774 (3.0.0)
if (cancel && !spark.sparkContext.isStopped)
spark.sparkContext.cancelJobGroup(statementId)
withLockRequired {
val errMsg = Utils.stringifyException(e)
diff --git
a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SQLOperationListener.scala
b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SQLOperationListener.scala
index 4342c0c7f..11c7225b3 100644
---
a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SQLOperationListener.scala
+++
b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SQLOperationListener.scala
@@ -72,10 +72,8 @@ class SQLOperationListener(
def getExecutionId: Option[Long] = executionId
- // For broadcast, Spark will introduce a new runId as SPARK_JOB_GROUP_ID,
see:
- // https://github.com/apache/spark/pull/24595, So we will miss these logs.
- // TODO: Fix this until the below ticket resolved
- // https://issues.apache.org/jira/browse/SPARK-34064
+ // Prior SPARK-43952 (3.5.0), broadcast jobs uses a different group id, so
we will
+ // miss those logs. See more details in SPARK-20774 (3.0.0)
private def sameGroupId(properties: Properties): Boolean = {
properties != null && properties.getProperty(KYUUBI_STATEMENT_ID_KEY) ==
operationId
}
