[
https://issues.apache.org/jira/browse/SPARK-48717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-48717.
----------------------------------
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 47093
[https://github.com/apache/spark/pull/47093]
> Python foreachBatch streaming query cannot be stopped gracefully after pin
> thread mode is enabled and is running spark queries
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-48717
> URL: https://issues.apache.org/jira/browse/SPARK-48717
> Project: Spark
> Issue Type: New Feature
> Components: PySpark, SS
> Affects Versions: 4.0.0
> Reporter: Wei Liu
> Assignee: Wei Liu
> Priority: Major
> Fix For: 4.0.0
>
>
> Followup of https://issues.apache.org/jira/browse/SPARK-39218
>
> It only considered the InterruptedException is thrown when time.sleep(10) is
> intercepted. But when a spark query is executing:
> {code:java}
> def func(batch_df, batch_id):
> batch_df.sparkSession.range(10000000).write.saveAsTable("oops")
> print(batch_df.count()) {code}
> the actual error would be:
>
> {code:java}
> py4j.protocol.Py4JJavaError: An error occurred while calling
> o2141502.saveAsTable.
> : org.apache.spark.SparkException: Job aborted.
> <REDACTED>
> <REDACTED>
> <REDACTED>
> ...
> at
> org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.$anonfun$run$2(StreamExecution.scala:262)
>
> at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
> at
> org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:262)
>
> *Caused by: java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1000)*
>
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1308)
> {code}
> We should also add consideration to this scenario
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
