juliuszsompolski commented on code in PR #41005:
URL: https://github.com/apache/spark/pull/41005#discussion_r1200924867
##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/ClientE2ETestSuite.scala:
##########
@@ -867,6 +872,71 @@ class ClientE2ETestSuite extends RemoteSparkSession with
SQLHelper {
assert(!df.filter(df("_2").endsWith(suffix)).isEmpty)
}
}
+
+ test("interrupt all - background queries, foreground interrupt") {
Review Comment:
@LuciferYang I tried reproducing in `UserDefinedFunctionE2ETestSuite` with
`test("Dataset typed map")`
I tried different variants like
```
test("Dataset typed map") {
val rows = spark.range(10).map(n => { Thread.sleep(1000); n /
2})(PrimitiveLongEncoder).collectAsList()
assert(rows == Arrays.asList[Long](0, 0, 1, 1, 2, 2, 3, 3, 4, 4))
test("Dataset typed map") {
val rows = spark.range(10).map(n => { Thread.sleep(1000); n /
2})(PrimitiveLongEncoder).collect()
assert(rows == Arrays.asList[Long](0, 0, 1, 1, 2, 2, 3, 3, 4, 4))
}
test("Dataset typed map") {
implicit val ec = ExecutionContext.global
val f = Future { spark.range(10).map(n => n /
2)(PrimitiveLongEncoder).collectAsList() }
val rows = Await.result(f, 30.seconds)
assert(rows == Arrays.asList[Long](0, 0, 1, 1, 2, 2, 3, 3, 4, 4))
}
test("Dataset typed map") {
implicit val ec = ExecutionContext.global
val f = Future { spark.range(10).map(n => { Thread.sleep(1000); n / 2
})(PrimitiveLongEncoder).collectAsList() }
val rows = Await.result(f, 30.seconds)
assert(rows == Arrays.asList[Long](0, 0, 1, 1, 2, 2, 3, 3, 4, 4))
}
```
and they all work there.
but if I copy these two `interrupt all` tests to
`UserDefinedFunctionE2ETestSuite`, they also work for me there:
```
UserDefinedFunctionE2ETestSuite:
- Dataset typed filter
- Dataset typed filter - java
- Dataset typed map
- interrupt all - background queries, foreground interrupt
- interrupt all - foreground queries, background interrupt
- filter with condition
...
```
I'm stumped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
