andygrove opened a new issue, #6643:
URL: https://github.com/apache/datafusion-comet/issues/6643
### Describe the bug
`CometIcebergWriteActionSuite` "a failed write job deletes the data files of
tasks that completed" (added in #5663) fails intermittently in CI. We've seen
both the `native=true` and `native=false` variants fail. The injected `boom`
failure never surfaces. Instead, the task that should throw it gives up waiting
in `JobAbortGate.awaitOthers()`:
```
- a failed write job deletes the data files of tasks that completed
(native=false) *** FAILED ***
... was false expected the injected task failure to surface, got
Some(org.apache.spark.SparkException: Job aborted due to stage failure: Task 1
in stage 397.0 failed 1 times, most recent failure: Lost task 1.0 in stage
397.0 (TID 524) (... executor driver): org.apache.spark.SparkException:
[FAILED_EXECUTE_UDF] Failed to execute user defined function
(CometIcebergWriteActionSuite$$Lambda$15258/0x0000000843a2c840: (int) => int).
at
org.apache.spark.sql.errors.QueryExecutionErrors$.failedExecuteUserDefinedFunctionError(QueryExecutionErrors.scala:217)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificCometBatchKernel.process(Unknown
Source)
at
org.apache.comet.udf.codegen.CometScalaUDFCodegen.liftedTree1$1(CometScalaUDFCodegen.scala:130)
at
org.apache.comet.udf.codegen.CometScalaUDFCodegen.evaluate(CometScalaUDFCodegen.scala:129)
at
org.apache.comet.udf.CometUdfBridge.evaluateInternal(CometUdfBridge.java:231)
at
org.apache.comet.udf.CometUdfBridge.evaluate(CometUdfBridge.java:144)
Caused by: java.lang.IllegalStateException: JobAbortGate: the other tasks
never finished
at
org.apache.comet.JobAbortGate$.awaitOthers(CometIcebergWriteActionSuite.scala:2766)
at
org.apache.comet.CometIcebergWriteActionSuite.$anonfun$new$292(CometIcebergWriteActionSuite.scala:1855)
```
This was seen on `branch-1.1`, where the test and `JobAbortGate` are the
same as on `main`.
### Likely cause
The `boom_after_others` UDF blocks in `JobAbortGate.awaitOthers()` (for up
to 2 minutes) until the other two tasks have succeeded. As the stack trace
shows, Comet evaluates the UDF through `CometUdfBridge`, called from native
code. So the blocked task holds a thread of the shared native (Tokio) runtime
while it waits. If that runtime has too few free workers for the other two
tasks to make progress, they never finish, and the gate times out before `boom`
is thrown.
The runtime's size and load depend on which suites ran earlier in the same
JVM. That would explain why the test passes on its own and fails only sometimes
in the full suite. This cause is inferred from the trace and hasn't been
confirmed.
### Steps to reproduce
The failure is intermittent. It shows up when running the whole JVM test
suite (for example `make test-jvm` with a Spark 3.4 profile), not when running
`CometIcebergWriteActionSuite` on its own.
### Expected behavior
The test passes regardless of which suites ran before it in the same JVM.
The test shouldn't block a native runtime thread while it waits for other
tasks. For example, it could do the waiting outside the UDF that Comet
evaluates natively, or make sure no native worker thread holds the wait.
### Additional context
The test was added in #5663.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
