andygrove commented on PR #5051:
URL:
https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5209716289
> Tested on Spark4.0 against current branch. Using PySpark Got this error on
the same pipeline that was testing earlier. And I can clearly see a regression
in performance. Same job with `.cache()` on default Spark took <10 min, Comet
ran >15min.
>
> Here is error:
>
> ```
> Py4JJavaError: An error occurred while calling o12641.createOrReplace.
> : org.apache.spark.SparkException: Job aborted due to stage failure: Task
92 in stage 27.0 failed 4 times, most recent failure: Lost task 92.3 in stage
27.0 (TID 15624) (1.1.1.1 executor 76): org.apache.comet.CometNativeException:
Invalid argument error: column types must match schema types, expected
List(Struct("col_1": Utf8, "col_2": Boolean, "col_3": Boolean)) but found
List(Struct("col_1": Utf8, "col_2": non-null Boolean, "col_3": Boolean)) at
column index 5
> at org.apache.comet.Native.executePlan(Native Method)
> at
org.apache.comet.CometExecIterator.$anonfun$getNextBatch$2(CometExecIterator.scala:155)
> at
org.apache.comet.CometExecIterator.$anonfun$getNextBatch$2$adapted(CometExecIterator.scala:154)
> at org.apache.comet.vector.NativeUtil.getNextBatch(NativeUtil.scala:189)
> at
org.apache.comet.CometExecIterator.$anonfun$getNextBatch$1(CometExecIterator.scala:154)
> at org.apache.comet.Tracing$.withTrace(Tracing.scala:31)
> at
org.apache.comet.CometExecIterator.getNextBatch(CometExecIterator.scala:152)
> at
org.apache.comet.CometExecIterator.hasNext(CometExecIterator.scala:196)
> ```
>
> (obfuscated column names)
I think the crash may get fixed once
https://github.com/apache/datafusion-comet/pull/5138 is merged. Looking into
perf regression...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
