felipepessoto commented on issue #9296: URL: https://github.com/apache/gluten/issues/9296#issuecomment-4693162104
I think Delta don't have many unit tests checking the query plan. I need to think how we can get a manageable report of errors, but looking some random failures they look real issues, like https://github.com/apache/gluten/actions/runs/27376177957/job/80905117213?pr=12278 incorrect results: ``` [info] - schema evolution - extra nested column in source - update, isPartitioned=true *** FAILED *** [info] == Results == [info] !== Correct Answer - 2 == == Spark Answer - 2 == [info] struct<key:int,x:struct<a:int,c:int,b:int>> struct<key:int,x:struct<a:int,c:int,b:int>> [info] ![1,[10,1000,100]] [1,[10,100,100]] [info] [2,[2,2000,null]] [2,[2,2000,null]] (QueryTest.scala:273) ``` ``` [info] - Time travel with schema changes [info] - data skipping still works with time travel *** FAILED *** [info] 426 was not less than 426 (DeltaHistoryManagerSuite.scala:476) [info] org.scalatest.exceptions.TestFailedException: [info] at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472) [info] at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471) ``` failures: ``` [info] - create table *** FAILED *** [info] org.apache.spark.SparkException: [TASK_WRITE_FAILED] Task failed while writing rows to file:/__w/gluten/gluten/delta/spark/spark-warehouse/org.apache.spark.sql.delta.DeltaArbitraryColumnNameSuite/source. SQLSTATE: 58030 [info] at org.apache.spark.sql.errors.QueryExecutionErrors$.taskFailedWhileWritingRowsError(QueryExecutionErrors.scala:814) [info] at org.apache.spark.sql.delta.files.GlutenDeltaFileFormatWriter$.executeTask(GlutenDeltaFileFormatWriter.scala:505) .... [info] Cause: java.util.concurrent.ExecutionException: org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Exception: VeloxUserError [info] Error Source: USER [info] Error Code: INVALID_ARGUMENT [info] Reason: Field not found: col-3c1789ba-5bac-4efe-b631-cbd8b227dfbd. Available fields are: c with special chars ,;{}() [info] =, d with special chars ,;{}() [info] =. [info] Retriable: False [info] Context: Top-level Expression: (node_value-stream:0_1).col-3c1789ba-5bac-4efe-b631-cbd8b227dfbd [info] Function: getChildIdx [info] File: /work/ep/build-velox/build/velox_ep/velox/type/Type.cpp [info] Line: 598 [info] Stack trace: [info] # 0 [info] # 1 [info] # 2 ``` Failures instead of fallback: ``` [info] - Variant type *** FAILED *** [info] org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1489.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1489.0 (TID 1267) (1c419bb89a89 executor driver): java.lang.UnsupportedOperationException: Unsupported data type: variant [info] at org.apache.spark.sql.utils.SparkArrowUtil$.toArrowType(SparkArrowUtil.scala:60) [info] at org.apache.spark.sql.utils.SparkArrowUtil$.toArrowField(SparkArrowUtil.scala:137) [info] at org.apache.spark.sql.utils.SparkArrowUtil$.$anonfun$toArrowSchema$1(SparkArrowUtil.scala:167) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
