Visorgood commented on PR #19017:
URL: https://github.com/apache/iceberg/pull/19017#issuecomment-6987188390

   Thanks for taking a look, @singhpk234. This comes from #18953, which 
@manuzhang filed while verifying Spark 4.2.1 (#18294). Spark has its own test 
for SPARK-60403 in `WholeStageCodegenSuite`. The Iceberg-specific part is 
MERGE: it runs through `MergeRowsExec` for both copy-on-write and 
merge-on-read, and when the generated code doesn't compile, Spark falls back 
from whole-stage codegen without failing the query, so the MERGE just gets 
slower.
   
   I agree the join tests and the checks on generated call names test Spark 
internals. I can drop `TestOuterJoinCodegen` and the `TestBase` helper and keep 
only the MERGE test, which runs with `spark.sql.codegen.fallback` disabled and 
checks the merged rows; it fails on Spark 4.2.0 and passes on 4.2.1. Or I can 
close this if Spark's coverage is enough. @manuzhang, what do you think?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to