malinjawi commented on issue #9003: URL: https://github.com/apache/gluten/issues/9003#issuecomment-5895372728
> [@philo-he](https://github.com/philo-he) [@zhztheplayer](https://github.com/zhztheplayer), [@malinjawi](https://github.com/malinjawi) I wanted to raise this issue as it is the main cause of Delta UT failures, 226 of 963 failures are because of this, besides the fact that it is providing incorrect metrics in the Delta Log. @felipepessoto thanks for the push on this, I picked it up. Root cause is on our side: the Delta project transformer strips every IncrementMetric and the metrics updater credits each one with the operator's output rows, so the counters MERGE keeps inside its CASE WHEN branches get every row. That's also why a no-op MERGE shows changes in CDF. Fix is up in #13158: we only offload an IncrementMetric where Spark would evaluate it once per output row, everything else stays on Spark. DELETE/UPDATE keep their offload, only MERGE's write projection falls back. @rui-mo to your question: the expression is Delta-only, but the bug is Gluten's accounting, no Velox change needed for this step. Counting it natively (so MERGE's projection can go back to native) is the follow-up. @zhztheplayer @philo-he @felipepessoto @zhouyuan would appreciate a review on #13158 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
