grorge123 commented on issue #5781: URL: https://github.com/apache/datafusion-comet/issues/5781#issuecomment-6017588599
@dwsmith1983 @mbutrovich agreed: let's take the dispatcher route from #5867 for `size`, `array_append`, `arrays_zip` and `map_from_arrays`. It already checks determinism before the `LAST_WIN` opt-in and gates the Spark 3.x `array_append` item, so I'll drop those four checks from #5526 when I split it. A few things from #5526 sit next to this and will go in their own changes: the stateful `coalesce` guard (it serializes its non-final children twice, and #5867 doesn't touch it), `arrays_zip` checking its arguments one WHEN at a time so a deterministic later argument isn't evaluated on rows Spark skips, and the handling of contexts where Spark evaluates through `eval`. Thanks for filing #6711 with #6712. I'll drop my duplicate per-occurrence implementation from #5526 and build on #6712. Refusing non-deterministic user functions, `reflect` and subqueries in the dispatcher stays a separate change, and I'll open an issue for it. As you say, narrowing which guards are emitted (`size` with `legacySizeOfNull` on or a non-nullable child, and `array_append` gating the item only when the array can be NULL) can follow either PR. Assisted-by: Claude Code (claude-opus-5-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
