ohad-definity commented on PR #59706: URL: https://github.com/apache/spark/pull/59706#issuecomment-6384026548
@IgorBerman yes please, that's a great list, and #5 especially: it's the subtlest path of the P1 fix and has no regression test yet. Same route as before (a PR into `ohad-definity/spark` `SPARK-47230`) works well. For #6, a plain comment in the suite's "not narrowed today" style rather than a TODO. Master's `SchemaPruningSuite` (from SPARK-57022/57175/57176) suggests a few more generator-input shapes, all with the same adversarial data: - `posexplode(slice(items, 1, 1))`: should still prune, like `reverse`; - `explode(slice(items, id + 1, 1))`: non-foldable bounds, should not prune through; - `explode(array_compact(team.members))` with null elements: prunes, nulls still dropped; - `explode(transform(filter(items, x -> x.a > 1), x -> x.c))`: the non-path fallback should still prune to `<a, c>`; - nested wrappers like `explode(filter(array_sort(items, cmp), pred))`: the union of both lambdas' reads. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
