HyukjinKwon commented on PR #59283:
URL: https://github.com/apache/spark/pull/59283#issuecomment-6181072263
Thanks, verified on the current head. CSE now descends into a With body and
into the definitions that body always reaches, and supportedExpression still
rejects candidates that hold unbound references or CommonExpressionDefs. The
new tests cover a shared struct input and a value shared from a With body, plus
the conditional, unused, fallback, lambda and shadowed cases. There is one
problem with the new traversal. For a nested With, alwaysEvaluatedDefinitions
walks the inner body once through childrenToRecurse(nested) and again in
collect, so the work doubles at every nesting level. Deep nested-path updates
on non-nullable structs now make subexpression elimination exponential in the
path depth. Details are in this review.
<!-- SPARK_DEV_REVIEW_REPLY
{"feedback_id":"conversation:6178417779","thread_id":"conversation:6178417779","verdict_sha256":"da4e342b83744e2a784cb95fd720aa39782cab886b1272d5195846b31f427d38"}
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
