ueshin commented on PR #59100: URL: https://github.com/apache/spark/pull/59100#issuecomment-5880522197
The earlier change (apache/spark#52253) addressed a performance regression caused by switching to fully lazy analysis in apache/spark#49386. Could we check whether always taking the lazy path for pipelines reintroduces that regression, particularly for long chains of `withColumn` calls? We may need a way to control this behavior, although re-enabling eager analysis would also bring back the dependency issue this PR fixes. cc @cloud-fan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
