[
https://issues.apache.org/jira/browse/SPARK-59855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59855:
-----------------------------------
Labels: pull-request-available (was: )
> Analyze long withColumns chains in one pass and remove the eager analysis in
> Spark Connect
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-59855
> URL: https://issues.apache.org/jira/browse/SPARK-59855
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Wenchen Fan
> Priority: Major
> Labels: pull-request-available
>
> Spark Connect plans each withColumns / withColumnsRenamed call lazily as
> Project(Seq(UnresolvedStarWithColumns(...)), child) (SPARK-50601), so a loop
> of df = df.withColumn(...) becomes a stack of such Projects. The fixed-point
> analyzer resolves at most one of them per iteration of the Resolution batch,
> and a chain of about 100 calls hits spark.sql.analyzer.maxIterations.
> SPARK-53505 worked around this by eagerly analyzing each withColumns in the
> Connect planner, which is quadratic and binds the plan when it's built (e.g.
> pipeline flows lose their dependency on datasets that already exist in the
> catalog).
> This ticket makes the analyzer resolve a lazily planned withColumns as soon
> as its child is resolved, in isolation from the child plan, so the whole
> chain is resolved in one pass, and removes the eager analysis from the
> Connect planner.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
