Wenchen Fan created SPARK-59855:
-----------------------------------
Summary: Analyze long withColumns chains in one pass and remove
the eager analysis in Spark Connect
Key: SPARK-59855
URL: https://issues.apache.org/jira/browse/SPARK-59855
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Wenchen Fan
Spark Connect plans each withColumns / withColumnsRenamed call lazily as
Project(Seq(UnresolvedStarWithColumns(...)), child) (SPARK-50601), so a loop of
df = df.withColumn(...) becomes a stack of such Projects. The fixed-point
analyzer resolves at most one of them per iteration of the Resolution batch,
and a chain of about 100 calls hits spark.sql.analyzer.maxIterations.
SPARK-53505 worked around this by eagerly analyzing each withColumns in the
Connect planner, which is quadratic and binds the plan when it's built (e.g.
pipeline flows lose their dependency on datasets that already exist in the
catalog).
This ticket makes the analyzer resolve a lazily planned withColumns as soon as
its child is resolved, in isolation from the child plan, so the whole chain is
resolved in one pass, and removes the eager analysis from the Connect planner.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
