[
https://issues.apache.org/jira/browse/SPARK-51692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18141234#comment-18141234
]
Artem Kupchinskiy commented on SPARK-51692:
-------------------------------------------
Still fails on 4.1.4 and 4.3.0, now as a plan validation error. A smaller
repro, failing on 4.3.0 with default settings:
{code:python}
spark.range(0, 1000, 1, 4).selectExpr("id % 7 AS g", "id AS
u").write.partitionBy("g").parquet("/tmp/fact")
spark.range(0, 1000, 1, 4).selectExpr("id % 7 AS g", "id % 5 AS
x").write.parquet("/tmp/dim")
spark.read.parquet("/tmp/fact").createOrReplaceTempView("fact")
spark.read.parquet("/tmp/dim").createOrReplaceTempView("dim")
spark.sql("""
WITH t AS (
SELECT f.g, f.u, rand() AS r
FROM fact f JOIN (SELECT DISTINCT g FROM dim WHERE x = 1 AND g < 3) d ON
f.g = d.g)
SELECT count(*) FROM t a JOIN t b ON a.u = b.u
""").show()
# [PLAN_VALIDATION_FAILED_RULE_IN_BATCH] Rule ...ReplaceCTERefWithRepartition
... generated an invalid plan:
# The plan was previously resolved and now became unresolved.
{code}
On master it also fails with a deterministic {{WITH t AS MATERIALIZED (...)}}
body. The cause is as described above: {{ReplaceCTERefWithRepartition}}
deduplicates the CTE body for a reference whose output attributes differ from
the definition's, which renews the plan inside the {{DynamicPruningSubquery}}
but not its {{buildKeys}}.
The closed PR removed the deduplication. When I emulated that on master, it
broke the {{CTEInlineSuite}} test "MATERIALIZED CTE with an inner CTE
correlated to its own relations" ("An alias reuses the same expression ID"). I
opened https://github.com/apache/spark/pull/60241, which keeps the
deduplication but leaves the build plan, build keys and expression ID of the
dynamic pruning subquery unchanged.
> can not find field for sql with partition prune and cte
> -------------------------------------------------------
>
> Key: SPARK-51692
> URL: https://issues.apache.org/jira/browse/SPARK-51692
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.5, 4.0.0
> Environment: spark3.5
> Reporter: lifulong
> Priority: Major
> Labels: pull-request-available
>
> java.lang.IllegalStateException: Couldn't find client_id#49 in
> [p_date#3543,client_id#3428,user_classify#3504]
> at
> org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1.applyOrElse(BoundAttribute.scala:80)
> at
> org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1.applyOrElse(BoundAttribute.scala:73)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:461)
> at
> org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:76)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:461)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:437)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:405)
> at
> org.apache.spark.sql.catalyst.expressions.BindReferences$.bindReference(BoundAttribute.scala:73)
> at
> org.apache.spark.sql.catalyst.expressions.BindReferences$.$anonfun$bindReferences$1(BoundAttribute.scala:94)
> at scala.collection.immutable.List.map(List.scala:293)
> at
> org.apache.spark.sql.catalyst.expressions.BindReferences$.bindReferences(BoundAttribute.scala:94)
> at
> org.apache.spark.sql.execution.adaptive.PlanAdaptiveDynamicPruningFilters$$anonfun$apply$2.applyOrElse(PlanAdaptiveDynamicPruningFilters.scala:45)
> at
> org.apache.spark.sql.execution.adaptive.PlanAdaptiveDynamicPruningFilters$$anonfun$apply$2.applyOrElse(PlanAdaptiveDynamicPruningFilters.scala:40)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:461)
>
> Rule PartitionPruning will add DynamicPruningSubquery Filter for partition
> prune, DynamicPruningSubquery Node contains prune joinKeys and
> filteringPlan, that info will finally used for gen
> SubqueryAdaptiveBroadcastExec node
>
> Rule ReplaceCTERefWithRepartition will gen new filed ref when replace CTE
> ref, but do not replace buildKeys in SubqueryAdaptiveBroadcastExec node which
> contains in CTE.
>
>
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
