[GitHub] [spark] pan3793 commented on pull request #36697: [SPARK-39313][SQL] `toCatalystOrdering` should fail if V2Expression can not be translated

[GitBox]

pan3793 commented on PR #36697:
URL: https://github.com/apache/spark/pull/36697#issuecomment-1140133553

   ```
   [info] - SPARK-30289 Create: partitioned by nested column *** FAILED *** 
(363 milliseconds)
   [info]   java.lang.RuntimeException: Once strategy's idempotence is broken 
for batch Early Filter and Projection Push-Down
   [info]  RelationV2[ts#1494] testcat.table_name   RelationV2[ts#1494] 
testcat.table_name
   [info]   at 
org.apache.spark.sql.errors.QueryExecutionErrors$.onceStrategyIdempotenceIsBrokenForBatchError(QueryExecutionErrors.scala:1302)
   [info]   at 
org.apache.spark.sql.catalyst.rules.RuleExecutor.checkBatchIdempotence(RuleExecutor.scala:168)
   [info]   at 
org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:254)
   [info]   at 
org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1$adapted(RuleExecutor.scala:200)
   [info]   at scala.collection.immutable.List.foreach(List.scala:431)
   [info]   at 
org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:200)
   [info]   at 
org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$executeAndTrack$1(RuleExecutor.scala:179)
   [info]   at 
org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:88)
   [info]   at 
org.apache.spark.sql.catalyst.rules.RuleExecutor.executeAndTrack(RuleExecutor.scala:179)
   [info]   at 
org.apache.spark.sql.execution.QueryExecution.$anonfun$optimizedPlan$1(QueryExecution.scala:125)
   [info]   at 
org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
   [info]   at 
org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:183)
   [info]   at 
org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
   [info]   at 
org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:183)
   [info]   at 
org.apache.spark.sql.execution.QueryExecution.optimizedPlan$lzycompute(QueryExecution.scala:121)
   [info]   at 
org.apache.spark.sql.execution.QueryExecution.optimizedPlan(QueryExecution.scala:117)
   [info]   at 
org.apache.spark.sql.QueryTest.assertEmptyMissingInput(QueryTest.scala:226)
   [info]   at org.apache.spark.sql.QueryTest.checkAnswer(QueryTest.scala:148)
   [info]   at 
org.apache.spark.sql.DataFrameWriterV2Suite.$anonfun$new$97(DataFrameWriterV2Suite.scala:692)
   ```
   
   In `RuleExecutor.checkBatchIdempotence`, the `Alias` node of 
`keyGroupedPartitioning` changes exprId.
   
   <img width="888" alt="image" 
src="https://user-images.githubusercontent.com/26535726/170804934-b36c3c7b-32f3-4303-9620-622b2d83a998.png";>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org