juliuszsompolski commented on PR #53173: URL: https://github.com/apache/spark/pull/53173#issuecomment-3573134680
> It would make me very happy to find a way to fix it without changing Spark code. It seems to me however that currently I lose all the ability to distinguish DFWV1 saveAsTable with mode overwrite vs. DFWV2 replace, because both operations create an identical logical plan. @dongjoon-hyun I investigated it more, raised a PR https://github.com/delta-io/delta/pull/5569 with a sketch of trying to move it to a Delta extension rule. But https://github.com/apache/spark/blob/master/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala#L3343 ``` writeOperation.getTable.getSaveMethod match { case proto.WriteOperation.SaveTable.TableSaveMethod.TABLE_SAVE_METHOD_SAVE_AS_TABLE => -> w.saveAsTableCommand(tableName) ``` is directly returning the ReplaceTableAsSelect plan created by DataFrameWriter, without running any resolution or analysis on it. At the point or returning it we already returned from DataFrameWriter call. It further gets analyzed and executed in https://github.com/apache/spark/blob/master/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/execution/SparkConnectPlanExecution.scala#L92 ``` case proto.Plan.OpTypeCase.COMMAND => val command = request.getPlan.getCommand planner.transformCommand(command) match { case Some(transformer) => val qe = new QueryExecution( session, transformer(tracker), tracker, shuffleCleanupMode = shuffleCleanupMode) -> qe.assertCommandExecuted() executeHolder.eventsManager.postFinished() ``` at which point the plan is identical for `saveAsTable` and `replace`, and it cannot be distinguished anymore by a similar stack trace call, as we have already returned from there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
