gengliangwang commented on code in PR #42036:
URL: https://github.com/apache/spark/pull/42036#discussion_r1270216567
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CTESubstitution.scala:
##########
@@ -46,42 +46,65 @@ import
org.apache.spark.sql.internal.SQLConf.{LEGACY_CTE_PRECEDENCE_POLICY, Lega
* dependency for any valid CTE query (i.e., given CTE definitions A and B
with B referencing A,
* A is guaranteed to appear before B). Otherwise, it must be an invalid user
query, and an
* analysis exception will be thrown later by relation resolving rules.
+ *
+ * If the query is a SQL command or DML statement (extends `CTEInChildren`),
+ * place `WithCTE` into their children.
*/
object CTESubstitution extends Rule[LogicalPlan] {
def apply(plan: LogicalPlan): LogicalPlan = {
if (!plan.containsPattern(UNRESOLVED_WITH)) {
return plan
}
- val isCommand = plan.exists {
- case _: Command | _: ParsedStatement | _: InsertIntoDir => true
- case _ => false
+
+ val forceInline = if (conf.getConf(SQLConf.LEGACY_INLINE_CTE_IN_COMMANDS))
{
+ // The legacy behavior always inlines the CTE relations for queries in
commands.
+ plan.exists {
+ case _: Command | _: ParsedStatement | _: InsertIntoDir => true
+ case _ => false
+ }
+ } else {
+ val commands = plan.collect {
+ case c @ (_: Command | _: ParsedStatement | _: InsertIntoDir) => c
+ }
+ if (commands.length == 1) {
+ // If there is only one command and it's `CTEInChildren`, we can
resolve
+ // CTE normally and don't need to force inline.
+ !commands.head.isInstanceOf[CTEInChildren]
+ } else if (commands.length > 1) {
+ // This can happen with the multi-insert statement. We should fall
back to
Review Comment:
Nit: There is duplicated logic here.
To make the code more readable, we can always collect the commands first.
If the length of commands is 1, there is a different behavior based on the
legacy conf. Otherwise the logic is determined.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
