beyond1920 commented on code in PR #10826:
URL: https://github.com/apache/hudi/pull/10826#discussion_r1513982842
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala:
##########
@@ -95,7 +95,9 @@ object InsertIntoHoodieTableCommand extends Logging with
ProvidesHoodieConfig wi
}
val config = buildHoodieInsertConfig(catalogTable, sparkSession,
isOverWritePartition, isOverWriteTable, partitionSpec, extraOptions,
staticOverwritePartitionPathOpt)
- val alignedQuery = alignQueryOutput(query, catalogTable, partitionSpec,
sparkSession.sessionState.conf)
+ val optimizer = sparkSession.sessionState.optimizer
+ val optimizerPlan = optimizer.execute(query)
+ val alignedQuery = alignQueryOutput(optimizerPlan, catalogTable,
partitionSpec, sparkSession.sessionState.conf)
Review Comment:
It's seems a little heavy to use optimizer just for case insensitive.
Besides, if wrap an optimize phase here, user might missed something in
spark sql WEB UI.
Is there any better solution?
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala:
##########
@@ -95,7 +95,9 @@ object InsertIntoHoodieTableCommand extends Logging with
ProvidesHoodieConfig wi
}
val config = buildHoodieInsertConfig(catalogTable, sparkSession,
isOverWritePartition, isOverWriteTable, partitionSpec, extraOptions,
staticOverwritePartitionPathOpt)
- val alignedQuery = alignQueryOutput(query, catalogTable, partitionSpec,
sparkSession.sessionState.conf)
+ val optimizer = sparkSession.sessionState.optimizer
+ val optimizerPlan = optimizer.execute(query)
+ val alignedQuery = alignQueryOutput(optimizerPlan, catalogTable,
partitionSpec, sparkSession.sessionState.conf)
Review Comment:
It seems a little heavy to use optimizer just for case insensitive.
Besides, if wrap an optimize phase here, user might missed something in
spark sql WEB UI.
Is there any better solution?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
