vladimirg-db commented on code in PR #53508:
URL: https://github.com/apache/spark/pull/53508#discussion_r2628449664
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala:
##########
@@ -707,7 +709,28 @@ object ViewSyncSchemaToMetaStore extends (LogicalPlan =>
Unit) {
viewFields.find(_.name == name).get.metadata)
}
StructType(newFields)
+ } else if
(session.sessionState.conf.viewSchemaEvolutionPreserveUserComments) {
+ // Adopt types/nullable/names from query, but preserve view
comments.
+ val newFields = viewQuery.schema.map { planField =>
+ val existingField = viewFields.find(_.name == planField.name)
+ val metadataToUse = existingField match {
+ case Some(viewField) =>
+ // Preserve existing view comment if it exists.
+ if (viewField.getComment().isDefined) {
+ viewField.metadata
+ } else {
+ // No existing comment, adopt from table.
+ planField.metadata
+ }
Review Comment:
Actually, we would have a mix of comments from the table and from the view
schema this way. I think this is not ok. cc @srielau to look at this code.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala:
##########
@@ -707,7 +709,28 @@ object ViewSyncSchemaToMetaStore extends (LogicalPlan =>
Unit) {
viewFields.find(_.name == name).get.metadata)
}
StructType(newFields)
+ } else if
(session.sessionState.conf.viewSchemaEvolutionPreserveUserComments) {
+ // Adopt types/nullable/names from query, but preserve view
comments.
+ val newFields = viewQuery.schema.map { planField =>
+ val existingField = viewFields.find(_.name == planField.name)
Review Comment:
This is O(n^2), will be super slow for wide views. Please improve
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
