zsxwing commented on code in PR #48986:
URL: https://github.com/apache/spark/pull/48986#discussion_r1866289021
##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:
##########
@@ -1130,10 +1133,6 @@ private[hive] object HiveClientImpl extends Logging {
Option(hc.getComment).map(field.withComment).getOrElse(field)
}
- private def verifyColumnDataType(schema: StructType): Unit = {
Review Comment:
It's better to keep this check since it doesn't hurt.
One of the side effects of this method is it will trigger the fallback code
path here:
https://github.com/apache/spark/blob/05728e4ff64e6684d7c6501f8a079e3b9aded9ed/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala#L283-L289
This would ensure even if we generate a bad schema (e.g., a bug in
`catalogString`), we will still write a schema that's working in Spark. Of
cause, the fallback code path will not work for Hive. However, the Hive
compatibility is the best effort and the Spark compatibility is more important.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
