attilapiros commented on a change in pull request #31133:
URL: https://github.com/apache/spark/pull/31133#discussion_r568483231
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala
##########
@@ -388,6 +394,9 @@ private[hive] object HiveTableUtil {
private[hive] object DeserializerLock
private[hive] object HadoopTableReader extends HiveInspectors with Logging {
+
+ val avroTableProperties =
AvroTableProperties.values().map(_.getPropName()).toSet
Review comment:
The `AvroTableProperties` enum is used for avoiding the hardcoding of
those property names which related to Avro and thus where the table properties
has higher priority when a value for that property is chosen.
So this PR does NOT set any of those properties if the user hasn't specified
one by using the `SERDEPROPERTIES` on the table or on the partition here we
just change which one is wining before passing the values further to the SerDe
initialization.
The `avro.schema.literal` on its own is not enough for example if the user
used different `avro.schema.url` on the table than on the partition we will
have the same issue. This can be true for all the properties it is good if they
are coming from the consistent source: either from the table or from the
partition. The advantage of using `AvroTableProperties` that we should not
investigate all the consistency between these properties but just take them all
from the same source and when a new one is introduced in the future which
matters for the serde we will be still safe and this part does not generate a
new bug silently (as we cannot have test for not existing future serde
properties).
What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
