parisni commented on code in PR #8683:
URL: https://github.com/apache/hudi/pull/8683#discussion_r1193606161
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala:
##########
@@ -59,32 +59,32 @@ private[sql] object SchemaConverters {
private val unionFieldMemberPrefix = "member"
private def toSqlTypeHelper(avroSchema: Schema, existingRecordNames:
Set[String]): SchemaType = {
- avroSchema.getType match {
- case INT => avroSchema.getLogicalType match {
- case _: Date => SchemaType(DateType, nullable = false)
- case _ => SchemaType(IntegerType, nullable = false)
+ (avroSchema.getType, Option(avroSchema.getDoc)) match {
Review Comment:
Likely spark also have this limitation when retrieving schema from avro. But
spark don't usually infer spark schema from avro. Hudi does, and that's the
reason of the patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
