parisni commented on code in PR #8683:
URL: https://github.com/apache/hudi/pull/8683#discussion_r1200536081
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala:
##########
@@ -59,32 +59,32 @@ private[sql] object SchemaConverters {
private val unionFieldMemberPrefix = "member"
private def toSqlTypeHelper(avroSchema: Schema, existingRecordNames:
Set[String]): SchemaType = {
- avroSchema.getType match {
- case INT => avroSchema.getLogicalType match {
- case _: Date => SchemaType(DateType, nullable = false)
- case _ => SchemaType(IntegerType, nullable = false)
+ (avroSchema.getType, Option(avroSchema.getDoc)) match {
Review Comment:
> Kind of feel there is no need to change each match for every data types
Any columns including nested columns also may have comments, so I don't see
why we should'nt look after all avro content for doc.
> which invokes toSqlTypeHelper firstly then fix the comment separately.
This would lead to walk thought the avro schema two times, and also lead to
complex merge of results. Am i missing something ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
