Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21984#discussion_r208138182
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala ---
@@ -92,7 +92,7 @@ class AvroSerializer(rootCatalystType: DataType,
rootAvroType: Schema, nullable:
case BinaryType =>
(getter, ordinal) => ByteBuffer.wrap(getter.getBinary(ordinal))
case DateType =>
- (getter, ordinal) => getter.getInt(ordinal) *
DateTimeUtils.MILLIS_PER_DAY
+ (getter, ordinal) => getter.getInt(ordinal)
--- End diff --
There are 2 kinds of compatibilities:
1. the file written by old avro data source can be read by the new avro
data source
2. the file written by new avro data source can be read by the old avro
data source
I think we should focus on 1) and ignore 2)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
