cxzl25 commented on PR #5287:
URL: https://github.com/apache/hudi/pull/5287#issuecomment-1095301076
Using Spark3.2.0+Hudi 0.11 Snapshot, the following exception was encountered.
This problem can be solved if `AvroDeserializer` supports SQLConf
configuration.
```java
You can set the SQL config
'spark.sql.avro.datetimeRebaseModeInRead' or the datasource option
'datetimeRebaseMode' to 'LEGACY' to rebase the datetime values
w.r.t. the calendar difference during reading. To read the datetime values
as it is,
set the SQL config 'spark.sql.avro.datetimeRebaseModeInRead' or the
datasource option 'datetimeRebaseMode' to 'CORRECTED'.
at
org.apache.spark.sql.errors.QueryExecutionErrors$.sparkUpgradeInReadingDatesError(QueryExecutionErrors.scala:474)
at
org.apache.spark.sql.execution.datasources.DataSourceUtils$.newRebaseExceptionInRead(DataSourceUtils.scala:161)
at
org.apache.spark.sql.execution.datasources.DataSourceUtils$.$anonfun$creteTimestampRebaseFuncInRead$1(DataSourceUtils.scala:203)
at
org.apache.hudi.spark.org.apache.spark.sql.avro.AvroDeserializer.$anonfun$newWriter$7(AvroDeserializer.scala:145)
at
org.apache.hudi.spark.org.apache.spark.sql.avro.AvroDeserializer.$anonfun$newWriter$7$adapted(AvroDeserializer.scala:143)
at
org.apache.hudi.spark.org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$1(AvroDeserializer.scala:360)
at
org.apache.hudi.spark.org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$1$adapted(AvroDeserializer.scala:356)
at
org.apache.hudi.spark.org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$2(AvroDeserializer.scala:378)
at
org.apache.hudi.spark.org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$2$adapted(AvroDeserializer.scala:374)
at
org.apache.hudi.spark.org.apache.spark.sql.avro.AvroDeserializer.$anonfun$converter$4(AvroDeserializer.scala:85)
at
org.apache.hudi.spark.org.apache.spark.sql.avro.AvroDeserializer.deserialize(AvroDeserializer.scala:103)
at
org.apache.hudi.spark.org.apache.spark.sql.avro.Spark3HoodieAvroDeserializer.doDeserialize(Spark3HoodieAvroDeserializer.scala:37)
at
org.apache.hudi.spark.org.apache.spark.sql.avro.HoodieAvroDeserializerTrait.deserialize(HoodieAvroDeserializerTrait.scala:29)
at
org.apache.hudi.spark.org.apache.spark.sql.avro.HoodieAvroDeserializerTrait.deserialize$(HoodieAvroDeserializerTrait.scala:28)
at
org.apache.hudi.spark.org.apache.spark.sql.avro.Spark3HoodieAvroDeserializer.deserialize(Spark3HoodieAvroDeserializer.scala:24)
at
org.apache.hudi.HoodieMergeOnReadRDD$$anon$3.hasNext(HoodieMergeOnReadRDD.scala:262)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
