Yann Byron created HUDI-3125:
--------------------------------
Summary: Spark SQL write timestamp type without disable
`spark.sql.datetime.java8API.enabled` manually
Key: HUDI-3125
URL: https://issues.apache.org/jira/browse/HUDI-3125
Project: Apache Hudi
Issue Type: Improvement
Components: Spark Integration
Reporter: Yann Byron
Fix For: 0.11.0, 0.10.1
{code:java}
create table h0_p(id int, name string, price double, dt timestamp) using hudi
partitioned by(dt) options(type = 'cow', primaryKey = 'id');
insert into h0_p values (3, 'a1', 10, cast('2021-05-08 00:00:00' as
timestamp)); {code}
By default, that run the sql above will throw exception:
{code:java}
Caused by: java.lang.ClassCastException: java.time.Instant cannot be cast to
java.sql.Timestamp
at
org.apache.hudi.AvroConversionHelper$.$anonfun$createConverterToAvro$8(AvroConversionHelper.scala:306)
at
org.apache.hudi.AvroConversionHelper$.$anonfun$createConverterToAvro$8$adapted(AvroConversionHelper.scala:306)
at scala.Option.map(Option.scala:230) {code}
We need disable `spark.sql.datetime.java8API.enabled` manually to make it work:
{code:java}
set spark.sql.datetime.java8API.enabled=false; {code}
And the command must be executed in the runtime. It can't work if provide this
by spark-sql command: `spark-sql --conf
spark.sql.datetime.java8API.enabled=false`. That's because this config is
forced to enable when launch spark-sql.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
