[
https://issues.apache.org/jira/browse/HUDI-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17472426#comment-17472426
]
Sagar Sumit commented on HUDI-2909:
-----------------------------------
[~h7kanna] To avoid the error that you're getting, the new config needs to be
enabled. However, that will lead to inconsistency between values of the
timestamp column before and after enabling the config. I would suggest to
recreate the dataset with the config enabled. If it's not an option then
probably we need to write a tool to migrate just the affected columns. I can
help you out with that.
> Partition field parsing fails due to KeyGenerator giving inconsistent value
> for logical timestamp type
> ------------------------------------------------------------------------------------------------------
>
> Key: HUDI-2909
> URL: https://issues.apache.org/jira/browse/HUDI-2909
> Project: Apache Hudi
> Issue Type: Bug
> Components: DeltaStreamer
> Reporter: Harsha Teja Kanna
> Assignee: Sagar Sumit
> Priority: Blocker
> Labels: core-flow-ds, pull-request-available, sev:critical
> Fix For: 0.10.1
>
>
> Existing table has timebased keygen config show below
> hoodie.deltastreamer.keygen.timebased.timestamp.type=SCALAR
> hoodie.deltastreamer.keygen.timebased.output.timezone=GMT
> hoodie.deltastreamer.keygen.timebased.output.dateformat=yyyy/MM/dd
> hoodie.deltastreamer.keygen.timebased.timestamp.scalar.time.unit=MICROSECONDS
> hoodie.deltastreamer.keygen.timebased.input.timezone=GMT
> hoodie.datasource.write.partitionpath.field=lastdate:timestamp
> hoodie.datasource.write.operation=upsert
> hoodie.deltastreamer.transformer.sql=SELECT session.id, session.rid,
> session.mid, to_timestamp(session.lastdate) as lastdate,
> to_timestamp(session.updatedate) as updatedate FROM <SRC> a
>
> Upgrading to 0.10.0 from 0.9.0 fails with exception
> org.apache.hudi.exception.HoodieKeyGeneratorException: Unable to parse input
> partition field :2021-12-01 10:13:34.702
> Caused by: org.apache.hudi.exception.HoodieNotSupportedException: Unexpected
> type for partition field: java.sql.Timestamp
> at
> org.apache.hudi.keygen.TimestampBasedAvroKeyGenerator.getPartitionPath(TimestampBasedAvroKeyGenerator.java:211)
> at
> org.apache.hudi.keygen.TimestampBasedAvroKeyGenerator.getPartitionPath(TimestampBasedAvroKeyGenerator.java:133)
> *Workaround fix:*
> Reverting this
> https://github.com/apache/hudi/pull/3944/files#diff-22fb52b5cf28727ba23cb8bd4be820432a4e396ce663ac472a4677e889b7491eR543
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
