berniedurfee-renaissance opened a new issue, #8196:
URL: https://github.com/apache/hudi/issues/8196
An older version of a table schema had a column `xyz` typed as
`DECIMAL(10,3)` while a newer data file has the same column typed to
`DECIMAL(28,6)`.
```
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Could not convert
field Type from DECIMAL(10,3) to DECIMAL(28,6) for field xyz
at
org.apache.hudi.hive.util.HiveSchemaUtil.getSchemaDifference(HiveSchemaUtil.java:103)
at org.apache.hudi.hive.HiveSyncTool.syncSchema(HiveSyncTool.java:249)
at
org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:184)
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:129)
at
org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:115)
... 48 more
```
I would expect this to work because values in the field with type
`DECIMAL(10,3)` will fit into the field of type `DECIMAL(28,6)`. I think this
change would be considered 'backward compatible' and Hudi should just update
the schema in Hive to set the column type to `DECIMAL(28,6)`, which would fit
the old values as well as new values.
Shouldn't the schema evolve if the new data type is 'bigger' than the
previous data type?
I'm running this a Glue 3 job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
