[
https://issues.apache.org/jira/browse/HUDI-7902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17863211#comment-17863211
]
Lokesh Jain commented on HUDI-7902:
-----------------------------------
The approach would be to reuse the key `hoodie.table.partition.fields` to store
the key type for custom key generator. We will be breaking the changes into
multiple parts here:-
1. Reader side change - We will ensure the reader is able to read the table in
a backward compatible manner based on the table version. If table version is
older then it should fall back to older method of using the key else the new
method. This would also ensure that other users of the key are handled properly.
2. Write side change - Here we will ensure that the new values are reflected in
the table config and updated in the hoodie.properties as part of upgrade
handling. We will also need to handle downgrade for this change.
3. We need to update file index so that _partitionSchemaFromProperties handles
the custom key generator and assigns string type for timestamp based partition
types.
> Add a new table property to store partition field types for custom key
> generator
> --------------------------------------------------------------------------------
>
> Key: HUDI-7902
> URL: https://issues.apache.org/jira/browse/HUDI-7902
> Project: Apache Hudi
> Issue Type: Improvement
> Reporter: Ethan Guo
> Assignee: Lokesh Jain
> Priority: Major
> Fix For: 1.0.0
>
>
> As of today, writing a partitioned table with CustomKeyGenerator requires
> write config of the partition field types (they are not stored as a table
> property). CustomKeyGenerator requires partition keys to be in the format of
> {{field:type}} (e.g. {{column1:SIMPLE}} ). However, only the field names are
> stored in {{hoodie.properties}} file. We need to store the field types too
> so that without the write config, the writer configure the correct partition
> field name and type in Spark datasource and SQL DML writes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
