[
https://issues.apache.org/jira/browse/HUDI-8220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lokesh Jain updated HUDI-8220:
------------------------------
Fix Version/s: 1.0.0
> CustomKeyGenerator can not be created with flink
> ------------------------------------------------
>
> Key: HUDI-8220
> URL: https://issues.apache.org/jira/browse/HUDI-8220
> Project: Apache Hudi
> Issue Type: Bug
> Affects Versions: 0.15.0, 1.0.0
> Reporter: Lokesh Jain
> Priority: Major
> Fix For: 1.0.0
>
>
> If we create a table in flink with custom key generator it fails with error
> message.
> {code:java}
> Caused by: org.apache.hudi.exception.HoodieKeyException: Unable to find field
> names for partition path in proper format at
> org.apache.hudi.keygen.CustomAvroKeyGenerator.lambda$getPartitionKeyGenerators$1(CustomAvroKeyGenerator.java:81)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
> at
> org.apache.hudi.keygen.CustomAvroKeyGenerator.getPartitionKeyGenerators(CustomAvroKeyGenerator.java:97)
> at
> org.apache.hudi.keygen.CustomAvroKeyGenerator.<init>(CustomAvroKeyGenerator.java:71)
> ... 21 more {code}
> {code:java}
> CREATE TABLE hudi_table(
> ts BIGINT,
> uuid VARCHAR(40) PRIMARY KEY NOT ENFORCED,
> rider VARCHAR(20),
> driver VARCHAR(20),
> fare DOUBLE,
> city VARCHAR(20)
> )
> WITH (
> 'connector' = 'hudi',
> 'path' = 'file:///tmp/hudi_table',
> 'table.type' = 'MERGE_ON_READ',
> 'hoodie.keygen.timebased.timestamp.type' = 'SCALAR',
> 'hoodie.keygen.timebased.output.dateformat' = 'yyyy/MM/dd',
> 'hoodie.keygen.timebased.timestamp.scalar.time.unit' = 'seconds',
> 'hoodie.datasource.write.keygenerator.class' =
> 'org.apache.hudi.keygen.CustomAvroKeyGenerator',
> 'hoodie.datasource.write.partitionpath.field' = 'ts:timestamp'
> ); {code}
> We need to support using custom keygen with flink engine.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
