Lokesh Jain created HUDI-8220:
---------------------------------
Summary: CustomKeyGenerator can not be created with flink
Key: HUDI-8220
URL: https://issues.apache.org/jira/browse/HUDI-8220
Project: Apache Hudi
Issue Type: Bug
Reporter: Lokesh Jain
If we create a table in flink with custom key generator it fails with error
message.
{code:java}
Caused by: org.apache.hudi.exception.HoodieKeyException: Unable to find field
names for partition path in proper format at
org.apache.hudi.keygen.CustomAvroKeyGenerator.lambda$getPartitionKeyGenerators$1(CustomAvroKeyGenerator.java:81)
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) at
org.apache.hudi.keygen.CustomAvroKeyGenerator.getPartitionKeyGenerators(CustomAvroKeyGenerator.java:97)
at
org.apache.hudi.keygen.CustomAvroKeyGenerator.<init>(CustomAvroKeyGenerator.java:71)
... 21 more {code}
{code:java}
CREATE TABLE hudi_table(
ts BIGINT,
uuid VARCHAR(40) PRIMARY KEY NOT ENFORCED,
rider VARCHAR(20),
driver VARCHAR(20),
fare DOUBLE,
city VARCHAR(20)
)
WITH (
'connector' = 'hudi',
'path' = 'file:///tmp/hudi_table',
'table.type' = 'MERGE_ON_READ',
'hoodie.keygen.timebased.timestamp.type' = 'SCALAR',
'hoodie.keygen.timebased.output.dateformat' = 'yyyy/MM/dd',
'hoodie.keygen.timebased.timestamp.scalar.time.unit' = 'seconds',
'hoodie.datasource.write.keygenerator.class' =
'org.apache.hudi.keygen.CustomAvroKeyGenerator',
'hoodie.datasource.write.partitionpath.field' = 'ts:timestamp'
); {code}
We need to support using custom keygen with flink engine.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
