[
https://issues.apache.org/jira/browse/HUDI-8220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lokesh Jain updated HUDI-8220:
------------------------------
Description:
If we create a table in flink with custom key generator it fails with error
message.
{code:java}
[ERROR] Could not execute SQL statement. Reason:
org.apache.hudi.exception.HoodieValidationException: Field ts:timestamp does
not exist{code}
{code:java}
CREATE TABLE hudi_table(
ts BIGINT,
uuid VARCHAR(40) PRIMARY KEY NOT ENFORCED,
rider VARCHAR(20),
driver VARCHAR(20),
fare DOUBLE,
city VARCHAR(20)
)
WITH (
'connector' = 'hudi',
'path' = 'file:///tmp/hudi_table',
'table.type' = 'MERGE_ON_READ',
'hoodie.keygen.timebased.timestamp.type' = 'SCALAR',
'hoodie.keygen.timebased.output.dateformat' = 'yyyy/MM/dd',
'hoodie.keygen.timebased.timestamp.scalar.time.unit' = 'seconds',
'hoodie.datasource.write.keygenerator.class' =
'org.apache.hudi.keygen.CustomAvroKeyGenerator',
'hoodie.datasource.write.partitionpath.field' = 'ts:timestamp'
); {code}
We need to support using custom keygen with flink engine.
was:
If we create a table in flink with custom key generator it fails with error
message.
{code:java}
Caused by: org.apache.hudi.exception.HoodieKeyException: Unable to find field
names for partition path in proper format at
org.apache.hudi.keygen.CustomAvroKeyGenerator.lambda$getPartitionKeyGenerators$1(CustomAvroKeyGenerator.java:81)
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) at
org.apache.hudi.keygen.CustomAvroKeyGenerator.getPartitionKeyGenerators(CustomAvroKeyGenerator.java:97)
at
org.apache.hudi.keygen.CustomAvroKeyGenerator.<init>(CustomAvroKeyGenerator.java:71)
... 21 more {code}
{code:java}
CREATE TABLE hudi_table(
ts BIGINT,
uuid VARCHAR(40) PRIMARY KEY NOT ENFORCED,
rider VARCHAR(20),
driver VARCHAR(20),
fare DOUBLE,
city VARCHAR(20)
)
WITH (
'connector' = 'hudi',
'path' = 'file:///tmp/hudi_table',
'table.type' = 'MERGE_ON_READ',
'hoodie.keygen.timebased.timestamp.type' = 'SCALAR',
'hoodie.keygen.timebased.output.dateformat' = 'yyyy/MM/dd',
'hoodie.keygen.timebased.timestamp.scalar.time.unit' = 'seconds',
'hoodie.datasource.write.keygenerator.class' =
'org.apache.hudi.keygen.CustomAvroKeyGenerator',
'hoodie.datasource.write.partitionpath.field' = 'ts:timestamp'
); {code}
We need to support using custom keygen with flink engine.
> CustomKeyGenerator can not be created with flink
> ------------------------------------------------
>
> Key: HUDI-8220
> URL: https://issues.apache.org/jira/browse/HUDI-8220
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Lokesh Jain
> Priority: Major
> Fix For: 1.0.0
>
>
> If we create a table in flink with custom key generator it fails with error
> message.
> {code:java}
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.hudi.exception.HoodieValidationException: Field ts:timestamp does
> not exist{code}
> {code:java}
> CREATE TABLE hudi_table(
> ts BIGINT,
> uuid VARCHAR(40) PRIMARY KEY NOT ENFORCED,
> rider VARCHAR(20),
> driver VARCHAR(20),
> fare DOUBLE,
> city VARCHAR(20)
> )
> WITH (
> 'connector' = 'hudi',
> 'path' = 'file:///tmp/hudi_table',
> 'table.type' = 'MERGE_ON_READ',
> 'hoodie.keygen.timebased.timestamp.type' = 'SCALAR',
> 'hoodie.keygen.timebased.output.dateformat' = 'yyyy/MM/dd',
> 'hoodie.keygen.timebased.timestamp.scalar.time.unit' = 'seconds',
> 'hoodie.datasource.write.keygenerator.class' =
> 'org.apache.hudi.keygen.CustomAvroKeyGenerator',
> 'hoodie.datasource.write.partitionpath.field' = 'ts:timestamp'
> ); {code}
> We need to support using custom keygen with flink engine.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
