Hi Vinoth - that was extremely helpful...I almost had it working, HOWEVER, it appears I have dates that some have the ms on the end and others don't....so if I pick adding a time format with them, then the ones without the fail and vice versa....Good times.
After I figure this out I'll see if I can put this information somewhere easy to find. On Mon, May 4, 2020 at 12:23 PM Vinoth Chandar <[email protected]> wrote: > Hi Allen, > > You are able to configure the key generator for deltastreamer using this > property (either via a file or --config ) > hoodie.datasource.write.keygenerator.class > > You might be interested in this built-in generator. > > https://github.com/apache/incubator-hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/keygen/TimestampBasedKeyGenerator.java#L64 > It takes let you configure a field as a recordKey, and if you can parse > your timestamp using Java SimpleDateFormat, you can specify the datetime > field and a pattern to parse it into.. > > Happy to make this work for you. > > community, any volunteers to faq/document this? :) > > > On Mon, May 4, 2020 at 9:11 AM Allen Underwood > <[email protected]> wrote: > > > I’ve tried to do my due diligence by googling / searching this slack and > > I’ve come up empty.Is there a way through configuration / deltastreamer > > to extract a custom partition key?Basically I have a datetime field in a > > Kafka Source that has an ISO8601 datetime….is there a way to extract a > > partition value out of that?I found this after some Googling, but this > > seems like it’d only be useful if I wanted to write my own writer > > application: > > > > > https://github.com/apache/incubator-hudi/blob/master/hudi-spark/src/main/java/org/apache/hudi/keygen/ComplexKeyGenerator.javaAny > > way to do what I need through configuration of the spark job / hudi > > configuration? > > hudi-spark/src/main/java/org/apache/hudi/keygen/ComplexKeyGenerator.java > > < > https://github.com/apache/incubator-hudi/blob/master/hudi-spark/src/main/java/org/apache/hudi/keygen/ComplexKeyGenerator.java > > > > > > -- > > *Allen Underwood* > > > -- *Allen Underwood* Principal Software Engineer Broadcom | Symantec Enterprise Division *Mobile*: 404.808.5926
smime.p7s
Description: S/MIME Cryptographic Signature
