Thanks Shahidha for the quick response. Pratyaksh, I am ok with making the behavior consistent with other Key generators. Please go ahead and submit a PR.
Thanks, Balaji.V On Thu, Dec 12, 2019 at 10:34 PM Pratyaksh Sharma <[email protected]> wrote: > Hi Shahida, > > Thank you for the clarification. Actually I was thinking about a corner > case where we define the partition field and in some incoming record, the > value for the corresponding defined partition field is not present. Such > cases would result in exception and job will get killed. > > On Fri, Dec 13, 2019 at 11:02 AM Shahida Khan <[email protected]> > wrote: > > > Hi Pratyaksh, > > > > As far as I understand, basic requirement of TimestampBasedKeyGenerator > is > > converting the partitions into timebased dateformat. > > *e.g.* your columns is in Unix Timestamp which need to convert to > > dateformat like '2019/12/10' > > > > There will never be scenario where you won't give partitions and use > > TimestampBasedKeyGenerator. > > Also, to use TimestampBasedKeyGenerator, mandate configs needs to be > define > > which actually is converting your field to partitions. > > e.g. > > hoodie.datasource.write.partitionpath.field= col_dtmDateTime > > > > > hoodie.datasource.write.keygenerator.class=org.apache.hudi.utilities.keygen.TimestampBasedKeyGenerator > > hoodie.deltastreamer.keygen.timebased.timestamp.type=EPOCHMILLISECONDS > > hoodie.deltastreamer.keygen.timebased.output.dateformat=yyyy/MM/dd > > > > I hope this help! > > > > *Regards,* > > *Shahida R. Khan* > > *+91 9167538366* > > > > > > On Thu, 12 Dec 2019 at 12:53, Pratyaksh Sharma <[email protected]> > > wrote: > > > > > Hi, > > > > > > If value for configured partitionPathField is not present, we are > > > defaulting to default partition path in all the key generator classes > > > except TimestampBasedKeyGenerator. In TimestampBasedKeyGenerator, we > > > directly throw exception if the value is null. > > > > > > I wanted to know if this behaviour is intentional. Ideally we should > > handle > > > such cases gracefully everywhere. > > > > > >
