Re: support for out of order processing

2021-01-31 Thread Anton Zuyeu
Thank you Nishith. On Sat, Jan 30, 2021 at 7:29 PM nishith agarwal wrote: > Anton, > > Yes, you can achieve this with Hudi. Hudi uses a HoodieRecordPayload > implementation to be able to merge old and new records. You can define a > source ordering field (here "sort_key") to govern which

Re: support for out of order processing

2021-01-30 Thread nishith agarwal
Anton, Yes, you can achieve this with Hudi. Hudi uses a HoodieRecordPayload implementation to be able to merge old and new records. You can define a source ordering field (here "sort_key") to govern which record should be chosen as the latest one. The DefaultHoodieRecordPayload supports this ->

support for out of order processing

2021-01-30 Thread Anton Zuyeu
Hi Hudi team, We are replicating database table by reading table change logs and applying them to Hudi table, we would like to implement our pipeline so it can process records out of order. Pretty much we want to introduce column "sort_key" and only update existing records in the hudi table if a