bjgrunner opened a new issue, #10335: URL: https://github.com/apache/hudi/issues/10335
We are using Spark 3.4 and scala 3.12, and hudi-spark3.4-bundle_2.12. I have a table partitioned by event_date, and have another column called previous_event_date. The unique key field is event_id, via bloom index. Suppose I have a record with: Event_id = 1 Event_date = 14dec2023 Previous_event_date = 13dec2023 A new record comes in for upsert: Event_id = 1 Event_date = 15dec2023 Previous_event_date = 14dec2023 This is a change to the existing record, but has a new event date. Is there a way I can tell Hudi to do something like: Update table Set event_date=15dec2023, previous_date = 14dec2023 Where event_id = 1 And event_date = new record previous event date (14dec2023) I'm basically wanting to move the record from one partition to another, without having to use a global index, or perform a delete first followed by upsert. I hope this is clear, I tried my best to explain the requirement. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
