noahtaite commented on issue #9805: URL: https://github.com/apache/hudi/issues/9805#issuecomment-1746882929
Hey @ad1happy2go, yeah I am just facing two minor issues with this approach of requiring a glue sync after every DELETE_PARTITION. 1- We have a requirement to generate a staging glue table off of a subset of data that is copied from our production Hudi lake. Since these copies are manual we also need to manually sync that to a new glue table from scratch periodically. In my dev environment trying to glue sync any table with .replacecommits from scratch causes the partitions in the replacecommit to be dropped regardless of them being ingested later. Meaning in the interim we need to be very robust with our approach and ensure to glue sync across all locations whenever a delete_partititon is performed. 2 - Fails on 25+ partitions which all of my DELETE_PARTITIONS would be operating on. I think we can workaround this with Glue sync using ./hudi-sync-tool though. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
