MonkeyCanCode commented on PR #225: URL: https://github.com/apache/polaris-tools/pull/225#issuecomment-4437737265
> > I do understand why this is discouraged as there is a potential a write can happen on both end (or table maintenances but this will be more like setup issues IMO). However, for large iceberg lakehouse that have multiple writers and they are responsible for a set of tables, it is not very possible to just migrate it all at one time. In that case, we do need this type of functionality to re-register a table. WDYT? > > > > Can we migrate batch by batch for this scenario? single batch can be all tables in one namespace. > > This tool supports identifier regex. So, we can use that to migrate specific group of tables. So, no need to attempt re-register IMO. > > > > Also, I would like to add that, if only 'K' tables migrated out of 'N' and you need to migrate remaining. You can call same cli for all 'N' tables. It will skip migration for registered tables (with the error that table already exist). But other tables will still be migrated in that process. It won't pause migration of other tables if current table already exists. So this may not be 100% valid as people can be using namespace as tenant/workflow separator for a given writer. Assuming one app is writing to 1k tables and each of the table has their own namespace, the purposed solution may not be useable in this case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
