kbendick commented on PR #4795: URL: https://github.com/apache/iceberg/pull/4795#issuecomment-1135125402
> I am using HadoopCatalog but not really getting around of lack of lock here (the lock problem was resolved by https://github.com/apache/iceberg/pull/3663 where you can use some DynamoDB based lockManager with HadoopTableOperation Ok. That's good you're using a lock. I'm not sure how well exercised that code path is in practice, but you are using it. So that's good. > we have multiple scenarios needing this local snapshot id, the simplest example might be we have multiple threads committing to the same table, we want to audit this information (with information like which thread generated which version)... Is there another possible way to solve this problem? I do worry that exposing the snapshot ID this way will be dangerous for other users, who might use the (outdated) snapshot ID for the wrong purpose. I understand that _you_ are using it for purposes that sound like they don't violate any integrity concerns (based on what you said), but we have to think of the general case. Exposing it as simply the latest snapshot id in the thread leaves it open to the user to have to do the correct thing with the no longer current snapshot ID. If you add the thread ID to the snapshot summary, you'd be able to query the `snapshots` table and see it there. Is that a possibility? Another possibility I can think of is emitting an event on commit. The TLDR of my concern is that while you might very well know what you're doing with this information, there will likely be others who come along and do not use it properly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
