jcellary opened a new pull request, #3121: URL: https://github.com/apache/iceberg-python/pull/3121
This PR introduces a new flag LOCK_ENABLED in hive catalog. It is set to true by default to allow for backwards compatibility. If set to false no locks are being created when writing/committing a table. # Rationale for this change In our production environment we are relying on our own external locking mechanism, so locks are not needed here and furthermore they are causing deadlocks. The current implementation creates a lock, writes data and then removes the lock. When the application dies during data write in a hard way (without a chance to run the finally clause of try catch) then the lock is never removed. There is no mechanism in the lib of removing stale old locks and checking their age. So the effect of that is that when one of the job dies in the wrong moment, all other jobs are stuck forever. Currently in our prod environment every few weeks we have to remove the contents of `HIVE_LOCKS` table to unblock jobs. To prevent this from happening we introduced a flag which allows skipping creation of locks. ## Are these changes tested? Yes, we are running a fork with these changes on our production cluster ## Are there any user-facing changes? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
