deepakpanda93 commented on issue #6226: URL: https://github.com/apache/hudi/issues/6226#issuecomment-4842648085
Closing this as addressed in Hudi 1.x. On the two parts here: 1. Locks serializing across non-overlapping partitions — as @fengjian428 and @nsivabalan noted, this is by design for OCC: a single table-level lock is taken at commit time regardless of which partitions are written, and conflict resolution only lets non-overlapping writes both succeed. For concurrent-writer workloads where this serialization is the bottleneck, 1.x adds Non-Blocking Concurrency Control (NBCC) (hoodie.write.concurrency.mode=NON_BLOCKING_CONCURRENCY_CONTROL, since 1.0.0, HUDI-6949), which avoids blocking on a single lock for the whole write (works best with MOR + bucket index). 2. HoodieLockException: Unable to acquire lock, lock object null — the LockManager lock-acquisition path was rewritten in 1.0.0: it now retries and, on failure, reports the current lock owner instead of that null message. The DynamoDB lock provider was also reworked since 0.11 — migrated to AWS SDK v2 (HUDI-6638) and given a new implicit / auto-generated-partition-key variant (HUDI-8005, HUDI-8490). Since the original report was on 0.11 and went without a repro/logs, and the relevant lock code has since been substantially rewritten, I'm closing this out. If you still see lock-acquisition failures or unexpected serialization on a recent 1.x release, please reopen with the Hudi version, lock configs, and DynamoDB access logs and we'll dig in. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
