lokeshj1703 commented on PR #17970:
URL: https://github.com/apache/hudi/pull/17970#issuecomment-4046450243
@nsivabalan Here is the commit message.
[HUDI-9159] Add support for StorageBasedLockProvider
Detailed commit msg:
Cherry-picks the following PRs into release-0.14.2:
- HUDI-9156: Add heartbeat manager for StorageBasedLockProvider
- HUDI-9158: Add storage based lock provider abstract implementation and
constructor support
- HUDI-9160: Add GCS implementation of StorageBasedLockProvider
- HUDI-9159: S3 implementation of StorageLock for StorageBasedLockProvider
- HUDI-9122: Add RFC updates and tweak to S3 storage lock client
- HUDI-9158: Rename config from
hoodie.write.lock.storage.heartbeat.poll.secs to
hoodie.write.lock.storage.renew.interval.secs
- HUDI-9742: Add lock release metrics
- HUDI-9752: Add metrics for storage based lock provider
- HUDI-9782: Add audit logging scaffolding and implementation for storage
based lock provider
- HUDI-9168: Refactor hudi-client-common to not import hudi-aws
Option 2:
Core Implementation (HUDI-9156, HUDI-9158, HUDI-9159, HUDI-9160):
- Added heartbeat manager for StorageBasedLockProvider to manage lock
renewals
- Implemented abstract storage based lock provider foundation
- Added S3 implementation of StorageLock with optimizations
- Added GCS implementation of StorageBasedLockProvider
- Added backward-compatible constructor supporting 0.14.x lock provider
format
- Refactored hudi-client-common to remove hudi-aws dependency
Configuration Changes (HUDI-9158):
- Renamed config: hoodie.write.lock.storage.heartbeat.poll.secs →
hoodie.write.lock.storage.renew.interval.secs
Metrics & Monitoring (HUDI-9742, HUDI-9752):
- Added lock release metrics
- Added comprehensive metrics for storage based lock provider
- Prevent lock metrics failure from failing the process
Audit & Validation (HUDI-9782):
- Added audit logging scaffolding for storage based lock provider
- Implemented audit service for storage based lock provider
- Added validation and cleanup APIs for storage lock provider audit
validation
Bug Fixes:
- Exit transaction with error when unlock failure due to lock acquired by
others
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
