pkgajulapalli opened a new pull request, #19574: URL: https://github.com/apache/hudi/pull/19574
### Describe the issue this Pull Request addresses `StorageBasedLockProvider#unlock()` has three distinct failure paths that all throw a **byte-identical** `HoodieLockException` message: ```java throw new HoodieLockException(generateLockStateMessage(FAILED_TO_RELEASE)); ``` All three also share a single `updateLockReleaseFailureMetric` counter. So when a lock release fails in production, neither the exception nor the metric tells you which path produced it — and the causes call for completely different fixes (stop the writer's GC pressure, back off from a storage rate limit, or fix clock skew between nodes). The individual storage outcomes *are* already logged inside `tryExpireCurrentLock`. What is missing is any way to attribute the thrown exception to a cause, and any signal at all on two of the three paths. This matters because a failed release leaves the lock file in storage without its `expired: true` flag — i.e. a dangling lock that blocks every other writer on that table until the lease elapses. ### Summary and Changelog Each `FAILED_TO_RELEASE` throw now names its cause via a new `generateLockStateMessage(LockState, String cause)` overload, and each logs the context needed to act on it. | Cause | Meaning | |---|---| | `HEARTBEAT_STOP_FAILED` | The heartbeat task would not stop, so the lock is deliberately left un-expired (the task could still renew it after we return). Logs the `interrupted` flag to separate the two sub-cases in `LockProviderHeartbeatManager#stopHeartbeat`. | | `INTERRUPTED_DURING_THROTTLE_BACKOFF` | Interrupted mid-backoff. Now also passes the `InterruptedException` so the stack trace survives. | | `THROTTLE_RETRIES_EXHAUSTED` | The retry budget was exhausted against a storage rate limit (e.g. the GCS 1-write/sec per-object limit). | | `EXPIRE_WRITE_FAILED` | A terminal `UNKNOWN_ERROR` / `ACQUIRED_BY_OTHERS` outcome. | Additionally, on `ACQUIRED_BY_OTHERS` we now log how long ago our lease should have ended: - **positive** → we overran our own lease, pointing at a starved heartbeat (long GC, thread-pool starvation); - **negative** → the lease had not elapsed by *our* clock, pointing at clock skew between nodes. Those two are indistinguishable today and need different fixes. Every new message also carries `lockFilePath`, so a lock left dangling in storage can be joined back to the writer that failed to release it. Detailed changes: - `StorageBasedLockProvider`: added the `generateLockStateMessage(state, cause)` overload; added a `logger.error` at each of the three throw sites with the cause-specific context; passed `ie` into the interrupted-path log so the stack trace is retained; added the lease-overrun delta log on `ACQUIRED_BY_OTHERS`. - `TestStorageBasedLockProvider`: added `testUnlockThrowsExceptionWhenInterruptedDuringThrottleBackoff` (this path previously had no coverage); tightened the three existing `FAILED_TO_RELEASE` assertions to also pin the specific cause label, so a future refactor that collapses them fails the build. No code was copied. ### Impact None on behaviour. Control flow is untouched — every edit either adds a `logger.error` call or appends `, cause <LABEL>` to an existing exception message. No public API, config, or metric change (`updateLockReleaseFailureMetric` deliberately remains a single counter; splitting it per cause would change the metrics surface and is left for a separate change). Anything parsing these exception strings verbatim would see the appended `, cause <LABEL>` suffix. The existing `FAILED_TO_RELEASE` substring is preserved. ### Risk Level none — logging and exception-message text only. ### Documentation Update none — no new configs, no user-facing feature change. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable Verified locally on JDK 17: `TestStorageBasedLockProvider` — **48 tests, 0 failures, 0 errors**. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
