nsivabalan opened a new issue, #20027:
URL: https://github.com/apache/hudi/issues/20027
### Describe the problem you faced
`StorageBasedLockProvider` can leave a dangling lock file when the storage
backend returns an HTTP 5xx on the lock-expire write.
Observed in production: a writer acquired the lock, completed everything it
needed to do inside the lock, and then hit `503 SERVICE_UNAVAILABLE` when
releasing. The lock file was left un-expired, and ~30s later a second writer
detected the dangling lock and alerted.
This is the same class of problem as #18438 (fixed in #18439 for HTTP 429),
but 5xx was never covered.
**Root cause.** All three storage lock clients classify HTTP 5xx as
`UNKNOWN_ERROR` — the 5xx branch logs a warning and then falls through to the
`UNKNOWN_ERROR` return:
```java
} else if (e.getCode() == RATE_LIMIT_ERROR_CODE) { // 429 ->
THROTTLED (fixed in #18439)
return Pair.of(LockUpsertResult.THROTTLED, Option.empty());
} else if (e.getCode() >= INTERNAL_SERVER_ERROR_CODE_MIN) { // 5xx -> falls
through
logger.warn(...);
}
return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty()); // <- 503
lands here
```
`tryExpireCurrentLock` maps `UNKNOWN_ERROR` to `ExpireLockResult.FAILED`,
and the retry loop in `unlock()` only engages while the result is `THROTTLED`:
```java
for (int attempt = 1; attempt <= THROTTLE_MAX_RETRIES && expireResult ==
ExpireLockResult.THROTTLED; attempt++) {
```
So a 503 fails the release on the **first** attempt with no retry at all.
The cloud SDKs do not cover for this either — both clients are deliberately
constructed with retries disabled (`S3StorageLockClient` uses `retryStrategy(r
-> r.maxAttempts(1))`, `GCSStorageLockClient` uses
`RetrySettings.setMaxAttempts(1)`), so a single transient 5xx dangles the lock
every time.
Affects `S3StorageLockClient`, `GCSStorageLockClient` and
`AzureStorageLockClient`.
### To Reproduce
1. Configure `StorageBasedLockProvider` against S3, GCS or Azure.
2. Acquire the lock and do some work.
3. Have the storage backend return HTTP 503 (or any 5xx) for the lock-expire
`PUT`.
4. `unlock()` throws `FAILED_TO_RELEASE` immediately, with no retry, and the
lock file remains un-expired until its lease elapses.
### Expected behavior
A 5xx should be treated as a retriable, server-side rejection rather than an
indeterminate lock state. The conditional write is known not to have taken
effect, so retrying the identical write is safe: if the write never landed the
precondition still matches and the retry succeeds; if it silently did land the
token no longer matches, yielding `412` -> `ACQUIRED_BY_OTHERS`, which is
already handled as released. Either way the retry converges.
### Environment Description
* Hudi version: master
* Storage: S3 / GCS / Azure Blob
* Running on Docker? : no
### Additional context
A fix is ready and will be linked shortly. It adds
`LockUpsertResult.TRANSIENT_ERROR` for HTTP 5xx and reuses the existing retry
machinery from #18439 on the release path, plus bounded retries on the acquire
and renew paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
