yihua commented on PR #18439: URL: https://github.com/apache/hudi/pull/18439#issuecomment-4494754391
> The fix LGTM, however in addressing HTTP status code 429. > > However, i was thinking of other HTTP status/error codes that other providers might return. (Which may not be in the scope in this PR, we can track it separately and fix it in another PR). > > # S3 returns 503 > AWS S3 documents **503 Slow Down** as the rate-limit response for object operations S3 rarely returns 429 for puts, so on the S3 path, this PR fixes a case that almost never fires while leaving the actual S3 throttle path (503 SlowDown) falling into **UNKNOWN_ERROR**. (CMIIW) > > Source: https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html#:~:text=you%20may%20see%20some%20503%20(Slow%20Down)%20errors > > Azure does something similar: https://learn.microsoft.com/en-us/rest/api/storageservices/common-rest-api-error-codes)) > > # 400 > GCS retry strategy doc (https://cloud.google.com/storage/docs/retry-strategy) lists both 408, 429, and 5xx as retryable. > > # 408 > No source, but 408 is a request timeout and might be due to slow or network jitters which we might want to retry too instead of throwing UNKNOWN_ERROR. > > We might want to address these separately. In production deployment of the storage-based lock provider, so far we only see throttling on conditional writes on the same object on GCS due to the maximum rate of writes to the same object name (one write per second) (https://docs.cloud.google.com/storage/quotas#objects). So these follow-ups are low-priority. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
