gavinchou opened a new pull request, #65974: URL: https://github.com/apache/doris/pull/65974
### What problem does this PR solve? Backport #64038 to `branch-4.1`. Add observability for S3 local rate limiting and S3/Azure 429 responses so throttling-related performance issues can be diagnosed and alerted more directly. ### What is changed? - Add explicit S3 local rate limiter bvars for sleep duration, sleep count, and rejected count. - Rename the old ambiguous S3 local rate limiter bvars so their meaning is clear. - Log the first and every N local S3 rate limiter throttled/rejected requests without reading bvar values for decisions. - Add S3 429 retry/failure bvars for S3/Azure object clients, including Azure batch delete failures. - Capture the limiter configuration and result under the limiter lock to avoid concurrent reset races. - Adapt the Cloud configuration change to `branch-4.1` without bringing in unrelated master-only fault-injection settings. ### New or renamed bvar metrics Renamed local S3 GET limiter metrics: - `get_rate_limit_ns` -> `s3_get_rate_limit_sleep_ns` - `get_rate_limit_exceed_req_num` -> `s3_get_rate_limit_sleep_count` Renamed local S3 PUT limiter metrics: - `put_rate_limit_ns` -> `s3_put_rate_limit_sleep_ns` - `put_rate_limit_exceed_req_num` -> `s3_put_rate_limit_sleep_count` New local S3 limiter rejection metrics: - `s3_get_rate_limit_rejected_count` - `s3_put_rate_limit_rejected_count` New S3/Azure 429 metrics: - `s3_request_retry_too_many_requests_count` - `s3_request_failed_too_many_requests_count` Metric examples: ```text s3_get_rate_limit_sleep_ns : 123456789 s3_get_rate_limit_sleep_count : 42 s3_get_rate_limit_rejected_count : 3 s3_put_rate_limit_sleep_ns : 987654321 s3_put_rate_limit_sleep_count : 27 s3_put_rate_limit_rejected_count : 1 s3_request_retry_too_many_requests_count : 8 s3_request_failed_too_many_requests_count : 2 ``` Example local limiter logs: ```text S3 GET request is throttled by local rate limiter, sleep_ms=12, sleep_count=1000, token_per_second=1000, bucket_tokens=2000, token_limit=5000 S3 PUT request is rejected by local rate limiter, rejected_count=1, token_per_second=1000, bucket_tokens=2000, token_limit=5000 ``` ### Tests - `sh run-cloud-ut.sh --run --filter='s3_rate_limiter_test:*'` (7/7 passed) - `sh run-be-ut.sh --run --filter='S3UTILTest.*:S3FileWriterTest.*'` (24/24 passed) - `sh run-be-ut.sh --run --filter='AzureObjStorageClientTlsHelperTest.*'` (2/2 passed) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
