deepakpanda93 commented on issue #7487: URL: https://github.com/apache/hudi/issues/7487#issuecomment-4844645594
This is S3 request-rate throttling (HTTP 503 / SlowDown on LIST/GET) rather than a hard quota — S3 allows ~5,500 GET and ~3,500 mutating requests per second per prefix, and heavy recursive LIST during file-system-view building (amplified by many small files / uncleaned log files) can exceed that on large tables. The main lever to cut the LIST volume is the metadata table, whose FILES partition serves file listings directly and avoids recursive S3 LIST. It's enabled by default on the writer (hoodie.metadata.enable=true); make sure it's also enabled on the reader/query side. On top of that, a series of changes have reduced unnecessary FS calls — e.g. HUDI-5477 ([#7561](https://github.com/apache/hudi/pull/7561), timeline loading in the sync client) and broader file-index/listing optimizations that continue through 1.x. Recommendations: - Run on a recent 1.x release with the metadata table enabled for both writes and reads. - Keep cleaning/compaction healthy to avoid small-file/log-file buildup (see the related [#7600](https://github.com/apache/hudi/issues/7600)); consider clustering if you have many small files. - For unavoidable bursts, rely on S3 request retries / increase prefix spread. Since this is throttling/tuning (no single code defect) and the original reports were on 0.11–0.12, I'm closing this out. If you still see 503 throttling on 1.x with the metadata table enabled, please reopen with S3 request-debug logs (log4j.logger.com.amazonaws.request=DEBUG) showing which prefix drives the LIST volume, and we'll dig in. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
