platinumhamburg commented on PR #3404: URL: https://github.com/apache/fluss/pull/3404#issuecomment-4716240199
> The RateLimiter in the job only throttles deletes. The job also issues many read-side FS calls — especially `listStatus` over log tablet dirs that can hold tens of thousands of segment dirs. Unthrottled, this can saturate the underlying store and impact cluster stability. Out of scope for this PR. Could you open a tracking issue for this and replace this TODO with a one-line pointer to it? @wuchong Thanks for catching this. I agree this should be fixed in this PR because unthrottled remote filesystem reads/lists can be a production stability risk on object stores with QPS limits. I updated the action to use a single shared remote filesystem operation limiter instead of limiting deletes only. The new option is --remote-fs-op-rate-limit-per-second, and the budget covers remote FS metadata reads, manifest reads, listStatus, and delete operations. For the distributed ScanAndClean stage, the configured rate is split across subtasks based on the runtime operator parallelism. This is a best-effort job-level target because Flink does not provide a cross-JVM global RateLimiter for this action. Scope enumeration uses the same option as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
