[
https://issues.apache.org/jira/browse/HDDS-11508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai resolved HDDS-11508.
-------------------------------------
Fix Version/s: 2.0.0
Resolution: Implemented
> Decouple delete batch limits from Ratis request size
> ----------------------------------------------------
>
> Key: HDDS-11508
> URL: https://issues.apache.org/jira/browse/HDDS-11508
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: OM, SCM
> Reporter: Ethan Rose
> Assignee: Sadanand Shenoy
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0
>
>
> Currently the background deletion services in OM and SCM send one Ratis
> request per iteration. This can result in very large protos if the task's
> limits are high, and Ratis may reject them due to their size. HDDS-8977 tried
> to fix this by check the Ratis message size and falling back to a hardcoded
> limit of 1000 entries if the message is too big, but this is too conservative
> and will over-throttle the service under heavy deletion load.
> For each run, these entries should collect as many items as is their
> configured limit, and then break them into multiple Ratis requests to send
> out. This way limit per task can be set purely based on the time taken to
> gather the entries, and not based on size of the resulting proto.
> This Jira may optionally be split into subtasks for each service:
> * OM open key cleanup
> * OM file delete
> * OM directory delete
> * SCM block delete
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
