xichen01 commented on PR #4939: URL: https://github.com/apache/ozone/pull/4939#issuecomment-1631787405
> > Our business requires a deletion speed of at least 1000+ QPS to meet our business needs. The current Ozone deletion speed is not stable enough to reach this speed. > > I have seen another PR for solving duplicate delete blocks handling. > > IMO, we can do below changes to improve deletion speed, > > 1. increase the item size, block.deletion.per-interval.max > 2. reduce the frequency for triggering frequent getting blocks, ozone.block.deleting.service.timeout, like 30 sec in your case. > -- tuning above parameter can help achieve same result, as having code to keep queue of commands. Thanks you for the suggestion, we have increase the size of `block.deletion.per-interval.max` , but we are increase the `ozone.block.deleting.service.timeout` too, because if we just reduce this value from 60s to 30s, the valid(no duplicate) "Block deletion transaction" will reduce from 1/3 to 1/6 even more less, maybe 5/6 "Block deletion transaction" are duplicate. And some DN are slower than others, that cause most (sometimes is all) of the "Block deletion transaction"'s are sent to the slow DN, which results in the other normal DN can not receiving the new transaction(figure below). this PR can resolve this issue.  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
