[
https://issues.apache.org/jira/browse/HDDS-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17242233#comment-17242233
]
Lokesh Jain commented on HDDS-4344:
-----------------------------------
Another test result:
ozone.key.deleting.limit.per.task - 20000 (OM deletes 20000 keys per minute)
hdds.scm.block.deletion.per-interval.max - 20000 (SCM processes 20000 blocks
for deletion every minute)
With 20000 keys/ minute processing speed at OM, deletion completes in about 70
minutes. I'll create a jira to update the default configs to 20000 keys/minute
for OM and 20000 blocks/min for SCM.
If we consider a datanode here. Every datanode has around 330GB of data after
writes are completed. In datanode the configs are such that it can delete a
maximum of 1000 blocks in 10 containers in a minute. Based on this number a
datanode can delete a maximum of 10GB per minute. It should take a minimum of
330 GB/10 = 33 minutes for every datanode to delete its data. Further the
recursive delete api in ozone fs takes around 25-30 mins to delete 1 million
keys.
> Block Deletion Performance Improvements
> ---------------------------------------
>
> Key: HDDS-4344
> URL: https://issues.apache.org/jira/browse/HDDS-4344
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Reporter: Lokesh Jain
> Assignee: Lokesh Jain
> Priority: Major
> Attachments: Block Deletion Performance.pdf
>
>
> In cluster deployments it was observed that block deletion can be slow. For
> example if a user writes a million keys in Ozone, the time it takes for those
> million keys to be deleted from datanodes can be high. The jira would cover
> various improvements which can be made for better deletion speeds.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
