[
https://issues.apache.org/jira/browse/HDDS-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xu Shao Hong updated HDDS-7328:
-------------------------------
Description:
Currently, the deleting speed of paths in the FSO bucket is too slow.
Every time the DirectoryDeletingService executes(every 1m by default), it
chooses only one path from DeletedDirectroyTable and tries adding its sub-files
and sub-dirs into the PurgeDirectories Request. The real deletion happens in
OMKeyDeleteResponseWithFSO, which moved all sub-dirs into
DeletedDirectroyTable. Then these sub-dirs will be chosen by the future
execution of DirectoryDeletingService.
In the real production environment, such deletion speed is not applicable.
There could be lots of dirs deleted without sub-dirs or sub-files. The dir
deletion speed would lag behind the dir creation speed.
I propose to optimize the logic of the DirectoryDeletingService, we could
consume all the quota (ozone.path.deleting.limit.per.task) to delete paths as
much as possible. The good news is the PB is already designed to do so, we
could add more PurgePathRequest in one PurgeDirectoriesRequest.
was:
Currently, the deleting speed of paths in the FSO bucket is too slow.
Every time the DirectoryDeletingService executes(every 1m by default), it
chooses only one path from DeletedDirectroyTable and tries adding its sub-files
and sub-dirs into the PurgeDirectories Request. The real deletion happens in
OMKeyDeleteResponseWithFSO, which moved all sub-dirs into
DeletedDirectroyTable. Then these sub-dirs will be chosen by the future
execution of DirectoryDeletingService.
In the real production environment, such deletion speed is not applicable.
There could be lots of dirs deleted without sub-dirs or sub-files. The dir
deletion speed would lag behind the dir creation speed.
In this PR, I propose to optimize the logic of the DirectoryDeletingService, we
could consume all the quota (ozone.path.deleting.limit.per.task) to delete
paths as much as possible. The good news is the PB is already designed to do
so, we could add more PurgePathRequest in one PurgeDirectoriesRequest.
Originally: 1 min. At most 1 path from deletedDirectoryTable.
Now: 1min. At most 10000 paths from deletedDirectoryTable.
> Improve Deletion of FSO Paths
> -----------------------------
>
> Key: HDDS-7328
> URL: https://issues.apache.org/jira/browse/HDDS-7328
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Xu Shao Hong
> Assignee: Xu Shao Hong
> Priority: Major
> Labels: pull-request-available
>
> Currently, the deleting speed of paths in the FSO bucket is too slow.
> Every time the DirectoryDeletingService executes(every 1m by default), it
> chooses only one path from DeletedDirectroyTable and tries adding its
> sub-files and sub-dirs into the PurgeDirectories Request. The real deletion
> happens in OMKeyDeleteResponseWithFSO, which moved all sub-dirs into
> DeletedDirectroyTable. Then these sub-dirs will be chosen by the future
> execution of DirectoryDeletingService.
> In the real production environment, such deletion speed is not applicable.
> There could be lots of dirs deleted without sub-dirs or sub-files. The dir
> deletion speed would lag behind the dir creation speed.
> I propose to optimize the logic of the DirectoryDeletingService, we could
> consume all the quota (ozone.path.deleting.limit.per.task) to delete paths as
> much as possible. The good news is the PB is already designed to do so, we
> could add more PurgePathRequest in one PurgeDirectoriesRequest.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
