[
https://issues.apache.org/jira/browse/HDDS-13242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Andika updated HDDS-13242:
-------------------------------
Description:
TrashOzoneFileSystem#listStatus (used by TrashOzoneFileSystem#getTrashRoots)
uses Integer.MAX_VALUE as the number of list entries. However, since HDDS-7290,
a single listStatus is limited to ozone.fs.listing.page.size.max configuration,
which defaults to 5000.
Although it is not common to have more than 5000 trash roots (e.g. which
implies 5000 unique users deletes to trash in the same bucket). This might
cause some trash roots to not be processed. We can first write a test to
validate the issue.
One solution is to use a listing iterator to send multiple smaller listStatus
API, similar to OzoneFileStatusIterator. We can improve this iterator by
persistently storing the current RocksDB iterator across listStatus calls
instead of creating one for every listStatus call.
Note: The FS client implementation BasicRootedOzoneFileSystem#listStatusAdapter
(used by BasicRootedOzoneFileSystem#getTrashRoots) also default to 5000, so it
will also have the same issues. However, since getTrashRoots are only used for
server-side (OM) trash cleanup, the client-side getTrashRoots are rarely used.
was:
TrashOzoneFileSystem#listStatus (used by TrashOzoneFileSystem#getTrashRoots)
uses Integer.MAX_VALUE as the number of list entries. However, since HDDS-7290,
a single listStatus is limited to ozone.fs.listing.page.size.max configuration,
which defaults to 1024.
Although it is not common to have more than 1024 trash roots (e.g. which
implies 1024 unique users deletes to trash in the same bucket). This might
cause some trash roots to not be processed. We can first write a test to
validate the issue.
One solution is to use a listing iterator to send multiple smaller listStatus
API, similar to OzoneFileStatusIterator. We can improve this iterator by
persistently storing the current RocksDB iterator across listStatus calls
instead of creating one for every listStatus call.
Note: The FS client implementation BasicRootedOzoneFileSystem#listStatusAdapter
(used by BasicRootedOzoneFileSystem#getTrashRoots) also default to 1024, so it
will also have the same issues. However, since getTrashRoots are only used for
server-side (OM) trash cleanup, the client-side getTrashRoots are rarely used.
> getTrashRoots is limited by the max list page size
> --------------------------------------------------
>
> Key: HDDS-13242
> URL: https://issues.apache.org/jira/browse/HDDS-13242
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Minor
> Labels: pull-request-available
>
> TrashOzoneFileSystem#listStatus (used by TrashOzoneFileSystem#getTrashRoots)
> uses Integer.MAX_VALUE as the number of list entries. However, since
> HDDS-7290, a single listStatus is limited to ozone.fs.listing.page.size.max
> configuration, which defaults to 5000.
> Although it is not common to have more than 5000 trash roots (e.g. which
> implies 5000 unique users deletes to trash in the same bucket). This might
> cause some trash roots to not be processed. We can first write a test to
> validate the issue.
> One solution is to use a listing iterator to send multiple smaller listStatus
> API, similar to OzoneFileStatusIterator. We can improve this iterator by
> persistently storing the current RocksDB iterator across listStatus calls
> instead of creating one for every listStatus call.
> Note: The FS client implementation
> BasicRootedOzoneFileSystem#listStatusAdapter (used by
> BasicRootedOzoneFileSystem#getTrashRoots) also default to 5000, so it will
> also have the same issues. However, since getTrashRoots are only used for
> server-side (OM) trash cleanup, the client-side getTrashRoots are rarely used.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
