[
https://issues.apache.org/jira/browse/HDDS-13242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Andika updated HDDS-13242:
-------------------------------
Description:
TrashOzoneFileSystem#listStatus (used by TrashOzoneFileSystem#getTrashRoots)
uses Integer.MAX_VALUE as the number of list entries. However, since HDDS-7290,
a single listStatus is limited to ozone.fs.listing.page.size.max configuration,
which defaults to 1024.
Although it is not common to have more than 1024 trash roots (e.g. which
implies 1024 users deleting the same bucket). This might cause some trash roots
to not be processed. We can first write a test to validate the issue.
One solution is to use a listing iterator to send multiple smaller listStatus
API, similar to OzoneFileStatusIterator.
Note: The FS client implementation BasicRootedOzoneFileSystem#listStatusAdapter
(used by BasicRootedOzoneFileSystem#getTrashRoots) also default to 1024, so it
will also have the same issues. However, since getTrashRoots are only used for
server-side (OM) trash cleanup, the client-side getTrashRoots are rarely used.
was:
TrashOzoneFileSystem#listStatus (used by TrashOzoneFileSystem#getTrashRoots)
uses Integer.MAX_VALUE as the number of list entries. However, since HDDS-7290,
a single listStatus is limited to ozone.fs.listing.page.size.max configuration,
which defaults to 1024.
Although it is not common to have more than 1024 trash roots (e.g. which
implies 1024 users deleting the same bucket). This might cause some trash roots
to not be processed.
One solution is to use a listing iterator to send multiple smaller listStatus
API, similar to OzoneFileStatusIterator.
Note: The FS client implementation BasicRootedOzoneFileSystem#listStatusAdapter
(used by BasicRootedOzoneFileSystem#getTrashRoots) also default to 1024, so it
will also have the same issues. However, since getTrashRoots are only used for
server-side (OM) trash cleanup, the client-side getTrashRoots are rarely used.
> getTrashRoots is limited by the max list page size
> --------------------------------------------------
>
> Key: HDDS-13242
> URL: https://issues.apache.org/jira/browse/HDDS-13242
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Ivan Andika
> Priority: Minor
>
> TrashOzoneFileSystem#listStatus (used by TrashOzoneFileSystem#getTrashRoots)
> uses Integer.MAX_VALUE as the number of list entries. However, since
> HDDS-7290, a single listStatus is limited to ozone.fs.listing.page.size.max
> configuration, which defaults to 1024.
> Although it is not common to have more than 1024 trash roots (e.g. which
> implies 1024 users deleting the same bucket). This might cause some trash
> roots to not be processed. We can first write a test to validate the issue.
> One solution is to use a listing iterator to send multiple smaller listStatus
> API, similar to OzoneFileStatusIterator.
> Note: The FS client implementation
> BasicRootedOzoneFileSystem#listStatusAdapter (used by
> BasicRootedOzoneFileSystem#getTrashRoots) also default to 1024, so it will
> also have the same issues. However, since getTrashRoots are only used for
> server-side (OM) trash cleanup, the client-side getTrashRoots are rarely used.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@ozone.apache.org
For additional commands, e-mail: issues-h...@ozone.apache.org
