[
https://issues.apache.org/jira/browse/HDDS-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795329#comment-16795329
]
Nanda kumar commented on HDDS-1302:
-----------------------------------
All the list calls (like list directories, files) in OzoneFS depends on list
key call of OzoneClient.
OzoneClient internally uses {{OzoneBucket#KeyIterator}} to iterate over the
list of keys in a bucket. This iterator has an internal cache whose size can be
configured using {{ozone.client.list.cache}} (default value is 1000). So after
every 1000 keys, we make a new RPC call to OM to get the next set of keys
(Iterator's internal logic).
Below is the call which the iterator uses:
{code:java|title=ClientProtocol.java}
/**
* Returns list of Keys in {Volume/Bucket} that matches the keyPrefix,
* size of the returned list depends on maxListResult. The caller has
* to make multiple calls to read all keys.
* @param volumeName Name of the Volume
* @param bucketName Name of the Bucket
* @param keyPrefix Bucket prefix to match
* @param prevKey Starting point of the list, this key is excluded
* @param maxListResult Max number of buckets to return.
* @return {@code List<OzoneKey>}
* @throws IOException
*/
List<OzoneKey> listKeys(String volumeName, String bucketName,
String keyPrefix, String prevKey, int maxListResult)
throws IOException;
{code}
As mentioned, the {{prevKey}} should be excluded in the result returned by OM,
else the iterator will get a duplicate value at the end of every 1000 key or
the value configured in {{ozone.client.list.cache}}.
* So if we do a list file operation on a directory which has more than 1000
(default value) files/directories, we will get some duplicates in the result.
* If we count the number of files in a directory, the result will be incorrect.
Note: Don't get confused by the name {{startId}} in OM code and {{prevKey}} in
client code, they are one and the same.
> Fix SCM CLI does not list container with id 1
> ---------------------------------------------
>
> Key: HDDS-1302
> URL: https://issues.apache.org/jira/browse/HDDS-1302
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Reporter: Bharat Viswanadham
> Assignee: Vivek Ratnavel Subramanian
> Priority: Minor
>
> In HDDS-1263 it is changed to handle the list containers with containerID 1
> by changing the actual logic of listContainers in ScmContainerManager.java.
> But now with this change, it is contradicting with the javadoc.
> From [~nandakumar131] comments
> https://issues.apache.org/jira/browse/HDDS-1263?focusedCommentId=16794865&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16794865
>
> I agree this will be the way to fix it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
