[
https://issues.apache.org/jira/browse/HDFS-15196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064094#comment-17064094
]
Fengnan Li commented on HDFS-15196:
-----------------------------------
[~ayushtkn] Thanks for the explanation. I agree that from a client's
perspective, it doesn't matter where the entries are from.
I was wrong in my previous reply and I thought twice about this. My point is
that adding this value doesn't make much difference.
Let's take one example. say we have namenode listing limit as 3 and there is
/parent/file-0, /parent/file-2, /parent/file-3, /parent/file-4 in namenodes,
/parent/file-1, /parent/file-5, /parent/file-6 in mount points.
Current patch:
In the first client call, namenode will return file-0, file-2 and file3, router
will insert file-1 to make the result file-[0-3] and remaining count as 1
(file-4). The client will see this non-zero value and send another call with
startAfter as file-3, this time namenode will return file-4 and router will
append file-5 and file-6. At this moment, remainingEntries is 0 and all results
are returned to clients. No further call will be made.
If we add remainingEntries, the first batch will be returned along with
remainingEntries 3 (1 from namenode + 2 from routers). The remainingEntries in
the second call would be 0, no further call as well.
In both cases clients would send the same amount of calls and get all results.
> RBF: RouterRpcServer getListing cannot list large dirs correctly
> ----------------------------------------------------------------
>
> Key: HDFS-15196
> URL: https://issues.apache.org/jira/browse/HDFS-15196
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Fengnan Li
> Assignee: Fengnan Li
> Priority: Critical
> Attachments: HDFS-15196.001.patch, HDFS-15196.002.patch,
> HDFS-15196.003.patch, HDFS-15196.003.patch, HDFS-15196.004.patch,
> HDFS-15196.005.patch, HDFS-15196.006.patch, HDFS-15196.007.patch,
> HDFS-15196.008.patch, HDFS-15196.009.patch, HDFS-15196.010.patch
>
>
> In RouterRpcServer, getListing function is handled as two parts:
> # Union all partial listings from destination ns + paths
> # Append mount points for the dir to be listed
> In the case of large dir which is bigger than DFSConfigKeys.DFS_LIST_LIMIT
> (with default value 1k), the batch listing will be used and the startAfter
> will be used to define the boundary of each batch listing. However, step 2
> here will add existing mount points, which will mess up with the boundary of
> the batch, thus making the next batch startAfter wrong.
> The fix is just to append the mount points when there is no more batch query
> necessary.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
