dlmarion commented on issue #6413: URL: https://github.com/apache/accumulo/issues/6413#issuecomment-4650627108
FWIW it look like there were two efforts to resolve this issue in the NameNode. Neither were merged. https://issues.apache.org/jira/browse/HDFS-7174 https://issues.apache.org/jira/browse/HDFS-9053 I think there might be a penalty for having too many subdirectories, not only in memory consumption in the NameNode, but also in time traversing the directories when trying to list files. https://github.com/apache/hadoop/pull/8064 made the `dfs.namenode.fs-limits.max-directory-items` property reconfigurable in Hadoop 3.5.0. Looking at that PR, there is a hard limit of 6.4 million objects. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
