[
https://issues.apache.org/jira/browse/HDFS-16891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17676798#comment-17676798
]
ASF GitHub Bot commented on HDFS-16891:
---------------------------------------
virajjasani opened a new pull request, #5300:
URL: https://github.com/apache/hadoop/pull/5300
If we enable parallel loading and persisting of inodes from/to fs image, we
get the benefit of improved performance. However, while loading sub-sections
INODE_DIR_SUB and INODE_SUB, if we encounter any errors, we use copy-on-write
list to maintain the list of exceptions. Since our usecase is not to iterate
over this list while executor threads are adding new elements to the list,
using copy-on-write is bit of an overhead for this usecase.
It would be better to synchronize adding new elements to the list rather
than having the list copy all elements over every time new element is added to
the list.
> Avoid the overhead of copy-on-write exception list while loading inodes sub
> sections in parallel
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-16891
> URL: https://issues.apache.org/jira/browse/HDFS-16891
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Affects Versions: 3.3.4
> Reporter: Viraj Jasani
> Assignee: Viraj Jasani
> Priority: Major
>
> If we enable parallel loading and persisting of inodes from/to fs image, we
> get the benefit of improved performance. However, while loading sub-sections
> INODE_DIR_SUB and INODE_SUB, if we encounter any errors, we use copy-on-write
> list to maintain the list of exceptions. Since our usecase is not to iterate
> over this list while executor threads are adding new elements to the list,
> using copy-on-write is bit of an overhead for this usecase.
> It would be better to synchronize adding new elements to the list rather than
> having the list copy all elements over every time new element is added to the
> list.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
