[jira] [Updated] (HDFS-13693) Remove unnecessary search in INodeDirectory.addChild during image loading
[ https://issues.apache.org/jira/browse/HDFS-13693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-13693: --- Fix Version/s: 3.2.2 3.1.4 > Remove unnecessary search in INodeDirectory.addChild during image loading > - > > Key: HDFS-13693 > URL: https://issues.apache.org/jira/browse/HDFS-13693 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: zhouyingchao >Assignee: Lisheng Sun >Priority: Major > Fix For: 3.3.0, 3.1.4, 3.2.2 > > Attachments: HDFS-13693-001.patch, HDFS-13693-002.patch, > HDFS-13693-003.patch, HDFS-13693-004.patch, HDFS-13693-005.patch > > > In FSImageFormatPBINode.loadINodeDirectorySection, all child INodes are added > to their parent INode's map one by one. The adding procedure will search a > position in the parent's map and then insert the child to the position. > However, during image loading, the search is unnecessary since the insert > position should always be at the end of the map given the sequence they are > serialized on disk. > Test this patch against a fsimage of a 70PB cluster (200million files and > 300million blocks), the image loading time be reduced from 1210 seconds to > 1138 seconds.So it can reduce up to about 10% of time. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13693) Remove unnecessary search in INodeDirectory.addChild during image loading
[ https://issues.apache.org/jira/browse/HDFS-13693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-13693: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.3.0 Status: Resolved (was: Patch Available) > Remove unnecessary search in INodeDirectory.addChild during image loading > - > > Key: HDFS-13693 > URL: https://issues.apache.org/jira/browse/HDFS-13693 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: zhouyingchao >Assignee: Lisheng Sun >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-13693-001.patch, HDFS-13693-002.patch, > HDFS-13693-003.patch, HDFS-13693-004.patch, HDFS-13693-005.patch > > > In FSImageFormatPBINode.loadINodeDirectorySection, all child INodes are added > to their parent INode's map one by one. The adding procedure will search a > position in the parent's map and then insert the child to the position. > However, during image loading, the search is unnecessary since the insert > position should always be at the end of the map given the sequence they are > serialized on disk. > Test this patch against a fsimage of a 70PB cluster (200million files and > 300million blocks), the image loading time be reduced from 1210 seconds to > 1138 seconds.So it can reduce up to about 10% of time. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13693) Remove unnecessary search in INodeDirectory.addChild during image loading
[ https://issues.apache.org/jira/browse/HDFS-13693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-13693: --- Attachment: HDFS-13693-004.patch > Remove unnecessary search in INodeDirectory.addChild during image loading > - > > Key: HDFS-13693 > URL: https://issues.apache.org/jira/browse/HDFS-13693 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: zhouyingchao >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-13693-001.patch, HDFS-13693-002.patch, > HDFS-13693-003.patch, HDFS-13693-004.patch, HDFS-13693-005.patch > > > In FSImageFormatPBINode.loadINodeDirectorySection, all child INodes are added > to their parent INode's map one by one. The adding procedure will search a > position in the parent's map and then insert the child to the position. > However, during image loading, the search is unnecessary since the insert > position should always be at the end of the map given the sequence they are > serialized on disk. > Test this patch against a fsimage of a 70PB cluster (200million files and > 300million blocks), the image loading time be reduced from 1210 seconds to > 1138 seconds.So it can reduce up to about 10% of time. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13693) Remove unnecessary search in INodeDirectory.addChild during image loading
[ https://issues.apache.org/jira/browse/HDFS-13693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-13693: --- Attachment: (was: HDFS-13693-004.patch) > Remove unnecessary search in INodeDirectory.addChild during image loading > - > > Key: HDFS-13693 > URL: https://issues.apache.org/jira/browse/HDFS-13693 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: zhouyingchao >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-13693-001.patch, HDFS-13693-002.patch, > HDFS-13693-003.patch, HDFS-13693-004.patch, HDFS-13693-005.patch > > > In FSImageFormatPBINode.loadINodeDirectorySection, all child INodes are added > to their parent INode's map one by one. The adding procedure will search a > position in the parent's map and then insert the child to the position. > However, during image loading, the search is unnecessary since the insert > position should always be at the end of the map given the sequence they are > serialized on disk. > Test this patch against a fsimage of a 70PB cluster (200million files and > 300million blocks), the image loading time be reduced from 1210 seconds to > 1138 seconds.So it can reduce up to about 10% of time. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13693) Remove unnecessary search in INodeDirectory.addChild during image loading
[ https://issues.apache.org/jira/browse/HDFS-13693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-13693: --- Attachment: HDFS-13693-005.patch > Remove unnecessary search in INodeDirectory.addChild during image loading > - > > Key: HDFS-13693 > URL: https://issues.apache.org/jira/browse/HDFS-13693 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: zhouyingchao >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-13693-001.patch, HDFS-13693-002.patch, > HDFS-13693-003.patch, HDFS-13693-004.patch, HDFS-13693-005.patch > > > In FSImageFormatPBINode.loadINodeDirectorySection, all child INodes are added > to their parent INode's map one by one. The adding procedure will search a > position in the parent's map and then insert the child to the position. > However, during image loading, the search is unnecessary since the insert > position should always be at the end of the map given the sequence they are > serialized on disk. > Test this patch against a fsimage of a 70PB cluster (200million files and > 300million blocks), the image loading time be reduced from 1210 seconds to > 1138 seconds.So it can reduce up to about 10% of time. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13693) Remove unnecessary search in INodeDirectory.addChild during image loading
[ https://issues.apache.org/jira/browse/HDFS-13693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-13693: --- Attachment: HDFS-13693-004.patch > Remove unnecessary search in INodeDirectory.addChild during image loading > - > > Key: HDFS-13693 > URL: https://issues.apache.org/jira/browse/HDFS-13693 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: zhouyingchao >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-13693-001.patch, HDFS-13693-002.patch, > HDFS-13693-003.patch, HDFS-13693-004.patch > > > In FSImageFormatPBINode.loadINodeDirectorySection, all child INodes are added > to their parent INode's map one by one. The adding procedure will search a > position in the parent's map and then insert the child to the position. > However, during image loading, the search is unnecessary since the insert > position should always be at the end of the map given the sequence they are > serialized on disk. > Test this patch against a fsimage of a 70PB cluster (200million files and > 300million blocks), the image loading time be reduced from 1210 seconds to > 1138 seconds.So it can reduce up to about 10% of time. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13693) Remove unnecessary search in INodeDirectory.addChild during image loading
[ https://issues.apache.org/jira/browse/HDFS-13693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-13693: --- Attachment: HDFS-13693-002.patch > Remove unnecessary search in INodeDirectory.addChild during image loading > - > > Key: HDFS-13693 > URL: https://issues.apache.org/jira/browse/HDFS-13693 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: zhouyingchao >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-13693-001.patch, HDFS-13693-002.patch > > > In FSImageFormatPBINode.loadINodeDirectorySection, all child INodes are added > to their parent INode's map one by one. The adding procedure will search a > position in the parent's map and then insert the child to the position. > However, during image loading, the search is unnecessary since the insert > position should always be at the end of the map given the sequence they are > serialized on disk. > Test this patch against a fsimage of a 70PB cluster (200million files and > 300million blocks), the image loading time be reduced from 1210 seconds to > 1138 seconds.So it can reduce up to about 10% of time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13693) Remove unnecessary search in INodeDirectory.addChild during image loading
[ https://issues.apache.org/jira/browse/HDFS-13693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-13693: --- Description: In FSImageFormatPBINode.loadINodeDirectorySection, all child INodes are added to their parent INode's map one by one. The adding procedure will search a position in the parent's map and then insert the child to the position. However, during image loading, the search is unnecessary since the insert position should always be at the end of the map given the sequence they are serialized on disk. Test this patch against a fsimage of a 70PB cluster (200million files and 300million blocks), the image loading time be reduced from 1210 seconds to 1138 seconds.So it can reduce up to about 10% of time. was: In FSImageFormatPBINode.loadINodeDirectorySection, all child INodes are added to their parent INode's map one by one. The adding procedure will search a position in the parent's map and then insert the child to the position. However, during image loading, the search is unnecessary since the insert position should always be at the end of the map given the sequence they are serialized on disk. Test this patch against a fsimage of a 70PB cluster (200million files and 300million blocks), the image loading time be reduced from 1210 seconds to 1138 seconds. > Remove unnecessary search in INodeDirectory.addChild during image loading > - > > Key: HDFS-13693 > URL: https://issues.apache.org/jira/browse/HDFS-13693 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: zhouyingchao >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-13693-001.patch > > > In FSImageFormatPBINode.loadINodeDirectorySection, all child INodes are added > to their parent INode's map one by one. The adding procedure will search a > position in the parent's map and then insert the child to the position. > However, during image loading, the search is unnecessary since the insert > position should always be at the end of the map given the sequence they are > serialized on disk. > Test this patch against a fsimage of a 70PB cluster (200million files and > 300million blocks), the image loading time be reduced from 1210 seconds to > 1138 seconds.So it can reduce up to about 10% of time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13693) Remove unnecessary search in INodeDirectory.addChild during image loading
[ https://issues.apache.org/jira/browse/HDFS-13693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-13693: --- Description: In FSImageFormatPBINode.loadINodeDirectorySection, all child INodes are added to their parent INode's map one by one. The adding procedure will search a position in the parent's map and then insert the child to the position. However, during image loading, the search is unnecessary since the insert position should always be at the end of the map given the sequence they are serialized on disk. Test this patch against a fsimage of a 70PB cluster (200million files and 300million blocks), the image loading time be reduced from 1210 seconds to 1138 seconds. was: In FSImageFormatPBINode.loadINodeDirectorySection, all child INodes are added to their parent INode's map one by one. The adding procedure will search a position in the parent's map and then insert the child to the position. However, during image loading, the search is unnecessary since the insert position should always be at the end of the map given the sequence they are serialized on disk. Test this patch against > Remove unnecessary search in INodeDirectory.addChild during image loading > - > > Key: HDFS-13693 > URL: https://issues.apache.org/jira/browse/HDFS-13693 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: zhouyingchao >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-13693-001.patch > > > In FSImageFormatPBINode.loadINodeDirectorySection, all child INodes are added > to their parent INode's map one by one. The adding procedure will search a > position in the parent's map and then insert the child to the position. > However, during image loading, the search is unnecessary since the insert > position should always be at the end of the map given the sequence they are > serialized on disk. > > Test this patch against a fsimage of a 70PB cluster (200million files and > 300million blocks), the image loading time be reduced from 1210 seconds to > 1138 seconds. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13693) Remove unnecessary search in INodeDirectory.addChild during image loading
[ https://issues.apache.org/jira/browse/HDFS-13693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-13693: --- Description: In FSImageFormatPBINode.loadINodeDirectorySection, all child INodes are added to their parent INode's map one by one. The adding procedure will search a position in the parent's map and then insert the child to the position. However, during image loading, the search is unnecessary since the insert position should always be at the end of the map given the sequence they are serialized on disk. Test this patch against was:In FSImageFormatPBINode.loadINodeDirectorySection, all child INodes are added to their parent INode's map one by one. The adding procedure will search a position in the parent's map and then insert the child to the position. However, during image loading, the search is unnecessary since the insert position should always be at the end of the map given the sequence they are serialized on disk. > Remove unnecessary search in INodeDirectory.addChild during image loading > - > > Key: HDFS-13693 > URL: https://issues.apache.org/jira/browse/HDFS-13693 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: zhouyingchao >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-13693-001.patch > > > In FSImageFormatPBINode.loadINodeDirectorySection, all child INodes are added > to their parent INode's map one by one. The adding procedure will search a > position in the parent's map and then insert the child to the position. > However, during image loading, the search is unnecessary since the insert > position should always be at the end of the map given the sequence they are > serialized on disk. > > Test this patch against -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13693) Remove unnecessary search in INodeDirectory.addChild during image loading
[ https://issues.apache.org/jira/browse/HDFS-13693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhouyingchao updated HDFS-13693: Attachment: HDFS-13693-001.patch Status: Patch Available (was: Open) Run all hdfs related unit tests and does not introduce new failures. > Remove unnecessary search in INodeDirectory.addChild during image loading > - > > Key: HDFS-13693 > URL: https://issues.apache.org/jira/browse/HDFS-13693 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: zhouyingchao >Priority: Major > Attachments: HDFS-13693-001.patch > > > In FSImageFormatPBINode.loadINodeDirectorySection, all child INodes are added > to their parent INode's map one by one. The adding procedure will search a > position in the parent's map and then insert the child to the position. > However, during image loading, the search is unnecessary since the insert > position should always be at the end of the map given the sequence they are > serialized on disk. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
