[jira] [Updated] (HDFS-12225) [SPS]: Optimize extended attributes for tracking SPS movements
[
https://issues.apache.org/jira/browse/HDFS-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-12225:
Fix Version/s: 3.2.0
> [SPS]: Optimize extended attributes for tracking SPS movements
> --
>
> Key: HDFS-12225
> URL: https://issues.apache.org/jira/browse/HDFS-12225
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Surendra Singh Lilhore
>Priority: Major
> Fix For: HDFS-10285, 3.2.0
>
> Attachments: HDFS-12225-HDFS-10285-01.patch,
> HDFS-12225-HDFS-10285-02.patch, HDFS-12225-HDFS-10285-03.patch,
> HDFS-12225-HDFS-10285-04.patch, HDFS-12225-HDFS-10285-05.patch,
> HDFS-12225-HDFS-10285-06.patch, HDFS-12225-HDFS-10285-07.patch,
> HDFS-12225-HDFS-10285-08.patch, HDFS-12225-HDFS-10285-09.patch
>
>
> We have discussed to optimize number extended attributes and asked to report
> separate JIRA while implementing [HDFS-11150 |
> https://issues.apache.org/jira/browse/HDFS-11150?focusedCommentId=15766127&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15766127]
> This is the JIRA to track that work
> For the context, comment copied from HDFS-11150
> {quote}
> [~yuanbo] wrote : I've tried that before. There is an issue here if we only
> mark the directory. When recovering from FsImage, the InodeMap isn't built
> up, so we don't know the sub-inode of a given inode, in the end, We cannot
> add these inodes to movement queue in FSDirectory#addToInodeMap, any
> thoughts?{quote}
> {quote}
> [~umamaheswararao] wrote: I got what you are saying. Ok for simplicity we can
> add for all Inodes now. For this to handle 100%, we may need intermittent
> processing, like first we should add them to some intermittentList while
> loading fsImage, once fully loaded and when starting active services, we
> should process that list and do required stuff. But it would add some
> additional complexity may be. Let's do with all file inodes now and we can
> revisit later if it is really creating issues. How about you raise a JIRA for
> it and think to optimize separately?
> {quote}
> {quote}
> [~andrew.wang] wrote in HDFS-10285 merge time review comment : HDFS-10899
> also the cursor of the iterator in the EZ root xattr to track progress and
> handle restarts. I wonder if we can do something similar here to avoid having
> an xattr-per-file being moved.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12225) [SPS]: Optimize extended attributes for tracking SPS movements
[
https://issues.apache.org/jira/browse/HDFS-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-12225:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: HDFS-10285
Status: Resolved (was: Patch Available)
I have just pushed it to branch. Thanks [~rakeshr] for the reviews
> [SPS]: Optimize extended attributes for tracking SPS movements
> --
>
> Key: HDFS-12225
> URL: https://issues.apache.org/jira/browse/HDFS-12225
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Surendra Singh Lilhore
> Fix For: HDFS-10285
>
> Attachments: HDFS-12225-HDFS-10285-01.patch,
> HDFS-12225-HDFS-10285-02.patch, HDFS-12225-HDFS-10285-03.patch,
> HDFS-12225-HDFS-10285-04.patch, HDFS-12225-HDFS-10285-05.patch,
> HDFS-12225-HDFS-10285-06.patch, HDFS-12225-HDFS-10285-07.patch,
> HDFS-12225-HDFS-10285-08.patch, HDFS-12225-HDFS-10285-09.patch
>
>
> We have discussed to optimize number extended attributes and asked to report
> separate JIRA while implementing [HDFS-11150 |
> https://issues.apache.org/jira/browse/HDFS-11150?focusedCommentId=15766127&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15766127]
> This is the JIRA to track that work
> For the context, comment copied from HDFS-11150
> {quote}
> [~yuanbo] wrote : I've tried that before. There is an issue here if we only
> mark the directory. When recovering from FsImage, the InodeMap isn't built
> up, so we don't know the sub-inode of a given inode, in the end, We cannot
> add these inodes to movement queue in FSDirectory#addToInodeMap, any
> thoughts?{quote}
> {quote}
> [~umamaheswararao] wrote: I got what you are saying. Ok for simplicity we can
> add for all Inodes now. For this to handle 100%, we may need intermittent
> processing, like first we should add them to some intermittentList while
> loading fsImage, once fully loaded and when starting active services, we
> should process that list and do required stuff. But it would add some
> additional complexity may be. Let's do with all file inodes now and we can
> revisit later if it is really creating issues. How about you raise a JIRA for
> it and think to optimize separately?
> {quote}
> {quote}
> [~andrew.wang] wrote in HDFS-10285 merge time review comment : HDFS-10899
> also the cursor of the iterator in the EZ root xattr to track progress and
> handle restarts. I wonder if we can do something similar here to avoid having
> an xattr-per-file being moved.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12225) [SPS]: Optimize extended attributes for tracking SPS movements
[
https://issues.apache.org/jira/browse/HDFS-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12225:
--
Attachment: HDFS-12225-HDFS-10285-09.patch
> [SPS]: Optimize extended attributes for tracking SPS movements
> --
>
> Key: HDFS-12225
> URL: https://issues.apache.org/jira/browse/HDFS-12225
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12225-HDFS-10285-01.patch,
> HDFS-12225-HDFS-10285-02.patch, HDFS-12225-HDFS-10285-03.patch,
> HDFS-12225-HDFS-10285-04.patch, HDFS-12225-HDFS-10285-05.patch,
> HDFS-12225-HDFS-10285-06.patch, HDFS-12225-HDFS-10285-07.patch,
> HDFS-12225-HDFS-10285-08.patch, HDFS-12225-HDFS-10285-09.patch
>
>
> We have discussed to optimize number extended attributes and asked to report
> separate JIRA while implementing [HDFS-11150 |
> https://issues.apache.org/jira/browse/HDFS-11150?focusedCommentId=15766127&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15766127]
> This is the JIRA to track that work
> For the context, comment copied from HDFS-11150
> {quote}
> [~yuanbo] wrote : I've tried that before. There is an issue here if we only
> mark the directory. When recovering from FsImage, the InodeMap isn't built
> up, so we don't know the sub-inode of a given inode, in the end, We cannot
> add these inodes to movement queue in FSDirectory#addToInodeMap, any
> thoughts?{quote}
> {quote}
> [~umamaheswararao] wrote: I got what you are saying. Ok for simplicity we can
> add for all Inodes now. For this to handle 100%, we may need intermittent
> processing, like first we should add them to some intermittentList while
> loading fsImage, once fully loaded and when starting active services, we
> should process that list and do required stuff. But it would add some
> additional complexity may be. Let's do with all file inodes now and we can
> revisit later if it is really creating issues. How about you raise a JIRA for
> it and think to optimize separately?
> {quote}
> {quote}
> [~andrew.wang] wrote in HDFS-10285 merge time review comment : HDFS-10899
> also the cursor of the iterator in the EZ root xattr to track progress and
> handle restarts. I wonder if we can do something similar here to avoid having
> an xattr-per-file being moved.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12225) [SPS]: Optimize extended attributes for tracking SPS movements
[
https://issues.apache.org/jira/browse/HDFS-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12225:
--
Attachment: HDFS-12225-HDFS-10285-08.patch
> [SPS]: Optimize extended attributes for tracking SPS movements
> --
>
> Key: HDFS-12225
> URL: https://issues.apache.org/jira/browse/HDFS-12225
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12225-HDFS-10285-01.patch,
> HDFS-12225-HDFS-10285-02.patch, HDFS-12225-HDFS-10285-03.patch,
> HDFS-12225-HDFS-10285-04.patch, HDFS-12225-HDFS-10285-05.patch,
> HDFS-12225-HDFS-10285-06.patch, HDFS-12225-HDFS-10285-07.patch,
> HDFS-12225-HDFS-10285-08.patch
>
>
> We have discussed to optimize number extended attributes and asked to report
> separate JIRA while implementing [HDFS-11150 |
> https://issues.apache.org/jira/browse/HDFS-11150?focusedCommentId=15766127&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15766127]
> This is the JIRA to track that work
> For the context, comment copied from HDFS-11150
> {quote}
> [~yuanbo] wrote : I've tried that before. There is an issue here if we only
> mark the directory. When recovering from FsImage, the InodeMap isn't built
> up, so we don't know the sub-inode of a given inode, in the end, We cannot
> add these inodes to movement queue in FSDirectory#addToInodeMap, any
> thoughts?{quote}
> {quote}
> [~umamaheswararao] wrote: I got what you are saying. Ok for simplicity we can
> add for all Inodes now. For this to handle 100%, we may need intermittent
> processing, like first we should add them to some intermittentList while
> loading fsImage, once fully loaded and when starting active services, we
> should process that list and do required stuff. But it would add some
> additional complexity may be. Let's do with all file inodes now and we can
> revisit later if it is really creating issues. How about you raise a JIRA for
> it and think to optimize separately?
> {quote}
> {quote}
> [~andrew.wang] wrote in HDFS-10285 merge time review comment : HDFS-10899
> also the cursor of the iterator in the EZ root xattr to track progress and
> handle restarts. I wonder if we can do something similar here to avoid having
> an xattr-per-file being moved.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12225) [SPS]: Optimize extended attributes for tracking SPS movements
[
https://issues.apache.org/jira/browse/HDFS-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12225:
--
Attachment: HDFS-12225-HDFS-10285-07.patch
Thanks [~rakeshr] for review..
Attached updated patch..
> [SPS]: Optimize extended attributes for tracking SPS movements
> --
>
> Key: HDFS-12225
> URL: https://issues.apache.org/jira/browse/HDFS-12225
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12225-HDFS-10285-01.patch,
> HDFS-12225-HDFS-10285-02.patch, HDFS-12225-HDFS-10285-03.patch,
> HDFS-12225-HDFS-10285-04.patch, HDFS-12225-HDFS-10285-05.patch,
> HDFS-12225-HDFS-10285-06.patch, HDFS-12225-HDFS-10285-07.patch
>
>
> We have discussed to optimize number extended attributes and asked to report
> separate JIRA while implementing [HDFS-11150 |
> https://issues.apache.org/jira/browse/HDFS-11150?focusedCommentId=15766127&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15766127]
> This is the JIRA to track that work
> For the context, comment copied from HDFS-11150
> {quote}
> [~yuanbo] wrote : I've tried that before. There is an issue here if we only
> mark the directory. When recovering from FsImage, the InodeMap isn't built
> up, so we don't know the sub-inode of a given inode, in the end, We cannot
> add these inodes to movement queue in FSDirectory#addToInodeMap, any
> thoughts?{quote}
> {quote}
> [~umamaheswararao] wrote: I got what you are saying. Ok for simplicity we can
> add for all Inodes now. For this to handle 100%, we may need intermittent
> processing, like first we should add them to some intermittentList while
> loading fsImage, once fully loaded and when starting active services, we
> should process that list and do required stuff. But it would add some
> additional complexity may be. Let's do with all file inodes now and we can
> revisit later if it is really creating issues. How about you raise a JIRA for
> it and think to optimize separately?
> {quote}
> {quote}
> [~andrew.wang] wrote in HDFS-10285 merge time review comment : HDFS-10899
> also the cursor of the iterator in the EZ root xattr to track progress and
> handle restarts. I wonder if we can do something similar here to avoid having
> an xattr-per-file being moved.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12225) [SPS]: Optimize extended attributes for tracking SPS movements
[
https://issues.apache.org/jira/browse/HDFS-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12225:
--
Attachment: HDFS-12225-HDFS-10285-06.patch
v6:
Fixed checkstyle warnings and some review comment given by Rakesh R.
Please review.
> [SPS]: Optimize extended attributes for tracking SPS movements
> --
>
> Key: HDFS-12225
> URL: https://issues.apache.org/jira/browse/HDFS-12225
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12225-HDFS-10285-01.patch,
> HDFS-12225-HDFS-10285-02.patch, HDFS-12225-HDFS-10285-03.patch,
> HDFS-12225-HDFS-10285-04.patch, HDFS-12225-HDFS-10285-05.patch,
> HDFS-12225-HDFS-10285-06.patch
>
>
> We have discussed to optimize number extended attributes and asked to report
> separate JIRA while implementing [HDFS-11150 |
> https://issues.apache.org/jira/browse/HDFS-11150?focusedCommentId=15766127&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15766127]
> This is the JIRA to track that work
> For the context, comment copied from HDFS-11150
> {quote}
> [~yuanbo] wrote : I've tried that before. There is an issue here if we only
> mark the directory. When recovering from FsImage, the InodeMap isn't built
> up, so we don't know the sub-inode of a given inode, in the end, We cannot
> add these inodes to movement queue in FSDirectory#addToInodeMap, any
> thoughts?{quote}
> {quote}
> [~umamaheswararao] wrote: I got what you are saying. Ok for simplicity we can
> add for all Inodes now. For this to handle 100%, we may need intermittent
> processing, like first we should add them to some intermittentList while
> loading fsImage, once fully loaded and when starting active services, we
> should process that list and do required stuff. But it would add some
> additional complexity may be. Let's do with all file inodes now and we can
> revisit later if it is really creating issues. How about you raise a JIRA for
> it and think to optimize separately?
> {quote}
> {quote}
> [~andrew.wang] wrote in HDFS-10285 merge time review comment : HDFS-10899
> also the cursor of the iterator in the EZ root xattr to track progress and
> handle restarts. I wonder if we can do something similar here to avoid having
> an xattr-per-file being moved.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12225) [SPS]: Optimize extended attributes for tracking SPS movements
[
https://issues.apache.org/jira/browse/HDFS-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12225:
--
Attachment: HDFS-12225-HDFS-10285-05.patch
Fixed checkstyle and findbug..
> [SPS]: Optimize extended attributes for tracking SPS movements
> --
>
> Key: HDFS-12225
> URL: https://issues.apache.org/jira/browse/HDFS-12225
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12225-HDFS-10285-01.patch,
> HDFS-12225-HDFS-10285-02.patch, HDFS-12225-HDFS-10285-03.patch,
> HDFS-12225-HDFS-10285-04.patch, HDFS-12225-HDFS-10285-05.patch
>
>
> We have discussed to optimize number extended attributes and asked to report
> separate JIRA while implementing [HDFS-11150 |
> https://issues.apache.org/jira/browse/HDFS-11150?focusedCommentId=15766127&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15766127]
> This is the JIRA to track that work
> For the context, comment copied from HDFS-11150
> {quote}
> [~yuanbo] wrote : I've tried that before. There is an issue here if we only
> mark the directory. When recovering from FsImage, the InodeMap isn't built
> up, so we don't know the sub-inode of a given inode, in the end, We cannot
> add these inodes to movement queue in FSDirectory#addToInodeMap, any
> thoughts?{quote}
> {quote}
> [~umamaheswararao] wrote: I got what you are saying. Ok for simplicity we can
> add for all Inodes now. For this to handle 100%, we may need intermittent
> processing, like first we should add them to some intermittentList while
> loading fsImage, once fully loaded and when starting active services, we
> should process that list and do required stuff. But it would add some
> additional complexity may be. Let's do with all file inodes now and we can
> revisit later if it is really creating issues. How about you raise a JIRA for
> it and think to optimize separately?
> {quote}
> {quote}
> [~andrew.wang] wrote in HDFS-10285 merge time review comment : HDFS-10899
> also the cursor of the iterator in the EZ root xattr to track progress and
> handle restarts. I wonder if we can do something similar here to avoid having
> an xattr-per-file being moved.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12225) [SPS]: Optimize extended attributes for tracking SPS movements
[
https://issues.apache.org/jira/browse/HDFS-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12225:
--
Attachment: HDFS-12225-HDFS-10285-04.patch
Thanks [~rakeshr] for review..
Attached updated patch..
> [SPS]: Optimize extended attributes for tracking SPS movements
> --
>
> Key: HDFS-12225
> URL: https://issues.apache.org/jira/browse/HDFS-12225
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12225-HDFS-10285-01.patch,
> HDFS-12225-HDFS-10285-02.patch, HDFS-12225-HDFS-10285-03.patch,
> HDFS-12225-HDFS-10285-04.patch
>
>
> We have discussed to optimize number extended attributes and asked to report
> separate JIRA while implementing [HDFS-11150 |
> https://issues.apache.org/jira/browse/HDFS-11150?focusedCommentId=15766127&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15766127]
> This is the JIRA to track that work
> For the context, comment copied from HDFS-11150
> {quote}
> [~yuanbo] wrote : I've tried that before. There is an issue here if we only
> mark the directory. When recovering from FsImage, the InodeMap isn't built
> up, so we don't know the sub-inode of a given inode, in the end, We cannot
> add these inodes to movement queue in FSDirectory#addToInodeMap, any
> thoughts?{quote}
> {quote}
> [~umamaheswararao] wrote: I got what you are saying. Ok for simplicity we can
> add for all Inodes now. For this to handle 100%, we may need intermittent
> processing, like first we should add them to some intermittentList while
> loading fsImage, once fully loaded and when starting active services, we
> should process that list and do required stuff. But it would add some
> additional complexity may be. Let's do with all file inodes now and we can
> revisit later if it is really creating issues. How about you raise a JIRA for
> it and think to optimize separately?
> {quote}
> {quote}
> [~andrew.wang] wrote in HDFS-10285 merge time review comment : HDFS-10899
> also the cursor of the iterator in the EZ root xattr to track progress and
> handle restarts. I wonder if we can do something similar here to avoid having
> an xattr-per-file being moved.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12225) [SPS]: Optimize extended attributes for tracking SPS movements
[
https://issues.apache.org/jira/browse/HDFS-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12225:
--
Attachment: HDFS-12225-HDFS-10285-03.patch
Thanks [~rakeshr] for review.
Attached updated patch. Please review..
> [SPS]: Optimize extended attributes for tracking SPS movements
> --
>
> Key: HDFS-12225
> URL: https://issues.apache.org/jira/browse/HDFS-12225
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12225-HDFS-10285-01.patch,
> HDFS-12225-HDFS-10285-02.patch, HDFS-12225-HDFS-10285-03.patch
>
>
> We have discussed to optimize number extended attributes and asked to report
> separate JIRA while implementing [HDFS-11150 |
> https://issues.apache.org/jira/browse/HDFS-11150?focusedCommentId=15766127&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15766127]
> This is the JIRA to track that work
> For the context, comment copied from HDFS-11150
> {quote}
> [~yuanbo] wrote : I've tried that before. There is an issue here if we only
> mark the directory. When recovering from FsImage, the InodeMap isn't built
> up, so we don't know the sub-inode of a given inode, in the end, We cannot
> add these inodes to movement queue in FSDirectory#addToInodeMap, any
> thoughts?{quote}
> {quote}
> [~umamaheswararao] wrote: I got what you are saying. Ok for simplicity we can
> add for all Inodes now. For this to handle 100%, we may need intermittent
> processing, like first we should add them to some intermittentList while
> loading fsImage, once fully loaded and when starting active services, we
> should process that list and do required stuff. But it would add some
> additional complexity may be. Let's do with all file inodes now and we can
> revisit later if it is really creating issues. How about you raise a JIRA for
> it and think to optimize separately?
> {quote}
> {quote}
> [~andrew.wang] wrote in HDFS-10285 merge time review comment : HDFS-10899
> also the cursor of the iterator in the EZ root xattr to track progress and
> handle restarts. I wonder if we can do something similar here to avoid having
> an xattr-per-file being moved.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12225) [SPS]: Optimize extended attributes for tracking SPS movements
[
https://issues.apache.org/jira/browse/HDFS-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12225:
--
Attachment: HDFS-12225-HDFS-10285-02.patch
Attached v2 patch.
Fixed checkstyle and test failure. Findbugs are unrelated to this patch
> [SPS]: Optimize extended attributes for tracking SPS movements
> --
>
> Key: HDFS-12225
> URL: https://issues.apache.org/jira/browse/HDFS-12225
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12225-HDFS-10285-01.patch,
> HDFS-12225-HDFS-10285-02.patch
>
>
> We have discussed to optimize number extended attributes and asked to report
> separate JIRA while implementing [HDFS-11150 |
> https://issues.apache.org/jira/browse/HDFS-11150?focusedCommentId=15766127&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15766127]
> This is the JIRA to track that work
> For the context, comment copied from HDFS-11150
> {quote}
> [~yuanbo] wrote : I've tried that before. There is an issue here if we only
> mark the directory. When recovering from FsImage, the InodeMap isn't built
> up, so we don't know the sub-inode of a given inode, in the end, We cannot
> add these inodes to movement queue in FSDirectory#addToInodeMap, any
> thoughts?{quote}
> {quote}
> [~umamaheswararao] wrote: I got what you are saying. Ok for simplicity we can
> add for all Inodes now. For this to handle 100%, we may need intermittent
> processing, like first we should add them to some intermittentList while
> loading fsImage, once fully loaded and when starting active services, we
> should process that list and do required stuff. But it would add some
> additional complexity may be. Let's do with all file inodes now and we can
> revisit later if it is really creating issues. How about you raise a JIRA for
> it and think to optimize separately?
> {quote}
> {quote}
> [~andrew.wang] wrote in HDFS-10285 merge time review comment : HDFS-10899
> also the cursor of the iterator in the EZ root xattr to track progress and
> handle restarts. I wonder if we can do something similar here to avoid having
> an xattr-per-file being moved.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12225) [SPS]: Optimize extended attributes for tracking SPS movements
[
https://issues.apache.org/jira/browse/HDFS-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12225:
--
Status: Patch Available (was: Open)
> [SPS]: Optimize extended attributes for tracking SPS movements
> --
>
> Key: HDFS-12225
> URL: https://issues.apache.org/jira/browse/HDFS-12225
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12225-HDFS-10285-01.patch
>
>
> We have discussed to optimize number extended attributes and asked to report
> separate JIRA while implementing [HDFS-11150 |
> https://issues.apache.org/jira/browse/HDFS-11150?focusedCommentId=15766127&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15766127]
> This is the JIRA to track that work
> For the context, comment copied from HDFS-11150
> {quote}
> [~yuanbo] wrote : I've tried that before. There is an issue here if we only
> mark the directory. When recovering from FsImage, the InodeMap isn't built
> up, so we don't know the sub-inode of a given inode, in the end, We cannot
> add these inodes to movement queue in FSDirectory#addToInodeMap, any
> thoughts?{quote}
> {quote}
> [~umamaheswararao] wrote: I got what you are saying. Ok for simplicity we can
> add for all Inodes now. For this to handle 100%, we may need intermittent
> processing, like first we should add them to some intermittentList while
> loading fsImage, once fully loaded and when starting active services, we
> should process that list and do required stuff. But it would add some
> additional complexity may be. Let's do with all file inodes now and we can
> revisit later if it is really creating issues. How about you raise a JIRA for
> it and think to optimize separately?
> {quote}
> {quote}
> [~andrew.wang] wrote in HDFS-10285 merge time review comment : HDFS-10899
> also the cursor of the iterator in the EZ root xattr to track progress and
> handle restarts. I wonder if we can do something similar here to avoid having
> an xattr-per-file being moved.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12225) [SPS]: Optimize extended attributes for tracking SPS movements
[
https://issues.apache.org/jira/browse/HDFS-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12225:
--
Attachment: HDFS-12225-HDFS-10285-01.patch
Thanks [~xiaochen] for suggestion.
Sorry for delay, I was on leave for last one week.
Attached initial patch. Please review..
> [SPS]: Optimize extended attributes for tracking SPS movements
> --
>
> Key: HDFS-12225
> URL: https://issues.apache.org/jira/browse/HDFS-12225
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12225-HDFS-10285-01.patch
>
>
> We have discussed to optimize number extended attributes and asked to report
> separate JIRA while implementing [HDFS-11150 |
> https://issues.apache.org/jira/browse/HDFS-11150?focusedCommentId=15766127&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15766127]
> This is the JIRA to track that work
> For the context, comment copied from HDFS-11150
> {quote}
> [~yuanbo] wrote : I've tried that before. There is an issue here if we only
> mark the directory. When recovering from FsImage, the InodeMap isn't built
> up, so we don't know the sub-inode of a given inode, in the end, We cannot
> add these inodes to movement queue in FSDirectory#addToInodeMap, any
> thoughts?{quote}
> {quote}
> [~umamaheswararao] wrote: I got what you are saying. Ok for simplicity we can
> add for all Inodes now. For this to handle 100%, we may need intermittent
> processing, like first we should add them to some intermittentList while
> loading fsImage, once fully loaded and when starting active services, we
> should process that list and do required stuff. But it would add some
> additional complexity may be. Let's do with all file inodes now and we can
> revisit later if it is really creating issues. How about you raise a JIRA for
> it and think to optimize separately?
> {quote}
> {quote}
> [~andrew.wang] wrote in HDFS-10285 merge time review comment : HDFS-10899
> also the cursor of the iterator in the EZ root xattr to track progress and
> handle restarts. I wonder if we can do something similar here to avoid having
> an xattr-per-file being moved.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12225) [SPS]: Optimize extended attributes for tracking SPS movements
[
https://issues.apache.org/jira/browse/HDFS-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-12225:
---
Description:
We have discussed to optimize number extended attributes and asked to report
separate JIRA while implementing [HDFS-11150 |
https://issues.apache.org/jira/browse/HDFS-11150?focusedCommentId=15766127&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15766127]
This is the JIRA to track that work
For the context, comment copied from HDFS-11150
{quote}
[~yuanbo] wrote : I've tried that before. There is an issue here if we only
mark the directory. When recovering from FsImage, the InodeMap isn't built up,
so we don't know the sub-inode of a given inode, in the end, We cannot add
these inodes to movement queue in FSDirectory#addToInodeMap, any
thoughts?{quote}
{quote}
[~umamaheswararao] wrote: I got what you are saying. Ok for simplicity we can
add for all Inodes now. For this to handle 100%, we may need intermittent
processing, like first we should add them to some intermittentList while
loading fsImage, once fully loaded and when starting active services, we should
process that list and do required stuff. But it would add some additional
complexity may be. Let's do with all file inodes now and we can revisit later
if it is really creating issues. How about you raise a JIRA for it and think to
optimize separately?
{quote}
{quote}
[~andrew.wang] wrote in HDFS-10285 merge time review comment : HDFS-10899 also
the cursor of the iterator in the EZ root xattr to track progress and handle
restarts. I wonder if we can do something similar here to avoid having an
xattr-per-file being moved.
{quote}
was:
We have discussed to optimize number extended attributes and asked to report
separate JIRA while implementing [HDFS-11150 |
https://issues.apache.org/jira/browse/HDFS-11150?focusedCommentId=15766127&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15766127]
This is the JIRA to track that work
For the context, comment copied from HDFS-11150
{quote}
[~yuanbo] wrote : I've tried that before. There is an issue here if we only
mark the directory. When recovering from FsImage, the InodeMap isn't built up,
so we don't know the sub-inode of a given inode, in the end, We cannot add
these inodes to movement queue in FSDirectory#addToInodeMap, any
thoughts?{quote}
[~umamaheswararao] wrote: I got what you are saying. Ok for simplicity we can
add for all Inodes now. For this to handle 100%, we may need intermittent
processing, like first we should add them to some intermittentList while
loading fsImage, once fully loaded and when starting active services, we should
process that list and do required stuff. But it would add some additional
complexity may be. Let's do with all file inodes now and we can revisit later
if it is really creating issues. How about you raise a JIRA for it and think to
optimize separately?
{quote}
> [SPS]: Optimize extended attributes for tracking SPS movements
> --
>
> Key: HDFS-12225
> URL: https://issues.apache.org/jira/browse/HDFS-12225
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Surendra Singh Lilhore
>
> We have discussed to optimize number extended attributes and asked to report
> separate JIRA while implementing [HDFS-11150 |
> https://issues.apache.org/jira/browse/HDFS-11150?focusedCommentId=15766127&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15766127]
> This is the JIRA to track that work
> For the context, comment copied from HDFS-11150
> {quote}
> [~yuanbo] wrote : I've tried that before. There is an issue here if we only
> mark the directory. When recovering from FsImage, the InodeMap isn't built
> up, so we don't know the sub-inode of a given inode, in the end, We cannot
> add these inodes to movement queue in FSDirectory#addToInodeMap, any
> thoughts?{quote}
> {quote}
> [~umamaheswararao] wrote: I got what you are saying. Ok for simplicity we can
> add for all Inodes now. For this to handle 100%, we may need intermittent
> processing, like first we should add them to some intermittentList while
> loading fsImage, once fully loaded and when starting active services, we
> should process that list and do required stuff. But it would add some
> additional complexity may be. Let's do with all file inodes now and we can
> revisit later if it is really creating issues. How about you raise a JIRA for
> it and think to optimize separately?
> {quote}
> {quote}
> [~andrew.wang] wrote in HDFS-10285 merge time review comment : HDFS-10899
> also the cursor of the iterator in the EZ root xattr to track progress and
> han
[jira] [Updated] (HDFS-12225) [SPS]: Optimize extended attributes for tracking SPS movements
[
https://issues.apache.org/jira/browse/HDFS-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-12225:
---
Summary: [SPS]: Optimize extended attributes for tracking SPS movements
(was: Optimize extended attributes for tracking SPS movements)
> [SPS]: Optimize extended attributes for tracking SPS movements
> --
>
> Key: HDFS-12225
> URL: https://issues.apache.org/jira/browse/HDFS-12225
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>
> We have discussed to optimize number extended attributes and asked to report
> separate JIRA while implementing [HDFS-11150 |
> https://issues.apache.org/jira/browse/HDFS-11150?focusedCommentId=15766127&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15766127]
> This is the JIRA to track that work
> For the context, comment copied from HDFS-11150
> {quote}
> [~yuanbo] wrote : I've tried that before. There is an issue here if we only
> mark the directory. When recovering from FsImage, the InodeMap isn't built
> up, so we don't know the sub-inode of a given inode, in the end, We cannot
> add these inodes to movement queue in FSDirectory#addToInodeMap, any
> thoughts?{quote}
> [~umamaheswararao] wrote: I got what you are saying. Ok for simplicity we can
> add for all Inodes now. For this to handle 100%, we may need intermittent
> processing, like first we should add them to some intermittentList while
> loading fsImage, once fully loaded and when starting active services, we
> should process that list and do required stuff. But it would add some
> additional complexity may be. Let's do with all file inodes now and we can
> revisit later if it is really creating issues. How about you raise a JIRA for
> it and think to optimize separately?
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
