[jira] [Updated] (HDFS-4420) Provide a way to exclude subtree from balancing process
[ https://issues.apache.org/jira/browse/HDFS-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HDFS-4420: -- Resolution: Duplicate Target Version/s: (was: 2.0.3-alpha, 1.3.0) Status: Resolved (was: Patch Available) Although not implemented with the same idea, HDFS-6133 resolves this need via DN block pinning. Reasoning versus using simple path to block lookup exclusion is efficiency, as detailed further in Daryn's comments at https://issues.apache.org/jira/browse/HDFS-6133?focusedCommentId=13980504page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13980504. Resolving as Duplicate. Thanks for the patch effort regardless Max! Provide a way to exclude subtree from balancing process --- Key: HDFS-4420 URL: https://issues.apache.org/jira/browse/HDFS-4420 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Reporter: Max Lapan Priority: Minor Attachments: Balancer-exclude-subtree-0.90.2.patch, Balancer-exclude-trunk-v2.patch, Balancer-exclude-trunk-v3.patch, Balancer-exclude-trunk.patch, HDFS-4420-v4.patch During balancer operation, it balances all blocks, regardless of their filesystem hierarchy. Sometimes, it would be usefull to exclude some subtree from balancing process. For example, regionservers data locality is cruical for HBase performance. Region's data is tied to regionservers, which reside on specific machines in cluster. During operation, regionservers reads and writes region's data, and after some time, all this data are reside on local machine, so, all reads become local, which is great for performance. Balancer breaks this locality during opertation by moving blocks around. This patch adds [-exclude path] switch, and, if path is provided, balancer will not move blocks under this path during operation. Attached patch have tested for 0.90.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-4420) Provide a way to exclude subtree from balancing process
[ https://issues.apache.org/jira/browse/HDFS-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Lapan updated HDFS-4420: Attachment: HDFS-4420-v4.patch New optimized version of patch. Now, full list of blocks to exclude built only on first iteration. On all further iterations, we only add newly added blocks (by timestamp). This reduces exclude list construction by a factor of 2-10, depending on exclude subtree size. On our cluster with 800TB /hbase, we now have this process take 10 seconds instead of 15 minutes. Provide a way to exclude subtree from balancing process --- Key: HDFS-4420 URL: https://issues.apache.org/jira/browse/HDFS-4420 Project: Hadoop HDFS Issue Type: Improvement Components: balancer Reporter: Max Lapan Priority: Minor Attachments: Balancer-exclude-subtree-0.90.2.patch, Balancer-exclude-trunk.patch, Balancer-exclude-trunk-v2.patch, Balancer-exclude-trunk-v3.patch, HDFS-4420-v4.patch During balancer operation, it balances all blocks, regardless of their filesystem hierarchy. Sometimes, it would be usefull to exclude some subtree from balancing process. For example, regionservers data locality is cruical for HBase performance. Region's data is tied to regionservers, which reside on specific machines in cluster. During operation, regionservers reads and writes region's data, and after some time, all this data are reside on local machine, so, all reads become local, which is great for performance. Balancer breaks this locality during opertation by moving blocks around. This patch adds [-exclude path] switch, and, if path is provided, balancer will not move blocks under this path during operation. Attached patch have tested for 0.90.2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4420) Provide a way to exclude subtree from balancing process
[ https://issues.apache.org/jira/browse/HDFS-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HDFS-4420: - Target Version/s: 2.0.3-alpha, 1.3.0 (was: 1.2.0, 2.0.3-alpha) Provide a way to exclude subtree from balancing process --- Key: HDFS-4420 URL: https://issues.apache.org/jira/browse/HDFS-4420 Project: Hadoop HDFS Issue Type: Improvement Components: balancer Reporter: Max Lapan Priority: Minor Attachments: Balancer-exclude-subtree-0.90.2.patch, Balancer-exclude-trunk.patch, Balancer-exclude-trunk-v2.patch, Balancer-exclude-trunk-v3.patch During balancer operation, it balances all blocks, regardless of their filesystem hierarchy. Sometimes, it would be usefull to exclude some subtree from balancing process. For example, regionservers data locality is cruical for HBase performance. Region's data is tied to regionservers, which reside on specific machines in cluster. During operation, regionservers reads and writes region's data, and after some time, all this data are reside on local machine, so, all reads become local, which is great for performance. Balancer breaks this locality during opertation by moving blocks around. This patch adds [-exclude path] switch, and, if path is provided, balancer will not move blocks under this path during operation. Attached patch have tested for 0.90.2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4420) Provide a way to exclude subtree from balancing process
[ https://issues.apache.org/jira/browse/HDFS-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Lapan updated HDFS-4420: Attachment: Balancer-exclude-trunk-v2.patch Now with test of excludePath feature Provide a way to exclude subtree from balancing process --- Key: HDFS-4420 URL: https://issues.apache.org/jira/browse/HDFS-4420 Project: Hadoop HDFS Issue Type: Improvement Components: balancer Reporter: Max Lapan Priority: Minor Attachments: Balancer-exclude-subtree-0.90.2.patch, Balancer-exclude-trunk.patch, Balancer-exclude-trunk-v2.patch During balancer operation, it balances all blocks, regardless of their filesystem hierarchy. Sometimes, it would be usefull to exclude some subtree from balancing process. For example, regionservers data locality is cruical for HBase performance. Region's data is tied to regionservers, which reside on specific machines in cluster. During operation, regionservers reads and writes region's data, and after some time, all this data are reside on local machine, so, all reads become local, which is great for performance. Balancer breaks this locality during opertation by moving blocks around. This patch adds [-exclude path] switch, and, if path is provided, balancer will not move blocks under this path during operation. Attached patch have tested for 0.90.2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4420) Provide a way to exclude subtree from balancing process
[ https://issues.apache.org/jira/browse/HDFS-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Lapan updated HDFS-4420: Status: Patch Available (was: Open) Provide a way to exclude subtree from balancing process --- Key: HDFS-4420 URL: https://issues.apache.org/jira/browse/HDFS-4420 Project: Hadoop HDFS Issue Type: Improvement Components: balancer Reporter: Max Lapan Priority: Minor Attachments: Balancer-exclude-subtree-0.90.2.patch, Balancer-exclude-trunk.patch, Balancer-exclude-trunk-v2.patch During balancer operation, it balances all blocks, regardless of their filesystem hierarchy. Sometimes, it would be usefull to exclude some subtree from balancing process. For example, regionservers data locality is cruical for HBase performance. Region's data is tied to regionservers, which reside on specific machines in cluster. During operation, regionservers reads and writes region's data, and after some time, all this data are reside on local machine, so, all reads become local, which is great for performance. Balancer breaks this locality during opertation by moving blocks around. This patch adds [-exclude path] switch, and, if path is provided, balancer will not move blocks under this path during operation. Attached patch have tested for 0.90.2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4420) Provide a way to exclude subtree from balancing process
[ https://issues.apache.org/jira/browse/HDFS-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Lapan updated HDFS-4420: Status: Open (was: Patch Available) Provide a way to exclude subtree from balancing process --- Key: HDFS-4420 URL: https://issues.apache.org/jira/browse/HDFS-4420 Project: Hadoop HDFS Issue Type: Improvement Components: balancer Reporter: Max Lapan Priority: Minor Attachments: Balancer-exclude-subtree-0.90.2.patch, Balancer-exclude-trunk.patch, Balancer-exclude-trunk-v2.patch During balancer operation, it balances all blocks, regardless of their filesystem hierarchy. Sometimes, it would be usefull to exclude some subtree from balancing process. For example, regionservers data locality is cruical for HBase performance. Region's data is tied to regionservers, which reside on specific machines in cluster. During operation, regionservers reads and writes region's data, and after some time, all this data are reside on local machine, so, all reads become local, which is great for performance. Balancer breaks this locality during opertation by moving blocks around. This patch adds [-exclude path] switch, and, if path is provided, balancer will not move blocks under this path during operation. Attached patch have tested for 0.90.2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4420) Provide a way to exclude subtree from balancing process
[ https://issues.apache.org/jira/browse/HDFS-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Lapan updated HDFS-4420: Status: Open (was: Patch Available) Provide a way to exclude subtree from balancing process --- Key: HDFS-4420 URL: https://issues.apache.org/jira/browse/HDFS-4420 Project: Hadoop HDFS Issue Type: Improvement Components: balancer Reporter: Max Lapan Priority: Minor Attachments: Balancer-exclude-subtree-0.90.2.patch, Balancer-exclude-trunk.patch, Balancer-exclude-trunk-v2.patch, Balancer-exclude-trunk-v3.patch During balancer operation, it balances all blocks, regardless of their filesystem hierarchy. Sometimes, it would be usefull to exclude some subtree from balancing process. For example, regionservers data locality is cruical for HBase performance. Region's data is tied to regionservers, which reside on specific machines in cluster. During operation, regionservers reads and writes region's data, and after some time, all this data are reside on local machine, so, all reads become local, which is great for performance. Balancer breaks this locality during opertation by moving blocks around. This patch adds [-exclude path] switch, and, if path is provided, balancer will not move blocks under this path during operation. Attached patch have tested for 0.90.2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4420) Provide a way to exclude subtree from balancing process
[ https://issues.apache.org/jira/browse/HDFS-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Lapan updated HDFS-4420: Attachment: Balancer-exclude-trunk-v3.patch Firebug and release errors fixed. Provide a way to exclude subtree from balancing process --- Key: HDFS-4420 URL: https://issues.apache.org/jira/browse/HDFS-4420 Project: Hadoop HDFS Issue Type: Improvement Components: balancer Reporter: Max Lapan Priority: Minor Attachments: Balancer-exclude-subtree-0.90.2.patch, Balancer-exclude-trunk.patch, Balancer-exclude-trunk-v2.patch, Balancer-exclude-trunk-v3.patch During balancer operation, it balances all blocks, regardless of their filesystem hierarchy. Sometimes, it would be usefull to exclude some subtree from balancing process. For example, regionservers data locality is cruical for HBase performance. Region's data is tied to regionservers, which reside on specific machines in cluster. During operation, regionservers reads and writes region's data, and after some time, all this data are reside on local machine, so, all reads become local, which is great for performance. Balancer breaks this locality during opertation by moving blocks around. This patch adds [-exclude path] switch, and, if path is provided, balancer will not move blocks under this path during operation. Attached patch have tested for 0.90.2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4420) Provide a way to exclude subtree from balancing process
[ https://issues.apache.org/jira/browse/HDFS-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Lapan updated HDFS-4420: Status: Patch Available (was: Open) Provide a way to exclude subtree from balancing process --- Key: HDFS-4420 URL: https://issues.apache.org/jira/browse/HDFS-4420 Project: Hadoop HDFS Issue Type: Improvement Components: balancer Reporter: Max Lapan Priority: Minor Attachments: Balancer-exclude-subtree-0.90.2.patch, Balancer-exclude-trunk.patch, Balancer-exclude-trunk-v2.patch, Balancer-exclude-trunk-v3.patch During balancer operation, it balances all blocks, regardless of their filesystem hierarchy. Sometimes, it would be usefull to exclude some subtree from balancing process. For example, regionservers data locality is cruical for HBase performance. Region's data is tied to regionservers, which reside on specific machines in cluster. During operation, regionservers reads and writes region's data, and after some time, all this data are reside on local machine, so, all reads become local, which is great for performance. Balancer breaks this locality during opertation by moving blocks around. This patch adds [-exclude path] switch, and, if path is provided, balancer will not move blocks under this path during operation. Attached patch have tested for 0.90.2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4420) Provide a way to exclude subtree from balancing process
[ https://issues.apache.org/jira/browse/HDFS-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Lapan updated HDFS-4420: Attachment: Balancer-exclude-trunk.patch Trunk version Provide a way to exclude subtree from balancing process --- Key: HDFS-4420 URL: https://issues.apache.org/jira/browse/HDFS-4420 Project: Hadoop HDFS Issue Type: Improvement Components: balancer Affects Versions: 0.20.2 Reporter: Max Lapan Priority: Minor Attachments: Balancer-exclude-subtree-0.90.2.patch, Balancer-exclude-trunk.patch During balancer operation, it balances all blocks, regardless of their filesystem hierarchy. Sometimes, it would be usefull to exclude some subtree from balancing process. For example, regionservers data locality is cruical for HBase performance. Region's data is tied to regionservers, which reside on specific machines in cluster. During operation, regionservers reads and writes region's data, and after some time, all this data are reside on local machine, so, all reads become local, which is great for performance. Balancer breaks this locality during opertation by moving blocks around. This patch adds [-exclude path] switch, and, if path is provided, balancer will not move blocks under this path during operation. Attached patch have tested for 0.90.2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4420) Provide a way to exclude subtree from balancing process
[ https://issues.apache.org/jira/browse/HDFS-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Lapan updated HDFS-4420: Affects Version/s: (was: 0.20.2) Status: Patch Available (was: Open) Provide a way to exclude subtree from balancing process --- Key: HDFS-4420 URL: https://issues.apache.org/jira/browse/HDFS-4420 Project: Hadoop HDFS Issue Type: Improvement Components: balancer Reporter: Max Lapan Priority: Minor Attachments: Balancer-exclude-subtree-0.90.2.patch, Balancer-exclude-trunk.patch During balancer operation, it balances all blocks, regardless of their filesystem hierarchy. Sometimes, it would be usefull to exclude some subtree from balancing process. For example, regionservers data locality is cruical for HBase performance. Region's data is tied to regionservers, which reside on specific machines in cluster. During operation, regionservers reads and writes region's data, and after some time, all this data are reside on local machine, so, all reads become local, which is great for performance. Balancer breaks this locality during opertation by moving blocks around. This patch adds [-exclude path] switch, and, if path is provided, balancer will not move blocks under this path during operation. Attached patch have tested for 0.90.2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4420) Provide a way to exclude subtree from balancing process
[ https://issues.apache.org/jira/browse/HDFS-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Lapan updated HDFS-4420: Attachment: Balancer-exclude-subtree-0.90.2.patch Provide a way to exclude subtree from balancing process --- Key: HDFS-4420 URL: https://issues.apache.org/jira/browse/HDFS-4420 Project: Hadoop HDFS Issue Type: Improvement Components: balancer Affects Versions: 0.20.2 Reporter: Max Lapan Priority: Minor Fix For: 0.20.2 Attachments: Balancer-exclude-subtree-0.90.2.patch During balancer operation, it balances all blocks, regardless of their filesystem hierarchy. Sometimes, it would be usefull to exclude some subtree from balancing process. For example, regionservers data locality is cruical for HBase performance. Region's data is tied to regionservers, which reside on specific machines in cluster. During operation, regionservers reads and writes region's data, and after some time, all this data are reside on local machine, so, all reads become local, which is great for performance. Balancer breaks this locality during opertation by moving blocks around. This patch adds [-exclude path] switch, and, if path is provided, balancer will not move blocks under this path during operation. Attached patch have tested for 0.90.2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4420) Provide a way to exclude subtree from balancing process
[ https://issues.apache.org/jira/browse/HDFS-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-4420: - Target Version/s: 1.2.0, 2.0.3-alpha Fix Version/s: (was: 0.20.2) Provide a way to exclude subtree from balancing process --- Key: HDFS-4420 URL: https://issues.apache.org/jira/browse/HDFS-4420 Project: Hadoop HDFS Issue Type: Improvement Components: balancer Affects Versions: 0.20.2 Reporter: Max Lapan Priority: Minor Attachments: Balancer-exclude-subtree-0.90.2.patch During balancer operation, it balances all blocks, regardless of their filesystem hierarchy. Sometimes, it would be usefull to exclude some subtree from balancing process. For example, regionservers data locality is cruical for HBase performance. Region's data is tied to regionservers, which reside on specific machines in cluster. During operation, regionservers reads and writes region's data, and after some time, all this data are reside on local machine, so, all reads become local, which is great for performance. Balancer breaks this locality during opertation by moving blocks around. This patch adds [-exclude path] switch, and, if path is provided, balancer will not move blocks under this path during operation. Attached patch have tested for 0.90.2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
