[jira] [Updated] (HDFS-8041) Consider remaining space during block blockplacement if dfs space is highly utilized
[
https://issues.apache.org/jira/browse/HDFS-8041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-8041:
-
Status: Open (was: Patch Available)
> Consider remaining space during block blockplacement if dfs space is highly
> utilized
>
>
> Key: HDFS-8041
> URL: https://issues.apache.org/jira/browse/HDFS-8041
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Labels: BlockPlacementPolicy
> Attachments: HDFS-8041.v1.patch, HDFS-8041.v2.patch,
> HDFS-8041.v3.patch, HDFS-8041.v4.patch
>
>
> This feature is helpful in avoiding smaller nodes (i.e. heterogeneous
> environment) getting constantly being full when the overall space utilization
> is over a certain threshold. When the utilization is low, balancer can keep
> up, but once the average per-node byte goes over the capacity of the smaller
> nodes, they get full so quickly even after perfect balance.
> This jira proposes an improvement that can be optionally enabled in order to
> slow down the rate of space usage growth of smaller nodes if the overall
> storage utilization is over a configured threshold. It will not replace
> balancer, rather will help balancer keep up. Also, the primary replica
> placement will not be affected. Only the replicas typically placed in a
> remote rack will be subject to this check.
> The appropriate threshold is cluster configuration specific. There is no
> generally good value to set, thus it is disabled by default. We have seen
> cases where the threshold of 85% - 90% would help. Figuring when
> {{totalSpaceUsed / numNodes}} becomes close to the capacity of a smaller node
> is helpful in determining the threshold.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8041) Consider remaining space during block blockplacement if dfs space is highly utilized
[
https://issues.apache.org/jira/browse/HDFS-8041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Priyank Rastogi updated HDFS-8041:
--
Labels: (was: BB2015-05-TBR)
Consider remaining space during block blockplacement if dfs space is highly
utilized
Key: HDFS-8041
URL: https://issues.apache.org/jira/browse/HDFS-8041
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Attachments: HDFS-8041.v1.patch, HDFS-8041.v2.patch,
HDFS-8041.v3.patch, HDFS-8041.v4.patch
This feature is helpful in avoiding smaller nodes (i.e. heterogeneous
environment) getting constantly being full when the overall space utilization
is over a certain threshold. When the utilization is low, balancer can keep
up, but once the average per-node byte goes over the capacity of the smaller
nodes, they get full so quickly even after perfect balance.
This jira proposes an improvement that can be optionally enabled in order to
slow down the rate of space usage growth of smaller nodes if the overall
storage utilization is over a configured threshold. It will not replace
balancer, rather will help balancer keep up. Also, the primary replica
placement will not be affected. Only the replicas typically placed in a
remote rack will be subject to this check.
The appropriate threshold is cluster configuration specific. There is no
generally good value to set, thus it is disabled by default. We have seen
cases where the threshold of 85% - 90% would help. Figuring when
{{totalSpaceUsed / numNodes}} becomes close to the capacity of a smaller node
is helpful in determining the threshold.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8041) Consider remaining space during block blockplacement if dfs space is highly utilized
[
https://issues.apache.org/jira/browse/HDFS-8041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Priyank Rastogi updated HDFS-8041:
--
Labels: BlockPlacementPolicy (was: )
Consider remaining space during block blockplacement if dfs space is highly
utilized
Key: HDFS-8041
URL: https://issues.apache.org/jira/browse/HDFS-8041
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Labels: BlockPlacementPolicy
Attachments: HDFS-8041.v1.patch, HDFS-8041.v2.patch,
HDFS-8041.v3.patch, HDFS-8041.v4.patch
This feature is helpful in avoiding smaller nodes (i.e. heterogeneous
environment) getting constantly being full when the overall space utilization
is over a certain threshold. When the utilization is low, balancer can keep
up, but once the average per-node byte goes over the capacity of the smaller
nodes, they get full so quickly even after perfect balance.
This jira proposes an improvement that can be optionally enabled in order to
slow down the rate of space usage growth of smaller nodes if the overall
storage utilization is over a configured threshold. It will not replace
balancer, rather will help balancer keep up. Also, the primary replica
placement will not be affected. Only the replicas typically placed in a
remote rack will be subject to this check.
The appropriate threshold is cluster configuration specific. There is no
generally good value to set, thus it is disabled by default. We have seen
cases where the threshold of 85% - 90% would help. Figuring when
{{totalSpaceUsed / numNodes}} becomes close to the capacity of a smaller node
is helpful in determining the threshold.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8041) Consider remaining space during block blockplacement if dfs space is highly utilized
[
https://issues.apache.org/jira/browse/HDFS-8041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HDFS-8041:
---
Labels: BB2015-05-TBR (was: )
Consider remaining space during block blockplacement if dfs space is highly
utilized
Key: HDFS-8041
URL: https://issues.apache.org/jira/browse/HDFS-8041
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Labels: BB2015-05-TBR
Attachments: HDFS-8041.v1.patch, HDFS-8041.v2.patch,
HDFS-8041.v3.patch, HDFS-8041.v4.patch
This feature is helpful in avoiding smaller nodes (i.e. heterogeneous
environment) getting constantly being full when the overall space utilization
is over a certain threshold. When the utilization is low, balancer can keep
up, but once the average per-node byte goes over the capacity of the smaller
nodes, they get full so quickly even after perfect balance.
This jira proposes an improvement that can be optionally enabled in order to
slow down the rate of space usage growth of smaller nodes if the overall
storage utilization is over a configured threshold. It will not replace
balancer, rather will help balancer keep up. Also, the primary replica
placement will not be affected. Only the replicas typically placed in a
remote rack will be subject to this check.
The appropriate threshold is cluster configuration specific. There is no
generally good value to set, thus it is disabled by default. We have seen
cases where the threshold of 85% - 90% would help. Figuring when
{{totalSpaceUsed / numNodes}} becomes close to the capacity of a smaller node
is helpful in determining the threshold.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8041) Consider remaining space during block blockplacement if dfs space is highly utilized
[
https://issues.apache.org/jira/browse/HDFS-8041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-8041:
-
Attachment: HDFS-8041.v4.patch
Existing patch does not apply after HDFS-8073. Attaching a new patch.
Consider remaining space during block blockplacement if dfs space is highly
utilized
Key: HDFS-8041
URL: https://issues.apache.org/jira/browse/HDFS-8041
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Attachments: HDFS-8041.v1.patch, HDFS-8041.v2.patch,
HDFS-8041.v3.patch, HDFS-8041.v4.patch
This feature is helpful in avoiding smaller nodes (i.e. heterogeneous
environment) getting constantly being full when the overall space utilization
is over a certain threshold. When the utilization is low, balancer can keep
up, but once the average per-node byte goes over the capacity of the smaller
nodes, they get full so quickly even after perfect balance.
This jira proposes an improvement that can be optionally enabled in order to
slow down the rate of space usage growth of smaller nodes if the overall
storage utilization is over a configured threshold. It will not replace
balancer, rather will help balancer keep up. Also, the primary replica
placement will not be affected. Only the replicas typically placed in a
remote rack will be subject to this check.
The appropriate threshold is cluster configuration specific. There is no
generally good value to set, thus it is disabled by default. We have seen
cases where the threshold of 85% - 90% would help. Figuring when
{{totalSpaceUsed / numNodes}} becomes close to the capacity of a smaller node
is helpful in determining the threshold.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8041) Consider remaining space during block blockplacement if dfs space is highly utilized
[
https://issues.apache.org/jira/browse/HDFS-8041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-8041:
-
Attachment: HDFS-8041.v3.patch
Attaching a new patch
Consider remaining space during block blockplacement if dfs space is highly
utilized
Key: HDFS-8041
URL: https://issues.apache.org/jira/browse/HDFS-8041
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Attachments: HDFS-8041.v1.patch, HDFS-8041.v2.patch,
HDFS-8041.v3.patch
This feature is helpful in avoiding smaller nodes (i.e. heterogeneous
environment) getting constantly being full when the overall space utilization
is over a certain threshold. When the utilization is low, balancer can keep
up, but once the average per-node byte goes over the capacity of the smaller
nodes, they get full so quickly even after perfect balance.
This jira proposes an improvement that can be optionally enabled in order to
slow down the rate of space usage growth of smaller nodes if the overall
storage utilization is over a configured threshold. It will not replace
balancer, rather will help balancer keep up. Also, the primary replica
placement will not be affected. Only the replicas typically placed in a
remote rack will be subject to this check.
The appropriate threshold is cluster configuration specific. There is no
generally good value to set, thus it is disabled by default. We have seen
cases where the threshold of 85% - 90% would help. Figuring when
{{totalSpaceUsed / numNodes}} becomes close to the capacity of a smaller node
is helpful in determining the threshold.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8041) Consider remaining space during block blockplacement if dfs space is highly utilized
[
https://issues.apache.org/jira/browse/HDFS-8041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-8041:
-
Attachment: HDFS-8041.v2.patch
Consider remaining space during block blockplacement if dfs space is highly
utilized
Key: HDFS-8041
URL: https://issues.apache.org/jira/browse/HDFS-8041
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Attachments: HDFS-8041.v1.patch, HDFS-8041.v2.patch
This feature is helpful in avoiding smaller nodes (i.e. heterogeneous
environment) getting constantly being full when the overall space utilization
is over a certain threshold. When the utilization is low, balancer can keep
up, but once the average per-node byte goes over the capacity of the smaller
nodes, they get full so quickly even after perfect balance.
This jira proposes an improvement that can be optionally enabled in order to
slow down the rate of space usage growth of smaller nodes if the overall
storage utilization is over a configured threshold. It will not replace
balancer, rather will help balancer keep up. Also, the primary replica
placement will not be affected. Only the replicas typically placed in a
remote rack will be subject to this check.
The appropriate threshold is cluster configuration specific. There is no
generally good value to set, thus it is disabled by default. We have seen
cases where the threshold of 85% - 90% would help. Figuring when
{{totalSpaceUsed / numNodes}} becomes close to the capacity of a smaller node
is helpful in determining the threshold.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8041) Consider remaining space during block blockplacement if dfs space is highly utilized
[
https://issues.apache.org/jira/browse/HDFS-8041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-8041:
-
Attachment: HDFS-8041.v1.patch
Consider remaining space during block blockplacement if dfs space is highly
utilized
Key: HDFS-8041
URL: https://issues.apache.org/jira/browse/HDFS-8041
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Attachments: HDFS-8041.v1.patch
This feature is helpful in avoiding smaller nodes (i.e. heterogeneous
environment) getting constantly being full when the overall space utilization
is over a certain threshold. When the utilization is low, balancer can keep
up, but once the average per-node byte goes over the capacity of the smaller
nodes, they get full so quickly even after perfect balance.
This jira proposes an improvement that can be optionally enabled in order to
slow down the rate of space usage growth of smaller nodes if the overall
storage utilization is over a configured threshold. It will not replace
balancer, rather will help balancer keep up. Also, the primary replica
placement will not be affected. Only the replicas typically placed in a
remote rack will be subject to this check.
The appropriate threshold is cluster configuration specific. There is no
generally good value to set, thus it is disabled by default. We have seen
cases where the threshold of 85% - 90% would help. Figuring when
{{totalSpaceUsed / numNodes}} becomes close to the capacity of a smaller node
is helpful in determining the threshold.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8041) Consider remaining space during block blockplacement if dfs space is highly utilized
[
https://issues.apache.org/jira/browse/HDFS-8041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-8041:
-
Status: Patch Available (was: Open)
Consider remaining space during block blockplacement if dfs space is highly
utilized
Key: HDFS-8041
URL: https://issues.apache.org/jira/browse/HDFS-8041
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Attachments: HDFS-8041.v1.patch
This feature is helpful in avoiding smaller nodes (i.e. heterogeneous
environment) getting constantly being full when the overall space utilization
is over a certain threshold. When the utilization is low, balancer can keep
up, but once the average per-node byte goes over the capacity of the smaller
nodes, they get full so quickly even after perfect balance.
This jira proposes an improvement that can be optionally enabled in order to
slow down the rate of space usage growth of smaller nodes if the overall
storage utilization is over a configured threshold. It will not replace
balancer, rather will help balancer keep up. Also, the primary replica
placement will not be affected. Only the replicas typically placed in a
remote rack will be subject to this check.
The appropriate threshold is cluster configuration specific. There is no
generally good value to set, thus it is disabled by default. We have seen
cases where the threshold of 85% - 90% would help. Figuring when
{{totalSpaceUsed / numNodes}} becomes close to the capacity of a smaller node
is helpful in determining the threshold.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
