[jira] [Updated] (HDFS-9322) Keep order of addresses to nameservice mapping from configuration file
[
https://issues.apache.org/jira/browse/HDFS-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rafal Wojdyla updated HDFS-9322:
Attachment: HDFS-9322.patch_1
> Keep order of addresses to nameservice mapping from configuration file
> --
>
> Key: HDFS-9322
> URL: https://issues.apache.org/jira/browse/HDFS-9322
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Rafal Wojdyla
>Assignee: Rafal Wojdyla
> Labels: has-patch
> Attachments: HDFS-9322.patch_1
>
>
> getAddressesForNameserviceId which is used by ConfiguredFailoverProxyProvider
> does not keep order of namenodes/addresses from configuration file - instead
> it relays on order given by HashMap (key is service id) which is misaligned
> with comment/doc in ConfiguredFailoverProxyProvider that says:
> {code}
> /**
> * A FailoverProxyProvider implementation which allows one to configure two
> URIs
> * to connect to during fail-over. The first configured address is tried
> first,
> * and on a fail-over event the other address is tried.
> */
> {code}
> One solution is to use LinkedHashMap which is insertion-ordered.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9322) Keep order of addresses to nameservice mapping from configuration file
[
https://issues.apache.org/jira/browse/HDFS-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rafal Wojdyla updated HDFS-9322:
Labels: has-patch (was: )
Status: Patch Available (was: Open)
> Keep order of addresses to nameservice mapping from configuration file
> --
>
> Key: HDFS-9322
> URL: https://issues.apache.org/jira/browse/HDFS-9322
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Rafal Wojdyla
>Assignee: Rafal Wojdyla
> Labels: has-patch
> Attachments: HDFS-9322.patch_1
>
>
> getAddressesForNameserviceId which is used by ConfiguredFailoverProxyProvider
> does not keep order of namenodes/addresses from configuration file - instead
> it relays on order given by HashMap (key is service id) which is misaligned
> with comment/doc in ConfiguredFailoverProxyProvider that says:
> {code}
> /**
> * A FailoverProxyProvider implementation which allows one to configure two
> URIs
> * to connect to during fail-over. The first configured address is tried
> first,
> * and on a fail-over event the other address is tried.
> */
> {code}
> One solution is to use LinkedHashMap which is insertion-ordered.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9321) Expose dispatcher parameters
Rafal Wojdyla created HDFS-9321: --- Summary: Expose dispatcher parameters Key: HDFS-9321 URL: https://issues.apache.org/jira/browse/HDFS-9321 Project: Hadoop HDFS Issue Type: Improvement Components: balancer & mover, HDFS Affects Versions: 2.7.1 Reporter: Rafal Wojdyla Assignee: Rafal Wojdyla To balance our cluster faster it was helpful to tune some hardcoded settings from dispatcher - which is used by balancer. Including: * max number of no-pending-move iterations * max iteration time * source wait time This patch adds exposes those parameters via configuration keys. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9321) Expose dispatcher parameters
[ https://issues.apache.org/jira/browse/HDFS-9321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rafal Wojdyla updated HDFS-9321: Labels: has-patch (was: ) Status: Patch Available (was: In Progress) > Expose dispatcher parameters > > > Key: HDFS-9321 > URL: https://issues.apache.org/jira/browse/HDFS-9321 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover, HDFS >Affects Versions: 2.7.1 >Reporter: Rafal Wojdyla >Assignee: Rafal Wojdyla > Labels: has-patch > Attachments: HDFS-9321.patch_1 > > > To balance our cluster faster it was helpful to tune some hardcoded settings > from dispatcher - which is used by balancer. Including: > * max number of no-pending-move iterations > * max iteration time > * source wait time > This patch adds exposes those parameters via configuration keys. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HDFS-9321) Expose dispatcher parameters
[ https://issues.apache.org/jira/browse/HDFS-9321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-9321 started by Rafal Wojdyla. --- > Expose dispatcher parameters > > > Key: HDFS-9321 > URL: https://issues.apache.org/jira/browse/HDFS-9321 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover, HDFS >Affects Versions: 2.7.1 >Reporter: Rafal Wojdyla >Assignee: Rafal Wojdyla > Attachments: HDFS-9321.patch_1 > > > To balance our cluster faster it was helpful to tune some hardcoded settings > from dispatcher - which is used by balancer. Including: > * max number of no-pending-move iterations > * max iteration time > * source wait time > This patch adds exposes those parameters via configuration keys. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9321) Expose dispatcher parameters
[ https://issues.apache.org/jira/browse/HDFS-9321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rafal Wojdyla updated HDFS-9321: Attachment: HDFS-9321.patch_1 > Expose dispatcher parameters > > > Key: HDFS-9321 > URL: https://issues.apache.org/jira/browse/HDFS-9321 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover, HDFS >Affects Versions: 2.7.1 >Reporter: Rafal Wojdyla >Assignee: Rafal Wojdyla > Attachments: HDFS-9321.patch_1 > > > To balance our cluster faster it was helpful to tune some hardcoded settings > from dispatcher - which is used by balancer. Including: > * max number of no-pending-move iterations > * max iteration time > * source wait time > This patch adds exposes those parameters via configuration keys. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9321) Expose extra dispatcher parameters
[ https://issues.apache.org/jira/browse/HDFS-9321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rafal Wojdyla updated HDFS-9321: Summary: Expose extra dispatcher parameters (was: Expose dispatcher parameters) > Expose extra dispatcher parameters > -- > > Key: HDFS-9321 > URL: https://issues.apache.org/jira/browse/HDFS-9321 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover, HDFS >Affects Versions: 2.7.1 >Reporter: Rafal Wojdyla >Assignee: Rafal Wojdyla > Labels: has-patch > Attachments: HDFS-9321.patch_1 > > > To balance our cluster faster it was helpful to tune some hardcoded settings > from dispatcher - which is used by balancer. Including: > * max number of no-pending-move iterations > * max iteration time > * source wait time > This patch adds exposes those parameters via configuration keys. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9322) Keep order of addresses to nameservice mapping from configuration file
Rafal Wojdyla created HDFS-9322:
---
Summary: Keep order of addresses to nameservice mapping from
configuration file
Key: HDFS-9322
URL: https://issues.apache.org/jira/browse/HDFS-9322
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.7.1
Reporter: Rafal Wojdyla
Assignee: Rafal Wojdyla
getAddressesForNameserviceId which is used by ConfiguredFailoverProxyProvider
does not keep order of namenodes/addresses from configuration file - instead it
relays on order given by HashMap (key is service id) which is misaligned with
comment/doc in ConfiguredFailoverProxyProvider that says:
{code}
/**
* A FailoverProxyProvider implementation which allows one to configure two URIs
* to connect to during fail-over. The first configured address is tried first,
* and on a fail-over event the other address is tried.
*/
{code}
One solution is to use LinkedHashMap which is insertion-ordered.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-2892) Some of property descriptions are not given(hdfs-default.xml)
[
https://issues.apache.org/jira/browse/HDFS-2892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rafal Wojdyla updated HDFS-2892:
Summary: Some of property descriptions are not given(hdfs-default.xml)
(was: Some of property descriptions are not given(hdfs-default.xml) )
Some of property descriptions are not given(hdfs-default.xml)
-
Key: HDFS-2892
URL: https://issues.apache.org/jira/browse/HDFS-2892
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 0.23.0
Reporter: Brahma Reddy Battula
Priority: Trivial
Hi..I taken 23.0 release form
http://hadoop.apache.org/common/releases.html#11+Nov%2C+2011%3A+release+0.23.0+available
I just gone through all properties provided in the hdfs-default.xml..Some of
the property description not mentioned..It's better to give description of
property and usage(how to configure ) and Only MapReduce related jars only
provided..Please check following two configurations
*No Description*
{noformat}
property
namedfs.datanode.https.address/name
value0.0.0.0:50475/value
/property
property
namedfs.namenode.https-address/name
value0.0.0.0:50470/value
/property
{noformat}
Better to mention example usage (what to configure...format(syntax))in
desc,here I did not get what default mean whether this name of n/w interface
or something else
property
namedfs.datanode.dns.interface/name
valuedefault/value
descriptionThe name of the Network Interface from which a data node
should
report its IP address.
/description
/property
The following property is commented..If it is not supported better to remove.
property
namedfs.cluster.administrators/name
valueACL for the admins/value
descriptionThis configuration is used to control who can access the
default servlets in the namenode, etc.
/description
/property
Small clarification for following property..if some value configured this
then NN will be safe mode upto this much time..
May I know usage of the following property...
property
namedfs.blockreport.initialDelay/name value0/value
descriptionDelay for first block report in seconds./description
/property
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7048) Incorrect Balancer#Source wait/notify leads to early termination
[ https://issues.apache.org/jira/browse/HDFS-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14132259#comment-14132259 ] Rafal Wojdyla commented on HDFS-7048: - [~yzhangal] I will update this ticket as soon as I get logs for this specific problem. Will be back soon. Incorrect Balancer#Source wait/notify leads to early termination Key: HDFS-7048 URL: https://issues.apache.org/jira/browse/HDFS-7048 Project: Hadoop HDFS Issue Type: Bug Components: balancer Affects Versions: 2.6.0 Reporter: Andrew Wang Split off from HDFS-6621. The Balancer attempts to wake up scheduler threads early as sources finish, but the synchronization with wait and notify is incorrect. This ticks the failure count, which can lead to early termination. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6621) Hadoop Balancer prematurely exits iterations
[
https://issues.apache.org/jira/browse/HDFS-6621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128713#comment-14128713
]
Rafal Wojdyla commented on HDFS-6621:
-
[~yzhangal] for problem 1 - patch is in [^HDFS-6621.patch]
Hadoop Balancer prematurely exits iterations
Key: HDFS-6621
URL: https://issues.apache.org/jira/browse/HDFS-6621
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer
Affects Versions: 2.2.0, 2.4.0
Environment: Red Hat Enterprise Linux Server release 5.8 with Hadoop
2.4.0
Reporter: Benjamin Bowman
Labels: balancer
Attachments: HDFS-6621.patch, HDFS-6621.patch_2, HDFS-6621.patch_3,
HDFS-6621.patch_4
I have been having an issue with the balancing being too slow. The issue was
not with the speed with which blocks were moved, but rather the balancer
would prematurely exit out of it's balancing iterations. It would move ~10
blocks or 100 MB then exit the current iteration (in which it said it was
planning on moving about 10 GB).
I looked in the Balancer.java code and believe I found and solved the issue.
In the dispatchBlocks() function there is a variable,
noPendingBlockIteration, which counts the number of iterations in which a
pending block to move cannot be found. Once this number gets to 5, the
balancer exits the overall balancing iteration. I believe the desired
functionality is 5 consecutive no pending block iterations - however this
variable is never reset to 0 upon block moves. So once this number reaches 5
- even if there have been thousands of blocks moved in between these no
pending block iterations - the overall balancing iteration will prematurely
end.
The fix I applied was to set noPendingBlockIteration = 0 when a pending block
is found and scheduled. In this way, my iterations do not prematurely exit
unless there is 5 consecutive no pending block iterations. Below is a copy
of my dispatchBlocks() function with the change I made.
{code}
private void dispatchBlocks() {
long startTime = Time.now();
long scheduledSize = getScheduledSize();
this.blocksToReceive = 2*scheduledSize;
boolean isTimeUp = false;
int noPendingBlockIteration = 0;
while(!isTimeUp getScheduledSize()0
(!srcBlockList.isEmpty() || blocksToReceive0)) {
PendingBlockMove pendingBlock = chooseNextBlockToMove();
if (pendingBlock != null) {
noPendingBlockIteration = 0;
// move the block
pendingBlock.scheduleBlockMove();
continue;
}
/* Since we can not schedule any block to move,
* filter any moved blocks from the source block list and
* check if we should fetch more blocks from the namenode
*/
filterMovedBlocks(); // filter already moved blocks
if (shouldFetchMoreBlocks()) {
// fetch new blocks
try {
blocksToReceive -= getBlockList();
continue;
} catch (IOException e) {
LOG.warn(Exception while getting block list, e);
return;
}
} else {
// source node cannot find a pendingBlockToMove, iteration +1
noPendingBlockIteration++;
// in case no blocks can be moved for source node's task,
// jump out of while-loop after 5 iterations.
if (noPendingBlockIteration = MAX_NO_PENDING_BLOCK_ITERATIONS) {
setScheduledSize(0);
}
}
// check if time is up or not
if (Time.now()-startTime MAX_ITERATION_TIME) {
isTimeUp = true;
continue;
}
/* Now we can not schedule any block to move and there are
* no new blocks added to the source block list, so we wait.
*/
try {
synchronized(Balancer.this) {
Balancer.this.wait(1000); // wait for targets/sources to be idle
}
} catch (InterruptedException ignored) {
}
}
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6621) Hadoop Balancer prematurely exits iterations
[
https://issues.apache.org/jira/browse/HDFS-6621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rafal Wojdyla updated HDFS-6621:
Attachment: HDFS-6621_problem1.patch
Hadoop Balancer prematurely exits iterations
Key: HDFS-6621
URL: https://issues.apache.org/jira/browse/HDFS-6621
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer
Affects Versions: 2.2.0, 2.4.0
Environment: Red Hat Enterprise Linux Server release 5.8 with Hadoop
2.4.0
Reporter: Benjamin Bowman
Labels: balancer
Attachments: HDFS-6621.patch, HDFS-6621.patch_2, HDFS-6621.patch_3,
HDFS-6621.patch_4, HDFS-6621_problem1.patch
I have been having an issue with the balancing being too slow. The issue was
not with the speed with which blocks were moved, but rather the balancer
would prematurely exit out of it's balancing iterations. It would move ~10
blocks or 100 MB then exit the current iteration (in which it said it was
planning on moving about 10 GB).
I looked in the Balancer.java code and believe I found and solved the issue.
In the dispatchBlocks() function there is a variable,
noPendingBlockIteration, which counts the number of iterations in which a
pending block to move cannot be found. Once this number gets to 5, the
balancer exits the overall balancing iteration. I believe the desired
functionality is 5 consecutive no pending block iterations - however this
variable is never reset to 0 upon block moves. So once this number reaches 5
- even if there have been thousands of blocks moved in between these no
pending block iterations - the overall balancing iteration will prematurely
end.
The fix I applied was to set noPendingBlockIteration = 0 when a pending block
is found and scheduled. In this way, my iterations do not prematurely exit
unless there is 5 consecutive no pending block iterations. Below is a copy
of my dispatchBlocks() function with the change I made.
{code}
private void dispatchBlocks() {
long startTime = Time.now();
long scheduledSize = getScheduledSize();
this.blocksToReceive = 2*scheduledSize;
boolean isTimeUp = false;
int noPendingBlockIteration = 0;
while(!isTimeUp getScheduledSize()0
(!srcBlockList.isEmpty() || blocksToReceive0)) {
PendingBlockMove pendingBlock = chooseNextBlockToMove();
if (pendingBlock != null) {
noPendingBlockIteration = 0;
// move the block
pendingBlock.scheduleBlockMove();
continue;
}
/* Since we can not schedule any block to move,
* filter any moved blocks from the source block list and
* check if we should fetch more blocks from the namenode
*/
filterMovedBlocks(); // filter already moved blocks
if (shouldFetchMoreBlocks()) {
// fetch new blocks
try {
blocksToReceive -= getBlockList();
continue;
} catch (IOException e) {
LOG.warn(Exception while getting block list, e);
return;
}
} else {
// source node cannot find a pendingBlockToMove, iteration +1
noPendingBlockIteration++;
// in case no blocks can be moved for source node's task,
// jump out of while-loop after 5 iterations.
if (noPendingBlockIteration = MAX_NO_PENDING_BLOCK_ITERATIONS) {
setScheduledSize(0);
}
}
// check if time is up or not
if (Time.now()-startTime MAX_ITERATION_TIME) {
isTimeUp = true;
continue;
}
/* Now we can not schedule any block to move and there are
* no new blocks added to the source block list, so we wait.
*/
try {
synchronized(Balancer.this) {
Balancer.this.wait(1000); // wait for targets/sources to be idle
}
} catch (InterruptedException ignored) {
}
}
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6621) Hadoop Balancer prematurely exits iterations
[
https://issues.apache.org/jira/browse/HDFS-6621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124903#comment-14124903
]
Rafal Wojdyla commented on HDFS-6621:
-
Hi [~yzhangal]
Thanks for comments, sorry for delay.
First of all - I agree that first problem is more important, and we should just
merge it in.
About solution to second problem, do we agree that the problem exists?
Especially with big number of threads such waking up for some threads may be
lethal even with fix for first problem. Is that correct?
It's been a while since I've made this change, and afair I tested both
problems/solutions and it they were separate problems, both of them cause
premature exists. First problem was more lethal tho.
About your comment with waiting - your are completely right! I missed this in
the patch. Now I see even more the value of pushing-patches/creating-tickets
right away ... not waiting till you have a bunch of changes.
Hadoop Balancer prematurely exits iterations
Key: HDFS-6621
URL: https://issues.apache.org/jira/browse/HDFS-6621
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer
Affects Versions: 2.2.0, 2.4.0
Environment: Red Hat Enterprise Linux Server release 5.8 with Hadoop
2.4.0
Reporter: Benjamin Bowman
Labels: balancer
Attachments: HDFS-6621.patch, HDFS-6621.patch_2
I have been having an issue with the balancing being too slow. The issue was
not with the speed with which blocks were moved, but rather the balancer
would prematurely exit out of it's balancing iterations. It would move ~10
blocks or 100 MB then exit the current iteration (in which it said it was
planning on moving about 10 GB).
I looked in the Balancer.java code and believe I found and solved the issue.
In the dispatchBlocks() function there is a variable,
noPendingBlockIteration, which counts the number of iterations in which a
pending block to move cannot be found. Once this number gets to 5, the
balancer exits the overall balancing iteration. I believe the desired
functionality is 5 consecutive no pending block iterations - however this
variable is never reset to 0 upon block moves. So once this number reaches 5
- even if there have been thousands of blocks moved in between these no
pending block iterations - the overall balancing iteration will prematurely
end.
The fix I applied was to set noPendingBlockIteration = 0 when a pending block
is found and scheduled. In this way, my iterations do not prematurely exit
unless there is 5 consecutive no pending block iterations. Below is a copy
of my dispatchBlocks() function with the change I made.
{code}
private void dispatchBlocks() {
long startTime = Time.now();
long scheduledSize = getScheduledSize();
this.blocksToReceive = 2*scheduledSize;
boolean isTimeUp = false;
int noPendingBlockIteration = 0;
while(!isTimeUp getScheduledSize()0
(!srcBlockList.isEmpty() || blocksToReceive0)) {
PendingBlockMove pendingBlock = chooseNextBlockToMove();
if (pendingBlock != null) {
noPendingBlockIteration = 0;
// move the block
pendingBlock.scheduleBlockMove();
continue;
}
/* Since we can not schedule any block to move,
* filter any moved blocks from the source block list and
* check if we should fetch more blocks from the namenode
*/
filterMovedBlocks(); // filter already moved blocks
if (shouldFetchMoreBlocks()) {
// fetch new blocks
try {
blocksToReceive -= getBlockList();
continue;
} catch (IOException e) {
LOG.warn(Exception while getting block list, e);
return;
}
} else {
// source node cannot find a pendingBlockToMove, iteration +1
noPendingBlockIteration++;
// in case no blocks can be moved for source node's task,
// jump out of while-loop after 5 iterations.
if (noPendingBlockIteration = MAX_NO_PENDING_BLOCK_ITERATIONS) {
setScheduledSize(0);
}
}
// check if time is up or not
if (Time.now()-startTime MAX_ITERATION_TIME) {
isTimeUp = true;
continue;
}
/* Now we can not schedule any block to move and there are
* no new blocks added to the source block list, so we wait.
*/
try {
synchronized(Balancer.this) {
Balancer.this.wait(1000); // wait for targets/sources to be idle
}
} catch (InterruptedException ignored) {
}
}
}
}
{code}
--
This message was
[jira] [Updated] (HDFS-6621) Hadoop Balancer prematurely exits iterations
[
https://issues.apache.org/jira/browse/HDFS-6621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rafal Wojdyla updated HDFS-6621:
Attachment: HDFS-6621.patch_3
Hadoop Balancer prematurely exits iterations
Key: HDFS-6621
URL: https://issues.apache.org/jira/browse/HDFS-6621
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer
Affects Versions: 2.2.0, 2.4.0
Environment: Red Hat Enterprise Linux Server release 5.8 with Hadoop
2.4.0
Reporter: Benjamin Bowman
Labels: balancer
Attachments: HDFS-6621.patch, HDFS-6621.patch_2, HDFS-6621.patch_3
I have been having an issue with the balancing being too slow. The issue was
not with the speed with which blocks were moved, but rather the balancer
would prematurely exit out of it's balancing iterations. It would move ~10
blocks or 100 MB then exit the current iteration (in which it said it was
planning on moving about 10 GB).
I looked in the Balancer.java code and believe I found and solved the issue.
In the dispatchBlocks() function there is a variable,
noPendingBlockIteration, which counts the number of iterations in which a
pending block to move cannot be found. Once this number gets to 5, the
balancer exits the overall balancing iteration. I believe the desired
functionality is 5 consecutive no pending block iterations - however this
variable is never reset to 0 upon block moves. So once this number reaches 5
- even if there have been thousands of blocks moved in between these no
pending block iterations - the overall balancing iteration will prematurely
end.
The fix I applied was to set noPendingBlockIteration = 0 when a pending block
is found and scheduled. In this way, my iterations do not prematurely exit
unless there is 5 consecutive no pending block iterations. Below is a copy
of my dispatchBlocks() function with the change I made.
{code}
private void dispatchBlocks() {
long startTime = Time.now();
long scheduledSize = getScheduledSize();
this.blocksToReceive = 2*scheduledSize;
boolean isTimeUp = false;
int noPendingBlockIteration = 0;
while(!isTimeUp getScheduledSize()0
(!srcBlockList.isEmpty() || blocksToReceive0)) {
PendingBlockMove pendingBlock = chooseNextBlockToMove();
if (pendingBlock != null) {
noPendingBlockIteration = 0;
// move the block
pendingBlock.scheduleBlockMove();
continue;
}
/* Since we can not schedule any block to move,
* filter any moved blocks from the source block list and
* check if we should fetch more blocks from the namenode
*/
filterMovedBlocks(); // filter already moved blocks
if (shouldFetchMoreBlocks()) {
// fetch new blocks
try {
blocksToReceive -= getBlockList();
continue;
} catch (IOException e) {
LOG.warn(Exception while getting block list, e);
return;
}
} else {
// source node cannot find a pendingBlockToMove, iteration +1
noPendingBlockIteration++;
// in case no blocks can be moved for source node's task,
// jump out of while-loop after 5 iterations.
if (noPendingBlockIteration = MAX_NO_PENDING_BLOCK_ITERATIONS) {
setScheduledSize(0);
}
}
// check if time is up or not
if (Time.now()-startTime MAX_ITERATION_TIME) {
isTimeUp = true;
continue;
}
/* Now we can not schedule any block to move and there are
* no new blocks added to the source block list, so we wait.
*/
try {
synchronized(Balancer.this) {
Balancer.this.wait(1000); // wait for targets/sources to be idle
}
} catch (InterruptedException ignored) {
}
}
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6621) Hadoop Balancer prematurely exits iterations
[
https://issues.apache.org/jira/browse/HDFS-6621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124982#comment-14124982
]
Rafal Wojdyla commented on HDFS-6621:
-
[~yzhangal] you're correct :D Sorry.
Reproducing error on real cluster - that's still feasible, reproducing this in
unit tests is kinda hard, I will try to come back with proof based on logs - is
that fine?
Hadoop Balancer prematurely exits iterations
Key: HDFS-6621
URL: https://issues.apache.org/jira/browse/HDFS-6621
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer
Affects Versions: 2.2.0, 2.4.0
Environment: Red Hat Enterprise Linux Server release 5.8 with Hadoop
2.4.0
Reporter: Benjamin Bowman
Labels: balancer
Attachments: HDFS-6621.patch, HDFS-6621.patch_2, HDFS-6621.patch_3
I have been having an issue with the balancing being too slow. The issue was
not with the speed with which blocks were moved, but rather the balancer
would prematurely exit out of it's balancing iterations. It would move ~10
blocks or 100 MB then exit the current iteration (in which it said it was
planning on moving about 10 GB).
I looked in the Balancer.java code and believe I found and solved the issue.
In the dispatchBlocks() function there is a variable,
noPendingBlockIteration, which counts the number of iterations in which a
pending block to move cannot be found. Once this number gets to 5, the
balancer exits the overall balancing iteration. I believe the desired
functionality is 5 consecutive no pending block iterations - however this
variable is never reset to 0 upon block moves. So once this number reaches 5
- even if there have been thousands of blocks moved in between these no
pending block iterations - the overall balancing iteration will prematurely
end.
The fix I applied was to set noPendingBlockIteration = 0 when a pending block
is found and scheduled. In this way, my iterations do not prematurely exit
unless there is 5 consecutive no pending block iterations. Below is a copy
of my dispatchBlocks() function with the change I made.
{code}
private void dispatchBlocks() {
long startTime = Time.now();
long scheduledSize = getScheduledSize();
this.blocksToReceive = 2*scheduledSize;
boolean isTimeUp = false;
int noPendingBlockIteration = 0;
while(!isTimeUp getScheduledSize()0
(!srcBlockList.isEmpty() || blocksToReceive0)) {
PendingBlockMove pendingBlock = chooseNextBlockToMove();
if (pendingBlock != null) {
noPendingBlockIteration = 0;
// move the block
pendingBlock.scheduleBlockMove();
continue;
}
/* Since we can not schedule any block to move,
* filter any moved blocks from the source block list and
* check if we should fetch more blocks from the namenode
*/
filterMovedBlocks(); // filter already moved blocks
if (shouldFetchMoreBlocks()) {
// fetch new blocks
try {
blocksToReceive -= getBlockList();
continue;
} catch (IOException e) {
LOG.warn(Exception while getting block list, e);
return;
}
} else {
// source node cannot find a pendingBlockToMove, iteration +1
noPendingBlockIteration++;
// in case no blocks can be moved for source node's task,
// jump out of while-loop after 5 iterations.
if (noPendingBlockIteration = MAX_NO_PENDING_BLOCK_ITERATIONS) {
setScheduledSize(0);
}
}
// check if time is up or not
if (Time.now()-startTime MAX_ITERATION_TIME) {
isTimeUp = true;
continue;
}
/* Now we can not schedule any block to move and there are
* no new blocks added to the source block list, so we wait.
*/
try {
synchronized(Balancer.this) {
Balancer.this.wait(1000); // wait for targets/sources to be idle
}
} catch (InterruptedException ignored) {
}
}
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6621) Hadoop Balancer prematurely exits iterations
[
https://issues.apache.org/jira/browse/HDFS-6621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rafal Wojdyla updated HDFS-6621:
Attachment: HDFS-6621.patch_4
Hadoop Balancer prematurely exits iterations
Key: HDFS-6621
URL: https://issues.apache.org/jira/browse/HDFS-6621
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer
Affects Versions: 2.2.0, 2.4.0
Environment: Red Hat Enterprise Linux Server release 5.8 with Hadoop
2.4.0
Reporter: Benjamin Bowman
Labels: balancer
Attachments: HDFS-6621.patch, HDFS-6621.patch_2, HDFS-6621.patch_3,
HDFS-6621.patch_4
I have been having an issue with the balancing being too slow. The issue was
not with the speed with which blocks were moved, but rather the balancer
would prematurely exit out of it's balancing iterations. It would move ~10
blocks or 100 MB then exit the current iteration (in which it said it was
planning on moving about 10 GB).
I looked in the Balancer.java code and believe I found and solved the issue.
In the dispatchBlocks() function there is a variable,
noPendingBlockIteration, which counts the number of iterations in which a
pending block to move cannot be found. Once this number gets to 5, the
balancer exits the overall balancing iteration. I believe the desired
functionality is 5 consecutive no pending block iterations - however this
variable is never reset to 0 upon block moves. So once this number reaches 5
- even if there have been thousands of blocks moved in between these no
pending block iterations - the overall balancing iteration will prematurely
end.
The fix I applied was to set noPendingBlockIteration = 0 when a pending block
is found and scheduled. In this way, my iterations do not prematurely exit
unless there is 5 consecutive no pending block iterations. Below is a copy
of my dispatchBlocks() function with the change I made.
{code}
private void dispatchBlocks() {
long startTime = Time.now();
long scheduledSize = getScheduledSize();
this.blocksToReceive = 2*scheduledSize;
boolean isTimeUp = false;
int noPendingBlockIteration = 0;
while(!isTimeUp getScheduledSize()0
(!srcBlockList.isEmpty() || blocksToReceive0)) {
PendingBlockMove pendingBlock = chooseNextBlockToMove();
if (pendingBlock != null) {
noPendingBlockIteration = 0;
// move the block
pendingBlock.scheduleBlockMove();
continue;
}
/* Since we can not schedule any block to move,
* filter any moved blocks from the source block list and
* check if we should fetch more blocks from the namenode
*/
filterMovedBlocks(); // filter already moved blocks
if (shouldFetchMoreBlocks()) {
// fetch new blocks
try {
blocksToReceive -= getBlockList();
continue;
} catch (IOException e) {
LOG.warn(Exception while getting block list, e);
return;
}
} else {
// source node cannot find a pendingBlockToMove, iteration +1
noPendingBlockIteration++;
// in case no blocks can be moved for source node's task,
// jump out of while-loop after 5 iterations.
if (noPendingBlockIteration = MAX_NO_PENDING_BLOCK_ITERATIONS) {
setScheduledSize(0);
}
}
// check if time is up or not
if (Time.now()-startTime MAX_ITERATION_TIME) {
isTimeUp = true;
continue;
}
/* Now we can not schedule any block to move and there are
* no new blocks added to the source block list, so we wait.
*/
try {
synchronized(Balancer.this) {
Balancer.this.wait(1000); // wait for targets/sources to be idle
}
} catch (InterruptedException ignored) {
}
}
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6621) Hadoop Balancer prematurely exits iterations
[
https://issues.apache.org/jira/browse/HDFS-6621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rafal Wojdyla updated HDFS-6621:
Attachment: HDFS-6621.patch_2
Hadoop Balancer prematurely exits iterations
Key: HDFS-6621
URL: https://issues.apache.org/jira/browse/HDFS-6621
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer
Affects Versions: 2.2.0, 2.4.0
Environment: Red Hat Enterprise Linux Server release 5.8 with Hadoop
2.4.0
Reporter: Benjamin Bowman
Labels: balancer
Attachments: HDFS-6621.patch, HDFS-6621.patch_2
I have been having an issue with the balancing being too slow. The issue was
not with the speed with which blocks were moved, but rather the balancer
would prematurely exit out of it's balancing iterations. It would move ~10
blocks or 100 MB then exit the current iteration (in which it said it was
planning on moving about 10 GB).
I looked in the Balancer.java code and believe I found and solved the issue.
In the dispatchBlocks() function there is a variable,
noPendingBlockIteration, which counts the number of iterations in which a
pending block to move cannot be found. Once this number gets to 5, the
balancer exits the overall balancing iteration. I believe the desired
functionality is 5 consecutive no pending block iterations - however this
variable is never reset to 0 upon block moves. So once this number reaches 5
- even if there have been thousands of blocks moved in between these no
pending block iterations - the overall balancing iteration will prematurely
end.
The fix I applied was to set noPendingBlockIteration = 0 when a pending block
is found and scheduled. In this way, my iterations do not prematurely exit
unless there is 5 consecutive no pending block iterations. Below is a copy
of my dispatchBlocks() function with the change I made.
{code}
private void dispatchBlocks() {
long startTime = Time.now();
long scheduledSize = getScheduledSize();
this.blocksToReceive = 2*scheduledSize;
boolean isTimeUp = false;
int noPendingBlockIteration = 0;
while(!isTimeUp getScheduledSize()0
(!srcBlockList.isEmpty() || blocksToReceive0)) {
PendingBlockMove pendingBlock = chooseNextBlockToMove();
if (pendingBlock != null) {
noPendingBlockIteration = 0;
// move the block
pendingBlock.scheduleBlockMove();
continue;
}
/* Since we can not schedule any block to move,
* filter any moved blocks from the source block list and
* check if we should fetch more blocks from the namenode
*/
filterMovedBlocks(); // filter already moved blocks
if (shouldFetchMoreBlocks()) {
// fetch new blocks
try {
blocksToReceive -= getBlockList();
continue;
} catch (IOException e) {
LOG.warn(Exception while getting block list, e);
return;
}
} else {
// source node cannot find a pendingBlockToMove, iteration +1
noPendingBlockIteration++;
// in case no blocks can be moved for source node's task,
// jump out of while-loop after 5 iterations.
if (noPendingBlockIteration = MAX_NO_PENDING_BLOCK_ITERATIONS) {
setScheduledSize(0);
}
}
// check if time is up or not
if (Time.now()-startTime MAX_ITERATION_TIME) {
isTimeUp = true;
continue;
}
/* Now we can not schedule any block to move and there are
* no new blocks added to the source block list, so we wait.
*/
try {
synchronized(Balancer.this) {
Balancer.this.wait(1000); // wait for targets/sources to be idle
}
} catch (InterruptedException ignored) {
}
}
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6621) Hadoop Balancer prematurely exits iterations
[
https://issues.apache.org/jira/browse/HDFS-6621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14109788#comment-14109788
]
Rafal Wojdyla commented on HDFS-6621:
-
Hi [~yzhangal],
The patch is pretty simple - but it's hard to reproduce it unit tests in sane
fashion.
Working on it.
Hadoop Balancer prematurely exits iterations
Key: HDFS-6621
URL: https://issues.apache.org/jira/browse/HDFS-6621
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer
Affects Versions: 2.2.0, 2.4.0
Environment: Red Hat Enterprise Linux Server release 5.8 with Hadoop
2.4.0
Reporter: Benjamin Bowman
Labels: balancer
Attachments: HDFS-6621.patch
I have been having an issue with the balancing being too slow. The issue was
not with the speed with which blocks were moved, but rather the balancer
would prematurely exit out of it's balancing iterations. It would move ~10
blocks or 100 MB then exit the current iteration (in which it said it was
planning on moving about 10 GB).
I looked in the Balancer.java code and believe I found and solved the issue.
In the dispatchBlocks() function there is a variable,
noPendingBlockIteration, which counts the number of iterations in which a
pending block to move cannot be found. Once this number gets to 5, the
balancer exits the overall balancing iteration. I believe the desired
functionality is 5 consecutive no pending block iterations - however this
variable is never reset to 0 upon block moves. So once this number reaches 5
- even if there have been thousands of blocks moved in between these no
pending block iterations - the overall balancing iteration will prematurely
end.
The fix I applied was to set noPendingBlockIteration = 0 when a pending block
is found and scheduled. In this way, my iterations do not prematurely exit
unless there is 5 consecutive no pending block iterations. Below is a copy
of my dispatchBlocks() function with the change I made.
{code}
private void dispatchBlocks() {
long startTime = Time.now();
long scheduledSize = getScheduledSize();
this.blocksToReceive = 2*scheduledSize;
boolean isTimeUp = false;
int noPendingBlockIteration = 0;
while(!isTimeUp getScheduledSize()0
(!srcBlockList.isEmpty() || blocksToReceive0)) {
PendingBlockMove pendingBlock = chooseNextBlockToMove();
if (pendingBlock != null) {
noPendingBlockIteration = 0;
// move the block
pendingBlock.scheduleBlockMove();
continue;
}
/* Since we can not schedule any block to move,
* filter any moved blocks from the source block list and
* check if we should fetch more blocks from the namenode
*/
filterMovedBlocks(); // filter already moved blocks
if (shouldFetchMoreBlocks()) {
// fetch new blocks
try {
blocksToReceive -= getBlockList();
continue;
} catch (IOException e) {
LOG.warn(Exception while getting block list, e);
return;
}
} else {
// source node cannot find a pendingBlockToMove, iteration +1
noPendingBlockIteration++;
// in case no blocks can be moved for source node's task,
// jump out of while-loop after 5 iterations.
if (noPendingBlockIteration = MAX_NO_PENDING_BLOCK_ITERATIONS) {
setScheduledSize(0);
}
}
// check if time is up or not
if (Time.now()-startTime MAX_ITERATION_TIME) {
isTimeUp = true;
continue;
}
/* Now we can not schedule any block to move and there are
* no new blocks added to the source block list, so we wait.
*/
try {
synchronized(Balancer.this) {
Balancer.this.wait(1000); // wait for targets/sources to be idle
}
} catch (InterruptedException ignored) {
}
}
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6648) Order of namenodes in ConfiguredFailoverProxyProvider is not defined by order in hdfs-site.xml
[
https://issues.apache.org/jira/browse/HDFS-6648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14099984#comment-14099984
]
Rafal Wojdyla commented on HDFS-6648:
-
Hi [~qwertymaniac], good to know that it wasn't a design goal - btw, what is
the best/easiest way to check what were the design goals for given
class/component - is Jira the only good place for that?
Java doc for ConfiguredFailoverProxyProvider says:
{code}
/**
* A FailoverProxyProvider implementation which allows one to configure two URIs
* to connect to during fail-over. The first configured address is tried first,
* and on a fail-over event the other address is tried.
*/
public class ConfiguredFailoverProxyProviderT extends
{code}
It says The first configured address is tried first - which is not true.
This was a major issue for us due to other bugs, including but not limited to:
* HDFS-5064
* HDFS-4858
So at the end of the day some clients were trying to connect to Standby
Namenode which sometimes was very unresponsive, it was killing the performance
big time.
Order taken from configuration file makes it more intuitive for administrator,
and makes it possible for administrator to mitigate bugs like the ones above by
explicitly defining order of namenodes.
Order of namenodes in ConfiguredFailoverProxyProvider is not defined by order
in hdfs-site.xml
--
Key: HDFS-6648
URL: https://issues.apache.org/jira/browse/HDFS-6648
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, hdfs-client
Affects Versions: 2.2.0
Reporter: Rafal Wojdyla
In org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider,
in the constructor, there's a map nameservice : service-id :
service-rpc-address (DFSUtil.getHaNnRpcAddresses). It's a LinkedHashMap
of HashMaps. The order is kept for _nameservices_. Then to find active
namenode, for nameservice, we get HashMap of service-id :
service-rpc-address for requested nameservice (taken from URI request), And
for this HashMap we get values - order of this collection is not strictly
defined! In the code:
{code}
CollectionInetSocketAddress addressesOfNns = addressesInNN.values();
{code}
And then we put these values (in not defined order) into ArrayList of
proxies, and then in getProxy we start from first proxy in the list and
failover to next if needed.
It would make sense for ConfiguredFailoverProxyProvider to keep order of
proxies/namenodes defined in hdfs-site.xml.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6621) Hadoop Balancer prematurely exits iterations
[
https://issues.apache.org/jira/browse/HDFS-6621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14058496#comment-14058496
]
Rafal Wojdyla commented on HDFS-6621:
-
We have also experience this problem with Balancer.
The problem in general is that balancer will prematurely finish iteration due
to noPendingBlockIteration = 5.
I was about to create JIRA ticket for this - but I have noticed this ticket.
The solutions that, we have applied is to:
1. noPendingBlockIteration = 0 when pendingBlock != null, exactly the way you
did
2. notify only on source object when block transfer finishes
Problem/Solutions 1 is well described above.
Problem/Solutions 2:
In org/apache/hadoop/hdfs/server/balancer/Balancer:
{code}
private void dispatch() {
...
synchronized (Balancer.this) {
Balancer.this.notifyAll();
}
}
{code}
this will notify all scheduling threads, even the ones that are waiting and
still have all 5 transfer threads occupied.
When occupied task wakes up, it will try to get next block to move, but because
all 5 transfer threads are occupied
it will get null as next block to move - which will increase
noPendingBlockIteration, and we are in the problem 1.
The solution is to notify threads waiting on source object and reset
PendingBlockMove object afterwords.
Should I provide patch in this ticket, or create a separate ticket?
Hadoop Balancer prematurely exits iterations
Key: HDFS-6621
URL: https://issues.apache.org/jira/browse/HDFS-6621
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer
Affects Versions: 2.2.0, 2.4.0
Environment: Red Hat Enterprise Linux Server release 5.8 with Hadoop
2.4.0
Reporter: Benjamin Bowman
Labels: balancer
Attachments: HDFS-6621.patch
I have been having an issue with the balancing being too slow. The issue was
not with the speed with which blocks were moved, but rather the balancer
would prematurely exit out of it's balancing iterations. It would move ~10
blocks or 100 MB then exit the current iteration (in which it said it was
planning on moving about 10 GB).
I looked in the Balancer.java code and believe I found and solved the issue.
In the dispatchBlocks() function there is a variable,
noPendingBlockIteration, which counts the number of iterations in which a
pending block to move cannot be found. Once this number gets to 5, the
balancer exits the overall balancing iteration. I believe the desired
functionality is 5 consecutive no pending block iterations - however this
variable is never reset to 0 upon block moves. So once this number reaches 5
- even if there have been thousands of blocks moved in between these no
pending block iterations - the overall balancing iteration will prematurely
end.
The fix I applied was to set noPendingBlockIteration = 0 when a pending block
is found and scheduled. In this way, my iterations do not prematurely exit
unless there is 5 consecutive no pending block iterations. Below is a copy
of my dispatchBlocks() function with the change I made.
private void dispatchBlocks() {
long startTime = Time.now();
long scheduledSize = getScheduledSize();
this.blocksToReceive = 2*scheduledSize;
boolean isTimeUp = false;
int noPendingBlockIteration = 0;
while(!isTimeUp getScheduledSize()0
(!srcBlockList.isEmpty() || blocksToReceive0)) {
PendingBlockMove pendingBlock = chooseNextBlockToMove();
if (pendingBlock != null) {
noPendingBlockIteration = 0;
// move the block
pendingBlock.scheduleBlockMove();
continue;
}
/* Since we can not schedule any block to move,
* filter any moved blocks from the source block list and
* check if we should fetch more blocks from the namenode
*/
filterMovedBlocks(); // filter already moved blocks
if (shouldFetchMoreBlocks()) {
// fetch new blocks
try {
blocksToReceive -= getBlockList();
continue;
} catch (IOException e) {
LOG.warn(Exception while getting block list, e);
return;
}
} else {
// source node cannot find a pendingBlockToMove, iteration +1
noPendingBlockIteration++;
// in case no blocks can be moved for source node's task,
// jump out of while-loop after 5 iterations.
if (noPendingBlockIteration = MAX_NO_PENDING_BLOCK_ITERATIONS) {
setScheduledSize(0);
}
}
// check if time is up or not
if (Time.now()-startTime MAX_ITERATION_TIME) {
isTimeUp = true;
continue;
}
[jira] [Created] (HDFS-6648) Order of namenodes in ConfiguredFailoverProxyProvider is not defined by order in hdfs-site.xml
Rafal Wojdyla created HDFS-6648:
---
Summary: Order of namenodes in ConfiguredFailoverProxyProvider is
not defined by order in hdfs-site.xml
Key: HDFS-6648
URL: https://issues.apache.org/jira/browse/HDFS-6648
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, hdfs-client
Affects Versions: 2.2.0
Reporter: Rafal Wojdyla
In org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider,
in the constructor, there's a map nameservice : service-id :
service-rpc-address (DFSUtil.getHaNnRpcAddresses). It's a LinkedHashMap of
HashMaps. The order is kept for _nameservices_. Then to find active namenode,
for nameservice, we get HashMap of service-id : service-rpc-address for
requested nameservice (taken from URI request), And for this HashMap we get
values - order of this collection is not strictly defined! In the code:
{code}
CollectionInetSocketAddress addressesOfNns = addressesInNN.values();
{code}
And then we put these values (in not defined order) into ArrayList of proxies,
and then in getProxy we start from first proxy in the list and failover to next
if needed.
It would make sense for ConfiguredFailoverProxyProvider to keep order of
proxies/namenodes defined in hdfs-site.xml.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6179) Synchronized
Rafal Wojdyla created HDFS-6179:
---
Summary: Synchronized
Key: HDFS-6179
URL: https://issues.apache.org/jira/browse/HDFS-6179
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode, namenode
Affects Versions: 2.2.0
Reporter: Rafal Wojdyla
Scenario:
* 600 ative DNs
* 1 *active* NN
* HA configuration
When we start SbNN because of huge number of blocks and relative small
initialDelay - SbNN during startup will go through multiple stop-the-world
garbage collections processes (in minutes - Namenode heap size is 75GB). We've
observed that SbNN slowness affects active NN so active NN is losing DNs (DNs
are considered dead due to lack of heartbeats). We assume that some DNs are
hanging.
When DN is considered dead by active Namenode, we've observed dead lock in DN
process, part of stack trace:
{noformat}
DataNode: [file:/disk1,file:/disk2] heartbeating to
standbynamenode.net/10.10.10.10:8020 daemon prio=10 tid=0x7ff429417800
nid=0x7f2a in Object.wait() [0x7ff42122c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.hadoop.ipc.Client.call(Client.java:1333)
- locked 0x0007db94e4c8 (a org.apache.hadoop.ipc.Client$Call)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at $Proxy9.registerDatanode(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at $Proxy9.registerDatanode(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.registerDatanode(DatanodeProtocolClientSideTranslatorPB.java:146)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:623)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.reRegister(BPServiceActor.java:740)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromStandby(BPOfferService.java:603)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:506)
- locked 0x000780006e08 (a
org.apache.hadoop.hdfs.server.datanode.BPOfferService)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:704)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:539)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:676)
at java.lang.Thread.run(Thread.java:662)
DataNode: [file:/disk1,file:/disk2] heartbeating to
activenamenode.net/10.10.10.11:8020 daemon prio=10 tid=0x7ff428a24000
nid=0x7f29 waiting for monitor entry [0x7ff42132e000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.updateActorStatesFromHeartbeat(BPOfferService.java:413)
- waiting to lock 0x000780006e08 (a
org.apache.hadoop.hdfs.server.datanode.BPOfferService)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:535)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:676)
at java.lang.Thread.run(Thread.java:662)
{noformat}
Notice that it's the same lock - due to synchronization at BPOfferService. The
problem is that command from standby can't be process due to unresponsive
standby Namenode, nevertheless DN is waiting for reply from SbNN, and is
waiting long enough to be considered dead by active namenode.
Info: if we kill SbNN, DN will instantly reconnect to active NN.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6179) Synchronized BPOfferService - datanode locks for slow namenode reply.
[
https://issues.apache.org/jira/browse/HDFS-6179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rafal Wojdyla updated HDFS-6179:
Summary: Synchronized BPOfferService - datanode locks for slow namenode
reply. (was: Synchronized )
Synchronized BPOfferService - datanode locks for slow namenode reply.
-
Key: HDFS-6179
URL: https://issues.apache.org/jira/browse/HDFS-6179
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode, namenode
Affects Versions: 2.2.0
Reporter: Rafal Wojdyla
Scenario:
* 600 ative DNs
* 1 *active* NN
* HA configuration
When we start SbNN because of huge number of blocks and relative small
initialDelay - SbNN during startup will go through multiple stop-the-world
garbage collections processes (in minutes - Namenode heap size is 75GB).
We've observed that SbNN slowness affects active NN so active NN is losing
DNs (DNs are considered dead due to lack of heartbeats). We assume that some
DNs are hanging.
When DN is considered dead by active Namenode, we've observed dead lock in
DN process, part of stack trace:
{noformat}
DataNode: [file:/disk1,file:/disk2] heartbeating to
standbynamenode.net/10.10.10.10:8020 daemon prio=10 tid=0x7ff429417800
nid=0x7f2a in Object.wait() [0x7ff42122c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.hadoop.ipc.Client.call(Client.java:1333)
- locked 0x0007db94e4c8 (a org.apache.hadoop.ipc.Client$Call)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at $Proxy9.registerDatanode(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at $Proxy9.registerDatanode(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.registerDatanode(DatanodeProtocolClientSideTranslatorPB.java:146)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:623)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.reRegister(BPServiceActor.java:740)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromStandby(BPOfferService.java:603)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:506)
- locked 0x000780006e08 (a
org.apache.hadoop.hdfs.server.datanode.BPOfferService)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:704)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:539)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:676)
at java.lang.Thread.run(Thread.java:662)
DataNode: [file:/disk1,file:/disk2] heartbeating to
activenamenode.net/10.10.10.11:8020 daemon prio=10 tid=0x7ff428a24000
nid=0x7f29 waiting for monitor entry [0x7ff42132e000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.updateActorStatesFromHeartbeat(BPOfferService.java:413)
- waiting to lock 0x000780006e08 (a
org.apache.hadoop.hdfs.server.datanode.BPOfferService)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:535)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:676)
at java.lang.Thread.run(Thread.java:662)
{noformat}
Notice that it's the same lock - due to synchronization at BPOfferService.
The problem is that command from standby can't be process due to unresponsive
standby Namenode, nevertheless DN is waiting for reply from SbNN, and is
waiting long enough to be considered dead by active namenode.
Info: if we kill SbNN, DN will instantly reconnect to active NN.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6179) Synchronized BPOfferService - datanode locks for slow namenode reply.
[
https://issues.apache.org/jira/browse/HDFS-6179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13957015#comment-13957015
]
Rafal Wojdyla commented on HDFS-6179:
-
Yes it's the same problem as HDFS-5014. Thank you!
Synchronized BPOfferService - datanode locks for slow namenode reply.
-
Key: HDFS-6179
URL: https://issues.apache.org/jira/browse/HDFS-6179
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode, namenode
Affects Versions: 2.2.0
Reporter: Rafal Wojdyla
Scenario:
* 600 ative DNs
* 1 *active* NN
* HA configuration
When we start SbNN because of huge number of blocks and relative small
initialDelay - SbNN during startup will go through multiple stop-the-world
garbage collections processes (in minutes - Namenode heap size is 75GB).
We've observed that SbNN slowness affects active NN so active NN is losing
DNs (DNs are considered dead due to lack of heartbeats). We assume that some
DNs are hanging.
When DN is considered dead by active Namenode, we've observed dead lock in
DN process, part of stack trace:
{noformat}
DataNode: [file:/disk1,file:/disk2] heartbeating to
standbynamenode.net/10.10.10.10:8020 daemon prio=10 tid=0x7ff429417800
nid=0x7f2a in Object.wait() [0x7ff42122c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.hadoop.ipc.Client.call(Client.java:1333)
- locked 0x0007db94e4c8 (a org.apache.hadoop.ipc.Client$Call)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at $Proxy9.registerDatanode(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at $Proxy9.registerDatanode(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.registerDatanode(DatanodeProtocolClientSideTranslatorPB.java:146)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:623)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.reRegister(BPServiceActor.java:740)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromStandby(BPOfferService.java:603)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:506)
- locked 0x000780006e08 (a
org.apache.hadoop.hdfs.server.datanode.BPOfferService)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:704)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:539)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:676)
at java.lang.Thread.run(Thread.java:662)
DataNode: [file:/disk1,file:/disk2] heartbeating to
activenamenode.net/10.10.10.11:8020 daemon prio=10 tid=0x7ff428a24000
nid=0x7f29 waiting for monitor entry [0x7ff42132e000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.updateActorStatesFromHeartbeat(BPOfferService.java:413)
- waiting to lock 0x000780006e08 (a
org.apache.hadoop.hdfs.server.datanode.BPOfferService)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:535)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:676)
at java.lang.Thread.run(Thread.java:662)
{noformat}
Notice that it's the same lock - due to synchronization at BPOfferService.
The problem is that command from standby can't be process due to unresponsive
standby Namenode, nevertheless DN is waiting for reply from SbNN, and is
waiting long enough to be considered dead by active namenode.
Info: if we kill SbNN, DN will instantly reconnect to active NN.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6082) List all NNs with state
[ https://issues.apache.org/jira/browse/HDFS-6082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rafal Wojdyla updated HDFS-6082: Priority: Minor (was: Major) List all NNs with state --- Key: HDFS-6082 URL: https://issues.apache.org/jira/browse/HDFS-6082 Project: Hadoop HDFS Issue Type: Improvement Components: ha, namenode Reporter: Rafal Wojdyla Priority: Minor HAAdmin let's you determine state of *given* service. It would be nice to have a call to determine states of all Namenodes (services?), something like: hdfs haadmin -ns foobar -getServicesState And the output would be: hostname | state This can be implemented at HAAdmin level - for all HA services or at DFSHAAdmin level - for Namenode HA only. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6082) List all NNs with state
Rafal Wojdyla created HDFS-6082: --- Summary: List all NNs with state Key: HDFS-6082 URL: https://issues.apache.org/jira/browse/HDFS-6082 Project: Hadoop HDFS Issue Type: Improvement Components: ha, namenode Reporter: Rafal Wojdyla HAAdmin let's you determine state of *given* service. It would be nice to have a call to determine states of all Namenodes (services?), something like: hdfs haadmin -getServicesState And the output would be: hostname | state This can be implemented at HAAdmin level - for all HA services or at DFSHAAdmin level - for Namenode HA only. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6082) List all NNs with state
[ https://issues.apache.org/jira/browse/HDFS-6082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rafal Wojdyla updated HDFS-6082: Description: HAAdmin let's you determine state of *given* service. It would be nice to have a call to determine states of all Namenodes (services?), something like: hdfs haadmin -ns foobar -getServicesState And the output would be: hostname | state This can be implemented at HAAdmin level - for all HA services or at DFSHAAdmin level - for Namenode HA only. was: HAAdmin let's you determine state of *given* service. It would be nice to have a call to determine states of all Namenodes (services?), something like: hdfs haadmin -getServicesState And the output would be: hostname | state This can be implemented at HAAdmin level - for all HA services or at DFSHAAdmin level - for Namenode HA only. List all NNs with state --- Key: HDFS-6082 URL: https://issues.apache.org/jira/browse/HDFS-6082 Project: Hadoop HDFS Issue Type: Improvement Components: ha, namenode Reporter: Rafal Wojdyla HAAdmin let's you determine state of *given* service. It would be nice to have a call to determine states of all Namenodes (services?), something like: hdfs haadmin -ns foobar -getServicesState And the output would be: hostname | state This can be implemented at HAAdmin level - for all HA services or at DFSHAAdmin level - for Namenode HA only. -- This message was sent by Atlassian JIRA (v6.2#6252)
