[jira] [Commented] (HBASE-6469) Failure on enable/disable table will cause table state in zk to be left as enabling/disabling until master is restarted
[
https://issues.apache.org/jira/browse/HBASE-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14134283#comment-14134283
]
Andrey Stepachev commented on HBASE-6469:
-
HBASE-7767 done, do we still need this?
Failure on enable/disable table will cause table state in zk to be left as
enabling/disabling until master is restarted
---
Key: HBASE-6469
URL: https://issues.apache.org/jira/browse/HBASE-6469
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.94.6
Reporter: Enis Soztutar
Assignee: rajeshbabu
Priority: Critical
Attachments: 6469-expose-force-r3.patch, HBASE-6469.patch,
HBASE-6469_2.patch, HBASE-6469_3.patch, HBASE-6469_4.patch,
HBASE-6469_retry_enable_or_disable.patch
In Enable/DisableTableHandler code, if something goes wrong in handling, the
table state in zk is left as ENABLING / DISABLING. After that we cannot force
any more action from the API or CLI, and the only recovery path is restarting
the master.
{code}
if (done) {
// Flip the table to enabled.
this.assignmentManager.getZKTable().setEnabledTable(
this.tableNameStr);
LOG.info(Table ' + this.tableNameStr
+ ' was successfully enabled. Status: done= + done);
} else {
LOG.warn(Table ' + this.tableNameStr
+ ' wasn't successfully enabled. Status: done= + done);
}
{code}
Here, if done is false, the table state is not changed. There is also no way
to set skipTableStateCheck from cli / api.
We have run into this issue a couple of times before.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-6469) Failure on enable/disable table will cause table state in zk to be left as enabling/disabling until master is restarted
[
https://issues.apache.org/jira/browse/HBASE-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14031151#comment-14031151
]
Mikhail Antonov commented on HBASE-6469:
Is that still critical and still in progress?
I guess if HBASE-7767 is done, this one would not be an issue.
Failure on enable/disable table will cause table state in zk to be left as
enabling/disabling until master is restarted
---
Key: HBASE-6469
URL: https://issues.apache.org/jira/browse/HBASE-6469
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.94.6
Reporter: Enis Soztutar
Assignee: rajeshbabu
Priority: Critical
Attachments: 6469-expose-force-r3.patch, HBASE-6469.patch,
HBASE-6469_2.patch, HBASE-6469_3.patch, HBASE-6469_4.patch,
HBASE-6469_retry_enable_or_disable.patch
In Enable/DisableTableHandler code, if something goes wrong in handling, the
table state in zk is left as ENABLING / DISABLING. After that we cannot force
any more action from the API or CLI, and the only recovery path is restarting
the master.
{code}
if (done) {
// Flip the table to enabled.
this.assignmentManager.getZKTable().setEnabledTable(
this.tableNameStr);
LOG.info(Table ' + this.tableNameStr
+ ' was successfully enabled. Status: done= + done);
} else {
LOG.warn(Table ' + this.tableNameStr
+ ' wasn't successfully enabled. Status: done= + done);
}
{code}
Here, if done is false, the table state is not changed. There is also no way
to set skipTableStateCheck from cli / api.
We have run into this issue a couple of times before.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-6469) Failure on enable/disable table will cause table state in zk to be left as enabling/disabling until master is restarted
[
https://issues.apache.org/jira/browse/HBASE-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618975#comment-13618975
]
rajeshbabu commented on HBASE-6469:
---
[~jxiang]
bq. I think it is because the ZKTable states are cached.
Here even zktable states are cached still they are same as znode states in zk.
bq. we can manually reset the table state in ZK, then retry will be able to
move on.
Can you give more details how we can do this? How HBASE-8233 really solves this
problem?
Failure on enable/disable table will cause table state in zk to be left as
enabling/disabling until master is restarted
---
Key: HBASE-6469
URL: https://issues.apache.org/jira/browse/HBASE-6469
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.0, 0.94.6
Reporter: Enis Soztutar
Assignee: rajeshbabu
Fix For: 0.95.0, 0.94.7
Attachments: 6469-expose-force-r3.patch, HBASE-6469_2.patch,
HBASE-6469_3.patch, HBASE-6469_4.patch, HBASE-6469.patch,
HBASE-6469_retry_enable_or_disable.patch
In Enable/DisableTableHandler code, if something goes wrong in handling, the
table state in zk is left as ENABLING / DISABLING. After that we cannot force
any more action from the API or CLI, and the only recovery path is restarting
the master.
{code}
if (done) {
// Flip the table to enabled.
this.assignmentManager.getZKTable().setEnabledTable(
this.tableNameStr);
LOG.info(Table ' + this.tableNameStr
+ ' was successfully enabled. Status: done= + done);
} else {
LOG.warn(Table ' + this.tableNameStr
+ ' wasn't successfully enabled. Status: done= + done);
}
{code}
Here, if done is false, the table state is not changed. There is also no way
to set skipTableStateCheck from cli / api.
We have run into this issue a couple of times before.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6469) Failure on enable/disable table will cause table state in zk to be left as enabling/disabling until master is restarted
[
https://issues.apache.org/jira/browse/HBASE-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618980#comment-13618980
]
Jimmy Xiang commented on HBASE-6469:
[~rajesh23], if we manually change the znode state in zk from zkcli, will the
zktable state be updated? Do we have a watcher on that znode? If so, we can
manually set the table to be disabled/enabled, then from hbase shell to retry
enable/disable the table. Will this work?
Failure on enable/disable table will cause table state in zk to be left as
enabling/disabling until master is restarted
---
Key: HBASE-6469
URL: https://issues.apache.org/jira/browse/HBASE-6469
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.0, 0.94.6
Reporter: Enis Soztutar
Assignee: rajeshbabu
Fix For: 0.95.0, 0.94.7
Attachments: 6469-expose-force-r3.patch, HBASE-6469_2.patch,
HBASE-6469_3.patch, HBASE-6469_4.patch, HBASE-6469.patch,
HBASE-6469_retry_enable_or_disable.patch
In Enable/DisableTableHandler code, if something goes wrong in handling, the
table state in zk is left as ENABLING / DISABLING. After that we cannot force
any more action from the API or CLI, and the only recovery path is restarting
the master.
{code}
if (done) {
// Flip the table to enabled.
this.assignmentManager.getZKTable().setEnabledTable(
this.tableNameStr);
LOG.info(Table ' + this.tableNameStr
+ ' was successfully enabled. Status: done= + done);
} else {
LOG.warn(Table ' + this.tableNameStr
+ ' wasn't successfully enabled. Status: done= + done);
}
{code}
Here, if done is false, the table state is not changed. There is also no way
to set skipTableStateCheck from cli / api.
We have run into this issue a couple of times before.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6469) Failure on enable/disable table will cause table state in zk to be left as enabling/disabling until master is restarted
[
https://issues.apache.org/jira/browse/HBASE-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619011#comment-13619011
]
rajeshbabu commented on HBASE-6469:
---
[~jxiang] We dont have watcher on these znodes. May be we can do this but it
will be little risky and difficult I feel.
Failure on enable/disable table will cause table state in zk to be left as
enabling/disabling until master is restarted
---
Key: HBASE-6469
URL: https://issues.apache.org/jira/browse/HBASE-6469
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.0, 0.94.6
Reporter: Enis Soztutar
Assignee: rajeshbabu
Fix For: 0.95.0, 0.94.7
Attachments: 6469-expose-force-r3.patch, HBASE-6469_2.patch,
HBASE-6469_3.patch, HBASE-6469_4.patch, HBASE-6469.patch,
HBASE-6469_retry_enable_or_disable.patch
In Enable/DisableTableHandler code, if something goes wrong in handling, the
table state in zk is left as ENABLING / DISABLING. After that we cannot force
any more action from the API or CLI, and the only recovery path is restarting
the master.
{code}
if (done) {
// Flip the table to enabled.
this.assignmentManager.getZKTable().setEnabledTable(
this.tableNameStr);
LOG.info(Table ' + this.tableNameStr
+ ' was successfully enabled. Status: done= + done);
} else {
LOG.warn(Table ' + this.tableNameStr
+ ' wasn't successfully enabled. Status: done= + done);
}
{code}
Here, if done is false, the table state is not changed. There is also no way
to set skipTableStateCheck from cli / api.
We have run into this issue a couple of times before.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6469) Failure on enable/disable table will cause table state in zk to be left as enabling/disabling until master is restarted
[
https://issues.apache.org/jira/browse/HBASE-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619027#comment-13619027
]
Jimmy Xiang commented on HBASE-6469:
I agree. I think it is not needed. That's why I suggested HBASE-8233, which
may be useful in other scenarios too.
Failure on enable/disable table will cause table state in zk to be left as
enabling/disabling until master is restarted
---
Key: HBASE-6469
URL: https://issues.apache.org/jira/browse/HBASE-6469
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.0, 0.94.6
Reporter: Enis Soztutar
Assignee: rajeshbabu
Fix For: 0.95.0, 0.94.7
Attachments: 6469-expose-force-r3.patch, HBASE-6469_2.patch,
HBASE-6469_3.patch, HBASE-6469_4.patch, HBASE-6469.patch,
HBASE-6469_retry_enable_or_disable.patch
In Enable/DisableTableHandler code, if something goes wrong in handling, the
table state in zk is left as ENABLING / DISABLING. After that we cannot force
any more action from the API or CLI, and the only recovery path is restarting
the master.
{code}
if (done) {
// Flip the table to enabled.
this.assignmentManager.getZKTable().setEnabledTable(
this.tableNameStr);
LOG.info(Table ' + this.tableNameStr
+ ' was successfully enabled. Status: done= + done);
} else {
LOG.warn(Table ' + this.tableNameStr
+ ' wasn't successfully enabled. Status: done= + done);
}
{code}
Here, if done is false, the table state is not changed. There is also no way
to set skipTableStateCheck from cli / api.
We have run into this issue a couple of times before.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6469) Failure on enable/disable table will cause table state in zk to be left as enabling/disabling until master is restarted
[
https://issues.apache.org/jira/browse/HBASE-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619039#comment-13619039
]
rajeshbabu commented on HBASE-6469:
---
bq. That's why I suggested HBASE-8233, which may be useful in other scenarios
too.
But I didnt see any state mismatch with cache and znodes(in that case
HBASE-8233 may solve). May be I understood wrongly.
Failure on enable/disable table will cause table state in zk to be left as
enabling/disabling until master is restarted
---
Key: HBASE-6469
URL: https://issues.apache.org/jira/browse/HBASE-6469
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.0, 0.94.6
Reporter: Enis Soztutar
Assignee: rajeshbabu
Fix For: 0.95.0, 0.94.7
Attachments: 6469-expose-force-r3.patch, HBASE-6469_2.patch,
HBASE-6469_3.patch, HBASE-6469_4.patch, HBASE-6469.patch,
HBASE-6469_retry_enable_or_disable.patch
In Enable/DisableTableHandler code, if something goes wrong in handling, the
table state in zk is left as ENABLING / DISABLING. After that we cannot force
any more action from the API or CLI, and the only recovery path is restarting
the master.
{code}
if (done) {
// Flip the table to enabled.
this.assignmentManager.getZKTable().setEnabledTable(
this.tableNameStr);
LOG.info(Table ' + this.tableNameStr
+ ' was successfully enabled. Status: done= + done);
} else {
LOG.warn(Table ' + this.tableNameStr
+ ' wasn't successfully enabled. Status: done= + done);
}
{code}
Here, if done is false, the table state is not changed. There is also no way
to set skipTableStateCheck from cli / api.
We have run into this issue a couple of times before.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6469) Failure on enable/disable table will cause table state in zk to be left as enabling/disabling until master is restarted
[
https://issues.apache.org/jira/browse/HBASE-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618465#comment-13618465
]
Hadoop QA commented on HBASE-6469:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12576298/HBASE-6469_retry_enable_or_disable.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.snapshot.TestRestoreFlushSnapshotFromClient
org.apache.hadoop.hbase.coprocessor.TestMasterObserver
org.apache.hadoop.hbase.util.TestMergeTable
org.apache.hadoop.hbase.util.TestHBaseFsck
org.apache.hadoop.hbase.procedure.TestZKProcedure
org.apache.hadoop.hbase.backup.TestHFileArchiving
org.apache.hadoop.hbase.master.handler.TestTableDeleteFamilyHandler
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
org.apache.hadoop.hbase.client.TestAdmin
org.apache.hadoop.hbase.client.TestCloneSnapshotFromClient
org.apache.hadoop.hbase.client.TestSnapshotMetadata
org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster
org.apache.hadoop.hbase.client.TestSnapshotCloneIndependence
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClient
org.apache.hadoop.hbase.master.TestDistributedLogSplitting
org.apache.hadoop.hbase.client.TestSnapshotFromClient
org.apache.hadoop.hbase.snapshot.TestExportSnapshot
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5075//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5075//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5075//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5075//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5075//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5075//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5075//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5075//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5075//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5075//console
This message is automatically generated.
Failure on enable/disable table will cause table state in zk to be left as
enabling/disabling until master is restarted
---
Key: HBASE-6469
URL: https://issues.apache.org/jira/browse/HBASE-6469
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.0, 0.94.6
Reporter: Enis Soztutar
Assignee: rajeshbabu
Fix For: 0.95.0, 0.94.7
Attachments: 6469-expose-force-r3.patch, HBASE-6469_2.patch,
HBASE-6469_3.patch, HBASE-6469_4.patch, HBASE-6469.patch,
HBASE-6469_retry_enable_or_disable.patch
In Enable/DisableTableHandler code, if something goes wrong in handling, the
table state in zk is left as ENABLING / DISABLING. After that we cannot force
[jira] [Commented] (HBASE-6469) Failure on enable/disable table will cause table state in zk to be left as enabling/disabling until master is restarted
[
https://issues.apache.org/jira/browse/HBASE-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618558#comment-13618558
]
Jimmy Xiang commented on HBASE-6469:
I think it is because the ZKTable states are cached. Otherwise, we can
manually reset the table state in ZK, then retry will be able to move on.
If there is a feature to poke the cache, it will be great, so we don't have to
restart the master. I just filed HBASE-8233, which should be helpful in this
case.
Failure on enable/disable table will cause table state in zk to be left as
enabling/disabling until master is restarted
---
Key: HBASE-6469
URL: https://issues.apache.org/jira/browse/HBASE-6469
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.0, 0.94.6
Reporter: Enis Soztutar
Assignee: rajeshbabu
Fix For: 0.95.0, 0.94.7
Attachments: 6469-expose-force-r3.patch, HBASE-6469_2.patch,
HBASE-6469_3.patch, HBASE-6469_4.patch, HBASE-6469.patch,
HBASE-6469_retry_enable_or_disable.patch
In Enable/DisableTableHandler code, if something goes wrong in handling, the
table state in zk is left as ENABLING / DISABLING. After that we cannot force
any more action from the API or CLI, and the only recovery path is restarting
the master.
{code}
if (done) {
// Flip the table to enabled.
this.assignmentManager.getZKTable().setEnabledTable(
this.tableNameStr);
LOG.info(Table ' + this.tableNameStr
+ ' was successfully enabled. Status: done= + done);
} else {
LOG.warn(Table ' + this.tableNameStr
+ ' wasn't successfully enabled. Status: done= + done);
}
{code}
Here, if done is false, the table state is not changed. There is also no way
to set skipTableStateCheck from cli / api.
We have run into this issue a couple of times before.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6469) Failure on enable/disable table will cause table state in zk to be left as enabling/disabling until master is restarted
[
https://issues.apache.org/jira/browse/HBASE-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618231#comment-13618231
]
Lars Hofhansl commented on HBASE-6469:
--
What is it that restarting the Master does that fixes the problem? Can we do
the same when we have another attempt at \{en|dis\}abling the table?
Failure on enable/disable table will cause table state in zk to be left as
enabling/disabling until master is restarted
---
Key: HBASE-6469
URL: https://issues.apache.org/jira/browse/HBASE-6469
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.0, 0.94.6
Reporter: Enis Soztutar
Assignee: rajeshbabu
Fix For: 0.95.0, 0.94.7
Attachments: 6469-expose-force-r3.patch, HBASE-6469_2.patch,
HBASE-6469_3.patch, HBASE-6469_4.patch, HBASE-6469.patch
In Enable/DisableTableHandler code, if something goes wrong in handling, the
table state in zk is left as ENABLING / DISABLING. After that we cannot force
any more action from the API or CLI, and the only recovery path is restarting
the master.
{code}
if (done) {
// Flip the table to enabled.
this.assignmentManager.getZKTable().setEnabledTable(
this.tableNameStr);
LOG.info(Table ' + this.tableNameStr
+ ' was successfully enabled. Status: done= + done);
} else {
LOG.warn(Table ' + this.tableNameStr
+ ' wasn't successfully enabled. Status: done= + done);
}
{code}
Here, if done is false, the table state is not changed. There is also no way
to set skipTableStateCheck from cli / api.
We have run into this issue a couple of times before.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6469) Failure on enable/disable table will cause table state in zk to be left as enabling/disabling until master is restarted
[
https://issues.apache.org/jira/browse/HBASE-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13597107#comment-13597107
]
ramkrishna.s.vasudevan commented on HBASE-6469:
---
I have similar concern as Lars has. Infact if you see latest changes N and
others are trying to remove this timing mechanisms. So introducing such a
thing will not welcomed.
Lets see what others say and check this out.
Failure on enable/disable table will cause table state in zk to be left as
enabling/disabling until master is restarted
---
Key: HBASE-6469
URL: https://issues.apache.org/jira/browse/HBASE-6469
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.0, 0.94.6
Reporter: Enis Soztutar
Assignee: rajeshbabu
Fix For: 0.95.0, 0.94.7
Attachments: 6469-expose-force-r3.patch, HBASE-6469_2.patch,
HBASE-6469_3.patch, HBASE-6469_4.patch, HBASE-6469.patch
In Enable/DisableTableHandler code, if something goes wrong in handling, the
table state in zk is left as ENABLING / DISABLING. After that we cannot force
any more action from the API or CLI, and the only recovery path is restarting
the master.
{code}
if (done) {
// Flip the table to enabled.
this.assignmentManager.getZKTable().setEnabledTable(
this.tableNameStr);
LOG.info(Table ' + this.tableNameStr
+ ' was successfully enabled. Status: done= + done);
} else {
LOG.warn(Table ' + this.tableNameStr
+ ' wasn't successfully enabled. Status: done= + done);
}
{code}
Here, if done is false, the table state is not changed. There is also no way
to set skipTableStateCheck from cli / api.
We have run into this issue a couple of times before.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6469) Failure on enable/disable table will cause table state in zk to be left as enabling/disabling until master is restarted
[
https://issues.apache.org/jira/browse/HBASE-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13585941#comment-13585941
]
ramkrishna.s.vasudevan commented on HBASE-6469:
---
@Rajesh
bq.Enable/disable the tables again if something wrong happened in the middle
Improve the comments for this class.
Saw some formatting related issues. Will go thro once again tomorrow and let
you know.
Failure on enable/disable table will cause table state in zk to be left as
enabling/disabling until master is restarted
---
Key: HBASE-6469
URL: https://issues.apache.org/jira/browse/HBASE-6469
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.2, 0.96.0
Reporter: Enis Soztutar
Assignee: rajeshbabu
Fix For: 0.96.0, 0.94.6
Attachments: 6469-expose-force-r3.patch, HBASE-6469_2.patch,
HBASE-6469.patch
In Enable/DisableTableHandler code, if something goes wrong in handling, the
table state in zk is left as ENABLING / DISABLING. After that we cannot force
any more action from the API or CLI, and the only recovery path is restarting
the master.
{code}
if (done) {
// Flip the table to enabled.
this.assignmentManager.getZKTable().setEnabledTable(
this.tableNameStr);
LOG.info(Table ' + this.tableNameStr
+ ' was successfully enabled. Status: done= + done);
} else {
LOG.warn(Table ' + this.tableNameStr
+ ' wasn't successfully enabled. Status: done= + done);
}
{code}
Here, if done is false, the table state is not changed. There is also no way
to set skipTableStateCheck from cli / api.
We have run into this issue a couple of times before.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6469) Failure on enable/disable table will cause table state in zk to be left as enabling/disabling until master is restarted
[
https://issues.apache.org/jira/browse/HBASE-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13585950#comment-13585950
]
Hadoop QA commented on HBASE-6469:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12570795/HBASE-6469_2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4527//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4527//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4527//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4527//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4527//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4527//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4527//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4527//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4527//console
This message is automatically generated.
Failure on enable/disable table will cause table state in zk to be left as
enabling/disabling until master is restarted
---
Key: HBASE-6469
URL: https://issues.apache.org/jira/browse/HBASE-6469
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.2, 0.96.0
Reporter: Enis Soztutar
Assignee: rajeshbabu
Fix For: 0.96.0, 0.94.6
Attachments: 6469-expose-force-r3.patch, HBASE-6469_2.patch,
HBASE-6469.patch
In Enable/DisableTableHandler code, if something goes wrong in handling, the
table state in zk is left as ENABLING / DISABLING. After that we cannot force
any more action from the API or CLI, and the only recovery path is restarting
the master.
{code}
if (done) {
// Flip the table to enabled.
this.assignmentManager.getZKTable().setEnabledTable(
this.tableNameStr);
LOG.info(Table ' + this.tableNameStr
+ ' was successfully enabled. Status: done= + done);
} else {
LOG.warn(Table ' + this.tableNameStr
+ ' wasn't successfully enabled. Status: done= + done);
}
{code}
Here, if done is false, the table state is not changed. There is also no way
to set skipTableStateCheck from cli / api.
We have run into this issue a couple of times before.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6469) Failure on enable/disable table will cause table state in zk to be left as enabling/disabling until master is restarted
[
https://issues.apache.org/jira/browse/HBASE-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586276#comment-13586276
]
Hadoop QA commented on HBASE-6469:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12570843/HBASE-6469_3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4535//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4535//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4535//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4535//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4535//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4535//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4535//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4535//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4535//console
This message is automatically generated.
Failure on enable/disable table will cause table state in zk to be left as
enabling/disabling until master is restarted
---
Key: HBASE-6469
URL: https://issues.apache.org/jira/browse/HBASE-6469
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.2, 0.96.0
Reporter: Enis Soztutar
Assignee: rajeshbabu
Fix For: 0.96.0, 0.94.6
Attachments: 6469-expose-force-r3.patch, HBASE-6469_2.patch,
HBASE-6469_3.patch, HBASE-6469.patch
In Enable/DisableTableHandler code, if something goes wrong in handling, the
table state in zk is left as ENABLING / DISABLING. After that we cannot force
any more action from the API or CLI, and the only recovery path is restarting
the master.
{code}
if (done) {
// Flip the table to enabled.
this.assignmentManager.getZKTable().setEnabledTable(
this.tableNameStr);
LOG.info(Table ' + this.tableNameStr
+ ' was successfully enabled. Status: done= + done);
} else {
LOG.warn(Table ' + this.tableNameStr
+ ' wasn't successfully enabled. Status: done= + done);
}
{code}
Here, if done is false, the table state is not changed. There is also no way
to set skipTableStateCheck from cli / api.
We have run into this issue a couple of times before.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6469) Failure on enable/disable table will cause table state in zk to be left as enabling/disabling until master is restarted
[
https://issues.apache.org/jira/browse/HBASE-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586523#comment-13586523
]
Lars Hofhansl commented on HBASE-6469:
--
I am not sure I like this approach.
The problem here is that the ZK state is either stuck in ENABLING or DISABLING.
A background thread checking on this does not seem to be that useful.
If a background can continue the transaction, why can't another (user
initiated) attempt to either enable or disable the table?
The expectation here is that a user won't care what care what state the table
is currently in. If a disable is triggered the result should be a disabled
table (and similar for an enable).
Can we just make so that we allow transitioning from ENABLING to both ENABLED
or DISABLED and from DISABLING to both ENABLED or DISABLED?
Failure on enable/disable table will cause table state in zk to be left as
enabling/disabling until master is restarted
---
Key: HBASE-6469
URL: https://issues.apache.org/jira/browse/HBASE-6469
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.2, 0.96.0
Reporter: Enis Soztutar
Assignee: rajeshbabu
Fix For: 0.96.0, 0.94.6
Attachments: 6469-expose-force-r3.patch, HBASE-6469_2.patch,
HBASE-6469_3.patch, HBASE-6469.patch
In Enable/DisableTableHandler code, if something goes wrong in handling, the
table state in zk is left as ENABLING / DISABLING. After that we cannot force
any more action from the API or CLI, and the only recovery path is restarting
the master.
{code}
if (done) {
// Flip the table to enabled.
this.assignmentManager.getZKTable().setEnabledTable(
this.tableNameStr);
LOG.info(Table ' + this.tableNameStr
+ ' was successfully enabled. Status: done= + done);
} else {
LOG.warn(Table ' + this.tableNameStr
+ ' wasn't successfully enabled. Status: done= + done);
}
{code}
Here, if done is false, the table state is not changed. There is also no way
to set skipTableStateCheck from cli / api.
We have run into this issue a couple of times before.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6469) Failure on enable/disable table will cause table state in zk to be left as enabling/disabling until master is restarted
[
https://issues.apache.org/jira/browse/HBASE-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586855#comment-13586855
]
rajeshbabu commented on HBASE-6469:
---
@Lars,
bq.If a background can continue the transaction, why can't another (user
initiated) attempt to either enable or disable the table?
Multiple calls of enable/disable may cause inconsistencies.
Here is the comment from EnableTableHandler/DisableTableHandler constructor.
{code}
// There could be multiple client requests trying to disable or enable
// the table at the same time. Ensure only the first request is honored
// After that, no other requests can be accepted until the table reaches
// DISABLED or ENABLED.
{code}
bq.The expectation here is that a user won't care what care what state the
table is currently in. If a disable is triggered the result should be a
disabled table (and similar for an enable).
I agree but user need to know about the current state of
assignments/unassignments(as of now) before calling force enable/disable.
bq.Can we just make so that we allow transitioning from ENABLING to both
ENABLED or DISABLED and from DISABLING to both ENABLED or DISABLED?
If we transition from ENABLING to DISABLED then the regions which are in
transtion or yet to start assignment wont be assigned. Some times even assigned
regions will be unassigned.
Below are the some code snippets which are used to handle inconsistencies with
disable and balance race.
{code}
private void assign(RegionState state,
final boolean setOfflineInZK, final boolean forceNewPlan) {
if (isDisabledorDisablingRegionInRIT(region)) {
return;
}
...
}
{code}
After completion of rit transition we will check once more that table is
disabling or disabled
{code}
boolean disabled = getZKTable().isDisablingOrDisabledTable(
regionInfo.getTableNameAsString());
if (!serverManager.isServerOnline(serverName) !disabled) {
LOG.info(Opened region + regionNameStr
+ but the region server is offline, reassign the region);
assign(regionInfo, true);
} else if (disabled) {
// if server is offline, no hurt to unassign again
LOG.info(Opened region + regionNameStr
+ but this table is disabled, triggering close of region);
unassign(regionInfo);
}
{code}
During master restart we will skip regions assignment of disabling/disabled
tables(This case is if master restarted after transition).
{code}
SetString disabledOrDisablingOrEnabling =
ZKTable.getDisabledOrDisablingTables(watcher);
disabledOrDisablingOrEnabling.addAll(ZKTable.getEnablingTables(watcher));
// Scan META for all user regions, skipping any disabled tables
MapHRegionInfo, ServerName allRegions = MetaReader.fullScan(
catalogTracker, disabledOrDisablingOrEnabling, true);
{code}
Exactly reverse(assign instead of unassign) will happen if we transition from
DISABLING to ENABLED.
code snippet from ClosedRegionHandler.process()
{code}
if (this.assignmentManager.getZKTable().
isDisablingOrDisabledTable(this.regionInfo.getTableNameAsString())) {
assignmentManager.offlineDisabledRegion(regionInfo);
return;
}
// ZK Node is in CLOSED state, assign it.
assignmentManager.getRegionStates().updateRegionState(
regionInfo, RegionState.State.CLOSED, null);
// This below has to do w/ online enable/disable of a table
assignmentManager.removeClosedRegion(regionInfo);
assignmentManager.assign(regionInfo, true);
{code}
Please correct me if I am wrong. Thanks Lars.
Failure on enable/disable table will cause table state in zk to be left as
enabling/disabling until master is restarted
---
Key: HBASE-6469
URL: https://issues.apache.org/jira/browse/HBASE-6469
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.2, 0.96.0
Reporter: Enis Soztutar
Assignee: rajeshbabu
Fix For: 0.96.0, 0.94.7
Attachments: 6469-expose-force-r3.patch, HBASE-6469_2.patch,
HBASE-6469_3.patch, HBASE-6469.patch
In Enable/DisableTableHandler code, if something goes wrong in handling, the
table state in zk is left as ENABLING / DISABLING. After that we cannot force
any more action from the API or CLI, and the only recovery path is restarting
the master.
{code}
if (done) {
// Flip the table to enabled.
this.assignmentManager.getZKTable().setEnabledTable(
this.tableNameStr);
LOG.info(Table ' + this.tableNameStr
+ ' was
[jira] [Commented] (HBASE-6469) Failure on enable/disable table will cause table state in zk to be left as enabling/disabling until master is restarted
[
https://issues.apache.org/jira/browse/HBASE-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13584743#comment-13584743
]
Ted Yu commented on HBASE-6469:
---
Overall, looks quite good.
{code}
+ public class TableStateUpdater extends Chore {
{code}
This class is internal to AssignmentManager and can be declared private.
{code}
+protected void chore() {
+ if (!tablesInPartialState.isEmpty()) {
{code}
If you write the condition differently, you should be able to save some
indentation.
{code}
+} catch (KeeperException e) {
+ LOG.error(Error trying to enable the table +
tableEntry.getKey(), e);
+}
{code}
With the current position of the catch block, you need some flag so that
'enable' and 'disable' can be selected in the log.
{code}
+ if (tableEntry.getValue() == regionStates.getRegionsOfTable(
+ Bytes.toBytes(tableEntry.getKey())).size()) {
{code}
Is it possible that region count gets bigger than tableEntry.getValue() ? If
not, add PreCondition ?
{code}
+MapString, Integer tablesInPartialState = new TreeMapString, Integer();
{code}
nit: if you use byte[] as key (with proper comparator), you can save the
Bytes.toBytes() call.
Failure on enable/disable table will cause table state in zk to be left as
enabling/disabling until master is restarted
---
Key: HBASE-6469
URL: https://issues.apache.org/jira/browse/HBASE-6469
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.2, 0.96.0
Reporter: Enis Soztutar
Assignee: rajeshbabu
Fix For: 0.96.0, 0.94.6
Attachments: 6469-expose-force-r3.patch, HBASE-6469.patch
In Enable/DisableTableHandler code, if something goes wrong in handling, the
table state in zk is left as ENABLING / DISABLING. After that we cannot force
any more action from the API or CLI, and the only recovery path is restarting
the master.
{code}
if (done) {
// Flip the table to enabled.
this.assignmentManager.getZKTable().setEnabledTable(
this.tableNameStr);
LOG.info(Table ' + this.tableNameStr
+ ' was successfully enabled. Status: done= + done);
} else {
LOG.warn(Table ' + this.tableNameStr
+ ' wasn't successfully enabled. Status: done= + done);
}
{code}
Here, if done is false, the table state is not changed. There is also no way
to set skipTableStateCheck from cli / api.
We have run into this issue a couple of times before.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
