[jira] [Commented] (HBASE-5702) MasterSchemaChangeTracker.excludeRegionServerForSchemaChanges leaks a MonitoredTask per call
[ https://issues.apache.org/jira/browse/HBASE-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13245793#comment-13245793 ] Jean-Daniel Cryans commented on HBASE-5702: --- Yeah when I opened this jira it had a narrow scope but I think it could be much bigger. Basically if you try to instant alter a table you'll see there's about two or three (sorry I can't be more precise at the moment, I'm testing something else right now) tasks that are leaked. I can see some issues just looking at the code: MasterSchemaChangeTracker - processCompletedSchemaChanges: creates a task, sets the status, never closes it. It's a misuse of MonitoredTask I think, a task is normally something that's long running and you need to report progress. - processAlterStatus: creates a task but stops it right away, also creates one then kills it. - handleFailedOrExpiredSchemaChanges: creates a task, sets the status, never closes it. Also it has an extra white space. - createSchemaChangeNode: creates a task then closes/aborts it right away SchemaChangeTracker - handleSchemaChange: creates a task, sets the status, never closes it. - reportAndLogSchemaRefreshError: can create task and set a status but doesn't close it. There really should only be 1 task throughout the alter process. > MasterSchemaChangeTracker.excludeRegionServerForSchemaChanges leaks a > MonitoredTask per call > > > Key: HBASE-5702 > URL: https://issues.apache.org/jira/browse/HBASE-5702 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.0 >Reporter: Jean-Daniel Cryans >Assignee: Subbu M Iyer >Priority: Critical > Fix For: 0.94.0, 0.96.0 > > > This bug is so easy to reproduce I'm wondering why it hasn't been reported > yet. Stop any number of region servers on a 0.94/6 cluster and you'll see in > the master interface one task per stopped region server saying the following: > |Processing schema change exclusion for region server = > sv4r27s44,62023,1333402175340|RUNNING (since 5sec ago)|No schema change in > progress. Skipping exclusion for server = sv4r27s44,62023,1333402175340 > (since 5sec ago)| > It's gonna stay there until the master cleans it: > bq. WARN org.apache.hadoop.hbase.monitoring.TaskMonitor: Status Processing > schema change exclusion for region server = sv4r27s44,62023,1333402175340: > status=No schema change in progress. Skipping exclusion for server = > sv4r27s44,62023,1333402175340, state=RUNNING, startTime=1333404636419, > completionTime=-1 appears to have been leaked > It's not clear to me why it's using a MonitoredTask in the first place. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5702) MasterSchemaChangeTracker.excludeRegionServerForSchemaChanges leaks a MonitoredTask per call
[ https://issues.apache.org/jira/browse/HBASE-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13245650#comment-13245650 ] Subbu M Iyer commented on HBASE-5702: - JD, Can you share with me the "many more issues with MonitoredTask" that you have mentioned? I believe the problems you have seen are all related to the MonitoredTask reporting during instant schema change process and not with the actual schema change process itself? Please let me know so we can categorize and address the issues accordingly. > MasterSchemaChangeTracker.excludeRegionServerForSchemaChanges leaks a > MonitoredTask per call > > > Key: HBASE-5702 > URL: https://issues.apache.org/jira/browse/HBASE-5702 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.0 >Reporter: Jean-Daniel Cryans >Assignee: Subbu M Iyer >Priority: Critical > Fix For: 0.94.0, 0.96.0 > > > This bug is so easy to reproduce I'm wondering why it hasn't been reported > yet. Stop any number of region servers on a 0.94/6 cluster and you'll see in > the master interface one task per stopped region server saying the following: > |Processing schema change exclusion for region server = > sv4r27s44,62023,1333402175340|RUNNING (since 5sec ago)|No schema change in > progress. Skipping exclusion for server = sv4r27s44,62023,1333402175340 > (since 5sec ago)| > It's gonna stay there until the master cleans it: > bq. WARN org.apache.hadoop.hbase.monitoring.TaskMonitor: Status Processing > schema change exclusion for region server = sv4r27s44,62023,1333402175340: > status=No schema change in progress. Skipping exclusion for server = > sv4r27s44,62023,1333402175340, state=RUNNING, startTime=1333404636419, > completionTime=-1 appears to have been leaked > It's not clear to me why it's using a MonitoredTask in the first place. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5702) MasterSchemaChangeTracker.excludeRegionServerForSchemaChanges leaks a MonitoredTask per call
[
https://issues.apache.org/jira/browse/HBASE-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13245496#comment-13245496
]
Zhihong Yu commented on HBASE-5702:
---
The following method in ServerManager.java doesn't check the current value for
hbase.instant.schema.alter.enabled config:
{code}
private void excludeRegionServerFromSchemaChanges(final ServerName
serverName) {
this.services.getSchemaChangeTracker()
.excludeRegionServerForSchemaChanges(serverName.getServerName());
}
{code}
> MasterSchemaChangeTracker.excludeRegionServerForSchemaChanges leaks a
> MonitoredTask per call
>
>
> Key: HBASE-5702
> URL: https://issues.apache.org/jira/browse/HBASE-5702
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.0
>Reporter: Jean-Daniel Cryans
>Priority: Critical
> Fix For: 0.94.0, 0.96.0
>
>
> This bug is so easy to reproduce I'm wondering why it hasn't been reported
> yet. Stop any number of region servers on a 0.94/6 cluster and you'll see in
> the master interface one task per stopped region server saying the following:
> |Processing schema change exclusion for region server =
> sv4r27s44,62023,1333402175340|RUNNING (since 5sec ago)|No schema change in
> progress. Skipping exclusion for server = sv4r27s44,62023,1333402175340
> (since 5sec ago)|
> It's gonna stay there until the master cleans it:
> bq. WARN org.apache.hadoop.hbase.monitoring.TaskMonitor: Status Processing
> schema change exclusion for region server = sv4r27s44,62023,1333402175340:
> status=No schema change in progress. Skipping exclusion for server =
> sv4r27s44,62023,1333402175340, state=RUNNING, startTime=1333404636419,
> completionTime=-1 appears to have been leaked
> It's not clear to me why it's using a MonitoredTask in the first place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5702) MasterSchemaChangeTracker.excludeRegionServerForSchemaChanges leaks a MonitoredTask per call
[ https://issues.apache.org/jira/browse/HBASE-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13244816#comment-13244816 ] Jean-Daniel Cryans commented on HBASE-5702: --- Now that I've tested "instant" schema updates, there's many more issues with MonitoredTask than I originally thought :( > MasterSchemaChangeTracker.excludeRegionServerForSchemaChanges leaks a > MonitoredTask per call > > > Key: HBASE-5702 > URL: https://issues.apache.org/jira/browse/HBASE-5702 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.0 >Reporter: Jean-Daniel Cryans > Fix For: 0.94.0, 0.96.0 > > > This bug is so easy to reproduce I'm wondering why it hasn't been reported > yet. Stop any number of region servers on a 0.94/6 cluster and you'll see in > the master interface one task per stopped region server saying the following: > |Processing schema change exclusion for region server = > sv4r27s44,62023,1333402175340|RUNNING (since 5sec ago)|No schema change in > progress. Skipping exclusion for server = sv4r27s44,62023,1333402175340 > (since 5sec ago)| > It's gonna stay there until the master cleans it: > bq. WARN org.apache.hadoop.hbase.monitoring.TaskMonitor: Status Processing > schema change exclusion for region server = sv4r27s44,62023,1333402175340: > status=No schema change in progress. Skipping exclusion for server = > sv4r27s44,62023,1333402175340, state=RUNNING, startTime=1333404636419, > completionTime=-1 appears to have been leaked > It's not clear to me why it's using a MonitoredTask in the first place. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5702) MasterSchemaChangeTracker.excludeRegionServerForSchemaChanges leaks a MonitoredTask per call
[ https://issues.apache.org/jira/browse/HBASE-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13244801#comment-13244801 ] Jean-Daniel Cryans commented on HBASE-5702: --- Looking at the code it seems it's leaking in other places... and that message shouldn't even be there in the first place because hbase.instant.schema.alter.enabled isn't enabled on this cluster. > MasterSchemaChangeTracker.excludeRegionServerForSchemaChanges leaks a > MonitoredTask per call > > > Key: HBASE-5702 > URL: https://issues.apache.org/jira/browse/HBASE-5702 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.0 >Reporter: Jean-Daniel Cryans > Fix For: 0.94.0, 0.96.0 > > > This bug is so easy to reproduce I'm wondering why it hasn't been reported > yet. Stop any number of region servers on a 0.94/6 cluster and you'll see in > the master interface one task per stopped region server saying the following: > |Processing schema change exclusion for region server = > sv4r27s44,62023,1333402175340|RUNNING (since 5sec ago)|No schema change in > progress. Skipping exclusion for server = sv4r27s44,62023,1333402175340 > (since 5sec ago)| > It's gonna stay there until the master cleans it: > bq. WARN org.apache.hadoop.hbase.monitoring.TaskMonitor: Status Processing > schema change exclusion for region server = sv4r27s44,62023,1333402175340: > status=No schema change in progress. Skipping exclusion for server = > sv4r27s44,62023,1333402175340, state=RUNNING, startTime=1333404636419, > completionTime=-1 appears to have been leaked > It's not clear to me why it's using a MonitoredTask in the first place. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
