[jira] [Commented] (HBASE-5136) Redundant MonitoredTask instances in case of distributed log splitting retry
[ https://issues.apache.org/jira/browse/HBASE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13484496#comment-13484496 ] Sergey Shelukhin commented on HBASE-5136: - Hi. Should this be ok to commit? Redundant MonitoredTask instances in case of distributed log splitting retry Key: HBASE-5136 URL: https://issues.apache.org/jira/browse/HBASE-5136 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Attachments: 5136-trunk.patch, 5136.txt In case of log splitting retry, the following code would be executed multiple times: {code} public long splitLogDistributed(final ListPath logDirs) throws IOException { MonitoredTask status = TaskMonitor.get().createStatus( Doing distributed log split in + logDirs); {code} leading to multiple MonitoredTask instances. User may get confused by multiple distributed log splitting entries for the same region server on master UI -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5136) Redundant MonitoredTask instances in case of distributed log splitting retry
[ https://issues.apache.org/jira/browse/HBASE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13484518#comment-13484518 ] Ted Yu commented on HBASE-5136: --- For Jimmy's patch: {code} +Integer logDirsHash = logDirs.hashCode(); {code} What if there is hash collision between two log directories ? Redundant MonitoredTask instances in case of distributed log splitting retry Key: HBASE-5136 URL: https://issues.apache.org/jira/browse/HBASE-5136 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Attachments: 5136-trunk.patch, 5136.txt In case of log splitting retry, the following code would be executed multiple times: {code} public long splitLogDistributed(final ListPath logDirs) throws IOException { MonitoredTask status = TaskMonitor.get().createStatus( Doing distributed log split in + logDirs); {code} leading to multiple MonitoredTask instances. User may get confused by multiple distributed log splitting entries for the same region server on master UI -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5136) Redundant MonitoredTask instances in case of distributed log splitting retry
[ https://issues.apache.org/jira/browse/HBASE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13484537#comment-13484537 ] Jimmy Xiang commented on HBASE-5136: @Ted, were you referring to 5136.txt? That's your patch. My patch is 5136-trunk.patch, which is already committed in HBASE-6357. I think we can close this one as a duplicate. Redundant MonitoredTask instances in case of distributed log splitting retry Key: HBASE-5136 URL: https://issues.apache.org/jira/browse/HBASE-5136 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Attachments: 5136-trunk.patch, 5136.txt In case of log splitting retry, the following code would be executed multiple times: {code} public long splitLogDistributed(final ListPath logDirs) throws IOException { MonitoredTask status = TaskMonitor.get().createStatus( Doing distributed log split in + logDirs); {code} leading to multiple MonitoredTask instances. User may get confused by multiple distributed log splitting entries for the same region server on master UI -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5136) Redundant MonitoredTask instances in case of distributed log splitting retry
[ https://issues.apache.org/jira/browse/HBASE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13185688#comment-13185688 ] Jimmy Xiang commented on HBASE-5136: Instead of reuse the same status object, can we abort the original one? {code} waitForSplittingCompletion(batch, status); if (batch.done != batch.installed) { batch.isDead = true; tot_mgr_log_split_batch_err.incrementAndGet(); LOG.warn(error while splitting logs in + logDirs + installed = + batch.installed + but only + batch.done + done); = update the status message and abort it here throw new IOException(error or interrupt while splitting logs in + logDirs + Task = + batch); } {code} Redundant MonitoredTask instances in case of distributed log splitting retry Key: HBASE-5136 URL: https://issues.apache.org/jira/browse/HBASE-5136 Project: HBase Issue Type: Task Reporter: Zhihong Yu Assignee: Zhihong Yu Attachments: 5136.txt In case of log splitting retry, the following code would be executed multiple times: {code} public long splitLogDistributed(final ListPath logDirs) throws IOException { MonitoredTask status = TaskMonitor.get().createStatus( Doing distributed log split in + logDirs); {code} leading to multiple MonitoredTask instances. User may get confused by multiple distributed log splitting entries for the same region server on master UI -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5136) Redundant MonitoredTask instances in case of distributed log splitting retry
[ https://issues.apache.org/jira/browse/HBASE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13185693#comment-13185693 ] Zhihong Yu commented on HBASE-5136: --- This is similar to Prakash's suggestion. In some situation, several (aborted) tasks for the same server would be shown on master UI at the same time since it takes 1 minute to clean up task display. Redundant MonitoredTask instances in case of distributed log splitting retry Key: HBASE-5136 URL: https://issues.apache.org/jira/browse/HBASE-5136 Project: HBase Issue Type: Task Reporter: Zhihong Yu Assignee: Zhihong Yu Attachments: 5136.txt In case of log splitting retry, the following code would be executed multiple times: {code} public long splitLogDistributed(final ListPath logDirs) throws IOException { MonitoredTask status = TaskMonitor.get().createStatus( Doing distributed log split in + logDirs); {code} leading to multiple MonitoredTask instances. User may get confused by multiple distributed log splitting entries for the same region server on master UI -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5136) Redundant MonitoredTask instances in case of distributed log splitting retry
[ https://issues.apache.org/jira/browse/HBASE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184377#comment-13184377 ] Zhihong Yu commented on HBASE-5136: --- Can someone review the patch ? Thanks Redundant MonitoredTask instances in case of distributed log splitting retry Key: HBASE-5136 URL: https://issues.apache.org/jira/browse/HBASE-5136 Project: HBase Issue Type: Task Reporter: Zhihong Yu Assignee: Zhihong Yu Attachments: 5136.txt In case of log splitting retry, the following code would be executed multiple times: {code} public long splitLogDistributed(final ListPath logDirs) throws IOException { MonitoredTask status = TaskMonitor.get().createStatus( Doing distributed log split in + logDirs); {code} leading to multiple MonitoredTask instances. User may get confused by multiple distributed log splitting entries for the same region server on master UI -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5136) Redundant MonitoredTask instances in case of distributed log splitting retry
[ https://issues.apache.org/jira/browse/HBASE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184507#comment-13184507 ] Zhihong Yu commented on HBASE-5136: --- Then this JIRA depends on HBASE-5174. Please comment on my proposal there. The patch in this JIRA is just specialized version of my proposal for HBASE-5174. Redundant MonitoredTask instances in case of distributed log splitting retry Key: HBASE-5136 URL: https://issues.apache.org/jira/browse/HBASE-5136 Project: HBase Issue Type: Task Reporter: Zhihong Yu Assignee: Zhihong Yu Attachments: 5136.txt In case of log splitting retry, the following code would be executed multiple times: {code} public long splitLogDistributed(final ListPath logDirs) throws IOException { MonitoredTask status = TaskMonitor.get().createStatus( Doing distributed log split in + logDirs); {code} leading to multiple MonitoredTask instances. User may get confused by multiple distributed log splitting entries for the same region server on master UI -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5136) Redundant MonitoredTask instances in case of distributed log splitting retry
[ https://issues.apache.org/jira/browse/HBASE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13183090#comment-13183090 ] Zhihong Yu commented on HBASE-5136: --- @Prakash: Can you kindly provide your opinion ? Redundant MonitoredTask instances in case of distributed log splitting retry Key: HBASE-5136 URL: https://issues.apache.org/jira/browse/HBASE-5136 Project: HBase Issue Type: Task Reporter: Zhihong Yu Assignee: Zhihong Yu Attachments: 5136.txt In case of log splitting retry, the following code would be executed multiple times: {code} public long splitLogDistributed(final ListPath logDirs) throws IOException { MonitoredTask status = TaskMonitor.get().createStatus( Doing distributed log split in + logDirs); {code} leading to multiple MonitoredTask instances. User may get confused by multiple distributed log splitting entries for the same region server on master UI -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5136) Redundant MonitoredTask instances in case of distributed log splitting retry
[ https://issues.apache.org/jira/browse/HBASE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181618#comment-13181618 ] Zhihong Yu commented on HBASE-5136: --- Can SplitLogManager maintain mapping from hash of logDirs to MonitoredTask ? If hash of logDirs isn't found in this mapping, we create MonitoredTask. @Prakash: What do you think ? Redundant MonitoredTask instances in case of distributed log splitting retry Key: HBASE-5136 URL: https://issues.apache.org/jira/browse/HBASE-5136 Project: HBase Issue Type: Task Reporter: Zhihong Yu In case of log splitting retry, the following code would be executed multiple times: {code} public long splitLogDistributed(final ListPath logDirs) throws IOException { MonitoredTask status = TaskMonitor.get().createStatus( Doing distributed log split in + logDirs); {code} leading to multiple MonitoredTask instances. User may get confused by multiple distributed log splitting entries for the same region server on master UI -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5136) Redundant MonitoredTask instances in case of distributed log splitting retry
[ https://issues.apache.org/jira/browse/HBASE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181686#comment-13181686 ] Zhihong Yu commented on HBASE-5136: --- Patch v1 is based on the assumption that the list of dead region servers for distributed log splitting stays the same across retries. Redundant MonitoredTask instances in case of distributed log splitting retry Key: HBASE-5136 URL: https://issues.apache.org/jira/browse/HBASE-5136 Project: HBase Issue Type: Task Reporter: Zhihong Yu Assignee: Zhihong Yu Attachments: 5136.txt In case of log splitting retry, the following code would be executed multiple times: {code} public long splitLogDistributed(final ListPath logDirs) throws IOException { MonitoredTask status = TaskMonitor.get().createStatus( Doing distributed log split in + logDirs); {code} leading to multiple MonitoredTask instances. User may get confused by multiple distributed log splitting entries for the same region server on master UI -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5136) Redundant MonitoredTask instances in case of distributed log splitting retry
[ https://issues.apache.org/jira/browse/HBASE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181721#comment-13181721 ] Hadoop QA commented on HBASE-5136: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509744/5136.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated -151 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 79 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/692//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/692//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/692//console This message is automatically generated. Redundant MonitoredTask instances in case of distributed log splitting retry Key: HBASE-5136 URL: https://issues.apache.org/jira/browse/HBASE-5136 Project: HBase Issue Type: Task Reporter: Zhihong Yu Assignee: Zhihong Yu Attachments: 5136.txt In case of log splitting retry, the following code would be executed multiple times: {code} public long splitLogDistributed(final ListPath logDirs) throws IOException { MonitoredTask status = TaskMonitor.get().createStatus( Doing distributed log split in + logDirs); {code} leading to multiple MonitoredTask instances. User may get confused by multiple distributed log splitting entries for the same region server on master UI -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5136) Redundant MonitoredTask instances in case of distributed log splitting retry
[ https://issues.apache.org/jira/browse/HBASE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181777#comment-13181777 ] Prakash Khemani commented on HBASE-5136: it will be lot simpler to do status.cleanup() in the finally block in splitLogDistributed() Redundant MonitoredTask instances in case of distributed log splitting retry Key: HBASE-5136 URL: https://issues.apache.org/jira/browse/HBASE-5136 Project: HBase Issue Type: Task Reporter: Zhihong Yu Assignee: Zhihong Yu Attachments: 5136.txt In case of log splitting retry, the following code would be executed multiple times: {code} public long splitLogDistributed(final ListPath logDirs) throws IOException { MonitoredTask status = TaskMonitor.get().createStatus( Doing distributed log split in + logDirs); {code} leading to multiple MonitoredTask instances. User may get confused by multiple distributed log splitting entries for the same region server on master UI -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5136) Redundant MonitoredTask instances in case of distributed log splitting retry
[ https://issues.apache.org/jira/browse/HBASE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181779#comment-13181779 ] Zhihong Yu commented on HBASE-5136: --- Thanks for the review, Prakash. So the suggestion is to add finally block to splitLogDistributed(). Here is the implementation for cleanup() where stateTime would be set to current time: {code} public void cleanup() { if (state == State.RUNNING) { setState(State.ABORTED); } } {code} TaskMonitor hardcodes expiration timeout as 60 seconds: {code} private static final long EXPIRATION_TIME = 60*1000; {code} The suggested approach would produce partially confusing result because there might be multiple MonitoredTask instances for the same logDirs within 60 second period. Please correct me if my interpretation of the task monitoring is wrong. Redundant MonitoredTask instances in case of distributed log splitting retry Key: HBASE-5136 URL: https://issues.apache.org/jira/browse/HBASE-5136 Project: HBase Issue Type: Task Reporter: Zhihong Yu Assignee: Zhihong Yu Attachments: 5136.txt In case of log splitting retry, the following code would be executed multiple times: {code} public long splitLogDistributed(final ListPath logDirs) throws IOException { MonitoredTask status = TaskMonitor.get().createStatus( Doing distributed log split in + logDirs); {code} leading to multiple MonitoredTask instances. User may get confused by multiple distributed log splitting entries for the same region server on master UI -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira