[jira] [Commented] (HBASE-5136) Redundant MonitoredTask instances in case of distributed log splitting retry
[
https://issues.apache.org/jira/browse/HBASE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13484496#comment-13484496
]
Sergey Shelukhin commented on HBASE-5136:
-
Hi. Should this be ok to commit?
Redundant MonitoredTask instances in case of distributed log splitting retry
Key: HBASE-5136
URL: https://issues.apache.org/jira/browse/HBASE-5136
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Attachments: 5136-trunk.patch, 5136.txt
In case of log splitting retry, the following code would be executed multiple
times:
{code}
public long splitLogDistributed(final ListPath logDirs) throws
IOException {
MonitoredTask status = TaskMonitor.get().createStatus(
Doing distributed log split in + logDirs);
{code}
leading to multiple MonitoredTask instances.
User may get confused by multiple distributed log splitting entries for the
same region server on master UI
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5136) Redundant MonitoredTask instances in case of distributed log splitting retry
[
https://issues.apache.org/jira/browse/HBASE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13484518#comment-13484518
]
Ted Yu commented on HBASE-5136:
---
For Jimmy's patch:
{code}
+Integer logDirsHash = logDirs.hashCode();
{code}
What if there is hash collision between two log directories ?
Redundant MonitoredTask instances in case of distributed log splitting retry
Key: HBASE-5136
URL: https://issues.apache.org/jira/browse/HBASE-5136
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Attachments: 5136-trunk.patch, 5136.txt
In case of log splitting retry, the following code would be executed multiple
times:
{code}
public long splitLogDistributed(final ListPath logDirs) throws
IOException {
MonitoredTask status = TaskMonitor.get().createStatus(
Doing distributed log split in + logDirs);
{code}
leading to multiple MonitoredTask instances.
User may get confused by multiple distributed log splitting entries for the
same region server on master UI
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5136) Redundant MonitoredTask instances in case of distributed log splitting retry
[
https://issues.apache.org/jira/browse/HBASE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13484537#comment-13484537
]
Jimmy Xiang commented on HBASE-5136:
@Ted, were you referring to 5136.txt? That's your patch. My patch is
5136-trunk.patch, which is already committed in HBASE-6357.
I think we can close this one as a duplicate.
Redundant MonitoredTask instances in case of distributed log splitting retry
Key: HBASE-5136
URL: https://issues.apache.org/jira/browse/HBASE-5136
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Attachments: 5136-trunk.patch, 5136.txt
In case of log splitting retry, the following code would be executed multiple
times:
{code}
public long splitLogDistributed(final ListPath logDirs) throws
IOException {
MonitoredTask status = TaskMonitor.get().createStatus(
Doing distributed log split in + logDirs);
{code}
leading to multiple MonitoredTask instances.
User may get confused by multiple distributed log splitting entries for the
same region server on master UI
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5136) Redundant MonitoredTask instances in case of distributed log splitting retry
[
https://issues.apache.org/jira/browse/HBASE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13185688#comment-13185688
]
Jimmy Xiang commented on HBASE-5136:
Instead of reuse the same status object, can we abort the original one?
{code}
waitForSplittingCompletion(batch, status);
if (batch.done != batch.installed) {
batch.isDead = true;
tot_mgr_log_split_batch_err.incrementAndGet();
LOG.warn(error while splitting logs in + logDirs +
installed = + batch.installed + but only + batch.done + done);
= update the status message and abort it here
throw new IOException(error or interrupt while splitting logs in
+ logDirs + Task = + batch);
}
{code}
Redundant MonitoredTask instances in case of distributed log splitting retry
Key: HBASE-5136
URL: https://issues.apache.org/jira/browse/HBASE-5136
Project: HBase
Issue Type: Task
Reporter: Zhihong Yu
Assignee: Zhihong Yu
Attachments: 5136.txt
In case of log splitting retry, the following code would be executed multiple
times:
{code}
public long splitLogDistributed(final ListPath logDirs) throws
IOException {
MonitoredTask status = TaskMonitor.get().createStatus(
Doing distributed log split in + logDirs);
{code}
leading to multiple MonitoredTask instances.
User may get confused by multiple distributed log splitting entries for the
same region server on master UI
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5136) Redundant MonitoredTask instances in case of distributed log splitting retry
[
https://issues.apache.org/jira/browse/HBASE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13185693#comment-13185693
]
Zhihong Yu commented on HBASE-5136:
---
This is similar to Prakash's suggestion.
In some situation, several (aborted) tasks for the same server would be shown
on master UI at the same time since it takes 1 minute to clean up task display.
Redundant MonitoredTask instances in case of distributed log splitting retry
Key: HBASE-5136
URL: https://issues.apache.org/jira/browse/HBASE-5136
Project: HBase
Issue Type: Task
Reporter: Zhihong Yu
Assignee: Zhihong Yu
Attachments: 5136.txt
In case of log splitting retry, the following code would be executed multiple
times:
{code}
public long splitLogDistributed(final ListPath logDirs) throws
IOException {
MonitoredTask status = TaskMonitor.get().createStatus(
Doing distributed log split in + logDirs);
{code}
leading to multiple MonitoredTask instances.
User may get confused by multiple distributed log splitting entries for the
same region server on master UI
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5136) Redundant MonitoredTask instances in case of distributed log splitting retry
[
https://issues.apache.org/jira/browse/HBASE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184377#comment-13184377
]
Zhihong Yu commented on HBASE-5136:
---
Can someone review the patch ?
Thanks
Redundant MonitoredTask instances in case of distributed log splitting retry
Key: HBASE-5136
URL: https://issues.apache.org/jira/browse/HBASE-5136
Project: HBase
Issue Type: Task
Reporter: Zhihong Yu
Assignee: Zhihong Yu
Attachments: 5136.txt
In case of log splitting retry, the following code would be executed multiple
times:
{code}
public long splitLogDistributed(final ListPath logDirs) throws
IOException {
MonitoredTask status = TaskMonitor.get().createStatus(
Doing distributed log split in + logDirs);
{code}
leading to multiple MonitoredTask instances.
User may get confused by multiple distributed log splitting entries for the
same region server on master UI
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5136) Redundant MonitoredTask instances in case of distributed log splitting retry
[
https://issues.apache.org/jira/browse/HBASE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184507#comment-13184507
]
Zhihong Yu commented on HBASE-5136:
---
Then this JIRA depends on HBASE-5174.
Please comment on my proposal there.
The patch in this JIRA is just specialized version of my proposal for
HBASE-5174.
Redundant MonitoredTask instances in case of distributed log splitting retry
Key: HBASE-5136
URL: https://issues.apache.org/jira/browse/HBASE-5136
Project: HBase
Issue Type: Task
Reporter: Zhihong Yu
Assignee: Zhihong Yu
Attachments: 5136.txt
In case of log splitting retry, the following code would be executed multiple
times:
{code}
public long splitLogDistributed(final ListPath logDirs) throws
IOException {
MonitoredTask status = TaskMonitor.get().createStatus(
Doing distributed log split in + logDirs);
{code}
leading to multiple MonitoredTask instances.
User may get confused by multiple distributed log splitting entries for the
same region server on master UI
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5136) Redundant MonitoredTask instances in case of distributed log splitting retry
[
https://issues.apache.org/jira/browse/HBASE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13183090#comment-13183090
]
Zhihong Yu commented on HBASE-5136:
---
@Prakash:
Can you kindly provide your opinion ?
Redundant MonitoredTask instances in case of distributed log splitting retry
Key: HBASE-5136
URL: https://issues.apache.org/jira/browse/HBASE-5136
Project: HBase
Issue Type: Task
Reporter: Zhihong Yu
Assignee: Zhihong Yu
Attachments: 5136.txt
In case of log splitting retry, the following code would be executed multiple
times:
{code}
public long splitLogDistributed(final ListPath logDirs) throws
IOException {
MonitoredTask status = TaskMonitor.get().createStatus(
Doing distributed log split in + logDirs);
{code}
leading to multiple MonitoredTask instances.
User may get confused by multiple distributed log splitting entries for the
same region server on master UI
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5136) Redundant MonitoredTask instances in case of distributed log splitting retry
[
https://issues.apache.org/jira/browse/HBASE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181618#comment-13181618
]
Zhihong Yu commented on HBASE-5136:
---
Can SplitLogManager maintain mapping from hash of logDirs to MonitoredTask ?
If hash of logDirs isn't found in this mapping, we create MonitoredTask.
@Prakash:
What do you think ?
Redundant MonitoredTask instances in case of distributed log splitting retry
Key: HBASE-5136
URL: https://issues.apache.org/jira/browse/HBASE-5136
Project: HBase
Issue Type: Task
Reporter: Zhihong Yu
In case of log splitting retry, the following code would be executed multiple
times:
{code}
public long splitLogDistributed(final ListPath logDirs) throws
IOException {
MonitoredTask status = TaskMonitor.get().createStatus(
Doing distributed log split in + logDirs);
{code}
leading to multiple MonitoredTask instances.
User may get confused by multiple distributed log splitting entries for the
same region server on master UI
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5136) Redundant MonitoredTask instances in case of distributed log splitting retry
[
https://issues.apache.org/jira/browse/HBASE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181686#comment-13181686
]
Zhihong Yu commented on HBASE-5136:
---
Patch v1 is based on the assumption that the list of dead region servers for
distributed log splitting stays the same across retries.
Redundant MonitoredTask instances in case of distributed log splitting retry
Key: HBASE-5136
URL: https://issues.apache.org/jira/browse/HBASE-5136
Project: HBase
Issue Type: Task
Reporter: Zhihong Yu
Assignee: Zhihong Yu
Attachments: 5136.txt
In case of log splitting retry, the following code would be executed multiple
times:
{code}
public long splitLogDistributed(final ListPath logDirs) throws
IOException {
MonitoredTask status = TaskMonitor.get().createStatus(
Doing distributed log split in + logDirs);
{code}
leading to multiple MonitoredTask instances.
User may get confused by multiple distributed log splitting entries for the
same region server on master UI
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5136) Redundant MonitoredTask instances in case of distributed log splitting retry
[
https://issues.apache.org/jira/browse/HBASE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181721#comment-13181721
]
Hadoop QA commented on HBASE-5136:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12509744/5136.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
-1 javadoc. The javadoc tool appears to have generated -151 warning
messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
-1 findbugs. The patch appears to introduce 79 new Findbugs (version
1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestImportTsv
org.apache.hadoop.hbase.mapred.TestTableMapReduce
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/692//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/692//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/692//console
This message is automatically generated.
Redundant MonitoredTask instances in case of distributed log splitting retry
Key: HBASE-5136
URL: https://issues.apache.org/jira/browse/HBASE-5136
Project: HBase
Issue Type: Task
Reporter: Zhihong Yu
Assignee: Zhihong Yu
Attachments: 5136.txt
In case of log splitting retry, the following code would be executed multiple
times:
{code}
public long splitLogDistributed(final ListPath logDirs) throws
IOException {
MonitoredTask status = TaskMonitor.get().createStatus(
Doing distributed log split in + logDirs);
{code}
leading to multiple MonitoredTask instances.
User may get confused by multiple distributed log splitting entries for the
same region server on master UI
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5136) Redundant MonitoredTask instances in case of distributed log splitting retry
[
https://issues.apache.org/jira/browse/HBASE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181777#comment-13181777
]
Prakash Khemani commented on HBASE-5136:
it will be lot simpler to do status.cleanup() in the finally block in
splitLogDistributed()
Redundant MonitoredTask instances in case of distributed log splitting retry
Key: HBASE-5136
URL: https://issues.apache.org/jira/browse/HBASE-5136
Project: HBase
Issue Type: Task
Reporter: Zhihong Yu
Assignee: Zhihong Yu
Attachments: 5136.txt
In case of log splitting retry, the following code would be executed multiple
times:
{code}
public long splitLogDistributed(final ListPath logDirs) throws
IOException {
MonitoredTask status = TaskMonitor.get().createStatus(
Doing distributed log split in + logDirs);
{code}
leading to multiple MonitoredTask instances.
User may get confused by multiple distributed log splitting entries for the
same region server on master UI
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5136) Redundant MonitoredTask instances in case of distributed log splitting retry
[
https://issues.apache.org/jira/browse/HBASE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181779#comment-13181779
]
Zhihong Yu commented on HBASE-5136:
---
Thanks for the review, Prakash.
So the suggestion is to add finally block to splitLogDistributed().
Here is the implementation for cleanup() where stateTime would be set to
current time:
{code}
public void cleanup() {
if (state == State.RUNNING) {
setState(State.ABORTED);
}
}
{code}
TaskMonitor hardcodes expiration timeout as 60 seconds:
{code}
private static final long EXPIRATION_TIME = 60*1000;
{code}
The suggested approach would produce partially confusing result because there
might be multiple MonitoredTask instances for the same logDirs within 60 second
period.
Please correct me if my interpretation of the task monitoring is wrong.
Redundant MonitoredTask instances in case of distributed log splitting retry
Key: HBASE-5136
URL: https://issues.apache.org/jira/browse/HBASE-5136
Project: HBase
Issue Type: Task
Reporter: Zhihong Yu
Assignee: Zhihong Yu
Attachments: 5136.txt
In case of log splitting retry, the following code would be executed multiple
times:
{code}
public long splitLogDistributed(final ListPath logDirs) throws
IOException {
MonitoredTask status = TaskMonitor.get().createStatus(
Doing distributed log split in + logDirs);
{code}
leading to multiple MonitoredTask instances.
User may get confused by multiple distributed log splitting entries for the
same region server on master UI
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
