[jira] [Commented] (HBASE-5782) Not all the regions are getting assigned after the log splitting.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254646#comment-13254646 ] ramkrishna.s.vasudevan commented on HBASE-5782: --- Found the problem and working on soln. Infact this problem would have introduced data loss easily in simple RS going down scenario. Not all the regions are getting assigned after the log splitting. - Key: HBASE-5782 URL: https://issues.apache.org/jira/browse/HBASE-5782 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.94.0 Reporter: Gopinathan A Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.94.0 Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table. Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created. Same scenario was verified mulitple times in 0.92.1, no issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting assigned after the log splitting.
[
https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254666#comment-13254666
]
ramkrishna.s.vasudevan commented on HBASE-5782:
---
The problem here is
{code}
HLogKey logKey = makeKey(encodedRegionName, tableName, seqNum, now, clusterId);
doWrite(info, logKey, edits, htd);
{code}
We form a key here which is unique based on the seqNum. Now as we do
appendNoSync() there is no sync called in append flow. But there is always the
LogSyncer thread which periodically syncs.
All the edits are added to
{code}
synchronized void append(Entry e) throws IOException {
pendingWrites.add(e);
}
{code}
Here the pendingWrites are obtained and then it is flushed one by one.
{code}
pending = logSyncerThread.getPendingWrites();
// First flush all the pending writes to HDFS. Then
// issue the sync to HDFS. If sync is successful, then update
// syncedTillHere to indicate that transactions till this
// number has been successfully synced.
logSyncerThread.hlogFlush(tempWriter, pending);
{code}
Here there is no synchronization and hence the pendingwrites are not in
sequence. Because it is not in sequence the recovered.edits file is also
created with no sequence in some cases.
{code}
if (key.getLogSeqNum() = currentEditSeqId) {
LOG.debug( key is + key.getLogSeqNum()
+ current edit is +
currentEditSeqId);
skippedEdits++;
continue;
}
{code}
I just added the above log while replayRecoveredEdits and i can find
{code}
key is 175current edit is 176
key is 174current edit is 176
key is 186current edit is 187
key is 188current edit is 190
*** key is 189current edit is 190
key is 204current edit is 205
key is 206current edit is 207
key is 217current edit is 218
key is 216current edit is 218
key is 255current edit is 256
{code}
This leads to dataloss. Correct me if am wrong.
Not all the regions are getting assigned after the log splitting.
-
Key: HBASE-5782
URL: https://issues.apache.org/jira/browse/HBASE-5782
Project: HBase
Issue Type: Bug
Components: wal
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
Fix For: 0.94.0
Create a table with 1000 splits, after the region assignemnt, kill the
regionserver wich contains META table.
Here few regions are missing after the log splitting and region assigment.
HBCK report shows multiple region holes are got created.
Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting assigned after the log splitting.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254669#comment-13254669 ] ramkrishna.s.vasudevan commented on HBASE-5782: --- We can easily reproduce this problem. I think it applies to any edit that happens and if it is synced by the log syncer thread. So we must have some sync mechanism here to ensure this problem does not happen. Based on one of our requirements i was thinking that the log syncer thread should be configurable like either to use it or not use it. Please provide your suggestions. Not all the regions are getting assigned after the log splitting. - Key: HBASE-5782 URL: https://issues.apache.org/jira/browse/HBASE-5782 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.94.0 Reporter: Gopinathan A Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.94.0 Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table. Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created. Same scenario was verified mulitple times in 0.92.1, no issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting assigned after the log splitting.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254782#comment-13254782 ] ramkrishna.s.vasudevan commented on HBASE-5782: --- This patch passes TestHLog and TestHRegion. Not able to run the testsuite as at home. The idea here is the edits till the seq id already persisted can be safely ignored. Now from this issue we are sure that the edits can appear in diff seq not necessary in the ascending order. So all the edits less than the seq id from the store files are skipped and all other edits are consdiered. Please provide your comments. Not all the regions are getting assigned after the log splitting. - Key: HBASE-5782 URL: https://issues.apache.org/jira/browse/HBASE-5782 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.94.0 Reporter: Gopinathan A Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.94.0 Attachments: HBASE-5782.patch Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table. Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created. Same scenario was verified mulitple times in 0.92.1, no issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting assigned after the log splitting.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254805#comment-13254805 ] Jean-Daniel Cryans commented on HBASE-5782: --- bq. Based on one of our requirements i was thinking that the log syncer thread should be configurable like either to use it or not use it. As long as it's enabled by default I'm good with that. Not all the regions are getting assigned after the log splitting. - Key: HBASE-5782 URL: https://issues.apache.org/jira/browse/HBASE-5782 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.94.0 Reporter: Gopinathan A Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.94.0 Attachments: HBASE-5782.patch Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table. Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created. Same scenario was verified mulitple times in 0.92.1, no issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting assigned after the log splitting.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254810#comment-13254810 ] Hadoop QA commented on HBASE-5782: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522794/HBASE-5782.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1537//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1537//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1537//console This message is automatically generated. Not all the regions are getting assigned after the log splitting. - Key: HBASE-5782 URL: https://issues.apache.org/jira/browse/HBASE-5782 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.94.0 Reporter: Gopinathan A Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.94.0 Attachments: HBASE-5782.patch Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table. Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created. Same scenario was verified mulitple times in 0.92.1, no issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting assigned after the log splitting.
[
https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254892#comment-13254892
]
Jean-Daniel Cryans commented on HBASE-5782:
---
Looking more into this, I think HBASE-4487 is the real issue. I think I can
also prove that you can get the issue even with a disabled {{LogSyncer}}.
t1 does {{appendNoSync}} of k1
t1 does {{syncer}} up to {{getPendingWrites}}
t2 does {{appendNoSync}} of k2
t2 does {{syncer}} up to the end
In the log you'd see k2 then k1 so what's really wrong to me is this:
{code}
// Done in parallel for all writer threads, thanks to HDFS-895
ListEntry pending = logSyncerThread.getPendingWrites();
{code}
Although accessing pending writes is done in sync, you can apply them in
whichever way.
Furthermore, {{logSyncerThread.hlogFlush}} can also append entries to the WAL
in any order. For example, if both t1 and t2 have multiple edits they could end
up intermingled in the WAL simply by doing {{hlogFlush}} at the same time.
If {{LogSyncer}} was really an issue then {{HRegion.put}} and
{{HRegion.delete}} would need to be disabled too since they don't use
{{appendNoSync}} and just sync everything :)
How this used to work is that threads could only append to the WAL under the
{{updateLock}} and that was done at the same time as the {{doWrite}} which
creates the key. The call to sync could be done by any number of threads at the
same time.
If this is right, then we should pull back HBASE-4487 or add more locks.
We should also change this Jira's title once we get a better understanding of
the problem because it's not a region assignment problem.
Not all the regions are getting assigned after the log splitting.
-
Key: HBASE-5782
URL: https://issues.apache.org/jira/browse/HBASE-5782
Project: HBase
Issue Type: Bug
Components: wal
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
Fix For: 0.94.0
Attachments: HBASE-5782.patch
Create a table with 1000 splits, after the region assignemnt, kill the
regionserver wich contains META table.
Here few regions are missing after the log splitting and region assigment.
HBCK report shows multiple region holes are got created.
Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting assigned after the log splitting.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254917#comment-13254917 ] Zhihong Yu commented on HBASE-5782: --- bq. If this is right, then we should pull back HBASE-4487 or add more locks Adding more locks would take more time to validate / test. In order to get 0.94.0 out the door, can we pull back HBASE-4487 in 0.94 and pursue the locking approach in trunk (or separate branch) ? Not all the regions are getting assigned after the log splitting. - Key: HBASE-5782 URL: https://issues.apache.org/jira/browse/HBASE-5782 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.94.0 Reporter: Gopinathan A Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.94.0 Attachments: HBASE-5782.patch Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table. Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created. Same scenario was verified mulitple times in 0.92.1, no issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting assigned after the log splitting.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254925#comment-13254925 ] Lars Hofhansl commented on HBASE-5782: -- So the problem is that logSyncerThread keeps the edit in order but the syncer then applies the pending batches potentially out of order? We might just need a sync lock to prevent two threads syncing at the same time. This is different from the update lock, which also prevents writing any edits. Not all the regions are getting assigned after the log splitting. - Key: HBASE-5782 URL: https://issues.apache.org/jira/browse/HBASE-5782 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.94.0 Reporter: Gopinathan A Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.94.0 Attachments: HBASE-5782.patch Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table. Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created. Same scenario was verified mulitple times in 0.92.1, no issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting assigned after the log splitting.
[
https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254932#comment-13254932
]
Jean-Daniel Cryans commented on HBASE-5782:
---
bq. In order to get 0.94.0 out the door, can we pull back HBASE-4487 in 0.94
and pursue the locking approach in trunk (or separate branch) ?
+1, we might want to review HBASE-4282 too as it seems to do something similar
with the transaction ids.
bq. So the problem is that logSyncerThread keeps the edit in order but the
syncer then applies the pending batches potentially out of order?
It's sad that the pending edits live in {{LogSyncer}}, that thread is really
just suppose to call sync... but yeah so they are added there in order but then
it's a free for all in {{syncer}}. Adding a sync there could solve the issue
but in the end what it does is moving the log from appending (pre HBASE-4487)
to syncing plus a _ton_ of new complexity in HLog. I'd prefer solution that
doesn't add a lock to patch something that's broken.
Not all the regions are getting assigned after the log splitting.
-
Key: HBASE-5782
URL: https://issues.apache.org/jira/browse/HBASE-5782
Project: HBase
Issue Type: Bug
Components: wal
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
Fix For: 0.94.0
Attachments: HBASE-5782.patch
Create a table with 1000 splits, after the region assignemnt, kill the
regionserver wich contains META table.
Here few regions are missing after the log splitting and region assigment.
HBCK report shows multiple region holes are got created.
Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting assigned after the log splitting.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254959#comment-13254959 ] Zhihong Yu commented on HBASE-5782: --- bq. I'd prefer solution that doesn't add a lock to patch something that's broken. I agree. I suggest the following actions: 1. pull back HBASE-4487 in 0.94 and trunk 2. a) spend major effort on HBASE-5699 (multiple WALs per region server) 2. b) make HBASE-4487 semantically correct 2.a and 2.b can proceed in parallel. But I think HBASE-5699 is the long-term solution. Not all the regions are getting assigned after the log splitting. - Key: HBASE-5782 URL: https://issues.apache.org/jira/browse/HBASE-5782 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.94.0 Reporter: Gopinathan A Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.94.0 Attachments: HBASE-5782.patch Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table. Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created. Same scenario was verified mulitple times in 0.92.1, no issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting assigned after the log splitting.
[
https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254994#comment-13254994
]
Zhihong Yu commented on HBASE-5782:
---
HLog.appendNoSync() is used by
HRegion.{append|doMiniBatchPut|mutateRowsWithLocks}.
Those methods would be affected when HLog.appendNoSync() is removed.
Not all the regions are getting assigned after the log splitting.
-
Key: HBASE-5782
URL: https://issues.apache.org/jira/browse/HBASE-5782
Project: HBase
Issue Type: Bug
Components: wal
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
Fix For: 0.94.0
Attachments: HBASE-5782.patch
Create a table with 1000 splits, after the region assignemnt, kill the
regionserver wich contains META table.
Here few regions are missing after the log splitting and region assigment.
HBCK report shows multiple region holes are got created.
Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting assigned after the log splitting.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255034#comment-13255034 ] stack commented on HBASE-5782: -- I just committed a tool over on HBASE-5792. It tests WALs. If you pass the -verify flag, you'll see that even w/ just three threads, sequence ids are out of order. Could be useful verifying whatever fix we have here. Not all the regions are getting assigned after the log splitting. - Key: HBASE-5782 URL: https://issues.apache.org/jira/browse/HBASE-5782 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.94.0 Reporter: Gopinathan A Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.94.0 Attachments: HBASE-5782.patch Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table. Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created. Same scenario was verified mulitple times in 0.92.1, no issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting assigned after the log splitting.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255094#comment-13255094 ] Lars Hofhansl commented on HBASE-5782: -- The short term choices we have are: # revert HBASE-4528, HBASE-4487, and HBASE-5541 (are there others?) # Partially revert HBASE-2467 (or devise other ways to have strictly one thread flushing an HLog). Maybe Todd as the author of HBASE-2467 could chime in... Todd? Not all the regions are getting assigned after the log splitting. - Key: HBASE-5782 URL: https://issues.apache.org/jira/browse/HBASE-5782 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.94.0 Reporter: Gopinathan A Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.94.0 Attachments: HBASE-5782.patch Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table. Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created. Same scenario was verified mulitple times in 0.92.1, no issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting assigned after the log splitting.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255121#comment-13255121 ] Zhihong Yu commented on HBASE-5782: --- Interesting patch. How do we measure / compare the following combinations: 1. HLog.appendNoSync() used with one sync thread doing flush 2. HLog.appendNoSync() not used, multiple sync threads doing flush in parallel Not all the regions are getting assigned after the log splitting. - Key: HBASE-5782 URL: https://issues.apache.org/jira/browse/HBASE-5782 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.94.0 Reporter: Gopinathan A Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.94.0 Attachments: 5782.txt, HBASE-5782.patch Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table. Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created. Same scenario was verified mulitple times in 0.92.1, no issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting assigned after the log splitting.
[
https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255130#comment-13255130
]
Jean-Daniel Cryans commented on HBASE-5782:
---
bq. Don't hate me, just throwing this out there.
LOL
My current concerns:
- Performance, and I'm not the only one concerned.
- OOME, what happens if you enable deferred log flush and HDFS is slowing
down? That's actually an issue with HBASE-4487 since the {{LinkedList}} of
pending writes is unbounded whereas before simply appending to the file would
slow you down.
- This patch will make it that threads can sync data from threads that came in
later. You need to check {{txid = this.syncedTillHere}} again once you are
past the {{synchronized(syncLock)}} and return if it was taken care of while
you were blocking.
- Deadlocks, you never know when adding new locks :)
Not all the regions are getting assigned after the log splitting.
-
Key: HBASE-5782
URL: https://issues.apache.org/jira/browse/HBASE-5782
Project: HBase
Issue Type: Bug
Components: wal
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
Fix For: 0.94.0
Attachments: 5782.txt, HBASE-5782.patch
Create a table with 1000 splits, after the region assignemnt, kill the
regionserver wich contains META table.
Here few regions are missing after the log splitting and region assigment.
HBCK report shows multiple region holes are got created.
Same scenario was verified mulitple times in 0.92.1, no issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting assigned after the log splitting.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254385#comment-13254385 ] ramkrishna.s.vasudevan commented on HBASE-5782: --- @LArs Tomorrow i can try writing a test with similar scenario. bq.(Although that might be difficult) I think it should be easier. Hopefully tomorrow can come up with some initial analysis. Not all the regions are getting assigned after the log splitting. - Key: HBASE-5782 URL: https://issues.apache.org/jira/browse/HBASE-5782 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.94.0 Reporter: Gopinathan A Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.94.0 Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table. Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created. Same scenario was verified mulitple times in 0.92.1, no issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting assigned after the log splitting.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254433#comment-13254433 ] Lars Hofhansl commented on HBASE-5782: -- Thanks Ram! Not all the regions are getting assigned after the log splitting. - Key: HBASE-5782 URL: https://issues.apache.org/jira/browse/HBASE-5782 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.94.0 Reporter: Gopinathan A Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.94.0 Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table. Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created. Same scenario was verified mulitple times in 0.92.1, no issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting assigned after the log splitting.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253418#comment-13253418 ] ramkrishna.s.vasudevan commented on HBASE-5782: --- @Lars I was checking this issue. I think it has to be fixed before RC? What you say? Not sure of the root cause still. Not all the regions are getting assigned after the log splitting. - Key: HBASE-5782 URL: https://issues.apache.org/jira/browse/HBASE-5782 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.94.0 Reporter: Gopinathan A Priority: Critical Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table. Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created. Same scenario was verified mulitple times in 0.92.1, no issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting assigned after the log splitting.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253886#comment-13253886 ] Lars Hofhansl commented on HBASE-5782: -- Did you have HBASE-5778 enabled? Not all the regions are getting assigned after the log splitting. - Key: HBASE-5782 URL: https://issues.apache.org/jira/browse/HBASE-5782 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.94.0 Reporter: Gopinathan A Priority: Critical Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table. Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created. Same scenario was verified mulitple times in 0.92.1, no issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting assigned after the log splitting.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253978#comment-13253978 ] Gopinathan A commented on HBASE-5782: - Initially I also doubted the same. I am able to reproduce same issue after disabling the wal compression. Not all the regions are getting assigned after the log splitting. - Key: HBASE-5782 URL: https://issues.apache.org/jira/browse/HBASE-5782 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.94.0 Reporter: Gopinathan A Priority: Critical Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table. Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created. Same scenario was verified mulitple times in 0.92.1, no issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting assigned after the log splitting.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253990#comment-13253990 ] Lars Hofhansl commented on HBASE-5782: -- You tried multiple times in 0.92.1, and no problem? Any chance to capture this in a test? (Although that might be difficult) Not all the regions are getting assigned after the log splitting. - Key: HBASE-5782 URL: https://issues.apache.org/jira/browse/HBASE-5782 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.94.0 Reporter: Gopinathan A Priority: Critical Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table. Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created. Same scenario was verified mulitple times in 0.92.1, no issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Not all the regions are getting assigned after the log splitting.
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253996#comment-13253996 ] Lars Hofhansl commented on HBASE-5782: -- And is there anything interesting in the log files? Not all the regions are getting assigned after the log splitting. - Key: HBASE-5782 URL: https://issues.apache.org/jira/browse/HBASE-5782 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.94.0 Reporter: Gopinathan A Priority: Critical Create a table with 1000 splits, after the region assignemnt, kill the regionserver wich contains META table. Here few regions are missing after the log splitting and region assigment. HBCK report shows multiple region holes are got created. Same scenario was verified mulitple times in 0.92.1, no issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
