[jira] [Commented] (HBASE-4335) Splits can create temporary holes in .META. that confuse clients and regionservers
[ https://issues.apache.org/jira/browse/HBASE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127122#comment-13127122 ] Jonathan Gray commented on HBASE-4335: -- @LarsH, in the future, please have your svn commit message be in the same format as the CHANGES.txt update (ie. HBASE- The title description (author [via committer]) Splits can create temporary holes in .META. that confuse clients and regionservers -- Key: HBASE-4335 URL: https://issues.apache.org/jira/browse/HBASE-4335 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.90.4 Reporter: Joe Pallas Assignee: Lars Hofhansl Priority: Critical Fix For: 0.92.0 Attachments: 4335-v2.txt, 4335-v3.txt, 4335-v4.txt, 4335-v5.txt, 4335.txt When a SplitTransaction is performed, three updates are done to .META.: 1. The parent region is marked as splitting (and hence offline) 2. The first daughter region is added (same start key as parent) 3. The second daughter region is added (split key is start key) (later, the original parent region is deleted, but that's not important to this discussion) Steps 2 and 3 are actually done concurrently by SplitTransaction.DaughterOpener threads. While the master is notified when a split is complete, the only visibility that clients have is whether the daughter regions have appeared in .META. If the second daughter is added to .META. first, then .META. will contain the (offline) parent region followed by the second daughter region. If the client looks up a key that is greater than (or equal to) the split, the client will find the second daughter region and use it. If the key is less than the split key, the client will find the parent region and see that it is offline, triggering a retry. If the first daughter is added to .META. before the second daughter, there is a window during which .META. has a hole: the first daughter effectively hides the parent region (same start key), but there is no entry for the second daughter. A region lookup will find the first daughter for all keys in the parent's range, but the first daughter does not include keys at or beyond the split key. See HBASE-4333 and HBASE-4334 for details on how this causes problems and suggestions for mitigating this in the client and regionserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4335) Splits can create temporary holes in .META. that confuse clients and regionservers
[ https://issues.apache.org/jira/browse/HBASE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127251#comment-13127251 ] Lars Hofhansl commented on HBASE-4335: -- Ted already pointed that out to me :) Splits can create temporary holes in .META. that confuse clients and regionservers -- Key: HBASE-4335 URL: https://issues.apache.org/jira/browse/HBASE-4335 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.90.4 Reporter: Joe Pallas Assignee: Lars Hofhansl Priority: Critical Fix For: 0.92.0 Attachments: 4335-v2.txt, 4335-v3.txt, 4335-v4.txt, 4335-v5.txt, 4335.txt When a SplitTransaction is performed, three updates are done to .META.: 1. The parent region is marked as splitting (and hence offline) 2. The first daughter region is added (same start key as parent) 3. The second daughter region is added (split key is start key) (later, the original parent region is deleted, but that's not important to this discussion) Steps 2 and 3 are actually done concurrently by SplitTransaction.DaughterOpener threads. While the master is notified when a split is complete, the only visibility that clients have is whether the daughter regions have appeared in .META. If the second daughter is added to .META. first, then .META. will contain the (offline) parent region followed by the second daughter region. If the client looks up a key that is greater than (or equal to) the split, the client will find the second daughter region and use it. If the key is less than the split key, the client will find the parent region and see that it is offline, triggering a retry. If the first daughter is added to .META. before the second daughter, there is a window during which .META. has a hole: the first daughter effectively hides the parent region (same start key), but there is no entry for the second daughter. A region lookup will find the first daughter for all keys in the parent's range, but the first daughter does not include keys at or beyond the split key. See HBASE-4333 and HBASE-4334 for details on how this causes problems and suggestions for mitigating this in the client and regionserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4335) Splits can create temporary holes in .META. that confuse clients and regionservers
[ https://issues.apache.org/jira/browse/HBASE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124728#comment-13124728 ] Lars Hofhansl commented on HBASE-4335: -- Results : Tests run: 1031, Failures: 0, Errors: 0, Skipped: 16 [INFO] [INFO] BUILD SUCCESSFUL [INFO] :) Splits can create temporary holes in .META. that confuse clients and regionservers -- Key: HBASE-4335 URL: https://issues.apache.org/jira/browse/HBASE-4335 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.90.4 Reporter: Joe Pallas Assignee: Lars Hofhansl Priority: Critical Fix For: 0.92.0 Attachments: 4335-v2.txt, 4335-v3.txt, 4335-v4.txt, 4335-v5.txt, 4335.txt When a SplitTransaction is performed, three updates are done to .META.: 1. The parent region is marked as splitting (and hence offline) 2. The first daughter region is added (same start key as parent) 3. The second daughter region is added (split key is start key) (later, the original parent region is deleted, but that's not important to this discussion) Steps 2 and 3 are actually done concurrently by SplitTransaction.DaughterOpener threads. While the master is notified when a split is complete, the only visibility that clients have is whether the daughter regions have appeared in .META. If the second daughter is added to .META. first, then .META. will contain the (offline) parent region followed by the second daughter region. If the client looks up a key that is greater than (or equal to) the split, the client will find the second daughter region and use it. If the key is less than the split key, the client will find the parent region and see that it is offline, triggering a retry. If the first daughter is added to .META. before the second daughter, there is a window during which .META. has a hole: the first daughter effectively hides the parent region (same start key), but there is no entry for the second daughter. A region lookup will find the first daughter for all keys in the parent's range, but the first daughter does not include keys at or beyond the split key. See HBASE-4333 and HBASE-4334 for details on how this causes problems and suggestions for mitigating this in the client and regionserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4335) Splits can create temporary holes in .META. that confuse clients and regionservers
[ https://issues.apache.org/jira/browse/HBASE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124734#comment-13124734 ] Ted Yu commented on HBASE-4335: --- Pass. +1. Splits can create temporary holes in .META. that confuse clients and regionservers -- Key: HBASE-4335 URL: https://issues.apache.org/jira/browse/HBASE-4335 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.90.4 Reporter: Joe Pallas Assignee: Lars Hofhansl Priority: Critical Fix For: 0.92.0 Attachments: 4335-v2.txt, 4335-v3.txt, 4335-v4.txt, 4335-v5.txt, 4335.txt When a SplitTransaction is performed, three updates are done to .META.: 1. The parent region is marked as splitting (and hence offline) 2. The first daughter region is added (same start key as parent) 3. The second daughter region is added (split key is start key) (later, the original parent region is deleted, but that's not important to this discussion) Steps 2 and 3 are actually done concurrently by SplitTransaction.DaughterOpener threads. While the master is notified when a split is complete, the only visibility that clients have is whether the daughter regions have appeared in .META. If the second daughter is added to .META. first, then .META. will contain the (offline) parent region followed by the second daughter region. If the client looks up a key that is greater than (or equal to) the split, the client will find the second daughter region and use it. If the key is less than the split key, the client will find the parent region and see that it is offline, triggering a retry. If the first daughter is added to .META. before the second daughter, there is a window during which .META. has a hole: the first daughter effectively hides the parent region (same start key), but there is no entry for the second daughter. A region lookup will find the first daughter for all keys in the parent's range, but the first daughter does not include keys at or beyond the split key. See HBASE-4333 and HBASE-4334 for details on how this causes problems and suggestions for mitigating this in the client and regionserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4335) Splits can create temporary holes in .META. that confuse clients and regionservers
[ https://issues.apache.org/jira/browse/HBASE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13125037#comment-13125037 ] Jonathan Hsieh commented on HBASE-4335: --- lgtm Splits can create temporary holes in .META. that confuse clients and regionservers -- Key: HBASE-4335 URL: https://issues.apache.org/jira/browse/HBASE-4335 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.90.4 Reporter: Joe Pallas Assignee: Lars Hofhansl Priority: Critical Fix For: 0.92.0 Attachments: 4335-v2.txt, 4335-v3.txt, 4335-v4.txt, 4335-v5.txt, 4335.txt When a SplitTransaction is performed, three updates are done to .META.: 1. The parent region is marked as splitting (and hence offline) 2. The first daughter region is added (same start key as parent) 3. The second daughter region is added (split key is start key) (later, the original parent region is deleted, but that's not important to this discussion) Steps 2 and 3 are actually done concurrently by SplitTransaction.DaughterOpener threads. While the master is notified when a split is complete, the only visibility that clients have is whether the daughter regions have appeared in .META. If the second daughter is added to .META. first, then .META. will contain the (offline) parent region followed by the second daughter region. If the client looks up a key that is greater than (or equal to) the split, the client will find the second daughter region and use it. If the key is less than the split key, the client will find the parent region and see that it is offline, triggering a retry. If the first daughter is added to .META. before the second daughter, there is a window during which .META. has a hole: the first daughter effectively hides the parent region (same start key), but there is no entry for the second daughter. A region lookup will find the first daughter for all keys in the parent's range, but the first daughter does not include keys at or beyond the split key. See HBASE-4333 and HBASE-4334 for details on how this causes problems and suggestions for mitigating this in the client and regionserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4335) Splits can create temporary holes in .META. that confuse clients and regionservers
[ https://issues.apache.org/jira/browse/HBASE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13125061#comment-13125061 ] ramkrishna.s.vasudevan commented on HBASE-4335: --- +1. Your test case comments looks good. May be you can also take a look at HBASE-4562 and HBASE-4563 as it is related to Splits. Splits can create temporary holes in .META. that confuse clients and regionservers -- Key: HBASE-4335 URL: https://issues.apache.org/jira/browse/HBASE-4335 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.90.4 Reporter: Joe Pallas Assignee: Lars Hofhansl Priority: Critical Fix For: 0.92.0 Attachments: 4335-v2.txt, 4335-v3.txt, 4335-v4.txt, 4335-v5.txt, 4335.txt When a SplitTransaction is performed, three updates are done to .META.: 1. The parent region is marked as splitting (and hence offline) 2. The first daughter region is added (same start key as parent) 3. The second daughter region is added (split key is start key) (later, the original parent region is deleted, but that's not important to this discussion) Steps 2 and 3 are actually done concurrently by SplitTransaction.DaughterOpener threads. While the master is notified when a split is complete, the only visibility that clients have is whether the daughter regions have appeared in .META. If the second daughter is added to .META. first, then .META. will contain the (offline) parent region followed by the second daughter region. If the client looks up a key that is greater than (or equal to) the split, the client will find the second daughter region and use it. If the key is less than the split key, the client will find the parent region and see that it is offline, triggering a retry. If the first daughter is added to .META. before the second daughter, there is a window during which .META. has a hole: the first daughter effectively hides the parent region (same start key), but there is no entry for the second daughter. A region lookup will find the first daughter for all keys in the parent's range, but the first daughter does not include keys at or beyond the split key. See HBASE-4333 and HBASE-4334 for details on how this causes problems and suggestions for mitigating this in the client and regionserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4335) Splits can create temporary holes in .META. that confuse clients and regionservers
[ https://issues.apache.org/jira/browse/HBASE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13125492#comment-13125492 ] Hudson commented on HBASE-4335: --- Integrated in HBase-0.92 #55 (See [https://builds.apache.org/job/HBase-0.92/55/]) Integrating HBASE-4335 larsh : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java Splits can create temporary holes in .META. that confuse clients and regionservers -- Key: HBASE-4335 URL: https://issues.apache.org/jira/browse/HBASE-4335 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.90.4 Reporter: Joe Pallas Assignee: Lars Hofhansl Priority: Critical Fix For: 0.92.0 Attachments: 4335-v2.txt, 4335-v3.txt, 4335-v4.txt, 4335-v5.txt, 4335.txt When a SplitTransaction is performed, three updates are done to .META.: 1. The parent region is marked as splitting (and hence offline) 2. The first daughter region is added (same start key as parent) 3. The second daughter region is added (split key is start key) (later, the original parent region is deleted, but that's not important to this discussion) Steps 2 and 3 are actually done concurrently by SplitTransaction.DaughterOpener threads. While the master is notified when a split is complete, the only visibility that clients have is whether the daughter regions have appeared in .META. If the second daughter is added to .META. first, then .META. will contain the (offline) parent region followed by the second daughter region. If the client looks up a key that is greater than (or equal to) the split, the client will find the second daughter region and use it. If the key is less than the split key, the client will find the parent region and see that it is offline, triggering a retry. If the first daughter is added to .META. before the second daughter, there is a window during which .META. has a hole: the first daughter effectively hides the parent region (same start key), but there is no entry for the second daughter. A region lookup will find the first daughter for all keys in the parent's range, but the first daughter does not include keys at or beyond the split key. See HBASE-4333 and HBASE-4334 for details on how this causes problems and suggestions for mitigating this in the client and regionserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4335) Splits can create temporary holes in .META. that confuse clients and regionservers
[ https://issues.apache.org/jira/browse/HBASE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13125539#comment-13125539 ] Hudson commented on HBASE-4335: --- Integrated in HBase-TRUNK #2318 (See [https://builds.apache.org/job/HBase-TRUNK/2318/]) Integrating HBASE-4335 larsh : Files : * /hbase/trunk/CHANGES.txt * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java Splits can create temporary holes in .META. that confuse clients and regionservers -- Key: HBASE-4335 URL: https://issues.apache.org/jira/browse/HBASE-4335 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.90.4 Reporter: Joe Pallas Assignee: Lars Hofhansl Priority: Critical Fix For: 0.92.0 Attachments: 4335-v2.txt, 4335-v3.txt, 4335-v4.txt, 4335-v5.txt, 4335.txt When a SplitTransaction is performed, three updates are done to .META.: 1. The parent region is marked as splitting (and hence offline) 2. The first daughter region is added (same start key as parent) 3. The second daughter region is added (split key is start key) (later, the original parent region is deleted, but that's not important to this discussion) Steps 2 and 3 are actually done concurrently by SplitTransaction.DaughterOpener threads. While the master is notified when a split is complete, the only visibility that clients have is whether the daughter regions have appeared in .META. If the second daughter is added to .META. first, then .META. will contain the (offline) parent region followed by the second daughter region. If the client looks up a key that is greater than (or equal to) the split, the client will find the second daughter region and use it. If the key is less than the split key, the client will find the parent region and see that it is offline, triggering a retry. If the first daughter is added to .META. before the second daughter, there is a window during which .META. has a hole: the first daughter effectively hides the parent region (same start key), but there is no entry for the second daughter. A region lookup will find the first daughter for all keys in the parent's range, but the first daughter does not include keys at or beyond the split key. See HBASE-4333 and HBASE-4334 for details on how this causes problems and suggestions for mitigating this in the client and regionserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4335) Splits can create temporary holes in .META. that confuse clients and regionservers
[ https://issues.apache.org/jira/browse/HBASE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13123882#comment-13123882 ] Ted Yu commented on HBASE-4335: --- @Lars: You would get +1's after declaring the passing of test suite. Splits can create temporary holes in .META. that confuse clients and regionservers -- Key: HBASE-4335 URL: https://issues.apache.org/jira/browse/HBASE-4335 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.90.4 Reporter: Joe Pallas Assignee: Lars Hofhansl Priority: Critical Fix For: 0.92.0 Attachments: 4335-v2.txt, 4335-v3.txt, 4335-v4.txt, 4335-v5.txt, 4335.txt When a SplitTransaction is performed, three updates are done to .META.: 1. The parent region is marked as splitting (and hence offline) 2. The first daughter region is added (same start key as parent) 3. The second daughter region is added (split key is start key) (later, the original parent region is deleted, but that's not important to this discussion) Steps 2 and 3 are actually done concurrently by SplitTransaction.DaughterOpener threads. While the master is notified when a split is complete, the only visibility that clients have is whether the daughter regions have appeared in .META. If the second daughter is added to .META. first, then .META. will contain the (offline) parent region followed by the second daughter region. If the client looks up a key that is greater than (or equal to) the split, the client will find the second daughter region and use it. If the key is less than the split key, the client will find the parent region and see that it is offline, triggering a retry. If the first daughter is added to .META. before the second daughter, there is a window during which .META. has a hole: the first daughter effectively hides the parent region (same start key), but there is no entry for the second daughter. A region lookup will find the first daughter for all keys in the parent's range, but the first daughter does not include keys at or beyond the split key. See HBASE-4333 and HBASE-4334 for details on how this causes problems and suggestions for mitigating this in the client and regionserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4335) Splits can create temporary holes in .META. that confuse clients and regionservers
[ https://issues.apache.org/jira/browse/HBASE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13122974#comment-13122974 ] Jonathan Hsieh commented on HBASE-4335: --- @Lars some nits / suggestions on v3. TestEndToEndSplitTransaction needs license. Maybe a more descriptive function names for phaseI, phaseII, phaseIII? Any reason for the (overly?) general Class... instead of just taking a single Class and checking for null when no exceptions expected? Or maybe just make 'test' return boolean and assertTrue/assertFalse? Splits can create temporary holes in .META. that confuse clients and regionservers -- Key: HBASE-4335 URL: https://issues.apache.org/jira/browse/HBASE-4335 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.90.4 Reporter: Joe Pallas Assignee: Lars Hofhansl Priority: Critical Fix For: 0.92.0 Attachments: 4335-v2.txt, 4335-v3.txt, 4335.txt When a SplitTransaction is performed, three updates are done to .META.: 1. The parent region is marked as splitting (and hence offline) 2. The first daughter region is added (same start key as parent) 3. The second daughter region is added (split key is start key) (later, the original parent region is deleted, but that's not important to this discussion) Steps 2 and 3 are actually done concurrently by SplitTransaction.DaughterOpener threads. While the master is notified when a split is complete, the only visibility that clients have is whether the daughter regions have appeared in .META. If the second daughter is added to .META. first, then .META. will contain the (offline) parent region followed by the second daughter region. If the client looks up a key that is greater than (or equal to) the split, the client will find the second daughter region and use it. If the key is less than the split key, the client will find the parent region and see that it is offline, triggering a retry. If the first daughter is added to .META. before the second daughter, there is a window during which .META. has a hole: the first daughter effectively hides the parent region (same start key), but there is no entry for the second daughter. A region lookup will find the first daughter for all keys in the parent's range, but the first daughter does not include keys at or beyond the split key. See HBASE-4333 and HBASE-4334 for details on how this causes problems and suggestions for mitigating this in the client and regionserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4335) Splits can create temporary holes in .META. that confuse clients and regionservers
[ https://issues.apache.org/jira/browse/HBASE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13123123#comment-13123123 ] Ted Yu commented on HBASE-4335: --- How about calling the first phase createDaughtersPhase, second openDaughtersPhase and the third phase transitionZKNodePhase ? Splits can create temporary holes in .META. that confuse clients and regionservers -- Key: HBASE-4335 URL: https://issues.apache.org/jira/browse/HBASE-4335 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.90.4 Reporter: Joe Pallas Assignee: Lars Hofhansl Priority: Critical Fix For: 0.92.0 Attachments: 4335-v2.txt, 4335-v3.txt, 4335.txt When a SplitTransaction is performed, three updates are done to .META.: 1. The parent region is marked as splitting (and hence offline) 2. The first daughter region is added (same start key as parent) 3. The second daughter region is added (split key is start key) (later, the original parent region is deleted, but that's not important to this discussion) Steps 2 and 3 are actually done concurrently by SplitTransaction.DaughterOpener threads. While the master is notified when a split is complete, the only visibility that clients have is whether the daughter regions have appeared in .META. If the second daughter is added to .META. first, then .META. will contain the (offline) parent region followed by the second daughter region. If the client looks up a key that is greater than (or equal to) the split, the client will find the second daughter region and use it. If the key is less than the split key, the client will find the parent region and see that it is offline, triggering a retry. If the first daughter is added to .META. before the second daughter, there is a window during which .META. has a hole: the first daughter effectively hides the parent region (same start key), but there is no entry for the second daughter. A region lookup will find the first daughter for all keys in the parent's range, but the first daughter does not include keys at or beyond the split key. See HBASE-4333 and HBASE-4334 for details on how this causes problems and suggestions for mitigating this in the client and regionserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4335) Splits can create temporary holes in .META. that confuse clients and regionservers
[ https://issues.apache.org/jira/browse/HBASE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13123342#comment-13123342 ] Lars Hofhansl commented on HBASE-4335: -- I like those names. Will do. @Jon. Initially i expected multiple different exceptions to thrown hence the general class approach. You're right here it does not make sense. Splits can create temporary holes in .META. that confuse clients and regionservers -- Key: HBASE-4335 URL: https://issues.apache.org/jira/browse/HBASE-4335 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.90.4 Reporter: Joe Pallas Assignee: Lars Hofhansl Priority: Critical Fix For: 0.92.0 Attachments: 4335-v2.txt, 4335-v3.txt, 4335.txt When a SplitTransaction is performed, three updates are done to .META.: 1. The parent region is marked as splitting (and hence offline) 2. The first daughter region is added (same start key as parent) 3. The second daughter region is added (split key is start key) (later, the original parent region is deleted, but that's not important to this discussion) Steps 2 and 3 are actually done concurrently by SplitTransaction.DaughterOpener threads. While the master is notified when a split is complete, the only visibility that clients have is whether the daughter regions have appeared in .META. If the second daughter is added to .META. first, then .META. will contain the (offline) parent region followed by the second daughter region. If the client looks up a key that is greater than (or equal to) the split, the client will find the second daughter region and use it. If the key is less than the split key, the client will find the parent region and see that it is offline, triggering a retry. If the first daughter is added to .META. before the second daughter, there is a window during which .META. has a hole: the first daughter effectively hides the parent region (same start key), but there is no entry for the second daughter. A region lookup will find the first daughter for all keys in the parent's range, but the first daughter does not include keys at or beyond the split key. See HBASE-4333 and HBASE-4334 for details on how this causes problems and suggestions for mitigating this in the client and regionserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4335) Splits can create temporary holes in .META. that confuse clients and regionservers
[ https://issues.apache.org/jira/browse/HBASE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118210#comment-13118210 ] Joe Pallas commented on HBASE-4335: --- @Lars, I think there is evidence that this is not the only way holes in meta might appear, so I would not mark HBASE-4333 and HBASE-4334 as duplicates. I think HBASE-4334 is the important one (the server should not assume the client is correct), and HBASE-4333 is more of an optimization (discover the problem at the client instead of the server). Splits can create temporary holes in .META. that confuse clients and regionservers -- Key: HBASE-4335 URL: https://issues.apache.org/jira/browse/HBASE-4335 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.90.4 Reporter: Joe Pallas Assignee: Lars Hofhansl Priority: Critical Fix For: 0.92.0 Attachments: 4335-v2.txt, 4335.txt When a SplitTransaction is performed, three updates are done to .META.: 1. The parent region is marked as splitting (and hence offline) 2. The first daughter region is added (same start key as parent) 3. The second daughter region is added (split key is start key) (later, the original parent region is deleted, but that's not important to this discussion) Steps 2 and 3 are actually done concurrently by SplitTransaction.DaughterOpener threads. While the master is notified when a split is complete, the only visibility that clients have is whether the daughter regions have appeared in .META. If the second daughter is added to .META. first, then .META. will contain the (offline) parent region followed by the second daughter region. If the client looks up a key that is greater than (or equal to) the split, the client will find the second daughter region and use it. If the key is less than the split key, the client will find the parent region and see that it is offline, triggering a retry. If the first daughter is added to .META. before the second daughter, there is a window during which .META. has a hole: the first daughter effectively hides the parent region (same start key), but there is no entry for the second daughter. A region lookup will find the first daughter for all keys in the parent's range, but the first daughter does not include keys at or beyond the split key. See HBASE-4333 and HBASE-4334 for details on how this causes problems and suggestions for mitigating this in the client and regionserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4335) Splits can create temporary holes in .META. that confuse clients and regionservers
[ https://issues.apache.org/jira/browse/HBASE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118237#comment-13118237 ] Lars Hofhansl commented on HBASE-4335: -- Agreed that HBASE-4334 is important. At least holes would not go silently. I'll work on that one too. Splits can create temporary holes in .META. that confuse clients and regionservers -- Key: HBASE-4335 URL: https://issues.apache.org/jira/browse/HBASE-4335 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.90.4 Reporter: Joe Pallas Assignee: Lars Hofhansl Priority: Critical Fix For: 0.92.0 Attachments: 4335-v2.txt, 4335.txt When a SplitTransaction is performed, three updates are done to .META.: 1. The parent region is marked as splitting (and hence offline) 2. The first daughter region is added (same start key as parent) 3. The second daughter region is added (split key is start key) (later, the original parent region is deleted, but that's not important to this discussion) Steps 2 and 3 are actually done concurrently by SplitTransaction.DaughterOpener threads. While the master is notified when a split is complete, the only visibility that clients have is whether the daughter regions have appeared in .META. If the second daughter is added to .META. first, then .META. will contain the (offline) parent region followed by the second daughter region. If the client looks up a key that is greater than (or equal to) the split, the client will find the second daughter region and use it. If the key is less than the split key, the client will find the parent region and see that it is offline, triggering a retry. If the first daughter is added to .META. before the second daughter, there is a window during which .META. has a hole: the first daughter effectively hides the parent region (same start key), but there is no entry for the second daughter. A region lookup will find the first daughter for all keys in the parent's range, but the first daughter does not include keys at or beyond the split key. See HBASE-4333 and HBASE-4334 for details on how this causes problems and suggestions for mitigating this in the client and regionserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4335) Splits can create temporary holes in .META. that confuse clients and regionservers
[ https://issues.apache.org/jira/browse/HBASE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117724#comment-13117724 ] Lars Hofhansl commented on HBASE-4335: -- I think when this is fixed we can mark HBASE-4333 and HBASE-4334 as dups. @Joe, do you agree? Or do you think there is still merit in fixing the specific issues mentioned there? Splits can create temporary holes in .META. that confuse clients and regionservers -- Key: HBASE-4335 URL: https://issues.apache.org/jira/browse/HBASE-4335 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.90.4 Reporter: Joe Pallas Assignee: Lars Hofhansl Priority: Critical Fix For: 0.92.0 Attachments: 4335-v2.txt, 4335.txt When a SplitTransaction is performed, three updates are done to .META.: 1. The parent region is marked as splitting (and hence offline) 2. The first daughter region is added (same start key as parent) 3. The second daughter region is added (split key is start key) (later, the original parent region is deleted, but that's not important to this discussion) Steps 2 and 3 are actually done concurrently by SplitTransaction.DaughterOpener threads. While the master is notified when a split is complete, the only visibility that clients have is whether the daughter regions have appeared in .META. If the second daughter is added to .META. first, then .META. will contain the (offline) parent region followed by the second daughter region. If the client looks up a key that is greater than (or equal to) the split, the client will find the second daughter region and use it. If the key is less than the split key, the client will find the parent region and see that it is offline, triggering a retry. If the first daughter is added to .META. before the second daughter, there is a window during which .META. has a hole: the first daughter effectively hides the parent region (same start key), but there is no entry for the second daughter. A region lookup will find the first daughter for all keys in the parent's range, but the first daughter does not include keys at or beyond the split key. See HBASE-4333 and HBASE-4334 for details on how this causes problems and suggestions for mitigating this in the client and regionserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4335) Splits can create temporary holes in .META. that confuse clients and regionservers
[ https://issues.apache.org/jira/browse/HBASE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117759#comment-13117759 ] Lars Hofhansl commented on HBASE-4335: -- I split up SplitTransaction.execute is three phases. I called them phaseI, phaseII, and phaseIII (please suggest better names). This is also nice because these can be separately documented now. All three are called from execute and the result is the same, but they can now be called separately (from classed in the same package, i.e. tests). This should allow me to add a test. Splits can create temporary holes in .META. that confuse clients and regionservers -- Key: HBASE-4335 URL: https://issues.apache.org/jira/browse/HBASE-4335 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.90.4 Reporter: Joe Pallas Assignee: Lars Hofhansl Priority: Critical Fix For: 0.92.0 Attachments: 4335-v2.txt, 4335.txt When a SplitTransaction is performed, three updates are done to .META.: 1. The parent region is marked as splitting (and hence offline) 2. The first daughter region is added (same start key as parent) 3. The second daughter region is added (split key is start key) (later, the original parent region is deleted, but that's not important to this discussion) Steps 2 and 3 are actually done concurrently by SplitTransaction.DaughterOpener threads. While the master is notified when a split is complete, the only visibility that clients have is whether the daughter regions have appeared in .META. If the second daughter is added to .META. first, then .META. will contain the (offline) parent region followed by the second daughter region. If the client looks up a key that is greater than (or equal to) the split, the client will find the second daughter region and use it. If the key is less than the split key, the client will find the parent region and see that it is offline, triggering a retry. If the first daughter is added to .META. before the second daughter, there is a window during which .META. has a hole: the first daughter effectively hides the parent region (same start key), but there is no entry for the second daughter. A region lookup will find the first daughter for all keys in the parent's range, but the first daughter does not include keys at or beyond the split key. See HBASE-4333 and HBASE-4334 for details on how this causes problems and suggestions for mitigating this in the client and regionserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4335) Splits can create temporary holes in .META. that confuse clients and regionservers
[
https://issues.apache.org/jira/browse/HBASE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114813#comment-13114813
]
stack commented on HBASE-4335:
--
Patch looks good to me. Shall I try make a test?
This is not you:
{code}
+if (stopped || stopping) {
+ // add 2nd daughter first (see HBASE-4335)
+ MetaEditor.addDaughter(server.getCatalogTracker(),
+ b.getRegionInfo(), null);
{code}
... but I'm trying to recall why this a good idea. We may be stopping because
cluster is going down and .META. usually is last to go down but thats not
guaranteed.
We could put in this patch as is and address this other item in another issue.
Splits can create temporary holes in .META. that confuse clients and
regionservers
--
Key: HBASE-4335
URL: https://issues.apache.org/jira/browse/HBASE-4335
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.90.4
Reporter: Joe Pallas
Assignee: Lars Hofhansl
Priority: Critical
Fix For: 0.92.0
Attachments: 4335-v2.txt, 4335.txt
When a SplitTransaction is performed, three updates are done to .META.:
1. The parent region is marked as splitting (and hence offline)
2. The first daughter region is added (same start key as parent)
3. The second daughter region is added (split key is start key)
(later, the original parent region is deleted, but that's not important to
this discussion)
Steps 2 and 3 are actually done concurrently by
SplitTransaction.DaughterOpener threads. While the master is notified when a
split is complete, the only visibility that clients have is whether the
daughter regions have appeared in .META.
If the second daughter is added to .META. first, then .META. will contain the
(offline) parent region followed by the second daughter region. If the
client looks up a key that is greater than (or equal to) the split, the
client will find the second daughter region and use it. If the key is less
than the split key, the client will find the parent region and see that it is
offline, triggering a retry.
If the first daughter is added to .META. before the second daughter, there is
a window during which .META. has a hole: the first daughter effectively hides
the parent region (same start key), but there is no entry for the second
daughter. A region lookup will find the first daughter for all keys in the
parent's range, but the first daughter does not include keys at or beyond the
split key.
See HBASE-4333 and HBASE-4334 for details on how this causes problems and
suggestions for mitigating this in the client and regionserver.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4335) Splits can create temporary holes in .META. that confuse clients and regionservers
[ https://issues.apache.org/jira/browse/HBASE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114829#comment-13114829 ] Ted Yu commented on HBASE-4335: --- Do all unit tests pass ? :-) Splits can create temporary holes in .META. that confuse clients and regionservers -- Key: HBASE-4335 URL: https://issues.apache.org/jira/browse/HBASE-4335 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.90.4 Reporter: Joe Pallas Assignee: Lars Hofhansl Priority: Critical Fix For: 0.92.0 Attachments: 4335-v2.txt, 4335.txt When a SplitTransaction is performed, three updates are done to .META.: 1. The parent region is marked as splitting (and hence offline) 2. The first daughter region is added (same start key as parent) 3. The second daughter region is added (split key is start key) (later, the original parent region is deleted, but that's not important to this discussion) Steps 2 and 3 are actually done concurrently by SplitTransaction.DaughterOpener threads. While the master is notified when a split is complete, the only visibility that clients have is whether the daughter regions have appeared in .META. If the second daughter is added to .META. first, then .META. will contain the (offline) parent region followed by the second daughter region. If the client looks up a key that is greater than (or equal to) the split, the client will find the second daughter region and use it. If the key is less than the split key, the client will find the parent region and see that it is offline, triggering a retry. If the first daughter is added to .META. before the second daughter, there is a window during which .META. has a hole: the first daughter effectively hides the parent region (same start key), but there is no entry for the second daughter. A region lookup will find the first daughter for all keys in the parent's range, but the first daughter does not include keys at or beyond the split key. See HBASE-4333 and HBASE-4334 for details on how this causes problems and suggestions for mitigating this in the client and regionserver. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4335) Splits can create temporary holes in .META. that confuse clients and regionservers
[ https://issues.apache.org/jira/browse/HBASE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114846#comment-13114846 ] Lars Hofhansl commented on HBASE-4335: -- @Stack... I was wondering about that too. It is checked in the beginning of execute and then was rechecked inside DautherOpener, but without any additional locks held (so not sure what additional guarantees we get.) Does it have to do with the order w.r.t. making zookeeper changes? When DaughterOpener is run we already made the zk changes, and that might be reason that now we have to go ahead and also write to .META. even if the RegionServer was stopped or is stopping... not sure... To be safe I moved it out and kept it. @Ted... Let me find out :) Splits can create temporary holes in .META. that confuse clients and regionservers -- Key: HBASE-4335 URL: https://issues.apache.org/jira/browse/HBASE-4335 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.90.4 Reporter: Joe Pallas Assignee: Lars Hofhansl Priority: Critical Fix For: 0.92.0 Attachments: 4335-v2.txt, 4335.txt When a SplitTransaction is performed, three updates are done to .META.: 1. The parent region is marked as splitting (and hence offline) 2. The first daughter region is added (same start key as parent) 3. The second daughter region is added (split key is start key) (later, the original parent region is deleted, but that's not important to this discussion) Steps 2 and 3 are actually done concurrently by SplitTransaction.DaughterOpener threads. While the master is notified when a split is complete, the only visibility that clients have is whether the daughter regions have appeared in .META. If the second daughter is added to .META. first, then .META. will contain the (offline) parent region followed by the second daughter region. If the client looks up a key that is greater than (or equal to) the split, the client will find the second daughter region and use it. If the key is less than the split key, the client will find the parent region and see that it is offline, triggering a retry. If the first daughter is added to .META. before the second daughter, there is a window during which .META. has a hole: the first daughter effectively hides the parent region (same start key), but there is no entry for the second daughter. A region lookup will find the first daughter for all keys in the parent's range, but the first daughter does not include keys at or beyond the split key. See HBASE-4333 and HBASE-4334 for details on how this causes problems and suggestions for mitigating this in the client and regionserver. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4335) Splits can create temporary holes in .META. that confuse clients and regionservers
[ https://issues.apache.org/jira/browse/HBASE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114964#comment-13114964 ] Lars Hofhansl commented on HBASE-4335: -- @Stack... Not sure how you can test the actual problem unless you either (1) instrument the code to introduce an artificial delay triggered by test code or (2) artificially create holes in meta. Neither are good options IMHO. Maybe a probabilistic test that just splits a table and calls get simultaneously. Splits can create temporary holes in .META. that confuse clients and regionservers -- Key: HBASE-4335 URL: https://issues.apache.org/jira/browse/HBASE-4335 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.90.4 Reporter: Joe Pallas Assignee: Lars Hofhansl Priority: Critical Fix For: 0.92.0 Attachments: 4335-v2.txt, 4335.txt When a SplitTransaction is performed, three updates are done to .META.: 1. The parent region is marked as splitting (and hence offline) 2. The first daughter region is added (same start key as parent) 3. The second daughter region is added (split key is start key) (later, the original parent region is deleted, but that's not important to this discussion) Steps 2 and 3 are actually done concurrently by SplitTransaction.DaughterOpener threads. While the master is notified when a split is complete, the only visibility that clients have is whether the daughter regions have appeared in .META. If the second daughter is added to .META. first, then .META. will contain the (offline) parent region followed by the second daughter region. If the client looks up a key that is greater than (or equal to) the split, the client will find the second daughter region and use it. If the key is less than the split key, the client will find the parent region and see that it is offline, triggering a retry. If the first daughter is added to .META. before the second daughter, there is a window during which .META. has a hole: the first daughter effectively hides the parent region (same start key), but there is no entry for the second daughter. A region lookup will find the first daughter for all keys in the parent's range, but the first daughter does not include keys at or beyond the split key. See HBASE-4333 and HBASE-4334 for details on how this causes problems and suggestions for mitigating this in the client and regionserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4335) Splits can create temporary holes in .META. that confuse clients and regionservers
[ https://issues.apache.org/jira/browse/HBASE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13115071#comment-13115071 ] Lars Hofhansl commented on HBASE-4335: -- @Joe... Understood. :) And you are right about mocking. In this case it also makes sense to break execute into three parts: (1) the setup before opening the region (2) opening the regions (3) the post open stuff. A test can then call 1 and 3 and mock 2 (for example by using the DaughterOpeners serially. @Stack... Did you want to work on the test or should I? Splits can create temporary holes in .META. that confuse clients and regionservers -- Key: HBASE-4335 URL: https://issues.apache.org/jira/browse/HBASE-4335 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.90.4 Reporter: Joe Pallas Assignee: Lars Hofhansl Priority: Critical Fix For: 0.92.0 Attachments: 4335-v2.txt, 4335.txt When a SplitTransaction is performed, three updates are done to .META.: 1. The parent region is marked as splitting (and hence offline) 2. The first daughter region is added (same start key as parent) 3. The second daughter region is added (split key is start key) (later, the original parent region is deleted, but that's not important to this discussion) Steps 2 and 3 are actually done concurrently by SplitTransaction.DaughterOpener threads. While the master is notified when a split is complete, the only visibility that clients have is whether the daughter regions have appeared in .META. If the second daughter is added to .META. first, then .META. will contain the (offline) parent region followed by the second daughter region. If the client looks up a key that is greater than (or equal to) the split, the client will find the second daughter region and use it. If the key is less than the split key, the client will find the parent region and see that it is offline, triggering a retry. If the first daughter is added to .META. before the second daughter, there is a window during which .META. has a hole: the first daughter effectively hides the parent region (same start key), but there is no entry for the second daughter. A region lookup will find the first daughter for all keys in the parent's range, but the first daughter does not include keys at or beyond the split key. See HBASE-4333 and HBASE-4334 for details on how this causes problems and suggestions for mitigating this in the client and regionserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4335) Splits can create temporary holes in .META. that confuse clients and regionservers
[ https://issues.apache.org/jira/browse/HBASE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13115075#comment-13115075 ] Lars Hofhansl commented on HBASE-4335: -- Ran all test. There are three failures locally (TestHTablePool, TestDistributedLogSplitting, TestAdmin), and they fail with or without my change. Splits can create temporary holes in .META. that confuse clients and regionservers -- Key: HBASE-4335 URL: https://issues.apache.org/jira/browse/HBASE-4335 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.90.4 Reporter: Joe Pallas Assignee: Lars Hofhansl Priority: Critical Fix For: 0.92.0 Attachments: 4335-v2.txt, 4335.txt When a SplitTransaction is performed, three updates are done to .META.: 1. The parent region is marked as splitting (and hence offline) 2. The first daughter region is added (same start key as parent) 3. The second daughter region is added (split key is start key) (later, the original parent region is deleted, but that's not important to this discussion) Steps 2 and 3 are actually done concurrently by SplitTransaction.DaughterOpener threads. While the master is notified when a split is complete, the only visibility that clients have is whether the daughter regions have appeared in .META. If the second daughter is added to .META. first, then .META. will contain the (offline) parent region followed by the second daughter region. If the client looks up a key that is greater than (or equal to) the split, the client will find the second daughter region and use it. If the key is less than the split key, the client will find the parent region and see that it is offline, triggering a retry. If the first daughter is added to .META. before the second daughter, there is a window during which .META. has a hole: the first daughter effectively hides the parent region (same start key), but there is no entry for the second daughter. A region lookup will find the first daughter for all keys in the parent's range, but the first daughter does not include keys at or beyond the split key. See HBASE-4333 and HBASE-4334 for details on how this causes problems and suggestions for mitigating this in the client and regionserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4335) Splits can create temporary holes in .META. that confuse clients and regionservers
[ https://issues.apache.org/jira/browse/HBASE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114083#comment-13114083 ] Lars Hofhansl commented on HBASE-4335: -- To restate the problem: If the first daughter is added to .META. first, any key lookup for a key = splitKey would incorrectly return the first daughter region. This seems like a legitimate, if rare, problem. Checking the end key would work for all point operation (put,get,delete,icv,cap,etc), and most already do that (except GET as you state in HBASE-4334). I don't think scans do that either, and not sure how it would work for scans. Hmm... Seems like it could work, and scans are serial and start the next region with the last value from the previous region, so if the startKey was checked we would catch this and could do a retry. I think it is better to avoid holes, though, overlap between active and offline-split regions seem fine. So what about splitting up the splitting process? The daughters are added to .META. in postOpenDeployTasks. That does not contain any long running operations. What if we removed that from the DaughterOpener threads and call it synchronous in the right order? (And also need to add the special stopped || stopping case). Splits can create temporary holes in .META. that confuse clients and regionservers -- Key: HBASE-4335 URL: https://issues.apache.org/jira/browse/HBASE-4335 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.90.4 Reporter: Joe Pallas Priority: Critical Fix For: 0.92.0 When a SplitTransaction is performed, three updates are done to .META.: 1. The parent region is marked as splitting (and hence offline) 2. The first daughter region is added (same start key as parent) 3. The second daughter region is added (split key is start key) (later, the original parent region is deleted, but that's not important to this discussion) Steps 2 and 3 are actually done concurrently by SplitTransaction.DaughterOpener threads. While the master is notified when a split is complete, the only visibility that clients have is whether the daughter regions have appeared in .META. If the second daughter is added to .META. first, then .META. will contain the (offline) parent region followed by the second daughter region. If the client looks up a key that is greater than (or equal to) the split, the client will find the second daughter region and use it. If the key is less than the split key, the client will find the parent region and see that it is offline, triggering a retry. If the first daughter is added to .META. before the second daughter, there is a window during which .META. has a hole: the first daughter effectively hides the parent region (same start key), but there is no entry for the second daughter. A region lookup will find the first daughter for all keys in the parent's range, but the first daughter does not include keys at or beyond the split key. See HBASE-4333 and HBASE-4334 for details on how this causes problems and suggestions for mitigating this in the client and regionserver. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4335) Splits can create temporary holes in .META. that confuse clients and regionservers
[ https://issues.apache.org/jira/browse/HBASE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114106#comment-13114106 ] Lars Hofhansl commented on HBASE-4335: -- Note: Obviously the change of order in which the threads are started/joined makes no difference. Just thought that might help with readability. Splits can create temporary holes in .META. that confuse clients and regionservers -- Key: HBASE-4335 URL: https://issues.apache.org/jira/browse/HBASE-4335 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.90.4 Reporter: Joe Pallas Assignee: Lars Hofhansl Priority: Critical Fix For: 0.92.0 Attachments: 4335.txt When a SplitTransaction is performed, three updates are done to .META.: 1. The parent region is marked as splitting (and hence offline) 2. The first daughter region is added (same start key as parent) 3. The second daughter region is added (split key is start key) (later, the original parent region is deleted, but that's not important to this discussion) Steps 2 and 3 are actually done concurrently by SplitTransaction.DaughterOpener threads. While the master is notified when a split is complete, the only visibility that clients have is whether the daughter regions have appeared in .META. If the second daughter is added to .META. first, then .META. will contain the (offline) parent region followed by the second daughter region. If the client looks up a key that is greater than (or equal to) the split, the client will find the second daughter region and use it. If the key is less than the split key, the client will find the parent region and see that it is offline, triggering a retry. If the first daughter is added to .META. before the second daughter, there is a window during which .META. has a hole: the first daughter effectively hides the parent region (same start key), but there is no entry for the second daughter. A region lookup will find the first daughter for all keys in the parent's range, but the first daughter does not include keys at or beyond the split key. See HBASE-4333 and HBASE-4334 for details on how this causes problems and suggestions for mitigating this in the client and regionserver. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4335) Splits can create temporary holes in .META. that confuse clients and regionservers
[ https://issues.apache.org/jira/browse/HBASE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114111#comment-13114111 ] Joe Pallas commented on HBASE-4335: --- That looks like the sort of solution I envisioned. I am not sure that changing the order of starting/joining the threads actually helps; I think it may just be a distraction for the reader. Splits can create temporary holes in .META. that confuse clients and regionservers -- Key: HBASE-4335 URL: https://issues.apache.org/jira/browse/HBASE-4335 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.90.4 Reporter: Joe Pallas Assignee: Lars Hofhansl Priority: Critical Fix For: 0.92.0 Attachments: 4335.txt When a SplitTransaction is performed, three updates are done to .META.: 1. The parent region is marked as splitting (and hence offline) 2. The first daughter region is added (same start key as parent) 3. The second daughter region is added (split key is start key) (later, the original parent region is deleted, but that's not important to this discussion) Steps 2 and 3 are actually done concurrently by SplitTransaction.DaughterOpener threads. While the master is notified when a split is complete, the only visibility that clients have is whether the daughter regions have appeared in .META. If the second daughter is added to .META. first, then .META. will contain the (offline) parent region followed by the second daughter region. If the client looks up a key that is greater than (or equal to) the split, the client will find the second daughter region and use it. If the key is less than the split key, the client will find the parent region and see that it is offline, triggering a retry. If the first daughter is added to .META. before the second daughter, there is a window during which .META. has a hole: the first daughter effectively hides the parent region (same start key), but there is no entry for the second daughter. A region lookup will find the first daughter for all keys in the parent's range, but the first daughter does not include keys at or beyond the split key. See HBASE-4333 and HBASE-4334 for details on how this causes problems and suggestions for mitigating this in the client and regionserver. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
