[jira] [Updated] (HBASE-4563) When split doing this.parent.close(false) occurs error,it'll cause the splited region cann't write read
[
https://issues.apache.org/jira/browse/HBASE-4563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
bluedavy updated HBASE-4563:
Attachment: HBASE-4563-test.report.txt
HBASE-4563.patch
patch test report.
When split doing this.parent.close(false) occurs error,it'll cause the
splited region cann't write read
-
Key: HBASE-4563
URL: https://issues.apache.org/jira/browse/HBASE-4563
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.90.4
Reporter: bluedavy
Priority: Blocker
Fix For: 0.90.5
Attachments: HBASE-4563-test.report.txt, HBASE-4563.patch
Follow below steps to replay the problem:
1. change the SplitTransaction.java as below,just like mock the hdfs error.
{code:title=SplitTransaction.java|borderStyle=solid}
ListStoreFile hstoreFilesToSplit = this.parent.close(false);
throw new IOException(some unexpected error in close store files);
{code}
2. update the regionserver code,restart;
3. create a table put some data to the table;
4. split the table;
5. scan the table,then it'll fail.
We can fix the bug just use below code:
{code:title=SplitTransaction.java|borderStyle=solid}
ListStoreFile hstoreFilesToSplit = null;
try{
hstoreFilesToSplit = this.parent.close(false);
}
catch(IOException e){
this.journal.add(JournalEntry.CLOSED_PARENT_REGION);
throw e;
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4562) When split doing offlineParentInMeta occurs error,it'll cause data loss
[
https://issues.apache.org/jira/browse/HBASE-4562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13123977#comment-13123977
]
bluedavy commented on HBASE-4562:
-
@Ted Yu
OK, I attached the patch again,and also provide the test suite results.
When split doing offlineParentInMeta occurs error,it'll cause data loss
---
Key: HBASE-4562
URL: https://issues.apache.org/jira/browse/HBASE-4562
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.90.4
Reporter: bluedavy
Priority: Blocker
Fix For: 0.90.5
Attachments: HBASE-4562-test.report.txt, HBASE-4562.patch
Follow below steps to replay the problem:
1. change the SplitTransaction.java as below,just like mock the timeout error.
{code:title=SplitTransaction.java|borderStyle=solid}
if (!testing) {
MetaEditor.offlineParentInMeta(server.getCatalogTracker(),
this.parent.getRegionInfo(), a.getRegionInfo(), b.getRegionInfo());
throw new IOException(some unexpected error in split);
}
{code}
2. update the regionserver code,restart;
3. create a table put some data to the table;
4. split the table;
5. kill the regionserver hosted the table;
6. wait some time after master ServerShutdownHandler.process execute,then
scan the table,u'll find the data wrote before lost.
We can fix the bug just use below code:
{code:title=SplitTransaction.java|borderStyle=solid}
this.journal.add(JournalEntry.PONR);
if (!testing) {
MetaEditor.offlineParentInMeta(server.getCatalogTracker(),
this.parent.getRegionInfo(), a.getRegionInfo(), b.getRegionInfo());
throw new IOException(some unexpected error in split);
}
{code}
{code:title=CompactSplitThread.java|borderStyle=solid}
if (st.rollback(this.server, this.server)) {
LOG.info(Successful rollback of failed split of +
parent.getRegionNameAsString());
}
else {
this.server.abort(Abort; we got an error after
point-of-no-return);
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4562) When split doing offlineParentInMeta occurs error,it'll cause data loss
[
https://issues.apache.org/jira/browse/HBASE-4562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
bluedavy updated HBASE-4562:
Attachment: HBASE-4562-test.report.txt
HBASE-4562.patch
When split doing offlineParentInMeta occurs error,it'll cause data loss
---
Key: HBASE-4562
URL: https://issues.apache.org/jira/browse/HBASE-4562
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.90.4
Reporter: bluedavy
Priority: Blocker
Fix For: 0.90.5
Attachments: HBASE-4562-test.report.txt, HBASE-4562.patch
Follow below steps to replay the problem:
1. change the SplitTransaction.java as below,just like mock the timeout error.
{code:title=SplitTransaction.java|borderStyle=solid}
if (!testing) {
MetaEditor.offlineParentInMeta(server.getCatalogTracker(),
this.parent.getRegionInfo(), a.getRegionInfo(), b.getRegionInfo());
throw new IOException(some unexpected error in split);
}
{code}
2. update the regionserver code,restart;
3. create a table put some data to the table;
4. split the table;
5. kill the regionserver hosted the table;
6. wait some time after master ServerShutdownHandler.process execute,then
scan the table,u'll find the data wrote before lost.
We can fix the bug just use below code:
{code:title=SplitTransaction.java|borderStyle=solid}
this.journal.add(JournalEntry.PONR);
if (!testing) {
MetaEditor.offlineParentInMeta(server.getCatalogTracker(),
this.parent.getRegionInfo(), a.getRegionInfo(), b.getRegionInfo());
throw new IOException(some unexpected error in split);
}
{code}
{code:title=CompactSplitThread.java|borderStyle=solid}
if (st.rollback(this.server, this.server)) {
LOG.info(Successful rollback of failed split of +
parent.getRegionNameAsString());
}
else {
this.server.abort(Abort; we got an error after
point-of-no-return);
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4563) When split doing this.parent.close(false) occurs error,it'll cause the splited region cann't write read
[
https://issues.apache.org/jira/browse/HBASE-4563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13123990#comment-13123990
]
ramkrishna.s.vasudevan commented on HBASE-4563:
---
Thanks for the patch.
{code}
+ catch(IOException e){
+ throw e;
+ }
{code}
Why should we catch and again throw exception?
When split doing this.parent.close(false) occurs error,it'll cause the
splited region cann't write read
-
Key: HBASE-4563
URL: https://issues.apache.org/jira/browse/HBASE-4563
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.90.4
Reporter: bluedavy
Priority: Blocker
Fix For: 0.90.5
Attachments: HBASE-4563-test.report.txt, HBASE-4563.patch
Follow below steps to replay the problem:
1. change the SplitTransaction.java as below,just like mock the hdfs error.
{code:title=SplitTransaction.java|borderStyle=solid}
ListStoreFile hstoreFilesToSplit = this.parent.close(false);
throw new IOException(some unexpected error in close store files);
{code}
2. update the regionserver code,restart;
3. create a table put some data to the table;
4. split the table;
5. scan the table,then it'll fail.
We can fix the bug just use below code:
{code:title=SplitTransaction.java|borderStyle=solid}
ListStoreFile hstoreFilesToSplit = null;
try{
hstoreFilesToSplit = this.parent.close(false);
}
catch(IOException e){
this.journal.add(JournalEntry.CLOSED_PARENT_REGION);
throw e;
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4563) When split doing this.parent.close(false) occurs error,it'll cause the splited region cann't write read
[
https://issues.apache.org/jira/browse/HBASE-4563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13123992#comment-13123992
]
ramkrishna.s.vasudevan commented on HBASE-4563:
---
Is it possible to write a testcase with the scenario you have told in the
description?
Minor comment.
{code}
+ }
+ catch(IOException e){
+ throw e;
+ }
+ finally{
{code}
Catch block, finally block should start in the same line like
+ } catch(IOException e){
+ throw e;
+ } finally{
When split doing this.parent.close(false) occurs error,it'll cause the
splited region cann't write read
-
Key: HBASE-4563
URL: https://issues.apache.org/jira/browse/HBASE-4563
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.90.4
Reporter: bluedavy
Priority: Blocker
Fix For: 0.90.5
Attachments: HBASE-4563-test.report.txt, HBASE-4563.patch
Follow below steps to replay the problem:
1. change the SplitTransaction.java as below,just like mock the hdfs error.
{code:title=SplitTransaction.java|borderStyle=solid}
ListStoreFile hstoreFilesToSplit = this.parent.close(false);
throw new IOException(some unexpected error in close store files);
{code}
2. update the regionserver code,restart;
3. create a table put some data to the table;
4. split the table;
5. scan the table,then it'll fail.
We can fix the bug just use below code:
{code:title=SplitTransaction.java|borderStyle=solid}
ListStoreFile hstoreFilesToSplit = null;
try{
hstoreFilesToSplit = this.parent.close(false);
}
catch(IOException e){
this.journal.add(JournalEntry.CLOSED_PARENT_REGION);
throw e;
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4562) When split doing offlineParentInMeta occurs error,it'll cause data loss
[
https://issues.apache.org/jira/browse/HBASE-4562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13123993#comment-13123993
]
ramkrishna.s.vasudevan commented on HBASE-4562:
---
Is it possible to write a testcase with the scenario you have told in the
description?
Minor comment:
{code}
+}
+else {
+ this.server.abort(Abort; we got an error after point-of-no-return);
+}
{code}
+} else {
+ this.server.abort(Abort; we got an error after point-of-no-return);
+}
else block should begin in the same line where if block ends.
When split doing offlineParentInMeta occurs error,it'll cause data loss
---
Key: HBASE-4562
URL: https://issues.apache.org/jira/browse/HBASE-4562
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.90.4
Reporter: bluedavy
Priority: Blocker
Fix For: 0.90.5
Attachments: HBASE-4562-test.report.txt, HBASE-4562.patch
Follow below steps to replay the problem:
1. change the SplitTransaction.java as below,just like mock the timeout error.
{code:title=SplitTransaction.java|borderStyle=solid}
if (!testing) {
MetaEditor.offlineParentInMeta(server.getCatalogTracker(),
this.parent.getRegionInfo(), a.getRegionInfo(), b.getRegionInfo());
throw new IOException(some unexpected error in split);
}
{code}
2. update the regionserver code,restart;
3. create a table put some data to the table;
4. split the table;
5. kill the regionserver hosted the table;
6. wait some time after master ServerShutdownHandler.process execute,then
scan the table,u'll find the data wrote before lost.
We can fix the bug just use below code:
{code:title=SplitTransaction.java|borderStyle=solid}
this.journal.add(JournalEntry.PONR);
if (!testing) {
MetaEditor.offlineParentInMeta(server.getCatalogTracker(),
this.parent.getRegionInfo(), a.getRegionInfo(), b.getRegionInfo());
throw new IOException(some unexpected error in split);
}
{code}
{code:title=CompactSplitThread.java|borderStyle=solid}
if (st.rollback(this.server, this.server)) {
LOG.info(Successful rollback of failed split of +
parent.getRegionNameAsString());
}
else {
this.server.abort(Abort; we got an error after
point-of-no-return);
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4480) Testing script to simplfy local testing
[ https://issues.apache.org/jira/browse/HBASE-4480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124027#comment-13124027 ] ramkrishna.s.vasudevan commented on HBASE-4480: --- @Jesse/Scott How should the file name passed for running the specified testcases? Could you provide a sample. Testing script to simplfy local testing --- Key: HBASE-4480 URL: https://issues.apache.org/jira/browse/HBASE-4480 Project: HBase Issue Type: Improvement Reporter: Jesse Yates Priority: Minor Labels: test Attachments: HBASE-4480.patch, HBASE-4480_v2.patch, runtest.sh, runtest2.sh As mentioned by http://search-hadoop.com/m/r2Ab624ES3e and http://search-hadoop.com/m/cZjDH1ykGIA it would be nice if we could have a script that would handle more of the finer points of running/checking our test suite. This script should: (1) Allow people to determine which tests are hanging/taking a long time to run (2) Allow rerunning of particular tests to make sure it wasn't an artifact of running the whole suite that caused the failure (3) Allow people to specify to run just unit tests or also integration tests (essentially wrapping calls to 'maven test' and 'maven verify'). This script should just be a convenience script - running tests directly from maven should not be impacted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4564) [book] book.xml,developer.xml - a few changes
[book] book.xml,developer.xml - a few changes - Key: HBASE-4564 URL: https://issues.apache.org/jira/browse/HBASE-4564 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor book.xml * Arch/Catalog - added comment that ROOT and META are actually HBase tables. That came up this week on the dist-list. The book already describes the structure but I'm making this even more explicit. developer.xml * adding entry for running multiple unit tests comma-listed (came up on dist-list this week) * adding link to ReviewBoard documentation in RB sub-section -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4564) [book] book.xml,developer.xml - a few changes
[ https://issues.apache.org/jira/browse/HBASE-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-4564: - Description: book.xml * Arch/Catalog - added comment that ROOT and META are actually HBase tables. That came up this week on the dist-list. The book already describes the structure but I'm making this even more explicit. * Added comment on explicit versioning (setting timestamp) that while this isn't preferred (also came up on dist-list this week). developer.xml * adding entry for running multiple unit tests comma-listed (came up on dist-list this week) * adding link to ReviewBoard documentation in RB sub-section was: book.xml * Arch/Catalog - added comment that ROOT and META are actually HBase tables. That came up this week on the dist-list. The book already describes the structure but I'm making this even more explicit. developer.xml * adding entry for running multiple unit tests comma-listed (came up on dist-list this week) * adding link to ReviewBoard documentation in RB sub-section [book] book.xml,developer.xml - a few changes - Key: HBASE-4564 URL: https://issues.apache.org/jira/browse/HBASE-4564 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: docbkx_HBASE_4564.patch book.xml * Arch/Catalog - added comment that ROOT and META are actually HBase tables. That came up this week on the dist-list. The book already describes the structure but I'm making this even more explicit. * Added comment on explicit versioning (setting timestamp) that while this isn't preferred (also came up on dist-list this week). developer.xml * adding entry for running multiple unit tests comma-listed (came up on dist-list this week) * adding link to ReviewBoard documentation in RB sub-section -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4564) [book] book.xml,developer.xml - a few changes
[ https://issues.apache.org/jira/browse/HBASE-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-4564: - Attachment: docbkx_HBASE_4564.patch [book] book.xml,developer.xml - a few changes - Key: HBASE-4564 URL: https://issues.apache.org/jira/browse/HBASE-4564 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: docbkx_HBASE_4564.patch book.xml * Arch/Catalog - added comment that ROOT and META are actually HBase tables. That came up this week on the dist-list. The book already describes the structure but I'm making this even more explicit. developer.xml * adding entry for running multiple unit tests comma-listed (came up on dist-list this week) * adding link to ReviewBoard documentation in RB sub-section -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4564) [book] book.xml,developer.xml - a few changes
[ https://issues.apache.org/jira/browse/HBASE-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-4564: - Description: book.xml * Arch/Catalog - added comment that ROOT and META are actually HBase tables. That came up this week on the dist-list. The book already describes the structure but I'm making this even more explicit. * Added comment on explicit versioning (setting timestamp) that this isn't preferred (also came up on dist-list this week). developer.xml * adding entry for running multiple unit tests comma-listed (came up on dist-list this week) * adding link to ReviewBoard documentation in RB sub-section was: book.xml * Arch/Catalog - added comment that ROOT and META are actually HBase tables. That came up this week on the dist-list. The book already describes the structure but I'm making this even more explicit. * Added comment on explicit versioning (setting timestamp) that while this isn't preferred (also came up on dist-list this week). developer.xml * adding entry for running multiple unit tests comma-listed (came up on dist-list this week) * adding link to ReviewBoard documentation in RB sub-section [book] book.xml,developer.xml - a few changes - Key: HBASE-4564 URL: https://issues.apache.org/jira/browse/HBASE-4564 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: docbkx_HBASE_4564.patch book.xml * Arch/Catalog - added comment that ROOT and META are actually HBase tables. That came up this week on the dist-list. The book already describes the structure but I'm making this even more explicit. * Added comment on explicit versioning (setting timestamp) that this isn't preferred (also came up on dist-list this week). developer.xml * adding entry for running multiple unit tests comma-listed (came up on dist-list this week) * adding link to ReviewBoard documentation in RB sub-section -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4564) [book] book.xml,developer.xml - a few changes
[ https://issues.apache.org/jira/browse/HBASE-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-4564: - Status: Patch Available (was: Open) [book] book.xml,developer.xml - a few changes - Key: HBASE-4564 URL: https://issues.apache.org/jira/browse/HBASE-4564 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: docbkx_HBASE_4564.patch book.xml * Arch/Catalog - added comment that ROOT and META are actually HBase tables. That came up this week on the dist-list. The book already describes the structure but I'm making this even more explicit. * Added comment on explicit versioning (setting timestamp) that this isn't preferred (also came up on dist-list this week). developer.xml * adding entry for running multiple unit tests comma-listed (came up on dist-list this week) * adding link to ReviewBoard documentation in RB sub-section -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4564) [book] book.xml,developer.xml - a few changes
[ https://issues.apache.org/jira/browse/HBASE-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-4564: - Resolution: Fixed Status: Resolved (was: Patch Available) [book] book.xml,developer.xml - a few changes - Key: HBASE-4564 URL: https://issues.apache.org/jira/browse/HBASE-4564 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: docbkx_HBASE_4564.patch book.xml * Arch/Catalog - added comment that ROOT and META are actually HBase tables. That came up this week on the dist-list. The book already describes the structure but I'm making this even more explicit. * Added comment on explicit versioning (setting timestamp) that this isn't preferred (also came up on dist-list this week). developer.xml * adding entry for running multiple unit tests comma-listed (came up on dist-list this week) * adding link to ReviewBoard documentation in RB sub-section -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4480) Testing script to simplfy local testing
[
https://issues.apache.org/jira/browse/HBASE-4480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124142#comment-13124142
]
Scott Kuehn commented on HBASE-4480:
Specify the filename as the option arg to -f, for example:
{code}
src/test/bin/test-util.sh -f tests.txt org.apache.hadoop.hbase.client.TestGet
{code}
If the file tests.txt contains the following:
{code}
org.apache.hadoop.hbase.TestCompare
org.apache.hadoop.hbase.client.replication.TestReplicationAdmin
{code}
The three tests will be executed: org.apache.hadoop.hbase.TestCompare,
org.apache.hadoop.hbase.client.replication.TestReplicationAdmin, and
org.apache.hadoop.hbase.client.TestGet
If you run:
{code}
src/test/bin/test-util.sh -f tests.txt
{code}
The two tests in tests.txt will be executed:
org.apache.hadoop.hbase.TestCompare,
org.apache.hadoop.hbase.client.replication.TestReplicationAdmin
Testing script to simplfy local testing
---
Key: HBASE-4480
URL: https://issues.apache.org/jira/browse/HBASE-4480
Project: HBase
Issue Type: Improvement
Reporter: Jesse Yates
Priority: Minor
Labels: test
Attachments: HBASE-4480.patch, HBASE-4480_v2.patch, runtest.sh,
runtest2.sh
As mentioned by http://search-hadoop.com/m/r2Ab624ES3e and
http://search-hadoop.com/m/cZjDH1ykGIA it would be nice if we could have a
script that would handle more of the finer points of running/checking our
test suite.
This script should:
(1) Allow people to determine which tests are hanging/taking a long time to
run
(2) Allow rerunning of particular tests to make sure it wasn't an artifact of
running the whole suite that caused the failure
(3) Allow people to specify to run just unit tests or also integration tests
(essentially wrapping calls to 'maven test' and 'maven verify').
This script should just be a convenience script - running tests directly from
maven should not be impacted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4480) Testing script to simplfy local testing
[ https://issues.apache.org/jira/browse/HBASE-4480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124157#comment-13124157 ] ramkrishna.s.vasudevan commented on HBASE-4480: --- Thanks Scott Is the package name needed for the testcases that needs to be run. Can you make this clear in the script? But when i try to run it is not reading from the file but it is running all the testcases. May be i need to check once again. Testing script to simplfy local testing --- Key: HBASE-4480 URL: https://issues.apache.org/jira/browse/HBASE-4480 Project: HBase Issue Type: Improvement Reporter: Jesse Yates Priority: Minor Labels: test Attachments: HBASE-4480.patch, HBASE-4480_v2.patch, runtest.sh, runtest2.sh As mentioned by http://search-hadoop.com/m/r2Ab624ES3e and http://search-hadoop.com/m/cZjDH1ykGIA it would be nice if we could have a script that would handle more of the finer points of running/checking our test suite. This script should: (1) Allow people to determine which tests are hanging/taking a long time to run (2) Allow rerunning of particular tests to make sure it wasn't an artifact of running the whole suite that caused the failure (3) Allow people to specify to run just unit tests or also integration tests (essentially wrapping calls to 'maven test' and 'maven verify'). This script should just be a convenience script - running tests directly from maven should not be impacted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog
[ https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124160#comment-13124160 ] jirapos...@reviews.apache.org commented on HBASE-4528: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2141/#review2481 --- This version looks good to me, with the exception of the need for a memstoreTS check when removing the memstore entries. I did a couple runs of TestLogRolling with this version and it passes. /src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java https://reviews.apache.org/r/2141/#comment5622 Agree with Ted that we should check memstoreTS against w.getWriteNumber() when removing, since this isn't a part of KeyValue.equals(). /src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java https://reviews.apache.org/r/2141/#comment5617 I think this line is still needed to ensure the WriteEntry is removed from the RWCC write queue in the case of a rollback. - Gary On 2011-10-08 07:50:25, Dhruba Borthakur wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2141/ bq. --- bq. bq. (Updated 2011-10-08 07:50:25) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. The changes the multiPut operation so that the sync to the wal occurs outside the rowlock. bq. bq. This enhancement is done only to HRegion.mut(Put[]) because this is the only method that gets invoked from an application. The HRegion.put(Put) is used only by unit tests and should possibly be deprecated. bq. bq. I have attached a unit test. I have not yet run all unit tests, but early feedback on this patch will be very helpful. bq. bq. bq. This addresses bug HBASE-4528. bq. https://issues.apache.org/jira/browse/HBASE-4528 bq. bq. bq. Diffs bq. - bq. bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1180314 bq./src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 1180314 bq. /src/main/java/org/apache/hadoop/hbase/regionserver/ReadWriteConsistencyControl.java 1180314 bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1180314 bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFlusher.java 1180314 bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestParallelPut.java PRE-CREATION bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java 1180314 bq. /src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 1180314 bq. bq. Diff: https://reviews.apache.org/r/2141/diff bq. bq. bq. Testing bq. --- bq. bq. Not yet run the full suite of unit tests. bq. bq. bq. Thanks, bq. bq. Dhruba bq. bq. The put operation can release the rowlock before sync-ing the Hlog -- Key: HBASE-4528 URL: https://issues.apache.org/jira/browse/HBASE-4528 Project: HBase Issue Type: Improvement Components: regionserver Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: appendNoSyncPut1.txt, appendNoSyncPut2.txt, appendNoSyncPut3.txt, appendNoSyncPut4.txt This allows for better throughput when there are hot rows. A single row update improves from 100 puts/sec/server to 5000 puts/sec/server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4480) Testing script to simplfy local testing
[
https://issues.apache.org/jira/browse/HBASE-4480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124173#comment-13124173
]
Scott Kuehn commented on HBASE-4480:
I've always used the full package name, which is how I typically invoke mvn
-Dtest='testname' from the command line. I'll update the docs to mention this.
Also, when I tried running with only the test class name ('TestCompare' instead
of 'org.apache.hadoop.hbase.TestCompare), I encountered the issue you
mentioned, where all tests were run. I'm not sure why this is the case, and
I'll take a closer look this evening.
Testing script to simplfy local testing
---
Key: HBASE-4480
URL: https://issues.apache.org/jira/browse/HBASE-4480
Project: HBase
Issue Type: Improvement
Reporter: Jesse Yates
Priority: Minor
Labels: test
Attachments: HBASE-4480.patch, HBASE-4480_v2.patch, runtest.sh,
runtest2.sh
As mentioned by http://search-hadoop.com/m/r2Ab624ES3e and
http://search-hadoop.com/m/cZjDH1ykGIA it would be nice if we could have a
script that would handle more of the finer points of running/checking our
test suite.
This script should:
(1) Allow people to determine which tests are hanging/taking a long time to
run
(2) Allow rerunning of particular tests to make sure it wasn't an artifact of
running the whole suite that caused the failure
(3) Allow people to specify to run just unit tests or also integration tests
(essentially wrapping calls to 'maven test' and 'maven verify').
This script should just be a convenience script - running tests directly from
maven should not be impacted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4388) Second start after migration from 90 to trunk crashes
[ https://issues.apache.org/jira/browse/HBASE-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124204#comment-13124204 ] Suraj Varma commented on HBASE-4388: @Ted Yu: This was a standalone install on cygwin. I had a 0.90.x hbase root dir which i was pointing to from 0.92.0-SNAPSHOT plus the patch. Procedure was to simply do start-hbase.sh on the 0.92 install, the master died each time with the NegativeArraySizeException. By explicitly catching this exception in HRegionInfo090x#readFields(DataInput) method rethrowing as IOException (similar to EOFException), I was able to get past this and complete the migration successfully. I'm attaching the master log from when it threw the error during migration. Second start after migration from 90 to trunk crashes - Key: HBASE-4388 URL: https://issues.apache.org/jira/browse/HBASE-4388 Project: HBase Issue Type: Bug Components: master, migration Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: stack Priority: Blocker Fix For: 0.92.0 Attachments: 4388-v2.txt, 4388-v3.txt, 4388.txt, meta.tgz I started a trunk cluster to upgrade from 90, inserted a ton of data, then did a clean shutdown. When I started again, I got the following exception: 11/09/13 12:29:09 INFO master.HMaster: Meta has HRI with HTDs. Updating meta now. 11/09/13 12:29:09 FATAL master.HMaster: Unhandled exception. Starting shutdown. java.lang.NegativeArraySizeException: -102 at org.apache.hadoop.hbase.util.Bytes.readByteArray(Bytes.java:147) at org.apache.hadoop.hbase.HTableDescriptor.readFields(HTableDescriptor.java:606) at org.apache.hadoop.hbase.migration.HRegionInfo090x.readFields(HRegionInfo090x.java:641) at org.apache.hadoop.hbase.util.Writables.getWritable(Writables.java:133) at org.apache.hadoop.hbase.util.Writables.getWritable(Writables.java:103) at org.apache.hadoop.hbase.util.Writables.getHRegionInfoForMigration(Writables.java:228) at org.apache.hadoop.hbase.catalog.MetaEditor.getHRegionInfoForMigration(MetaEditor.java:350) at org.apache.hadoop.hbase.catalog.MetaEditor$1.visit(MetaEditor.java:273) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:633) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:255) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:235) at org.apache.hadoop.hbase.catalog.MetaEditor.updateMetaWithNewRegionInfo(MetaEditor.java:284) at org.apache.hadoop.hbase.catalog.MetaEditor.migrateRootAndMeta(MetaEditor.java:298) at org.apache.hadoop.hbase.master.HMaster.updateMetaWithNewHRI(HMaster.java:529) at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:472) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4560) Script to find hanging test cases in build
[
https://issues.apache.org/jira/browse/HBASE-4560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124205#comment-13124205
]
Ted Yu commented on HBASE-4560:
---
Using the attached script, the following test was found to hang in build 2314:
{code}
Running org.apache.hadoop.hbase.client.TestShell
Running org.apache.hadoop.hbase.client.TestFromClientSide
Tests run: 46, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 128.669 sec
{code}
Script to find hanging test cases in build
--
Key: HBASE-4560
URL: https://issues.apache.org/jira/browse/HBASE-4560
Project: HBase
Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
Priority: Minor
Attachments: findHangingTest.sh
A script that parses the console output to get the hanging test names. This
will be very useful to know the hanging test case names when we see in some
builds all the test cases run but still the build shows failed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4388) Second start after migration from 90 to trunk crashes
[ https://issues.apache.org/jira/browse/HBASE-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suraj Varma updated HBASE-4388: --- Attachment: hbase-master-nase.log Attaching master log snippet captured during the NegativeArraySizeException run. Second start after migration from 90 to trunk crashes - Key: HBASE-4388 URL: https://issues.apache.org/jira/browse/HBASE-4388 Project: HBase Issue Type: Bug Components: master, migration Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: stack Priority: Blocker Fix For: 0.92.0 Attachments: 4388-v2.txt, 4388-v3.txt, 4388.txt, hbase-master-nase.log, meta.tgz I started a trunk cluster to upgrade from 90, inserted a ton of data, then did a clean shutdown. When I started again, I got the following exception: 11/09/13 12:29:09 INFO master.HMaster: Meta has HRI with HTDs. Updating meta now. 11/09/13 12:29:09 FATAL master.HMaster: Unhandled exception. Starting shutdown. java.lang.NegativeArraySizeException: -102 at org.apache.hadoop.hbase.util.Bytes.readByteArray(Bytes.java:147) at org.apache.hadoop.hbase.HTableDescriptor.readFields(HTableDescriptor.java:606) at org.apache.hadoop.hbase.migration.HRegionInfo090x.readFields(HRegionInfo090x.java:641) at org.apache.hadoop.hbase.util.Writables.getWritable(Writables.java:133) at org.apache.hadoop.hbase.util.Writables.getWritable(Writables.java:103) at org.apache.hadoop.hbase.util.Writables.getHRegionInfoForMigration(Writables.java:228) at org.apache.hadoop.hbase.catalog.MetaEditor.getHRegionInfoForMigration(MetaEditor.java:350) at org.apache.hadoop.hbase.catalog.MetaEditor$1.visit(MetaEditor.java:273) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:633) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:255) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:235) at org.apache.hadoop.hbase.catalog.MetaEditor.updateMetaWithNewRegionInfo(MetaEditor.java:284) at org.apache.hadoop.hbase.catalog.MetaEditor.migrateRootAndMeta(MetaEditor.java:298) at org.apache.hadoop.hbase.master.HMaster.updateMetaWithNewHRI(HMaster.java:529) at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:472) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4562) When split doing offlineParentInMeta occurs error,it'll cause data loss
[
https://issues.apache.org/jira/browse/HBASE-4562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124242#comment-13124242
]
Ted Yu commented on HBASE-4562:
---
Apache Jenkins build for 0.90 has 678 tests. The attached test report had fewer
test cases.
Also, patch for 0.92/TRUNK is needed since the fix would go to those places as
well.
When split doing offlineParentInMeta occurs error,it'll cause data loss
---
Key: HBASE-4562
URL: https://issues.apache.org/jira/browse/HBASE-4562
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.90.4
Reporter: bluedavy
Priority: Blocker
Fix For: 0.90.5
Attachments: HBASE-4562-test.report.txt, HBASE-4562.patch
Follow below steps to replay the problem:
1. change the SplitTransaction.java as below,just like mock the timeout error.
{code:title=SplitTransaction.java|borderStyle=solid}
if (!testing) {
MetaEditor.offlineParentInMeta(server.getCatalogTracker(),
this.parent.getRegionInfo(), a.getRegionInfo(), b.getRegionInfo());
throw new IOException(some unexpected error in split);
}
{code}
2. update the regionserver code,restart;
3. create a table put some data to the table;
4. split the table;
5. kill the regionserver hosted the table;
6. wait some time after master ServerShutdownHandler.process execute,then
scan the table,u'll find the data wrote before lost.
We can fix the bug just use below code:
{code:title=SplitTransaction.java|borderStyle=solid}
this.journal.add(JournalEntry.PONR);
if (!testing) {
MetaEditor.offlineParentInMeta(server.getCatalogTracker(),
this.parent.getRegionInfo(), a.getRegionInfo(), b.getRegionInfo());
throw new IOException(some unexpected error in split);
}
{code}
{code:title=CompactSplitThread.java|borderStyle=solid}
if (st.rollback(this.server, this.server)) {
LOG.info(Successful rollback of failed split of +
parent.getRegionNameAsString());
}
else {
this.server.abort(Abort; we got an error after
point-of-no-return);
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4565) Maven HBase build broken on cygwin with copynativelib.sh call.
Maven HBase build broken on cygwin with copynativelib.sh call.
--
Key: HBASE-4565
URL: https://issues.apache.org/jira/browse/HBASE-4565
Project: HBase
Issue Type: Bug
Components: build
Affects Versions: 0.92.0
Environment: cygwin (on xp and win7)
Reporter: Suraj Varma
Fix For: 0.92.0
This is broken in both 0.92 as well as trunk pom.xml
Here's a sample maven log snippet from trunk (from Mayuresh on user mailing
list)
[INFO] [antrun:run {execution: package}]
[INFO] Executing tasks
main:
[mkdir] Created dir:
D:\workspace\mkshirsa\hbase-trunk\target\hbase-0.93-SNAPSHOT\hbase-0.93-SNAPSHOT\lib\native\${build.platform}
[exec] ls: cannot access D:workspacemkshirsahbase-trunktarget/nativelib: No
such file or directory
[exec] tar (child): Cannot connect to D: resolve failed
[INFO]
[ERROR] BUILD ERROR
[INFO]
[INFO] An Ant BuildException has occured: exec returned: 3328
There are two issues:
1) The ant run task below doesn't resolve the windows file separator returned
by the project.build.directory - this causes the above resolve failed.
!-- Using Unix cp to preserve symlinks, using script to handle wildcards --
echo file=${project.build.directory}/copynativelibs.sh
if [ `ls ${project.build.directory}/nativelib | wc -l` -ne 0]; then
2) The tar argument value below also has a similar issue in that the path arg
doesn't resolve right.
!-- Using Unix tar to preserve symlinks --
exec executable=tar failonerror=yes
dir=${project.build.directory}/${project.artifactId}-${project.version}
arg value=czf/
arg
value=/cygdrive/c/workspaces/hbase-0.92-svn/target/${project.artifactId}-${project.version}.tar.gz/
arg value=./
/exec
In both cases, the fix would probably be to use a cross-platform way to handle
the directory locations.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4565) Maven HBase build broken on cygwin with copynativelib.sh call.
[
https://issues.apache.org/jira/browse/HBASE-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124284#comment-13124284
]
stack commented on HBASE-4565:
--
@Suraj Do you have a patch for us?
Maven HBase build broken on cygwin with copynativelib.sh call.
--
Key: HBASE-4565
URL: https://issues.apache.org/jira/browse/HBASE-4565
Project: HBase
Issue Type: Bug
Components: build
Affects Versions: 0.92.0
Environment: cygwin (on xp and win7)
Reporter: Suraj Varma
Labels: build, maven
Fix For: 0.92.0
This is broken in both 0.92 as well as trunk pom.xml
Here's a sample maven log snippet from trunk (from Mayuresh on user mailing
list)
[INFO] [antrun:run {execution: package}]
[INFO] Executing tasks
main:
[mkdir] Created dir:
D:\workspace\mkshirsa\hbase-trunk\target\hbase-0.93-SNAPSHOT\hbase-0.93-SNAPSHOT\lib\native\${build.platform}
[exec] ls: cannot access D:workspacemkshirsahbase-trunktarget/nativelib:
No such file or directory
[exec] tar (child): Cannot connect to D: resolve failed
[INFO]
[ERROR] BUILD ERROR
[INFO]
[INFO] An Ant BuildException has occured: exec returned: 3328
There are two issues:
1) The ant run task below doesn't resolve the windows file separator returned
by the project.build.directory - this causes the above resolve failed.
!-- Using Unix cp to preserve symlinks, using script to handle wildcards --
echo file=${project.build.directory}/copynativelibs.sh
if [ `ls ${project.build.directory}/nativelib | wc -l` -ne 0]; then
2) The tar argument value below also has a similar issue in that the path arg
doesn't resolve right.
!-- Using Unix tar to preserve symlinks --
exec executable=tar failonerror=yes
dir=${project.build.directory}/${project.artifactId}-${project.version}
arg value=czf/
arg
value=/cygdrive/c/workspaces/hbase-0.92-svn/target/${project.artifactId}-${project.version}.tar.gz/
arg value=./
/exec
In both cases, the fix would probably be to use a cross-platform way to
handle the directory locations.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4430) Disable TestSlabCache and TestSingleSizedCache temporarily to see if these are cause of build box failure though all tests pass
[
https://issues.apache.org/jira/browse/HBASE-4430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124289#comment-13124289
]
stack commented on HBASE-4430:
--
From Li up on IRC:
{code}
17:56 li something interesting:
17:57 li so
17:57 li under osx, the thing never ever fails now
17:57 li before 4482, we could get it to fail about 1/600 times
17:57 li under osx
17:57 li under linux, it fails every time
17:57 li for a different reason though
17:57 li heres a stacktrace
17:57 li http://pastebin.com/zsbGSy5P
17:58 li readbytes seems to be blocking
17:58 li i don't really know hy
17:58 li i'm not sure what readbytes is used for - i'm guessing its part of
mvn or junit?
17:58 li it would be funny if we somehow stumbled upon a race condition in
junit/mvn.
{code}
Disable TestSlabCache and TestSingleSizedCache temporarily to see if these
are cause of build box failure though all tests pass
---
Key: HBASE-4430
URL: https://issues.apache.org/jira/browse/HBASE-4430
Project: HBase
Issue Type: Task
Components: test
Reporter: stack
Assignee: Li Pi
Fix For: 0.92.0
Attachments: TestSlabCache.trace
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4430) Disable TestSlabCache and TestSingleSizedCache temporarily to see if these are cause of build box failure though all tests pass
[ https://issues.apache.org/jira/browse/HBASE-4430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124303#comment-13124303 ] Li Pi commented on HBASE-4430: -- According to todd, it looks something like mvn's stack trace. top didn't reveal any other java processes. I'll run this test again when I get home. I'm running ubuntu within vmware. On Mon, Oct 10, 2011 at 9:57 AM, stack (Commented) (JIRA) Disable TestSlabCache and TestSingleSizedCache temporarily to see if these are cause of build box failure though all tests pass --- Key: HBASE-4430 URL: https://issues.apache.org/jira/browse/HBASE-4430 Project: HBase Issue Type: Task Components: test Reporter: stack Assignee: Li Pi Fix For: 0.92.0 Attachments: TestSlabCache.trace -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4566) [book] Documentation on KeyValue, section on capacity planning
[book] Documentation on KeyValue, section on capacity planning -- Key: HBASE-4566 URL: https://issues.apache.org/jira/browse/HBASE-4566 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor book.xml * Added section in Arch/Regions on KeyValue and what it takes to store a KV. * Added reference in Schema Design to new KV section. ops_mgt.xml * Created start of Capacity Planning section, emphasizing that KV storage dominates the storage profile, with link to new KV section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4566) [book] Documentation on KeyValue, section on capacity planning
[ https://issues.apache.org/jira/browse/HBASE-4566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-4566: - Status: Patch Available (was: Open) [book] Documentation on KeyValue, section on capacity planning -- Key: HBASE-4566 URL: https://issues.apache.org/jira/browse/HBASE-4566 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: docbkx_HBASE_4566.patch book.xml * Added section in Arch/Regions on KeyValue and what it takes to store a KV. * Added reference in Schema Design to new KV section. ops_mgt.xml * Created start of Capacity Planning section, emphasizing that KV storage dominates the storage profile, with link to new KV section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4566) [book] Documentation on KeyValue, section on capacity planning
[ https://issues.apache.org/jira/browse/HBASE-4566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-4566: - Attachment: docbkx_HBASE_4566.patch [book] Documentation on KeyValue, section on capacity planning -- Key: HBASE-4566 URL: https://issues.apache.org/jira/browse/HBASE-4566 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: docbkx_HBASE_4566.patch book.xml * Added section in Arch/Regions on KeyValue and what it takes to store a KV. * Added reference in Schema Design to new KV section. ops_mgt.xml * Created start of Capacity Planning section, emphasizing that KV storage dominates the storage profile, with link to new KV section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4566) [book] Documentation on KeyValue, section on capacity planning
[ https://issues.apache.org/jira/browse/HBASE-4566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-4566: - Resolution: Fixed Status: Resolved (was: Patch Available) [book] Documentation on KeyValue, section on capacity planning -- Key: HBASE-4566 URL: https://issues.apache.org/jira/browse/HBASE-4566 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: docbkx_HBASE_4566.patch book.xml * Added section in Arch/Regions on KeyValue and what it takes to store a KV. * Added reference in Schema Design to new KV section. ops_mgt.xml * Created start of Capacity Planning section, emphasizing that KV storage dominates the storage profile, with link to new KV section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4556) Fix all incorrect uses of InternalScanner.next(...)
[
https://issues.apache.org/jira/browse/HBASE-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124349#comment-13124349
]
Jonathan Gray commented on HBASE-4556:
--
Why do we not see bugs because of this? Should the contract be how we actually
use it since it seems to work?
Fix all incorrect uses of InternalScanner.next(...)
---
Key: HBASE-4556
URL: https://issues.apache.org/jira/browse/HBASE-4556
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
There are cases all over the code where InternalScanner.next(...) is not used
correctly.
I see this a lot:
{code}
while(scanner.next(...)) {
}
{code}
The correct pattern is:
{code}
boolean more = false;
do {
more = scanner.next(...);
} while (more);
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog
[ https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124352#comment-13124352 ] jirapos...@reviews.apache.org commented on HBASE-4528: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2141/#review2488 --- /src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java https://reviews.apache.org/r/2141/#comment5629 Isn't the memstore snapshot taken here (in flusher.prepare())? And if so, what prevents new transactions from sneaking into the memstore between line 1144 (where you take the rwcc point) and this one? Maybe lines 1143-1144 be moved inside the writeLock().lock()? - Kannan On 2011-10-08 07:50:25, Dhruba Borthakur wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2141/ bq. --- bq. bq. (Updated 2011-10-08 07:50:25) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. The changes the multiPut operation so that the sync to the wal occurs outside the rowlock. bq. bq. This enhancement is done only to HRegion.mut(Put[]) because this is the only method that gets invoked from an application. The HRegion.put(Put) is used only by unit tests and should possibly be deprecated. bq. bq. I have attached a unit test. I have not yet run all unit tests, but early feedback on this patch will be very helpful. bq. bq. bq. This addresses bug HBASE-4528. bq. https://issues.apache.org/jira/browse/HBASE-4528 bq. bq. bq. Diffs bq. - bq. bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1180314 bq./src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 1180314 bq. /src/main/java/org/apache/hadoop/hbase/regionserver/ReadWriteConsistencyControl.java 1180314 bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1180314 bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFlusher.java 1180314 bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestParallelPut.java PRE-CREATION bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java 1180314 bq. /src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 1180314 bq. bq. Diff: https://reviews.apache.org/r/2141/diff bq. bq. bq. Testing bq. --- bq. bq. Not yet run the full suite of unit tests. bq. bq. bq. Thanks, bq. bq. Dhruba bq. bq. The put operation can release the rowlock before sync-ing the Hlog -- Key: HBASE-4528 URL: https://issues.apache.org/jira/browse/HBASE-4528 Project: HBase Issue Type: Improvement Components: regionserver Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: appendNoSyncPut1.txt, appendNoSyncPut2.txt, appendNoSyncPut3.txt, appendNoSyncPut4.txt This allows for better throughput when there are hot rows. A single row update improves from 100 puts/sec/server to 5000 puts/sec/server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2856) TestAcidGuarantee broken on trunk
[ https://issues.apache.org/jira/browse/HBASE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124369#comment-13124369 ] jirapos...@reviews.apache.org commented on HBASE-2856: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2224/ --- (Updated 2011-10-10 18:27:58.815672) Review request for Ted Yu, Michael Stack, Kannan Muthukkaruppan, and Karthik Ranganathan. Changes --- use variable length encoding to store the memstoreTS. Summary --- address the 2856 issues by writing the memstoreTS to the disk. version v11 of the patch. uploading it here for easier review process. This addresses bug HBASE-2856. https://issues.apache.org/jira/browse/HBASE-2856 Diffs (updated) - /src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 1179910 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1179910 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 1179910 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 1179910 /src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1179910 /src/main/java/org/apache/hadoop/hbase/regionserver/ReadWriteConsistencyControl.java 1179910 /src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 1179910 /src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1179910 /src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1179910 /src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 1179910 /src/test/java/org/apache/hadoop/hbase/TestAcidGuarantees.java 1179910 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 1179910 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 1179910 /src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 1179910 Diff: https://reviews.apache.org/r/2224/diff Testing --- mvn test Thanks, Amitanand TestAcidGuarantee broken on trunk -- Key: HBASE-2856 URL: https://issues.apache.org/jira/browse/HBASE-2856 Project: HBase Issue Type: Bug Affects Versions: 0.89.20100621 Reporter: ryan rawson Assignee: Amitanand Aiyer Priority: Blocker Fix For: 0.94.0 Attachments: 2856-v2.txt, 2856-v3.txt, 2856-v4.txt, 2856-v5.txt, acid.txt TestAcidGuarantee has a test whereby it attempts to read a number of columns from a row, and every so often the first column of N is different, when it should be the same. This is a bug deep inside the scanner whereby the first peek() of a row is done at time T then the rest of the read is done at T+1 after a flush, thus the memstoreTS data is lost, and previously 'uncommitted' data becomes committed and flushed to disk. One possible solution is to introduce the memstoreTS (or similarly equivalent value) to the HFile thus allowing us to preserve read consistency past flushes. Another solution involves fixing the scanners so that peek() is not destructive (and thus might return different things at different times alas). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4536) Allow CF to retain deleted rows
[ https://issues.apache.org/jira/browse/HBASE-4536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124372#comment-13124372 ] Lars Hofhansl commented on HBASE-4536: -- This issue is pretty resistant to all my attempts to solve it. :( Was toying with keeping track of the first key written to a file (we currently keep track of the last key). However, since the family delete hides all puts with a TS less than *or equal to* the delete TS, this won't help as the delete marker itself eventually would be the first key and it still might affect puts later in the file. Also KVs are sorted in reverse time order (except family delete markers), so I cannot just look at the put directly following the delete marker, because it'll be the newest put rather then the oldest. So there are three options I think: 1. Only allow the new flag set on CFs with TTL set. MIN_VERSIONS would not apply to deleted rows or delete marker rows (wouldn't know how long to keep family deletes in that case). (MAX)VERSIONS would still be enforced on all rows types except for family delete markers. 2. Translate family delete markers to column delete marker at (major) compaction time. 3. Change HFileWriterV* to keep track of the earliest put TS in a store and write it to the file metadata. Use that use expire delete marker that are older and hence can't affect any puts in the file. None of these are particularly attractive. #1 is limiting, #2 might get expensive if there're many columns, #3 would require the FileWriters to understand KVs. Allow CF to retain deleted rows --- Key: HBASE-4536 URL: https://issues.apache.org/jira/browse/HBASE-4536 Project: HBase Issue Type: Sub-task Components: regionserver Affects Versions: 0.92.0 Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Parent allows for a cluster to retain rows for a TTL or keep a minimum number of versions. However, if a client deletes a row all version older than the delete tomb stone will be remove at the next major compaction (and even at memstore flush - see HBASE-4241). There should be a way to retain those version to guard against software error. I see two options here: 1. Add a new flag HColumnDescriptor. Something like RETAIN_DELETED. 2. Folds this into the parent change. I.e. keep minimum-number-of-versions of versions even past the delete marker. #1 would allow for more flexibility. #2 comes somewhat naturally with parent (from a user viewpoint) Comments? Any other options? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4536) Allow CF to retain deleted rows
[ https://issues.apache.org/jira/browse/HBASE-4536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124376#comment-13124376 ] Lars Hofhansl commented on HBASE-4536: -- and 3a. Have Store.java keep track of the earliest put in internalFlushCache and compactStore and then append it to the file metadata. That way HFileWriterV* would not need to know about KVs. Allow CF to retain deleted rows --- Key: HBASE-4536 URL: https://issues.apache.org/jira/browse/HBASE-4536 Project: HBase Issue Type: Sub-task Components: regionserver Affects Versions: 0.92.0 Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Parent allows for a cluster to retain rows for a TTL or keep a minimum number of versions. However, if a client deletes a row all version older than the delete tomb stone will be remove at the next major compaction (and even at memstore flush - see HBASE-4241). There should be a way to retain those version to guard against software error. I see two options here: 1. Add a new flag HColumnDescriptor. Something like RETAIN_DELETED. 2. Folds this into the parent change. I.e. keep minimum-number-of-versions of versions even past the delete marker. #1 would allow for more flexibility. #2 comes somewhat naturally with parent (from a user viewpoint) Comments? Any other options? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4556) Fix all incorrect uses of InternalScanner.next(...)
[
https://issues.apache.org/jira/browse/HBASE-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124387#comment-13124387
]
Lars Hofhansl commented on HBASE-4556:
--
Many scanners only return false when there are no more current KVs to return,
i.e. the passed kv list is also empty. But does not follow the contract of
InternalScanner.
Fix all incorrect uses of InternalScanner.next(...)
---
Key: HBASE-4556
URL: https://issues.apache.org/jira/browse/HBASE-4556
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
There are cases all over the code where InternalScanner.next(...) is not used
correctly.
I see this a lot:
{code}
while(scanner.next(...)) {
}
{code}
The correct pattern is:
{code}
boolean more = false;
do {
more = scanner.next(...);
} while (more);
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4567) Divide by zero exception in RegionSplitter
Divide by zero exception in RegionSplitter -- Key: HBASE-4567 URL: https://issues.apache.org/jira/browse/HBASE-4567 Project: HBase Issue Type: Bug Reporter: Dave Revell Priority: Minor When doing a rolling split of existing regions using RegionSplitter, a divide by zero exception occurs. I'll be posting a junit test soon that reproduces the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4568) Make zk dump jsp response more quickly
Make zk dump jsp response more quickly -- Key: HBASE-4568 URL: https://issues.apache.org/jira/browse/HBASE-4568 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang For each zk dump, currently hbase will create a zk client instance every time. This is quite slow when any machines in the quorum is dead. Because it will connect to each machine in the zk quorum again. HMaster master = (HMaster)getServletContext().getAttribute(HMaster.MASTER); Configuration conf = master.getConfiguration(); HBaseAdmin hbadmin = new HBaseAdmin(conf); HConnection connection = hbadmin.getConnection(); ZooKeeperWatcher watcher = connection.getZooKeeperWatcher(); So we can simplify this: HMaster master = (HMaster)getServletContext().getAttribute(HMaster.MASTER); ZooKeeperWatcher watcher = master.getZooKeeperWatcher(); Also when hbase call getServerStats() for each machine in the zk quorum, it hard coded the default time out as 1 min. It would be nice to make this configurable and set it to a low time out. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4566) [book] Documentation on KeyValue, section on capacity planning
[ https://issues.apache.org/jira/browse/HBASE-4566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124448#comment-13124448 ] Hudson commented on HBASE-4566: --- Integrated in HBase-TRUNK #2315 (See [https://builds.apache.org/job/HBase-TRUNK/2315/]) HBASE-4566 book.xml,ops_mgt.xml - KeyValue documentation dmeil : Files : * /hbase/trunk/src/docbkx/book.xml * /hbase/trunk/src/docbkx/ops_mgt.xml [book] Documentation on KeyValue, section on capacity planning -- Key: HBASE-4566 URL: https://issues.apache.org/jira/browse/HBASE-4566 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: docbkx_HBASE_4566.patch book.xml * Added section in Arch/Regions on KeyValue and what it takes to store a KV. * Added reference in Schema Design to new KV section. ops_mgt.xml * Created start of Capacity Planning section, emphasizing that KV storage dominates the storage profile, with link to new KV section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4489) Better key splitting in RegionSplitter
[ https://issues.apache.org/jira/browse/HBASE-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Revell updated HBASE-4489: --- Attachment: HBASE-4489-branch0.90-v2.patch HBASE-4489-trunk-v2.patch New patches ending in -v2. These have new tests for RegionSplitter. Some weirdness: RegionSplitter.rollingSplit() seems to be broken, so it doesn't have any test cases in my code. I opened HBASE-4567 to focus on this. I also included a test case in TestRegionSplitter.java called reproduceDivByZeroFailure() that reproduces the problem. I think fixing this bug is outside the scope of this ticket. Better key splitting in RegionSplitter -- Key: HBASE-4489 URL: https://issues.apache.org/jira/browse/HBASE-4489 Project: HBase Issue Type: Improvement Affects Versions: 0.90.4 Reporter: Dave Revell Assignee: Dave Revell Attachments: HBASE-4489-branch0.90-v1.patch, HBASE-4489-branch0.90-v2.patch, HBASE-4489-trunk-v1.patch, HBASE-4489-trunk-v2.patch The RegionSplitter utility allows users to create a pre-split table from the command line or do a rolling split on an existing table. It supports pluggable split algorithms that implement the SplitAlgorithm interface. The only/default SplitAlgorithm is one that assumes keys fall in the range from ASCII string to ASCII string 7FFF. This is not a sane default, and seems useless to most users. Users are likely to be surprised by the fact that all the region splits occur in in the byte range of ASCII characters. A better default split algorithm would be one that evenly divides the space of all bytes, which is what this patch does. Making a table with five regions would split at \x33\x33..., \x66\x66, \x99\x99..., \xCC\xCC..., and \xFF\xFF. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4569) Simpler implementation for slab cache
[ https://issues.apache.org/jira/browse/HBASE-4569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HBASE-4569: --- Attachment: hbase-4569.txt here's a draft patch. doesn't pass the heapsize tests, and lacks some metrics stuff, but passes the actual cache tests. Interested to see if people think this is the right direction. Simpler implementation for slab cache - Key: HBASE-4569 URL: https://issues.apache.org/jira/browse/HBASE-4569 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.92.0 Reporter: Todd Lipcon Attachments: hbase-4569.txt I spent a little bit of time last night hacking on the slab cache implementation to make it a little simpler. The change is: - no longer has the composition of SlabCache containing a SingleSizeCache per slab size. SlabCache holds its own slabs - no longer use guava's map implementations to handle a size-bounded cache. Instead, manages its own LRU linked list - significantly less clever about synchronization. since this is an L2 cache, it should be less contended than the L1 cache, and I think we can afford to be dumb. - should have less memory usage since there's only one map entry per key instead of several. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4569) Simpler implementation for slab cache
Simpler implementation for slab cache - Key: HBASE-4569 URL: https://issues.apache.org/jira/browse/HBASE-4569 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.92.0 Reporter: Todd Lipcon Attachments: hbase-4569.txt I spent a little bit of time last night hacking on the slab cache implementation to make it a little simpler. The change is: - no longer has the composition of SlabCache containing a SingleSizeCache per slab size. SlabCache holds its own slabs - no longer use guava's map implementations to handle a size-bounded cache. Instead, manages its own LRU linked list - significantly less clever about synchronization. since this is an L2 cache, it should be less contended than the L1 cache, and I think we can afford to be dumb. - should have less memory usage since there's only one map entry per key instead of several. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4422) Move block cache parameters and references into single CacheConf class
[ https://issues.apache.org/jira/browse/HBASE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124455#comment-13124455 ] jirapos...@reviews.apache.org commented on HBASE-4422: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2089/ --- (Updated 2011-10-10 20:19:30.766633) Review request for hbase, Dhruba Borthakur, Michael Stack, and Li Pi. Changes --- After another review from Mikhail, this version brings back the local block cache instance variable and makes all of the static variables and methods private. Summary --- Creates a new CacheConfig class and moves almost everything block cache related into this single class. Adding new configuration params and booleans and such should be much better. All tests are NOT passing yet, still working on it, but wanted to have something up today. Basically code complete but broken :) This addresses bug HBASE-4422. https://issues.apache.org/jira/browse/HBASE-4422 Diffs (updated) - /src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java 1179008 /src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 1179008 /src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java 1179008 /src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 1179008 /src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java PRE-CREATION /src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java 1179008 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 1179008 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java 1179008 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1179008 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 1179008 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java 1179008 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 1179008 /src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1179008 /src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 1179008 /src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java 1179008 /src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java 1179008 /src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java 1179008 /src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 1179008 /src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1179008 /src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1179008 /src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1179008 /src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1179008 /src/main/java/org/apache/hadoop/hbase/util/BloomFilterFactory.java 1179008 /src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java 1179008 /src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java 1179008 /src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java 1179008 /src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java 1179008 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 1179008 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java 1179008 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java 1179008 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java 1179008 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java 1179008 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java 1179008 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 1179008 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java 1179008 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java 1179008 /src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java 1179008 /src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java 1179008 /src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 1179008 /src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java 1179008 /src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java 1179008 /src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java 1179008 /src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 1179008 /src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java 1179008 /src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 1179008 /src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java 1179008
[jira] [Commented] (HBASE-4422) Move block cache parameters and references into single CacheConf class
[ https://issues.apache.org/jira/browse/HBASE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124474#comment-13124474 ] jirapos...@reviews.apache.org commented on HBASE-4422: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2089/ --- (Updated 2011-10-10 21:08:15.751510) Review request for hbase, Dhruba Borthakur, Michael Stack, and Li Pi. Changes --- Fix to TestCacheOnWrite (not sure how this got through to now?) Summary --- Creates a new CacheConfig class and moves almost everything block cache related into this single class. Adding new configuration params and booleans and such should be much better. All tests are NOT passing yet, still working on it, but wanted to have something up today. Basically code complete but broken :) This addresses bug HBASE-4422. https://issues.apache.org/jira/browse/HBASE-4422 Diffs (updated) - /src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java 1179008 /src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 1179008 /src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java 1179008 /src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 1179008 /src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java PRE-CREATION /src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java 1179008 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 1179008 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java 1179008 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1179008 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 1179008 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java 1179008 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 1179008 /src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1179008 /src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 1179008 /src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java 1179008 /src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java 1179008 /src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java 1179008 /src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 1179008 /src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1179008 /src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1179008 /src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1179008 /src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1179008 /src/main/java/org/apache/hadoop/hbase/util/BloomFilterFactory.java 1179008 /src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java 1179008 /src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java 1179008 /src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java 1179008 /src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java 1179008 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 1179008 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java 1179008 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java 1179008 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java 1179008 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java 1179008 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java 1179008 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 1179008 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java 1179008 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java 1179008 /src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java 1179008 /src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java 1179008 /src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 1179008 /src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java 1179008 /src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java 1179008 /src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java 1179008 /src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 1179008 /src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java 1179008 /src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 1179008 /src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java 1179008 Diff: https://reviews.apache.org/r/2089/diff Testing --- Still working through some tests
[jira] [Work started] (HBASE-4486) Improve Javadoc for HTableDescriptor
[ https://issues.apache.org/jira/browse/HBASE-4486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-4486 started by Akash Ashok. Improve Javadoc for HTableDescriptor Key: HBASE-4486 URL: https://issues.apache.org/jira/browse/HBASE-4486 Project: HBase Issue Type: Improvement Components: client, documentation Reporter: Akash Ashok Assignee: Akash Ashok Priority: Minor -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4422) Move block cache parameters and references into single CacheConf class
[ https://issues.apache.org/jira/browse/HBASE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124480#comment-13124480 ] jirapos...@reviews.apache.org commented on HBASE-4422: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2089/#review2494 --- /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java https://reviews.apache.org/r/2089/#comment5653 nit: add a space after getBlockCache() /src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java https://reviews.apache.org/r/2089/#comment5654 How is this related to block cache? - Mikhail On 2011-10-10 21:08:15, Jonathan Gray wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2089/ bq. --- bq. bq. (Updated 2011-10-10 21:08:15) bq. bq. bq. Review request for hbase, Dhruba Borthakur, Michael Stack, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Creates a new CacheConfig class and moves almost everything block cache related into this single class. Adding new configuration params and booleans and such should be much better. bq. bq. All tests are NOT passing yet, still working on it, but wanted to have something up today. Basically code complete but broken :) bq. bq. bq. This addresses bug HBASE-4422. bq. https://issues.apache.org/jira/browse/HBASE-4422 bq. bq. bq. Diffs bq. - bq. bq./src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java PRE-CREATION bq./src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 1179008 bq. /src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java 1179008 bq. /src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/util/BloomFilterFactory.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java 1179008 bq. /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java 1179008 bq. /src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java 1179008 bq.
[jira] [Commented] (HBASE-4422) Move block cache parameters and references into single CacheConf class
[ https://issues.apache.org/jira/browse/HBASE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124481#comment-13124481 ] jirapos...@reviews.apache.org commented on HBASE-4422: -- bq. On 2011-10-10 21:19:18, Mikhail Bautin wrote: bq. Also, the /src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java diff is still not showing up. - Mikhail --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2089/#review2494 --- On 2011-10-10 21:08:15, Jonathan Gray wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2089/ bq. --- bq. bq. (Updated 2011-10-10 21:08:15) bq. bq. bq. Review request for hbase, Dhruba Borthakur, Michael Stack, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Creates a new CacheConfig class and moves almost everything block cache related into this single class. Adding new configuration params and booleans and such should be much better. bq. bq. All tests are NOT passing yet, still working on it, but wanted to have something up today. Basically code complete but broken :) bq. bq. bq. This addresses bug HBASE-4422. bq. https://issues.apache.org/jira/browse/HBASE-4422 bq. bq. bq. Diffs bq. - bq. bq./src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java PRE-CREATION bq./src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 1179008 bq. /src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java 1179008 bq. /src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/util/BloomFilterFactory.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java 1179008 bq. /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java 1179008 bq. /src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java 1179008 bq. /src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 1179008 bq.
[jira] [Commented] (HBASE-1744) Thrift server to match the new java api.
[
https://issues.apache.org/jira/browse/HBASE-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124482#comment-13124482
]
Bob Copeland commented on HBASE-1744:
-
I think it would be useful to take care of this todo:
{quote}
+ * TODO: This is missing the KeyValue.Type.DeleteColumn semantic. I could add
a DeleteType or something like that
{quote}
Right now thrift2 is using deleteColumn() instead of deleteColumns() which I
find to be the least useful of the two (due to re-appearing old versions). I
made a candidate incremental patch on my tree adding a TDeleteType following
the TODO's hint. It's kind of ugly but works for me. I would also be happy
with just using deleteColumns() by default.
Thrift server to match the new java api.
Key: HBASE-1744
URL: https://issues.apache.org/jira/browse/HBASE-1744
Project: HBase
Issue Type: Improvement
Components: thrift
Reporter: Tim Sell
Assignee: Tim Sell
Priority: Critical
Fix For: 0.94.0
Attachments: HBASE-1744.2.patch, HBASE-1744.3.patch,
HBASE-1744.4.patch, HBASE-1744.5.patch, HBASE-1744.6.patch,
HBASE-1744.7.patch, HBASE-1744.preview.1.patch, thriftexperiment.patch
This mutateRows, etc.. is a little confusing compared to the new cleaner java
client.
Thinking of ways to make a thrift client that is just as elegant. something
like:
void put(1:Bytes table, 2:TPut put) throws (1:IOError io)
with:
struct TColumn {
1:Bytes family,
2:Bytes qualifier,
3:i64 timestamp
}
struct TPut {
1:Bytes row,
2:mapTColumn, Bytes values
}
This creates more verbose rpc than if the columns in TPut were just
mapBytes, mapBytes, Bytes, but that is harder to fit timestamps into and
still be intuitive from say python.
Presumably the goal of a thrift gateway is to be easy first.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4422) Move block cache parameters and references into single CacheConf class
[ https://issues.apache.org/jira/browse/HBASE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124485#comment-13124485 ] jirapos...@reviews.apache.org commented on HBASE-4422: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2089/#review2496 --- Ship it! Looks good to me! Reviewed the StoreFile.java diff from the patch on the JIRA. - Mikhail On 2011-10-10 21:08:15, Jonathan Gray wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2089/ bq. --- bq. bq. (Updated 2011-10-10 21:08:15) bq. bq. bq. Review request for hbase, Dhruba Borthakur, Michael Stack, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Creates a new CacheConfig class and moves almost everything block cache related into this single class. Adding new configuration params and booleans and such should be much better. bq. bq. All tests are NOT passing yet, still working on it, but wanted to have something up today. Basically code complete but broken :) bq. bq. bq. This addresses bug HBASE-4422. bq. https://issues.apache.org/jira/browse/HBASE-4422 bq. bq. bq. Diffs bq. - bq. bq./src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java PRE-CREATION bq./src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 1179008 bq. /src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java 1179008 bq. /src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/util/BloomFilterFactory.java 1179008 bq./src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java 1179008 bq. /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java 1179008 bq. /src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java 1179008 bq. /src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java 1179008 bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 1179008 bq. /src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java
[jira] [Updated] (HBASE-1744) Thrift server to match the new java api.
[
https://issues.apache.org/jira/browse/HBASE-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bob Copeland updated HBASE-1744:
Attachment: 0001-thrift2-enable-usage-of-.deleteColumns-for-thrift.patch
Incremental patch adding TDeleteType enum
Thrift server to match the new java api.
Key: HBASE-1744
URL: https://issues.apache.org/jira/browse/HBASE-1744
Project: HBase
Issue Type: Improvement
Components: thrift
Reporter: Tim Sell
Assignee: Tim Sell
Priority: Critical
Fix For: 0.94.0
Attachments:
0001-thrift2-enable-usage-of-.deleteColumns-for-thrift.patch,
HBASE-1744.2.patch, HBASE-1744.3.patch, HBASE-1744.4.patch,
HBASE-1744.5.patch, HBASE-1744.6.patch, HBASE-1744.7.patch,
HBASE-1744.preview.1.patch, thriftexperiment.patch
This mutateRows, etc.. is a little confusing compared to the new cleaner java
client.
Thinking of ways to make a thrift client that is just as elegant. something
like:
void put(1:Bytes table, 2:TPut put) throws (1:IOError io)
with:
struct TColumn {
1:Bytes family,
2:Bytes qualifier,
3:i64 timestamp
}
struct TPut {
1:Bytes row,
2:mapTColumn, Bytes values
}
This creates more verbose rpc than if the columns in TPut were just
mapBytes, mapBytes, Bytes, but that is harder to fit timestamps into and
still be intuitive from say python.
Presumably the goal of a thrift gateway is to be easy first.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4486) Improve Javadoc for HTableDescriptor
[ https://issues.apache.org/jira/browse/HBASE-4486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akash Ashok updated HBASE-4486: --- Attachment: HTableDescriptor.html HBase-4486.patch Have also attached the compiled javadoc. Small edits on the MasterFileSystem.java Related Tests run: TestCatalogJanitor Still in the process of running the entire test suite Improve Javadoc for HTableDescriptor Key: HBASE-4486 URL: https://issues.apache.org/jira/browse/HBASE-4486 Project: HBase Issue Type: Improvement Components: client, documentation Reporter: Akash Ashok Assignee: Akash Ashok Priority: Minor Attachments: HBase-4486.patch, HTableDescriptor.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4552) multi-CF bulk load is not atomic across column families
[
https://issues.apache.org/jira/browse/HBASE-4552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124506#comment-13124506
]
Ted Yu commented on HBASE-4552:
---
One solution is to add the following method to HRegionInterface:
{code}
public void bulkLoadHFile(ListPairbyte[], String familyPaths, byte[]
regionName)
throws IOException;
{code}
familyPaths is a list of family, hfilePath pairs for the same region
identified by regionName.
LoadIncrementalHFiles would need to group HFiles for the same region together
before calling the above method.
multi-CF bulk load is not atomic across column families
---
Key: HBASE-4552
URL: https://issues.apache.org/jira/browse/HBASE-4552
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Fix For: 0.92.0
Currently the bulk load API simply imports one HFile at a time. With
multi-column-family support, this is inappropriate, since different CFs show
up separately. Instead, the IPC endpoint should take a of CF - HFiles, so we
can online them all under a single region-wide lock.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4552) multi-CF bulk load is not atomic across column families
[ https://issues.apache.org/jira/browse/HBASE-4552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124511#comment-13124511 ] Todd Lipcon commented on HBASE-4552: yep, that's what I meant, but the implementation isn't quite trivial since we have to do the locking at a higher level, so the change is made visible atomically. multi-CF bulk load is not atomic across column families --- Key: HBASE-4552 URL: https://issues.apache.org/jira/browse/HBASE-4552 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.92.0 Reporter: Todd Lipcon Fix For: 0.92.0 Currently the bulk load API simply imports one HFile at a time. With multi-column-family support, this is inappropriate, since different CFs show up separately. Instead, the IPC endpoint should take a of CF - HFiles, so we can online them all under a single region-wide lock. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4552) multi-CF bulk load is not atomic across column families
[
https://issues.apache.org/jira/browse/HBASE-4552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124521#comment-13124521
]
Ted Yu commented on HBASE-4552:
---
For HRegion, we can introduce the following new method:
{code}
public void bulkLoadHFile(ListPairbyte[], String familyPaths)
throws IOException {
{code}
where familyPaths is a list of family, hfilePath pairs identifying HFile path
and the underlying family the HFile should be loaded to.
multi-CF bulk load is not atomic across column families
---
Key: HBASE-4552
URL: https://issues.apache.org/jira/browse/HBASE-4552
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Fix For: 0.92.0
Currently the bulk load API simply imports one HFile at a time. With
multi-column-family support, this is inappropriate, since different CFs show
up separately. Instead, the IPC endpoint should take a of CF - HFiles, so we
can online them all under a single region-wide lock.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4568) Make zk dump jsp response more quickly
[ https://issues.apache.org/jira/browse/HBASE-4568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liyin Tang updated HBASE-4568: -- Description: For each zk dump, currently hbase will create a zk client instance every time. This is quite slow when any machines in the quorum is dead. Because it will connect to each machine in the zk quorum again. HMaster master = (HMaster)getServletContext().getAttribute(HMaster.MASTER); Configuration conf = master.getConfiguration(); HBaseAdmin hbadmin = new HBaseAdmin(conf); HConnection connection = hbadmin.getConnection(); ZooKeeperWatcher watcher = connection.getZooKeeperWatcher(); So we can simplify this: HMaster master = (HMaster)getServletContext().getAttribute(HMaster.MASTER); ZooKeeperWatcher watcher = master.getZooKeeperWatcher(); Also when hbase call getServerStats() for each machine in the zk quorum, it hard coded the default time out as 1 min. It would be nice to make this configurable and set it to a low time out. When hbase tries to connect to each machine in the zk quorum, it will create the socket, and then set the socket time out, and read it with this time out. It means hbase will create a socket and connect to the zk server with 0 time out at first, which will take a long time. Because a timeout of zero is interpreted as an infinite timeout. The connection will then block until established or an error occurs. was: For each zk dump, currently hbase will create a zk client instance every time. This is quite slow when any machines in the quorum is dead. Because it will connect to each machine in the zk quorum again. HMaster master = (HMaster)getServletContext().getAttribute(HMaster.MASTER); Configuration conf = master.getConfiguration(); HBaseAdmin hbadmin = new HBaseAdmin(conf); HConnection connection = hbadmin.getConnection(); ZooKeeperWatcher watcher = connection.getZooKeeperWatcher(); So we can simplify this: HMaster master = (HMaster)getServletContext().getAttribute(HMaster.MASTER); ZooKeeperWatcher watcher = master.getZooKeeperWatcher(); Also when hbase call getServerStats() for each machine in the zk quorum, it hard coded the default time out as 1 min. It would be nice to make this configurable and set it to a low time out. Make zk dump jsp response more quickly -- Key: HBASE-4568 URL: https://issues.apache.org/jira/browse/HBASE-4568 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang For each zk dump, currently hbase will create a zk client instance every time. This is quite slow when any machines in the quorum is dead. Because it will connect to each machine in the zk quorum again. HMaster master = (HMaster)getServletContext().getAttribute(HMaster.MASTER); Configuration conf = master.getConfiguration(); HBaseAdmin hbadmin = new HBaseAdmin(conf); HConnection connection = hbadmin.getConnection(); ZooKeeperWatcher watcher = connection.getZooKeeperWatcher(); So we can simplify this: HMaster master = (HMaster)getServletContext().getAttribute(HMaster.MASTER); ZooKeeperWatcher watcher = master.getZooKeeperWatcher(); Also when hbase call getServerStats() for each machine in the zk quorum, it hard coded the default time out as 1 min. It would be nice to make this configurable and set it to a low time out. When hbase tries to connect to each machine in the zk quorum, it will create the socket, and then set the socket time out, and read it with this time out. It means hbase will create a socket and connect to the zk server with 0 time out at first, which will take a long time. Because a timeout of zero is interpreted as an infinite timeout. The connection will then block until established or an error occurs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4570) Scan ACID problem with concurrent puts.
Scan ACID problem with concurrent puts.
---
Key: HBASE-4570
URL: https://issues.apache.org/jira/browse/HBASE-4570
Project: HBase
Issue Type: Bug
Components: client, regionserver
Affects Versions: 0.90.3, 0.90.1
Reporter: Jonathan Hsieh
When scanning a table sometimes rows that have multiple column families get
split into two rows if there are concurrent writes. In this particular case we
are overwriting the contents of a Get directly back onto itself as a Put.
For example, this is a two cf row (with f1, f2, .. f9 cfs). It is
actually returned as two rows (#55 and #56). Interestingly if the two were
merged we would have a single proper row.
Row row024461 had time stamps: [55:
keyvalues={row024461/f0:data/1318200440867/Put/vlen=1000,
row024461/f0:qual/1318200440867/Put/vlen=10,
row024461/f1:data/1318200440867/Put/vlen=1000,
row024461/f1:qual/1318200440867/Put/vlen=10,
row024461/f2:data/1318200440867/Put/vlen=1000,
row024461/f2:qual/1318200440867/Put/vlen=10,
row024461/f3:data/1318200440867/Put/vlen=1000,
row024461/f3:qual/1318200440867/Put/vlen=10,
row024461/f4:data/1318200440867/Put/vlen=1000,
row024461/f4:qual/1318200440867/Put/vlen=10},
56: keyvalues={row024461/f5:data/1318200440867/Put/vlen=1000,
row024461/f5:qual/1318200440867/Put/vlen=10,
row024461/f6:data/1318200440867/Put/vlen=1000,
row024461/f6:qual/1318200440867/Put/vlen=10,
row024461/f7:data/1318200440867/Put/vlen=1000,
row024461/f7:qual/1318200440867/Put/vlen=10,
row024461/f8:data/1318200440867/Put/vlen=1000,
row024461/f8:qual/1318200440867/Put/vlen=10,
row024461/f9:data/1318200440867/Put/vlen=1000,
row024461/f9:qual/1318200440867/Put/vlen=10}]
I've only tested this on 0.90.1+patches and 0.90.3+patches, but it is
consistent and duplicatable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2856) TestAcidGuarantee broken on trunk
[ https://issues.apache.org/jira/browse/HBASE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124561#comment-13124561 ] jirapos...@reviews.apache.org commented on HBASE-2856: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2224/ --- (Updated 2011-10-10 23:14:05.211809) Review request for Ted Yu, Michael Stack, Kannan Muthukkaruppan, and Karthik Ranganathan. Changes --- address some of the comments given. Summary --- address the 2856 issues by writing the memstoreTS to the disk. version v11 of the patch. uploading it here for easier review process. This addresses bug HBASE-2856. https://issues.apache.org/jira/browse/HBASE-2856 Diffs (updated) - /src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 1181113 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 1181113 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 1181113 /src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1181113 /src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 1181113 /src/test/java/org/apache/hadoop/hbase/TestAcidGuarantees.java 1181113 /src/main/java/org/apache/hadoop/hbase/regionserver/ReadWriteConsistencyControl.java 1181113 /src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 1181113 /src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1181113 /src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1181113 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 1181113 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 1181113 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1181113 /src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 1181113 Diff: https://reviews.apache.org/r/2224/diff Testing --- mvn test Thanks, Amitanand TestAcidGuarantee broken on trunk -- Key: HBASE-2856 URL: https://issues.apache.org/jira/browse/HBASE-2856 Project: HBase Issue Type: Bug Affects Versions: 0.89.20100621 Reporter: ryan rawson Assignee: Amitanand Aiyer Priority: Blocker Fix For: 0.94.0 Attachments: 2856-v2.txt, 2856-v3.txt, 2856-v4.txt, 2856-v5.txt, acid.txt TestAcidGuarantee has a test whereby it attempts to read a number of columns from a row, and every so often the first column of N is different, when it should be the same. This is a bug deep inside the scanner whereby the first peek() of a row is done at time T then the rest of the read is done at T+1 after a flush, thus the memstoreTS data is lost, and previously 'uncommitted' data becomes committed and flushed to disk. One possible solution is to introduce the memstoreTS (or similarly equivalent value) to the HFile thus allowing us to preserve read consistency past flushes. Another solution involves fixing the scanners so that peek() is not destructive (and thus might return different things at different times alas). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2856) TestAcidGuarantee broken on trunk
[ https://issues.apache.org/jira/browse/HBASE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124564#comment-13124564 ] jirapos...@reviews.apache.org commented on HBASE-2856: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2224/#review2497 --- /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java https://reviews.apache.org/r/2224/#comment5655 convert tab to spaces. /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java https://reviews.apache.org/r/2224/#comment5656 Would MAX_MEMSTORE_TS_KEY be a better name ? - Ted On 2011-10-10 23:14:05, Amitanand Aiyer wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2224/ bq. --- bq. bq. (Updated 2011-10-10 23:14:05) bq. bq. bq. Review request for Ted Yu, Michael Stack, Kannan Muthukkaruppan, and Karthik Ranganathan. bq. bq. bq. Summary bq. --- bq. bq. address the 2856 issues by writing the memstoreTS to the disk. bq. bq. version v11 of the patch. bq. bq. uploading it here for easier review process. bq. bq. bq. This addresses bug HBASE-2856. bq. https://issues.apache.org/jira/browse/HBASE-2856 bq. bq. bq. Diffs bq. - bq. bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 1181113 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 1181113 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 1181113 bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1181113 bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 1181113 bq./src/test/java/org/apache/hadoop/hbase/TestAcidGuarantees.java 1181113 bq. /src/main/java/org/apache/hadoop/hbase/regionserver/ReadWriteConsistencyControl.java 1181113 bq. /src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 1181113 bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1181113 bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1181113 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 1181113 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 1181113 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1181113 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 1181113 bq. bq. Diff: https://reviews.apache.org/r/2224/diff bq. bq. bq. Testing bq. --- bq. bq. mvn test bq. bq. bq. Thanks, bq. bq. Amitanand bq. bq. TestAcidGuarantee broken on trunk -- Key: HBASE-2856 URL: https://issues.apache.org/jira/browse/HBASE-2856 Project: HBase Issue Type: Bug Affects Versions: 0.89.20100621 Reporter: ryan rawson Assignee: Amitanand Aiyer Priority: Blocker Fix For: 0.94.0 Attachments: 2856-v2.txt, 2856-v3.txt, 2856-v4.txt, 2856-v5.txt, acid.txt TestAcidGuarantee has a test whereby it attempts to read a number of columns from a row, and every so often the first column of N is different, when it should be the same. This is a bug deep inside the scanner whereby the first peek() of a row is done at time T then the rest of the read is done at T+1 after a flush, thus the memstoreTS data is lost, and previously 'uncommitted' data becomes committed and flushed to disk. One possible solution is to introduce the memstoreTS (or similarly equivalent value) to the HFile thus allowing us to preserve read consistency past flushes. Another solution involves fixing the scanners so that peek() is not destructive (and thus might return different things at different times alas). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4102) atomicAppend: A put that appends to the latest version of a cell; i.e. reads current value then adds the bytes offered by the client to the tail and writes out a new entr
[ https://issues.apache.org/jira/browse/HBASE-4102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-4102: - Attachment: 4102-v1.txt Thanks Ted... New change addressing all your points. atomicAppend: A put that appends to the latest version of a cell; i.e. reads current value then adds the bytes offered by the client to the tail and writes out a new entry --- Key: HBASE-4102 URL: https://issues.apache.org/jira/browse/HBASE-4102 Project: HBase Issue Type: New Feature Reporter: stack Assignee: Lars Hofhansl Attachments: 4102-v1.txt, 4102.txt Its come up a few times that clients want to add to an existing cell rather than make a new cell each time. At our place, the frontend keeps a list of urls a user has visited -- their md5s -- and updates it as user progresses. Rather than read, modify client-side, then write new value back to hbase, it would be sweet if could do it all in one operation in hbase server. TSDB aims to be space efficient. Rather than pay the cost of the KV wrapper per metric, it would rather have a KV for an interval an in this KV have a value that is all the metrics for the period. It could be done as a coprocessor but this feels more like a fundamental feature. Benoît suggests that atomicAppend take a flag to indicate whether or not the client wants to see the resulting cell; often a client won't want to see the result and in this case, why pay the price formulating and delivering a response that client just drops. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2856) TestAcidGuarantee broken on trunk
[ https://issues.apache.org/jira/browse/HBASE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124572#comment-13124572 ] jirapos...@reviews.apache.org commented on HBASE-2856: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2224/ --- (Updated 2011-10-10 23:30:10.189214) Review request for Ted Yu, Michael Stack, Kannan Muthukkaruppan, and Karthik Ranganathan. Changes --- fix whitespace issues. Summary --- address the 2856 issues by writing the memstoreTS to the disk. version v11 of the patch. uploading it here for easier review process. This addresses bug HBASE-2856. https://issues.apache.org/jira/browse/HBASE-2856 Diffs (updated) - /src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 1181113 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 1181113 /src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1181113 /src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 1181113 /src/test/java/org/apache/hadoop/hbase/TestAcidGuarantees.java 1181113 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 1181113 /src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 1181113 /src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1181113 /src/main/java/org/apache/hadoop/hbase/regionserver/ReadWriteConsistencyControl.java 1181113 /src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1181113 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 1181113 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 1181113 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1181113 /src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 1181113 Diff: https://reviews.apache.org/r/2224/diff Testing --- mvn test Thanks, Amitanand TestAcidGuarantee broken on trunk -- Key: HBASE-2856 URL: https://issues.apache.org/jira/browse/HBASE-2856 Project: HBase Issue Type: Bug Affects Versions: 0.89.20100621 Reporter: ryan rawson Assignee: Amitanand Aiyer Priority: Blocker Fix For: 0.94.0 Attachments: 2856-v2.txt, 2856-v3.txt, 2856-v4.txt, 2856-v5.txt, acid.txt TestAcidGuarantee has a test whereby it attempts to read a number of columns from a row, and every so often the first column of N is different, when it should be the same. This is a bug deep inside the scanner whereby the first peek() of a row is done at time T then the rest of the read is done at T+1 after a flush, thus the memstoreTS data is lost, and previously 'uncommitted' data becomes committed and flushed to disk. One possible solution is to introduce the memstoreTS (or similarly equivalent value) to the HFile thus allowing us to preserve read consistency past flushes. Another solution involves fixing the scanners so that peek() is not destructive (and thus might return different things at different times alas). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2856) TestAcidGuarantee broken on trunk
[ https://issues.apache.org/jira/browse/HBASE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124575#comment-13124575 ] jirapos...@reviews.apache.org commented on HBASE-2856: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2224/ --- (Updated 2011-10-10 23:35:00.552025) Review request for Ted Yu, Michael Stack, Kannan Muthukkaruppan, and Karthik Ranganathan. Changes --- rename MAX_MEMSTORE_KEY to MAX_MEMSTORE_TS_KEY Summary --- address the 2856 issues by writing the memstoreTS to the disk. version v11 of the patch. uploading it here for easier review process. This addresses bug HBASE-2856. https://issues.apache.org/jira/browse/HBASE-2856 Diffs (updated) - /src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 1181113 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1181113 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 1181113 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 1181113 /src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1181113 /src/main/java/org/apache/hadoop/hbase/regionserver/ReadWriteConsistencyControl.java 1181113 /src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 1181113 /src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1181113 /src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1181113 /src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 1181113 /src/test/java/org/apache/hadoop/hbase/TestAcidGuarantees.java 1181113 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 1181113 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 1181113 /src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 1181113 Diff: https://reviews.apache.org/r/2224/diff Testing --- mvn test Thanks, Amitanand TestAcidGuarantee broken on trunk -- Key: HBASE-2856 URL: https://issues.apache.org/jira/browse/HBASE-2856 Project: HBase Issue Type: Bug Affects Versions: 0.89.20100621 Reporter: ryan rawson Assignee: Amitanand Aiyer Priority: Blocker Fix For: 0.94.0 Attachments: 2856-v2.txt, 2856-v3.txt, 2856-v4.txt, 2856-v5.txt, acid.txt TestAcidGuarantee has a test whereby it attempts to read a number of columns from a row, and every so often the first column of N is different, when it should be the same. This is a bug deep inside the scanner whereby the first peek() of a row is done at time T then the rest of the read is done at T+1 after a flush, thus the memstoreTS data is lost, and previously 'uncommitted' data becomes committed and flushed to disk. One possible solution is to introduce the memstoreTS (or similarly equivalent value) to the HFile thus allowing us to preserve read consistency past flushes. Another solution involves fixing the scanners so that peek() is not destructive (and thus might return different things at different times alas). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2856) TestAcidGuarantee broken on trunk
[ https://issues.apache.org/jira/browse/HBASE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124581#comment-13124581 ] jirapos...@reviews.apache.org commented on HBASE-2856: -- bq. On 2011-10-05 23:23:04, Michael Stack wrote: bq. /src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java, line 556 bq. https://reviews.apache.org/r/2224/diff/1/?file=48330#file48330line556 bq. bq. Is this a good name? Its come back in from the storefile metadata. There is no memstore in this context. No biggie. Just saying. any suggestions? - Amitanand --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2224/#review2374 --- On 2011-10-10 23:35:00, Amitanand Aiyer wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2224/ bq. --- bq. bq. (Updated 2011-10-10 23:35:00) bq. bq. bq. Review request for Ted Yu, Michael Stack, Kannan Muthukkaruppan, and Karthik Ranganathan. bq. bq. bq. Summary bq. --- bq. bq. address the 2856 issues by writing the memstoreTS to the disk. bq. bq. version v11 of the patch. bq. bq. uploading it here for easier review process. bq. bq. bq. This addresses bug HBASE-2856. bq. https://issues.apache.org/jira/browse/HBASE-2856 bq. bq. bq. Diffs bq. - bq. bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 1181113 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1181113 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 1181113 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 1181113 bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1181113 bq. /src/main/java/org/apache/hadoop/hbase/regionserver/ReadWriteConsistencyControl.java 1181113 bq. /src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 1181113 bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1181113 bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1181113 bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 1181113 bq./src/test/java/org/apache/hadoop/hbase/TestAcidGuarantees.java 1181113 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 1181113 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 1181113 bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 1181113 bq. bq. Diff: https://reviews.apache.org/r/2224/diff bq. bq. bq. Testing bq. --- bq. bq. mvn test bq. bq. bq. Thanks, bq. bq. Amitanand bq. bq. TestAcidGuarantee broken on trunk -- Key: HBASE-2856 URL: https://issues.apache.org/jira/browse/HBASE-2856 Project: HBase Issue Type: Bug Affects Versions: 0.89.20100621 Reporter: ryan rawson Assignee: Amitanand Aiyer Priority: Blocker Fix For: 0.94.0 Attachments: 2856-v2.txt, 2856-v3.txt, 2856-v4.txt, 2856-v5.txt, acid.txt TestAcidGuarantee has a test whereby it attempts to read a number of columns from a row, and every so often the first column of N is different, when it should be the same. This is a bug deep inside the scanner whereby the first peek() of a row is done at time T then the rest of the read is done at T+1 after a flush, thus the memstoreTS data is lost, and previously 'uncommitted' data becomes committed and flushed to disk. One possible solution is to introduce the memstoreTS (or similarly equivalent value) to the HFile thus allowing us to preserve read consistency past flushes. Another solution involves fixing the scanners so that peek() is not destructive (and thus might return different things at different times alas). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4571) [book] book.xml - added Schema Design Smackdown section
[book] book.xml - added Schema Design Smackdown section --- Key: HBASE-4571 URL: https://issues.apache.org/jira/browse/HBASE-4571 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor book.xml * added Schema Design Smackdown section in the Schema Design chapter. Common schema design questions on the dist-list. * Rows vs. Versions * Rows vs. Columns -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4571) [book] book.xml - added Schema Design Smackdown section
[ https://issues.apache.org/jira/browse/HBASE-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-4571: - Resolution: Fixed Status: Resolved (was: Patch Available) [book] book.xml - added Schema Design Smackdown section --- Key: HBASE-4571 URL: https://issues.apache.org/jira/browse/HBASE-4571 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: book_HBASE_4571.xml.patch book.xml * added Schema Design Smackdown section in the Schema Design chapter. Common schema design questions on the dist-list. * Rows vs. Versions * Rows vs. Columns -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4571) [book] book.xml - added Schema Design Smackdown section
[ https://issues.apache.org/jira/browse/HBASE-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-4571: - Attachment: book_HBASE_4571.xml.patch [book] book.xml - added Schema Design Smackdown section --- Key: HBASE-4571 URL: https://issues.apache.org/jira/browse/HBASE-4571 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: book_HBASE_4571.xml.patch book.xml * added Schema Design Smackdown section in the Schema Design chapter. Common schema design questions on the dist-list. * Rows vs. Versions * Rows vs. Columns -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4563) When split doing this.parent.close(false) occurs error,it'll cause the splited region cann't write read
[
https://issues.apache.org/jira/browse/HBASE-4563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124626#comment-13124626
]
bluedavy commented on HBASE-4563:
-
throw exception so CompactSplitThread can catch it then do rollback...
When split doing this.parent.close(false) occurs error,it'll cause the
splited region cann't write read
-
Key: HBASE-4563
URL: https://issues.apache.org/jira/browse/HBASE-4563
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.90.4
Reporter: bluedavy
Priority: Blocker
Fix For: 0.90.5
Attachments: HBASE-4563-test.report.txt, HBASE-4563.patch
Follow below steps to replay the problem:
1. change the SplitTransaction.java as below,just like mock the hdfs error.
{code:title=SplitTransaction.java|borderStyle=solid}
ListStoreFile hstoreFilesToSplit = this.parent.close(false);
throw new IOException(some unexpected error in close store files);
{code}
2. update the regionserver code,restart;
3. create a table put some data to the table;
4. split the table;
5. scan the table,then it'll fail.
We can fix the bug just use below code:
{code:title=SplitTransaction.java|borderStyle=solid}
ListStoreFile hstoreFilesToSplit = null;
try{
hstoreFilesToSplit = this.parent.close(false);
}
catch(IOException e){
this.journal.add(JournalEntry.CLOSED_PARENT_REGION);
throw e;
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4563) When split doing this.parent.close(false) occurs error,it'll cause the splited region cann't write read
[
https://issues.apache.org/jira/browse/HBASE-4563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124630#comment-13124630
]
Ted Yu commented on HBASE-4563:
---
Ramkrishna's comment @ 10/Oct/11 10:35 was saying that the IOException caught
didn't get transformed into another exception type.
So the catch block isn't needed.
When split doing this.parent.close(false) occurs error,it'll cause the
splited region cann't write read
-
Key: HBASE-4563
URL: https://issues.apache.org/jira/browse/HBASE-4563
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.90.4
Reporter: bluedavy
Priority: Blocker
Fix For: 0.90.5
Attachments: HBASE-4563-test.report.txt, HBASE-4563.patch
Follow below steps to replay the problem:
1. change the SplitTransaction.java as below,just like mock the hdfs error.
{code:title=SplitTransaction.java|borderStyle=solid}
ListStoreFile hstoreFilesToSplit = this.parent.close(false);
throw new IOException(some unexpected error in close store files);
{code}
2. update the regionserver code,restart;
3. create a table put some data to the table;
4. split the table;
5. scan the table,then it'll fail.
We can fix the bug just use below code:
{code:title=SplitTransaction.java|borderStyle=solid}
ListStoreFile hstoreFilesToSplit = null;
try{
hstoreFilesToSplit = this.parent.close(false);
}
catch(IOException e){
this.journal.add(JournalEntry.CLOSED_PARENT_REGION);
throw e;
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4563) When split doing this.parent.close(false) occurs error,it'll cause the splited region cann't write read
[
https://issues.apache.org/jira/browse/HBASE-4563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124635#comment-13124635
]
bluedavy commented on HBASE-4563:
-
@Ted Yu
Yes,:),I'll change it attach the patch again.
When split doing this.parent.close(false) occurs error,it'll cause the
splited region cann't write read
-
Key: HBASE-4563
URL: https://issues.apache.org/jira/browse/HBASE-4563
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.90.4
Reporter: bluedavy
Priority: Blocker
Fix For: 0.90.5
Attachments: HBASE-4563-test.report.txt, HBASE-4563.patch
Follow below steps to replay the problem:
1. change the SplitTransaction.java as below,just like mock the hdfs error.
{code:title=SplitTransaction.java|borderStyle=solid}
ListStoreFile hstoreFilesToSplit = this.parent.close(false);
throw new IOException(some unexpected error in close store files);
{code}
2. update the regionserver code,restart;
3. create a table put some data to the table;
4. split the table;
5. scan the table,then it'll fail.
We can fix the bug just use below code:
{code:title=SplitTransaction.java|borderStyle=solid}
ListStoreFile hstoreFilesToSplit = null;
try{
hstoreFilesToSplit = this.parent.close(false);
}
catch(IOException e){
this.journal.add(JournalEntry.CLOSED_PARENT_REGION);
throw e;
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4562) When split doing offlineParentInMeta occurs error,it'll cause data loss
[
https://issues.apache.org/jira/browse/HBASE-4562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124638#comment-13124638
]
bluedavy commented on HBASE-4562:
-
@Ted Yu
I'll check the tests...
Also I'll attach the patch for 0.92 TRUNK.
When split doing offlineParentInMeta occurs error,it'll cause data loss
---
Key: HBASE-4562
URL: https://issues.apache.org/jira/browse/HBASE-4562
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.90.4
Reporter: bluedavy
Priority: Blocker
Fix For: 0.90.5
Attachments: HBASE-4562-test.report.txt, HBASE-4562.patch
Follow below steps to replay the problem:
1. change the SplitTransaction.java as below,just like mock the timeout error.
{code:title=SplitTransaction.java|borderStyle=solid}
if (!testing) {
MetaEditor.offlineParentInMeta(server.getCatalogTracker(),
this.parent.getRegionInfo(), a.getRegionInfo(), b.getRegionInfo());
throw new IOException(some unexpected error in split);
}
{code}
2. update the regionserver code,restart;
3. create a table put some data to the table;
4. split the table;
5. kill the regionserver hosted the table;
6. wait some time after master ServerShutdownHandler.process execute,then
scan the table,u'll find the data wrote before lost.
We can fix the bug just use below code:
{code:title=SplitTransaction.java|borderStyle=solid}
this.journal.add(JournalEntry.PONR);
if (!testing) {
MetaEditor.offlineParentInMeta(server.getCatalogTracker(),
this.parent.getRegionInfo(), a.getRegionInfo(), b.getRegionInfo());
throw new IOException(some unexpected error in split);
}
{code}
{code:title=CompactSplitThread.java|borderStyle=solid}
if (st.rollback(this.server, this.server)) {
LOG.info(Successful rollback of failed split of +
parent.getRegionNameAsString());
}
else {
this.server.abort(Abort; we got an error after
point-of-no-return);
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2856) TestAcidGuarantee broken on trunk
[ https://issues.apache.org/jira/browse/HBASE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124639#comment-13124639 ] jirapos...@reviews.apache.org commented on HBASE-2856: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2224/ --- (Updated 2011-10-11 01:09:52.483911) Review request for Ted Yu, Michael Stack, Kannan Muthukkaruppan, and Karthik Ranganathan. Changes --- fix the TestAcidGuarantees test. Had changed a condition from 0 to == HFileWriterV2.KEY_VALUE_VER_WITH_MEMSTORE make sure it is == Summary --- address the 2856 issues by writing the memstoreTS to the disk. version v11 of the patch. uploading it here for easier review process. This addresses bug HBASE-2856. https://issues.apache.org/jira/browse/HBASE-2856 Diffs (updated) - /src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 1181113 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1181113 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 1181113 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 1181113 /src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1181113 /src/main/java/org/apache/hadoop/hbase/regionserver/ReadWriteConsistencyControl.java 1181113 /src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 1181113 /src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1181113 /src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1181113 /src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 1181113 /src/test/java/org/apache/hadoop/hbase/TestAcidGuarantees.java 1181113 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 1181113 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 1181113 /src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 1181113 Diff: https://reviews.apache.org/r/2224/diff Testing --- mvn test Thanks, Amitanand TestAcidGuarantee broken on trunk -- Key: HBASE-2856 URL: https://issues.apache.org/jira/browse/HBASE-2856 Project: HBase Issue Type: Bug Affects Versions: 0.89.20100621 Reporter: ryan rawson Assignee: Amitanand Aiyer Priority: Blocker Fix For: 0.94.0 Attachments: 2856-v2.txt, 2856-v3.txt, 2856-v4.txt, 2856-v5.txt, acid.txt TestAcidGuarantee has a test whereby it attempts to read a number of columns from a row, and every so often the first column of N is different, when it should be the same. This is a bug deep inside the scanner whereby the first peek() of a row is done at time T then the rest of the read is done at T+1 after a flush, thus the memstoreTS data is lost, and previously 'uncommitted' data becomes committed and flushed to disk. One possible solution is to introduce the memstoreTS (or similarly equivalent value) to the HFile thus allowing us to preserve read consistency past flushes. Another solution involves fixing the scanners so that peek() is not destructive (and thus might return different things at different times alas). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2856) TestAcidGuarantee broken on trunk
[ https://issues.apache.org/jira/browse/HBASE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124643#comment-13124643 ] jirapos...@reviews.apache.org commented on HBASE-2856: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2224/#review2500 --- /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java https://reviews.apache.org/r/2224/#comment5659 :) new revision looks good. - Kannan On 2011-10-11 01:09:52, Amitanand Aiyer wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2224/ bq. --- bq. bq. (Updated 2011-10-11 01:09:52) bq. bq. bq. Review request for Ted Yu, Michael Stack, Kannan Muthukkaruppan, and Karthik Ranganathan. bq. bq. bq. Summary bq. --- bq. bq. address the 2856 issues by writing the memstoreTS to the disk. bq. bq. version v11 of the patch. bq. bq. uploading it here for easier review process. bq. bq. bq. This addresses bug HBASE-2856. bq. https://issues.apache.org/jira/browse/HBASE-2856 bq. bq. bq. Diffs bq. - bq. bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 1181113 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1181113 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 1181113 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 1181113 bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1181113 bq. /src/main/java/org/apache/hadoop/hbase/regionserver/ReadWriteConsistencyControl.java 1181113 bq. /src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 1181113 bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1181113 bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1181113 bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 1181113 bq./src/test/java/org/apache/hadoop/hbase/TestAcidGuarantees.java 1181113 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 1181113 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 1181113 bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 1181113 bq. bq. Diff: https://reviews.apache.org/r/2224/diff bq. bq. bq. Testing bq. --- bq. bq. mvn test bq. bq. bq. Thanks, bq. bq. Amitanand bq. bq. TestAcidGuarantee broken on trunk -- Key: HBASE-2856 URL: https://issues.apache.org/jira/browse/HBASE-2856 Project: HBase Issue Type: Bug Affects Versions: 0.89.20100621 Reporter: ryan rawson Assignee: Amitanand Aiyer Priority: Blocker Fix For: 0.94.0 Attachments: 2856-v2.txt, 2856-v3.txt, 2856-v4.txt, 2856-v5.txt, acid.txt TestAcidGuarantee has a test whereby it attempts to read a number of columns from a row, and every so often the first column of N is different, when it should be the same. This is a bug deep inside the scanner whereby the first peek() of a row is done at time T then the rest of the read is done at T+1 after a flush, thus the memstoreTS data is lost, and previously 'uncommitted' data becomes committed and flushed to disk. One possible solution is to introduce the memstoreTS (or similarly equivalent value) to the HFile thus allowing us to preserve read consistency past flushes. Another solution involves fixing the scanners so that peek() is not destructive (and thus might return different things at different times alas). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4536) Allow CF to retain deleted rows
[ https://issues.apache.org/jira/browse/HBASE-4536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124644#comment-13124644 ] Ted Yu commented on HBASE-4536: --- 3a stands out as first choice so far. Allow CF to retain deleted rows --- Key: HBASE-4536 URL: https://issues.apache.org/jira/browse/HBASE-4536 Project: HBase Issue Type: Sub-task Components: regionserver Affects Versions: 0.92.0 Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Parent allows for a cluster to retain rows for a TTL or keep a minimum number of versions. However, if a client deletes a row all version older than the delete tomb stone will be remove at the next major compaction (and even at memstore flush - see HBASE-4241). There should be a way to retain those version to guard against software error. I see two options here: 1. Add a new flag HColumnDescriptor. Something like RETAIN_DELETED. 2. Folds this into the parent change. I.e. keep minimum-number-of-versions of versions even past the delete marker. #1 would allow for more flexibility. #2 comes somewhat naturally with parent (from a user viewpoint) Comments? Any other options? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2856) TestAcidGuarantee broken on trunk
[ https://issues.apache.org/jira/browse/HBASE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124645#comment-13124645 ] Ted Yu commented on HBASE-2856: --- I think we should tackle TestAcidGuarantees failure(s) next. TestAcidGuarantee broken on trunk -- Key: HBASE-2856 URL: https://issues.apache.org/jira/browse/HBASE-2856 Project: HBase Issue Type: Bug Affects Versions: 0.89.20100621 Reporter: ryan rawson Assignee: Amitanand Aiyer Priority: Blocker Fix For: 0.94.0 Attachments: 2856-v2.txt, 2856-v3.txt, 2856-v4.txt, 2856-v5.txt, acid.txt TestAcidGuarantee has a test whereby it attempts to read a number of columns from a row, and every so often the first column of N is different, when it should be the same. This is a bug deep inside the scanner whereby the first peek() of a row is done at time T then the rest of the read is done at T+1 after a flush, thus the memstoreTS data is lost, and previously 'uncommitted' data becomes committed and flushed to disk. One possible solution is to introduce the memstoreTS (or similarly equivalent value) to the HFile thus allowing us to preserve read consistency past flushes. Another solution involves fixing the scanners so that peek() is not destructive (and thus might return different things at different times alas). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4571) [book] book.xml - added Schema Design Smackdown section
[ https://issues.apache.org/jira/browse/HBASE-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124653#comment-13124653 ] Hudson commented on HBASE-4571: --- Integrated in HBase-TRUNK #2316 (See [https://builds.apache.org/job/HBase-TRUNK/2316/]) HBASE-4571. book.xml, adding Schema Design Smackdown section. dmeil : Files : * /hbase/trunk/src/docbkx/book.xml [book] book.xml - added Schema Design Smackdown section --- Key: HBASE-4571 URL: https://issues.apache.org/jira/browse/HBASE-4571 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: book_HBASE_4571.xml.patch book.xml * added Schema Design Smackdown section in the Schema Design chapter. Common schema design questions on the dist-list. * Rows vs. Versions * Rows vs. Columns -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4572) Multithread compaction in regionserver
Multithread compaction in regionserver -- Key: HBASE-4572 URL: https://issues.apache.org/jira/browse/HBASE-4572 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.90.4 Reporter: Aaron Guo Fix For: 0.92.0 RegionServer only have one thread doing region compaction staff. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4572) Multithread compaction in regionserver
[ https://issues.apache.org/jira/browse/HBASE-4572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Guo updated HBASE-4572: - Description: RegionServer only have one thread doing region compaction staff. Could We adjust the RegionServer compation thread num and priority online. was:RegionServer only have one thread doing region compaction staff. Multithread compaction in regionserver -- Key: HBASE-4572 URL: https://issues.apache.org/jira/browse/HBASE-4572 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.90.4 Reporter: Aaron Guo Fix For: 0.92.0 RegionServer only have one thread doing region compaction staff. Could We adjust the RegionServer compation thread num and priority online. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4552) multi-CF bulk load is not atomic across column families
[ https://issues.apache.org/jira/browse/HBASE-4552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124660#comment-13124660 ] Ted Yu commented on HBASE-4552: --- Toward the end of LoadIncrementalHFiles.tryLoad() we can utilize startEndKeys of the underlying table to group (possibly split) HFiles by their first row keys. Then the new bulkLoadHFile() method would be called by doBulkLoad(). multi-CF bulk load is not atomic across column families --- Key: HBASE-4552 URL: https://issues.apache.org/jira/browse/HBASE-4552 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.92.0 Reporter: Todd Lipcon Fix For: 0.92.0 Currently the bulk load API simply imports one HFile at a time. With multi-column-family support, this is inappropriate, since different CFs show up separately. Instead, the IPC endpoint should take a of CF - HFiles, so we can online them all under a single region-wide lock. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-1744) Thrift server to match the new java api.
[
https://issues.apache.org/jira/browse/HBASE-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124663#comment-13124663
]
Ted Yu commented on HBASE-1744:
---
@Bob:
Can you combine your patch with Tim's patch v7 ?
Thanks
Thrift server to match the new java api.
Key: HBASE-1744
URL: https://issues.apache.org/jira/browse/HBASE-1744
Project: HBase
Issue Type: Improvement
Components: thrift
Reporter: Tim Sell
Assignee: Tim Sell
Priority: Critical
Fix For: 0.94.0
Attachments:
0001-thrift2-enable-usage-of-.deleteColumns-for-thrift.patch,
HBASE-1744.2.patch, HBASE-1744.3.patch, HBASE-1744.4.patch,
HBASE-1744.5.patch, HBASE-1744.6.patch, HBASE-1744.7.patch,
HBASE-1744.preview.1.patch, thriftexperiment.patch
This mutateRows, etc.. is a little confusing compared to the new cleaner java
client.
Thinking of ways to make a thrift client that is just as elegant. something
like:
void put(1:Bytes table, 2:TPut put) throws (1:IOError io)
with:
struct TColumn {
1:Bytes family,
2:Bytes qualifier,
3:i64 timestamp
}
struct TPut {
1:Bytes row,
2:mapTColumn, Bytes values
}
This creates more verbose rpc than if the columns in TPut were just
mapBytes, mapBytes, Bytes, but that is harder to fit timestamps into and
still be intuitive from say python.
Presumably the goal of a thrift gateway is to be easy first.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4536) Allow CF to retain deleted rows
[ https://issues.apache.org/jira/browse/HBASE-4536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124665#comment-13124665 ] Lars Hofhansl commented on HBASE-4536: -- That's what I concluded shortly after I posted that. I made the changes, also added a raw option to Scan. All tests pass, including a bunch of new tests. Allow CF to retain deleted rows --- Key: HBASE-4536 URL: https://issues.apache.org/jira/browse/HBASE-4536 Project: HBase Issue Type: Sub-task Components: regionserver Affects Versions: 0.92.0 Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Parent allows for a cluster to retain rows for a TTL or keep a minimum number of versions. However, if a client deletes a row all version older than the delete tomb stone will be remove at the next major compaction (and even at memstore flush - see HBASE-4241). There should be a way to retain those version to guard against software error. I see two options here: 1. Add a new flag HColumnDescriptor. Something like RETAIN_DELETED. 2. Folds this into the parent change. I.e. keep minimum-number-of-versions of versions even past the delete marker. #1 would allow for more flexibility. #2 comes somewhat naturally with parent (from a user viewpoint) Comments? Any other options? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4536) Allow CF to retain deleted rows
[ https://issues.apache.org/jira/browse/HBASE-4536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124667#comment-13124667 ] jirapos...@reviews.apache.org commented on HBASE-4536: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2178/ --- (Updated 2011-10-11 04:02:54.234402) Review request for hbase. Changes --- Added a specific flag for HColumnFamilty to enable this. Family delete markers are now also collected. The way that is done is a bit more complicated than I had hoped. o StoreFile.writer keeps track of the lowest timestamp of any Put seen and stores in the files metadata. o Upon major compaction the lowest of all put timestamps of all involved storefiles is passed to the StoreScanner, which can that remove any family delete older than that timestamp. Tests: ... Results : Tests run: 1033, Failures: 0, Errors: 0, Skipped: 16 Summary (updated) --- HBase timerange Gets and Scans allow to do timetravel in HBase. I.e. look at the state of the data at any point in the past, provided the data is still around. This did not work for deletes, however. Deletes would always mask all puts in the past. This change adds a flag that can be on HColumnDescriptor to enable retention of deleted rows. These rows are still subject to TTL and/or VERSIONS. This changes the following: 1. There is a new flag on HColumnDescriptor enabling that behavior. 2. Allow gets/scans with a timerange to retrieve rows hidden by a delete marker, if the timerange does not include the delete marker. 3. Do not unconditionally collect all deleted rows during a compaction. 4. Allow a raw Scan, which retrieves all delete markers and deleted rows. The change is small'ish, but the logic is intricate, so please review carefully. This addresses bug HBASE-4536. https://issues.apache.org/jira/browse/HBASE-4536 Diffs (updated) - http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 1181286 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Scan.java 1181286 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ColumnTracker.java 1181286 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ExplicitColumnTracker.java 1181286 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 1181286 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanWildcardColumnTracker.java 1181286 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1181286 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1181286 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 1181286 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java 1181286 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestExplicitColumnTracker.java 1181286 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestKeepDeletes.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMinVersions.java 1181286 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWildcardColumnTracker.java 1181286 Diff: https://reviews.apache.org/r/2178/diff Testing (updated) --- All tests pass now. Thanks, Lars Allow CF to retain deleted rows --- Key: HBASE-4536 URL: https://issues.apache.org/jira/browse/HBASE-4536 Project: HBase Issue Type: Sub-task Components: regionserver Affects Versions: 0.92.0 Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Parent allows for a cluster to retain rows for a TTL or keep a minimum number of versions. However, if a client deletes a row all version older than the delete tomb stone will be remove at the next major compaction (and even at memstore flush - see HBASE-4241). There should be a way to retain those version to guard against software error. I see two options here: 1. Add a new flag HColumnDescriptor. Something like RETAIN_DELETED. 2. Folds this into the parent change. I.e. keep minimum-number-of-versions of versions even past the delete marker.
[jira] [Reopened] (HBASE-4540) OpenedRegionHandler is not enforcing atomicity of the operation it is performing
[
https://issues.apache.org/jira/browse/HBASE-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan reopened HBASE-4540:
---
Reopening the issue to backport to 0.90 so that the fix can be available in
0.90 branch also.
OpenedRegionHandler is not enforcing atomicity of the operation it is
performing
Key: HBASE-4540
URL: https://issues.apache.org/jira/browse/HBASE-4540
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.92.0
Attachments: HBASE-4540_1.patch
- OpenedRegionHandler has not yet deleted the znode of the region R1 opened
by RS1.
- RS1 goes down.
- Servershutdownhandler assigns the region R1 to RS2.
- The znode of R1 is moved to OFFLINE state by master or OPENING state by
RS2 if RS2 has started opening the region.
- Now the first OpenedRegionHandler tries to delete the znode thinking its
in OPENED state but fails.
- Though it fails it removes the node from RIT and adds RS1 as the owner of
R1 in master's memory.
- Now when RS2 completes opening the region the master is not able to open
the region as already the reigon has been deleted from RIT.
{code}
Master
==
2011-10-05 20:49:45,301 INFO
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Finished
processing of shutdown of linux146,60020,1317827727647
2011-10-05 20:49:54,177 DEBUG org.apache.hadoop.hbase.master.HMaster: Not
running balancer because 1 region(s) in transition:
{3e69d628a8bd8e9b7c5e7a2a6e03aad9=t1,,1317827883842.3e69d628a8bd8e9b7c5e7a2a6e03aad9.
state=PENDING_OPEN, ts=1317827985272, server=linux76,60020,1317827746847}
2011-10-05 20:49:57,720 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=M_ZK_REGION_OFFLINE, server=linux76,6,1317827742012,
region=3e69d628a8bd8e9b7c5e7a2a6e03aad9
2011-10-05 20:50:14,501 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:6-0x132d3dc13090023 Deleting existing unassigned node for
3e69d628a8bd8e9b7c5e7a2a6e03aad9 that is in expected state RS_ZK_REGION_OPENED
2011-10-05 20:50:14,505 WARN org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:6-0x132d3dc13090023 Attempting to delete unassigned node
3e69d628a8bd8e9b7c5e7a2a6e03aad9 in RS_ZK_REGION_OPENED state but node is in
RS_ZK_REGION_OPENING state
After the region is opened in RS2
=
2011-10-05 20:50:48,066 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=linux76,60020,1317827746847,
region=3e69d628a8bd8e9b7c5e7a2a6e03aad9, which is more than 15 seconds late
2011-10-05 20:50:48,290 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received OPENING for region
3e69d628a8bd8e9b7c5e7a2a6e03aad9 from server linux76,60020,1317827746847 but
region was in the state null and not in expected PENDING_OPEN or OPENING
states
2011-10-05 20:50:53,743 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=linux76,60020,1317827746847,
region=3e69d628a8bd8e9b7c5e7a2a6e03aad9
2011-10-05 20:50:54,182 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor:
Scanned 1 catalog row(s) and gc'd 0 unreferenced parent region(s)
2011-10-05 20:50:54,397 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received OPENING for region
3e69d628a8bd8e9b7c5e7a2a6e03aad9 from server linux76,60020,1317827746847 but
region was in the state null and not in expected PENDING_OPEN or OPENING
states
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4536) Allow CF to retain deleted rows
[ https://issues.apache.org/jira/browse/HBASE-4536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124674#comment-13124674 ] jirapos...@reviews.apache.org commented on HBASE-4536: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2178/ --- (Updated 2011-10-11 04:21:07.770408) Review request for hbase. Changes --- Addressing anticipated nits :) Summary --- HBase timerange Gets and Scans allow to do timetravel in HBase. I.e. look at the state of the data at any point in the past, provided the data is still around. This did not work for deletes, however. Deletes would always mask all puts in the past. This change adds a flag that can be on HColumnDescriptor to enable retention of deleted rows. These rows are still subject to TTL and/or VERSIONS. This changes the following: 1. There is a new flag on HColumnDescriptor enabling that behavior. 2. Allow gets/scans with a timerange to retrieve rows hidden by a delete marker, if the timerange does not include the delete marker. 3. Do not unconditionally collect all deleted rows during a compaction. 4. Allow a raw Scan, which retrieves all delete markers and deleted rows. The change is small'ish, but the logic is intricate, so please review carefully. This addresses bug HBASE-4536. https://issues.apache.org/jira/browse/HBASE-4536 Diffs (updated) - http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 1181616 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Scan.java 1181616 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ColumnTracker.java 1181616 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ExplicitColumnTracker.java 1181616 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 1181616 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanWildcardColumnTracker.java 1181616 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1181616 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1181616 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 1181616 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java 1181616 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestExplicitColumnTracker.java 1181616 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestKeepDeletes.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMinVersions.java 1181616 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWildcardColumnTracker.java 1181616 Diff: https://reviews.apache.org/r/2178/diff Testing --- All tests pass now. Thanks, Lars Allow CF to retain deleted rows --- Key: HBASE-4536 URL: https://issues.apache.org/jira/browse/HBASE-4536 Project: HBase Issue Type: Sub-task Components: regionserver Affects Versions: 0.92.0 Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Parent allows for a cluster to retain rows for a TTL or keep a minimum number of versions. However, if a client deletes a row all version older than the delete tomb stone will be remove at the next major compaction (and even at memstore flush - see HBASE-4241). There should be a way to retain those version to guard against software error. I see two options here: 1. Add a new flag HColumnDescriptor. Something like RETAIN_DELETED. 2. Folds this into the parent change. I.e. keep minimum-number-of-versions of versions even past the delete marker. #1 would allow for more flexibility. #2 comes somewhat naturally with parent (from a user viewpoint) Comments? Any other options? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4480) Testing script to simplfy local testing
[ https://issues.apache.org/jira/browse/HBASE-4480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Kuehn updated HBASE-4480: --- Attachment: HBASE-4480_v3.patch Testing script to simplfy local testing --- Key: HBASE-4480 URL: https://issues.apache.org/jira/browse/HBASE-4480 Project: HBase Issue Type: Improvement Reporter: Jesse Yates Priority: Minor Labels: test Attachments: HBASE-4480.patch, HBASE-4480_v2.patch, HBASE-4480_v3.patch, runtest.sh, runtest2.sh As mentioned by http://search-hadoop.com/m/r2Ab624ES3e and http://search-hadoop.com/m/cZjDH1ykGIA it would be nice if we could have a script that would handle more of the finer points of running/checking our test suite. This script should: (1) Allow people to determine which tests are hanging/taking a long time to run (2) Allow rerunning of particular tests to make sure it wasn't an artifact of running the whole suite that caused the failure (3) Allow people to specify to run just unit tests or also integration tests (essentially wrapping calls to 'maven test' and 'maven verify'). This script should just be a convenience script - running tests directly from maven should not be impacted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4480) Testing script to simplfy local testing
[ https://issues.apache.org/jira/browse/HBASE-4480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124700#comment-13124700 ] Scott Kuehn commented on HBASE-4480: @Ram - fixed an issue where empty lines in the test file would be interpreted as -Dtest='', and thus triggering a full mvn test run. This might have fixed the prob you encountered w/running all tests. v3 patch posted. Testing script to simplfy local testing --- Key: HBASE-4480 URL: https://issues.apache.org/jira/browse/HBASE-4480 Project: HBase Issue Type: Improvement Reporter: Jesse Yates Priority: Minor Labels: test Attachments: HBASE-4480.patch, HBASE-4480_v2.patch, HBASE-4480_v3.patch, runtest.sh, runtest2.sh As mentioned by http://search-hadoop.com/m/r2Ab624ES3e and http://search-hadoop.com/m/cZjDH1ykGIA it would be nice if we could have a script that would handle more of the finer points of running/checking our test suite. This script should: (1) Allow people to determine which tests are hanging/taking a long time to run (2) Allow rerunning of particular tests to make sure it wasn't an artifact of running the whole suite that caused the failure (3) Allow people to specify to run just unit tests or also integration tests (essentially wrapping calls to 'maven test' and 'maven verify'). This script should just be a convenience script - running tests directly from maven should not be impacted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4556) Fix all incorrect uses of InternalScanner.next(...)
[
https://issues.apache.org/jira/browse/HBASE-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124706#comment-13124706
]
stack commented on HBASE-4556:
--
Do our tests need to be more strict Lars?
Fix all incorrect uses of InternalScanner.next(...)
---
Key: HBASE-4556
URL: https://issues.apache.org/jira/browse/HBASE-4556
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
There are cases all over the code where InternalScanner.next(...) is not used
correctly.
I see this a lot:
{code}
while(scanner.next(...)) {
}
{code}
The correct pattern is:
{code}
boolean more = false;
do {
more = scanner.next(...);
} while (more);
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4562) When split doing offlineParentInMeta occurs error,it'll cause data loss
[
https://issues.apache.org/jira/browse/HBASE-4562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124715#comment-13124715
]
bluedavy commented on HBASE-4562:
-
@Ted Yu
I count all tests in src/test,I'm sure all test are ran...
When split doing offlineParentInMeta occurs error,it'll cause data loss
---
Key: HBASE-4562
URL: https://issues.apache.org/jira/browse/HBASE-4562
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.90.4
Reporter: bluedavy
Priority: Blocker
Fix For: 0.90.5
Attachments: HBASE-4562-test.report.txt, HBASE-4562.patch
Follow below steps to replay the problem:
1. change the SplitTransaction.java as below,just like mock the timeout error.
{code:title=SplitTransaction.java|borderStyle=solid}
if (!testing) {
MetaEditor.offlineParentInMeta(server.getCatalogTracker(),
this.parent.getRegionInfo(), a.getRegionInfo(), b.getRegionInfo());
throw new IOException(some unexpected error in split);
}
{code}
2. update the regionserver code,restart;
3. create a table put some data to the table;
4. split the table;
5. kill the regionserver hosted the table;
6. wait some time after master ServerShutdownHandler.process execute,then
scan the table,u'll find the data wrote before lost.
We can fix the bug just use below code:
{code:title=SplitTransaction.java|borderStyle=solid}
this.journal.add(JournalEntry.PONR);
if (!testing) {
MetaEditor.offlineParentInMeta(server.getCatalogTracker(),
this.parent.getRegionInfo(), a.getRegionInfo(), b.getRegionInfo());
throw new IOException(some unexpected error in split);
}
{code}
{code:title=CompactSplitThread.java|borderStyle=solid}
if (st.rollback(this.server, this.server)) {
LOG.info(Successful rollback of failed split of +
parent.getRegionNameAsString());
}
else {
this.server.abort(Abort; we got an error after
point-of-no-return);
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4563) When split doing this.parent.close(false) occurs error,it'll cause the splited region cann't write read
[
https://issues.apache.org/jira/browse/HBASE-4563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
bluedavy updated HBASE-4563:
Attachment: HBASE-4563.patch
When split doing this.parent.close(false) occurs error,it'll cause the
splited region cann't write read
-
Key: HBASE-4563
URL: https://issues.apache.org/jira/browse/HBASE-4563
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.90.4
Reporter: bluedavy
Priority: Blocker
Fix For: 0.90.5
Attachments: HBASE-4563-test.report.txt, HBASE-4563.patch
Follow below steps to replay the problem:
1. change the SplitTransaction.java as below,just like mock the hdfs error.
{code:title=SplitTransaction.java|borderStyle=solid}
ListStoreFile hstoreFilesToSplit = this.parent.close(false);
throw new IOException(some unexpected error in close store files);
{code}
2. update the regionserver code,restart;
3. create a table put some data to the table;
4. split the table;
5. scan the table,then it'll fail.
We can fix the bug just use below code:
{code:title=SplitTransaction.java|borderStyle=solid}
ListStoreFile hstoreFilesToSplit = null;
try{
hstoreFilesToSplit = this.parent.close(false);
}
catch(IOException e){
this.journal.add(JournalEntry.CLOSED_PARENT_REGION);
throw e;
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4563) When split doing this.parent.close(false) occurs error,it'll cause the splited region cann't write read
[
https://issues.apache.org/jira/browse/HBASE-4563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
bluedavy updated HBASE-4563:
Attachment: (was: HBASE-4563.patch)
When split doing this.parent.close(false) occurs error,it'll cause the
splited region cann't write read
-
Key: HBASE-4563
URL: https://issues.apache.org/jira/browse/HBASE-4563
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.90.4
Reporter: bluedavy
Priority: Blocker
Fix For: 0.90.5
Attachments: HBASE-4563-test.report.txt, HBASE-4563.patch
Follow below steps to replay the problem:
1. change the SplitTransaction.java as below,just like mock the hdfs error.
{code:title=SplitTransaction.java|borderStyle=solid}
ListStoreFile hstoreFilesToSplit = this.parent.close(false);
throw new IOException(some unexpected error in close store files);
{code}
2. update the regionserver code,restart;
3. create a table put some data to the table;
4. split the table;
5. scan the table,then it'll fail.
We can fix the bug just use below code:
{code:title=SplitTransaction.java|borderStyle=solid}
ListStoreFile hstoreFilesToSplit = null;
try{
hstoreFilesToSplit = this.parent.close(false);
}
catch(IOException e){
this.journal.add(JournalEntry.CLOSED_PARENT_REGION);
throw e;
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4563) When split doing this.parent.close(false) occurs error,it'll cause the splited region cann't write read
[
https://issues.apache.org/jira/browse/HBASE-4563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
bluedavy updated HBASE-4563:
Description:
Follow below steps to replay the problem:
1. change the SplitTransaction.java as below,just like mock the hdfs error.
{code:title=SplitTransaction.java|borderStyle=solid}
ListStoreFile hstoreFilesToSplit = this.parent.close(false);
throw new IOException(some unexpected error in close store files);
{code}
2. update the regionserver code,restart;
3. create a table put some data to the table;
4. split the table;
5. scan the table,then it'll fail.
We can fix the bug just use the patch.
was:
Follow below steps to replay the problem:
1. change the SplitTransaction.java as below,just like mock the hdfs error.
{code:title=SplitTransaction.java|borderStyle=solid}
ListStoreFile hstoreFilesToSplit = this.parent.close(false);
throw new IOException(some unexpected error in close store files);
{code}
2. update the regionserver code,restart;
3. create a table put some data to the table;
4. split the table;
5. scan the table,then it'll fail.
We can fix the bug just use below code:
{code:title=SplitTransaction.java|borderStyle=solid}
ListStoreFile hstoreFilesToSplit = null;
try{
hstoreFilesToSplit = this.parent.close(false);
}
catch(IOException e){
this.journal.add(JournalEntry.CLOSED_PARENT_REGION);
throw e;
}
{code}
When split doing this.parent.close(false) occurs error,it'll cause the
splited region cann't write read
-
Key: HBASE-4563
URL: https://issues.apache.org/jira/browse/HBASE-4563
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.90.4
Reporter: bluedavy
Priority: Blocker
Fix For: 0.90.5
Attachments: HBASE-4563-test.report.txt, HBASE-4563.patch
Follow below steps to replay the problem:
1. change the SplitTransaction.java as below,just like mock the hdfs error.
{code:title=SplitTransaction.java|borderStyle=solid}
ListStoreFile hstoreFilesToSplit = this.parent.close(false);
throw new IOException(some unexpected error in close store files);
{code}
2. update the regionserver code,restart;
3. create a table put some data to the table;
4. split the table;
5. scan the table,then it'll fail.
We can fix the bug just use the patch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4562) When split doing offlineParentInMeta occurs error,it'll cause data loss
[
https://issues.apache.org/jira/browse/HBASE-4562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
bluedavy updated HBASE-4562:
Description:
Follow below steps to replay the problem:
1. change the SplitTransaction.java as below,just like mock the timeout error.
{code:title=SplitTransaction.java|borderStyle=solid}
if (!testing) {
MetaEditor.offlineParentInMeta(server.getCatalogTracker(),
this.parent.getRegionInfo(), a.getRegionInfo(), b.getRegionInfo());
throw new IOException(some unexpected error in split);
}
{code}
2. update the regionserver code,restart;
3. create a table put some data to the table;
4. split the table;
5. kill the regionserver hosted the table;
6. wait some time after master ServerShutdownHandler.process execute,then scan
the table,u'll find the data wrote before lost.
We can fix the bug just use the patch.
was:
Follow below steps to replay the problem:
1. change the SplitTransaction.java as below,just like mock the timeout error.
{code:title=SplitTransaction.java|borderStyle=solid}
if (!testing) {
MetaEditor.offlineParentInMeta(server.getCatalogTracker(),
this.parent.getRegionInfo(), a.getRegionInfo(), b.getRegionInfo());
throw new IOException(some unexpected error in split);
}
{code}
2. update the regionserver code,restart;
3. create a table put some data to the table;
4. split the table;
5. kill the regionserver hosted the table;
6. wait some time after master ServerShutdownHandler.process execute,then scan
the table,u'll find the data wrote before lost.
We can fix the bug just use below code:
{code:title=SplitTransaction.java|borderStyle=solid}
this.journal.add(JournalEntry.PONR);
if (!testing) {
MetaEditor.offlineParentInMeta(server.getCatalogTracker(),
this.parent.getRegionInfo(), a.getRegionInfo(), b.getRegionInfo());
throw new IOException(some unexpected error in split);
}
{code}
{code:title=CompactSplitThread.java|borderStyle=solid}
if (st.rollback(this.server, this.server)) {
LOG.info(Successful rollback of failed split of +
parent.getRegionNameAsString());
}
else {
this.server.abort(Abort; we got an error after point-of-no-return);
}
{code}
When split doing offlineParentInMeta occurs error,it'll cause data loss
---
Key: HBASE-4562
URL: https://issues.apache.org/jira/browse/HBASE-4562
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.90.4
Reporter: bluedavy
Priority: Blocker
Fix For: 0.90.5
Attachments: HBASE-4562-test.report.txt, HBASE-4562.patch
Follow below steps to replay the problem:
1. change the SplitTransaction.java as below,just like mock the timeout error.
{code:title=SplitTransaction.java|borderStyle=solid}
if (!testing) {
MetaEditor.offlineParentInMeta(server.getCatalogTracker(),
this.parent.getRegionInfo(), a.getRegionInfo(), b.getRegionInfo());
throw new IOException(some unexpected error in split);
}
{code}
2. update the regionserver code,restart;
3. create a table put some data to the table;
4. split the table;
5. kill the regionserver hosted the table;
6. wait some time after master ServerShutdownHandler.process execute,then
scan the table,u'll find the data wrote before lost.
We can fix the bug just use the patch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
