[jira] [Commented] (HBASE-8067) TestHFileArchiving.testArchiveOnTableDelete sometimes fails
[
https://issues.apache.org/jira/browse/HBASE-8067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13661806#comment-13661806
]
Hadoop QA commented on HBASE-8067:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12583792/8067-trunk.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5746//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5746//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5746//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5746//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5746//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5746//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5746//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5746//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5746//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5746//console
This message is automatically generated.
TestHFileArchiving.testArchiveOnTableDelete sometimes fails
---
Key: HBASE-8067
URL: https://issues.apache.org/jira/browse/HBASE-8067
Project: HBase
Issue Type: Bug
Components: Admin, master, test
Affects Versions: 0.94.6, 0.95.2
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Fix For: 0.98.0, 0.94.8, 0.95.1
Attachments: 8067-0.94.txt, 8067-trunk.txt, HBASE-8067-debug.patch,
HBASE-8067-v0.patch
it seems that testArchiveOnTableDelete() fails because the archiving in
DeleteTableHandler is still in progress when admin.deleteTable() returns.
{code}
Error Message
Archived files are missing some of the store files!
Stacktrace
java.lang.AssertionError: Archived files are missing some of the store files!
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.hbase.backup.TestHFileArchiving.testArchiveOnTableDelete(TestHFileArchiving.java:262)
{code}
(Looking at the problem in a more generic way, we don't have any way to
inform the client when an async operation is completed)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5844) Delete the region servers znode after a regions server crash
[ https://issues.apache.org/jira/browse/HBASE-5844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13661824#comment-13661824 ] Liang Lee commented on HBASE-5844: -- Hi,stack ,Could you please provide some document about how to use this pach like HBASE-7404? Where does the core configution HBASE_ZNODE_FILE should be configured? Thanks Delete the region servers znode after a regions server crash Key: HBASE-5844 URL: https://issues.apache.org/jira/browse/HBASE-5844 Project: HBase Issue Type: Improvement Components: regionserver, scripts Affects Versions: 0.95.2 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.95.0 Attachments: 5844.v1.patch, 5844.v2.patch, 5844.v3.patch, 5844.v3.patch, 5844.v4.patch today, if the regions server crashes, its znode is not deleted in ZooKeeper. So the recovery process will stop only after a timeout, usually 30s. By deleting the znode in start script, we remove this delay and the recovery starts immediately. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8555) FilterList correctness was dominated by sub-filter(list) ordering randomly
[ https://issues.apache.org/jira/browse/HBASE-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13661843#comment-13661843 ] Liang Xie commented on HBASE-8555: -- bq. How about call the Rowfilter#filterRowKey() in the Rowfilter#filterKeyValue() if it haven't been called. em, good suggestion, sound great for me:) FilterList correctness was dominated by sub-filter(list) ordering randomly -- Key: HBASE-8555 URL: https://issues.apache.org/jira/browse/HBASE-8555 Project: HBase Issue Type: Bug Components: Filters Affects Versions: 0.94.3 Reporter: Liang Xie Assignee: Liang Xie Priority: Critical Attachments: 8555-trunk-v1.txt, HBASE-8555-0.94.txt say, ther're 10 rows, column value is i%2: row0 0 row1 1 row2 0 row3 1 row4 0 row5 1 row6 0 row7 1 row8 0 row9 1 1: filter : row filter row4 === row5 row6 row7 row8 row9 2: subFilterList: row filter = row4 column==0=== row0 row2 row4 3.1 filterlist[expected] filter || subFilterList === row0 row2 row4 row5 row6 row7 row8 row9 3.2 filterlist[BUGON!] subFilterList || filter === row0 row1 row2 row3 row4 row5 row6 row7 row8 row9 (Please refer to the new testNestedFilterListWithSCVF case) It was found when i managed to transform the following SQL into HBase scan statement: select xxx from xxx where (pk = xxx and column1 = xxx) or pk xxx My finding is that we had an assumption for filter methods call sequence: e.g. filterRowKey() should be called before filterKeyValue(). and the orignial filterList.filterRowKey impl broke it due to fast short-circuit returning. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8555) FilterList correctness was dominated by sub-filter(list) ordering randomly
[ https://issues.apache.org/jira/browse/HBASE-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13661842#comment-13661842 ] Liang Xie commented on HBASE-8555: -- bq. Patch makes sense. Good example and a very good testcase. So you will iterate the sublist tree so that all the filters that has filterRowKey() passes through and finally decide if to include it or no yes, absolutely correct:) FilterList correctness was dominated by sub-filter(list) ordering randomly -- Key: HBASE-8555 URL: https://issues.apache.org/jira/browse/HBASE-8555 Project: HBase Issue Type: Bug Components: Filters Affects Versions: 0.94.3 Reporter: Liang Xie Assignee: Liang Xie Priority: Critical Attachments: 8555-trunk-v1.txt, HBASE-8555-0.94.txt say, ther're 10 rows, column value is i%2: row0 0 row1 1 row2 0 row3 1 row4 0 row5 1 row6 0 row7 1 row8 0 row9 1 1: filter : row filter row4 === row5 row6 row7 row8 row9 2: subFilterList: row filter = row4 column==0=== row0 row2 row4 3.1 filterlist[expected] filter || subFilterList === row0 row2 row4 row5 row6 row7 row8 row9 3.2 filterlist[BUGON!] subFilterList || filter === row0 row1 row2 row3 row4 row5 row6 row7 row8 row9 (Please refer to the new testNestedFilterListWithSCVF case) It was found when i managed to transform the following SQL into HBase scan statement: select xxx from xxx where (pk = xxx and column1 = xxx) or pk xxx My finding is that we had an assumption for filter methods call sequence: e.g. filterRowKey() should be called before filterKeyValue(). and the orignial filterList.filterRowKey impl broke it due to fast short-circuit returning. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8559) increase unit-test coverage of package org.apache.hadoop.hbase.coprocessor
[ https://issues.apache.org/jira/browse/HBASE-8559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan A. Veselovsky updated HBASE-8559: -- Affects Version/s: 0.95.2 0.94.8 0.98.0 Status: Patch Available (was: Open) increase unit-test coverage of package org.apache.hadoop.hbase.coprocessor -- Key: HBASE-8559 URL: https://issues.apache.org/jira/browse/HBASE-8559 Project: HBase Issue Type: Test Affects Versions: 0.98.0, 0.94.8, 0.95.2 Reporter: Ivan A. Veselovsky Attachments: HBASE-8559-0.94--N2.patch, HBASE-8559-trunk--N2.patch increase unit-test coverage of package org.apache.hadoop.hbase.coprocessor up to 80%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8559) increase unit-test coverage of package org.apache.hadoop.hbase.coprocessor
[ https://issues.apache.org/jira/browse/HBASE-8559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan A. Veselovsky updated HBASE-8559: -- Attachment: HBASE-8559-trunk--N2.patch HBASE-8559-0.94--N2.patch HBASE-8559-trunk--N2.patch is also applicable to branch 0.95. increase unit-test coverage of package org.apache.hadoop.hbase.coprocessor -- Key: HBASE-8559 URL: https://issues.apache.org/jira/browse/HBASE-8559 Project: HBase Issue Type: Test Reporter: Ivan A. Veselovsky Attachments: HBASE-8559-0.94--N2.patch, HBASE-8559-trunk--N2.patch increase unit-test coverage of package org.apache.hadoop.hbase.coprocessor up to 80%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8559) increase unit-test coverage of package org.apache.hadoop.hbase.coprocessor
[
https://issues.apache.org/jira/browse/HBASE-8559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13661920#comment-13661920
]
Hadoop QA commented on HBASE-8559:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12583815/HBASE-8559-trunk--N2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestTableLockManager
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5748//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5748//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5748//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5748//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5748//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5748//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5748//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5748//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5748//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5748//console
This message is automatically generated.
increase unit-test coverage of package org.apache.hadoop.hbase.coprocessor
--
Key: HBASE-8559
URL: https://issues.apache.org/jira/browse/HBASE-8559
Project: HBase
Issue Type: Test
Affects Versions: 0.98.0, 0.94.8, 0.95.2
Reporter: Ivan A. Veselovsky
Attachments: HBASE-8559-0.94--N2.patch, HBASE-8559-trunk--N2.patch
increase unit-test coverage of package org.apache.hadoop.hbase.coprocessor up
to 80%.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8471) Server-side, remove convertion from pb type to client type before we call method
[ https://issues.apache.org/jira/browse/HBASE-8471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-8471: -- Attachment: HBASE-8471.patch Server-side, remove convertion from pb type to client type before we call method Key: HBASE-8471 URL: https://issues.apache.org/jira/browse/HBASE-8471 Project: HBase Issue Type: Improvement Components: IPC/RPC, regionserver Reporter: stack Assignee: Anoop Sam John Priority: Critical Attachments: HBASE-8471.patch In the regionserver, when the rpc receives a call, the call is described using protobufs. Before we make the server-side invocation, we do a transform on the pb param objects to make a native pojo -- e.g. from a pb Puts into an hbase o.a.h.h.client.Put -- and only then do we make the call against the server. On the way out, similar, before putting the result on the wire, we will do a convertion from o.a.h.h.client.Result into pb Result. This issue is about our first INVESTIGATING if it is possible to do away w/ this marshalling/unmarshalling serverside especially given the pb objects themselves are rich in accessor and getters, etc. If it is possible to do w/ pbs alone serverside, then we should go ahead and rip out all the serverside convertions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8471) Server-side, remove convertion from pb type to client type before we call method
[ https://issues.apache.org/jira/browse/HBASE-8471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13661934#comment-13661934 ] Anoop Sam John commented on HBASE-8471: --- Patch with avoiding Result copying. Will see later how/whether we can change Mutations case also. With PerformanceEvaluation tool scanRange1000, I am able to see 10-15% reduction in latency with patch.(Just 1 client running.) And for this there is no change needed in any CP hooks. Server-side, remove convertion from pb type to client type before we call method Key: HBASE-8471 URL: https://issues.apache.org/jira/browse/HBASE-8471 Project: HBase Issue Type: Improvement Components: IPC/RPC, regionserver Reporter: stack Assignee: Anoop Sam John Priority: Critical Attachments: HBASE-8471.patch In the regionserver, when the rpc receives a call, the call is described using protobufs. Before we make the server-side invocation, we do a transform on the pb param objects to make a native pojo -- e.g. from a pb Puts into an hbase o.a.h.h.client.Put -- and only then do we make the call against the server. On the way out, similar, before putting the result on the wire, we will do a convertion from o.a.h.h.client.Result into pb Result. This issue is about our first INVESTIGATING if it is possible to do away w/ this marshalling/unmarshalling serverside especially given the pb objects themselves are rich in accessor and getters, etc. If it is possible to do w/ pbs alone serverside, then we should go ahead and rip out all the serverside convertions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8471) Server-side, remove convertion from pb type to client type before we call method
[ https://issues.apache.org/jira/browse/HBASE-8471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-8471: -- Affects Version/s: 0.95.0 Status: Patch Available (was: Open) Server-side, remove convertion from pb type to client type before we call method Key: HBASE-8471 URL: https://issues.apache.org/jira/browse/HBASE-8471 Project: HBase Issue Type: Improvement Components: IPC/RPC, regionserver Affects Versions: 0.95.0 Reporter: stack Assignee: Anoop Sam John Priority: Critical Attachments: HBASE-8471.patch In the regionserver, when the rpc receives a call, the call is described using protobufs. Before we make the server-side invocation, we do a transform on the pb param objects to make a native pojo -- e.g. from a pb Puts into an hbase o.a.h.h.client.Put -- and only then do we make the call against the server. On the way out, similar, before putting the result on the wire, we will do a convertion from o.a.h.h.client.Result into pb Result. This issue is about our first INVESTIGATING if it is possible to do away w/ this marshalling/unmarshalling serverside especially given the pb objects themselves are rich in accessor and getters, etc. If it is possible to do w/ pbs alone serverside, then we should go ahead and rip out all the serverside convertions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8471) Server-side, remove convertion from pb type to client type before we call method
[
https://issues.apache.org/jira/browse/HBASE-8471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13661998#comment-13661998
]
Hadoop QA commented on HBASE-8471:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12583825/HBASE-8471.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.master.TestAssignmentManager.testSSHTimesOutOpeningRegionTransition(TestAssignmentManager.java:967)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5749//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5749//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5749//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5749//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5749//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5749//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5749//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5749//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5749//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5749//console
This message is automatically generated.
Server-side, remove convertion from pb type to client type before we call
method
Key: HBASE-8471
URL: https://issues.apache.org/jira/browse/HBASE-8471
Project: HBase
Issue Type: Improvement
Components: IPC/RPC, regionserver
Affects Versions: 0.95.0
Reporter: stack
Assignee: Anoop Sam John
Priority: Critical
Attachments: HBASE-8471.patch
In the regionserver, when the rpc receives a call, the call is described
using protobufs. Before we make the server-side invocation, we do a
transform on the pb param objects to make a native pojo -- e.g. from a pb
Puts into an hbase o.a.h.h.client.Put -- and only then do we make the call
against the server.
On the way out, similar, before putting the result on the wire, we will do a
convertion from o.a.h.h.client.Result into pb Result.
This issue is about our first INVESTIGATING if it is possible to do away w/
this marshalling/unmarshalling serverside especially given the pb objects
themselves are rich in accessor and getters, etc. If it is possible to do w/
pbs alone serverside, then we should go ahead and rip out all the serverside
convertions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8450) Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc.
[ https://issues.apache.org/jira/browse/HBASE-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662095#comment-13662095 ] stack commented on HBASE-8450: -- [~lhofhansl] On delete marker never getting deleted, my thinking is that this would not be the end of the world and second, frequently compactions are promoted to major compaction because all files are selected so deletes would get purged (The general call is that disabling major compactions will be overall better than retention of deletes; do you disagree w/ this?). On the suggestion that FAST_DIFF slows down scan, let me put up a new patch that does not enable it; your comment puts it into that bucket of configs that we should change only after we do some testing. Let me make a new patch. Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc. Key: HBASE-8450 URL: https://issues.apache.org/jira/browse/HBASE-8450 Project: HBase Issue Type: Task Components: Usability Reporter: stack Assignee: stack Priority: Critical Fix For: 0.95.1 Attachments: 8450.txt, 8450v2.txt This is a critical task we need to do before we release; review our defaults. On cursory review, there are configs in hbase-default.xml that no longer have matching code; there are some that should be changed because we know better now and others that should be amended because hardware and deploys are bigger than they used to be. We could also move stuff around so that the must-edit stuff is near the top (zk quorum config. is mid-way down the page) and beef up the descriptions -- especially since these descriptions shine through in the refguide. Lastly, I notice that our tgz does not include an hbase-default.xml other than the one bundled up in the jar. Maybe we should make it more accessible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8573) Store last flushed sequence id for each store of region for Distributed Log Replay
[ https://issues.apache.org/jira/browse/HBASE-8573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662103#comment-13662103 ] stack commented on HBASE-8573: -- bq. Distributed Log Replay at this stage is experimental. No it is not. We want it enabled as the default for 0.95/trunk. [~jeffreyz] What you think of [~zjushch]'s note above? On [~sershe]'s comment, the answer is if the cluster crashed and all zk data was erased, the end result would be an overplay of edits only. In other words, we can 'safely' lose what is noted in zk; we'll just do more work if we don't have it. Store last flushed sequence id for each store of region for Distributed Log Replay -- Key: HBASE-8573 URL: https://issues.apache.org/jira/browse/HBASE-8573 Project: HBase Issue Type: Improvement Reporter: Ted Yu HBASE-7006 stores last flushed sequence id of the region in zookeeper. To prevent deleted data from appearing again, we should store last flushed sequence id for each store of region in zookeeper. See discussion here: https://issues.apache.org/jira/browse/HBASE-7006?focusedCommentId=13660428page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13660428 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8471) Server-side, remove convertion from pb type to client type before we call method
[ https://issues.apache.org/jira/browse/HBASE-8471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662060#comment-13662060 ] stack commented on HBASE-8471: -- This patch looks great. Small change. Nice improvement. Good on you [~anoop.hbase]. Want to retry the hadoopqa? Server-side, remove convertion from pb type to client type before we call method Key: HBASE-8471 URL: https://issues.apache.org/jira/browse/HBASE-8471 Project: HBase Issue Type: Improvement Components: IPC/RPC, regionserver Affects Versions: 0.95.0 Reporter: stack Assignee: Anoop Sam John Priority: Critical Attachments: HBASE-8471.patch In the regionserver, when the rpc receives a call, the call is described using protobufs. Before we make the server-side invocation, we do a transform on the pb param objects to make a native pojo -- e.g. from a pb Puts into an hbase o.a.h.h.client.Put -- and only then do we make the call against the server. On the way out, similar, before putting the result on the wire, we will do a convertion from o.a.h.h.client.Result into pb Result. This issue is about our first INVESTIGATING if it is possible to do away w/ this marshalling/unmarshalling serverside especially given the pb objects themselves are rich in accessor and getters, etc. If it is possible to do w/ pbs alone serverside, then we should go ahead and rip out all the serverside convertions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8522) Archived hfiles and old hlogs may be deleted immediatly by HFileCleaner, LogCleaner in HMaster
[ https://issues.apache.org/jira/browse/HBASE-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-8522: - Attachment: HBASE-8522-trunk-v2.patch Retry hadoopqa Archived hfiles and old hlogs may be deleted immediatly by HFileCleaner, LogCleaner in HMaster -- Key: HBASE-8522 URL: https://issues.apache.org/jira/browse/HBASE-8522 Project: HBase Issue Type: Bug Affects Versions: 0.94.3 Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Attachments: HBASE-8522-0.94.patch, HBASE-8522-0.94-v2.patch, HBASE-8522-trunk.patch, HBASE-8522-trunk-v2.patch, HBASE-8522-trunk-v2.patch TimeToLiveHFileCleaner is configed to 'hbase.master.hfilecleaner.plugins' in hbase-default.xml. And timeToLiveHFileCleaner uses the modify time of the hfile to determine if it should be deleted. But, the modify time of the hdfs file is time when its writer is closed. The rename op will not change the modify time of the hfile. So the hfile may be deleted immediatly by HFileCleaner after it is moved to archives. See log: 2013-05-08 08:15:46,053 DEBUG org.apache.hadoop.hbase.master.cleaner.CleanerChore: Checking directory: hdfs://hbase/.archive/table/4e48ffc1ec089082c66e6d1b5f018fb5/M/729e8bc1430540cb9b2c147c90039cdc 2013-05-08 08:15:46,055 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine: Call: getFileInfo took 1ms 2013-05-08 08:15:46,055 DEBUG org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner: Life:40033567, ttl:30, current:1367972146054, from: 1367932112487 2013-05-08 08:15:46,055 DEBUG org.apache.hadoop.hbase.master.cleaner.CleanerChore: Removing:hdfs://hbase/.archive/table/4e48ffc1ec089082c66e6d1b5f018fb5/M/729e8bc1430540cb9b2c147c90039cdc from archive The same to old hlogs. And my solution is very simple: When hfiles and hlogs are archived, we set the modify time of files after rename. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8522) Archived hfiles and old hlogs may be deleted immediatly by HFileCleaner, LogCleaner in HMaster
[ https://issues.apache.org/jira/browse/HBASE-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662083#comment-13662083 ] Matteo Bertozzi commented on HBASE-8522: Hey [~lhofhansl] does my answer make sense to you? are you ok with the patches? Archived hfiles and old hlogs may be deleted immediatly by HFileCleaner, LogCleaner in HMaster -- Key: HBASE-8522 URL: https://issues.apache.org/jira/browse/HBASE-8522 Project: HBase Issue Type: Bug Affects Versions: 0.94.3 Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Attachments: HBASE-8522-0.94.patch, HBASE-8522-0.94-v2.patch, HBASE-8522-trunk.patch, HBASE-8522-trunk-v2.patch TimeToLiveHFileCleaner is configed to 'hbase.master.hfilecleaner.plugins' in hbase-default.xml. And timeToLiveHFileCleaner uses the modify time of the hfile to determine if it should be deleted. But, the modify time of the hdfs file is time when its writer is closed. The rename op will not change the modify time of the hfile. So the hfile may be deleted immediatly by HFileCleaner after it is moved to archives. See log: 2013-05-08 08:15:46,053 DEBUG org.apache.hadoop.hbase.master.cleaner.CleanerChore: Checking directory: hdfs://hbase/.archive/table/4e48ffc1ec089082c66e6d1b5f018fb5/M/729e8bc1430540cb9b2c147c90039cdc 2013-05-08 08:15:46,055 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine: Call: getFileInfo took 1ms 2013-05-08 08:15:46,055 DEBUG org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner: Life:40033567, ttl:30, current:1367972146054, from: 1367932112487 2013-05-08 08:15:46,055 DEBUG org.apache.hadoop.hbase.master.cleaner.CleanerChore: Removing:hdfs://hbase/.archive/table/4e48ffc1ec089082c66e6d1b5f018fb5/M/729e8bc1430540cb9b2c147c90039cdc from archive The same to old hlogs. And my solution is very simple: When hfiles and hlogs are archived, we set the modify time of files after rename. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8067) TestHFileArchiving.testArchiveOnTableDelete sometimes fails
[
https://issues.apache.org/jira/browse/HBASE-8067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-8067:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed Lars' change to the test to 0.94 and to trunk. I don't think this
fixes all issues w/ this test but will open new issue for subsequent
TestHFileArchiving failures... this one has enough going on in it already.
TestHFileArchiving.testArchiveOnTableDelete sometimes fails
---
Key: HBASE-8067
URL: https://issues.apache.org/jira/browse/HBASE-8067
Project: HBase
Issue Type: Bug
Components: Admin, master, test
Affects Versions: 0.94.6, 0.95.2
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Fix For: 0.98.0, 0.94.8, 0.95.1
Attachments: 8067-0.94.txt, 8067-trunk.txt, HBASE-8067-debug.patch,
HBASE-8067-v0.patch
it seems that testArchiveOnTableDelete() fails because the archiving in
DeleteTableHandler is still in progress when admin.deleteTable() returns.
{code}
Error Message
Archived files are missing some of the store files!
Stacktrace
java.lang.AssertionError: Archived files are missing some of the store files!
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.hbase.backup.TestHFileArchiving.testArchiveOnTableDelete(TestHFileArchiving.java:262)
{code}
(Looking at the problem in a more generic way, we don't have any way to
inform the client when an async operation is completed)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8450) Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc.
[ https://issues.apache.org/jira/browse/HBASE-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-8450: - Attachment: 8450v3.txt Undid our setting of FAST_DIFF and set max versions to 1 as default. Here is complete list: + max versions now 1 instead of 3 + row blooms on by default + handlers 30 instead of 10 + upped memstore lower limit from .35 to .38 + zookeeper timeout default is 90seconds instead of 180 + client pause is 100ms instead of 1000ms + retries are now 20 instead of 10 (so overall we still wait same amount of time) + bulkload retries is 10 instead of infinite + major compactions are off + blockingstorefiles is 10 instead of 7 + block cache is 0.4 instead of 0.25 + rpc timeout is 10seconds instead of 60seconds Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc. Key: HBASE-8450 URL: https://issues.apache.org/jira/browse/HBASE-8450 Project: HBase Issue Type: Task Components: Usability Reporter: stack Assignee: stack Priority: Critical Fix For: 0.95.1 Attachments: 8450.txt, 8450v2.txt, 8450v3.txt This is a critical task we need to do before we release; review our defaults. On cursory review, there are configs in hbase-default.xml that no longer have matching code; there are some that should be changed because we know better now and others that should be amended because hardware and deploys are bigger than they used to be. We could also move stuff around so that the must-edit stuff is near the top (zk quorum config. is mid-way down the page) and beef up the descriptions -- especially since these descriptions shine through in the refguide. Lastly, I notice that our tgz does not include an hbase-default.xml other than the one bundled up in the jar. Maybe we should make it more accessible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8282) User triggered flushes does not allow compaction to get triggered even if compaction criteria is met
[
https://issues.apache.org/jira/browse/HBASE-8282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662123#comment-13662123
]
ramkrishna.s.vasudevan commented on HBASE-8282:
---
Committed to trunk and 0.95 Lars.
User triggered flushes does not allow compaction to get triggered even if
compaction criteria is met
Key: HBASE-8282
URL: https://issues.apache.org/jira/browse/HBASE-8282
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.98.0, 0.94.8, 0.95.1
Attachments: HBASE-8282_trunk_1.patch, HBASE-8282_trunk.patch
Observe the below logs
{code}
2013-04-04 17:43:45,825 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Flushing
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9.
2013-04-04 17:43:45,826 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Started memstore flush for
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9., current region
memstore size 42.3 M
2013-04-04 17:43:45,826 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished snapshotting
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9., commencing wait for
mvcc, flushsize=44388936
2013-04-04 17:43:45,826 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished snapshotting, commencing flushing stores
2013-04-04 17:43:45,831 DEBUG org.apache.hadoop.hbase.util.FSUtils: Creating
file=hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/.tmp/7020db24b38246a5818e7eb4e130ad9e
with permission=rwxrwxrwx
2013-04-04 17:43:45,834 DEBUG org.apache.hadoop.hbase.io.hfile.HFileWriterV2:
Initialized with CacheConfig:enabled [cacheDataOnRead=true]
[cacheDataOnWrite=false] [cacheIndexesOnWrite=false]
[cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
2013-04-04 17:43:45,835 INFO org.apache.hadoop.hbase.regionserver.StoreFile:
Delete Family Bloom filter type for
hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/.tmp/7020db24b38246a5818e7eb4e130ad9e:
CompoundBloomFilterWriter
2013-04-04 17:43:46,074 INFO org.apache.hadoop.hbase.regionserver.StoreFile:
NO General Bloom and NO DeleteFamily was added to HFile
(hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/.tmp/7020db24b38246a5818e7eb4e130ad9e)
2013-04-04 17:43:46,074 INFO org.apache.hadoop.hbase.regionserver.HStore:
Flushed , sequenceid=1051817, memsize=42.3 M, into tmp file
hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/.tmp/7020db24b38246a5818e7eb4e130ad9e
2013-04-04 17:43:46,093 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionFileSystem: Committing store file
hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/.tmp/7020db24b38246a5818e7eb4e130ad9e
as
hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/f1/7020db24b38246a5818e7eb4e130ad9e
2013-04-04 17:43:46,103 INFO org.apache.hadoop.hbase.regionserver.HStore:
Added
hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/f1/7020db24b38246a5818e7eb4e130ad9e,
entries=54911, sequenceid=1051817, filesize=35.1 M
2013-04-04 17:43:46,103 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Finished memstore flush of ~42.3 M/44388936, currentsize=0/0 for region
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9. in 278ms,
sequenceid=-1, compaction requested=true
2013-04-04 17:43:56,335 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache:
Stats: total=394.09 MB, free=3.61 GB, max=4.00 GB, blocks=5263,
accesses=244882, hits=42988, hitRatio=17.55%, , cachingAccesses=49869,
cachingHits=24251, cachingHitsRatio=48.63%, , evictions=0, evicted=20349,
evictedPerRun=Infinity
2013-04-04 17:44:31,119 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Flushing
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9.
2013-04-04 17:44:31,120 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Started memstore flush for
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9., current region
memstore size 42.7 M
2013-04-04 17:44:31,120 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished snapshotting
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9., commencing wait for
mvcc, flushsize=44764248
2013-04-04 17:44:31,120 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished snapshotting, commencing flushing stores
2013-04-04 17:44:31,136 DEBUG org.apache.hadoop.hbase.util.FSUtils: Creating
file=hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/.tmp/c5c2cd6aaf1644daa9c858149da39081
with permission=rwxrwxrwx
2013-04-04 17:44:31,139 DEBUG
[jira] [Created] (HBASE-8578) Improve flushCache to distinguish flush and compaction needed scenarios
ramkrishna.s.vasudevan created HBASE-8578: - Summary: Improve flushCache to distinguish flush and compaction needed scenarios Key: HBASE-8578 URL: https://issues.apache.org/jira/browse/HBASE-8578 Project: HBase Issue Type: Improvement Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Priority: Minor Fix For: 0.98.0, 0.95.1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8282) User triggered flushes does not allow compaction to get triggered even if compaction criteria is met
[
https://issues.apache.org/jira/browse/HBASE-8282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662127#comment-13662127
]
ramkrishna.s.vasudevan commented on HBASE-8282:
---
Raised HBASE-8578 for improvements.
User triggered flushes does not allow compaction to get triggered even if
compaction criteria is met
Key: HBASE-8282
URL: https://issues.apache.org/jira/browse/HBASE-8282
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.98.0, 0.94.8, 0.95.1
Attachments: HBASE-8282_trunk_1.patch, HBASE-8282_trunk.patch
Observe the below logs
{code}
2013-04-04 17:43:45,825 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Flushing
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9.
2013-04-04 17:43:45,826 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Started memstore flush for
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9., current region
memstore size 42.3 M
2013-04-04 17:43:45,826 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished snapshotting
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9., commencing wait for
mvcc, flushsize=44388936
2013-04-04 17:43:45,826 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished snapshotting, commencing flushing stores
2013-04-04 17:43:45,831 DEBUG org.apache.hadoop.hbase.util.FSUtils: Creating
file=hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/.tmp/7020db24b38246a5818e7eb4e130ad9e
with permission=rwxrwxrwx
2013-04-04 17:43:45,834 DEBUG org.apache.hadoop.hbase.io.hfile.HFileWriterV2:
Initialized with CacheConfig:enabled [cacheDataOnRead=true]
[cacheDataOnWrite=false] [cacheIndexesOnWrite=false]
[cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
2013-04-04 17:43:45,835 INFO org.apache.hadoop.hbase.regionserver.StoreFile:
Delete Family Bloom filter type for
hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/.tmp/7020db24b38246a5818e7eb4e130ad9e:
CompoundBloomFilterWriter
2013-04-04 17:43:46,074 INFO org.apache.hadoop.hbase.regionserver.StoreFile:
NO General Bloom and NO DeleteFamily was added to HFile
(hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/.tmp/7020db24b38246a5818e7eb4e130ad9e)
2013-04-04 17:43:46,074 INFO org.apache.hadoop.hbase.regionserver.HStore:
Flushed , sequenceid=1051817, memsize=42.3 M, into tmp file
hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/.tmp/7020db24b38246a5818e7eb4e130ad9e
2013-04-04 17:43:46,093 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionFileSystem: Committing store file
hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/.tmp/7020db24b38246a5818e7eb4e130ad9e
as
hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/f1/7020db24b38246a5818e7eb4e130ad9e
2013-04-04 17:43:46,103 INFO org.apache.hadoop.hbase.regionserver.HStore:
Added
hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/f1/7020db24b38246a5818e7eb4e130ad9e,
entries=54911, sequenceid=1051817, filesize=35.1 M
2013-04-04 17:43:46,103 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Finished memstore flush of ~42.3 M/44388936, currentsize=0/0 for region
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9. in 278ms,
sequenceid=-1, compaction requested=true
2013-04-04 17:43:56,335 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache:
Stats: total=394.09 MB, free=3.61 GB, max=4.00 GB, blocks=5263,
accesses=244882, hits=42988, hitRatio=17.55%, , cachingAccesses=49869,
cachingHits=24251, cachingHitsRatio=48.63%, , evictions=0, evicted=20349,
evictedPerRun=Infinity
2013-04-04 17:44:31,119 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Flushing
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9.
2013-04-04 17:44:31,120 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Started memstore flush for
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9., current region
memstore size 42.7 M
2013-04-04 17:44:31,120 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished snapshotting
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9., commencing wait for
mvcc, flushsize=44764248
2013-04-04 17:44:31,120 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished snapshotting, commencing flushing stores
2013-04-04 17:44:31,136 DEBUG org.apache.hadoop.hbase.util.FSUtils: Creating
file=hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/.tmp/c5c2cd6aaf1644daa9c858149da39081
with permission=rwxrwxrwx
2013-04-04 17:44:31,139 DEBUG
[jira] [Commented] (HBASE-8545) Meta stuck in transition when it is assigned to a just restarted dead region sever
[
https://issues.apache.org/jira/browse/HBASE-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662133#comment-13662133
]
ramkrishna.s.vasudevan commented on HBASE-8545:
---
Going thro the code once again, it is bit difficult to know whether the new
connection is to the same server but with different host or entirely a new
server.
If the below code
{code}
if (isDeadServer(serverName)) {
throw new RegionServerStoppedException(serverName + is dead.);
}
{code}
works then things will be fine. Will see if anything can be done here if not we
can go with the current patch itself.
Meta stuck in transition when it is assigned to a just restarted dead region
sever
---
Key: HBASE-8545
URL: https://issues.apache.org/jira/browse/HBASE-8545
Project: HBase
Issue Type: Bug
Components: Region Assignment
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Attachments: trunk-8545.patch, trunk-8545_v2.patch
Support the meta region server is down, and the SSH tries to re-assign it.
This could happen:
1. AM plans to assign meta to a region server (R_old);
2. Now R_old is dead, the new region server (R_new) starts up on the same
host, port, but gets a different start code;
3. AM sends the open region request to R_new and the Meta is opened on it;
4. AM gets ZK event, but it is from a different region server instance
(R_new), not the expected one (R_old), so it sends a close region request to
R_new;
5. Now, the meta is stuck in transition and won't be assigned.
This won't happen to a user region since the SSH for R_old will find out the
user region stuck in transition and re-assign it. For meta, it is a little
different. AM checks if a dead region server carries the meta based on the
ZK info, which is changed to the new region server R_new at step 3 by the
open region handler.
The fix I was thinking about is:
1. In checking if a region server carries a region, uses the region
transition information if it exists (which is the source of truth, to
master), if not, checks the ZK data as before;
2. In open region handler, when transition assign zk node from offline to
opening, make sure the current region server is the expected one
(ZK#transitionNode, existing code doesn't check the target server name).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8450) Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc.
[ https://issues.apache.org/jira/browse/HBASE-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662134#comment-13662134 ] Nicolas Liochon commented on HBASE-8450: I would even increase the number of retries to 30. +1 for zookeeper timeout. I don't understand this one: + rpc timeout is 10seconds instead of 60seconds From what I've seen, it's not that uncommon to have something above 60s, and retrying just adds some workload on the server? Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc. Key: HBASE-8450 URL: https://issues.apache.org/jira/browse/HBASE-8450 Project: HBase Issue Type: Task Components: Usability Reporter: stack Assignee: stack Priority: Critical Fix For: 0.95.1 Attachments: 8450.txt, 8450v2.txt, 8450v3.txt This is a critical task we need to do before we release; review our defaults. On cursory review, there are configs in hbase-default.xml that no longer have matching code; there are some that should be changed because we know better now and others that should be amended because hardware and deploys are bigger than they used to be. We could also move stuff around so that the must-edit stuff is near the top (zk quorum config. is mid-way down the page) and beef up the descriptions -- especially since these descriptions shine through in the refguide. Lastly, I notice that our tgz does not include an hbase-default.xml other than the one bundled up in the jar. Maybe we should make it more accessible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8450) Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc.
[
https://issues.apache.org/jira/browse/HBASE-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662142#comment-13662142
]
Hadoop QA commented on HBASE-8450:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12583846/8450v3.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestColumnSeeking
org.apache.hadoop.hbase.filter.TestColumnPrefixFilter
org.apache.hadoop.hbase.filter.TestDependentColumnFilter
org.apache.hadoop.hbase.filter.TestMultipleColumnPrefixFilter
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5751//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5751//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5751//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5751//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5751//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5751//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5751//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5751//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5751//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5751//console
This message is automatically generated.
Update hbase-default.xml and general recommendations to better suit current
hw, h2, experience, etc.
Key: HBASE-8450
URL: https://issues.apache.org/jira/browse/HBASE-8450
Project: HBase
Issue Type: Task
Components: Usability
Reporter: stack
Assignee: stack
Priority: Critical
Fix For: 0.95.1
Attachments: 8450.txt, 8450v2.txt, 8450v3.txt
This is a critical task we need to do before we release; review our defaults.
On cursory review, there are configs in hbase-default.xml that no longer have
matching code; there are some that should be changed because we know better
now and others that should be amended because hardware and deploys are bigger
than they used to be.
We could also move stuff around so that the must-edit stuff is near the top
(zk quorum config. is mid-way down the page) and beef up the descriptions --
especially since these descriptions shine through in the refguide.
Lastly, I notice that our tgz does not include an hbase-default.xml other
than the one bundled up in the jar. Maybe we should make it more accessible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5583) Master restart on create table with splitkeys does not recreate table with all the splitkey regions
[ https://issues.apache.org/jira/browse/HBASE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662145#comment-13662145 ] ramkrishna.s.vasudevan commented on HBASE-5583: --- Its more than one month, is it ok if we can revive now? Ted agreed to it a month ago itself. What other feel? Master restart on create table with splitkeys does not recreate table with all the splitkey regions --- Key: HBASE-5583 URL: https://issues.apache.org/jira/browse/HBASE-5583 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.95.1 Attachments: HBASE-5583_new_1.patch, HBASE-5583_new_1_review.patch, HBASE-5583_new_2.patch, HBASE-5583_new_4_WIP.patch, HBASE-5583_new_5_WIP_using_tableznode.patch - Create table using splitkeys - MAster goes down before all regions are added to meta - On master restart the table is again enabled but with less number of regions than specified in splitkeys Anyway client will get an exception if i had called sync create table. But table exists or not check will say table exists. Is this scenario to be handled by client only or can we have some mechanism on the master side for this? Pls suggest. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8522) Archived hfiles and old hlogs may be deleted immediatly by HFileCleaner, LogCleaner in HMaster
[
https://issues.apache.org/jira/browse/HBASE-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662146#comment-13662146
]
Hadoop QA commented on HBASE-8522:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12583841/HBASE-8522-trunk-v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5750//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5750//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5750//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5750//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5750//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5750//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5750//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5750//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5750//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5750//console
This message is automatically generated.
Archived hfiles and old hlogs may be deleted immediatly by HFileCleaner,
LogCleaner in HMaster
--
Key: HBASE-8522
URL: https://issues.apache.org/jira/browse/HBASE-8522
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.3
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
Attachments: HBASE-8522-0.94.patch, HBASE-8522-0.94-v2.patch,
HBASE-8522-trunk.patch, HBASE-8522-trunk-v2.patch, HBASE-8522-trunk-v2.patch
TimeToLiveHFileCleaner is configed to 'hbase.master.hfilecleaner.plugins' in
hbase-default.xml. And timeToLiveHFileCleaner uses the modify time of the
hfile to determine if it should be deleted. But, the modify time of the hdfs
file is time when its writer is closed. The rename op will not change the
modify time of the hfile. So the hfile may be deleted immediatly by
HFileCleaner after it is moved to archives. See log:
2013-05-08 08:15:46,053 DEBUG
org.apache.hadoop.hbase.master.cleaner.CleanerChore: Checking directory:
hdfs://hbase/.archive/table/4e48ffc1ec089082c66e6d1b5f018fb5/M/729e8bc1430540cb9b2c147c90039cdc
2013-05-08 08:15:46,055 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine: Call:
getFileInfo took 1ms
2013-05-08 08:15:46,055 DEBUG
org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner: Life:40033567,
ttl:30, current:1367972146054, from: 1367932112487
2013-05-08 08:15:46,055 DEBUG
org.apache.hadoop.hbase.master.cleaner.CleanerChore:
Removing:hdfs://hbase/.archive/table/4e48ffc1ec089082c66e6d1b5f018fb5/M/729e8bc1430540cb9b2c147c90039cdc
from archive
The same to old hlogs.
And my solution is very simple: When hfiles and hlogs are archived, we set
the modify time of files after rename.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:
[jira] [Commented] (HBASE-8545) Meta stuck in transition when it is assigned to a just restarted dead region sever
[ https://issues.apache.org/jira/browse/HBASE-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662149#comment-13662149 ] Jimmy Xiang commented on HBASE-8545: This is a very rare racing issue. It really depends how thing happens at that moment. Since the region plan uses the old server, it means isDeadServer(serveName) could return false, while the server is already dead because there is a short time in the middle. Meta stuck in transition when it is assigned to a just restarted dead region sever --- Key: HBASE-8545 URL: https://issues.apache.org/jira/browse/HBASE-8545 Project: HBase Issue Type: Bug Components: Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: trunk-8545.patch, trunk-8545_v2.patch Support the meta region server is down, and the SSH tries to re-assign it. This could happen: 1. AM plans to assign meta to a region server (R_old); 2. Now R_old is dead, the new region server (R_new) starts up on the same host, port, but gets a different start code; 3. AM sends the open region request to R_new and the Meta is opened on it; 4. AM gets ZK event, but it is from a different region server instance (R_new), not the expected one (R_old), so it sends a close region request to R_new; 5. Now, the meta is stuck in transition and won't be assigned. This won't happen to a user region since the SSH for R_old will find out the user region stuck in transition and re-assign it. For meta, it is a little different. AM checks if a dead region server carries the meta based on the ZK info, which is changed to the new region server R_new at step 3 by the open region handler. The fix I was thinking about is: 1. In checking if a region server carries a region, uses the region transition information if it exists (which is the source of truth, to master), if not, checks the ZK data as before; 2. In open region handler, when transition assign zk node from offline to opening, make sure the current region server is the expected one (ZK#transitionNode, existing code doesn't check the target server name). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8545) Meta stuck in transition when it is assigned to a just restarted dead region sever
[ https://issues.apache.org/jira/browse/HBASE-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-8545: --- Status: Open (was: Patch Available) Meta stuck in transition when it is assigned to a just restarted dead region sever --- Key: HBASE-8545 URL: https://issues.apache.org/jira/browse/HBASE-8545 Project: HBase Issue Type: Bug Components: Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: trunk-8545.patch, trunk-8545_v2.patch, trunk-8545_v3.patch Support the meta region server is down, and the SSH tries to re-assign it. This could happen: 1. AM plans to assign meta to a region server (R_old); 2. Now R_old is dead, the new region server (R_new) starts up on the same host, port, but gets a different start code; 3. AM sends the open region request to R_new and the Meta is opened on it; 4. AM gets ZK event, but it is from a different region server instance (R_new), not the expected one (R_old), so it sends a close region request to R_new; 5. Now, the meta is stuck in transition and won't be assigned. This won't happen to a user region since the SSH for R_old will find out the user region stuck in transition and re-assign it. For meta, it is a little different. AM checks if a dead region server carries the meta based on the ZK info, which is changed to the new region server R_new at step 3 by the open region handler. The fix I was thinking about is: 1. In checking if a region server carries a region, uses the region transition information if it exists (which is the source of truth, to master), if not, checks the ZK data as before; 2. In open region handler, when transition assign zk node from offline to opening, make sure the current region server is the expected one (ZK#transitionNode, existing code doesn't check the target server name). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8545) Meta stuck in transition when it is assigned to a just restarted dead region sever
[ https://issues.apache.org/jira/browse/HBASE-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-8545: --- Attachment: trunk-8545_v3.patch Meta stuck in transition when it is assigned to a just restarted dead region sever --- Key: HBASE-8545 URL: https://issues.apache.org/jira/browse/HBASE-8545 Project: HBase Issue Type: Bug Components: Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: trunk-8545.patch, trunk-8545_v2.patch, trunk-8545_v3.patch Support the meta region server is down, and the SSH tries to re-assign it. This could happen: 1. AM plans to assign meta to a region server (R_old); 2. Now R_old is dead, the new region server (R_new) starts up on the same host, port, but gets a different start code; 3. AM sends the open region request to R_new and the Meta is opened on it; 4. AM gets ZK event, but it is from a different region server instance (R_new), not the expected one (R_old), so it sends a close region request to R_new; 5. Now, the meta is stuck in transition and won't be assigned. This won't happen to a user region since the SSH for R_old will find out the user region stuck in transition and re-assign it. For meta, it is a little different. AM checks if a dead region server carries the meta based on the ZK info, which is changed to the new region server R_new at step 3 by the open region handler. The fix I was thinking about is: 1. In checking if a region server carries a region, uses the region transition information if it exists (which is the source of truth, to master), if not, checks the ZK data as before; 2. In open region handler, when transition assign zk node from offline to opening, make sure the current region server is the expected one (ZK#transitionNode, existing code doesn't check the target server name). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8545) Meta stuck in transition when it is assigned to a just restarted dead region sever
[ https://issues.apache.org/jira/browse/HBASE-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-8545: --- Status: Patch Available (was: Open) Attached version v3 which has a little change to make sure most of the error messages are the same as before (the patch). Since when a version mismatches, or the etype mismatches, most likely, the region server name doesn't match. There is no logic change here. Meta stuck in transition when it is assigned to a just restarted dead region sever --- Key: HBASE-8545 URL: https://issues.apache.org/jira/browse/HBASE-8545 Project: HBase Issue Type: Bug Components: Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: trunk-8545.patch, trunk-8545_v2.patch, trunk-8545_v3.patch Support the meta region server is down, and the SSH tries to re-assign it. This could happen: 1. AM plans to assign meta to a region server (R_old); 2. Now R_old is dead, the new region server (R_new) starts up on the same host, port, but gets a different start code; 3. AM sends the open region request to R_new and the Meta is opened on it; 4. AM gets ZK event, but it is from a different region server instance (R_new), not the expected one (R_old), so it sends a close region request to R_new; 5. Now, the meta is stuck in transition and won't be assigned. This won't happen to a user region since the SSH for R_old will find out the user region stuck in transition and re-assign it. For meta, it is a little different. AM checks if a dead region server carries the meta based on the ZK info, which is changed to the new region server R_new at step 3 by the open region handler. The fix I was thinking about is: 1. In checking if a region server carries a region, uses the region transition information if it exists (which is the source of truth, to master), if not, checks the ZK data as before; 2. In open region handler, when transition assign zk node from offline to opening, make sure the current region server is the expected one (ZK#transitionNode, existing code doesn't check the target server name). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8450) Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc.
[ https://issues.apache.org/jira/browse/HBASE-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-8450: - Attachment: 8450v5.txt I put the rpc timeout back to 60 at @nkeywal query Left retries at 20 for now. Fixed tests that depend on maxversions == 3 Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc. Key: HBASE-8450 URL: https://issues.apache.org/jira/browse/HBASE-8450 Project: HBase Issue Type: Task Components: Usability Reporter: stack Assignee: stack Priority: Critical Fix For: 0.95.1 Attachments: 8450.txt, 8450v2.txt, 8450v3.txt, 8450v5.txt This is a critical task we need to do before we release; review our defaults. On cursory review, there are configs in hbase-default.xml that no longer have matching code; there are some that should be changed because we know better now and others that should be amended because hardware and deploys are bigger than they used to be. We could also move stuff around so that the must-edit stuff is near the top (zk quorum config. is mid-way down the page) and beef up the descriptions -- especially since these descriptions shine through in the refguide. Lastly, I notice that our tgz does not include an hbase-default.xml other than the one bundled up in the jar. Maybe we should make it more accessible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8471) Server-side, remove convertion from pb type to client type before we call method
[ https://issues.apache.org/jira/browse/HBASE-8471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-8471: - Attachment: HBASE-8471.patch Retry hadoopqa Server-side, remove convertion from pb type to client type before we call method Key: HBASE-8471 URL: https://issues.apache.org/jira/browse/HBASE-8471 Project: HBase Issue Type: Improvement Components: IPC/RPC, regionserver Affects Versions: 0.95.0 Reporter: stack Assignee: Anoop Sam John Priority: Critical Attachments: HBASE-8471.patch, HBASE-8471.patch In the regionserver, when the rpc receives a call, the call is described using protobufs. Before we make the server-side invocation, we do a transform on the pb param objects to make a native pojo -- e.g. from a pb Puts into an hbase o.a.h.h.client.Put -- and only then do we make the call against the server. On the way out, similar, before putting the result on the wire, we will do a convertion from o.a.h.h.client.Result into pb Result. This issue is about our first INVESTIGATING if it is possible to do away w/ this marshalling/unmarshalling serverside especially given the pb objects themselves are rich in accessor and getters, etc. If it is possible to do w/ pbs alone serverside, then we should go ahead and rip out all the serverside convertions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8545) Meta stuck in transition when it is assigned to a just restarted dead region sever
[ https://issues.apache.org/jira/browse/HBASE-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662167#comment-13662167 ] Jimmy Xiang commented on HBASE-8545: bq. Ideally we should ensure that we don't create the connection to the new RS as the original request is for a dead RS. I agree. However, this is hard because we probably don't have a cached connection (especially when master fails over), while the master hasn't detected the region server is already restarted. Meta stuck in transition when it is assigned to a just restarted dead region sever --- Key: HBASE-8545 URL: https://issues.apache.org/jira/browse/HBASE-8545 Project: HBase Issue Type: Bug Components: Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: trunk-8545.patch, trunk-8545_v2.patch, trunk-8545_v3.patch Support the meta region server is down, and the SSH tries to re-assign it. This could happen: 1. AM plans to assign meta to a region server (R_old); 2. Now R_old is dead, the new region server (R_new) starts up on the same host, port, but gets a different start code; 3. AM sends the open region request to R_new and the Meta is opened on it; 4. AM gets ZK event, but it is from a different region server instance (R_new), not the expected one (R_old), so it sends a close region request to R_new; 5. Now, the meta is stuck in transition and won't be assigned. This won't happen to a user region since the SSH for R_old will find out the user region stuck in transition and re-assign it. For meta, it is a little different. AM checks if a dead region server carries the meta based on the ZK info, which is changed to the new region server R_new at step 3 by the open region handler. The fix I was thinking about is: 1. In checking if a region server carries a region, uses the region transition information if it exists (which is the source of truth, to master), if not, checks the ZK data as before; 2. In open region handler, when transition assign zk node from offline to opening, make sure the current region server is the expected one (ZK#transitionNode, existing code doesn't check the target server name). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8450) Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc.
[ https://issues.apache.org/jira/browse/HBASE-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662170#comment-13662170 ] Lars Hofhansl commented on HBASE-8450: -- I'd still be concerned about disabling major compactions. If kvs are deleted the delete markers will hang around forever slowing down scans. Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc. Key: HBASE-8450 URL: https://issues.apache.org/jira/browse/HBASE-8450 Project: HBase Issue Type: Task Components: Usability Reporter: stack Assignee: stack Priority: Critical Fix For: 0.95.1 Attachments: 8450.txt, 8450v2.txt, 8450v3.txt, 8450v5.txt This is a critical task we need to do before we release; review our defaults. On cursory review, there are configs in hbase-default.xml that no longer have matching code; there are some that should be changed because we know better now and others that should be amended because hardware and deploys are bigger than they used to be. We could also move stuff around so that the must-edit stuff is near the top (zk quorum config. is mid-way down the page) and beef up the descriptions -- especially since these descriptions shine through in the refguide. Lastly, I notice that our tgz does not include an hbase-default.xml other than the one bundled up in the jar. Maybe we should make it more accessible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8579) TestDelayedRpc falis from time to time
stack created HBASE-8579: Summary: TestDelayedRpc falis from time to time Key: HBASE-8579 URL: https://issues.apache.org/jira/browse/HBASE-8579 Project: HBase Issue Type: Bug Components: IPC/RPC Reporter: stack Fix For: 0.98.0, 0.95.1 Attachments: 8579.txt It failed here http://54.241.6.143/job/HBase-0.95/291/ Looks like basic rpc timeout (timeout set to 1 second and complaint was that it had waited 1.6 seconds). I suppose this could happen on loaded server. Let me try upping timeouts for now. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8579) TestDelayedRpc falis from time to time
[ https://issues.apache.org/jira/browse/HBASE-8579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-8579: - Attachment: 8579.txt TestDelayedRpc falis from time to time -- Key: HBASE-8579 URL: https://issues.apache.org/jira/browse/HBASE-8579 Project: HBase Issue Type: Bug Components: IPC/RPC Reporter: stack Fix For: 0.98.0, 0.95.1 Attachments: 8579.txt It failed here http://54.241.6.143/job/HBase-0.95/291/ Looks like basic rpc timeout (timeout set to 1 second and complaint was that it had waited 1.6 seconds). I suppose this could happen on loaded server. Let me try upping timeouts for now. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8579) TestDelayedRpc falis from time to time
[ https://issues.apache.org/jira/browse/HBASE-8579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-8579: - Status: Patch Available (was: Open) TestDelayedRpc falis from time to time -- Key: HBASE-8579 URL: https://issues.apache.org/jira/browse/HBASE-8579 Project: HBase Issue Type: Bug Components: IPC/RPC Reporter: stack Fix For: 0.98.0, 0.95.1 Attachments: 8579.txt It failed here http://54.241.6.143/job/HBase-0.95/291/ Looks like basic rpc timeout (timeout set to 1 second and complaint was that it had waited 1.6 seconds). I suppose this could happen on loaded server. Let me try upping timeouts for now. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8579) TestDelayedRpc falis from time to time
[ https://issues.apache.org/jira/browse/HBASE-8579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662197#comment-13662197 ] Sergey Shelukhin commented on HBASE-8579: - lgtm TestDelayedRpc falis from time to time -- Key: HBASE-8579 URL: https://issues.apache.org/jira/browse/HBASE-8579 Project: HBase Issue Type: Bug Components: IPC/RPC Reporter: stack Fix For: 0.98.0, 0.95.1 Attachments: 8579.txt It failed here http://54.241.6.143/job/HBase-0.95/291/ Looks like basic rpc timeout (timeout set to 1 second and complaint was that it had waited 1.6 seconds). I suppose this could happen on loaded server. Let me try upping timeouts for now. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8450) Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc.
[ https://issues.apache.org/jira/browse/HBASE-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662205#comment-13662205 ] stack commented on HBASE-8450: -- [~lhofhansl] Then what would you suggest for a major compaction interval? If you think a week, then I'd say might as well just turn it off. If you say a day, i'd suggest that is too frequent rewriting all of a deploys' data. I can add to the refguide a section on If your scans start to slow over time... Would that help? Looks like there are more tests that expect max versions of 3 as default. Fixing... new patch coming. Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc. Key: HBASE-8450 URL: https://issues.apache.org/jira/browse/HBASE-8450 Project: HBase Issue Type: Task Components: Usability Reporter: stack Assignee: stack Priority: Critical Fix For: 0.95.1 Attachments: 8450.txt, 8450v2.txt, 8450v3.txt, 8450v5.txt This is a critical task we need to do before we release; review our defaults. On cursory review, there are configs in hbase-default.xml that no longer have matching code; there are some that should be changed because we know better now and others that should be amended because hardware and deploys are bigger than they used to be. We could also move stuff around so that the must-edit stuff is near the top (zk quorum config. is mid-way down the page) and beef up the descriptions -- especially since these descriptions shine through in the refguide. Lastly, I notice that our tgz does not include an hbase-default.xml other than the one bundled up in the jar. Maybe we should make it more accessible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HBASE-8067) TestHFileArchiving.testArchiveOnTableDelete sometimes fails
[
https://issues.apache.org/jira/browse/HBASE-8067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack reopened HBASE-8067:
--
Was going to open a new issue after this test failed on the first build after
its commit but instead, will just do reopen of this one again (the second time).
Here is the fail:
http://54.241.6.143/job/HBase-0.95/org.apache.hbase$hbase-server/294/
TestHFileArchiving.testArchiveOnTableDelete sometimes fails
---
Key: HBASE-8067
URL: https://issues.apache.org/jira/browse/HBASE-8067
Project: HBase
Issue Type: Bug
Components: Admin, master, test
Affects Versions: 0.94.6, 0.95.2
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Fix For: 0.98.0, 0.94.8, 0.95.1
Attachments: 8067-0.94.txt, 8067-trunk.txt, HBASE-8067-debug.patch,
HBASE-8067-v0.patch
it seems that testArchiveOnTableDelete() fails because the archiving in
DeleteTableHandler is still in progress when admin.deleteTable() returns.
{code}
Error Message
Archived files are missing some of the store files!
Stacktrace
java.lang.AssertionError: Archived files are missing some of the store files!
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.hbase.backup.TestHFileArchiving.testArchiveOnTableDelete(TestHFileArchiving.java:262)
{code}
(Looking at the problem in a more generic way, we don't have any way to
inform the client when an async operation is completed)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8067) TestHFileArchiving.testArchiveOnTableDelete sometimes fails
[
https://issues.apache.org/jira/browse/HBASE-8067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-8067:
-
Priority: Critical (was: Major)
TestHFileArchiving.testArchiveOnTableDelete sometimes fails
---
Key: HBASE-8067
URL: https://issues.apache.org/jira/browse/HBASE-8067
Project: HBase
Issue Type: Bug
Components: Admin, master, test
Affects Versions: 0.94.6, 0.95.2
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Critical
Fix For: 0.98.0, 0.94.8, 0.95.1
Attachments: 8067-0.94.txt, 8067-trunk.txt, HBASE-8067-debug.patch,
HBASE-8067-v0.patch
it seems that testArchiveOnTableDelete() fails because the archiving in
DeleteTableHandler is still in progress when admin.deleteTable() returns.
{code}
Error Message
Archived files are missing some of the store files!
Stacktrace
java.lang.AssertionError: Archived files are missing some of the store files!
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.hbase.backup.TestHFileArchiving.testArchiveOnTableDelete(TestHFileArchiving.java:262)
{code}
(Looking at the problem in a more generic way, we don't have any way to
inform the client when an async operation is completed)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8450) Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc.
[
https://issues.apache.org/jira/browse/HBASE-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662234#comment-13662234
]
Hadoop QA commented on HBASE-8450:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12583855/8450v5.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.io.encoding.TestEncodedSeekers
org.apache.hadoop.hbase.regionserver.TestHRegionBusyWait
org.apache.hadoop.hbase.security.access.TestAccessController
org.apache.hadoop.hbase.TestMultiVersions
org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor
org.apache.hadoop.hbase.regionserver.TestHRegion
org.apache.hadoop.hbase.replication.TestReplicationSmallTests
org.apache.hadoop.hbase.client.TestFromClientSide
org.apache.hadoop.hbase.regionserver.TestGetClosestAtOrBefore
org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler
org.apache.hadoop.hbase.mapreduce.TestImportExport
org.apache.hadoop.hbase.rest.client.TestRemoteTable
org.apache.hadoop.hbase.regionserver.TestSeekOptimizations
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5752//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5752//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5752//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5752//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5752//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5752//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5752//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5752//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5752//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5752//console
This message is automatically generated.
Update hbase-default.xml and general recommendations to better suit current
hw, h2, experience, etc.
Key: HBASE-8450
URL: https://issues.apache.org/jira/browse/HBASE-8450
Project: HBase
Issue Type: Task
Components: Usability
Reporter: stack
Assignee: stack
Priority: Critical
Fix For: 0.95.1
Attachments: 8450.txt, 8450v2.txt, 8450v3.txt, 8450v5.txt
This is a critical task we need to do before we release; review our defaults.
On cursory review, there are configs in hbase-default.xml that no longer have
matching code; there are some that should be changed because we know better
now and others that should be amended because hardware and deploys are bigger
than they used to be.
We could also move stuff around so that the must-edit stuff is near the top
(zk quorum config. is mid-way down the page) and beef up the descriptions --
especially since these
[jira] [Commented] (HBASE-8545) Meta stuck in transition when it is assigned to a just restarted dead region sever
[
https://issues.apache.org/jira/browse/HBASE-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662241#comment-13662241
]
Hadoop QA commented on HBASE-8545:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12583854/trunk-8545_v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5753//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5753//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5753//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5753//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5753//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5753//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5753//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5753//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5753//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5753//console
This message is automatically generated.
Meta stuck in transition when it is assigned to a just restarted dead region
sever
---
Key: HBASE-8545
URL: https://issues.apache.org/jira/browse/HBASE-8545
Project: HBase
Issue Type: Bug
Components: Region Assignment
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Attachments: trunk-8545.patch, trunk-8545_v2.patch,
trunk-8545_v3.patch
Support the meta region server is down, and the SSH tries to re-assign it.
This could happen:
1. AM plans to assign meta to a region server (R_old);
2. Now R_old is dead, the new region server (R_new) starts up on the same
host, port, but gets a different start code;
3. AM sends the open region request to R_new and the Meta is opened on it;
4. AM gets ZK event, but it is from a different region server instance
(R_new), not the expected one (R_old), so it sends a close region request to
R_new;
5. Now, the meta is stuck in transition and won't be assigned.
This won't happen to a user region since the SSH for R_old will find out the
user region stuck in transition and re-assign it. For meta, it is a little
different. AM checks if a dead region server carries the meta based on the
ZK info, which is changed to the new region server R_new at step 3 by the
open region handler.
The fix I was thinking about is:
1. In checking if a region server carries a region, uses the region
transition information if it exists (which is the source of truth, to
master), if not, checks the ZK data as before;
2. In open region handler, when transition assign zk node from offline to
opening, make sure the current region server is the expected one
(ZK#transitionNode, existing code doesn't check the target server name).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:
[jira] [Commented] (HBASE-8471) Server-side, remove convertion from pb type to client type before we call method
[
https://issues.apache.org/jira/browse/HBASE-8471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662242#comment-13662242
]
Hadoop QA commented on HBASE-8471:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12583856/HBASE-8471.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.master.TestAssignmentManager.testSSHTimesOutOpeningRegionTransition(TestAssignmentManager.java:967)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5754//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5754//console
This message is automatically generated.
Server-side, remove convertion from pb type to client type before we call
method
Key: HBASE-8471
URL: https://issues.apache.org/jira/browse/HBASE-8471
Project: HBase
Issue Type: Improvement
Components: IPC/RPC, regionserver
Affects Versions: 0.95.0
Reporter: stack
Assignee: Anoop Sam John
Priority: Critical
Attachments: HBASE-8471.patch, HBASE-8471.patch
In the regionserver, when the rpc receives a call, the call is described
using protobufs. Before we make the server-side invocation, we do a
transform on the pb param objects to make a native pojo -- e.g. from a pb
Puts into an hbase o.a.h.h.client.Put -- and only then do we make the call
against the server.
On the way out, similar, before putting the result on the wire, we will do a
convertion from o.a.h.h.client.Result into pb Result.
This issue is about our first INVESTIGATING if it is possible to do away w/
this marshalling/unmarshalling serverside especially given the pb objects
themselves are rich in accessor and getters, etc. If it is possible to do w/
pbs alone serverside, then we should go ahead and rip out all the serverside
convertions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4462) Properly treating SocketTimeoutException
[ https://issues.apache.org/jira/browse/HBASE-4462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4462: - Attachment: unittest_that_shows_us_retrying_sockettimeout.txt Here is unit test that seems to show us retrying socketimeouts. Properly treating SocketTimeoutException Key: HBASE-4462 URL: https://issues.apache.org/jira/browse/HBASE-4462 Project: HBase Issue Type: Improvement Affects Versions: 0.90.4 Reporter: Jean-Daniel Cryans Assignee: ramkrishna.s.vasudevan Fix For: 0.90.8 Attachments: HBASE-4462_0.90.x.patch, unittest_that_shows_us_retrying_sockettimeout.txt SocketTimeoutException is currently treated like any IOE inside of HCM.getRegionServerWithRetries and I think this is a problem. This method should only do retries in cases where we are pretty sure the operation will complete, but with STE we already waited for (by default) 60 seconds and nothing happened. I found this while debugging Douglas Campbell's problem on the mailing list where it seemed like he was using the same scanner from multiple threads, but actually it was just the same client doing retries while the first run didn't even finish yet (that's another problem). You could see the first scanner, then up to two other handlers waiting for it to finish in order to run (because of the synchronization on RegionScanner). So what should we do? We could treat STE as a DoNotRetryException and let the client deal with it, or we could retry only once. There's also the option of having a different behavior for get/put/icv/scan, the issue with operations that modify a cell is that you don't know if the operation completed or not (same when a RS dies hard after completing let's say a Put but just before returning to the client). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4462) Properly treating SocketTimeoutException
[
https://issues.apache.org/jira/browse/HBASE-4462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662252#comment-13662252
]
stack commented on HBASE-4462:
--
Test fails like this:
{code}
3 Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 5.943 sec
FAILURE!
4
testNoRetryOnSocketTimeoutException(org.apache.hadoop.hbase.client.TestClientNoCluster)
Time elapsed: 5.726 sec ERROR!
5 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=5, exceptions:
6 Mon May 20 11:30:57 PDT 2013,
org.apache.hadoop.hbase.client.ScannerCallable@275e538e,
java.net.SocketTimeoutException: From Mockito
7 Mon May 20 11:30:58 PDT 2013,
org.apache.hadoop.hbase.client.ScannerCallable@275e538e,
java.net.SocketTimeoutException: From Mockito
8 Mon May 20 11:30:59 PDT 2013,
org.apache.hadoop.hbase.client.ScannerCallable@275e538e,
java.net.SocketTimeoutException: From Mockito
9 Mon May 20 11:31:00 PDT 2013,
org.apache.hadoop.hbase.client.ScannerCallable@275e538e,
java.net.SocketTimeoutException: From Mockito
10 Mon May 20 11:31:02 PDT 2013,
org.apache.hadoop.hbase.client.ScannerCallable@275e538e,
java.net.SocketTimeoutException: From Mockito
{code}
Properly treating SocketTimeoutException
Key: HBASE-4462
URL: https://issues.apache.org/jira/browse/HBASE-4462
Project: HBase
Issue Type: Improvement
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
Assignee: ramkrishna.s.vasudevan
Fix For: 0.90.8
Attachments: HBASE-4462_0.90.x.patch,
unittest_that_shows_us_retrying_sockettimeout.txt
SocketTimeoutException is currently treated like any IOE inside of
HCM.getRegionServerWithRetries and I think this is a problem. This method
should only do retries in cases where we are pretty sure the operation will
complete, but with STE we already waited for (by default) 60 seconds and
nothing happened.
I found this while debugging Douglas Campbell's problem on the mailing list
where it seemed like he was using the same scanner from multiple threads, but
actually it was just the same client doing retries while the first run didn't
even finish yet (that's another problem). You could see the first scanner,
then up to two other handlers waiting for it to finish in order to run
(because of the synchronization on RegionScanner).
So what should we do? We could treat STE as a DoNotRetryException and let the
client deal with it, or we could retry only once.
There's also the option of having a different behavior for get/put/icv/scan,
the issue with operations that modify a cell is that you don't know if the
operation completed or not (same when a RS dies hard after completing let's
say a Put but just before returning to the client).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8471) Server-side, remove convertion from pb type to client type before we call method
[ https://issues.apache.org/jira/browse/HBASE-8471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662261#comment-13662261 ] stack commented on HBASE-8471: -- [~anoop.hbase] looks like an issue w/ patch... good stuff. Server-side, remove convertion from pb type to client type before we call method Key: HBASE-8471 URL: https://issues.apache.org/jira/browse/HBASE-8471 Project: HBase Issue Type: Improvement Components: IPC/RPC, regionserver Affects Versions: 0.95.0 Reporter: stack Assignee: Anoop Sam John Priority: Critical Attachments: HBASE-8471.patch, HBASE-8471.patch In the regionserver, when the rpc receives a call, the call is described using protobufs. Before we make the server-side invocation, we do a transform on the pb param objects to make a native pojo -- e.g. from a pb Puts into an hbase o.a.h.h.client.Put -- and only then do we make the call against the server. On the way out, similar, before putting the result on the wire, we will do a convertion from o.a.h.h.client.Result into pb Result. This issue is about our first INVESTIGATING if it is possible to do away w/ this marshalling/unmarshalling serverside especially given the pb objects themselves are rich in accessor and getters, etc. If it is possible to do w/ pbs alone serverside, then we should go ahead and rip out all the serverside convertions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4462) Properly treating SocketTimeoutException
[
https://issues.apache.org/jira/browse/HBASE-4462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662278#comment-13662278
]
Hadoop QA commented on HBASE-4462:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12583869/unittest_that_shows_us_retrying_sockettimeout.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestClientNoCluster
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5756//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5756//console
This message is automatically generated.
Properly treating SocketTimeoutException
Key: HBASE-4462
URL: https://issues.apache.org/jira/browse/HBASE-4462
Project: HBase
Issue Type: Improvement
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
Assignee: ramkrishna.s.vasudevan
Fix For: 0.90.8
Attachments: HBASE-4462_0.90.x.patch,
unittest_that_shows_us_retrying_sockettimeout.txt
SocketTimeoutException is currently treated like any IOE inside of
HCM.getRegionServerWithRetries and I think this is a problem. This method
should only do retries in cases where we are pretty sure the operation will
complete, but with STE we already waited for (by default) 60 seconds and
nothing happened.
I found this while debugging Douglas Campbell's problem on the mailing list
where it seemed like he was using the same scanner from multiple threads, but
actually it was just the same client doing retries while the first run didn't
even finish yet (that's another problem). You could see the first scanner,
then up to two other handlers waiting for it to finish in order to run
(because of the synchronization on RegionScanner).
So what should we do? We could treat STE as a DoNotRetryException and let the
client deal with it, or we could retry only once.
There's also the option of having a different behavior for get/put/icv/scan,
the issue with operations that modify a cell is that you don't know if the
operation completed or not (same when a RS dies hard after completing let's
say a Put but just before returning to the client).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8450) Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc.
[ https://issues.apache.org/jira/browse/HBASE-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662290#comment-13662290 ] Lars Hofhansl commented on HBASE-8450: -- This is a bit of a fundamental flaw in HBase. True, if you never delete stuff, you don't need major compactions. But if you do, especially when you place a lot of delete markers, you definitely want major compactions. Sergey's striped compaction will make this a moot point, but until then it seems that we need to be able to clean up. Major compactions also help to eventually regain data locality if regions were moved. Whether it's a day or a week almost does not matter. A week seems fine to me as default. Can always document to disable major compactions if you do not delete a lot. Otherwise we need to introduce a Routine Maintenance section to the book and instruct to run major compactions when it makes sense (they have no metric to go by really). Maybe a way to tackle this is to have remove delete marker compaction, which will still read all files but only rewrites HFile when a delete marker was found. Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc. Key: HBASE-8450 URL: https://issues.apache.org/jira/browse/HBASE-8450 Project: HBase Issue Type: Task Components: Usability Reporter: stack Assignee: stack Priority: Critical Fix For: 0.95.1 Attachments: 8450.txt, 8450v2.txt, 8450v3.txt, 8450v5.txt This is a critical task we need to do before we release; review our defaults. On cursory review, there are configs in hbase-default.xml that no longer have matching code; there are some that should be changed because we know better now and others that should be amended because hardware and deploys are bigger than they used to be. We could also move stuff around so that the must-edit stuff is near the top (zk quorum config. is mid-way down the page) and beef up the descriptions -- especially since these descriptions shine through in the refguide. Lastly, I notice that our tgz does not include an hbase-default.xml other than the one bundled up in the jar. Maybe we should make it more accessible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8067) TestHFileArchiving.testArchiveOnTableDelete sometimes fails
[
https://issues.apache.org/jira/browse/HBASE-8067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662293#comment-13662293
]
Lars Hofhansl commented on HBASE-8067:
--
Huh? The fail is TestMasterFailover.testMasterFailoverWithMockedRITOnDeadRS,
but this issue is about TestHFileArchiving.
TestHFileArchiving.testArchiveOnTableDelete sometimes fails
---
Key: HBASE-8067
URL: https://issues.apache.org/jira/browse/HBASE-8067
Project: HBase
Issue Type: Bug
Components: Admin, master, test
Affects Versions: 0.94.6, 0.95.2
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Critical
Fix For: 0.98.0, 0.94.8, 0.95.1
Attachments: 8067-0.94.txt, 8067-trunk.txt, HBASE-8067-debug.patch,
HBASE-8067-v0.patch
it seems that testArchiveOnTableDelete() fails because the archiving in
DeleteTableHandler is still in progress when admin.deleteTable() returns.
{code}
Error Message
Archived files are missing some of the store files!
Stacktrace
java.lang.AssertionError: Archived files are missing some of the store files!
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.hbase.backup.TestHFileArchiving.testArchiveOnTableDelete(TestHFileArchiving.java:262)
{code}
(Looking at the problem in a more generic way, we don't have any way to
inform the client when an async operation is completed)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8450) Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc.
[ https://issues.apache.org/jira/browse/HBASE-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662295#comment-13662295 ] Lars Hofhansl commented on HBASE-8450: -- Just saw this comment: bq. second, frequently compactions are promoted to major compaction because all files are selected so deletes would get purged That is true too. How frequently does that happen? Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc. Key: HBASE-8450 URL: https://issues.apache.org/jira/browse/HBASE-8450 Project: HBase Issue Type: Task Components: Usability Reporter: stack Assignee: stack Priority: Critical Fix For: 0.95.1 Attachments: 8450.txt, 8450v2.txt, 8450v3.txt, 8450v5.txt This is a critical task we need to do before we release; review our defaults. On cursory review, there are configs in hbase-default.xml that no longer have matching code; there are some that should be changed because we know better now and others that should be amended because hardware and deploys are bigger than they used to be. We could also move stuff around so that the must-edit stuff is near the top (zk quorum config. is mid-way down the page) and beef up the descriptions -- especially since these descriptions shine through in the refguide. Lastly, I notice that our tgz does not include an hbase-default.xml other than the one bundled up in the jar. Maybe we should make it more accessible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8450) Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc.
[ https://issues.apache.org/jira/browse/HBASE-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662296#comment-13662296 ] Jean-Daniel Cryans commented on HBASE-8450: --- bq. That is true too. How frequently does that happen? Unfortunately it really depends on how big the files are in the region such that it would decide to include all the files. Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc. Key: HBASE-8450 URL: https://issues.apache.org/jira/browse/HBASE-8450 Project: HBase Issue Type: Task Components: Usability Reporter: stack Assignee: stack Priority: Critical Fix For: 0.95.1 Attachments: 8450.txt, 8450v2.txt, 8450v3.txt, 8450v5.txt This is a critical task we need to do before we release; review our defaults. On cursory review, there are configs in hbase-default.xml that no longer have matching code; there are some that should be changed because we know better now and others that should be amended because hardware and deploys are bigger than they used to be. We could also move stuff around so that the must-edit stuff is near the top (zk quorum config. is mid-way down the page) and beef up the descriptions -- especially since these descriptions shine through in the refguide. Lastly, I notice that our tgz does not include an hbase-default.xml other than the one bundled up in the jar. Maybe we should make it more accessible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8579) TestDelayedRpc falis from time to time
[
https://issues.apache.org/jira/browse/HBASE-8579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662304#comment-13662304
]
Hadoop QA commented on HBASE-8579:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12583863/8579.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.security.access.TestAccessController
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5755//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5755//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5755//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5755//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5755//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5755//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5755//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5755//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5755//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5755//console
This message is automatically generated.
TestDelayedRpc falis from time to time
--
Key: HBASE-8579
URL: https://issues.apache.org/jira/browse/HBASE-8579
Project: HBase
Issue Type: Bug
Components: IPC/RPC
Reporter: stack
Fix For: 0.98.0, 0.95.1
Attachments: 8579.txt
It failed here http://54.241.6.143/job/HBase-0.95/291/
Looks like basic rpc timeout (timeout set to 1 second and complaint was that
it had waited 1.6 seconds). I suppose this could happen on loaded server.
Let me try upping timeouts for now.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8310) HBase snapshot timeout default values and TableLockManger timeout
[ https://issues.apache.org/jira/browse/HBASE-8310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662329#comment-13662329 ] Jerry He commented on HBASE-8310: - bq. In this case the snapshot is still in progress and a next request will return Rejected taking snapshot because we are already running another snapshot from the snapshotManager and you should wait ... Agree. This is a defense against really messing up on the server. Still this would contradict the synchronized purpose of the client snapshot request? bq. but the table lock is lost because has a timeout? ? Could you provide some details? HBase snapshot timeout default values and TableLockManger timeout - Key: HBASE-8310 URL: https://issues.apache.org/jira/browse/HBASE-8310 Project: HBase Issue Type: Bug Components: snapshots Affects Versions: 0.95.0 Reporter: Jerry He Assignee: Jerry He Priority: Minor Fix For: 0.98.0, 0.94.8, 0.95.2 There are a few timeout values and defaults being used by HBase snapshot. DEFAULT_MAX_WAIT_TIME (6 milli sec, 1 min) for client response TIMEOUT_MILLIS_DEFAULT (6 milli sec, 1 min) for Procedure timeout SNAPSHOT_TIMEOUT_MILLIS_DEFAULT (6 milli sec, 1 min) for region server subprocedure There is also other timeout involved, for example, DEFAULT_TABLE_WRITE_LOCK_TIMEOUT_MS (10 mins) for TakeSnapshotHandler#prepare() We could have this case: The user issues a sync snapshot request, waits for 1 min, and gets an exception. In the meantime the snapshot handler is blocked on the table lock, and the snapshot may continue to finish after 10 mins. But the user will probably re-issue the snapshot request during the 10 mins. This is a little confusing and messy when this happens. To be more reasonable, we should either increase the DEFAULT_MAX_WAIT_TIME or decrease the table lock waiting time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8580) Ensure that there are 3 replicas when Flushing or Compacting
Elliott Clark created HBASE-8580: Summary: Ensure that there are 3 replicas when Flushing or Compacting Key: HBASE-8580 URL: https://issues.apache.org/jira/browse/HBASE-8580 Project: HBase Issue Type: Improvement Reporter: Elliott Clark It is possible that datanodes in a DFS pipeline can be dropped. When this happens it possible for there to be fewer than the number of replicas expected. After a close on the file the namenode tries to re-replicate the file. On flush and compaction we should block waiting for the number of datanodes reporting that they have the needed blocks to be 3. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8067) TestHFileArchiving.testArchiveOnTableDelete sometimes fails
[
https://issues.apache.org/jira/browse/HBASE-8067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662345#comment-13662345
]
Hudson commented on HBASE-8067:
---
Integrated in HBase-TRUNK #4132 (See
[https://builds.apache.org/job/HBase-TRUNK/4132/])
HBASE-8067 TestHFileArchiving.testArchiveOnTableDelete sometimes fails
(Revision 1484503)
Result = SUCCESS
stack :
Files :
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchiving.java
TestHFileArchiving.testArchiveOnTableDelete sometimes fails
---
Key: HBASE-8067
URL: https://issues.apache.org/jira/browse/HBASE-8067
Project: HBase
Issue Type: Bug
Components: Admin, master, test
Affects Versions: 0.94.6, 0.95.2
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Critical
Fix For: 0.98.0, 0.94.8, 0.95.1
Attachments: 8067-0.94.txt, 8067-trunk.txt, HBASE-8067-debug.patch,
HBASE-8067-v0.patch
it seems that testArchiveOnTableDelete() fails because the archiving in
DeleteTableHandler is still in progress when admin.deleteTable() returns.
{code}
Error Message
Archived files are missing some of the store files!
Stacktrace
java.lang.AssertionError: Archived files are missing some of the store files!
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.hbase.backup.TestHFileArchiving.testArchiveOnTableDelete(TestHFileArchiving.java:262)
{code}
(Looking at the problem in a more generic way, we don't have any way to
inform the client when an async operation is completed)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8282) User triggered flushes does not allow compaction to get triggered even if compaction criteria is met
[
https://issues.apache.org/jira/browse/HBASE-8282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662346#comment-13662346
]
Hudson commented on HBASE-8282:
---
Integrated in HBase-TRUNK #4132 (See
[https://builds.apache.org/job/HBase-TRUNK/4132/])
HBASE-8282-User triggered flushes does not allow compaction to get
triggered even if compaction criteria is met (Ram) (Revision 1484519)
Result = SUCCESS
ramkrishna :
Files :
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
User triggered flushes does not allow compaction to get triggered even if
compaction criteria is met
Key: HBASE-8282
URL: https://issues.apache.org/jira/browse/HBASE-8282
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.98.0, 0.94.8, 0.95.1
Attachments: HBASE-8282_trunk_1.patch, HBASE-8282_trunk.patch
Observe the below logs
{code}
2013-04-04 17:43:45,825 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Flushing
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9.
2013-04-04 17:43:45,826 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Started memstore flush for
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9., current region
memstore size 42.3 M
2013-04-04 17:43:45,826 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished snapshotting
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9., commencing wait for
mvcc, flushsize=44388936
2013-04-04 17:43:45,826 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished snapshotting, commencing flushing stores
2013-04-04 17:43:45,831 DEBUG org.apache.hadoop.hbase.util.FSUtils: Creating
file=hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/.tmp/7020db24b38246a5818e7eb4e130ad9e
with permission=rwxrwxrwx
2013-04-04 17:43:45,834 DEBUG org.apache.hadoop.hbase.io.hfile.HFileWriterV2:
Initialized with CacheConfig:enabled [cacheDataOnRead=true]
[cacheDataOnWrite=false] [cacheIndexesOnWrite=false]
[cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
2013-04-04 17:43:45,835 INFO org.apache.hadoop.hbase.regionserver.StoreFile:
Delete Family Bloom filter type for
hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/.tmp/7020db24b38246a5818e7eb4e130ad9e:
CompoundBloomFilterWriter
2013-04-04 17:43:46,074 INFO org.apache.hadoop.hbase.regionserver.StoreFile:
NO General Bloom and NO DeleteFamily was added to HFile
(hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/.tmp/7020db24b38246a5818e7eb4e130ad9e)
2013-04-04 17:43:46,074 INFO org.apache.hadoop.hbase.regionserver.HStore:
Flushed , sequenceid=1051817, memsize=42.3 M, into tmp file
hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/.tmp/7020db24b38246a5818e7eb4e130ad9e
2013-04-04 17:43:46,093 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionFileSystem: Committing store file
hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/.tmp/7020db24b38246a5818e7eb4e130ad9e
as
hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/f1/7020db24b38246a5818e7eb4e130ad9e
2013-04-04 17:43:46,103 INFO org.apache.hadoop.hbase.regionserver.HStore:
Added
hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/f1/7020db24b38246a5818e7eb4e130ad9e,
entries=54911, sequenceid=1051817, filesize=35.1 M
2013-04-04 17:43:46,103 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Finished memstore flush of ~42.3 M/44388936, currentsize=0/0 for region
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9. in 278ms,
sequenceid=-1, compaction requested=true
2013-04-04 17:43:56,335 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache:
Stats: total=394.09 MB, free=3.61 GB, max=4.00 GB, blocks=5263,
accesses=244882, hits=42988, hitRatio=17.55%, , cachingAccesses=49869,
cachingHits=24251, cachingHitsRatio=48.63%, , evictions=0, evicted=20349,
evictedPerRun=Infinity
2013-04-04 17:44:31,119 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Flushing
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9.
2013-04-04 17:44:31,120 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Started memstore flush for
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9., current region
memstore size 42.7 M
2013-04-04 17:44:31,120 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished snapshotting
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9., commencing wait for
mvcc, flushsize=44764248
[jira] [Commented] (HBASE-8579) TestDelayedRpc falis from time to time
[ https://issues.apache.org/jira/browse/HBASE-8579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662350#comment-13662350 ] stack commented on HBASE-8579: -- Committed to 0.95 and trunk. Leaving issue open to see if this takes care of the sporadic fails. TestDelayedRpc falis from time to time -- Key: HBASE-8579 URL: https://issues.apache.org/jira/browse/HBASE-8579 Project: HBase Issue Type: Bug Components: IPC/RPC Reporter: stack Fix For: 0.98.0, 0.95.1 Attachments: 8579.txt It failed here http://54.241.6.143/job/HBase-0.95/291/ Looks like basic rpc timeout (timeout set to 1 second and complaint was that it had waited 1.6 seconds). I suppose this could happen on loaded server. Let me try upping timeouts for now. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8573) Store last flushed sequence id for each store of region for Distributed Log Replay
[
https://issues.apache.org/jira/browse/HBASE-8573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662354#comment-13662354
]
Jeffrey Zhong commented on HBASE-8573:
--
First of all the exact description of HBase-6059 won't happen in
distributedLogReplay while the make deleted data exist again could still
happen in another race condition.(happen very rare though).
Below are steps how this could happen:
The situation is that we have two store files hstore1(sequence id = id1) and
hstore2(sequence id = id1) assuming hstore1 flushed successfully while hstore2
failed for some reasons. So we have hstore1(id1 bump up to id3) and s2(id1).
Assuming the flush of s1 contains a new put on row1. Currently distributed log
replay uses the min(flushed sequence ids of all stores) so we could replay the
put since we only skip edits since id1.
While if we had a delete on row1 before sequence id(id1) (a delete before a put
on row1), then the put replay could bring back the deleted data back because a
major compaction erased the delete.
hbase-6059 triggered a long discussion on this, it convinced me that
distributedLogReplay also needs flushed sequence if per store for replay.
[~zjushch]
{quote}
Maybe the only potential problem is the duplication of some data
{quote}
I think it's all right in the situation without compromising data correctness.
Store last flushed sequence id for each store of region for Distributed Log
Replay
--
Key: HBASE-8573
URL: https://issues.apache.org/jira/browse/HBASE-8573
Project: HBase
Issue Type: Improvement
Reporter: Ted Yu
HBASE-7006 stores last flushed sequence id of the region in zookeeper.
To prevent deleted data from appearing again, we should store last flushed
sequence id for each store of region in zookeeper.
See discussion here:
https://issues.apache.org/jira/browse/HBASE-7006?focusedCommentId=13660428page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13660428
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8067) TestHFileArchiving.testArchiveOnTableDelete sometimes fails
[
https://issues.apache.org/jira/browse/HBASE-8067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662356#comment-13662356
]
stack commented on HBASE-8067:
--
Pardon me [~lhofhansl], should have pasted this
http://54.241.6.143/job/HBase-TRUNK/306/
TestHFileArchiving.testArchiveOnTableDelete sometimes fails
---
Key: HBASE-8067
URL: https://issues.apache.org/jira/browse/HBASE-8067
Project: HBase
Issue Type: Bug
Components: Admin, master, test
Affects Versions: 0.94.6, 0.95.2
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Critical
Fix For: 0.98.0, 0.94.8, 0.95.1
Attachments: 8067-0.94.txt, 8067-trunk.txt, HBASE-8067-debug.patch,
HBASE-8067-v0.patch
it seems that testArchiveOnTableDelete() fails because the archiving in
DeleteTableHandler is still in progress when admin.deleteTable() returns.
{code}
Error Message
Archived files are missing some of the store files!
Stacktrace
java.lang.AssertionError: Archived files are missing some of the store files!
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.hbase.backup.TestHFileArchiving.testArchiveOnTableDelete(TestHFileArchiving.java:262)
{code}
(Looking at the problem in a more generic way, we don't have any way to
inform the client when an async operation is completed)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-7298) Upgrade hadoop 1 dependency to hadoop 1.1.2
[ https://issues.apache.org/jira/browse/HBASE-7298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-7298. --- Resolution: Implemented Upgrade hadoop 1 dependency to hadoop 1.1.2 --- Key: HBASE-7298 URL: https://issues.apache.org/jira/browse/HBASE-7298 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Ted Yu hadoop 1.1.2 release fixes a critical bug: https://issues.apache.org/jira/browse/HADOOP-9115 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8581) rpc refactor dropped passing the operation timeout through to the rpcclient
stack created HBASE-8581: Summary: rpc refactor dropped passing the operation timeout through to the rpcclient Key: HBASE-8581 URL: https://issues.apache.org/jira/browse/HBASE-8581 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack jd was poking and noticed that we were not passing the operation timeout down as rpc timeout as we used to in 0.94. Let me fix. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8067) TestHFileArchiving.testArchiveOnTableDelete sometimes fails
[
https://issues.apache.org/jira/browse/HBASE-8067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662366#comment-13662366
]
Lars Hofhansl commented on HBASE-8067:
--
Sigh. Yes. Was hoping you made a mistake :) Not so lucky. Do you think we
should revert the change?
TestHFileArchiving.testArchiveOnTableDelete sometimes fails
---
Key: HBASE-8067
URL: https://issues.apache.org/jira/browse/HBASE-8067
Project: HBase
Issue Type: Bug
Components: Admin, master, test
Affects Versions: 0.94.6, 0.95.2
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Critical
Fix For: 0.98.0, 0.94.8, 0.95.1
Attachments: 8067-0.94.txt, 8067-trunk.txt, HBASE-8067-debug.patch,
HBASE-8067-v0.patch
it seems that testArchiveOnTableDelete() fails because the archiving in
DeleteTableHandler is still in progress when admin.deleteTable() returns.
{code}
Error Message
Archived files are missing some of the store files!
Stacktrace
java.lang.AssertionError: Archived files are missing some of the store files!
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.hbase.backup.TestHFileArchiving.testArchiveOnTableDelete(TestHFileArchiving.java:262)
{code}
(Looking at the problem in a more generic way, we don't have any way to
inform the client when an async operation is completed)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8067) TestHFileArchiving.testArchiveOnTableDelete sometimes fails
[
https://issues.apache.org/jira/browse/HBASE-8067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662368#comment-13662368
]
Hudson commented on HBASE-8067:
---
Integrated in HBase-0.94-security #147 (See
[https://builds.apache.org/job/HBase-0.94-security/147/])
HBASE-8067 TestHFileArchiving.testArchiveOnTableDelete sometimes fails
(Revision 1484504)
Result = FAILURE
stack :
Files :
*
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchiving.java
TestHFileArchiving.testArchiveOnTableDelete sometimes fails
---
Key: HBASE-8067
URL: https://issues.apache.org/jira/browse/HBASE-8067
Project: HBase
Issue Type: Bug
Components: Admin, master, test
Affects Versions: 0.94.6, 0.95.2
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Critical
Fix For: 0.98.0, 0.94.8, 0.95.1
Attachments: 8067-0.94.txt, 8067-trunk.txt, HBASE-8067-debug.patch,
HBASE-8067-v0.patch
it seems that testArchiveOnTableDelete() fails because the archiving in
DeleteTableHandler is still in progress when admin.deleteTable() returns.
{code}
Error Message
Archived files are missing some of the store files!
Stacktrace
java.lang.AssertionError: Archived files are missing some of the store files!
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.hbase.backup.TestHFileArchiving.testArchiveOnTableDelete(TestHFileArchiving.java:262)
{code}
(Looking at the problem in a more generic way, we don't have any way to
inform the client when an async operation is completed)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8450) Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc.
[ https://issues.apache.org/jira/browse/HBASE-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662369#comment-13662369 ] Jean-Daniel Cryans commented on HBASE-8450: --- bq. From what I've seen, it's not that uncommon to have something above 60s, and retrying just adds some workload on the server? So originally I opened HBASE-4462 so that we stop retrying those timeouts, but people agreed that HBASE-2937 (the operation timeout included in 0.92.0) was covering that case. Passing an operation timeout actually changes the rpc timeout at the same time, and it does cancel the retries in ServerCallable (although in trunk I found out we don't change the rpc timeout, see HBASE-8581). I agree with you that setting that down to 10 would currently mean that we might retry timeouts on the region servers. What I'm wondering now is how we can make this better. Should we also tweak the default operation timeout lower? Maybe it set it higher for MR jobs? Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc. Key: HBASE-8450 URL: https://issues.apache.org/jira/browse/HBASE-8450 Project: HBase Issue Type: Task Components: Usability Reporter: stack Assignee: stack Priority: Critical Fix For: 0.95.1 Attachments: 8450.txt, 8450v2.txt, 8450v3.txt, 8450v5.txt This is a critical task we need to do before we release; review our defaults. On cursory review, there are configs in hbase-default.xml that no longer have matching code; there are some that should be changed because we know better now and others that should be amended because hardware and deploys are bigger than they used to be. We could also move stuff around so that the must-edit stuff is near the top (zk quorum config. is mid-way down the page) and beef up the descriptions -- especially since these descriptions shine through in the refguide. Lastly, I notice that our tgz does not include an hbase-default.xml other than the one bundled up in the jar. Maybe we should make it more accessible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8067) TestHFileArchiving.testArchiveOnTableDelete sometimes fails
[
https://issues.apache.org/jira/browse/HBASE-8067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662372#comment-13662372
]
stack commented on HBASE-8067:
--
[~lhofhansl] Nah. It makes sense. Matteo gave me a rundown on the race
condition. Was going to dig in on it.
TestHFileArchiving.testArchiveOnTableDelete sometimes fails
---
Key: HBASE-8067
URL: https://issues.apache.org/jira/browse/HBASE-8067
Project: HBase
Issue Type: Bug
Components: Admin, master, test
Affects Versions: 0.94.6, 0.95.2
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Critical
Fix For: 0.98.0, 0.94.8, 0.95.1
Attachments: 8067-0.94.txt, 8067-trunk.txt, HBASE-8067-debug.patch,
HBASE-8067-v0.patch
it seems that testArchiveOnTableDelete() fails because the archiving in
DeleteTableHandler is still in progress when admin.deleteTable() returns.
{code}
Error Message
Archived files are missing some of the store files!
Stacktrace
java.lang.AssertionError: Archived files are missing some of the store files!
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.hbase.backup.TestHFileArchiving.testArchiveOnTableDelete(TestHFileArchiving.java:262)
{code}
(Looking at the problem in a more generic way, we don't have any way to
inform the client when an async operation is completed)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8580) Ensure that there are 3 replicas when Flushing or Compacting
[ https://issues.apache.org/jira/browse/HBASE-8580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662376#comment-13662376 ] Sergey Shelukhin commented on HBASE-8580: - We are not guaranteed that replicas will not fail during, or immediately after, we finish writing, or later, so we are only privileging some small window of time to ensure 3 replicas, and trying to marginally improve reliability over HDFS by poking around in the (semi-)internals. How is this different from having 3 replicas at other time? Is there some specific bug/scenario you saw? Ensure that there are 3 replicas when Flushing or Compacting Key: HBASE-8580 URL: https://issues.apache.org/jira/browse/HBASE-8580 Project: HBase Issue Type: Improvement Reporter: Elliott Clark It is possible that datanodes in a DFS pipeline can be dropped. When this happens it possible for there to be fewer than the number of replicas expected. After a close on the file the namenode tries to re-replicate the file. On flush and compaction we should block waiting for the number of datanodes reporting that they have the needed blocks to be 3. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8067) TestHFileArchiving.testArchiveOnTableDelete sometimes fails
[
https://issues.apache.org/jira/browse/HBASE-8067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-8067:
-
Fix Version/s: (was: 0.94.8)
0.94.9
Moving out to 0.94.9
TestHFileArchiving.testArchiveOnTableDelete sometimes fails
---
Key: HBASE-8067
URL: https://issues.apache.org/jira/browse/HBASE-8067
Project: HBase
Issue Type: Bug
Components: Admin, master, test
Affects Versions: 0.94.6, 0.95.2
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Critical
Fix For: 0.98.0, 0.95.1, 0.94.9
Attachments: 8067-0.94.txt, 8067-trunk.txt, HBASE-8067-debug.patch,
HBASE-8067-v0.patch
it seems that testArchiveOnTableDelete() fails because the archiving in
DeleteTableHandler is still in progress when admin.deleteTable() returns.
{code}
Error Message
Archived files are missing some of the store files!
Stacktrace
java.lang.AssertionError: Archived files are missing some of the store files!
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.hbase.backup.TestHFileArchiving.testArchiveOnTableDelete(TestHFileArchiving.java:262)
{code}
(Looking at the problem in a more generic way, we don't have any way to
inform the client when an async operation is completed)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8282) User triggered flushes does not allow compaction to get triggered even if compaction criteria is met
[
https://issues.apache.org/jira/browse/HBASE-8282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662378#comment-13662378
]
Lars Hofhansl commented on HBASE-8282:
--
0.94?
User triggered flushes does not allow compaction to get triggered even if
compaction criteria is met
Key: HBASE-8282
URL: https://issues.apache.org/jira/browse/HBASE-8282
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.98.0, 0.94.8, 0.95.1
Attachments: HBASE-8282_trunk_1.patch, HBASE-8282_trunk.patch
Observe the below logs
{code}
2013-04-04 17:43:45,825 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Flushing
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9.
2013-04-04 17:43:45,826 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Started memstore flush for
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9., current region
memstore size 42.3 M
2013-04-04 17:43:45,826 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished snapshotting
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9., commencing wait for
mvcc, flushsize=44388936
2013-04-04 17:43:45,826 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished snapshotting, commencing flushing stores
2013-04-04 17:43:45,831 DEBUG org.apache.hadoop.hbase.util.FSUtils: Creating
file=hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/.tmp/7020db24b38246a5818e7eb4e130ad9e
with permission=rwxrwxrwx
2013-04-04 17:43:45,834 DEBUG org.apache.hadoop.hbase.io.hfile.HFileWriterV2:
Initialized with CacheConfig:enabled [cacheDataOnRead=true]
[cacheDataOnWrite=false] [cacheIndexesOnWrite=false]
[cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
2013-04-04 17:43:45,835 INFO org.apache.hadoop.hbase.regionserver.StoreFile:
Delete Family Bloom filter type for
hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/.tmp/7020db24b38246a5818e7eb4e130ad9e:
CompoundBloomFilterWriter
2013-04-04 17:43:46,074 INFO org.apache.hadoop.hbase.regionserver.StoreFile:
NO General Bloom and NO DeleteFamily was added to HFile
(hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/.tmp/7020db24b38246a5818e7eb4e130ad9e)
2013-04-04 17:43:46,074 INFO org.apache.hadoop.hbase.regionserver.HStore:
Flushed , sequenceid=1051817, memsize=42.3 M, into tmp file
hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/.tmp/7020db24b38246a5818e7eb4e130ad9e
2013-04-04 17:43:46,093 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionFileSystem: Committing store file
hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/.tmp/7020db24b38246a5818e7eb4e130ad9e
as
hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/f1/7020db24b38246a5818e7eb4e130ad9e
2013-04-04 17:43:46,103 INFO org.apache.hadoop.hbase.regionserver.HStore:
Added
hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/f1/7020db24b38246a5818e7eb4e130ad9e,
entries=54911, sequenceid=1051817, filesize=35.1 M
2013-04-04 17:43:46,103 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Finished memstore flush of ~42.3 M/44388936, currentsize=0/0 for region
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9. in 278ms,
sequenceid=-1, compaction requested=true
2013-04-04 17:43:56,335 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache:
Stats: total=394.09 MB, free=3.61 GB, max=4.00 GB, blocks=5263,
accesses=244882, hits=42988, hitRatio=17.55%, , cachingAccesses=49869,
cachingHits=24251, cachingHitsRatio=48.63%, , evictions=0, evicted=20349,
evictedPerRun=Infinity
2013-04-04 17:44:31,119 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Flushing
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9.
2013-04-04 17:44:31,120 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Started memstore flush for
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9., current region
memstore size 42.7 M
2013-04-04 17:44:31,120 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished snapshotting
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9., commencing wait for
mvcc, flushsize=44764248
2013-04-04 17:44:31,120 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished snapshotting, commencing flushing stores
2013-04-04 17:44:31,136 DEBUG org.apache.hadoop.hbase.util.FSUtils: Creating
file=hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/.tmp/c5c2cd6aaf1644daa9c858149da39081
with permission=rwxrwxrwx
2013-04-04 17:44:31,139 DEBUG org.apache.hadoop.hbase.io.hfile.HFileWriterV2:
Initialized with
[jira] [Commented] (HBASE-8580) Ensure that there are 3 replicas when Flushing or Compacting
[ https://issues.apache.org/jira/browse/HBASE-8580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662384#comment-13662384 ] Enis Soztutar commented on HBASE-8580: -- We do log rolling if we detect that the replica count has gone below hbase.regionserver.hlog.tolerable.lowreplication (which is 3 on hdfs) for every sync in WAL. For flushes, and compactions, we do not care about the replication count for the blocks that are being written, but we might care for the min replication for all the blocks in the file after finalizing the file and before committing the results. For compaction for example, we probably do not want to replace files with replication 3 with compacted file with replication 2. But as Sergey said, datanodes can still fail just after we checked etc. Ensure that there are 3 replicas when Flushing or Compacting Key: HBASE-8580 URL: https://issues.apache.org/jira/browse/HBASE-8580 Project: HBase Issue Type: Improvement Reporter: Elliott Clark It is possible that datanodes in a DFS pipeline can be dropped. When this happens it possible for there to be fewer than the number of replicas expected. After a close on the file the namenode tries to re-replicate the file. On flush and compaction we should block waiting for the number of datanodes reporting that they have the needed blocks to be 3. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8420) Port HBASE-6874 Implement prefetching for scanners from 0.89-fb
[ https://issues.apache.org/jira/browse/HBASE-8420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-8420: --- Attachment: trunk-8420_v5.patch Port HBASE-6874 Implement prefetching for scanners from 0.89-fb - Key: HBASE-8420 URL: https://issues.apache.org/jira/browse/HBASE-8420 Project: HBase Issue Type: Improvement Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: 0.94-8420_v1.patch, 0.94-8420_v2.patch, trunk-8420_v1.patch, trunk-8420_v2.patch, trunk-8420_v3.patch, trunk-8420_v4.patch, trunk-8420_v5.patch This should help scanner performance. We should have it in trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8420) Port HBASE-6874 Implement prefetching for scanners from 0.89-fb
[ https://issues.apache.org/jira/browse/HBASE-8420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-8420: --- Status: Patch Available (was: Open) Attached v5. In case a submitted task is interrupted by cancel(), the prefetched result size counter is reduced if it is already increased. Also added some change to make the test not flaky by making sure the right region scanner holder is used to check prefetching flag/counter in the test. Port HBASE-6874 Implement prefetching for scanners from 0.89-fb - Key: HBASE-8420 URL: https://issues.apache.org/jira/browse/HBASE-8420 Project: HBase Issue Type: Improvement Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: 0.94-8420_v1.patch, 0.94-8420_v2.patch, trunk-8420_v1.patch, trunk-8420_v2.patch, trunk-8420_v3.patch, trunk-8420_v4.patch, trunk-8420_v5.patch This should help scanner performance. We should have it in trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8497) Protobuf WAL also needs a trailer
[ https://issues.apache.org/jira/browse/HBASE-8497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-8497: --- Attachment: HBASE-8497-v5.patch I did the following changes in this patch: 1) Renamed WALTrailerOffset to lastPositionToRead (as I commented on the rb). This is the last byte position of WALEdits, so, the next() method should read up to this point. 2) Make the maximum trailer size configurable. To *avoid any surprise* for user, if the trailer size configured_max_allow_size, a WARN message is printed in the log, but the trailer is respected (it is written/read). 3) Incorporated the style nits suggested by jon. mvn test TestHLog* pass. Protobuf WAL also needs a trailer -- Key: HBASE-8497 URL: https://issues.apache.org/jira/browse/HBASE-8497 Project: HBase Issue Type: Sub-task Components: Protobufs, wal Affects Versions: 0.95.1 Reporter: Enis Soztutar Assignee: Himanshu Vashishtha Fix For: 0.98.0, 0.95.1 Attachments: HBASE-8497-v0.patch, HBASE-8497-v2.patch, HBASE-8497-v3.patch, HBASE-8497-v4.patch, HBASE-8497-v5.patch New Protobuf WAL has a header, but we will probably need a trailer as well, reserved for later usage. Right now, we can we just serialize an empty trailer, but putting more metadata there, like range of sequence_id's, region names, table names etc might be needed in the future. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8067) TestHFileArchiving.testArchiveOnTableDelete sometimes fails
[
https://issues.apache.org/jira/browse/HBASE-8067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662408#comment-13662408
]
Hudson commented on HBASE-8067:
---
Integrated in HBase-0.94 #991 (See
[https://builds.apache.org/job/HBase-0.94/991/])
HBASE-8067 TestHFileArchiving.testArchiveOnTableDelete sometimes fails
(Revision 1484504)
Result = SUCCESS
stack :
Files :
*
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchiving.java
TestHFileArchiving.testArchiveOnTableDelete sometimes fails
---
Key: HBASE-8067
URL: https://issues.apache.org/jira/browse/HBASE-8067
Project: HBase
Issue Type: Bug
Components: Admin, master, test
Affects Versions: 0.94.6, 0.95.2
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Critical
Fix For: 0.98.0, 0.95.1, 0.94.9
Attachments: 8067-0.94.txt, 8067-trunk.txt, HBASE-8067-debug.patch,
HBASE-8067-v0.patch
it seems that testArchiveOnTableDelete() fails because the archiving in
DeleteTableHandler is still in progress when admin.deleteTable() returns.
{code}
Error Message
Archived files are missing some of the store files!
Stacktrace
java.lang.AssertionError: Archived files are missing some of the store files!
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.hbase.backup.TestHFileArchiving.testArchiveOnTableDelete(TestHFileArchiving.java:262)
{code}
(Looking at the problem in a more generic way, we don't have any way to
inform the client when an async operation is completed)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8420) Port HBASE-6874 Implement prefetching for scanners from 0.89-fb
[ https://issues.apache.org/jira/browse/HBASE-8420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-8420: --- Status: Open (was: Patch Available) There still could be racing issue in resetting the counter. Will keep it simple. Port HBASE-6874 Implement prefetching for scanners from 0.89-fb - Key: HBASE-8420 URL: https://issues.apache.org/jira/browse/HBASE-8420 Project: HBase Issue Type: Improvement Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: 0.94-8420_v1.patch, 0.94-8420_v2.patch, trunk-8420_v1.patch, trunk-8420_v2.patch, trunk-8420_v3.patch, trunk-8420_v4.patch, trunk-8420_v5.patch This should help scanner performance. We should have it in trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8580) Ensure that there are 3 replicas when Flushing or Compacting
[ https://issues.apache.org/jira/browse/HBASE-8580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662413#comment-13662413 ] Elliott Clark commented on HBASE-8580: -- Yes there is no guarantee that things won't degrade. But before we move files in from tmp we should know that they were on 3 datanodes at least when we did the move. Mostly this came from a user in #hbase irc. He was seeing hfile corruption when region servers were crashing on heavy compactions. Ensure that there are 3 replicas when Flushing or Compacting Key: HBASE-8580 URL: https://issues.apache.org/jira/browse/HBASE-8580 Project: HBase Issue Type: Improvement Reporter: Elliott Clark It is possible that datanodes in a DFS pipeline can be dropped. When this happens it possible for there to be fewer than the number of replicas expected. After a close on the file the namenode tries to re-replicate the file. On flush and compaction we should block waiting for the number of datanodes reporting that they have the needed blocks to be 3. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8420) Port HBASE-6874 Implement prefetching for scanners from 0.89-fb
[
https://issues.apache.org/jira/browse/HBASE-8420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662417#comment-13662417
]
Jean-Daniel Cryans commented on HBASE-8420:
---
The latest patch looks good. Only thing I don't like so much is the handling of
those exceptions since they would bubble up as-is to the client and the
interrupt isn't reset:
{code}
} catch (InterruptedException ie) {
throw new IOException(ie);
} catch (ExecutionException ee) {
throw new IOException(ee);
}
{code}
Port HBASE-6874 Implement prefetching for scanners from 0.89-fb
-
Key: HBASE-8420
URL: https://issues.apache.org/jira/browse/HBASE-8420
Project: HBase
Issue Type: Improvement
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Attachments: 0.94-8420_v1.patch, 0.94-8420_v2.patch,
trunk-8420_v1.patch, trunk-8420_v2.patch, trunk-8420_v3.patch,
trunk-8420_v4.patch, trunk-8420_v5.patch
This should help scanner performance. We should have it in trunk.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8564) TestMetricsRegionServer depends on test order
[ https://issues.apache.org/jira/browse/HBASE-8564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662419#comment-13662419 ] Jeffrey Zhong commented on HBASE-8564: -- I think the latest check in on this JIRA broke the integration test run. Today I tried to run an integration test like following: mvn failsafe:integration-test -Dit.test=IntegrationTestBigLinkedList Got following errors: Tests in error: testContinuousIngest(org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList): org/apache/hadoop/hbase/test/MetricsAssertHelper testContinuousIngest(org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList): null not an instance of org.apache.hadoop.hbase.MiniHBaseCluster The same happened in jenkins as well http://54.241.6.143/job/HBase-TRUNK/305/consoleFull. Thanks, -Jeffrey TestMetricsRegionServer depends on test order - Key: HBASE-8564 URL: https://issues.apache.org/jira/browse/HBASE-8564 Project: HBase Issue Type: Bug Affects Versions: 0.98.0, 0.95.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.98.0, 0.95.1 Attachments: HBASE-8564-0.patch, HBASE-8564-1.patch, HBASE-8564-ADD-0.patch TestMetricsRegionServer has failed a few times on trunk. It's passing depends upon test ordering so will fail sometimes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-8261) Backport HBASE-7718 TestClassLoading needs to consider runtime classpath in buildCoprocessorJar
[ https://issues.apache.org/jira/browse/HBASE-8261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-8261. -- Resolution: Duplicate Fix Version/s: (was: 0.94.8) Looks like the code has changed a lot 0.94 and this is no longer relevant. [~jmspaggi] Please reopen if my assessment is incorrect. Backport HBASE-7718 TestClassLoading needs to consider runtime classpath in buildCoprocessorJar --- Key: HBASE-8261 URL: https://issues.apache.org/jira/browse/HBASE-8261 Project: HBase Issue Type: Task Reporter: stack Assignee: Jean-Marc Spaggiari See patch attached to HBASE-7718 for 0.94. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8282) User triggered flushes does not allow compaction to get triggered even if compaction criteria is met
[
https://issues.apache.org/jira/browse/HBASE-8282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662421#comment-13662421
]
Lars Hofhansl commented on HBASE-8282:
--
I'm happy to either push to 0.94.9 or say that we won't fix this in 0.94.
Opinions?
User triggered flushes does not allow compaction to get triggered even if
compaction criteria is met
Key: HBASE-8282
URL: https://issues.apache.org/jira/browse/HBASE-8282
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.98.0, 0.94.8, 0.95.1
Attachments: HBASE-8282_trunk_1.patch, HBASE-8282_trunk.patch
Observe the below logs
{code}
2013-04-04 17:43:45,825 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Flushing
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9.
2013-04-04 17:43:45,826 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Started memstore flush for
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9., current region
memstore size 42.3 M
2013-04-04 17:43:45,826 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished snapshotting
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9., commencing wait for
mvcc, flushsize=44388936
2013-04-04 17:43:45,826 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished snapshotting, commencing flushing stores
2013-04-04 17:43:45,831 DEBUG org.apache.hadoop.hbase.util.FSUtils: Creating
file=hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/.tmp/7020db24b38246a5818e7eb4e130ad9e
with permission=rwxrwxrwx
2013-04-04 17:43:45,834 DEBUG org.apache.hadoop.hbase.io.hfile.HFileWriterV2:
Initialized with CacheConfig:enabled [cacheDataOnRead=true]
[cacheDataOnWrite=false] [cacheIndexesOnWrite=false]
[cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
2013-04-04 17:43:45,835 INFO org.apache.hadoop.hbase.regionserver.StoreFile:
Delete Family Bloom filter type for
hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/.tmp/7020db24b38246a5818e7eb4e130ad9e:
CompoundBloomFilterWriter
2013-04-04 17:43:46,074 INFO org.apache.hadoop.hbase.regionserver.StoreFile:
NO General Bloom and NO DeleteFamily was added to HFile
(hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/.tmp/7020db24b38246a5818e7eb4e130ad9e)
2013-04-04 17:43:46,074 INFO org.apache.hadoop.hbase.regionserver.HStore:
Flushed , sequenceid=1051817, memsize=42.3 M, into tmp file
hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/.tmp/7020db24b38246a5818e7eb4e130ad9e
2013-04-04 17:43:46,093 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionFileSystem: Committing store file
hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/.tmp/7020db24b38246a5818e7eb4e130ad9e
as
hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/f1/7020db24b38246a5818e7eb4e130ad9e
2013-04-04 17:43:46,103 INFO org.apache.hadoop.hbase.regionserver.HStore:
Added
hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/f1/7020db24b38246a5818e7eb4e130ad9e,
entries=54911, sequenceid=1051817, filesize=35.1 M
2013-04-04 17:43:46,103 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Finished memstore flush of ~42.3 M/44388936, currentsize=0/0 for region
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9. in 278ms,
sequenceid=-1, compaction requested=true
2013-04-04 17:43:56,335 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache:
Stats: total=394.09 MB, free=3.61 GB, max=4.00 GB, blocks=5263,
accesses=244882, hits=42988, hitRatio=17.55%, , cachingAccesses=49869,
cachingHits=24251, cachingHitsRatio=48.63%, , evictions=0, evicted=20349,
evictedPerRun=Infinity
2013-04-04 17:44:31,119 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Flushing
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9.
2013-04-04 17:44:31,120 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Started memstore flush for
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9., current region
memstore size 42.7 M
2013-04-04 17:44:31,120 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished snapshotting
MyTable,,1365111467842.f6792086ad3518dee244e7bf2761a1f9., commencing wait for
mvcc, flushsize=44764248
2013-04-04 17:44:31,120 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished snapshotting, commencing flushing stores
2013-04-04 17:44:31,136 DEBUG org.apache.hadoop.hbase.util.FSUtils: Creating
file=hdfs://localhost:9010/hbase/MyTable/f6792086ad3518dee244e7bf2761a1f9/.tmp/c5c2cd6aaf1644daa9c858149da39081
with permission=rwxrwxrwx
2013-04-04 17:44:31,139
[jira] [Commented] (HBASE-8564) TestMetricsRegionServer depends on test order
[ https://issues.apache.org/jira/browse/HBASE-8564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662425#comment-13662425 ] Elliott Clark commented on HBASE-8564: -- Hmmm I'm running this on a real cluster for integration tests. Maybe there's somthing with integration test + MiniCluster ? TestMetricsRegionServer depends on test order - Key: HBASE-8564 URL: https://issues.apache.org/jira/browse/HBASE-8564 Project: HBase Issue Type: Bug Affects Versions: 0.98.0, 0.95.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.98.0, 0.95.1 Attachments: HBASE-8564-0.patch, HBASE-8564-1.patch, HBASE-8564-ADD-0.patch TestMetricsRegionServer has failed a few times on trunk. It's passing depends upon test ordering so will fail sometimes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8564) TestMetricsRegionServer depends on test order
[ https://issues.apache.org/jira/browse/HBASE-8564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662444#comment-13662444 ] Jeffrey Zhong commented on HBASE-8564: -- Yeah, it failed against miniCluster runs because MetricsAssertHelper class can't be found. TestMetricsRegionServer depends on test order - Key: HBASE-8564 URL: https://issues.apache.org/jira/browse/HBASE-8564 Project: HBase Issue Type: Bug Affects Versions: 0.98.0, 0.95.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.98.0, 0.95.1 Attachments: HBASE-8564-0.patch, HBASE-8564-1.patch, HBASE-8564-ADD-0.patch TestMetricsRegionServer has failed a few times on trunk. It's passing depends upon test ordering so will fail sometimes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8497) Protobuf WAL also needs a trailer
[
https://issues.apache.org/jira/browse/HBASE-8497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662446#comment-13662446
]
Hadoop QA commented on HBASE-8497:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12583896/HBASE-8497-v5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestFromClientSide
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5757//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5757//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5757//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5757//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5757//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5757//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5757//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5757//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5757//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5757//console
This message is automatically generated.
Protobuf WAL also needs a trailer
--
Key: HBASE-8497
URL: https://issues.apache.org/jira/browse/HBASE-8497
Project: HBase
Issue Type: Sub-task
Components: Protobufs, wal
Affects Versions: 0.95.1
Reporter: Enis Soztutar
Assignee: Himanshu Vashishtha
Fix For: 0.98.0, 0.95.1
Attachments: HBASE-8497-v0.patch, HBASE-8497-v2.patch,
HBASE-8497-v3.patch, HBASE-8497-v4.patch, HBASE-8497-v5.patch
New Protobuf WAL has a header, but we will probably need a trailer as well,
reserved for later usage.
Right now, we can we just serialize an empty trailer, but putting more
metadata there, like range of sequence_id's, region names, table names etc
might be needed in the future.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8420) Port HBASE-6874 Implement prefetching for scanners from 0.89-fb
[
https://issues.apache.org/jira/browse/HBASE-8420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662447#comment-13662447
]
Hadoop QA commented on HBASE-8420:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12583891/trunk-8420_v5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestTableLockManager
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5758//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5758//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5758//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5758//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5758//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5758//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5758//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5758//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5758//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5758//console
This message is automatically generated.
Port HBASE-6874 Implement prefetching for scanners from 0.89-fb
-
Key: HBASE-8420
URL: https://issues.apache.org/jira/browse/HBASE-8420
Project: HBase
Issue Type: Improvement
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Attachments: 0.94-8420_v1.patch, 0.94-8420_v2.patch,
trunk-8420_v1.patch, trunk-8420_v2.patch, trunk-8420_v3.patch,
trunk-8420_v4.patch, trunk-8420_v5.patch
This should help scanner performance. We should have it in trunk.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8420) Port HBASE-6874 Implement prefetching for scanners from 0.89-fb
[ https://issues.apache.org/jira/browse/HBASE-8420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662454#comment-13662454 ] Jimmy Xiang commented on HBASE-8420: bq. Only thing I don't like so much is the handling of those exceptions since they would bubble up as-is to the client and the interrupt isn't reset I will address this in the next patch. Thanks a lot. Port HBASE-6874 Implement prefetching for scanners from 0.89-fb - Key: HBASE-8420 URL: https://issues.apache.org/jira/browse/HBASE-8420 Project: HBase Issue Type: Improvement Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: 0.94-8420_v1.patch, 0.94-8420_v2.patch, trunk-8420_v1.patch, trunk-8420_v2.patch, trunk-8420_v3.patch, trunk-8420_v4.patch, trunk-8420_v5.patch This should help scanner performance. We should have it in trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8497) Protobuf WAL also needs a trailer
[ https://issues.apache.org/jira/browse/HBASE-8497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662458#comment-13662458 ] Himanshu Vashishtha commented on HBASE-8497: org.apache.hadoop.hbase.client.TestFromClientSide passes for me at local; also, its cousin org.apache.hadoop.hbase.client.TestFromClientSideWithCP passes on this qa run. So, it looks more of the test run issue here. Protobuf WAL also needs a trailer -- Key: HBASE-8497 URL: https://issues.apache.org/jira/browse/HBASE-8497 Project: HBase Issue Type: Sub-task Components: Protobufs, wal Affects Versions: 0.95.1 Reporter: Enis Soztutar Assignee: Himanshu Vashishtha Fix For: 0.98.0, 0.95.1 Attachments: HBASE-8497-v0.patch, HBASE-8497-v2.patch, HBASE-8497-v3.patch, HBASE-8497-v4.patch, HBASE-8497-v5.patch New Protobuf WAL has a header, but we will probably need a trailer as well, reserved for later usage. Right now, we can we just serialize an empty trailer, but putting more metadata there, like range of sequence_id's, region names, table names etc might be needed in the future. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8564) TestMetricsRegionServer depends on test order
[ https://issues.apache.org/jira/browse/HBASE-8564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662465#comment-13662465 ] Elliott Clark commented on HBASE-8564: -- That's really weird it should be on the classpath, I thought. TestMetricsRegionServer depends on test order - Key: HBASE-8564 URL: https://issues.apache.org/jira/browse/HBASE-8564 Project: HBase Issue Type: Bug Affects Versions: 0.98.0, 0.95.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.98.0, 0.95.1 Attachments: HBASE-8564-0.patch, HBASE-8564-1.patch, HBASE-8564-ADD-0.patch TestMetricsRegionServer has failed a few times on trunk. It's passing depends upon test ordering so will fail sometimes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8581) rpc refactor dropped passing the operation timeout through to the rpcclient
[
https://issues.apache.org/jira/browse/HBASE-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-8581:
-
Attachment: 8581.txt
Here is a patch to restore what we had in 0.94 passing down an operation
timeout.
{code}
1 In HTable, allow setting an operation timeout on meta. Needed testing.
2
3 In ServerCallable, do a little refactor so cleaner around checking for
timeout.
4
5 In RpcClient, add back the getRpcTimeout that takes a default timeout and
6 returns minimum of default rpc timeout and whatever is set in the thread
local.
7 Call this new method on setup of a 'Service' the first time (In 0.94 it was
8 called when we set up a new proxy, which equates to about the same thing).
9
10 In TestClientNoCluster, add a test that verifies we get
SocketTimeoutException
11 when the operation timeout is reached rather than a retries exhausted
12 exception which is what we were getting when before this patch.
13
14 Add a log4j.properties to hbase-client so we get a bit of logging.
{code}
rpc refactor dropped passing the operation timeout through to the rpcclient
---
Key: HBASE-8581
URL: https://issues.apache.org/jira/browse/HBASE-8581
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: stack
Attachments: 8581.txt
jd was poking and noticed that we were not passing the operation timeout down
as rpc timeout as we used to in 0.94. Let me fix.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8581) rpc refactor dropped passing the operation timeout through to the rpcclient
[ https://issues.apache.org/jira/browse/HBASE-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-8581: - Status: Patch Available (was: Open) rpc refactor dropped passing the operation timeout through to the rpcclient --- Key: HBASE-8581 URL: https://issues.apache.org/jira/browse/HBASE-8581 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Attachments: 8581.txt jd was poking and noticed that we were not passing the operation timeout down as rpc timeout as we used to in 0.94. Let me fix. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8581) rpc refactor dropped passing the operation timeout through to the rpcclient
[ https://issues.apache.org/jira/browse/HBASE-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-8581: - Fix Version/s: 0.95.1 0.98.0 rpc refactor dropped passing the operation timeout through to the rpcclient --- Key: HBASE-8581 URL: https://issues.apache.org/jira/browse/HBASE-8581 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Fix For: 0.98.0, 0.95.1 Attachments: 8581.txt jd was poking and noticed that we were not passing the operation timeout down as rpc timeout as we used to in 0.94. Let me fix. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8497) Protobuf WAL also needs a trailer
[ https://issues.apache.org/jira/browse/HBASE-8497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662479#comment-13662479 ] Sergey Shelukhin commented on HBASE-8497: - left some comments. Main things - I had left a comment before about some discrepancy (Hmm... trailerSizeOffset points to the first byte of size 4-byte buffer). Also, the size limit was added as safety check. Making it a config parameter whose only effect is to output a warning that user has no idea what to do with is not helpful imho. There's no safety check. There's nothing for user to do really - he can ignore the warnings, or bump the config which basically means ignore the warning. Might as well remove the entire limit thing :) Protobuf WAL also needs a trailer -- Key: HBASE-8497 URL: https://issues.apache.org/jira/browse/HBASE-8497 Project: HBase Issue Type: Sub-task Components: Protobufs, wal Affects Versions: 0.95.1 Reporter: Enis Soztutar Assignee: Himanshu Vashishtha Fix For: 0.98.0, 0.95.1 Attachments: HBASE-8497-v0.patch, HBASE-8497-v2.patch, HBASE-8497-v3.patch, HBASE-8497-v4.patch, HBASE-8497-v5.patch New Protobuf WAL has a header, but we will probably need a trailer as well, reserved for later usage. Right now, we can we just serialize an empty trailer, but putting more metadata there, like range of sequence_id's, region names, table names etc might be needed in the future. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8581) rpc refactor dropped passing the operation timeout through to the rpcclient
[ https://issues.apache.org/jira/browse/HBASE-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662480#comment-13662480 ] stack commented on HBASE-8581: -- Looking at 0.94, scanners have different timeout mechanism than this one. rpc refactor dropped passing the operation timeout through to the rpcclient --- Key: HBASE-8581 URL: https://issues.apache.org/jira/browse/HBASE-8581 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Fix For: 0.98.0, 0.95.1 Attachments: 8581.txt jd was poking and noticed that we were not passing the operation timeout down as rpc timeout as we used to in 0.94. Let me fix. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8581) rpc refactor dropped passing the operation timeout through to the rpcclient
[
https://issues.apache.org/jira/browse/HBASE-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662482#comment-13662482
]
Hadoop QA commented on HBASE-8581:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12583917/8581.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5759//console
This message is automatically generated.
rpc refactor dropped passing the operation timeout through to the rpcclient
---
Key: HBASE-8581
URL: https://issues.apache.org/jira/browse/HBASE-8581
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: stack
Fix For: 0.98.0, 0.95.1
Attachments: 8581.txt
jd was poking and noticed that we were not passing the operation timeout down
as rpc timeout as we used to in 0.94. Let me fix.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8580) Ensure that there are 3 replicas when Flushing or Compacting
[ https://issues.apache.org/jira/browse/HBASE-8580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662483#comment-13662483 ] Sergey Shelukhin commented on HBASE-8580: - Interesting... RSes crashing, or you mean data nodes crashing? Ensure that there are 3 replicas when Flushing or Compacting Key: HBASE-8580 URL: https://issues.apache.org/jira/browse/HBASE-8580 Project: HBase Issue Type: Improvement Reporter: Elliott Clark It is possible that datanodes in a DFS pipeline can be dropped. When this happens it possible for there to be fewer than the number of replicas expected. After a close on the file the namenode tries to re-replicate the file. On flush and compaction we should block waiting for the number of datanodes reporting that they have the needed blocks to be 3. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7942) Make use of the plumbing in HBASE-7932 to provide location hints when region files are created
[
https://issues.apache.org/jira/browse/HBASE-7942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662490#comment-13662490
]
Ted Yu commented on HBASE-7942:
---
{code}
+interface FavoredNodesForRegion {
+ void updateRegionFavoredNodesMapping(String encodedRegionName,
ListServerName favoredNodes);
+ InetSocketAddress[] getFavoredNodesForRegion(String encodedRegionName);
{code}
Please add javadoc for the methods above.
Make use of the plumbing in HBASE-7932 to provide location hints when region
files are created
--
Key: HBASE-7942
URL: https://issues.apache.org/jira/browse/HBASE-7942
Project: HBase
Issue Type: Sub-task
Reporter: Devaraj Das
Assignee: Devaraj Das
Priority: Critical
Fix For: 0.95.1
Attachments: 7942-trunk-1.3.patch, 7942-trunk-1.4.patch
This functionality is blocked by the fixes in HDFS. Reflection is going to
be used to figure out whether the underlying Hadoop version has the new API.
Things will behave the same way if the underlying HDFS version doesn't have
the new API.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8581) rpc refactor dropped passing the operation timeout through to the rpcclient
[
https://issues.apache.org/jira/browse/HBASE-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662492#comment-13662492
]
Jean-Daniel Cryans commented on HBASE-8581:
---
That's a sweet test Stack, I would just add a fail() in
RetriesExhaustedException to make it easier to debug. Another nit, {{duration}}
is not super descriptive and it's the duration of a single call and there could
be many of them. Someone could think it's the duration of the whole
{{ServerCallable}}.
rpc refactor dropped passing the operation timeout through to the rpcclient
---
Key: HBASE-8581
URL: https://issues.apache.org/jira/browse/HBASE-8581
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: stack
Fix For: 0.98.0, 0.95.1
Attachments: 8581.txt
jd was poking and noticed that we were not passing the operation timeout down
as rpc timeout as we used to in 0.94. Let me fix.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7942) Make use of the plumbing in HBASE-7932 to provide location hints when region files are created
[ https://issues.apache.org/jira/browse/HBASE-7942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662496#comment-13662496 ] Sergey Shelukhin commented on HBASE-7942: - or add LOG.trace, for the latter case. Otherwise looks ok. Did you try it on real cluster? Make use of the plumbing in HBASE-7932 to provide location hints when region files are created -- Key: HBASE-7942 URL: https://issues.apache.org/jira/browse/HBASE-7942 Project: HBase Issue Type: Sub-task Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.95.1 Attachments: 7942-trunk-1.3.patch, 7942-trunk-1.4.patch This functionality is blocked by the fixes in HDFS. Reflection is going to be used to figure out whether the underlying Hadoop version has the new API. Things will behave the same way if the underlying HDFS version doesn't have the new API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8582) Possible NullPointerException in ZKInterProcessLockBase#visitLocks
Ted Yu created HBASE-8582:
-
Summary: Possible NullPointerException in
ZKInterProcessLockBase#visitLocks
Key: HBASE-8582
URL: https://issues.apache.org/jira/browse/HBASE-8582
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Attachments: 8582-v1.txt
Running test suite on hadoop 2.0 I saw the following test failure:
{code}
testErrorReporter(org.apache.hadoop.hbase.util.TestHBaseFsck) Time elapsed:
0.003 sec ERROR!
java.lang.NullPointerException
at
org.apache.hadoop.hbase.zookeeper.lock.ZKInterProcessLockBase.visitLocks(ZKInterProcessLockBase.java:426)
at
org.apache.hadoop.hbase.master.TableLockManager$ZKTableLockManager.visitAllLocks(TableLockManager.java:386)
at
org.apache.hadoop.hbase.util.hbck.TableLockChecker.checkTableLocks(TableLockChecker.java:76)
at
org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixTableLocks(HBaseFsck.java:2480)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:460)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:65)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:41)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:36)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testErrorReporter(TestHBaseFsck.java:1868)
{code}
Here is related code:
{code}
public void visitLocks(MetadataHandler handler) throws IOException {
ListString children;
try {
children = ZKUtil.listChildrenNoWatch(zkWatcher, parentLockNode);
{code}
Looks like children was null.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-8582) Possible NullPointerException in ZKInterProcessLockBase#visitLocks
[
https://issues.apache.org/jira/browse/HBASE-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu reassigned HBASE-8582:
-
Assignee: Ted Yu
Possible NullPointerException in ZKInterProcessLockBase#visitLocks
--
Key: HBASE-8582
URL: https://issues.apache.org/jira/browse/HBASE-8582
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Attachments: 8582-v1.txt
Running test suite on hadoop 2.0 I saw the following test failure:
{code}
testErrorReporter(org.apache.hadoop.hbase.util.TestHBaseFsck) Time elapsed:
0.003 sec ERROR!
java.lang.NullPointerException
at
org.apache.hadoop.hbase.zookeeper.lock.ZKInterProcessLockBase.visitLocks(ZKInterProcessLockBase.java:426)
at
org.apache.hadoop.hbase.master.TableLockManager$ZKTableLockManager.visitAllLocks(TableLockManager.java:386)
at
org.apache.hadoop.hbase.util.hbck.TableLockChecker.checkTableLocks(TableLockChecker.java:76)
at
org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixTableLocks(HBaseFsck.java:2480)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:460)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:65)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:41)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:36)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testErrorReporter(TestHBaseFsck.java:1868)
{code}
Here is related code:
{code}
public void visitLocks(MetadataHandler handler) throws IOException {
ListString children;
try {
children = ZKUtil.listChildrenNoWatch(zkWatcher, parentLockNode);
{code}
Looks like children was null.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8582) Possible NullPointerException in ZKInterProcessLockBase#visitLocks
[
https://issues.apache.org/jira/browse/HBASE-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-8582:
--
Attachment: 8582-v1.txt
Possible NullPointerException in ZKInterProcessLockBase#visitLocks
--
Key: HBASE-8582
URL: https://issues.apache.org/jira/browse/HBASE-8582
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Attachments: 8582-v1.txt
Running test suite on hadoop 2.0 I saw the following test failure:
{code}
testErrorReporter(org.apache.hadoop.hbase.util.TestHBaseFsck) Time elapsed:
0.003 sec ERROR!
java.lang.NullPointerException
at
org.apache.hadoop.hbase.zookeeper.lock.ZKInterProcessLockBase.visitLocks(ZKInterProcessLockBase.java:426)
at
org.apache.hadoop.hbase.master.TableLockManager$ZKTableLockManager.visitAllLocks(TableLockManager.java:386)
at
org.apache.hadoop.hbase.util.hbck.TableLockChecker.checkTableLocks(TableLockChecker.java:76)
at
org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixTableLocks(HBaseFsck.java:2480)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:460)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:65)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:41)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:36)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testErrorReporter(TestHBaseFsck.java:1868)
{code}
Here is related code:
{code}
public void visitLocks(MetadataHandler handler) throws IOException {
ListString children;
try {
children = ZKUtil.listChildrenNoWatch(zkWatcher, parentLockNode);
{code}
Looks like children was null.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
