[jira] [Updated] (HBASE-19990) Create remote wal directory when transitting to state S
[ https://issues.apache.org/jira/browse/HBASE-19990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19990: -- Attachment: HBASE-19990-HBASE-19064-v1.patch > Create remote wal directory when transitting to state S > --- > > Key: HBASE-19990 > URL: https://issues.apache.org/jira/browse/HBASE-19990 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-19990-HBASE-19064-v1.patch, > HBASE-19990-HBASE-19064.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-19999) Remove the SYNC_REPLICATION_ENABLED flag
Duo Zhang created HBASE-1: - Summary: Remove the SYNC_REPLICATION_ENABLED flag Key: HBASE-1 URL: https://issues.apache.org/jira/browse/HBASE-1 Project: HBase Issue Type: Sub-task Reporter: Duo Zhang It is a bit strange since we can not guard all the sync replication related code with it. We'd better change its name and only use it within the WAL construction. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19990) Create remote wal directory when transitting to state S
[ https://issues.apache.org/jira/browse/HBASE-19990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363614#comment-16363614 ] Duo Zhang commented on HBASE-19990: --- Filed HBASE-1 for the flag. > Create remote wal directory when transitting to state S > --- > > Key: HBASE-19990 > URL: https://issues.apache.org/jira/browse/HBASE-19990 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-19990-HBASE-19064-v1.patch, > HBASE-19990-HBASE-19064.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19965) Fix flaky TestAsyncRegionAdminApi
[ https://issues.apache.org/jira/browse/HBASE-19965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363655#comment-16363655 ] Hudson commented on HBASE-19965: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4582 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4582/]) HBASE-19965 Fix flaky TestAsyncRegionAdminApi; ADDENDUM do same to (stack: rev 6e35f5eab9799e5d1450f97dec9335800dd39c03) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncTableAdminApi.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncTableAdminApi3.java > Fix flaky TestAsyncRegionAdminApi > - > > Key: HBASE-19965 > URL: https://issues.apache.org/jira/browse/HBASE-19965 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19965.branch-2.001.patch > > > See > [https://builds.apache.org/job/HBase%20Nightly/job/branch-2/284/testReport/junit/org.apache.hadoop.hbase.client/TestAsyncRegionAdminApi/testMergeRegions_0_/] > > java.lang.AssertionError: expected:<2> but was:<3> at > org.apache.hadoop.hbase.client.TestAsyncRegionAdminApi.testMergeRegions(TestAsyncRegionAdminApi.java:359) > > Merge regions not work. The table still have 3 regions after the > MergeRegionsProcedure finished. > The master start balance region 9e2773ba1efba79a2defa276e9a26ed4. But because > the MergeRegionsProcedure pid=138 start work first, so the balance need wait > for the lock. But after merge regions finished, the MoveRegionProcedure > pid=139 start work and assign 9e2773ba1efba79a2defa276e9a26ed4 to a new > region server. This is not right. The MoveRegionProcedure should skip to > assign a region which was marked as offline. Or we should clear the merged > regions' procedure when MergeRegionsProcedure finished. > > Logs: > 2018-02-08 16:24:44,608 INFO [master/cd4730e3eae2:0.Chore.1] > master.HMaster(1454): balance > hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., > source=cd4730e3eae2,39077,1518106776411, > destination=cd4730e3eae2,40578,1518106776318 > 2018-02-08 16:24:44,608 DEBUG > [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37885] > procedure2.ProcedureExecutor(868): Stored pid=138, > state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE; MergeTableRegionsProcedure > table=testMergeRegions, regions=[9e2773ba1efba79a2defa276e9a26ed4, > 8f8fd5cd032313e1aadb83e31e1b7479], forcibly=false > .. > 2018-02-08 16:24:50,111 INFO [PEWorker-13] > procedure2.ProcedureExecutor(1249): Finished pid=138, state=SUCCESS; > MergeTableRegionsProcedure table=testMergeRegions, > regions=[9e2773ba1efba79a2defa276e9a26ed4, 8f8fd5cd032313e1aadb83e31e1b7479], > forcibly=false in 5.5710sec > 2018-02-08 16:24:50,113 INFO [PEWorker-13] > procedure.MasterProcedureScheduler(813): pid=139, > state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure > hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., > source=cd4730e3eae2,39077,1518106776411, > destination=cd4730e3eae2,40578,1518106776318 testMergeRegions > testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19863) java.lang.IllegalStateException: isDelete failed when SingleColumnValueFilter is used
[
https://issues.apache.org/jira/browse/HBASE-19863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363671#comment-16363671
]
Hadoop QA commented on HBASE-19863:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
37s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
27s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
10s{color} | {color:red} hbase-server: The patch generated 7 new + 327
unchanged - 0 fixed = 334 total (was 327) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
15s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
15m 23s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}106m
3s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}138m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:9f2f2db |
| JIRA Issue | HBASE-19863 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910523/HBASE-19863-branch-2.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 0561738b1214 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | branch-2 / 02bba3b788 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/11518/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/11518/testReport/ |
| Max. process+thread count | 4823 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/11518/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apach
[jira] [Commented] (HBASE-19863) java.lang.IllegalStateException: isDelete failed when SingleColumnValueFilter is used
[
https://issues.apache.org/jira/browse/HBASE-19863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363698#comment-16363698
]
Ted Yu commented on HBASE-19863:
Lgtm
Sergey:
Can you check findbugs warnings ?
> java.lang.IllegalStateException: isDelete failed when SingleColumnValueFilter
> is used
> -
>
> Key: HBASE-19863
> URL: https://issues.apache.org/jira/browse/HBASE-19863
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 1.4.1
>Reporter: Sergey Soldatov
>Assignee: Sergey Soldatov
>Priority: Major
> Attachments: HBASE-19863-branch-2.patch, HBASE-19863-branch1.patch,
> HBASE-19863-test.patch
>
>
> Under some circumstances scan with SingleColumnValueFilter may fail with an
> exception
> {noformat}
> java.lang.IllegalStateException: isDelete failed: deleteBuffer=C3,
> qualifier=C2, timestamp=1516433595543, comparison result: 1
> at
> org.apache.hadoop.hbase.regionserver.ScanDeleteTracker.isDeleted(ScanDeleteTracker.java:149)
> at
> org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.match(ScanQueryMatcher.java:386)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:545)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:147)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5876)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6027)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5814)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2552)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32385)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
> {noformat}
> Conditions:
> table T with a single column family 0 that uses ROWCOL bloom filter
> (important) and column qualifiers C1,C2,C3,C4,C5.
> When we fill the table for every row we put deleted cell for C3.
> The table has a single region with two HStore:
> A: start row: 0, stop row: 99
> B: start row: 10 stop row: 99
> B has newer versions of rows 10-99. Store files have several blocks each
> (important).
> Store A is the result of major compaction, so it doesn't have any deleted
> cells (important).
> So, we are running a scan like:
> {noformat}
> scan 'T', { COLUMNS => ['0:C3','0:C5'], FILTER => "SingleColumnValueFilter
> ('0','C5',=,'binary:whatever')"}
> {noformat}
> How the scan performs:
> First, we iterate A for rows 0 and 1 without any problems.
> Next, we start to iterate A for row 10, so read the first cell and set hfs
> scanner to A :
> 10:0/C1/0/Put/x but found that we have a newer version of the cell in B :
> 10:0/C1/1/Put/x,
> so we make B as our current store scanner. Since we are looking for
> particular columns
> C3 and C5, we perform the optimization StoreScanner.seekOrSkipToNextColumn
> which
> would run reseek for all store scanners.
> For store A the following magic would happen in requestSeek:
> 1. bloom filter check passesGeneralBloomFilter would set haveToSeek to
> false because row 10 doesn't have C3 qualifier in store A.
> 2. Since we don't have to seek we just create a fake row
> 10:0/C3/OLDEST_TIMESTAMP/Maximum, an optimization that is quite important for
> us and it commented with :
> {noformat}
> // Multi-column Bloom filter optimization.
> // Create a fake key/value, so that this scanner only bubbles up to the
> top
> // of the KeyValueHeap in StoreScanner after we scanned this row/column in
> // all other store files. The query matcher will then just skip this fake
> // key/value and the store scanner will progress to the next column. This
> // is obviously not a "real real" seek, but unlike the fake KV earlier in
> // this method, we want this to be propagated to ScanQueryMatcher.
> {noformat}
>
> For store B we would set it to fake 10:0/C3/createFirstOnRowColTS()/Maximum
> to skip C3 entirely.
> After that we start searching for qualifier C5 using seekOrSkipToNextColumn
> which run first trySkipToNextColumn:
> {noformat}
> protected boolean trySkipToNextColumn(Cell cell) throws IOException {
> Cell nextCell = null;
> do {
> Cell nextIndexedKey = getNextIndexedKey();
> if (nextIndexedKey != null && nextIndexedKey
[jira] [Updated] (HBASE-19863) java.lang.IllegalStateException: isDelete failed when SingleColumnValueFilter is used
[
https://issues.apache.org/jira/browse/HBASE-19863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Soldatov updated HBASE-19863:
Attachment: HBASE-19863.v2-branch-2.patch
> java.lang.IllegalStateException: isDelete failed when SingleColumnValueFilter
> is used
> -
>
> Key: HBASE-19863
> URL: https://issues.apache.org/jira/browse/HBASE-19863
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 1.4.1
>Reporter: Sergey Soldatov
>Assignee: Sergey Soldatov
>Priority: Major
> Attachments: HBASE-19863-branch-2.patch, HBASE-19863-branch1.patch,
> HBASE-19863-test.patch, HBASE-19863.v2-branch-2.patch
>
>
> Under some circumstances scan with SingleColumnValueFilter may fail with an
> exception
> {noformat}
> java.lang.IllegalStateException: isDelete failed: deleteBuffer=C3,
> qualifier=C2, timestamp=1516433595543, comparison result: 1
> at
> org.apache.hadoop.hbase.regionserver.ScanDeleteTracker.isDeleted(ScanDeleteTracker.java:149)
> at
> org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.match(ScanQueryMatcher.java:386)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:545)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:147)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5876)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6027)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5814)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2552)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32385)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
> {noformat}
> Conditions:
> table T with a single column family 0 that uses ROWCOL bloom filter
> (important) and column qualifiers C1,C2,C3,C4,C5.
> When we fill the table for every row we put deleted cell for C3.
> The table has a single region with two HStore:
> A: start row: 0, stop row: 99
> B: start row: 10 stop row: 99
> B has newer versions of rows 10-99. Store files have several blocks each
> (important).
> Store A is the result of major compaction, so it doesn't have any deleted
> cells (important).
> So, we are running a scan like:
> {noformat}
> scan 'T', { COLUMNS => ['0:C3','0:C5'], FILTER => "SingleColumnValueFilter
> ('0','C5',=,'binary:whatever')"}
> {noformat}
> How the scan performs:
> First, we iterate A for rows 0 and 1 without any problems.
> Next, we start to iterate A for row 10, so read the first cell and set hfs
> scanner to A :
> 10:0/C1/0/Put/x but found that we have a newer version of the cell in B :
> 10:0/C1/1/Put/x,
> so we make B as our current store scanner. Since we are looking for
> particular columns
> C3 and C5, we perform the optimization StoreScanner.seekOrSkipToNextColumn
> which
> would run reseek for all store scanners.
> For store A the following magic would happen in requestSeek:
> 1. bloom filter check passesGeneralBloomFilter would set haveToSeek to
> false because row 10 doesn't have C3 qualifier in store A.
> 2. Since we don't have to seek we just create a fake row
> 10:0/C3/OLDEST_TIMESTAMP/Maximum, an optimization that is quite important for
> us and it commented with :
> {noformat}
> // Multi-column Bloom filter optimization.
> // Create a fake key/value, so that this scanner only bubbles up to the
> top
> // of the KeyValueHeap in StoreScanner after we scanned this row/column in
> // all other store files. The query matcher will then just skip this fake
> // key/value and the store scanner will progress to the next column. This
> // is obviously not a "real real" seek, but unlike the fake KV earlier in
> // this method, we want this to be propagated to ScanQueryMatcher.
> {noformat}
>
> For store B we would set it to fake 10:0/C3/createFirstOnRowColTS()/Maximum
> to skip C3 entirely.
> After that we start searching for qualifier C5 using seekOrSkipToNextColumn
> which run first trySkipToNextColumn:
> {noformat}
> protected boolean trySkipToNextColumn(Cell cell) throws IOException {
> Cell nextCell = null;
> do {
> Cell nextIndexedKey = getNextIndexedKey();
> if (nextIndexedKey != null && nextIndexedKey !=
> KeyVa
[jira] [Commented] (HBASE-19863) java.lang.IllegalStateException: isDelete failed when SingleColumnValueFilter is used
[
https://issues.apache.org/jira/browse/HBASE-19863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363718#comment-16363718
]
Sergey Soldatov commented on HBASE-19863:
-
[~yuzhih...@gmail.com] you mean checkstyle? Addressed that in v2 patch.
> java.lang.IllegalStateException: isDelete failed when SingleColumnValueFilter
> is used
> -
>
> Key: HBASE-19863
> URL: https://issues.apache.org/jira/browse/HBASE-19863
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 1.4.1
>Reporter: Sergey Soldatov
>Assignee: Sergey Soldatov
>Priority: Major
> Attachments: HBASE-19863-branch-2.patch, HBASE-19863-branch1.patch,
> HBASE-19863-test.patch, HBASE-19863.v2-branch-2.patch
>
>
> Under some circumstances scan with SingleColumnValueFilter may fail with an
> exception
> {noformat}
> java.lang.IllegalStateException: isDelete failed: deleteBuffer=C3,
> qualifier=C2, timestamp=1516433595543, comparison result: 1
> at
> org.apache.hadoop.hbase.regionserver.ScanDeleteTracker.isDeleted(ScanDeleteTracker.java:149)
> at
> org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.match(ScanQueryMatcher.java:386)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:545)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:147)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5876)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6027)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5814)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2552)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32385)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
> {noformat}
> Conditions:
> table T with a single column family 0 that uses ROWCOL bloom filter
> (important) and column qualifiers C1,C2,C3,C4,C5.
> When we fill the table for every row we put deleted cell for C3.
> The table has a single region with two HStore:
> A: start row: 0, stop row: 99
> B: start row: 10 stop row: 99
> B has newer versions of rows 10-99. Store files have several blocks each
> (important).
> Store A is the result of major compaction, so it doesn't have any deleted
> cells (important).
> So, we are running a scan like:
> {noformat}
> scan 'T', { COLUMNS => ['0:C3','0:C5'], FILTER => "SingleColumnValueFilter
> ('0','C5',=,'binary:whatever')"}
> {noformat}
> How the scan performs:
> First, we iterate A for rows 0 and 1 without any problems.
> Next, we start to iterate A for row 10, so read the first cell and set hfs
> scanner to A :
> 10:0/C1/0/Put/x but found that we have a newer version of the cell in B :
> 10:0/C1/1/Put/x,
> so we make B as our current store scanner. Since we are looking for
> particular columns
> C3 and C5, we perform the optimization StoreScanner.seekOrSkipToNextColumn
> which
> would run reseek for all store scanners.
> For store A the following magic would happen in requestSeek:
> 1. bloom filter check passesGeneralBloomFilter would set haveToSeek to
> false because row 10 doesn't have C3 qualifier in store A.
> 2. Since we don't have to seek we just create a fake row
> 10:0/C3/OLDEST_TIMESTAMP/Maximum, an optimization that is quite important for
> us and it commented with :
> {noformat}
> // Multi-column Bloom filter optimization.
> // Create a fake key/value, so that this scanner only bubbles up to the
> top
> // of the KeyValueHeap in StoreScanner after we scanned this row/column in
> // all other store files. The query matcher will then just skip this fake
> // key/value and the store scanner will progress to the next column. This
> // is obviously not a "real real" seek, but unlike the fake KV earlier in
> // this method, we want this to be propagated to ScanQueryMatcher.
> {noformat}
>
> For store B we would set it to fake 10:0/C3/createFirstOnRowColTS()/Maximum
> to skip C3 entirely.
> After that we start searching for qualifier C5 using seekOrSkipToNextColumn
> which run first trySkipToNextColumn:
> {noformat}
> protected boolean trySkipToNextColumn(Cell cell) throws IOException {
> Cell nextCell = null;
> do {
> Cell nextIndexedKey = g
[jira] [Commented] (HBASE-19990) Create remote wal directory when transitting to state S
[
https://issues.apache.org/jira/browse/HBASE-19990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363778#comment-16363778
]
Hadoop QA commented on HBASE-19990:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HBASE-19064 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
6s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
40s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
49s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} HBASE-19064 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch hbase-replication passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} The patch hbase-procedure passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
14s{color} | {color:green} hbase-server: The patch generated 0 new + 12
unchanged - 7 fixed = 12 total (was 19) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
33s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
21m 53s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
23s{color} | {color:green} hbase-replication in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
59s{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}108m
45s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
57s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}160m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19990 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910531/HBASE-19990-HBASE-19064-v1.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoo
[jira] [Updated] (HBASE-19980) NullPointerException when restoring a snapshot after splitting a region
[
https://issues.apache.org/jira/browse/HBASE-19980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Toshihiro Suzuki updated HBASE-19980:
-
Attachment: HBASE-19980.master.001.patch
> NullPointerException when restoring a snapshot after splitting a region
> ---
>
> Key: HBASE-19980
> URL: https://issues.apache.org/jira/browse/HBASE-19980
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Toshihiro Suzuki
>Priority: Major
> Attachments: HBASE-19980-UT.patch, HBASE-19980.master.001.patch
>
>
> Investigating HBASE-19893, I'm facing the following NullPointerException when
> restoring a snapshot after splitting a region:
> {code}
> 2018-02-13 01:59:56,033 ERROR [PEWorker-1]
> procedure.RestoreSnapshotProcedure(403): restore snapshot={
> ss=snaptb1-1518454780480
> table=testRestoreSnapshotAfterSplittingRegions-1518454780480 type=FLUSH }
> failed in on-disk restore. Try re-running the restore command.
> java.io.IOException
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils.editRegions(ModifyRegionUtils.java:222)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:424)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:227)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:177)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.restoreSnapshot(RestoreSnapshotProcedure.java:395)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.executeFromState(RestoreSnapshotProcedure.java:143)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.executeFromState(RestoreSnapshotProcedure.java:58)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:180)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1455)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1224)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1734)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreReferenceFile(RestoreSnapshotHelper.java:751)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreStoreFile(RestoreSnapshotHelper.java:675)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreRegion(RestoreSnapshotHelper.java:521)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreRegion(RestoreSnapshotHelper.java:464)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.access$300(RestoreSnapshotHelper.java:116)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper$2.editRegion(RestoreSnapshotHelper.java:427)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$2.call(ModifyRegionUtils.java:209)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$2.call(ModifyRegionUtils.java:206)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19980) NullPointerException when restoring a snapshot after splitting a region
[
https://issues.apache.org/jira/browse/HBASE-19980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Toshihiro Suzuki updated HBASE-19980:
-
Assignee: Toshihiro Suzuki
Status: Patch Available (was: Open)
> NullPointerException when restoring a snapshot after splitting a region
> ---
>
> Key: HBASE-19980
> URL: https://issues.apache.org/jira/browse/HBASE-19980
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Major
> Attachments: HBASE-19980-UT.patch, HBASE-19980.master.001.patch
>
>
> Investigating HBASE-19893, I'm facing the following NullPointerException when
> restoring a snapshot after splitting a region:
> {code}
> 2018-02-13 01:59:56,033 ERROR [PEWorker-1]
> procedure.RestoreSnapshotProcedure(403): restore snapshot={
> ss=snaptb1-1518454780480
> table=testRestoreSnapshotAfterSplittingRegions-1518454780480 type=FLUSH }
> failed in on-disk restore. Try re-running the restore command.
> java.io.IOException
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils.editRegions(ModifyRegionUtils.java:222)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:424)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:227)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:177)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.restoreSnapshot(RestoreSnapshotProcedure.java:395)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.executeFromState(RestoreSnapshotProcedure.java:143)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.executeFromState(RestoreSnapshotProcedure.java:58)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:180)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1455)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1224)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1734)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreReferenceFile(RestoreSnapshotHelper.java:751)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreStoreFile(RestoreSnapshotHelper.java:675)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreRegion(RestoreSnapshotHelper.java:521)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreRegion(RestoreSnapshotHelper.java:464)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.access$300(RestoreSnapshotHelper.java:116)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper$2.editRegion(RestoreSnapshotHelper.java:427)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$2.call(ModifyRegionUtils.java:209)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$2.call(ModifyRegionUtils.java:206)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19980) NullPointerException when restoring a snapshot after splitting a region
[
https://issues.apache.org/jira/browse/HBASE-19980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363810#comment-16363810
]
Toshihiro Suzuki commented on HBASE-19980:
--
I just attached v1 patch.
> NullPointerException when restoring a snapshot after splitting a region
> ---
>
> Key: HBASE-19980
> URL: https://issues.apache.org/jira/browse/HBASE-19980
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Major
> Attachments: HBASE-19980-UT.patch, HBASE-19980.master.001.patch
>
>
> Investigating HBASE-19893, I'm facing the following NullPointerException when
> restoring a snapshot after splitting a region:
> {code}
> 2018-02-13 01:59:56,033 ERROR [PEWorker-1]
> procedure.RestoreSnapshotProcedure(403): restore snapshot={
> ss=snaptb1-1518454780480
> table=testRestoreSnapshotAfterSplittingRegions-1518454780480 type=FLUSH }
> failed in on-disk restore. Try re-running the restore command.
> java.io.IOException
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils.editRegions(ModifyRegionUtils.java:222)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:424)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:227)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:177)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.restoreSnapshot(RestoreSnapshotProcedure.java:395)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.executeFromState(RestoreSnapshotProcedure.java:143)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.executeFromState(RestoreSnapshotProcedure.java:58)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:180)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1455)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1224)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1734)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreReferenceFile(RestoreSnapshotHelper.java:751)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreStoreFile(RestoreSnapshotHelper.java:675)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreRegion(RestoreSnapshotHelper.java:521)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreRegion(RestoreSnapshotHelper.java:464)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.access$300(RestoreSnapshotHelper.java:116)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper$2.editRegion(RestoreSnapshotHelper.java:427)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$2.call(ModifyRegionUtils.java:209)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$2.call(ModifyRegionUtils.java:206)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-18294: -- Release Note: A region is flushed if its memory component exceeds the region flush threshold. A flush policy decides which stores to flush by comparing the size of the store to a column-family-flush threshold. If the overall size of all memstores in the machine exceeds the bounds defined by the administrator (denoted global pressure) a region is selected and flushed. HBASE-18294 changes flush decisions to be based on heap-occupancy and not data (key-value) size, consistently across levels. This rolls back some of the changes by HBASE-16747. Specifically, (1) RSs, Regions and stores track their overall on-heap and off-heap occupancy, (2) A region is flushed when its on-heap+off-heap size exceeds the region flush threshold specified in hbase.hregion.memstore.flush.size, (3) The store to be flushed is chosen based on its on-heap+off-heap size (4) At the RS level, a flush is triggered when the overall on-heap exceeds the on-heap limit, or when the overall off-heap size exceeds the off-heap limit (low/high water marks). Note that when the region flush size is set to XXmb a region flush may be triggered even before writing keys and values of size XX because the total heap occupancy of the region which includes additional metadata exceeded the threshold. was: A region is flushed if its memory component exceeds the region flush threshold. A flush policy decides which stores to flush by comparing the size of the store to a column-family-flush threshold. If the overall size of all memstores in the machine exceeds the bounds defined by the administrator (denoted global pressure) a region is selected and flush. HBASE-18294 changes flush decisions to be based on heap-occupancy and not data (key-value) size, consistently across levels. This rolls back some of the changes by HBASE-16747. Specifically, (1) RSs, Regions and stores track their overall on-heap and off-heap occupancy, (2) A region is flushed when its on-heap+off-heap size exceeds the region flush threshold, (3) The store to be flushed is chosen based on its on-heap+off-heap size (4) At the RS level, a flush is triggered when the overall on-heap exceeds the on-heap limit, or when the overall off-heap size exceeds the off-heap limit (low/high water marks). Note that when the region flush size is set to XXmb a region flush may be triggered even before writing keys and values of size XX because the total heap occupancy of the region which includes additional metadata exceeded the threshold. > Reduce global heap pressure: flush based on heap occupancy > -- > > Key: HBASE-18294 > URL: https://issues.apache.org/jira/browse/HBASE-18294 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.01.patch, HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.01.patch, HBASE-18294.01.patch, HBASE-18294.02.patch, > HBASE-18294.03.patch, HBASE-18294.04.patch, HBASE-18294.05.patch, > HBASE-18294.06.patch, HBASE-18294.07.patch, HBASE-18294.07.patch, > HBASE-18294.08.patch, HBASE-18294.09.patch, HBASE-18294.10.patch, > HBASE-18294.11.patch, HBASE-18294.11.patch, HBASE-18294.12.patch, > HBASE-18294.13.patch, HBASE-18294.15.patch, HBASE-18294.16.patch, > HBASE-18294.master.01.patch, HBASE-18294.master.01.patch > > > A region is flushed if its memory component exceed a threshold (default size > is 128MB). > A flush policy decides whether to flush a store by comparing the size of the > store to another threshold (that can be configured with > hbase.hregion.percolumnfamilyflush.size.lower.bound). > Currently the implementation (in both cases) compares the data size > (key-value only) to the threshold where it should compare the heap size > (which includes index size, and metadata). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[
https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363845#comment-16363845
]
Eshcar Hillel commented on HBASE-18294:
---
Stack the barriers for the RS (global) level are more complicated than just a
single configuration property. It is some computation over parameters defined by
{color:#008000}hbase.regionserver.global.memstore.size{color}
{color:#008000}hbase.regionserver.global.memstore.size.lower.limit{color}
{color:#008000}hbase.regionserver.offheap.global.memstore.size{color}
DEFAULT_MEMSTORE_SIZE
DEFAULT_MEMSTORE_SIZE_LOWER_LIMIT
and some settings from hbase-env.sh
I think it deserves a separate documentation since this Jira did not change the
barriers computation but what we compare against them.
Regarding the comments on the code I changed the implementation of
getMemStoreSize to return region.getMemStoreDataSize() since that what was
eventually returned in the previous implementation. I then went and changed the
call from getMemStoreSize to getMemStoreOnHeapSize or getMemStoreOffHeapSize
based on what I though appropriate for the case.
If all is ok, I will commit later today.
> Reduce global heap pressure: flush based on heap occupancy
> --
>
> Key: HBASE-18294
> URL: https://issues.apache.org/jira/browse/HBASE-18294
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-18294.01.patch, HBASE-18294.01.patch,
> HBASE-18294.01.patch, HBASE-18294.01.patch, HBASE-18294.01.patch,
> HBASE-18294.01.patch, HBASE-18294.01.patch, HBASE-18294.02.patch,
> HBASE-18294.03.patch, HBASE-18294.04.patch, HBASE-18294.05.patch,
> HBASE-18294.06.patch, HBASE-18294.07.patch, HBASE-18294.07.patch,
> HBASE-18294.08.patch, HBASE-18294.09.patch, HBASE-18294.10.patch,
> HBASE-18294.11.patch, HBASE-18294.11.patch, HBASE-18294.12.patch,
> HBASE-18294.13.patch, HBASE-18294.15.patch, HBASE-18294.16.patch,
> HBASE-18294.master.01.patch, HBASE-18294.master.01.patch
>
>
> A region is flushed if its memory component exceed a threshold (default size
> is 128MB).
> A flush policy decides whether to flush a store by comparing the size of the
> store to another threshold (that can be configured with
> hbase.hregion.percolumnfamilyflush.size.lower.bound).
> Currently the implementation (in both cases) compares the data size
> (key-value only) to the threshold where it should compare the heap size
> (which includes index size, and metadata).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19863) java.lang.IllegalStateException: isDelete failed when SingleColumnValueFilter is used
[
https://issues.apache.org/jira/browse/HBASE-19863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363871#comment-16363871
]
Hadoop QA commented on HBASE-19863:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
53s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
10s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
19s{color} | {color:red} hbase-server: The patch generated 2 new + 324
unchanged - 3 fixed = 326 total (was 327) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
29s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
17m 4s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}113m 1s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}150m 41s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestAsyncTable |
| | hadoop.hbase.master.TestDLSFSHLog |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:9f2f2db |
| JIRA Issue | HBASE-19863 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910558/HBASE-19863.v2-branch-2.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux d61b9e2a58cf 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | branch-2 / 02bba3b788 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/11521/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/11521/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/11521/testReport/ |
|
[jira] [Updated] (HBASE-19985) Redundant instanceof check in ProtobufUtil#getServiceException
[
https://issues.apache.org/jira/browse/HBASE-19985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-19985:
---
Labels: (was: be)
> Redundant instanceof check in ProtobufUtil#getServiceException
> --
>
> Key: HBASE-19985
> URL: https://issues.apache.org/jira/browse/HBASE-19985
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.1
>Reporter: Ted Yu
>Priority: Trivial
>
> {code}
> public static IOException getServiceException(ServiceException e) {
> Throwable t = e;
> if (e instanceof ServiceException) {
> t = e.getCause();
> {code}
> The instanceof check would always return true.
> This was reported by
> https://builds.apache.org/job/PreCommit-HBASE-Build/11495/artifact/patchprocess/branch-findbugs-hbase-client-warnings.html#Warnings_STYLE
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19985) Redundant instanceof check in ProtobufUtil#getServiceException
[
https://issues.apache.org/jira/browse/HBASE-19985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-19985:
---
Labels: be (was: )
> Redundant instanceof check in ProtobufUtil#getServiceException
> --
>
> Key: HBASE-19985
> URL: https://issues.apache.org/jira/browse/HBASE-19985
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.1
>Reporter: Ted Yu
>Priority: Trivial
>
> {code}
> public static IOException getServiceException(ServiceException e) {
> Throwable t = e;
> if (e instanceof ServiceException) {
> t = e.getCause();
> {code}
> The instanceof check would always return true.
> This was reported by
> https://builds.apache.org/job/PreCommit-HBASE-Build/11495/artifact/patchprocess/branch-findbugs-hbase-client-warnings.html#Warnings_STYLE
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19985) Redundant instanceof check in ProtobufUtil#getServiceException
[
https://issues.apache.org/jira/browse/HBASE-19985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363966#comment-16363966
]
Chia-Ping Tsai commented on HBASE-19985:
The code in master is shown below.
{code:java}
public static IOException
getServiceException(org.apache.hbase.thirdparty.com.google.protobuf.ServiceException
e) {
Throwable t = e.getCause();
if (ExceptionUtil.isInterrupt(t)) {
return ExceptionUtil.asInterrupt(t);
}
return t instanceof IOException ? (IOException) t : new HBaseIOException(t);
}{code}
ping [~lhofhansl]
> Redundant instanceof check in ProtobufUtil#getServiceException
> --
>
> Key: HBASE-19985
> URL: https://issues.apache.org/jira/browse/HBASE-19985
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.1
>Reporter: Ted Yu
>Priority: Trivial
> Fix For: 1.5.0, 1.4.2
>
>
> {code}
> public static IOException getServiceException(ServiceException e) {
> Throwable t = e;
> if (e instanceof ServiceException) {
> t = e.getCause();
> {code}
> The instanceof check would always return true.
> This was reported by
> https://builds.apache.org/job/PreCommit-HBASE-Build/11495/artifact/patchprocess/branch-findbugs-hbase-client-warnings.html#Warnings_STYLE
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19985) Redundant instanceof check in ProtobufUtil#getServiceException
[
https://issues.apache.org/jira/browse/HBASE-19985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-19985:
---
Fix Version/s: 1.4.2
1.5.0
> Redundant instanceof check in ProtobufUtil#getServiceException
> --
>
> Key: HBASE-19985
> URL: https://issues.apache.org/jira/browse/HBASE-19985
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.1
>Reporter: Ted Yu
>Priority: Trivial
> Fix For: 1.5.0, 1.4.2
>
>
> {code}
> public static IOException getServiceException(ServiceException e) {
> Throwable t = e;
> if (e instanceof ServiceException) {
> t = e.getCause();
> {code}
> The instanceof check would always return true.
> This was reported by
> https://builds.apache.org/job/PreCommit-HBASE-Build/11495/artifact/patchprocess/branch-findbugs-hbase-client-warnings.html#Warnings_STYLE
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19894) Add seePastDeleteMarkers() option in scan
[ https://issues.apache.org/jira/browse/HBASE-19894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364004#comment-16364004 ] Chia-Ping Tsai commented on HBASE-19894: What is the difference between seePastDeleteMarkers option and raw option? retrieve only the Delete cells? > Add seePastDeleteMarkers() option in scan > - > > Key: HBASE-19894 > URL: https://issues.apache.org/jira/browse/HBASE-19894 > Project: HBase > Issue Type: Sub-task > Components: scan >Reporter: Ankit Singhal >Assignee: Ankit Singhal >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19724) Fix Checkstyle errors in hbase-hadoop2-compat
[ https://issues.apache.org/jira/browse/HBASE-19724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364014#comment-16364014 ] Chia-Ping Tsai commented on HBASE-19724: +1 if the comment is addressed > Fix Checkstyle errors in hbase-hadoop2-compat > - > > Key: HBASE-19724 > URL: https://issues.apache.org/jira/browse/HBASE-19724 > Project: HBase > Issue Type: Sub-task >Reporter: Jan Hentschel >Assignee: Jan Hentschel >Priority: Minor > Attachments: HBASE-19724.master.001.patch > > > Fix the remaining Checkstyle errors in the *hbase-hadoop2-compat* module and > enable Checkstyle to fail on violations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19980) NullPointerException when restoring a snapshot after splitting a region
[
https://issues.apache.org/jira/browse/HBASE-19980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364095#comment-16364095
]
Hadoop QA commented on HBASE-19980:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 9m
26s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
52s{color} | {color:green} hbase-server: The patch generated 0 new + 19
unchanged - 1 fixed = 19 total (was 20) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 7m
15s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 46s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}121m 47s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}182m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaSameHosts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19980 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910565/HBASE-19980.master.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 3c7ee35663a0 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 2f1b3eab67 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/11522/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/11522/testReport/ |
| Max. process+thread count | 5227 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://build
[jira] [Updated] (HBASE-19980) NullPointerException when restoring a snapshot after splitting a region
[
https://issues.apache.org/jira/browse/HBASE-19980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Toshihiro Suzuki updated HBASE-19980:
-
Attachment: HBASE-19980.master.002.patch
> NullPointerException when restoring a snapshot after splitting a region
> ---
>
> Key: HBASE-19980
> URL: https://issues.apache.org/jira/browse/HBASE-19980
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Major
> Attachments: HBASE-19980-UT.patch, HBASE-19980.master.001.patch,
> HBASE-19980.master.002.patch
>
>
> Investigating HBASE-19893, I'm facing the following NullPointerException when
> restoring a snapshot after splitting a region:
> {code}
> 2018-02-13 01:59:56,033 ERROR [PEWorker-1]
> procedure.RestoreSnapshotProcedure(403): restore snapshot={
> ss=snaptb1-1518454780480
> table=testRestoreSnapshotAfterSplittingRegions-1518454780480 type=FLUSH }
> failed in on-disk restore. Try re-running the restore command.
> java.io.IOException
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils.editRegions(ModifyRegionUtils.java:222)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:424)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:227)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:177)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.restoreSnapshot(RestoreSnapshotProcedure.java:395)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.executeFromState(RestoreSnapshotProcedure.java:143)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.executeFromState(RestoreSnapshotProcedure.java:58)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:180)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1455)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1224)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1734)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreReferenceFile(RestoreSnapshotHelper.java:751)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreStoreFile(RestoreSnapshotHelper.java:675)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreRegion(RestoreSnapshotHelper.java:521)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreRegion(RestoreSnapshotHelper.java:464)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.access$300(RestoreSnapshotHelper.java:116)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper$2.editRegion(RestoreSnapshotHelper.java:427)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$2.call(ModifyRegionUtils.java:209)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$2.call(ModifyRegionUtils.java:206)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19980) NullPointerException when restoring a snapshot after splitting a region
[
https://issues.apache.org/jira/browse/HBASE-19980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364113#comment-16364113
]
Toshihiro Suzuki commented on HBASE-19980:
--
I attached v2 patch with small change.
> NullPointerException when restoring a snapshot after splitting a region
> ---
>
> Key: HBASE-19980
> URL: https://issues.apache.org/jira/browse/HBASE-19980
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Major
> Attachments: HBASE-19980-UT.patch, HBASE-19980.master.001.patch,
> HBASE-19980.master.002.patch
>
>
> Investigating HBASE-19893, I'm facing the following NullPointerException when
> restoring a snapshot after splitting a region:
> {code}
> 2018-02-13 01:59:56,033 ERROR [PEWorker-1]
> procedure.RestoreSnapshotProcedure(403): restore snapshot={
> ss=snaptb1-1518454780480
> table=testRestoreSnapshotAfterSplittingRegions-1518454780480 type=FLUSH }
> failed in on-disk restore. Try re-running the restore command.
> java.io.IOException
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils.editRegions(ModifyRegionUtils.java:222)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:424)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:227)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:177)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.restoreSnapshot(RestoreSnapshotProcedure.java:395)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.executeFromState(RestoreSnapshotProcedure.java:143)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.executeFromState(RestoreSnapshotProcedure.java:58)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:180)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1455)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1224)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1734)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreReferenceFile(RestoreSnapshotHelper.java:751)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreStoreFile(RestoreSnapshotHelper.java:675)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreRegion(RestoreSnapshotHelper.java:521)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreRegion(RestoreSnapshotHelper.java:464)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.access$300(RestoreSnapshotHelper.java:116)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper$2.editRegion(RestoreSnapshotHelper.java:427)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$2.call(ModifyRegionUtils.java:209)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$2.call(ModifyRegionUtils.java:206)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20000) Remove the quantum logic in FairQueue, always put high priority queue in front
Duo Zhang created HBASE-2: - Summary: Remove the quantum logic in FairQueue, always put high priority queue in front Key: HBASE-2 URL: https://issues.apache.org/jira/browse/HBASE-2 Project: HBase Issue Type: Sub-task Components: proc-v2 Reporter: Duo Zhang Assignee: Duo Zhang Fix For: 2.0.0-beta-2 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19930) fix ImmutableMemStoreLAB#forceCopyOfBigCellInto
[ https://issues.apache.org/jira/browse/HBASE-19930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364195#comment-16364195 ] ramkrishna.s.vasudevan commented on HBASE-19930: +1 to commit. > fix ImmutableMemStoreLAB#forceCopyOfBigCellInto > --- > > Key: HBASE-19930 > URL: https://issues.apache.org/jira/browse/HBASE-19930 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-1 >Reporter: Gali Sheffi >Assignee: Gali Sheffi >Priority: Major > Attachments: HBASE-19930-V01.patch, HBASE-19930-V02.patch, > HBASE-19930-V03.patch, HBASE-19930-V04.patch, HBASE-19930-V05.patch, > HBASE-19930-V06.patch > > > This issue is about fixing ImmutableMemStoreLAB#forceCopyOfBigCellInto. > Following a comment in HBASE-19133 regarding a bug in > ImmutableMemStoreLAB#forceCopyOfBigCellInto (assuming this method is never > called for an ImmutableMemStoreLAB, and just throwing an > IllegalStateException whenever called), the forceCopyOfBigCellInto method now > performs the copy of big cells on the first MSLABImpl in its mslabs > linked-list. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20000) Remove the quantum logic in FairQueue, always put high priority queue in front
[ https://issues.apache.org/jira/browse/HBASE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-2: -- Attachment: HBASE-2.patch > Remove the quantum logic in FairQueue, always put high priority queue in front > -- > > Key: HBASE-2 > URL: https://issues.apache.org/jira/browse/HBASE-2 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19980) NullPointerException when restoring a snapshot after splitting a region
[
https://issues.apache.org/jira/browse/HBASE-19980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364197#comment-16364197
]
Ted Yu commented on HBASE-19980:
lgtm
> NullPointerException when restoring a snapshot after splitting a region
> ---
>
> Key: HBASE-19980
> URL: https://issues.apache.org/jira/browse/HBASE-19980
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Major
> Attachments: HBASE-19980-UT.patch, HBASE-19980.master.001.patch,
> HBASE-19980.master.002.patch
>
>
> Investigating HBASE-19893, I'm facing the following NullPointerException when
> restoring a snapshot after splitting a region:
> {code}
> 2018-02-13 01:59:56,033 ERROR [PEWorker-1]
> procedure.RestoreSnapshotProcedure(403): restore snapshot={
> ss=snaptb1-1518454780480
> table=testRestoreSnapshotAfterSplittingRegions-1518454780480 type=FLUSH }
> failed in on-disk restore. Try re-running the restore command.
> java.io.IOException
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils.editRegions(ModifyRegionUtils.java:222)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:424)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:227)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:177)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.restoreSnapshot(RestoreSnapshotProcedure.java:395)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.executeFromState(RestoreSnapshotProcedure.java:143)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.executeFromState(RestoreSnapshotProcedure.java:58)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:180)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1455)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1224)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1734)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreReferenceFile(RestoreSnapshotHelper.java:751)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreStoreFile(RestoreSnapshotHelper.java:675)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreRegion(RestoreSnapshotHelper.java:521)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreRegion(RestoreSnapshotHelper.java:464)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.access$300(RestoreSnapshotHelper.java:116)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper$2.editRegion(RestoreSnapshotHelper.java:427)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$2.call(ModifyRegionUtils.java:209)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$2.call(ModifyRegionUtils.java:206)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20000) Remove the quantum logic in FairQueue, always put high priority queue in front
[ https://issues.apache.org/jira/browse/HBASE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364202#comment-16364202 ] Duo Zhang commented on HBASE-2: --- Part of the job. First, let's always put meta and system table in front of other tables. And also done some refactoring which moves out lots of internal classes in MasterProcedureScheduler to make it shorter. Also remove the configurable priorities for table, it is now hard coded to be 1~3. After removing of quantum logic then it is useless to make it configurable, and the new implementation use this as an assumption. > Remove the quantum logic in FairQueue, always put high priority queue in front > -- > > Key: HBASE-2 > URL: https://issues.apache.org/jira/browse/HBASE-2 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19863) java.lang.IllegalStateException: isDelete failed when SingleColumnValueFilter is used
[
https://issues.apache.org/jira/browse/HBASE-19863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364201#comment-16364201
]
ramkrishna.s.vasudevan commented on HBASE-19863:
Patch looks good to me. If the test is taking time to run probably you can
avoid the spinning of cluster and just create Hregion and add data to the
region directly. Else +1. Lets wait for other reviews if you want to get it in.
[~chia7712] - you have any comments?
> java.lang.IllegalStateException: isDelete failed when SingleColumnValueFilter
> is used
> -
>
> Key: HBASE-19863
> URL: https://issues.apache.org/jira/browse/HBASE-19863
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 1.4.1
>Reporter: Sergey Soldatov
>Assignee: Sergey Soldatov
>Priority: Major
> Attachments: HBASE-19863-branch-2.patch, HBASE-19863-branch1.patch,
> HBASE-19863-test.patch, HBASE-19863.v2-branch-2.patch
>
>
> Under some circumstances scan with SingleColumnValueFilter may fail with an
> exception
> {noformat}
> java.lang.IllegalStateException: isDelete failed: deleteBuffer=C3,
> qualifier=C2, timestamp=1516433595543, comparison result: 1
> at
> org.apache.hadoop.hbase.regionserver.ScanDeleteTracker.isDeleted(ScanDeleteTracker.java:149)
> at
> org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.match(ScanQueryMatcher.java:386)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:545)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:147)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5876)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6027)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5814)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2552)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32385)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
> {noformat}
> Conditions:
> table T with a single column family 0 that uses ROWCOL bloom filter
> (important) and column qualifiers C1,C2,C3,C4,C5.
> When we fill the table for every row we put deleted cell for C3.
> The table has a single region with two HStore:
> A: start row: 0, stop row: 99
> B: start row: 10 stop row: 99
> B has newer versions of rows 10-99. Store files have several blocks each
> (important).
> Store A is the result of major compaction, so it doesn't have any deleted
> cells (important).
> So, we are running a scan like:
> {noformat}
> scan 'T', { COLUMNS => ['0:C3','0:C5'], FILTER => "SingleColumnValueFilter
> ('0','C5',=,'binary:whatever')"}
> {noformat}
> How the scan performs:
> First, we iterate A for rows 0 and 1 without any problems.
> Next, we start to iterate A for row 10, so read the first cell and set hfs
> scanner to A :
> 10:0/C1/0/Put/x but found that we have a newer version of the cell in B :
> 10:0/C1/1/Put/x,
> so we make B as our current store scanner. Since we are looking for
> particular columns
> C3 and C5, we perform the optimization StoreScanner.seekOrSkipToNextColumn
> which
> would run reseek for all store scanners.
> For store A the following magic would happen in requestSeek:
> 1. bloom filter check passesGeneralBloomFilter would set haveToSeek to
> false because row 10 doesn't have C3 qualifier in store A.
> 2. Since we don't have to seek we just create a fake row
> 10:0/C3/OLDEST_TIMESTAMP/Maximum, an optimization that is quite important for
> us and it commented with :
> {noformat}
> // Multi-column Bloom filter optimization.
> // Create a fake key/value, so that this scanner only bubbles up to the
> top
> // of the KeyValueHeap in StoreScanner after we scanned this row/column in
> // all other store files. The query matcher will then just skip this fake
> // key/value and the store scanner will progress to the next column. This
> // is obviously not a "real real" seek, but unlike the fake KV earlier in
> // this method, we want this to be propagated to ScanQueryMatcher.
> {noformat}
>
> For store B we would set it to fake 10:0/C3/createFirstOnRowColTS()/Maximum
> to skip C3 entirely.
> After that we start searching for qualifier C5 using seekOrSki
[jira] [Commented] (HBASE-20000) Remove the quantum logic in FairQueue, always put high priority queue in front
[ https://issues.apache.org/jira/browse/HBASE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364203#comment-16364203 ] Duo Zhang commented on HBASE-2: --- Will implement the reservation for procedure worker thread in the next patch since this should be a separated work. > Remove the quantum logic in FairQueue, always put high priority queue in front > -- > > Key: HBASE-2 > URL: https://issues.apache.org/jira/browse/HBASE-2 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20000) Remove the quantum logic in FairQueue, always put high priority queue in front
[ https://issues.apache.org/jira/browse/HBASE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-2: -- Status: Patch Available (was: Open) > Remove the quantum logic in FairQueue, always put high priority queue in front > -- > > Key: HBASE-2 > URL: https://issues.apache.org/jira/browse/HBASE-2 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20000) Remove the quantum logic in FairQueue, always put high priority queue in front
[ https://issues.apache.org/jira/browse/HBASE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364213#comment-16364213 ] Duo Zhang commented on HBASE-2: --- [~stack] FYI. > Remove the quantum logic in FairQueue, always put high priority queue in front > -- > > Key: HBASE-2 > URL: https://issues.apache.org/jira/browse/HBASE-2 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20000) Remove the quantum logic in FairQueue, always put high priority queue in front
[ https://issues.apache.org/jira/browse/HBASE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364212#comment-16364212 ] Duo Zhang commented on HBASE-2: --- Review board link: https://reviews.apache.org/r/65653/ > Remove the quantum logic in FairQueue, always put high priority queue in front > -- > > Key: HBASE-2 > URL: https://issues.apache.org/jira/browse/HBASE-2 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-15911) NPE in AssignmentManager.onRegionTransition after Master restart
[ https://issues.apache.org/jira/browse/HBASE-15911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364274#comment-16364274 ] Pankaj Kumar commented on HBASE-15911: -- Hi [~benlau], please feel free to take this Jira. I too fixed in our version but didn't find time to write test case. > NPE in AssignmentManager.onRegionTransition after Master restart > > > Key: HBASE-15911 > URL: https://issues.apache.org/jira/browse/HBASE-15911 > Project: HBase > Issue Type: Bug > Components: master, Region Assignment >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Major > > 16/05/27 17:49:18 ERROR ipc.RpcServer: Unexpected throwable object > java.lang.NullPointerException > at > org.apache.hadoop.hbase.master.AssignmentManager.onRegionTransition(AssignmentManager.java:4364) > at > org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionStateTransition(MasterRpcServices.java:1421) > at > org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8623) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2239) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:116) > at > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:137) > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:112) > at java.lang.Thread.run(Thread.java:745) > I'm pretty sure I've seen it before and more than once, but never got to dig > in. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20000) Remove the quantum logic in FairQueue, always put high priority queue in front
[ https://issues.apache.org/jira/browse/HBASE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364285#comment-16364285 ] Dave Latham commented on HBASE-2: - The new HBASE 20,000! It slices and dices! > Remove the quantum logic in FairQueue, always put high priority queue in front > -- > > Key: HBASE-2 > URL: https://issues.apache.org/jira/browse/HBASE-2 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19980) NullPointerException when restoring a snapshot after splitting a region
[
https://issues.apache.org/jira/browse/HBASE-19980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364297#comment-16364297
]
Mike Drob commented on HBASE-19980:
---
Can you add a test for mob regions too? Some of that code moved around and I'm
having some trouble following the intent.
> NullPointerException when restoring a snapshot after splitting a region
> ---
>
> Key: HBASE-19980
> URL: https://issues.apache.org/jira/browse/HBASE-19980
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Major
> Attachments: HBASE-19980-UT.patch, HBASE-19980.master.001.patch,
> HBASE-19980.master.002.patch
>
>
> Investigating HBASE-19893, I'm facing the following NullPointerException when
> restoring a snapshot after splitting a region:
> {code}
> 2018-02-13 01:59:56,033 ERROR [PEWorker-1]
> procedure.RestoreSnapshotProcedure(403): restore snapshot={
> ss=snaptb1-1518454780480
> table=testRestoreSnapshotAfterSplittingRegions-1518454780480 type=FLUSH }
> failed in on-disk restore. Try re-running the restore command.
> java.io.IOException
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils.editRegions(ModifyRegionUtils.java:222)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:424)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:227)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:177)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.restoreSnapshot(RestoreSnapshotProcedure.java:395)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.executeFromState(RestoreSnapshotProcedure.java:143)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.executeFromState(RestoreSnapshotProcedure.java:58)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:180)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1455)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1224)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1734)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreReferenceFile(RestoreSnapshotHelper.java:751)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreStoreFile(RestoreSnapshotHelper.java:675)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreRegion(RestoreSnapshotHelper.java:521)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreRegion(RestoreSnapshotHelper.java:464)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.access$300(RestoreSnapshotHelper.java:116)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper$2.editRegion(RestoreSnapshotHelper.java:427)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$2.call(ModifyRegionUtils.java:209)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$2.call(ModifyRegionUtils.java:206)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19980) NullPointerException when restoring a snapshot after splitting a region
[
https://issues.apache.org/jira/browse/HBASE-19980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364302#comment-16364302
]
Toshihiro Suzuki commented on HBASE-19980:
--
[~mdrob] We have TestMobRestoreSnapshotFromClient. Isn't it enough?
> NullPointerException when restoring a snapshot after splitting a region
> ---
>
> Key: HBASE-19980
> URL: https://issues.apache.org/jira/browse/HBASE-19980
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Major
> Attachments: HBASE-19980-UT.patch, HBASE-19980.master.001.patch,
> HBASE-19980.master.002.patch
>
>
> Investigating HBASE-19893, I'm facing the following NullPointerException when
> restoring a snapshot after splitting a region:
> {code}
> 2018-02-13 01:59:56,033 ERROR [PEWorker-1]
> procedure.RestoreSnapshotProcedure(403): restore snapshot={
> ss=snaptb1-1518454780480
> table=testRestoreSnapshotAfterSplittingRegions-1518454780480 type=FLUSH }
> failed in on-disk restore. Try re-running the restore command.
> java.io.IOException
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils.editRegions(ModifyRegionUtils.java:222)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:424)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:227)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:177)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.restoreSnapshot(RestoreSnapshotProcedure.java:395)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.executeFromState(RestoreSnapshotProcedure.java:143)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.executeFromState(RestoreSnapshotProcedure.java:58)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:180)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1455)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1224)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1734)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreReferenceFile(RestoreSnapshotHelper.java:751)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreStoreFile(RestoreSnapshotHelper.java:675)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreRegion(RestoreSnapshotHelper.java:521)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreRegion(RestoreSnapshotHelper.java:464)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.access$300(RestoreSnapshotHelper.java:116)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper$2.editRegion(RestoreSnapshotHelper.java:427)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$2.call(ModifyRegionUtils.java:209)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$2.call(ModifyRegionUtils.java:206)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19116) Currently the tail of hfiles with CellComparator* classname makes it so hbase1 can't open hbase2 written hfiles; fix
[ https://issues.apache.org/jira/browse/HBASE-19116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19116: -- Resolution: Fixed Hadoop Flags: Reviewed Release Note: hbase-2.x sets KeyValue Comparators into the tail of hfiles rather than CellComparator, what it uses internally, just so hbase-1.x can continue to read hbase-2.x written hfiles. Status: Resolved (was: Patch Available) Pushed to master and branch-2. > Currently the tail of hfiles with CellComparator* classname makes it so > hbase1 can't open hbase2 written hfiles; fix > > > Key: HBASE-19116 > URL: https://issues.apache.org/jira/browse/HBASE-19116 > Project: HBase > Issue Type: Sub-task > Components: HFile, migration >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19116.branch-2.001.patch, > HBASE-19116.branch-2.002.patch, HBASE-19116.branch-2.003.patch, > HBASE-19116.branch-2.004.patch > > > See tail of HBASE-19052 for discussion which concludes we should try and make > it so operators do not have to go to latest hbase version before they > upgrade, at least if we can avoid it. > The necessary change of our default comparator from KV to Cell naming has > hfiles with tails that have the classname CellComparator in them in place of > KeyValueComparator. If an hbase1 tries to open them, it will fail not having > a CellComparator in its classpath (We have name of comparator in tail because > different files require different comparators... perhaps we write an alias > instead of a class one day... TODO). HBASE-16189 and HBASE-19052 are about > trying to carry knowledge of hbase2 back to hbase1, a brittle approach making > it so operators will have to upgrade to the latest branch-1 before they can > go to hbase2. > This issue is about undoing our writing of an incompatible (to hbase1) tail, > not unless we really have to (and it sounds like we could do without writing > an incompatible tail) to see if we can avoid requiring operators go to > lastest branch-1 (we may end up needing this but lets a have a really good > reason for it if we do). > Oh, let this filing be an answer to our [~anoop.hbase]'s old high-level > question over in HBASE-16189: > bq. ...means when rolling upgrade done to 2.0, first users have to upgrade to > some 1.x versions which is having this fix and then to 2.0.. What do you guys > think Whether we should avoid this kind of indirection? cc Enis Soztutar, > Stack, Ted Yu, Matteo Bertozzi > Yeah, lets try to avoid this if we can... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19980) NullPointerException when restoring a snapshot after splitting a region
[
https://issues.apache.org/jira/browse/HBASE-19980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364304#comment-16364304
]
Mike Drob commented on HBASE-19980:
---
Thanks for pointing me there, I will run it and take a look.
> NullPointerException when restoring a snapshot after splitting a region
> ---
>
> Key: HBASE-19980
> URL: https://issues.apache.org/jira/browse/HBASE-19980
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Major
> Attachments: HBASE-19980-UT.patch, HBASE-19980.master.001.patch,
> HBASE-19980.master.002.patch
>
>
> Investigating HBASE-19893, I'm facing the following NullPointerException when
> restoring a snapshot after splitting a region:
> {code}
> 2018-02-13 01:59:56,033 ERROR [PEWorker-1]
> procedure.RestoreSnapshotProcedure(403): restore snapshot={
> ss=snaptb1-1518454780480
> table=testRestoreSnapshotAfterSplittingRegions-1518454780480 type=FLUSH }
> failed in on-disk restore. Try re-running the restore command.
> java.io.IOException
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils.editRegions(ModifyRegionUtils.java:222)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:424)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:227)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:177)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.restoreSnapshot(RestoreSnapshotProcedure.java:395)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.executeFromState(RestoreSnapshotProcedure.java:143)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.executeFromState(RestoreSnapshotProcedure.java:58)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:180)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1455)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1224)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1734)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreReferenceFile(RestoreSnapshotHelper.java:751)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreStoreFile(RestoreSnapshotHelper.java:675)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreRegion(RestoreSnapshotHelper.java:521)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreRegion(RestoreSnapshotHelper.java:464)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.access$300(RestoreSnapshotHelper.java:116)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper$2.editRegion(RestoreSnapshotHelper.java:427)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$2.call(ModifyRegionUtils.java:209)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$2.call(ModifyRegionUtils.java:206)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364325#comment-16364325 ] stack commented on HBASE-18294: --- bq. I think it deserves a separate documentation since this Jira did not change the barriers computation but what we compare against them. Ok. bq. If all is ok, I will commit later today. Sounds good to me. > Reduce global heap pressure: flush based on heap occupancy > -- > > Key: HBASE-18294 > URL: https://issues.apache.org/jira/browse/HBASE-18294 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.01.patch, HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.01.patch, HBASE-18294.01.patch, HBASE-18294.02.patch, > HBASE-18294.03.patch, HBASE-18294.04.patch, HBASE-18294.05.patch, > HBASE-18294.06.patch, HBASE-18294.07.patch, HBASE-18294.07.patch, > HBASE-18294.08.patch, HBASE-18294.09.patch, HBASE-18294.10.patch, > HBASE-18294.11.patch, HBASE-18294.11.patch, HBASE-18294.12.patch, > HBASE-18294.13.patch, HBASE-18294.15.patch, HBASE-18294.16.patch, > HBASE-18294.master.01.patch, HBASE-18294.master.01.patch > > > A region is flushed if its memory component exceed a threshold (default size > is 128MB). > A flush policy decides whether to flush a store by comparing the size of the > store to another threshold (that can be configured with > hbase.hregion.percolumnfamilyflush.size.lower.bound). > Currently the implementation (in both cases) compares the data size > (key-value only) to the threshold where it should compare the heap size > (which includes index size, and metadata). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19506) Support variable sized chunks from ChunkCreator
[ https://issues.apache.org/jira/browse/HBASE-19506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gali Sheffi updated HBASE-19506: Attachment: HBASE-19506-V06.patch > Support variable sized chunks from ChunkCreator > --- > > Key: HBASE-19506 > URL: https://issues.apache.org/jira/browse/HBASE-19506 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Priority: Major > Attachments: HBASE-19506-V01.patch, HBASE-19506-V02.patch, > HBASE-19506-V03.patch, HBASE-19506-V04.patch, HBASE-19506-V05.patch, > HBASE-19506-V06.patch > > > When CellChunkMap is created it allocates a special index chunk (or chunks) > where array of cell-representations is stored. When the number of > cell-representations is small, it is preferable to allocate a chunk smaller > than a default value which is 2MB. > On the other hand, those "non-standard size" chunks can not be used in pool. > On-demand allocations in off-heap are costly. So this JIRA is about to > investigate the trade of between memory usage and the final performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19980) NullPointerException when restoring a snapshot after splitting a region
[
https://issues.apache.org/jira/browse/HBASE-19980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364431#comment-16364431
]
Hadoop QA commented on HBASE-19980:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
45s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} hbase-server: The patch generated 0 new + 19
unchanged - 1 fixed = 19 total (was 20) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
42s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
18m 16s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}116m
18s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}153m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19980 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910585/HBASE-19980.master.002.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 3590f1bcc7f5 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 2f1b3eab67 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/11523/testReport/ |
| Max. process+thread count | 5429 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/11523/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> NullPointerException when restoring a snapshot after splitting a region
>
[jira] [Commented] (HBASE-19980) NullPointerException when restoring a snapshot after splitting a region
[
https://issues.apache.org/jira/browse/HBASE-19980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364435#comment-16364435
]
Mike Drob commented on HBASE-19980:
---
I still don't 100% understand why we needed to move how we handle MOB restores,
but the tests pass so I'm reasonably happy with this.
> NullPointerException when restoring a snapshot after splitting a region
> ---
>
> Key: HBASE-19980
> URL: https://issues.apache.org/jira/browse/HBASE-19980
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Major
> Attachments: HBASE-19980-UT.patch, HBASE-19980.master.001.patch,
> HBASE-19980.master.002.patch
>
>
> Investigating HBASE-19893, I'm facing the following NullPointerException when
> restoring a snapshot after splitting a region:
> {code}
> 2018-02-13 01:59:56,033 ERROR [PEWorker-1]
> procedure.RestoreSnapshotProcedure(403): restore snapshot={
> ss=snaptb1-1518454780480
> table=testRestoreSnapshotAfterSplittingRegions-1518454780480 type=FLUSH }
> failed in on-disk restore. Try re-running the restore command.
> java.io.IOException
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils.editRegions(ModifyRegionUtils.java:222)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:424)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:227)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:177)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.restoreSnapshot(RestoreSnapshotProcedure.java:395)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.executeFromState(RestoreSnapshotProcedure.java:143)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.executeFromState(RestoreSnapshotProcedure.java:58)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:180)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1455)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1224)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1734)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreReferenceFile(RestoreSnapshotHelper.java:751)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreStoreFile(RestoreSnapshotHelper.java:675)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreRegion(RestoreSnapshotHelper.java:521)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreRegion(RestoreSnapshotHelper.java:464)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.access$300(RestoreSnapshotHelper.java:116)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper$2.editRegion(RestoreSnapshotHelper.java:427)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$2.call(ModifyRegionUtils.java:209)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$2.call(ModifyRegionUtils.java:206)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19980) NullPointerException when restoring a snapshot after splitting a region
[
https://issues.apache.org/jira/browse/HBASE-19980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364441#comment-16364441
]
Ted Yu commented on HBASE-19980:
[~mdrob]:
I interpret the above as +1
> NullPointerException when restoring a snapshot after splitting a region
> ---
>
> Key: HBASE-19980
> URL: https://issues.apache.org/jira/browse/HBASE-19980
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Major
> Attachments: HBASE-19980-UT.patch, HBASE-19980.master.001.patch,
> HBASE-19980.master.002.patch
>
>
> Investigating HBASE-19893, I'm facing the following NullPointerException when
> restoring a snapshot after splitting a region:
> {code}
> 2018-02-13 01:59:56,033 ERROR [PEWorker-1]
> procedure.RestoreSnapshotProcedure(403): restore snapshot={
> ss=snaptb1-1518454780480
> table=testRestoreSnapshotAfterSplittingRegions-1518454780480 type=FLUSH }
> failed in on-disk restore. Try re-running the restore command.
> java.io.IOException
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils.editRegions(ModifyRegionUtils.java:222)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:424)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:227)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:177)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.restoreSnapshot(RestoreSnapshotProcedure.java:395)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.executeFromState(RestoreSnapshotProcedure.java:143)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.executeFromState(RestoreSnapshotProcedure.java:58)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:180)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1455)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1224)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1734)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreReferenceFile(RestoreSnapshotHelper.java:751)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreStoreFile(RestoreSnapshotHelper.java:675)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreRegion(RestoreSnapshotHelper.java:521)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreRegion(RestoreSnapshotHelper.java:464)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.access$300(RestoreSnapshotHelper.java:116)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper$2.editRegion(RestoreSnapshotHelper.java:427)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$2.call(ModifyRegionUtils.java:209)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$2.call(ModifyRegionUtils.java:206)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-18294: -- Attachment: HBASE-18294.01.patch > Reduce global heap pressure: flush based on heap occupancy > -- > > Key: HBASE-18294 > URL: https://issues.apache.org/jira/browse/HBASE-18294 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.01.patch, HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.01.patch, HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.02.patch, HBASE-18294.03.patch, HBASE-18294.04.patch, > HBASE-18294.05.patch, HBASE-18294.06.patch, HBASE-18294.07.patch, > HBASE-18294.07.patch, HBASE-18294.08.patch, HBASE-18294.09.patch, > HBASE-18294.10.patch, HBASE-18294.11.patch, HBASE-18294.11.patch, > HBASE-18294.12.patch, HBASE-18294.13.patch, HBASE-18294.15.patch, > HBASE-18294.16.patch, HBASE-18294.master.01.patch, HBASE-18294.master.01.patch > > > A region is flushed if its memory component exceed a threshold (default size > is 128MB). > A flush policy decides whether to flush a store by comparing the size of the > store to another threshold (that can be configured with > hbase.hregion.percolumnfamilyflush.size.lower.bound). > Currently the implementation (in both cases) compares the data size > (key-value only) to the threshold where it should compare the heap size > (which includes index size, and metadata). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19320) document the mysterious direct memory leak in hbase
[ https://issues.apache.org/jira/browse/HBASE-19320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364451#comment-16364451 ] stack commented on HBASE-19320: --- It is beautiful [~huaxiang] +1 How about enabling it by default? > document the mysterious direct memory leak in hbase > > > Key: HBASE-19320 > URL: https://issues.apache.org/jira/browse/HBASE-19320 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0, 1.2.6 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Major > Attachments: HBASE-19320-master-v001.patch, Screen Shot 2017-11-21 at > 4.43.36 PM.png, Screen Shot 2017-11-21 at 4.44.22 PM.png > > > Recently we run into a direct memory leak case, which takes some time to > trace and debug. Internally discussed with our [~saint@gmail.com], we > thought we had some findings and want to share with the community. > Basically, it is the issue described in > http://www.evanjones.ca/java-bytebuffer-leak.html and it happened to one of > our hbase clusters. > Create the jira first and will fill in more details later. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19996) Some nonce procs might not be cleaned up (follow up HBASE-19756)
[ https://issues.apache.org/jira/browse/HBASE-19996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364457#comment-16364457 ] Ted Yu commented on HBASE-19996: Please attach patch for branch-2 > Some nonce procs might not be cleaned up (follow up HBASE-19756) > > > Key: HBASE-19996 > URL: https://issues.apache.org/jira/browse/HBASE-19996 > Project: HBase > Issue Type: Bug >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Fix For: 2.0.0, 1.3.2, 1.5.0, 1.4.2 > > Attachments: HBASE-19996.branch-1.4.001.patch, > HBASE-19996.branch-1.4.001.patch > > > Follow up to HBASE-19756 which dealt with NPEs during proc cleanup. > Unfortunately, the patch for branch-1 might not remove some valid procs too. > The branch-2 patch doesn't have this problem. This fixes the branch-1 bug and > also adds another test to branch-2. Thanks to [~toffer] for flagging this > internally. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19980) NullPointerException when restoring a snapshot after splitting a region
[
https://issues.apache.org/jira/browse/HBASE-19980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364454#comment-16364454
]
Mike Drob commented on HBASE-19980:
---
+0, but I'm not going to keep you from committing this.
> NullPointerException when restoring a snapshot after splitting a region
> ---
>
> Key: HBASE-19980
> URL: https://issues.apache.org/jira/browse/HBASE-19980
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Major
> Attachments: HBASE-19980-UT.patch, HBASE-19980.master.001.patch,
> HBASE-19980.master.002.patch
>
>
> Investigating HBASE-19893, I'm facing the following NullPointerException when
> restoring a snapshot after splitting a region:
> {code}
> 2018-02-13 01:59:56,033 ERROR [PEWorker-1]
> procedure.RestoreSnapshotProcedure(403): restore snapshot={
> ss=snaptb1-1518454780480
> table=testRestoreSnapshotAfterSplittingRegions-1518454780480 type=FLUSH }
> failed in on-disk restore. Try re-running the restore command.
> java.io.IOException
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils.editRegions(ModifyRegionUtils.java:222)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:424)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:227)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:177)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.restoreSnapshot(RestoreSnapshotProcedure.java:395)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.executeFromState(RestoreSnapshotProcedure.java:143)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.executeFromState(RestoreSnapshotProcedure.java:58)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:180)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1455)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1224)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1734)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreReferenceFile(RestoreSnapshotHelper.java:751)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreStoreFile(RestoreSnapshotHelper.java:675)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreRegion(RestoreSnapshotHelper.java:521)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreRegion(RestoreSnapshotHelper.java:464)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.access$300(RestoreSnapshotHelper.java:116)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper$2.editRegion(RestoreSnapshotHelper.java:427)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$2.call(ModifyRegionUtils.java:209)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$2.call(ModifyRegionUtils.java:206)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364463#comment-16364463 ] Eshcar Hillel commented on HBASE-18294: --- Following a comment by Anoop on RB I made some additional changes. Specifically propagating the flush type which is the result of the global pressure checks to the MemStoreFlusher deciding which region to flush. This allows choosing the region based on its on/off heap size w.r.t. the global heap pressure triggering the flush. I think this also answers an earlier question by stack regarding the code in the MemStoreFlusher. Anoop the data size tracking at the RS is redundant as you suggested, but since it is already used in several tests I prefer to keep it. Let's see how QA ends up. > Reduce global heap pressure: flush based on heap occupancy > -- > > Key: HBASE-18294 > URL: https://issues.apache.org/jira/browse/HBASE-18294 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.01.patch, HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.01.patch, HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.02.patch, HBASE-18294.03.patch, HBASE-18294.04.patch, > HBASE-18294.05.patch, HBASE-18294.06.patch, HBASE-18294.07.patch, > HBASE-18294.07.patch, HBASE-18294.08.patch, HBASE-18294.09.patch, > HBASE-18294.10.patch, HBASE-18294.11.patch, HBASE-18294.11.patch, > HBASE-18294.12.patch, HBASE-18294.13.patch, HBASE-18294.15.patch, > HBASE-18294.16.patch, HBASE-18294.master.01.patch, HBASE-18294.master.01.patch > > > A region is flushed if its memory component exceed a threshold (default size > is 128MB). > A flush policy decides whether to flush a store by comparing the size of the > store to another threshold (that can be configured with > hbase.hregion.percolumnfamilyflush.size.lower.bound). > Currently the implementation (in both cases) compares the data size > (key-value only) to the threshold where it should compare the heap size > (which includes index size, and metadata). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19320) document the mysterious direct memory leak in hbase
[ https://issues.apache.org/jira/browse/HBASE-19320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364473#comment-16364473 ] huaxiang sun commented on HBASE-19320: -- thanks [~saint@gmail.com] for review. It needs some manual work as the fix from JDK only applies to JDK8u102+, enable it by default sometimes may work (sometimes may not if jvm version is not right). > document the mysterious direct memory leak in hbase > > > Key: HBASE-19320 > URL: https://issues.apache.org/jira/browse/HBASE-19320 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0, 1.2.6 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Major > Attachments: HBASE-19320-master-v001.patch, Screen Shot 2017-11-21 at > 4.43.36 PM.png, Screen Shot 2017-11-21 at 4.44.22 PM.png > > > Recently we run into a direct memory leak case, which takes some time to > trace and debug. Internally discussed with our [~saint@gmail.com], we > thought we had some findings and want to share with the community. > Basically, it is the issue described in > http://www.evanjones.ca/java-bytebuffer-leak.html and it happened to one of > our hbase clusters. > Create the jira first and will fill in more details later. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19980) NullPointerException when restoring a snapshot after splitting a region
[
https://issues.apache.org/jira/browse/HBASE-19980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-19980:
---
Hadoop Flags: Reviewed
Fix Version/s: 2.0.0-beta-2
Please attach patch for branch-1
> NullPointerException when restoring a snapshot after splitting a region
> ---
>
> Key: HBASE-19980
> URL: https://issues.apache.org/jira/browse/HBASE-19980
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19980-UT.patch, HBASE-19980.master.001.patch,
> HBASE-19980.master.002.patch
>
>
> Investigating HBASE-19893, I'm facing the following NullPointerException when
> restoring a snapshot after splitting a region:
> {code}
> 2018-02-13 01:59:56,033 ERROR [PEWorker-1]
> procedure.RestoreSnapshotProcedure(403): restore snapshot={
> ss=snaptb1-1518454780480
> table=testRestoreSnapshotAfterSplittingRegions-1518454780480 type=FLUSH }
> failed in on-disk restore. Try re-running the restore command.
> java.io.IOException
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils.editRegions(ModifyRegionUtils.java:222)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:424)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:227)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:177)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.restoreSnapshot(RestoreSnapshotProcedure.java:395)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.executeFromState(RestoreSnapshotProcedure.java:143)
> at
> org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.executeFromState(RestoreSnapshotProcedure.java:58)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:180)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1455)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1224)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1734)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreReferenceFile(RestoreSnapshotHelper.java:751)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreStoreFile(RestoreSnapshotHelper.java:675)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreRegion(RestoreSnapshotHelper.java:521)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreRegion(RestoreSnapshotHelper.java:464)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.access$300(RestoreSnapshotHelper.java:116)
> at
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper$2.editRegion(RestoreSnapshotHelper.java:427)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$2.call(ModifyRegionUtils.java:209)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$2.call(ModifyRegionUtils.java:206)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19998) Flakey TestVisibilityLabelsWithDefaultVisLabelService
[
https://issues.apache.org/jira/browse/HBASE-19998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364511#comment-16364511
]
stack commented on HBASE-19998:
---
Debug turned up the culprit 'mover':
{code}
2018-02-14 12:35:07,975 DEBUG [PEWorker-6] procedure.ServerCrashProcedure(217):
Splitting WALs pid=9, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS;
ServerCrashProcedure server=asf903.gq1.ygridcore.net,56387,1518611700720,
splitWal=true, meta=false
2018-02-14 12:35:07,978 INFO [Thread-213] assignment.MoveRegionProcedure(57):
REMOVE
java.lang.Throwable: REMOVE: Just to see who is calling Move!!!
at
org.apache.hadoop.hbase.master.assignment.MoveRegionProcedure.(MoveRegionProcedure.java:57)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.createMoveRegionProcedure(AssignmentManager.java:740)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.moveAsync(AssignmentManager.java:545)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.lambda$checkIfShouldMoveSystemRegionAsync$0(AssignmentManager.java:481)
at java.lang.Thread.run(Thread.java:748)
2018-02-14 12:35:07,986 DEBUG [PEWorker-6] master.MasterWalManager(257):
Renamed region directory:
hdfs://localhost:39302/user/jenkins/test-data/4b64143d-6b3f-4516-9efb-0c6c1a438493/WALs/asf903.gq1.ygridcore.net,56387,1518611700720-splitting
2018-02-14 12:35:07,986 INFO [PEWorker-6] master.SplitLogManager(461): dead
splitlog workers [asf903.gq1.ygridcore.net,56387,1518611700720]
2018-02-14 12:35:07,986 DEBUG [Time-limited test-EventThread]
procedure2.ProcedureExecutor(868): Stored pid=10,
state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure
server=asf903.gq1.ygridcore.net,51305,1518611700792, splitWal=true, meta=true
2018-02-14 12:35:07,987 DEBUG [Time-limited test-EventThread]
assignment.AssignmentManager(1382):
Added=asf903.gq1.ygridcore.net,51305,1518611700792 to dead servers, submitted
shutdown handler to be executed meta=true
2018-02-14 12:35:07,988 DEBUG [Time-limited test-EventThread]
zookeeper.ZKUtil(355): regionserver:35367-0x161944f90b6000c,
quorum=localhost:59400, baseZNode=/hbase Set watcher on existing
znode=/hbase/rs/asf903.gq1.ygridcore.net,35367,1518611707634
2018-02-14 12:35:07,988 INFO [PEWorker-8] procedure.ServerCrashProcedure(118):
Start pid=10, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure
server=asf903.gq1.ygridcore.net,51305,1518611700792, splitWal=true, meta=true
2018-02-14 12:35:07,992 INFO [PEWorker-6] master.SplitLogManager(241): Started
splitting 1 logs in
[hdfs://localhost:39302/user/jenkins/test-data/4b64143d-6b3f-4516-9efb-0c6c1a438493/WALs/asf903.gq1.ygridcore.net,56387,1518611700720-splitting]
for [asf903.gq1.ygridcore.net,56387,1518611700720]
2018-02-14 12:35:07,999 DEBUG [Thread-213] procedure2.ProcedureExecutor(868):
Stored pid=11, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure
hri=hbase:meta,,1.1588230740,
source=asf903.gq1.ygridcore.net,51305,1518611700792, destination=
2018-02-14 12:35:08,003 DEBUG [PEWorker-9] procedure.ServerCrashProcedure(192):
pid=10, state=RUNNABLE:SERVER_CRASH_PROCESS_META; ServerCrashProcedure
server=asf903.gq1.ygridcore.net,51305,1518611700792, splitWal=true, meta=true;
processing hbase:meta
2018-02-14 12:35:08,003 INFO [PEWorker-10]
procedure.MasterProcedureScheduler(813): pid=11,
state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure
hri=hbase:meta,,1.1588230740,
source=asf903.gq1.ygridcore.net,51305,1518611700792, destination=,
table=hbase:meta, hbase:meta,,1.1588230740
2018-02-14 12:35:08,003 INFO [PEWorker-9] procedure2.ProcedureExecutor(1498):
Initialized subprocedures=[{pid=12, ppid=10,
state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure
failedMetaServer=asf903.gq1.ygridcore.net,51305,1518611700792, splitWal=true}]
2018-02-14 12:35:08,005 INFO [PEWorker-10] procedure2.ProcedureExecutor(1498):
Initialized subprocedures=[{pid=13, ppid=11,
state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=hbase:meta,
region=1588230740, server=asf903.gq1.ygridcore.net,51305,1518611700792}]
2018-02-14 12:35:08,006 INFO [Thread-213] assignment.MoveRegionProcedure(57):
REMOVE
java.lang.Throwable: REMOVE: Just to see who is calling Move!!!
at
org.apache.hadoop.hbase.master.assignment.MoveRegionProcedure.(MoveRegionProcedure.java:57)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.createMoveRegionProcedure(AssignmentManager.java:740)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.moveAsync(AssignmentManager.java:545)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.lambda$checkIfShouldMoveSystemRegionAsync$0(AssignmentManager.java:488)
at java.lang.Thread.run(Thread.java:748)
{code}
Its the facility added over in HBASE-18494 that is cutting in. In this case two
RegionServers are killed and
[jira] [Commented] (HBASE-20000) Remove the quantum logic in FairQueue, always put high priority queue in front
[
https://issues.apache.org/jira/browse/HBASE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364512#comment-16364512
]
Hadoop QA commented on HBASE-2:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
16s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
10s{color} | {color:red} hbase-server: The patch generated 5 new + 18 unchanged
- 17 fixed = 23 total (was 35) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
11s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
21m 46s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
28s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}117m 4s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}164m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-2 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910592/HBASE-2.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 808b7b9ddc33 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 2f1b3eab67 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-B
[jira] [Commented] (HBASE-19320) document the mysterious direct memory leak in hbase
[ https://issues.apache.org/jira/browse/HBASE-19320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364523#comment-16364523 ] stack commented on HBASE-19320: --- Is it harmless if it is in place? Can be different issue. +1 on committing this. Thanks [~huaxiang] > document the mysterious direct memory leak in hbase > > > Key: HBASE-19320 > URL: https://issues.apache.org/jira/browse/HBASE-19320 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0, 1.2.6 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Major > Attachments: HBASE-19320-master-v001.patch, Screen Shot 2017-11-21 at > 4.43.36 PM.png, Screen Shot 2017-11-21 at 4.44.22 PM.png > > > Recently we run into a direct memory leak case, which takes some time to > trace and debug. Internally discussed with our [~saint@gmail.com], we > thought we had some findings and want to share with the community. > Basically, it is the issue described in > http://www.evanjones.ca/java-bytebuffer-leak.html and it happened to one of > our hbase clusters. > Create the jira first and will fill in more details later. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19863) java.lang.IllegalStateException: isDelete failed when SingleColumnValueFilter is used
[
https://issues.apache.org/jira/browse/HBASE-19863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364544#comment-16364544
]
Chia-Ping Tsai commented on HBASE-19863:
{quote}[~chia7712] - you have any comments?
{quote}
Working at comprehending the root causeWill come back asap.
> java.lang.IllegalStateException: isDelete failed when SingleColumnValueFilter
> is used
> -
>
> Key: HBASE-19863
> URL: https://issues.apache.org/jira/browse/HBASE-19863
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 1.4.1
>Reporter: Sergey Soldatov
>Assignee: Sergey Soldatov
>Priority: Major
> Attachments: HBASE-19863-branch-2.patch, HBASE-19863-branch1.patch,
> HBASE-19863-test.patch, HBASE-19863.v2-branch-2.patch
>
>
> Under some circumstances scan with SingleColumnValueFilter may fail with an
> exception
> {noformat}
> java.lang.IllegalStateException: isDelete failed: deleteBuffer=C3,
> qualifier=C2, timestamp=1516433595543, comparison result: 1
> at
> org.apache.hadoop.hbase.regionserver.ScanDeleteTracker.isDeleted(ScanDeleteTracker.java:149)
> at
> org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.match(ScanQueryMatcher.java:386)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:545)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:147)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5876)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6027)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5814)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2552)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32385)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
> {noformat}
> Conditions:
> table T with a single column family 0 that uses ROWCOL bloom filter
> (important) and column qualifiers C1,C2,C3,C4,C5.
> When we fill the table for every row we put deleted cell for C3.
> The table has a single region with two HStore:
> A: start row: 0, stop row: 99
> B: start row: 10 stop row: 99
> B has newer versions of rows 10-99. Store files have several blocks each
> (important).
> Store A is the result of major compaction, so it doesn't have any deleted
> cells (important).
> So, we are running a scan like:
> {noformat}
> scan 'T', { COLUMNS => ['0:C3','0:C5'], FILTER => "SingleColumnValueFilter
> ('0','C5',=,'binary:whatever')"}
> {noformat}
> How the scan performs:
> First, we iterate A for rows 0 and 1 without any problems.
> Next, we start to iterate A for row 10, so read the first cell and set hfs
> scanner to A :
> 10:0/C1/0/Put/x but found that we have a newer version of the cell in B :
> 10:0/C1/1/Put/x,
> so we make B as our current store scanner. Since we are looking for
> particular columns
> C3 and C5, we perform the optimization StoreScanner.seekOrSkipToNextColumn
> which
> would run reseek for all store scanners.
> For store A the following magic would happen in requestSeek:
> 1. bloom filter check passesGeneralBloomFilter would set haveToSeek to
> false because row 10 doesn't have C3 qualifier in store A.
> 2. Since we don't have to seek we just create a fake row
> 10:0/C3/OLDEST_TIMESTAMP/Maximum, an optimization that is quite important for
> us and it commented with :
> {noformat}
> // Multi-column Bloom filter optimization.
> // Create a fake key/value, so that this scanner only bubbles up to the
> top
> // of the KeyValueHeap in StoreScanner after we scanned this row/column in
> // all other store files. The query matcher will then just skip this fake
> // key/value and the store scanner will progress to the next column. This
> // is obviously not a "real real" seek, but unlike the fake KV earlier in
> // this method, we want this to be propagated to ScanQueryMatcher.
> {noformat}
>
> For store B we would set it to fake 10:0/C3/createFirstOnRowColTS()/Maximum
> to skip C3 entirely.
> After that we start searching for qualifier C5 using seekOrSkipToNextColumn
> which run first trySkipToNextColumn:
> {noformat}
> protected boolean trySkipToNextColumn(Cell cell) throws IOException {
> Cell nextCell = null;
[jira] [Commented] (HBASE-19506) Support variable sized chunks from ChunkCreator
[
https://issues.apache.org/jira/browse/HBASE-19506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364580#comment-16364580
]
Hadoop QA commented on HBASE-19506:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
52s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
54s{color} | {color:red} hbase-server: The patch generated 1 new + 29 unchanged
- 3 fixed = 30 total (was 32) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
3s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m
59s{color} | {color:red} The patch causes 10 errors with Hadoop v2.6.5. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 7m
55s{color} | {color:red} The patch causes 10 errors with Hadoop v2.7.4. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 9m
57s{color} | {color:red} The patch causes 10 errors with Hadoop v3.0.0. {color}
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}122m 41s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}149m 3s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.regionserver.TestWalAndCompactingMemStoreFlush |
| | hadoop.hbase.client.TestSnapshotFromClientWithRegionReplicas |
| | hadoop.hbase.client.TestMobSnapshotFromClient |
| | hadoop.hbase.master.cleaner.TestSnapshotFromMaster |
| | hadoop.hbase.client.TestSnapshotMetadata |
| | hadoop.hbase.regionserver.TestCompactSplitThread |
| | hadoop.hbase.client.TestSnapshotCloneIndependence |
| | hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd |
| | hadoop.hbase.client.TestSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19506 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910600/HBASE-19506-V06.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 77b272599238 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
[jira] [Commented] (HBASE-18282) ReplicationLogCleaner can delete WALs not yet replicated in case of a KeeperException
[
https://issues.apache.org/jira/browse/HBASE-18282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364666#comment-16364666
]
Ben Lau commented on HBASE-18282:
-
Thanks [~apurtell]. It looks like this code is rewritten in master now so the
bug is present only in branch-2 and branch-1. I will attach forward ports for
those branches.
> ReplicationLogCleaner can delete WALs not yet replicated in case of a
> KeeperException
> -
>
> Key: HBASE-18282
> URL: https://issues.apache.org/jira/browse/HBASE-18282
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.1, 1.2.6, 1.1.11, 2.0.0-alpha-1
>Reporter: Ashu Pachauri
>Priority: Critical
>
> ReplicationStateZKBase#getListOfReplicators does not rethrow a
> KeeperException and returns null in such a case. ReplicationLogCleaner just
> assumes that there are no replicators and deletes everything.
> ReplicationStateZKBase:
> {code:java}
> public List getListOfReplicators() {
> List result = null;
> try {
> result = ZKUtil.listChildrenNoWatch(this.zookeeper, this.queuesZNode);
> } catch (KeeperException e) {
> this.abortable.abort("Failed to get list of replicators", e);
> }
> return result;
> }
> {code}
> ReplicationLogCleaner:
> {code:java}
> private Set loadWALsFromQueues() throws KeeperException {
> for (int retry = 0; ; retry++) {
> int v0 = replicationQueues.getQueuesZNodeCversion();
> List rss = replicationQueues.getListOfReplicators();
> if (rss == null) {
> LOG.debug("Didn't find any region server that replicates, won't
> prevent any deletions.");
> return ImmutableSet.of();
> }
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19984) Add hadoop 2.8 and 2.9 to precommit
[ https://issues.apache.org/jira/browse/HBASE-19984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364670#comment-16364670 ] Mike Drob commented on HBASE-19984: --- it sounds like there is enough reason to add both of those to our precommit checks > Add hadoop 2.8 and 2.9 to precommit > --- > > Key: HBASE-19984 > URL: https://issues.apache.org/jira/browse/HBASE-19984 > Project: HBase > Issue Type: Sub-task >Reporter: Mike Drob >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20000) Remove the quantum logic in FairQueue, always put high priority queue in front
[
https://issues.apache.org/jira/browse/HBASE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364693#comment-16364693
]
Appy commented on HBASE-2:
--
Just an idea to discuss (other comments in Review board)
We have different impl of isAvailable() for each queue which are more or large
the same.
But as we get more types, it'll become increasingly hard to reason about system
if the queue types grow different from each other.
Luckily in this case, it doesn't have to be this way.
ServerQueue uses Queue class' impl since there are no server procedures which
spawn child procedure which try to take lock on same server. So the simple
check is sufficient. But it can also do with same impl has peerqueue.
Can we make our lifes easier by making a simple invariant in procedure
framework -
"Any exclusive lock held by a parent proc P will not block it's child procedure
trying to get the same lock"
That makes it easy to reason about the framework. Implementation wise, it's
just one common implementation of isAvailable().
TableQueue can do
pubilc boolean isAvailable() {
if (namespaceLockStat.hasExclusiveLock()) {
return false;
} else {
return super.isAvailable();
}
}
> Remove the quantum logic in FairQueue, always put high priority queue in front
> --
>
> Key: HBASE-2
> URL: https://issues.apache.org/jira/browse/HBASE-2
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-2.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[
https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364705#comment-16364705
]
Hadoop QA commented on HBASE-18294:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 17 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 8m
28s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
43s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
40s{color} | {color:red} hbase-server: The patch generated 30 new + 807
unchanged - 7 fixed = 837 total (was 814) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
14s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
18m 53s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
17s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
54s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}109m 49s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m
18s{color} | {color:green} hbase-mapreduce in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}180m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestSizeFailures |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-18294 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910610/HBASE-18294.01.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux e50ddc190aa4 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/L
[jira] [Commented] (HBASE-19953) Avoid calling post* hook when procedure fails
[
https://issues.apache.org/jira/browse/HBASE-19953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364707#comment-16364707
]
Josh Elser commented on HBASE-19953:
Mighty [~stack]: one more (basic?) question for you. I'm seeing some
discrepancies between CreateTableProcedure and DeleteNamespaceProcedure in how
'pre-checks' are handled. Specifically the difference between
{{CreateTableProcedure#prepareCreate(..)}} and
{{DeleteNamespaceProcedure#prepareDelete(..)}}. The former calls
{{setFailure(..)}} with exceptions that would prevent a table-creation from
happening (e.g. table already exists) while the latter just throws the
Exception.
Are these logically equivalent at a higher level? Or is this a sign that
DeleteNS needs to be updated?
> Avoid calling post* hook when procedure fails
> -
>
> Key: HBASE-19953
> URL: https://issues.apache.org/jira/browse/HBASE-19953
> Project: HBase
> Issue Type: Bug
> Components: master, proc-v2
>Reporter: Ramesh Mani
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-2
>
>
> Ramesh pointed out a case where I think we're mishandling some post\*
> MasterObserver hooks. Specifically, I'm looking at the deleteNamespace.
> We synchronously execute the DeleteNamespace procedure. When the user
> provides a namespace that isn't empty, the procedure does a rollback (which
> is just a no-op), but this doesn't propagate an exception up to the
> NonceProcedureRunnable in {{HMaster#deleteNamespace}}. It took Ramesh
> pointing it out a bit better to me that the code executes a bit differently
> than we actually expect.
> I think we need to double-check our post hooks and make sure we aren't
> invoking them when the procedure actually failed. cc/ [~Apache9], [~stack].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20000) Remove the quantum logic in FairQueue, always put high priority queue in front
[ https://issues.apache.org/jira/browse/HBASE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364717#comment-16364717 ] Appy commented on HBASE-2: -- Also, if this is not fixing a known bug, let's not push unwanted core framework changes into 2.0 beta. [~stack] if it's not much work, can we please make branch-2.0 sooner than beta2. This is n'th time i'm in same dilemma - Can't commit new stuff to beta2; and if commit to just 3.0 and forget, things won't make to 2.1. > Remove the quantum logic in FairQueue, always put high priority queue in front > -- > > Key: HBASE-2 > URL: https://issues.apache.org/jira/browse/HBASE-2 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19984) Add hadoop 2.8 and 2.9 to precommit
[ https://issues.apache.org/jira/browse/HBASE-19984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364751#comment-16364751 ] Appy commented on HBASE-19984: -- Just adding to precommit doesn't give us anything - except that things compile, which can always be checked one-off. Until we do the work of fully vetting a cluster against those versions, we can't call it supported. And if can't call them supported, there's no point of putting them in precommit. If it was just 2-3 min overhead, i might have been onboard, but with ~10-15 min overhead, doesn't seem something that should be done. > Add hadoop 2.8 and 2.9 to precommit > --- > > Key: HBASE-19984 > URL: https://issues.apache.org/jira/browse/HBASE-19984 > Project: HBase > Issue Type: Sub-task >Reporter: Mike Drob >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19991) lots of hbase-rest test failures against hadoop 3
[ https://issues.apache.org/jira/browse/HBASE-19991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-19991: -- Attachment: HBASE-19991.v1.patch > lots of hbase-rest test failures against hadoop 3 > - > > Key: HBASE-19991 > URL: https://issues.apache.org/jira/browse/HBASE-19991 > Project: HBase > Issue Type: Bug > Components: REST, test >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-19991.WIP.patch, HBASE-19991.v1.patch > > > mvn clean test -pl hbase-rest -Dhadoop.profile=3.0 > [ERROR] Tests run: 106, Failures: 95, Errors: 8, Skipped: 1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19991) lots of hbase-rest test failures against hadoop 3
[ https://issues.apache.org/jira/browse/HBASE-19991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-19991: -- Fix Version/s: (was: 2.0.0) 2.0.0-beta-2 Status: Patch Available (was: Open) > lots of hbase-rest test failures against hadoop 3 > - > > Key: HBASE-19991 > URL: https://issues.apache.org/jira/browse/HBASE-19991 > Project: HBase > Issue Type: Bug > Components: REST, test >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19991.WIP.patch, HBASE-19991.v1.patch > > > mvn clean test -pl hbase-rest -Dhadoop.profile=3.0 > [ERROR] Tests run: 106, Failures: 95, Errors: 8, Skipped: 1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19991) lots of hbase-rest test failures against hadoop 3
[ https://issues.apache.org/jira/browse/HBASE-19991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364799#comment-16364799 ] Mike Drob commented on HBASE-19991: --- careful debugging revealed a few more places where the jersey deps were leaking in. I also need to add a comment somewhere that upgrading to jersey 2.26 (from our current 2.25.1) will likely need an upgrade to jetty 9.4, so should be done with great care, as that was one of the things i tried here and it didn't work as well as I thought it would > lots of hbase-rest test failures against hadoop 3 > - > > Key: HBASE-19991 > URL: https://issues.apache.org/jira/browse/HBASE-19991 > Project: HBase > Issue Type: Bug > Components: REST, test >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19991.WIP.patch, HBASE-19991.v1.patch > > > mvn clean test -pl hbase-rest -Dhadoop.profile=3.0 > [ERROR] Tests run: 106, Failures: 95, Errors: 8, Skipped: 1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18282) ReplicationLogCleaner can delete WALs not yet replicated in case of a KeeperException
[
https://issues.apache.org/jira/browse/HBASE-18282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ben Lau updated HBASE-18282:
Attachment: HBASE-18282-branch-2.patch
HBASE-18282-branch-1.patch
> ReplicationLogCleaner can delete WALs not yet replicated in case of a
> KeeperException
> -
>
> Key: HBASE-18282
> URL: https://issues.apache.org/jira/browse/HBASE-18282
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.1, 1.2.6, 1.1.11, 2.0.0-alpha-1
>Reporter: Ashu Pachauri
>Priority: Critical
> Attachments: HBASE-18282-branch-1.patch, HBASE-18282-branch-2.patch
>
>
> ReplicationStateZKBase#getListOfReplicators does not rethrow a
> KeeperException and returns null in such a case. ReplicationLogCleaner just
> assumes that there are no replicators and deletes everything.
> ReplicationStateZKBase:
> {code:java}
> public List getListOfReplicators() {
> List result = null;
> try {
> result = ZKUtil.listChildrenNoWatch(this.zookeeper, this.queuesZNode);
> } catch (KeeperException e) {
> this.abortable.abort("Failed to get list of replicators", e);
> }
> return result;
> }
> {code}
> ReplicationLogCleaner:
> {code:java}
> private Set loadWALsFromQueues() throws KeeperException {
> for (int retry = 0; ; retry++) {
> int v0 = replicationQueues.getQueuesZNodeCversion();
> List rss = replicationQueues.getListOfReplicators();
> if (rss == null) {
> LOG.debug("Didn't find any region server that replicates, won't
> prevent any deletions.");
> return ImmutableSet.of();
> }
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18282) ReplicationLogCleaner can delete WALs not yet replicated in case of a KeeperException
[
https://issues.apache.org/jira/browse/HBASE-18282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364808#comment-16364808
]
Ted Yu commented on HBASE-18282:
{code}
+ public void setConf(Configuration conf, ZKWatcher zk,
ReplicationQueuesClient replicationQueuesClient) {
{code}
Wrap long line.
{code}
+ try {
+Thread.sleep(3 + ThreadLocalRandom.current().nextInt(3));
{code}
Can we use shorter sleep interval ?
{code}
+ } catch (InterruptedException e2) {
+LOG.warn("Interrupted while waiting for list of replicators to be
available", e2);
{code}
Restore interrupt status and break.
> ReplicationLogCleaner can delete WALs not yet replicated in case of a
> KeeperException
> -
>
> Key: HBASE-18282
> URL: https://issues.apache.org/jira/browse/HBASE-18282
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.1, 1.2.6, 1.1.11, 2.0.0-alpha-1
>Reporter: Ashu Pachauri
>Priority: Critical
> Attachments: HBASE-18282-branch-1.patch, HBASE-18282-branch-2.patch
>
>
> ReplicationStateZKBase#getListOfReplicators does not rethrow a
> KeeperException and returns null in such a case. ReplicationLogCleaner just
> assumes that there are no replicators and deletes everything.
> ReplicationStateZKBase:
> {code:java}
> public List getListOfReplicators() {
> List result = null;
> try {
> result = ZKUtil.listChildrenNoWatch(this.zookeeper, this.queuesZNode);
> } catch (KeeperException e) {
> this.abortable.abort("Failed to get list of replicators", e);
> }
> return result;
> }
> {code}
> ReplicationLogCleaner:
> {code:java}
> private Set loadWALsFromQueues() throws KeeperException {
> for (int retry = 0; ; retry++) {
> int v0 = replicationQueues.getQueuesZNodeCversion();
> List rss = replicationQueues.getListOfReplicators();
> if (rss == null) {
> LOG.debug("Didn't find any region server that replicates, won't
> prevent any deletions.");
> return ImmutableSet.of();
> }
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19988) HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while waiting for a row lock
[
https://issues.apache.org/jira/browse/HBASE-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364827#comment-16364827
]
Umesh Agashe commented on HBASE-19988:
--
Thanks [~appy] for digging into the code and posting your thoughts, comments.
{quote}Currently the comment says "// We will retry when other exceptions, but
we should stop if we timeout ."
Should be updated with reasons why we break out for each type.
{quote}
I agree, I have modified the comment and uploaded the new patch. Please see
patch 002.
However I prefer current exception handling over suggested as the general
guideline is to handle specific exception first and then generic exceptions. As
for the log message, getRowLockInternal() itself logs following warning message:
{code:java}
2018-02-10 04:24:25,503 WARN [PutThread] regionserver.HRegion(5636): Thread
interrupted waiting for lock on row: row0{code}
I think this message has details required for debugging without being too
chatty. Additionally, with run time type checking code will look like:
{code:java}
if (isAtomic() or ioe instanceof IIOE or ioe instanceof TIOE) { throw ioe;
}{code}
As you can see, readability will suffer, if we identify additional exception/
condition for retrying the operation in future.
> HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while
> waiting for a row lock
> ---
>
> Key: HBASE-19988
> URL: https://issues.apache.org/jira/browse/HBASE-19988
> Project: HBase
> Issue Type: Improvement
> Components: amv2
>Affects Versions: 2.0.0-beta-1
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
>Priority: Minor
> Fix For: 2.0.0-beta-2
>
> Attachments: hbase-19988.master.001.patch,
> hbase-19988.master.001.patch, hbase-19988.master.002.patch
>
>
> See HBASE-19970, TestHRegionWithInMemoryFlush created 4.2g log file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19988) HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while waiting for a row lock
[ https://issues.apache.org/jira/browse/HBASE-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Umesh Agashe updated HBASE-19988: - Attachment: hbase-19988.master.002.patch > HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while > waiting for a row lock > --- > > Key: HBASE-19988 > URL: https://issues.apache.org/jira/browse/HBASE-19988 > Project: HBase > Issue Type: Improvement > Components: amv2 >Affects Versions: 2.0.0-beta-1 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Minor > Fix For: 2.0.0-beta-2 > > Attachments: hbase-19988.master.001.patch, > hbase-19988.master.001.patch, hbase-19988.master.002.patch > > > See HBASE-19970, TestHRegionWithInMemoryFlush created 4.2g log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19995) Current Jetty 9 version in HBase master branch can memory leak under high traffic
[ https://issues.apache.org/jira/browse/HBASE-19995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Lau updated HBASE-19995: Attachment: HBASE-19995.patch > Current Jetty 9 version in HBase master branch can memory leak under high > traffic > - > > Key: HBASE-19995 > URL: https://issues.apache.org/jira/browse/HBASE-19995 > Project: HBase > Issue Type: Bug > Components: REST >Affects Versions: 2.0 >Reporter: Ben Lau >Priority: Major > Attachments: HBASE-19995.patch > > > There is a memory-leak in Jetty 9 that manifests whenever you hit the call > queue limit in HBase REST. The memory-leak leaks both on-heap and off-heap > objects permanently. It happens because whenever the call queue for Jetty > server overflows, the task that is rejected runs a 'reject' method if it is a > Rejectable to do any cleanup. This clean up is necessary to for example close > the connection, deallocate any buffers, etc. Unfortunately, in Jetty 9, they > implemented the 'reject' / cleanup method of the SelectChannelEndpoint as a > non-blocking call that is not guaranteed to run. This was later fixed in > Jetty 9.4 and later backported however the version of Jetty 9 pulled in HBase > for REST comes before this fix. See > [https://github.com/eclipse/jetty.project/issues/1804] and > [https://github.com/apache/hbase/blob/master/pom.xml#L1416.] > If we want to stay on 9.3.X we could update to > [9.3.22.v20171030|https://mvnrepository.com/artifact/org.eclipse.jetty/jetty-server/9.3.22.v20171030] > which is the latest version of 9.3. Thoughts? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19995) Current Jetty 9 version in HBase master branch can memory leak under high traffic
[ https://issues.apache.org/jira/browse/HBASE-19995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364831#comment-16364831 ] Ted Yu commented on HBASE-19995: +1 > Current Jetty 9 version in HBase master branch can memory leak under high > traffic > - > > Key: HBASE-19995 > URL: https://issues.apache.org/jira/browse/HBASE-19995 > Project: HBase > Issue Type: Bug > Components: REST >Affects Versions: 2.0 >Reporter: Ben Lau >Priority: Major > Attachments: HBASE-19995.patch > > > There is a memory-leak in Jetty 9 that manifests whenever you hit the call > queue limit in HBase REST. The memory-leak leaks both on-heap and off-heap > objects permanently. It happens because whenever the call queue for Jetty > server overflows, the task that is rejected runs a 'reject' method if it is a > Rejectable to do any cleanup. This clean up is necessary to for example close > the connection, deallocate any buffers, etc. Unfortunately, in Jetty 9, they > implemented the 'reject' / cleanup method of the SelectChannelEndpoint as a > non-blocking call that is not guaranteed to run. This was later fixed in > Jetty 9.4 and later backported however the version of Jetty 9 pulled in HBase > for REST comes before this fix. See > [https://github.com/eclipse/jetty.project/issues/1804] and > [https://github.com/apache/hbase/blob/master/pom.xml#L1416.] > If we want to stay on 9.3.X we could update to > [9.3.22.v20171030|https://mvnrepository.com/artifact/org.eclipse.jetty/jetty-server/9.3.22.v20171030] > which is the latest version of 9.3. Thoughts? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19991) lots of hbase-rest test failures against hadoop 3
[
https://issues.apache.org/jira/browse/HBASE-19991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364832#comment-16364832
]
Mike Drob commented on HBASE-19991:
---
These changes made tests pass but {{mvn package -Dhadoop.profile=3.0 &&
bin/hbase-daemon.sh start rest}} still failed
> lots of hbase-rest test failures against hadoop 3
> -
>
> Key: HBASE-19991
> URL: https://issues.apache.org/jira/browse/HBASE-19991
> Project: HBase
> Issue Type: Bug
> Components: REST, test
>Reporter: Mike Drob
>Assignee: Mike Drob
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19991.WIP.patch, HBASE-19991.v1.patch
>
>
> mvn clean test -pl hbase-rest -Dhadoop.profile=3.0
> [ERROR] Tests run: 106, Failures: 95, Errors: 8, Skipped: 1
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19116) Currently the tail of hfiles with CellComparator* classname makes it so hbase1 can't open hbase2 written hfiles; fix
[ https://issues.apache.org/jira/browse/HBASE-19116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364863#comment-16364863 ] Hudson commented on HBASE-19116: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4585 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4585/]) HBASE-19116 Currently the tail of hfiles with CellComparator* classname (stack: rev 8d26736bc2b0c28efd5caa3be7d8c9037dba633a) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestFixedFileTrailer.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java > Currently the tail of hfiles with CellComparator* classname makes it so > hbase1 can't open hbase2 written hfiles; fix > > > Key: HBASE-19116 > URL: https://issues.apache.org/jira/browse/HBASE-19116 > Project: HBase > Issue Type: Sub-task > Components: HFile, migration >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19116.branch-2.001.patch, > HBASE-19116.branch-2.002.patch, HBASE-19116.branch-2.003.patch, > HBASE-19116.branch-2.004.patch > > > See tail of HBASE-19052 for discussion which concludes we should try and make > it so operators do not have to go to latest hbase version before they > upgrade, at least if we can avoid it. > The necessary change of our default comparator from KV to Cell naming has > hfiles with tails that have the classname CellComparator in them in place of > KeyValueComparator. If an hbase1 tries to open them, it will fail not having > a CellComparator in its classpath (We have name of comparator in tail because > different files require different comparators... perhaps we write an alias > instead of a class one day... TODO). HBASE-16189 and HBASE-19052 are about > trying to carry knowledge of hbase2 back to hbase1, a brittle approach making > it so operators will have to upgrade to the latest branch-1 before they can > go to hbase2. > This issue is about undoing our writing of an incompatible (to hbase1) tail, > not unless we really have to (and it sounds like we could do without writing > an incompatible tail) to see if we can avoid requiring operators go to > lastest branch-1 (we may end up needing this but lets a have a really good > reason for it if we do). > Oh, let this filing be an answer to our [~anoop.hbase]'s old high-level > question over in HBASE-16189: > bq. ...means when rolling upgrade done to 2.0, first users have to upgrade to > some 1.x versions which is having this fix and then to 2.0.. What do you guys > think Whether we should avoid this kind of indirection? cc Enis Soztutar, > Stack, Ted Yu, Matteo Bertozzi > Yeah, lets try to avoid this if we can... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19991) lots of hbase-rest test failures against hadoop 3
[ https://issues.apache.org/jira/browse/HBASE-19991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364872#comment-16364872 ] Mike Drob commented on HBASE-19991: --- although starting the rest server outside of the test context fails for me before the patch as well, so I'd rather deal with that in a follow on > lots of hbase-rest test failures against hadoop 3 > - > > Key: HBASE-19991 > URL: https://issues.apache.org/jira/browse/HBASE-19991 > Project: HBase > Issue Type: Bug > Components: REST, test >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19991.WIP.patch, HBASE-19991.v1.patch > > > mvn clean test -pl hbase-rest -Dhadoop.profile=3.0 > [ERROR] Tests run: 106, Failures: 95, Errors: 8, Skipped: 1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19991) lots of hbase-rest test failures against hadoop 3
[ https://issues.apache.org/jira/browse/HBASE-19991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-19991: -- Attachment: HBASE-19991.v2.patch > lots of hbase-rest test failures against hadoop 3 > - > > Key: HBASE-19991 > URL: https://issues.apache.org/jira/browse/HBASE-19991 > Project: HBase > Issue Type: Bug > Components: REST, test >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19991.WIP.patch, HBASE-19991.v1.patch, > HBASE-19991.v2.patch > > > mvn clean test -pl hbase-rest -Dhadoop.profile=3.0 > [ERROR] Tests run: 106, Failures: 95, Errors: 8, Skipped: 1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19991) lots of hbase-rest test failures against hadoop 3
[ https://issues.apache.org/jira/browse/HBASE-19991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364874#comment-16364874 ] Mike Drob commented on HBASE-19991: --- v2: accidentally included a dep change that caused license issues. looks like we might not need it anyway. [~tedyu], [~elserj] - you lads run into any of this? care to review? > lots of hbase-rest test failures against hadoop 3 > - > > Key: HBASE-19991 > URL: https://issues.apache.org/jira/browse/HBASE-19991 > Project: HBase > Issue Type: Bug > Components: REST, test >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19991.WIP.patch, HBASE-19991.v1.patch, > HBASE-19991.v2.patch > > > mvn clean test -pl hbase-rest -Dhadoop.profile=3.0 > [ERROR] Tests run: 106, Failures: 95, Errors: 8, Skipped: 1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20000) Remove the quantum logic in FairQueue, always put high priority queue in front
[ https://issues.apache.org/jira/browse/HBASE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364885#comment-16364885 ] Duo Zhang commented on HBASE-2: --- Unfortunately this is a bug fix... See the parent issue... The quantum logic does not always work, we need to have strong priority. And second we need to reserve threads for high priority procedures. Your suggestions are good but are improvements, so let’s open new issues to land them. Thanks. > Remove the quantum logic in FairQueue, always put high priority queue in front > -- > > Key: HBASE-2 > URL: https://issues.apache.org/jira/browse/HBASE-2 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-5034) Remove non distributed log splitting option
[ https://issues.apache.org/jira/browse/HBASE-5034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364891#comment-16364891 ] Appy edited comment on HBASE-5034 at 2/14/18 10:52 PM: --- --We removed distributed log splitting. Closing this one.-- Stupid me, It was distributed log replay that was removed. was (Author: appy): We removed distributed log splitting. Closing this one. > Remove non distributed log splitting option > --- > > Key: HBASE-5034 > URL: https://issues.apache.org/jira/browse/HBASE-5034 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.94.0 >Reporter: Jonathan Hsieh >Priority: Major > Labels: beginner > > As a suggestion: > Distributed log splitting the default setting for recovery in 0.92. To > reduce the number of paths necessary for testing, we should remove the > original non-distributed version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-5034) Remove non distributed log splitting option
[ https://issues.apache.org/jira/browse/HBASE-5034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364891#comment-16364891 ] Appy commented on HBASE-5034: - We removed distributed log splitting. Closing this one. > Remove non distributed log splitting option > --- > > Key: HBASE-5034 > URL: https://issues.apache.org/jira/browse/HBASE-5034 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.94.0 >Reporter: Jonathan Hsieh >Priority: Major > Labels: beginner > > As a suggestion: > Distributed log splitting the default setting for recovery in 0.92. To > reduce the number of paths necessary for testing, we should remove the > original non-distributed version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19863) java.lang.IllegalStateException: isDelete failed when SingleColumnValueFilter is used
[
https://issues.apache.org/jira/browse/HBASE-19863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364898#comment-16364898
]
Chia-Ping Tsai commented on HBASE-19863:
The trouble is caused by that the current kv scanner in kv heap doesn’t do the
real seek. The KeyValueHeap#pollRealKV do the check through
KeyValueScanner#realSeekDone, hence it assumes the cell from current kv scanner
is located at or after the seek cell. StoreFileScanner doesn’t tell the true so
the cell returned by StoreFileScanner may be located before the seek key. v2
patch is fine to me. Although, it may be better to do the following change to
ensure kv heap always does the seek when updating current kv scanner.
{code:title=StoreFileScanner.java}
setCurrentCell(PrivateCellUtil.createLastOnRowCol(kv));
realSeekDone = true; // we should make it false
{code}
A side-effect, however, is that one more disk seek is generated when store
scanner fails to collect enough column-value cells, since kv heap will enforce
the last (lazy)file scanner to do the real seek…
> java.lang.IllegalStateException: isDelete failed when SingleColumnValueFilter
> is used
> -
>
> Key: HBASE-19863
> URL: https://issues.apache.org/jira/browse/HBASE-19863
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 1.4.1
>Reporter: Sergey Soldatov
>Assignee: Sergey Soldatov
>Priority: Major
> Attachments: HBASE-19863-branch-2.patch, HBASE-19863-branch1.patch,
> HBASE-19863-test.patch, HBASE-19863.v2-branch-2.patch
>
>
> Under some circumstances scan with SingleColumnValueFilter may fail with an
> exception
> {noformat}
> java.lang.IllegalStateException: isDelete failed: deleteBuffer=C3,
> qualifier=C2, timestamp=1516433595543, comparison result: 1
> at
> org.apache.hadoop.hbase.regionserver.ScanDeleteTracker.isDeleted(ScanDeleteTracker.java:149)
> at
> org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.match(ScanQueryMatcher.java:386)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:545)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:147)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5876)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6027)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5814)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2552)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32385)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
> {noformat}
> Conditions:
> table T with a single column family 0 that uses ROWCOL bloom filter
> (important) and column qualifiers C1,C2,C3,C4,C5.
> When we fill the table for every row we put deleted cell for C3.
> The table has a single region with two HStore:
> A: start row: 0, stop row: 99
> B: start row: 10 stop row: 99
> B has newer versions of rows 10-99. Store files have several blocks each
> (important).
> Store A is the result of major compaction, so it doesn't have any deleted
> cells (important).
> So, we are running a scan like:
> {noformat}
> scan 'T', { COLUMNS => ['0:C3','0:C5'], FILTER => "SingleColumnValueFilter
> ('0','C5',=,'binary:whatever')"}
> {noformat}
> How the scan performs:
> First, we iterate A for rows 0 and 1 without any problems.
> Next, we start to iterate A for row 10, so read the first cell and set hfs
> scanner to A :
> 10:0/C1/0/Put/x but found that we have a newer version of the cell in B :
> 10:0/C1/1/Put/x,
> so we make B as our current store scanner. Since we are looking for
> particular columns
> C3 and C5, we perform the optimization StoreScanner.seekOrSkipToNextColumn
> which
> would run reseek for all store scanners.
> For store A the following magic would happen in requestSeek:
> 1. bloom filter check passesGeneralBloomFilter would set haveToSeek to
> false because row 10 doesn't have C3 qualifier in store A.
> 2. Since we don't have to seek we just create a fake row
> 10:0/C3/OLDEST_TIMESTAMP/Maximum, an optimization that is quite important for
> us and it commented with :
> {noformat}
> // Multi-column Bloom filter optimization.
> // Create a fake key/value, so that this scanner only bu
[jira] [Updated] (HBASE-18282) ReplicationLogCleaner can delete WALs not yet replicated in case of a KeeperException
[
https://issues.apache.org/jira/browse/HBASE-18282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-18282:
---
Fix Version/s: 1.4.2
1.5.0
1.3.2
> ReplicationLogCleaner can delete WALs not yet replicated in case of a
> KeeperException
> -
>
> Key: HBASE-18282
> URL: https://issues.apache.org/jira/browse/HBASE-18282
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.1, 1.2.6, 1.1.11, 2.0.0-alpha-1
>Reporter: Ashu Pachauri
>Priority: Critical
> Fix For: 2.0.0, 1.3.2, 1.5.0, 1.4.2
>
> Attachments: HBASE-18282-branch-1.patch, HBASE-18282-branch-2.patch
>
>
> ReplicationStateZKBase#getListOfReplicators does not rethrow a
> KeeperException and returns null in such a case. ReplicationLogCleaner just
> assumes that there are no replicators and deletes everything.
> ReplicationStateZKBase:
> {code:java}
> public List getListOfReplicators() {
> List result = null;
> try {
> result = ZKUtil.listChildrenNoWatch(this.zookeeper, this.queuesZNode);
> } catch (KeeperException e) {
> this.abortable.abort("Failed to get list of replicators", e);
> }
> return result;
> }
> {code}
> ReplicationLogCleaner:
> {code:java}
> private Set loadWALsFromQueues() throws KeeperException {
> for (int retry = 0; ; retry++) {
> int v0 = replicationQueues.getQueuesZNodeCversion();
> List rss = replicationQueues.getListOfReplicators();
> if (rss == null) {
> LOG.debug("Didn't find any region server that replicates, won't
> prevent any deletions.");
> return ImmutableSet.of();
> }
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-18282) ReplicationLogCleaner can delete WALs not yet replicated in case of a KeeperException
[
https://issues.apache.org/jira/browse/HBASE-18282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-18282:
---
Fix Version/s: 2.0.0
> ReplicationLogCleaner can delete WALs not yet replicated in case of a
> KeeperException
> -
>
> Key: HBASE-18282
> URL: https://issues.apache.org/jira/browse/HBASE-18282
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.1, 1.2.6, 1.1.11, 2.0.0-alpha-1
>Reporter: Ashu Pachauri
>Priority: Critical
> Fix For: 2.0.0, 1.3.2, 1.5.0, 1.4.2
>
> Attachments: HBASE-18282-branch-1.patch, HBASE-18282-branch-2.patch
>
>
> ReplicationStateZKBase#getListOfReplicators does not rethrow a
> KeeperException and returns null in such a case. ReplicationLogCleaner just
> assumes that there are no replicators and deletes everything.
> ReplicationStateZKBase:
> {code:java}
> public List getListOfReplicators() {
> List result = null;
> try {
> result = ZKUtil.listChildrenNoWatch(this.zookeeper, this.queuesZNode);
> } catch (KeeperException e) {
> this.abortable.abort("Failed to get list of replicators", e);
> }
> return result;
> }
> {code}
> ReplicationLogCleaner:
> {code:java}
> private Set loadWALsFromQueues() throws KeeperException {
> for (int retry = 0; ; retry++) {
> int v0 = replicationQueues.getQueuesZNodeCversion();
> List rss = replicationQueues.getListOfReplicators();
> if (rss == null) {
> LOG.debug("Didn't find any region server that replicates, won't
> prevent any deletions.");
> return ImmutableSet.of();
> }
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18282) ReplicationLogCleaner can delete WALs not yet replicated in case of a KeeperException
[
https://issues.apache.org/jira/browse/HBASE-18282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364914#comment-16364914
]
Andrew Purtell commented on HBASE-18282:
After addressing above comments, happy to commit. Holding 1.4.2 until we can
get this in.
> ReplicationLogCleaner can delete WALs not yet replicated in case of a
> KeeperException
> -
>
> Key: HBASE-18282
> URL: https://issues.apache.org/jira/browse/HBASE-18282
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.1, 1.2.6, 1.1.11, 2.0.0-alpha-1
>Reporter: Ashu Pachauri
>Assignee: Ben Lau
>Priority: Critical
> Fix For: 2.0.0, 1.3.2, 1.5.0, 1.4.2
>
> Attachments: HBASE-18282-branch-1.patch, HBASE-18282-branch-2.patch
>
>
> ReplicationStateZKBase#getListOfReplicators does not rethrow a
> KeeperException and returns null in such a case. ReplicationLogCleaner just
> assumes that there are no replicators and deletes everything.
> ReplicationStateZKBase:
> {code:java}
> public List getListOfReplicators() {
> List result = null;
> try {
> result = ZKUtil.listChildrenNoWatch(this.zookeeper, this.queuesZNode);
> } catch (KeeperException e) {
> this.abortable.abort("Failed to get list of replicators", e);
> }
> return result;
> }
> {code}
> ReplicationLogCleaner:
> {code:java}
> private Set loadWALsFromQueues() throws KeeperException {
> for (int retry = 0; ; retry++) {
> int v0 = replicationQueues.getQueuesZNodeCversion();
> List rss = replicationQueues.getListOfReplicators();
> if (rss == null) {
> LOG.debug("Didn't find any region server that replicates, won't
> prevent any deletions.");
> return ImmutableSet.of();
> }
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-18282) ReplicationLogCleaner can delete WALs not yet replicated in case of a KeeperException
[
https://issues.apache.org/jira/browse/HBASE-18282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell reassigned HBASE-18282:
--
Assignee: Ben Lau
> ReplicationLogCleaner can delete WALs not yet replicated in case of a
> KeeperException
> -
>
> Key: HBASE-18282
> URL: https://issues.apache.org/jira/browse/HBASE-18282
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.1, 1.2.6, 1.1.11, 2.0.0-alpha-1
>Reporter: Ashu Pachauri
>Assignee: Ben Lau
>Priority: Critical
> Fix For: 2.0.0, 1.3.2, 1.5.0, 1.4.2
>
> Attachments: HBASE-18282-branch-1.patch, HBASE-18282-branch-2.patch
>
>
> ReplicationStateZKBase#getListOfReplicators does not rethrow a
> KeeperException and returns null in such a case. ReplicationLogCleaner just
> assumes that there are no replicators and deletes everything.
> ReplicationStateZKBase:
> {code:java}
> public List getListOfReplicators() {
> List result = null;
> try {
> result = ZKUtil.listChildrenNoWatch(this.zookeeper, this.queuesZNode);
> } catch (KeeperException e) {
> this.abortable.abort("Failed to get list of replicators", e);
> }
> return result;
> }
> {code}
> ReplicationLogCleaner:
> {code:java}
> private Set loadWALsFromQueues() throws KeeperException {
> for (int retry = 0; ; retry++) {
> int v0 = replicationQueues.getQueuesZNodeCversion();
> List rss = replicationQueues.getListOfReplicators();
> if (rss == null) {
> LOG.debug("Didn't find any region server that replicates, won't
> prevent any deletions.");
> return ImmutableSet.of();
> }
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19985) Redundant instanceof check in ProtobufUtil#getServiceException
[
https://issues.apache.org/jira/browse/HBASE-19985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-19985:
---
Fix Version/s: (was: 1.4.2)
1.4.3
> Redundant instanceof check in ProtobufUtil#getServiceException
> --
>
> Key: HBASE-19985
> URL: https://issues.apache.org/jira/browse/HBASE-19985
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.1
>Reporter: Ted Yu
>Priority: Trivial
> Fix For: 1.5.0, 1.4.3
>
>
> {code}
> public static IOException getServiceException(ServiceException e) {
> Throwable t = e;
> if (e instanceof ServiceException) {
> t = e.getCause();
> {code}
> The instanceof check would always return true.
> This was reported by
> https://builds.apache.org/job/PreCommit-HBASE-Build/11495/artifact/patchprocess/branch-findbugs-hbase-client-warnings.html#Warnings_STYLE
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-18639) nightly jobs that need to do jdk7 + jdk8 need more than 6 hours
[ https://issues.apache.org/jira/browse/HBASE-18639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18639: --- Fix Version/s: (was: 1.4.2) 1.4.3 > nightly jobs that need to do jdk7 + jdk8 need more than 6 hours > --- > > Key: HBASE-18639 > URL: https://issues.apache.org/jira/browse/HBASE-18639 > Project: HBase > Issue Type: Task > Components: test >Reporter: Sean Busbey >Priority: Major > Labels: beginner > Fix For: 1.3.2, 1.5.0, 1.2.8, 1.4.3 > > > looks like the branches that need to do 2 jdks will need more than 6 hours, > at least until we can parallelize things. > e.g. > * > [branch-1.2|https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/job/branch-1.2/buildTimeTrend] > * > [branch-1.3|https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/job/branch-1.3/buildTimeTrend] > branch-1.4 and branch-1 are failing in jdk7 checks, so they don't go that > long yet, but once we get those tests fixed presumably they'll need the time > as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19444) RSGroups test units cannot be concurrently executed
[ https://issues.apache.org/jira/browse/HBASE-19444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-19444: --- Fix Version/s: (was: 1.4.2) 1.4.3 > RSGroups test units cannot be concurrently executed > --- > > Key: HBASE-19444 > URL: https://issues.apache.org/jira/browse/HBASE-19444 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.0 >Reporter: Andrew Purtell >Priority: Minor > Fix For: 2.0.0, 1.5.0, 1.4.3 > > > TestRSGroups and friends cannot be concurrently executed or they are very > likely to flake, failing with constraint exceptions. If executed serially all > units pass. Fix for concurrent execution. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18985) Fix the building warning of missing version in hbase-shaded-check-invariants module
[
https://issues.apache.org/jira/browse/HBASE-18985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-18985:
---
Fix Version/s: (was: 1.4.2)
1.4.3
> Fix the building warning of missing version in hbase-shaded-check-invariants
> module
> ---
>
> Key: HBASE-18985
> URL: https://issues.apache.org/jira/browse/HBASE-18985
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Priority: Minor
> Labels: beginner
> Fix For: 1.3.2, 1.5.0, 1.2.8, 1.4.3
>
>
> {quote}
> 19:20:38 [WARNING]
> 19:20:38 [WARNING] Some problems were encountered while building the
> effective model for
> org.apache.hbase:hbase-shaded-check-invariants:pom:1.2.7-SNAPSHOT
> 19:20:38 [WARNING] 'build.plugins.plugin.version' for
> org.codehaus.mojo:exec-maven-plugin is missing. @
> org.apache.hbase:hbase-shaded-check-invariants:[unknown-version],
> /var/lib/jenkins/workspace/PreCommit-HBASE-BRANCH-1.2/hbase-shaded/hbase-shaded-check-invariants/pom.xml,
> line 161, column 15
> 19:20:38 [WARNING]
> 19:20:38 [WARNING] It is highly recommended to fix these problems because
> they threaten the stability of your build.
> 19:20:38 [WARNING]
> 19:20:38 [WARNING] For this reason, future Maven versions might no longer
> support building such malformed projects.
> 19:20:38 [WARNING]
> {quote}
> This warning doesn't occur on master because it had been fixed by HBASE-18723.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19663) site build fails complaining "javadoc: error - class file for javax.annotation.meta.TypeQualifierNickname not found"
[
https://issues.apache.org/jira/browse/HBASE-19663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-19663:
---
Fix Version/s: (was: 1.4.2)
> site build fails complaining "javadoc: error - class file for
> javax.annotation.meta.TypeQualifierNickname not found"
>
>
> Key: HBASE-19663
> URL: https://issues.apache.org/jira/browse/HBASE-19663
> Project: HBase
> Issue Type: Bug
> Components: site
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0, 1.5.0
>
> Attachments: script.sh
>
>
> Cryptic failure trying to build beta-1 RC. Fails like this:
> {code}
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 03:54 min
> [INFO] Finished at: 2017-12-29T01:13:15-08:00
> [INFO] Final Memory: 381M/9165M
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-site-plugin:3.4:site (default-site) on project
> hbase: Error generating maven-javadoc-plugin:2.10.3:aggregate:
> [ERROR] Exit code: 1 - warning: unknown enum constant When.ALWAYS
> [ERROR] reason: class file for javax.annotation.meta.When not found
> [ERROR] warning: unknown enum constant When.UNKNOWN
> [ERROR] warning: unknown enum constant When.MAYBE
> [ERROR]
> /home/stack/hbase.git/hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java:762:
> warning - Tag @link: malformed: "#matchingRows(Cell, byte[]))"
> [ERROR]
> /home/stack/hbase.git/hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java:762:
> warning - Tag @link: reference not found: #matchingRows(Cell, byte[]))
> [ERROR]
> /home/stack/hbase.git/hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java:762:
> warning - Tag @link: reference not found: #matchingRows(Cell, byte[]))
> [ERROR] javadoc: warning - Class javax.annotation.Nonnull not found.
> [ERROR] javadoc: error - class file for
> javax.annotation.meta.TypeQualifierNickname not found
> [ERROR]
> [ERROR] Command line was: /home/stack/bin/jdk1.8.0_151/jre/../bin/javadoc
> -J-Xmx2G @options @packages
> [ERROR]
> [ERROR] Refer to the generated Javadoc files in
> '/home/stack/hbase.git/target/site/apidocs' dir.
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> {code}
> javax.annotation.meta.TypeQualifierNickname is out of jsr305 but we don't
> include this anywhere according to mvn dependency.
> Happens building the User API both test and main.
> Excluding these lines gets us passing again:
> {code}
> 3511
> 3512
> org.apache.yetus.audience.tools.IncludePublicAnnotationsStandardDoclet
> 3513
> 3514
> 3515 org.apache.yetus
> 3516 audience-annotations
> 3517 ${audience-annotations.version}
> 3518
> + 3519 true
> {code}
> Tried upgrading to newer mvn site (ours is three years old) but that a
> different set of problems.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-18549) Unclaimed replication queues can go undetected
[ https://issues.apache.org/jira/browse/HBASE-18549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18549: --- Fix Version/s: (was: 1.4.2) 1.4.3 > Unclaimed replication queues can go undetected > -- > > Key: HBASE-18549 > URL: https://issues.apache.org/jira/browse/HBASE-18549 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: Ashu Pachauri >Priority: Critical > Fix For: 1.3.2, 1.5.0, 1.4.3 > > > We have come across this situation multiple times where a zookeeper issues > can cause NodeFailoverWorker to fail picking up replication queue for a dead > region server silently. One example is when the znode size for a particular > queue exceed jute.maxBuffer value. > There can be other situations that may lead to this and just go undetected. > We need to have a metric for number of unclaimed replication queues. This > will help in mitigating the problem through alerting on the metric and > identifying underlying issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18282) ReplicationLogCleaner can delete WALs not yet replicated in case of a KeeperException
[
https://issues.apache.org/jira/browse/HBASE-18282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-18282:
---
Status: Patch Available (was: Open)
> ReplicationLogCleaner can delete WALs not yet replicated in case of a
> KeeperException
> -
>
> Key: HBASE-18282
> URL: https://issues.apache.org/jira/browse/HBASE-18282
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0-alpha-1, 1.1.11, 1.2.6, 1.3.1
>Reporter: Ashu Pachauri
>Assignee: Ben Lau
>Priority: Critical
> Fix For: 2.0.0, 1.3.2, 1.5.0, 1.4.2
>
> Attachments: HBASE-18282-branch-1.patch, HBASE-18282-branch-2.patch
>
>
> ReplicationStateZKBase#getListOfReplicators does not rethrow a
> KeeperException and returns null in such a case. ReplicationLogCleaner just
> assumes that there are no replicators and deletes everything.
> ReplicationStateZKBase:
> {code:java}
> public List getListOfReplicators() {
> List result = null;
> try {
> result = ZKUtil.listChildrenNoWatch(this.zookeeper, this.queuesZNode);
> } catch (KeeperException e) {
> this.abortable.abort("Failed to get list of replicators", e);
> }
> return result;
> }
> {code}
> ReplicationLogCleaner:
> {code:java}
> private Set loadWALsFromQueues() throws KeeperException {
> for (int retry = 0; ; retry++) {
> int v0 = replicationQueues.getQueuesZNodeCversion();
> List rss = replicationQueues.getListOfReplicators();
> if (rss == null) {
> LOG.debug("Didn't find any region server that replicates, won't
> prevent any deletions.");
> return ImmutableSet.of();
> }
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-18415) The local timeout may cause Admin to submit duplicate request
[ https://issues.apache.org/jira/browse/HBASE-18415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18415: --- Fix Version/s: (was: 1.4.2) (was: 1.2.8) 1.4.3 Status: Open (was: Patch Available) No update since sept 2017, cancelling patch > The local timeout may cause Admin to submit duplicate request > - > > Key: HBASE-18415 > URL: https://issues.apache.org/jira/browse/HBASE-18415 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Major > Fix For: 2.0.0, 1.3.2, 1.5.0, 1.4.3 > > Attachments: HBASE-18415.branch-1.ut.patch, > HBASE-18415.branch-1.v0.patch, HBASE-18415.branch-1.v1.patch, > HBASE-18415.branch-1.v2.patch, HBASE-18415.branch-1.v3.patch, > HBASE-18415.branch-1.v3.patch, HBASE-18415.branch-1.v3.patch, > HBASE-18415.branch-1.v4.patch, HBASE-18415.branch-1.v4.patch, > HBASE-18415.branch-1.v4.patch > > > After a timeout occurs on first request, client will retry the request with > distinct group/nonce. The second request may bring the TableXXXException back > if the first request have changed the table state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18228) HBCK improvements
[ https://issues.apache.org/jira/browse/HBASE-18228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18228: --- Fix Version/s: (was: 1.4.2) 1.4.3 > HBCK improvements > - > > Key: HBASE-18228 > URL: https://issues.apache.org/jira/browse/HBASE-18228 > Project: HBase > Issue Type: Improvement > Components: hbck >Reporter: Lars Hofhansl >Assignee: Karan Mehta >Priority: Critical > Fix For: 2.0.0, 3.0.0, 1.5.0, 1.4.3 > > Attachments: HBASE-18228.branch-1.3.patch > > > We just had a prod issue and running HBCK the way we did actually causes more > problems. > In part HBCK did stuff we did not expect, in part we had little visibility > into what HBCK was doing, and in part the logging was confusing. > I'm proposing 2 improvements: > 1. A dry-run mode. Run, and just list what would have been done. > 2. An interactive mode. Run, and for each action request Y/N user input. So > that a user can opt-out of stuff. > [~jmhsieh], FYI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18182) Setting HBASE_REGIONSERVER_OPTS with UseG1GC will cause regionserver start error
[
https://issues.apache.org/jira/browse/HBASE-18182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-18182:
---
Fix Version/s: (was: 1.4.2)
1.4.3
> Setting HBASE_REGIONSERVER_OPTS with UseG1GC will cause regionserver start
> error
>
>
> Key: HBASE-18182
> URL: https://issues.apache.org/jira/browse/HBASE-18182
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 2.0.0, 1.1.6, 1.3.1, 1.2.6
>Reporter: Fangyuan Deng
>Assignee: Fangyuan Deng
>Priority: Major
> Fix For: 2.1.0, 1.3.2, 1.2.8, 1.4.3
>
> Attachments: HBASE-18182.0.patch
>
>
> when we set in hbase-env.sh to use G1GC
> HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:+UseG1GC -Xms96g
> -Xmx96g -XX:MaxGCPauseMillis=100"
> then run ./hbase-daemon.sh start regionserver
> Conflicting collector combinations in option list; please refer to the
> release notes for the combinations allowed
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> because the HBASE_OPTS default use CMS and is conflicted with
> HBASE_REGIONSERVER_OPTS
> so I add a patch to modify bin/hbase like this
> elif [ "$COMMAND" = "regionserver" ] ; then
> CLASS='org.apache.hadoop.hbase.regionserver.HRegionServer'
> if [ "$1" != "stop" ] ; then
> HBASE_OPTS="$HBASE_OPTS $HBASE_REGIONSERVER_OPTS"
> if [[ $HBASE_REGIONSERVER_OPTS == *UseG1GC* ]] ; then
> HBASE_OPTS=${HBASE_OPTS/'-XX:+UseConcMarkSweepGC'/''}
> fi
> fi
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-18182) Setting HBASE_REGIONSERVER_OPTS with UseG1GC will cause regionserver start error
[
https://issues.apache.org/jira/browse/HBASE-18182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-18182:
---
Fix Version/s: (was: 2.1.0)
1.5.0
2.0.0
Status: Open (was: Patch Available)
[~busbey]'s comment remains unaddressed, cancelling patch
> Setting HBASE_REGIONSERVER_OPTS with UseG1GC will cause regionserver start
> error
>
>
> Key: HBASE-18182
> URL: https://issues.apache.org/jira/browse/HBASE-18182
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.2.6, 1.3.1, 1.1.6, 2.0.0
>Reporter: Fangyuan Deng
>Assignee: Fangyuan Deng
>Priority: Major
> Fix For: 2.0.0, 1.3.2, 1.5.0, 1.2.8, 1.4.3
>
> Attachments: HBASE-18182.0.patch
>
>
> when we set in hbase-env.sh to use G1GC
> HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:+UseG1GC -Xms96g
> -Xmx96g -XX:MaxGCPauseMillis=100"
> then run ./hbase-daemon.sh start regionserver
> Conflicting collector combinations in option list; please refer to the
> release notes for the combinations allowed
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> because the HBASE_OPTS default use CMS and is conflicted with
> HBASE_REGIONSERVER_OPTS
> so I add a patch to modify bin/hbase like this
> elif [ "$COMMAND" = "regionserver" ] ; then
> CLASS='org.apache.hadoop.hbase.regionserver.HRegionServer'
> if [ "$1" != "stop" ] ; then
> HBASE_OPTS="$HBASE_OPTS $HBASE_REGIONSERVER_OPTS"
> if [[ $HBASE_REGIONSERVER_OPTS == *UseG1GC* ]] ; then
> HBASE_OPTS=${HBASE_OPTS/'-XX:+UseConcMarkSweepGC'/''}
> fi
> fi
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-17960) IntegrationTestReplication fails in successive runs due to lack of appropriate cleanup
[ https://issues.apache.org/jira/browse/HBASE-17960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-17960: --- Fix Version/s: (was: 1.4.2) 1.4.3 > IntegrationTestReplication fails in successive runs due to lack of > appropriate cleanup > -- > > Key: HBASE-17960 > URL: https://issues.apache.org/jira/browse/HBASE-17960 > Project: HBase > Issue Type: Bug > Components: integration tests >Reporter: Ashu Pachauri >Priority: Major > Fix For: 2.0.0, 3.0.0, 1.3.2, 1.5.0, 1.4.3 > > > The way ITR works right now is that it adds a peer named 'TestPeer' for the > replication destination cluster. The name of the peer is same across runs. > Also, it removes the peer in the beginning of each run. However, it does not > wait for the queues corresponding to the peer to get cleaned up (which is an > asynchronous operation and can take 10s of seconds). This causes the next run > to fail and so on. > The test setup should wait for a non-trivial amount of time to cleanup the > queues corresponding to the peer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-17960) IntegrationTestReplication fails in successive runs due to lack of appropriate cleanup
[ https://issues.apache.org/jira/browse/HBASE-17960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell reassigned HBASE-17960: -- Assignee: (was: Ashu Pachauri) > IntegrationTestReplication fails in successive runs due to lack of > appropriate cleanup > -- > > Key: HBASE-17960 > URL: https://issues.apache.org/jira/browse/HBASE-17960 > Project: HBase > Issue Type: Bug > Components: integration tests >Reporter: Ashu Pachauri >Priority: Major > Fix For: 2.0.0, 3.0.0, 1.3.2, 1.5.0, 1.4.3 > > > The way ITR works right now is that it adds a peer named 'TestPeer' for the > replication destination cluster. The name of the peer is same across runs. > Also, it removes the peer in the beginning of each run. However, it does not > wait for the queues corresponding to the peer to get cleaned up (which is an > asynchronous operation and can take 10s of seconds). This causes the next run > to fail and so on. > The test setup should wait for a non-trivial amount of time to cleanup the > queues corresponding to the peer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-17890) FuzzyRowFilter fail if unaligned support is false
[
https://issues.apache.org/jira/browse/HBASE-17890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-17890:
---
Fix Version/s: (was: 1.4.2)
1.4.3
> FuzzyRowFilter fail if unaligned support is false
> -
>
> Key: HBASE-17890
> URL: https://issues.apache.org/jira/browse/HBASE-17890
> Project: HBase
> Issue Type: Sub-task
> Components: util
>Affects Versions: 2.0.0, 1.2.5
>Reporter: Jerry He
>Assignee: Chia-Ping Tsai
>Priority: Major
> Fix For: 2.0.0, 3.0.0, 1.3.2, 1.2.8, 1.4.3
>
> Attachments: HBASE-17890.v0.branch-1.patch, HBASE-17890.v0.patch,
> HBASE-17890.v1.branch-1.patch, HBASE-17890.v1.patch, HBASE-17890.v2.patch,
> HBASE-17890.v3.patch, HBASE-17890.v3.patch, HBASE-17890.v3.patch,
> HBASE-17890.v3.patch, HBASE-17890.v3.patch
>
>
> When unaligned support is false, FuzzyRow tests fail:
> {noformat}
> Failed tests:
> TestFuzzyRowAndColumnRangeFilter.Test:134->runTest:157->runScanner:186
> expected:<10> but was:<0>
> TestFuzzyRowFilter.testSatisfiesForward:81 expected: but was:
> TestFuzzyRowFilter.testSatisfiesReverse:121 expected: but
> was:
> TestFuzzyRowFilterEndToEnd.testEndToEnd:247->runTest1:278->runScanner:343
> expected:<6250> but was:<0>
> TestFuzzyRowFilterEndToEnd.testFilterList:385->runTest:417->runScanner:445
> expected:<5> but was:<0>
> TestFuzzyRowFilterEndToEnd.testHBASE14782:204 expected:<6> but was:<0>
> {noformat}
> This can be reproduced in the case described in HBASE-17869. Or on a platform
> really without unaligned support.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
