[jira] [Commented] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362160#comment-16362160 ] Eshcar Hillel commented on HBASE-18294: --- I have updated the release notes, please see if this addresses your concern #1. Regarding concern #2: my intention across the patch was to have symmetric decision making for on-heap and off-heap cases. Please let me know if you find any inconsistency in the code so I can fix it. (code is in RB) > Reduce global heap pressure: flush based on heap occupancy > -- > > Key: HBASE-18294 > URL: https://issues.apache.org/jira/browse/HBASE-18294 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.01.patch, HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.01.patch, HBASE-18294.01.patch, HBASE-18294.02.patch, > HBASE-18294.03.patch, HBASE-18294.04.patch, HBASE-18294.05.patch, > HBASE-18294.06.patch, HBASE-18294.07.patch, HBASE-18294.07.patch, > HBASE-18294.08.patch, HBASE-18294.09.patch, HBASE-18294.10.patch, > HBASE-18294.11.patch, HBASE-18294.11.patch, HBASE-18294.12.patch, > HBASE-18294.13.patch, HBASE-18294.15.patch, HBASE-18294.16.patch, > HBASE-18294.master.01.patch, HBASE-18294.master.01.patch > > > A region is flushed if its memory component exceed a threshold (default size > is 128MB). > A flush policy decides whether to flush a store by comparing the size of the > store to another threshold (that can be configured with > hbase.hregion.percolumnfamilyflush.size.lower.bound). > Currently the implementation (in both cases) compares the data size > (key-value only) to the threshold where it should compare the heap size > (which includes index size, and metadata). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19971) TestMajorCompaction#testDataBlockEncodingInCacheOnly sometimes fails with IllegalArgumentException
[
https://issues.apache.org/jira/browse/HBASE-19971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362202#comment-16362202
]
ramkrishna.s.vasudevan commented on HBASE-19971:
The stack trace seems fine to me.
{code}
createStoreFile(r);
for (int i = 0; i < compactionThreshold; i++) {
createStoreFile(r);
}
// Add more content.
HBaseTestCase.addContent(new RegionAsTable(r),
Bytes.toString(COLUMN_FAMILY));
{code}
We are adding some data and inturn it is creating some in memory flushes and
that is where this problem is. IMHO we need to investigate to see if there is
any real issue or a test case issue only.
> TestMajorCompaction#testDataBlockEncodingInCacheOnly sometimes fails with
> IllegalArgumentException
> --
>
> Key: HBASE-19971
> URL: https://issues.apache.org/jira/browse/HBASE-19971
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Major
> Attachments: 19971.v1.txt, testDataBlockEncodingInCacheOnly.out
>
>
> From
> https://builds.apache.org/job/HBASE-Flaky-Tests/25784/testReport/junit/org.apache.hadoop.hbase.regionserver/TestMajorCompaction/testDataBlockEncodingInCacheOnly_2_/
> :
> {code}
> java.io.IOException: java.lang.IllegalArgumentException: In CellChunkMap,
> cell must be associated with chunk.. We were looking for a cell at index 0
> at
> org.apache.hadoop.hbase.regionserver.TestMajorCompaction.majorCompaction(TestMajorCompaction.java:206)
> at
> org.apache.hadoop.hbase.regionserver.TestMajorCompaction.majorCompactionWithDataBlockEncoding(TestMajorCompaction.java:186)
> at
> org.apache.hadoop.hbase.regionserver.TestMajorCompaction.testDataBlockEncodingInCacheOnly(TestMajorCompaction.java:166)
> Caused by: java.lang.IllegalArgumentException: In CellChunkMap, cell must be
> associated with chunk.. We were looking for a cell at index 0
> at
> org.apache.hadoop.hbase.regionserver.TestMajorCompaction.majorCompaction(TestMajorCompaction.java:206)
> at
> org.apache.hadoop.hbase.regionserver.TestMajorCompaction.majorCompactionWithDataBlockEncoding(TestMajorCompaction.java:186)
> at
> org.apache.hadoop.hbase.regionserver.TestMajorCompaction.testDataBlockEncodingInCacheOnly(TestMajorCompaction.java:166)
> {code}
> From the index of the test, EAGER policy was used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19979) ReplicationSyncUp tool may leak Zookeeper connection
[
https://issues.apache.org/jira/browse/HBASE-19979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362262#comment-16362262
]
Pankaj Kumar commented on HBASE-19979:
--
Findbug error is not relevant.
> ReplicationSyncUp tool may leak Zookeeper connection
>
>
> Key: HBASE-19979
> URL: https://issues.apache.org/jira/browse/HBASE-19979
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 1.5.0, 2.0.0-beta-2
>
> Attachments: HBASE-19979-branch-1.001.patch, HBASE-19979.patch
>
>
> ReplicationSyncUp tool may leak Zookeeper connection in the following code
> snippet,
> {code}
> try {
> int numberOfOldSource = 1; // default wait once
> while (numberOfOldSource > 0) {
> Thread.sleep(SLEEP_TIME);
> numberOfOldSource = manager.getOldSources().size();
> }
> } catch (InterruptedException e) {
> System.err.println("didn't wait long enough:" + e);
> return (-1);
> }
> manager.join();
> zkw.close();
> {code}
> ZooKeeperWatcher will not be closed in case of InterruptedException.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19876) The exception happening in converting pb mutation to hbase.mutation messes up the CellScanner
[
https://issues.apache.org/jira/browse/HBASE-19876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-19876:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Push this patch to all active branches. Thanks for all reviews.
> The exception happening in converting pb mutation to hbase.mutation messes up
> the CellScanner
> -
>
> Key: HBASE-19876
> URL: https://issues.apache.org/jira/browse/HBASE-19876
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 1.3.2, 1.5.0, 1.2.7, 2.0.0-beta-2, 1.4.2
>
> Attachments: HBASE-19876.branch-1.2.v0.patch,
> HBASE-19876.master.001.patch, HBASE-19876.v0.patch, HBASE-19876.v1.patch,
> HBASE-19876.v2.patch, HBASE-19876.v3.patch, HBASE-19876.v3.patch,
> HBASE-19876.v3.patch, HBASE-19876.v3.patch, HBASE-19876.v4.patch,
> HBASE-19876.v5.patch, HBASE-19876.v6.patch
>
>
> {code:java}
> 2018-01-27 22:51:43,794 INFO [hconnection-0x3291b443-shared-pool11-t6]
> client.AsyncRequestFutureImpl(778): id=5, table=testQuotaStatusFromMaster3,
> attempt=6/16 failed=20ops, last
> exception=org.apache.hadoop.hbase.client.WrongRowIOException:
> org.apache.hadoop.hbase.client.WrongRowIOException: The row in xxx doesn't
> match the original one aaa
> at org.apache.hadoop.hbase.client.Mutation.add(Mutation.java:776)
> at org.apache.hadoop.hbase.client.Put.add(Put.java:282)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toPut(ProtobufUtil.java:642)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:952)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:896)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2591)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304){code}
> I noticed this bug when testing the table space quota.
> When rs are converting pb mutation to hbase.mutation, the quota exception or
> cell exception may be thrown.
> {code}
> Unable to find source-code formatter for language:
> rsrpcservices#dobatchop.java. Available languages are: actionscript, ada,
> applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html,
> java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r,
> rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml for
> (ClientProtos.Action action: mutations) {
> MutationProto m = action.getMutation();
> Mutation mutation;
> if (m.getMutateType() == MutationType.PUT) {
> mutation = ProtobufUtil.toPut(m, cells);
> batchContainsPuts = true;
> } else {
> mutation = ProtobufUtil.toDelete(m, cells);
> batchContainsDelete = true;
> }
> mutationActionMap.put(mutation, action);
> mArray[i++] = mutation;
> checkCellSizeLimit(region, mutation);
> // Check if a space quota disallows this mutation
> spaceQuotaEnforcement.getPolicyEnforcement(region).check(mutation);
> quota.addMutation(mutation);
> }
> {code}
> rs has caught the exception but it doesn't have the cellscanner skip the
> failed cells.
> {code:java}
> } catch (IOException ie) {
> if (atomic) {
> throw ie;
> }
> for (Action mutation : mutations) {
> builder.addResultOrException(getResultOrException(ie,
> mutation.getIndex()));
> }
> }
> {code}
> The bug results in the WrongRowIOException to remaining mutations since they
> refer to invalid cells.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19990) Create remote wal directory when transitting to state S
[ https://issues.apache.org/jira/browse/HBASE-19990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362230#comment-16362230 ] Duo Zhang commented on HBASE-19990: --- [~zghaobac] FYI. > Create remote wal directory when transitting to state S > --- > > Key: HBASE-19990 > URL: https://issues.apache.org/jira/browse/HBASE-19990 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-19990-HBASE-19064.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19979) ReplicationSyncUp tool may leak Zookeeper connection
[
https://issues.apache.org/jira/browse/HBASE-19979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pankaj Kumar updated HBASE-19979:
-
Fix Version/s: 2.0.0-beta-2
1.5.0
> ReplicationSyncUp tool may leak Zookeeper connection
>
>
> Key: HBASE-19979
> URL: https://issues.apache.org/jira/browse/HBASE-19979
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 1.5.0, 2.0.0-beta-2
>
> Attachments: HBASE-19979-branch-1.001.patch, HBASE-19979.patch
>
>
> ReplicationSyncUp tool may leak Zookeeper connection in the following code
> snippet,
> {code}
> try {
> int numberOfOldSource = 1; // default wait once
> while (numberOfOldSource > 0) {
> Thread.sleep(SLEEP_TIME);
> numberOfOldSource = manager.getOldSources().size();
> }
> } catch (InterruptedException e) {
> System.err.println("didn't wait long enough:" + e);
> return (-1);
> }
> manager.join();
> zkw.close();
> {code}
> ZooKeeperWatcher will not be closed in case of InterruptedException.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19990) Create remote wal directory when transitting to state S
[ https://issues.apache.org/jira/browse/HBASE-19990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362336#comment-16362336 ] Guanghao Zhang commented on HBASE-19990: We have a config for sync replication enabled. So no need to create remoteWALs dir when the config is false? > Create remote wal directory when transitting to state S > --- > > Key: HBASE-19990 > URL: https://issues.apache.org/jira/browse/HBASE-19990 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-19990-HBASE-19064.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19973) Implement a procedure to replay sync replication wal for standby cluster
[
https://issues.apache.org/jira/browse/HBASE-19973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362136#comment-16362136
]
Hadoop QA commented on HBASE-19973:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HBASE-19064 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
16s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
19s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} HBASE-19064 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
59s{color} | {color:red} hbase-server: The patch generated 6 new + 417
unchanged - 4 fixed = 423 total (was 421) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
6s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m
6s{color} | {color:red} The patch causes 10 errors with Hadoop v2.6.5. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m
5s{color} | {color:red} The patch causes 10 errors with Hadoop v2.7.4. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 10m
10s{color} | {color:red} The patch causes 10 errors with Hadoop v3.0.0. {color}
|
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
0m 50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
25s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 96m
23s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}127m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19973 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910357/HBASE-19973.HBASE-19064.002.patch
|
| Optional Tests |
[jira] [Commented] (HBASE-19979) ReplicationSyncUp tool may leak Zookeeper connection
[
https://issues.apache.org/jira/browse/HBASE-19979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362236#comment-16362236
]
Hadoop QA commented on HBASE-19979:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
42s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} branch-1 passed with JDK v1.8.0_162 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} branch-1 passed with JDK v1.7.0_171 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
35s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
9s{color} | {color:red} hbase-server in branch-1 has 1 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} branch-1 passed with JDK v1.8.0_162 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} branch-1 passed with JDK v1.7.0_171 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} the patch passed with JDK v1.8.0_162 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} the patch passed with JDK v1.7.0_171 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
18s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m
6s{color} | {color:red} The patch causes 44 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m
52s{color} | {color:red} The patch causes 44 errors with Hadoop v2.5.2. {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed with JDK v1.8.0_162 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} the patch passed with JDK v1.7.0_171 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 99m 41s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}124m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:36a7029 |
| JIRA Issue | HBASE-19979 |
|
[jira] [Resolved] (HBASE-11622) completebulkload/loadIncrementalHFiles cannot specify table with namespace
[
https://issues.apache.org/jira/browse/HBASE-11622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai resolved HBASE-11622.
Resolution: Duplicate
> completebulkload/loadIncrementalHFiles cannot specify table with namespace
> --
>
> Key: HBASE-11622
> URL: https://issues.apache.org/jira/browse/HBASE-11622
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Jianshi Huang
>Priority: Major
>
> I'm using completebulkload to load 500GB of data to a table (presplitted).
> However, it reports the following errors:
> Looks like completebulkload didn't recognize the namespace part
> (namespace:table).
> Caused by: java.net.URISyntaxException: Relative path in absolute URI:
> grapple:vertices,37.bottom
> at java.net.URI.checkPath(URI.java:1804)
> at java.net.URI.(URI.java:752)
> at org.apache.hadoop.fs.Path.initialize(Path.java:203)
> By looking at the source code of LoadIncrementalHFiles.java, it seems the
> temporary path created for splitting will contain ':',
> The error part should be this:
> String uniqueName = getUniqueName(table.getName());
> HColumnDescriptor familyDesc =
> table.getTableDescriptor().getFamily(item.family);
> Path botOut = new Path(tmpDir, uniqueName + ".bottom");
> Path topOut = new Path(tmpDir, uniqueName + ".top");
> splitStoreFile(getConf(), hfilePath, familyDesc, splitKey,
> botOut, topOut);
> uniqueName will be "namespce:table" so "new Path(...)" will fail.
> A bug?
> P.S.
> Comment from Matteo Bertozzi:
> we don't need the name to be related to the table name.
> We can replace the getUniqueName() using something like this
> String getUniqueName(TableName tableName) {
> String name = UUID.randomUUID().toString().replaceAll("-", "") +
> "," + regionCount.incrementAndGet();
> return name;
> }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19990) Create remote wal directory when transitting to state S
[ https://issues.apache.org/jira/browse/HBASE-19990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362350#comment-16362350 ] Duo Zhang commented on HBASE-19990: --- I think the flag a bit strange. We can not guard all the sync replication related code with the flag, and also, if there are sync replication peer then it is not possible to disable it I think? Maybe it should only be used in WALFactory, where we sometimes do not need to create provider for sync replication. > Create remote wal directory when transitting to state S > --- > > Key: HBASE-19990 > URL: https://issues.apache.org/jira/browse/HBASE-19990 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-19990-HBASE-19064.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19977) FileMmapEngine allocation of byte buffers should be synchronized
[ https://issues.apache.org/jira/browse/HBASE-19977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361989#comment-16361989 ] Anoop Sam John commented on HBASE-19977: +1 > FileMmapEngine allocation of byte buffers should be synchronized > > > Key: HBASE-19977 > URL: https://issues.apache.org/jira/browse/HBASE-19977 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-alpha-4 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19977.patch > > > Recently we have been testing bucket cache with mmap mode. We found that > after the multi threading way of allocating the byte buffers for offheap > bucket cache, the creation of the mmapped byte buffers needs to be > synchronized for allocating the right sized mmapped buffers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19930) fix ImmutableMemStoreLAB#forceCopyOfBigCellInto
[ https://issues.apache.org/jira/browse/HBASE-19930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362102#comment-16362102 ] Ted Yu commented on HBASE-19930: Gali: Can you run IntegrationTestIngestWithMOB with the patch ? > fix ImmutableMemStoreLAB#forceCopyOfBigCellInto > --- > > Key: HBASE-19930 > URL: https://issues.apache.org/jira/browse/HBASE-19930 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-1 >Reporter: Gali Sheffi >Assignee: Gali Sheffi >Priority: Major > Attachments: HBASE-19930-V01.patch, HBASE-19930-V02.patch, > HBASE-19930-V03.patch, HBASE-19930-V04.patch > > > This issue is about fixing ImmutableMemStoreLAB#forceCopyOfBigCellInto. > Following a comment in HBASE-19133 regarding a bug in > ImmutableMemStoreLAB#forceCopyOfBigCellInto (assuming this method is never > called for an ImmutableMemStoreLAB, and just throwing an > IllegalStateException whenever called), the forceCopyOfBigCellInto method now > performs the copy of big cells on the first MSLABImpl in its mslabs > linked-list. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19945) Make TestRSGroups LargeTest to prevent timeout
[
https://issues.apache.org/jira/browse/HBASE-19945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-19945:
---
Description:
TestRSGroups is annotated as MediumTests. It times out on Jenkins:
https://builds.apache.org/job/HBase-Trunk_matrix/4537/jdk=JDK%201.8%20(latest),label=(Hadoop%20&&%20!H5)/testReport/junit/org.apache.hadoop.hbase.rsgroup/TestRSGroups/org_apache_hadoop_hbase_rsgroup_TestRSGroups/
{code}
org.junit.runners.model.TestTimedOutException: test timed out after 180 seconds
{code}
The above is reproducible on Linux locally.
TestRSGroups should be made LargeTest.
was:
TestRSGroups is annotated as MediumTests. It times out on Jenkins:
https://builds.apache.org/job/HBase-Trunk_matrix/4537/jdk=JDK%201.8%20(latest),label=(Hadoop%20&&%20!H5)/testReport/junit/org.apache.hadoop.hbase.rsgroup/TestRSGroups/org_apache_hadoop_hbase_rsgroup_TestRSGroups/
{code}
org.junit.runners.model.TestTimedOutException: test timed out after 180 seconds
{code}
The above is reproducible on Linux locally.
The tests from TestRSGroups should be separated into two classes.
> Make TestRSGroups LargeTest to prevent timeout
> --
>
> Key: HBASE-19945
> URL: https://issues.apache.org/jira/browse/HBASE-19945
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 19945.v1.txt, 19945.v2.txt
>
>
> TestRSGroups is annotated as MediumTests. It times out on Jenkins:
> https://builds.apache.org/job/HBase-Trunk_matrix/4537/jdk=JDK%201.8%20(latest),label=(Hadoop%20&&%20!H5)/testReport/junit/org.apache.hadoop.hbase.rsgroup/TestRSGroups/org_apache_hadoop_hbase_rsgroup_TestRSGroups/
> {code}
> org.junit.runners.model.TestTimedOutException: test timed out after 180
> seconds
> {code}
> The above is reproducible on Linux locally.
> TestRSGroups should be made LargeTest.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19979) ReplicationSyncUp tool may leak Zookeeper connection
[
https://issues.apache.org/jira/browse/HBASE-19979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pankaj Kumar updated HBASE-19979:
-
Attachment: HBASE-19979-branch-1.001.patch
> ReplicationSyncUp tool may leak Zookeeper connection
>
>
> Key: HBASE-19979
> URL: https://issues.apache.org/jira/browse/HBASE-19979
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Attachments: HBASE-19979-branch-1.001.patch
>
>
> ReplicationSyncUp tool may leak Zookeeper connection in the following code
> snippet,
> {code}
> try {
> int numberOfOldSource = 1; // default wait once
> while (numberOfOldSource > 0) {
> Thread.sleep(SLEEP_TIME);
> numberOfOldSource = manager.getOldSources().size();
> }
> } catch (InterruptedException e) {
> System.err.println("didn't wait long enough:" + e);
> return (-1);
> }
> manager.join();
> zkw.close();
> {code}
> ZooKeeperWatcher will not be closed in case of InterruptedException.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19977) FileMmapEngine allocation of byte buffers should be synchronized
[
https://issues.apache.org/jira/browse/HBASE-19977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362123#comment-16362123
]
Hadoop QA commented on HBASE-19977:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
51s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
41s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
18m 43s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}118m
0s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}156m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19977 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910355/HBASE-19977.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 2147af8d6f6a 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / b4622ffad7 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/11504/testReport/ |
| Max. process+thread count | 5045 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/11504/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> FileMmapEngine allocation of byte buffers should
[jira] [Commented] (HBASE-19965) Fix flaky TestAsyncRegionAdminApi
[ https://issues.apache.org/jira/browse/HBASE-19965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362227#comment-16362227 ] Hudson commented on HBASE-19965: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4577 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4577/]) HBASE-19965 Fix flaky TestAsyncRegionAdminApi (stack: rev cf57ea15f1044d69fcfc8edee3c6f710ed78a7e0) * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncRegionAdminApi2.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncRegionAdminApi.java HBASE-19965 Fix flaky TestAsyncRegionAdminApi; ADDENDUM do same to (stack: rev b4622ffad7011b3d29bce6376edb896de6df9f6e) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncTableAdminApi.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncTableAdminApi2.java > Fix flaky TestAsyncRegionAdminApi > - > > Key: HBASE-19965 > URL: https://issues.apache.org/jira/browse/HBASE-19965 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19965.branch-2.001.patch > > > See > [https://builds.apache.org/job/HBase%20Nightly/job/branch-2/284/testReport/junit/org.apache.hadoop.hbase.client/TestAsyncRegionAdminApi/testMergeRegions_0_/] > > java.lang.AssertionError: expected:<2> but was:<3> at > org.apache.hadoop.hbase.client.TestAsyncRegionAdminApi.testMergeRegions(TestAsyncRegionAdminApi.java:359) > > Merge regions not work. The table still have 3 regions after the > MergeRegionsProcedure finished. > The master start balance region 9e2773ba1efba79a2defa276e9a26ed4. But because > the MergeRegionsProcedure pid=138 start work first, so the balance need wait > for the lock. But after merge regions finished, the MoveRegionProcedure > pid=139 start work and assign 9e2773ba1efba79a2defa276e9a26ed4 to a new > region server. This is not right. The MoveRegionProcedure should skip to > assign a region which was marked as offline. Or we should clear the merged > regions' procedure when MergeRegionsProcedure finished. > > Logs: > 2018-02-08 16:24:44,608 INFO [master/cd4730e3eae2:0.Chore.1] > master.HMaster(1454): balance > hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., > source=cd4730e3eae2,39077,1518106776411, > destination=cd4730e3eae2,40578,1518106776318 > 2018-02-08 16:24:44,608 DEBUG > [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37885] > procedure2.ProcedureExecutor(868): Stored pid=138, > state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE; MergeTableRegionsProcedure > table=testMergeRegions, regions=[9e2773ba1efba79a2defa276e9a26ed4, > 8f8fd5cd032313e1aadb83e31e1b7479], forcibly=false > .. > 2018-02-08 16:24:50,111 INFO [PEWorker-13] > procedure2.ProcedureExecutor(1249): Finished pid=138, state=SUCCESS; > MergeTableRegionsProcedure table=testMergeRegions, > regions=[9e2773ba1efba79a2defa276e9a26ed4, 8f8fd5cd032313e1aadb83e31e1b7479], > forcibly=false in 5.5710sec > 2018-02-08 16:24:50,113 INFO [PEWorker-13] > procedure.MasterProcedureScheduler(813): pid=139, > state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure > hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., > source=cd4730e3eae2,39077,1518106776411, > destination=cd4730e3eae2,40578,1518106776318 testMergeRegions > testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19990) Create remote wal directory when transitting to state S
[ https://issues.apache.org/jira/browse/HBASE-19990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362342#comment-16362342 ] Guanghao Zhang commented on HBASE-19990: bq. mfs.getWALFileSystem().mkdirs(remoteWALDirForPeer) Use walFs directly. > Create remote wal directory when transitting to state S > --- > > Key: HBASE-19990 > URL: https://issues.apache.org/jira/browse/HBASE-19990 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-19990-HBASE-19064.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19979) ReplicationSyncUp tool may leak Zookeeper connection
[
https://issues.apache.org/jira/browse/HBASE-19979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pankaj Kumar updated HBASE-19979:
-
Affects Version/s: (was: 1.5.0)
> ReplicationSyncUp tool may leak Zookeeper connection
>
>
> Key: HBASE-19979
> URL: https://issues.apache.org/jira/browse/HBASE-19979
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Attachments: HBASE-19979-branch-1.001.patch, HBASE-19979.patch
>
>
> ReplicationSyncUp tool may leak Zookeeper connection in the following code
> snippet,
> {code}
> try {
> int numberOfOldSource = 1; // default wait once
> while (numberOfOldSource > 0) {
> Thread.sleep(SLEEP_TIME);
> numberOfOldSource = manager.getOldSources().size();
> }
> } catch (InterruptedException e) {
> System.err.println("didn't wait long enough:" + e);
> return (-1);
> }
> manager.join();
> zkw.close();
> {code}
> ZooKeeperWatcher will not be closed in case of InterruptedException.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19970) Remove unused functions from TableAuthManager
[
https://issues.apache.org/jira/browse/HBASE-19970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362305#comment-16362305
]
Ashish Singhi commented on HBASE-19970:
---
This jira seems to have introduced a find bug warning..
[https://builds.apache.org/job/PreCommit-HBASE-Build/11507/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html]
{noformat}
Bug type DLS_DEAD_LOCAL_STORE (click for details)
In class org.apache.hadoop.hbase.security.access.AccessController
In method
org.apache.hadoop.hbase.security.access.AccessController.initialize(RegionCoprocessorEnvironment)
Local variable named conf
At AccessController.java:[line 237]{noformat}
> Remove unused functions from TableAuthManager
> -
>
> Key: HBASE-19970
> URL: https://issues.apache.org/jira/browse/HBASE-19970
> Project: HBase
> Issue Type: Task
>Reporter: Appy
>Assignee: Appy
>Priority: Minor
> Fix For: 1.5.0, 2.0.0-beta-2
>
> Attachments: HBASE-19970.master.001.patch
>
>
> Functions deleted in TableAuthManager:
> - setTableUserPermissions
> - setTableGroupPermissions
> - setNamespaceUserPermissions
> - setNamespaceGroupPermissions
> - writeTableToZooKeeper
> - writeNamespaceToZooKeeper
> To make sure it was not a bug, and that relevant functionality moved to some
> alternate code path, tried to find out why and when these functions went out
> of use. But just couldn't figure out...until i reached the patch which added
> them. Looks like they were dead functions to start with :)
> Jira which added them: HBASE-8409. Commit id:
> ac10b3c13d6b66e12d0c9601204b01dfa525ed19
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-18294: -- Release Note: A region is flushed if its memory component exceeds the region flush threshold. A flush policy decides which stores to flush by comparing the size of the store to a column-family-flush threshold. If the overall size of all memstores in the machine exceeds the bounds defined by the administrator (denoted global pressure) a region is selected and flush. HBASE-18294 changes flush decisions to be based on heap-occupancy and not data (key-value) size, consistently across levels. This rolls back some of the changes by HBASE-16747. Specifically, (1) RSs, Regions and stores track their overall on-heap and off-heap occupancy, (2) A region is flushed when its on-heap+off-heap size exceeds the region flush threshold, (3) The store to be flushed is chosen based on its on-heap+off-heap size (4) At the RS level, a flush is triggered when the overall on-heap exceeds the on-heap limit, or when the overall off-heap size exceeds the off-heap limit (low/high water marks). Note that when the region flush size is set to XXmb a region flush may be triggered even before writing keys and values of size XX because the total heap occupancy of the region which includes additional metadata exceeded the threshold. > Reduce global heap pressure: flush based on heap occupancy > -- > > Key: HBASE-18294 > URL: https://issues.apache.org/jira/browse/HBASE-18294 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.01.patch, HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.01.patch, HBASE-18294.01.patch, HBASE-18294.02.patch, > HBASE-18294.03.patch, HBASE-18294.04.patch, HBASE-18294.05.patch, > HBASE-18294.06.patch, HBASE-18294.07.patch, HBASE-18294.07.patch, > HBASE-18294.08.patch, HBASE-18294.09.patch, HBASE-18294.10.patch, > HBASE-18294.11.patch, HBASE-18294.11.patch, HBASE-18294.12.patch, > HBASE-18294.13.patch, HBASE-18294.15.patch, HBASE-18294.16.patch, > HBASE-18294.master.01.patch, HBASE-18294.master.01.patch > > > A region is flushed if its memory component exceed a threshold (default size > is 128MB). > A flush policy decides whether to flush a store by comparing the size of the > store to another threshold (that can be configured with > hbase.hregion.percolumnfamilyflush.size.lower.bound). > Currently the implementation (in both cases) compares the data size > (key-value only) to the threshold where it should compare the heap size > (which includes index size, and metadata). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-19990) Create remote wal directory when transitting to state S
Duo Zhang created HBASE-19990: - Summary: Create remote wal directory when transitting to state S Key: HBASE-19990 URL: https://issues.apache.org/jira/browse/HBASE-19990 Project: HBase Issue Type: Sub-task Reporter: Duo Zhang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-16693) The commit of HBASE-16604 creates a unrelated folder
[ https://issues.apache.org/jira/browse/HBASE-16693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai reassigned HBASE-16693: -- Assignee: Enis Soztutar > The commit of HBASE-16604 creates a unrelated folder > > > Key: HBASE-16693 > URL: https://issues.apache.org/jira/browse/HBASE-16693 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Enis Soztutar >Priority: Critical > Fix For: 2.0.0 > > > The folder's name is "hbase-thrift\036src", and it causes the git-clone > failure on windows. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (HBASE-11622) completebulkload/loadIncrementalHFiles cannot specify table with namespace
[
https://issues.apache.org/jira/browse/HBASE-11622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai reopened HBASE-11622:
> completebulkload/loadIncrementalHFiles cannot specify table with namespace
> --
>
> Key: HBASE-11622
> URL: https://issues.apache.org/jira/browse/HBASE-11622
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Jianshi Huang
>Priority: Major
>
> I'm using completebulkload to load 500GB of data to a table (presplitted).
> However, it reports the following errors:
> Looks like completebulkload didn't recognize the namespace part
> (namespace:table).
> Caused by: java.net.URISyntaxException: Relative path in absolute URI:
> grapple:vertices,37.bottom
> at java.net.URI.checkPath(URI.java:1804)
> at java.net.URI.(URI.java:752)
> at org.apache.hadoop.fs.Path.initialize(Path.java:203)
> By looking at the source code of LoadIncrementalHFiles.java, it seems the
> temporary path created for splitting will contain ':',
> The error part should be this:
> String uniqueName = getUniqueName(table.getName());
> HColumnDescriptor familyDesc =
> table.getTableDescriptor().getFamily(item.family);
> Path botOut = new Path(tmpDir, uniqueName + ".bottom");
> Path topOut = new Path(tmpDir, uniqueName + ".top");
> splitStoreFile(getConf(), hfilePath, familyDesc, splitKey,
> botOut, topOut);
> uniqueName will be "namespce:table" so "new Path(...)" will fail.
> A bug?
> P.S.
> Comment from Matteo Bertozzi:
> we don't need the name to be related to the table name.
> We can replace the getUniqueName() using something like this
> String getUniqueName(TableName tableName) {
> String name = UUID.randomUUID().toString().replaceAll("-", "") +
> "," + regionCount.incrementAndGet();
> return name;
> }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19990) Create remote wal directory when transitting to state S
[ https://issues.apache.org/jira/browse/HBASE-19990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19990: -- Assignee: Duo Zhang Status: Patch Available (was: Open) > Create remote wal directory when transitting to state S > --- > > Key: HBASE-19990 > URL: https://issues.apache.org/jira/browse/HBASE-19990 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-19990-HBASE-19064.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19990) Create remote wal directory when transitting to state S
[ https://issues.apache.org/jira/browse/HBASE-19990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19990: -- Attachment: HBASE-19990-HBASE-19064.patch > Create remote wal directory when transitting to state S > --- > > Key: HBASE-19990 > URL: https://issues.apache.org/jira/browse/HBASE-19990 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Priority: Major > Attachments: HBASE-19990-HBASE-19064.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19981) Boolean#getBoolean is used to parse value
[
https://issues.apache.org/jira/browse/HBASE-19981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362357#comment-16362357
]
Chia-Ping Tsai commented on HBASE-19981:
This bug is also in both of branch-1.3 and branch-1.2, right?
> Boolean#getBoolean is used to parse value
> -
>
> Key: HBASE-19981
> URL: https://issues.apache.org/jira/browse/HBASE-19981
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Janos Gub
>Priority: Major
> Fix For: 1.4.2
>
> Attachments: HBASE-19981.branch-1.001.patch
>
>
> In HColumnDescriptor of branch-1:
> {code}
> value.set(Bytes.toBytes(
> Boolean.getBoolean(Bytes.toString(value.get()))
> {code}
> According to
> https://docs.oracle.com/javase/7/docs/api/java/lang/Boolean.html#getBoolean(java.lang.String):
> {code}
> Returns true if and only if the system property named by the argument exists
> and is equal to the string "true"
> {code}
> This was not the intention of the quoted code.
> This was discovered by Fortify.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19930) fix ImmutableMemStoreLAB#forceCopyOfBigCellInto
[ https://issues.apache.org/jira/browse/HBASE-19930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362070#comment-16362070 ] Anoop Sam John commented on HBASE-19930: Fine.. add enough comments.. Hope we have tests to cover all these scenarios. Also when it is data compaction(not just index) any such possible issues? Better we have tests for all possible combinations. > fix ImmutableMemStoreLAB#forceCopyOfBigCellInto > --- > > Key: HBASE-19930 > URL: https://issues.apache.org/jira/browse/HBASE-19930 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-1 >Reporter: Gali Sheffi >Assignee: Gali Sheffi >Priority: Major > Attachments: HBASE-19930-V01.patch, HBASE-19930-V02.patch, > HBASE-19930-V03.patch, HBASE-19930-V04.patch > > > This issue is about fixing ImmutableMemStoreLAB#forceCopyOfBigCellInto. > Following a comment in HBASE-19133 regarding a bug in > ImmutableMemStoreLAB#forceCopyOfBigCellInto (assuming this method is never > called for an ImmutableMemStoreLAB, and just throwing an > IllegalStateException whenever called), the forceCopyOfBigCellInto method now > performs the copy of big cells on the first MSLABImpl in its mslabs > linked-list. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19971) TestMajorCompaction#testDataBlockEncodingInCacheOnly sometimes fails with IllegalArgumentException
[
https://issues.apache.org/jira/browse/HBASE-19971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362010#comment-16362010
]
Anastasia Braginsky commented on HBASE-19971:
-
Hey,
I do not see enough information to debug and to investigate the problem. The
exception stack is unclear for me from majorCompaction() what is the path to
getCell() from CCM (which is the only place to throw this exception)? Please
let me know in case the problem can be reproduced.
Generally speaking, I saw this exception once when there were no more chunks in
pool, but the problem was fixed long ago. What version of the code is used
here?
> TestMajorCompaction#testDataBlockEncodingInCacheOnly sometimes fails with
> IllegalArgumentException
> --
>
> Key: HBASE-19971
> URL: https://issues.apache.org/jira/browse/HBASE-19971
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Major
> Attachments: 19971.v1.txt, testDataBlockEncodingInCacheOnly.out
>
>
> From
> https://builds.apache.org/job/HBASE-Flaky-Tests/25784/testReport/junit/org.apache.hadoop.hbase.regionserver/TestMajorCompaction/testDataBlockEncodingInCacheOnly_2_/
> :
> {code}
> java.io.IOException: java.lang.IllegalArgumentException: In CellChunkMap,
> cell must be associated with chunk.. We were looking for a cell at index 0
> at
> org.apache.hadoop.hbase.regionserver.TestMajorCompaction.majorCompaction(TestMajorCompaction.java:206)
> at
> org.apache.hadoop.hbase.regionserver.TestMajorCompaction.majorCompactionWithDataBlockEncoding(TestMajorCompaction.java:186)
> at
> org.apache.hadoop.hbase.regionserver.TestMajorCompaction.testDataBlockEncodingInCacheOnly(TestMajorCompaction.java:166)
> Caused by: java.lang.IllegalArgumentException: In CellChunkMap, cell must be
> associated with chunk.. We were looking for a cell at index 0
> at
> org.apache.hadoop.hbase.regionserver.TestMajorCompaction.majorCompaction(TestMajorCompaction.java:206)
> at
> org.apache.hadoop.hbase.regionserver.TestMajorCompaction.majorCompactionWithDataBlockEncoding(TestMajorCompaction.java:186)
> at
> org.apache.hadoop.hbase.regionserver.TestMajorCompaction.testDataBlockEncodingInCacheOnly(TestMajorCompaction.java:166)
> {code}
> From the index of the test, EAGER policy was used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19979) ReplicationSyncUp tool may leak Zookeeper connection
[
https://issues.apache.org/jira/browse/HBASE-19979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362117#comment-16362117
]
Ashish Singhi commented on HBASE-19979:
---
+1
> ReplicationSyncUp tool may leak Zookeeper connection
>
>
> Key: HBASE-19979
> URL: https://issues.apache.org/jira/browse/HBASE-19979
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.5.0
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Attachments: HBASE-19979-branch-1.001.patch
>
>
> ReplicationSyncUp tool may leak Zookeeper connection in the following code
> snippet,
> {code}
> try {
> int numberOfOldSource = 1; // default wait once
> while (numberOfOldSource > 0) {
> Thread.sleep(SLEEP_TIME);
> numberOfOldSource = manager.getOldSources().size();
> }
> } catch (InterruptedException e) {
> System.err.println("didn't wait long enough:" + e);
> return (-1);
> }
> manager.join();
> zkw.close();
> {code}
> ZooKeeperWatcher will not be closed in case of InterruptedException.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19986) If HBaseTestClassRule timesout a test, thread dump.
[ https://issues.apache.org/jira/browse/HBASE-19986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362011#comment-16362011 ] Hudson commented on HBASE-19986: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4576 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4576/]) HBASE-19986 If HBaseTestClassRule timesout a test, thread dump; (stack: rev 24bed6b3fbb54a20ef67201c160de649421cdd51) * (edit) hbase-common/src/test/java/org/apache/hadoop/hbase/TestTimeout.java * (edit) hbase-common/src/test/java/org/apache/hadoop/hbase/HBaseClassTestRule.java * (edit) hbase-common/src/test/java/org/apache/hadoop/hbase/TimedOutTestsListener.java > If HBaseTestClassRule timesout a test, thread dump. > --- > > Key: HBASE-19986 > URL: https://issues.apache.org/jira/browse/HBASE-19986 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19986.branch-2.001.patch, > HBASE-19986.branch-2.002.patch, HBASE-19986.branch-2.003.patch > > > We set look for stuck thread in our timeout rule but it is super conservative > in what it prints.. it looks for a RUNNABLE thread and prints first found > ONLY. Pretty useless for us. If a test timesout, often the printing has > stopped in the stderr/stdout. > I'm trying to debug TestAsyncRegionAdminApi. It says test timed out after 10 > minutes but we've stopped printing to the logs and here is what junit prints: > --- > Test set: org.apache.hadoop.hbase.client.TestAsyncRegionAdminApi > --- > Tests run: 25, Failures: 0, Errors: 2, Skipped: 2, Time elapsed: 572.508 s > <<< FAILURE! - in org.apache.hadoop.hbase.client.TestAsyncRegionAdminApi > org.apache.hadoop.hbase.client.TestAsyncRegionAdminApi Time elapsed: 14.642 > s <<< ERROR! > org.junit.runners.model.TestTimedOutException: test timed out after 600 > seconds > at > org.apache.hadoop.hbase.client.TestAsyncRegionAdminApi.testMergeRegions(TestAsyncRegionAdminApi.java:363) > org.apache.hadoop.hbase.client.TestAsyncRegionAdminApi Time elapsed: 14.642 > s <<< ERROR! > java.lang.Exception: Appears to be stuck in thread Socket Reader #1 for port > 35917 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19960) Doc test timeouts and test categories in hbase2
[ https://issues.apache.org/jira/browse/HBASE-19960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362012#comment-16362012 ] Hudson commented on HBASE-19960: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4576 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4576/]) HBASE-19960 Doc test timeouts and test categories in hbase2 (stack: rev f8c3d4568e02ba7c65d0ff82a644e8757c55ff7b) * (edit) hbase-annotations/src/test/java/org/apache/hadoop/hbase/testclassification/LargeTests.java * (edit) hbase-annotations/src/test/java/org/apache/hadoop/hbase/testclassification/MediumTests.java * (edit) hbase-annotations/src/test/java/org/apache/hadoop/hbase/testclassification/SmallTests.java * (edit) src/main/asciidoc/_chapters/developer.adoc > Doc test timeouts and test categories in hbase2 > --- > > Key: HBASE-19960 > URL: https://issues.apache.org/jira/browse/HBASE-19960 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19960.branch-2.001.patch > > > Write up that Categories are no longer acted upon, that we no longer timeout > test methods. Write up that if a test goes longer than ten minutes, it is > killed. Make passing reference to how it used to be but don't spend much time > on it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19973) Implement a procedure to replay sync replication wal for standby cluster
[
https://issues.apache.org/jira/browse/HBASE-19973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361950#comment-16361950
]
Hadoop QA commented on HBASE-19973:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HBASE-19064 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
34s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
10s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} HBASE-19064 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 1m
40s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
29s{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 29s{color} |
{color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 29s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
11s{color} | {color:red} hbase-server: The patch generated 14 new + 417
unchanged - 4 fixed = 431 total (was 421) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 3m
18s{color} | {color:red} patch has 24 errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m
15s{color} | {color:red} The patch causes 24 errors with Hadoop v2.6.5. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 7m
4s{color} | {color:red} The patch causes 24 errors with Hadoop v2.7.4. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 9m
1s{color} | {color:red} The patch causes 24 errors with Hadoop v3.0.0. {color} |
| {color:red}-1{color} | {color:red} hbaseprotoc {color} | {color:red} 0m
34s{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
32s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 32s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
20s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19973 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910341/HBASE-19973.HBASE-19064.001.patch
|
| Optional Tests
[jira] [Updated] (HBASE-19973) Implement a procedure to replay sync replication wal for standby cluster
[ https://issues.apache.org/jira/browse/HBASE-19973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-19973: --- Attachment: HBASE-19973.HBASE-19064.002.patch > Implement a procedure to replay sync replication wal for standby cluster > > > Key: HBASE-19973 > URL: https://issues.apache.org/jira/browse/HBASE-19973 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Attachments: HBASE-19973.HBASE-19064.001.patch, > HBASE-19973.HBASE-19064.002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19971) TestMajorCompaction#testDataBlockEncodingInCacheOnly sometimes fails with IllegalArgumentException
[
https://issues.apache.org/jira/browse/HBASE-19971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362092#comment-16362092
]
Ted Yu commented on HBASE-19971:
Build #25784 was based off 0593dda66359dbd1ba3a06dbd12b8f2521eef6b9
> TestMajorCompaction#testDataBlockEncodingInCacheOnly sometimes fails with
> IllegalArgumentException
> --
>
> Key: HBASE-19971
> URL: https://issues.apache.org/jira/browse/HBASE-19971
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Major
> Attachments: 19971.v1.txt, testDataBlockEncodingInCacheOnly.out
>
>
> From
> https://builds.apache.org/job/HBASE-Flaky-Tests/25784/testReport/junit/org.apache.hadoop.hbase.regionserver/TestMajorCompaction/testDataBlockEncodingInCacheOnly_2_/
> :
> {code}
> java.io.IOException: java.lang.IllegalArgumentException: In CellChunkMap,
> cell must be associated with chunk.. We were looking for a cell at index 0
> at
> org.apache.hadoop.hbase.regionserver.TestMajorCompaction.majorCompaction(TestMajorCompaction.java:206)
> at
> org.apache.hadoop.hbase.regionserver.TestMajorCompaction.majorCompactionWithDataBlockEncoding(TestMajorCompaction.java:186)
> at
> org.apache.hadoop.hbase.regionserver.TestMajorCompaction.testDataBlockEncodingInCacheOnly(TestMajorCompaction.java:166)
> Caused by: java.lang.IllegalArgumentException: In CellChunkMap, cell must be
> associated with chunk.. We were looking for a cell at index 0
> at
> org.apache.hadoop.hbase.regionserver.TestMajorCompaction.majorCompaction(TestMajorCompaction.java:206)
> at
> org.apache.hadoop.hbase.regionserver.TestMajorCompaction.majorCompactionWithDataBlockEncoding(TestMajorCompaction.java:186)
> at
> org.apache.hadoop.hbase.regionserver.TestMajorCompaction.testDataBlockEncodingInCacheOnly(TestMajorCompaction.java:166)
> {code}
> From the index of the test, EAGER policy was used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19979) ReplicationSyncUp tool may leak Zookeeper connection
[
https://issues.apache.org/jira/browse/HBASE-19979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pankaj Kumar updated HBASE-19979:
-
Affects Version/s: 1.5.0
Status: Patch Available (was: Open)
Simple patch, please review.
> ReplicationSyncUp tool may leak Zookeeper connection
>
>
> Key: HBASE-19979
> URL: https://issues.apache.org/jira/browse/HBASE-19979
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.5.0
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Attachments: HBASE-19979-branch-1.001.patch
>
>
> ReplicationSyncUp tool may leak Zookeeper connection in the following code
> snippet,
> {code}
> try {
> int numberOfOldSource = 1; // default wait once
> while (numberOfOldSource > 0) {
> Thread.sleep(SLEEP_TIME);
> numberOfOldSource = manager.getOldSources().size();
> }
> } catch (InterruptedException e) {
> System.err.println("didn't wait long enough:" + e);
> return (-1);
> }
> manager.join();
> zkw.close();
> {code}
> ZooKeeperWatcher will not be closed in case of InterruptedException.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19979) ReplicationSyncUp tool may leak Zookeeper connection
[
https://issues.apache.org/jira/browse/HBASE-19979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362128#comment-16362128
]
Hadoop QA commented on HBASE-19979:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HBASE-19979 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.7.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-19979 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910373/HBASE-19979.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/11508/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> ReplicationSyncUp tool may leak Zookeeper connection
>
>
> Key: HBASE-19979
> URL: https://issues.apache.org/jira/browse/HBASE-19979
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.5.0
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Attachments: HBASE-19979-branch-1.001.patch, HBASE-19979.patch
>
>
> ReplicationSyncUp tool may leak Zookeeper connection in the following code
> snippet,
> {code}
> try {
> int numberOfOldSource = 1; // default wait once
> while (numberOfOldSource > 0) {
> Thread.sleep(SLEEP_TIME);
> numberOfOldSource = manager.getOldSources().size();
> }
> } catch (InterruptedException e) {
> System.err.println("didn't wait long enough:" + e);
> return (-1);
> }
> manager.join();
> zkw.close();
> {code}
> ZooKeeperWatcher will not be closed in case of InterruptedException.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19977) FileMmapEngine allocation of byte buffers should be synchronized
[ https://issues.apache.org/jira/browse/HBASE-19977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-19977: --- Attachment: HBASE-19977.patch > FileMmapEngine allocation of byte buffers should be synchronized > > > Key: HBASE-19977 > URL: https://issues.apache.org/jira/browse/HBASE-19977 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-alpha-4 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19977.patch > > > Recently we have been testing bucket cache with mmap mode. We found that > after the multi threading way of allocating the byte buffers for offheap > bucket cache, the creation of the mmapped byte buffers needs to be > synchronized for allocating the right sized mmapped buffers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19945) Separate tests of TestRSGroups into two classes
[
https://issues.apache.org/jira/browse/HBASE-19945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362097#comment-16362097
]
Ted Yu commented on HBASE-19945:
TestRSGroups is marked MediumTests, hence the 10 minute bound didn't apply.
19945.v2.txt would change it to LargeTests.
> Separate tests of TestRSGroups into two classes
> ---
>
> Key: HBASE-19945
> URL: https://issues.apache.org/jira/browse/HBASE-19945
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 19945.v1.txt, 19945.v2.txt
>
>
> TestRSGroups is annotated as MediumTests. It times out on Jenkins:
> https://builds.apache.org/job/HBase-Trunk_matrix/4537/jdk=JDK%201.8%20(latest),label=(Hadoop%20&&%20!H5)/testReport/junit/org.apache.hadoop.hbase.rsgroup/TestRSGroups/org_apache_hadoop_hbase_rsgroup_TestRSGroups/
> {code}
> org.junit.runners.model.TestTimedOutException: test timed out after 180
> seconds
> {code}
> The above is reproducible on Linux locally.
> The tests from TestRSGroups should be separated into two classes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19930) fix ImmutableMemStoreLAB#forceCopyOfBigCellInto
[ https://issues.apache.org/jira/browse/HBASE-19930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362094#comment-16362094 ] Anastasia Braginsky commented on HBASE-19930: - Any type of merge/compaction goes through the same code path, so it is all the same... > fix ImmutableMemStoreLAB#forceCopyOfBigCellInto > --- > > Key: HBASE-19930 > URL: https://issues.apache.org/jira/browse/HBASE-19930 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-1 >Reporter: Gali Sheffi >Assignee: Gali Sheffi >Priority: Major > Attachments: HBASE-19930-V01.patch, HBASE-19930-V02.patch, > HBASE-19930-V03.patch, HBASE-19930-V04.patch > > > This issue is about fixing ImmutableMemStoreLAB#forceCopyOfBigCellInto. > Following a comment in HBASE-19133 regarding a bug in > ImmutableMemStoreLAB#forceCopyOfBigCellInto (assuming this method is never > called for an ImmutableMemStoreLAB, and just throwing an > IllegalStateException whenever called), the forceCopyOfBigCellInto method now > performs the copy of big cells on the first MSLABImpl in its mslabs > linked-list. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19844) Shell should support to flush by regionserver
[ https://issues.apache.org/jira/browse/HBASE-19844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362109#comment-16362109 ] Ted Yu commented on HBASE-19844: +1 Thanks for the work, Reid. Happy New Year. > Shell should support to flush by regionserver > - > > Key: HBASE-19844 > URL: https://issues.apache.org/jira/browse/HBASE-19844 > Project: HBase > Issue Type: New Feature > Components: shell >Reporter: Chia-Ping Tsai >Assignee: Reid Chan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-19844.master.001.patch, > HBASE-19844.master.002.patch, HBASE-19844.master.003.patch, > HBASE-19844.master.004.patch > > > HBASE-4224 add a method to admin that can do the flush by regionserver. As > with other Admin methods, we should enable shell to use the flush method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19977) FileMmapEngine allocation of byte buffers should be synchronized
[ https://issues.apache.org/jira/browse/HBASE-19977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-19977: --- Status: Patch Available (was: Open) > FileMmapEngine allocation of byte buffers should be synchronized > > > Key: HBASE-19977 > URL: https://issues.apache.org/jira/browse/HBASE-19977 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-alpha-4 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19977.patch > > > Recently we have been testing bucket cache with mmap mode. We found that > after the multi threading way of allocating the byte buffers for offheap > bucket cache, the creation of the mmapped byte buffers needs to be > synchronized for allocating the right sized mmapped buffers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19950) Introduce a ColumnValueFilter
[
https://issues.apache.org/jira/browse/HBASE-19950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362120#comment-16362120
]
Hadoop QA commented on HBASE-19950:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
54s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch hbase-protocol-shaded passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} The patch hbase-protocol passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} hbase-client: The patch generated 0 new + 71
unchanged - 2 fixed = 71 total (was 73) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} The patch hbase-server passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
50s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
18m 48s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
1m 35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
21s{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
55s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}100m
40s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
[jira] [Updated] (HBASE-19979) ReplicationSyncUp tool may leak Zookeeper connection
[
https://issues.apache.org/jira/browse/HBASE-19979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pankaj Kumar updated HBASE-19979:
-
Attachment: HBASE-19979.patch
> ReplicationSyncUp tool may leak Zookeeper connection
>
>
> Key: HBASE-19979
> URL: https://issues.apache.org/jira/browse/HBASE-19979
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.5.0
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Attachments: HBASE-19979-branch-1.001.patch, HBASE-19979.patch
>
>
> ReplicationSyncUp tool may leak Zookeeper connection in the following code
> snippet,
> {code}
> try {
> int numberOfOldSource = 1; // default wait once
> while (numberOfOldSource > 0) {
> Thread.sleep(SLEEP_TIME);
> numberOfOldSource = manager.getOldSources().size();
> }
> } catch (InterruptedException e) {
> System.err.println("didn't wait long enough:" + e);
> return (-1);
> }
> manager.join();
> zkw.close();
> {code}
> ZooKeeperWatcher will not be closed in case of InterruptedException.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19977) FileMmapEngine allocation of byte buffers should be synchronized
[ https://issues.apache.org/jira/browse/HBASE-19977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361962#comment-16361962 ] ramkrishna.s.vasudevan commented on HBASE-19977: Patch for branch-2 and master. The integer 'pos' is now AtomicInteger. > FileMmapEngine allocation of byte buffers should be synchronized > > > Key: HBASE-19977 > URL: https://issues.apache.org/jira/browse/HBASE-19977 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-alpha-4 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19977.patch > > > Recently we have been testing bucket cache with mmap mode. We found that > after the multi threading way of allocating the byte buffers for offheap > bucket cache, the creation of the mmapped byte buffers needs to be > synchronized for allocating the right sized mmapped buffers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-9081) Online split for an reserved empty region
[ https://issues.apache.org/jira/browse/HBASE-9081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362016#comment-16362016 ] Pankaj Kumar commented on HBASE-9081: - Thanks [~jeason] for raising this Jira. Multiple split points (pre-split) can be defined only at table creation and thereafter region splits only into two daughter regions either manually (using HBaseAdmin APIs) or automatically (based on the split policy). Currently there is no way to split a region into multiple daughter regions, user need to send multiple RPCs to retrieve table regions and send split request. Based on the customer experiences, there is a need of multiple split of region in a single operation. We can say "Region Multi Split" instead of "Online split for an reserved empty region". There can be multiple scenario where multi split is very much useful, 1) In the beginning user can't predict the incoming data behavior, so create the table with default region (without pre-split). After some data load into the table, user can predict the data distribution and define the split points efficiently. But currently to split the region into multiple regions (let say 500) is not easy with existing APIs. User has to retrieve and split the region multiple times. 2) In case where the incoming data rate is too high, with current region split (2 daughter regions), multiple times splits is going to happen which will cause lot of I/O and cpu resources till it reaches to its desired number of regions (let say 500). But with the new feature, directly region can be split into the desirable number of regions in single operation. Let me know your thought over this, will attach the design doc soon. > Online split for an reserved empty region > - > > Key: HBASE-9081 > URL: https://issues.apache.org/jira/browse/HBASE-9081 > Project: HBase > Issue Type: New Feature > Components: master, regionserver >Reporter: Jieshan Bean >Assignee: Jieshan Bean >Priority: Major > > We already have a region splitter tool. But it can only provide limited > functions: > 1. Create table with a specified region number without give any splits. > 2. Roll-Split on an exist region. > We have such user scenario: > Table was created with splits like below: > abcdefgo > g~o is a reserved empty region. Will use it only after some days. So we don't > know the rowkey distribution currently. Will split it only when it get used. > Say, we want to split g~o with 10 new regions, likes g, g1, g2, g3, g4, > g5...,g9, o. > I didn't find similar function has already been there. Please tell me if I am > wrong. > Hope to hear your ideas on this:) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19977) FileMmapEngine allocation of byte buffers should be synchronized
[ https://issues.apache.org/jira/browse/HBASE-19977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362085#comment-16362085 ] Ted Yu commented on HBASE-19977: +1 > FileMmapEngine allocation of byte buffers should be synchronized > > > Key: HBASE-19977 > URL: https://issues.apache.org/jira/browse/HBASE-19977 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-alpha-4 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19977.patch > > > Recently we have been testing bucket cache with mmap mode. We found that > after the multi threading way of allocating the byte buffers for offheap > bucket cache, the creation of the mmapped byte buffers needs to be > synchronized for allocating the right sized mmapped buffers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19979) ReplicationSyncUp tool may leak Zookeeper connection
[
https://issues.apache.org/jira/browse/HBASE-19979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362115#comment-16362115
]
Ted Yu commented on HBASE-19979:
+1
> ReplicationSyncUp tool may leak Zookeeper connection
>
>
> Key: HBASE-19979
> URL: https://issues.apache.org/jira/browse/HBASE-19979
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.5.0
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Attachments: HBASE-19979-branch-1.001.patch
>
>
> ReplicationSyncUp tool may leak Zookeeper connection in the following code
> snippet,
> {code}
> try {
> int numberOfOldSource = 1; // default wait once
> while (numberOfOldSource > 0) {
> Thread.sleep(SLEEP_TIME);
> numberOfOldSource = manager.getOldSources().size();
> }
> } catch (InterruptedException e) {
> System.err.println("didn't wait long enough:" + e);
> return (-1);
> }
> manager.join();
> zkw.close();
> {code}
> ZooKeeperWatcher will not be closed in case of InterruptedException.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19950) Introduce a ColumnValueFilter
[
https://issues.apache.org/jira/browse/HBASE-19950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361956#comment-16361956
]
Reid Chan commented on HBASE-19950:
---
Fix two checkstyle warnings.
> Introduce a ColumnValueFilter
> -
>
> Key: HBASE-19950
> URL: https://issues.apache.org/jira/browse/HBASE-19950
> Project: HBase
> Issue Type: Improvement
> Components: Filters
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Minor
> Attachments: HBASE-19950.master.001.patch,
> HBASE-19950.master.002.patch, HBASE-19950.master.003.patch,
> HBASE-19950.master.004.patch, HBASE-19950.master.005.patch,
> HBASE-19950.master.006.patch, HBASE-19950.master.007.patch,
> HBASE-19950.master.008.patch, HBASE-19950.master.009.patch
>
>
> Different from {{SingleColumnValueFilter}} which returns an entire row when
> specified condition is matched, this new filter will return the matched cell
> only. There're already some discussions in HBASE-19824.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19950) Introduce a ColumnValueFilter
[
https://issues.apache.org/jira/browse/HBASE-19950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reid Chan updated HBASE-19950:
--
Attachment: HBASE-19950.master.009.patch
> Introduce a ColumnValueFilter
> -
>
> Key: HBASE-19950
> URL: https://issues.apache.org/jira/browse/HBASE-19950
> Project: HBase
> Issue Type: Improvement
> Components: Filters
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Minor
> Attachments: HBASE-19950.master.001.patch,
> HBASE-19950.master.002.patch, HBASE-19950.master.003.patch,
> HBASE-19950.master.004.patch, HBASE-19950.master.005.patch,
> HBASE-19950.master.006.patch, HBASE-19950.master.007.patch,
> HBASE-19950.master.008.patch, HBASE-19950.master.009.patch
>
>
> Different from {{SingleColumnValueFilter}} which returns an entire row when
> specified condition is matched, this new filter will return the matched cell
> only. There're already some discussions in HBASE-19824.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19977) FileMmapEngine allocation of byte buffers should be synchronized
[ https://issues.apache.org/jira/browse/HBASE-19977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361977#comment-16361977 ] Duo Zhang commented on HBASE-19977: --- +1. > FileMmapEngine allocation of byte buffers should be synchronized > > > Key: HBASE-19977 > URL: https://issues.apache.org/jira/browse/HBASE-19977 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-alpha-4 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19977.patch > > > Recently we have been testing bucket cache with mmap mode. We found that > after the multi threading way of allocating the byte buffers for offheap > bucket cache, the creation of the mmapped byte buffers needs to be > synchronized for allocating the right sized mmapped buffers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19930) fix ImmutableMemStoreLAB#forceCopyOfBigCellInto
[ https://issues.apache.org/jira/browse/HBASE-19930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362047#comment-16362047 ] Anastasia Braginsky commented on HBASE-19930: - [~anoop.hbase] and [~ram_krish], what [~chia7712] wrote is absolutely correct and this is the reason for the fix. Indeed it sounds odd that ImmutableMSLAB is updated, but this is what need to happen in this very special case. Gali will add the comment. But after that I'll commit since enough people have taken a look on the fix. > fix ImmutableMemStoreLAB#forceCopyOfBigCellInto > --- > > Key: HBASE-19930 > URL: https://issues.apache.org/jira/browse/HBASE-19930 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-1 >Reporter: Gali Sheffi >Assignee: Gali Sheffi >Priority: Major > Attachments: HBASE-19930-V01.patch, HBASE-19930-V02.patch, > HBASE-19930-V03.patch, HBASE-19930-V04.patch > > > This issue is about fixing ImmutableMemStoreLAB#forceCopyOfBigCellInto. > Following a comment in HBASE-19133 regarding a bug in > ImmutableMemStoreLAB#forceCopyOfBigCellInto (assuming this method is never > called for an ImmutableMemStoreLAB, and just throwing an > IllegalStateException whenever called), the forceCopyOfBigCellInto method now > performs the copy of big cells on the first MSLABImpl in its mslabs > linked-list. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19977) FileMmapEngine allocation of byte buffers should be synchronized
[ https://issues.apache.org/jira/browse/HBASE-19977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-19977: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master and branch-2. Thanks for all the reviews. > FileMmapEngine allocation of byte buffers should be synchronized > > > Key: HBASE-19977 > URL: https://issues.apache.org/jira/browse/HBASE-19977 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-alpha-4 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19977.patch > > > Recently we have been testing bucket cache with mmap mode. We found that > after the multi threading way of allocating the byte buffers for offheap > bucket cache, the creation of the mmapped byte buffers needs to be > synchronized for allocating the right sized mmapped buffers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19945) Make TestRSGroups LargeTest to prevent timeout
[
https://issues.apache.org/jira/browse/HBASE-19945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-19945:
---
Summary: Make TestRSGroups LargeTest to prevent timeout (was: Separate
tests of TestRSGroups into two classes)
> Make TestRSGroups LargeTest to prevent timeout
> --
>
> Key: HBASE-19945
> URL: https://issues.apache.org/jira/browse/HBASE-19945
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 19945.v1.txt, 19945.v2.txt
>
>
> TestRSGroups is annotated as MediumTests. It times out on Jenkins:
> https://builds.apache.org/job/HBase-Trunk_matrix/4537/jdk=JDK%201.8%20(latest),label=(Hadoop%20&&%20!H5)/testReport/junit/org.apache.hadoop.hbase.rsgroup/TestRSGroups/org_apache_hadoop_hbase_rsgroup_TestRSGroups/
> {code}
> org.junit.runners.model.TestTimedOutException: test timed out after 180
> seconds
> {code}
> The above is reproducible on Linux locally.
> The tests from TestRSGroups should be separated into two classes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19876) The exception happening in converting pb mutation to hbase.mutation messes up the CellScanner
[
https://issues.apache.org/jira/browse/HBASE-19876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362410#comment-16362410
]
Hudson commented on HBASE-19876:
SUCCESS: Integrated in Jenkins build HBase-1.2-IT #1068 (See
[https://builds.apache.org/job/HBase-1.2-IT/1068/])
HBASE-19876 The exception happening in converting pb mutation to (chia7712: rev
b59c3151dfe41926d242c5c581a248e272834aa1)
* (add)
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMalformedCellFromClient.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
> The exception happening in converting pb mutation to hbase.mutation messes up
> the CellScanner
> -
>
> Key: HBASE-19876
> URL: https://issues.apache.org/jira/browse/HBASE-19876
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 1.3.2, 1.5.0, 1.2.7, 2.0.0-beta-2, 1.4.2
>
> Attachments: HBASE-19876.branch-1.2.v0.patch,
> HBASE-19876.master.001.patch, HBASE-19876.v0.patch, HBASE-19876.v1.patch,
> HBASE-19876.v2.patch, HBASE-19876.v3.patch, HBASE-19876.v3.patch,
> HBASE-19876.v3.patch, HBASE-19876.v3.patch, HBASE-19876.v4.patch,
> HBASE-19876.v5.patch, HBASE-19876.v6.patch
>
>
> {code:java}
> 2018-01-27 22:51:43,794 INFO [hconnection-0x3291b443-shared-pool11-t6]
> client.AsyncRequestFutureImpl(778): id=5, table=testQuotaStatusFromMaster3,
> attempt=6/16 failed=20ops, last
> exception=org.apache.hadoop.hbase.client.WrongRowIOException:
> org.apache.hadoop.hbase.client.WrongRowIOException: The row in xxx doesn't
> match the original one aaa
> at org.apache.hadoop.hbase.client.Mutation.add(Mutation.java:776)
> at org.apache.hadoop.hbase.client.Put.add(Put.java:282)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toPut(ProtobufUtil.java:642)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:952)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:896)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2591)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304){code}
> I noticed this bug when testing the table space quota.
> When rs are converting pb mutation to hbase.mutation, the quota exception or
> cell exception may be thrown.
> {code}
> Unable to find source-code formatter for language:
> rsrpcservices#dobatchop.java. Available languages are: actionscript, ada,
> applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html,
> java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r,
> rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml for
> (ClientProtos.Action action: mutations) {
> MutationProto m = action.getMutation();
> Mutation mutation;
> if (m.getMutateType() == MutationType.PUT) {
> mutation = ProtobufUtil.toPut(m, cells);
> batchContainsPuts = true;
> } else {
> mutation = ProtobufUtil.toDelete(m, cells);
> batchContainsDelete = true;
> }
> mutationActionMap.put(mutation, action);
> mArray[i++] = mutation;
> checkCellSizeLimit(region, mutation);
> // Check if a space quota disallows this mutation
> spaceQuotaEnforcement.getPolicyEnforcement(region).check(mutation);
> quota.addMutation(mutation);
> }
> {code}
> rs has caught the exception but it doesn't have the cellscanner skip the
> failed cells.
> {code:java}
> } catch (IOException ie) {
> if (atomic) {
> throw ie;
> }
> for (Action mutation : mutations) {
> builder.addResultOrException(getResultOrException(ie,
> mutation.getIndex()));
> }
> }
> {code}
> The bug results in the WrongRowIOException to remaining mutations since they
> refer to invalid cells.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19968) MapReduce test fails with NoClassDefFoundError against hadoop3
[
https://issues.apache.org/jira/browse/HBASE-19968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362468#comment-16362468
]
Peter Somogyi commented on HBASE-19968:
---
[~yuzhih...@gmail.com], I won't create an issue. I looked around it looks like
both -P and -D options are valid.
> MapReduce test fails with NoClassDefFoundError against hadoop3
> --
>
> Key: HBASE-19968
> URL: https://issues.apache.org/jira/browse/HBASE-19968
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: 19968.v1.txt
>
>
> When running mapreduce tests against hadoop3, I observed the following:
> {code}
> [ERROR]
> testWithMockedMapReduceSingleRegion(org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat)
> Time elapsed: 0.024 s <<< ERROR!
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo
> at
> org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat.testWithMockedMapReduce(TestTableSnapshotInputFormat.java:178)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo
> at
> org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat.testWithMockedMapReduce(TestTableSnapshotInputFormat.java:178)
> {code}
> This was due to lack of dependency on hadoop-hdfs module in the hadoop-3.0
> profile of hbase-mapreduce/pom.xml
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19876) The exception happening in converting pb mutation to hbase.mutation messes up the CellScanner
[
https://issues.apache.org/jira/browse/HBASE-19876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362371#comment-16362371
]
Hudson commented on HBASE-19876:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #348 (See
[https://builds.apache.org/job/HBase-1.3-IT/348/])
HBASE-19876 The exception happening in converting pb mutation to (chia7712: rev
e36dca55bcd5ad95cd99a19a3630d6c143acfcab)
* (add)
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMalformedCellFromClient.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
> The exception happening in converting pb mutation to hbase.mutation messes up
> the CellScanner
> -
>
> Key: HBASE-19876
> URL: https://issues.apache.org/jira/browse/HBASE-19876
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 1.3.2, 1.5.0, 1.2.7, 2.0.0-beta-2, 1.4.2
>
> Attachments: HBASE-19876.branch-1.2.v0.patch,
> HBASE-19876.master.001.patch, HBASE-19876.v0.patch, HBASE-19876.v1.patch,
> HBASE-19876.v2.patch, HBASE-19876.v3.patch, HBASE-19876.v3.patch,
> HBASE-19876.v3.patch, HBASE-19876.v3.patch, HBASE-19876.v4.patch,
> HBASE-19876.v5.patch, HBASE-19876.v6.patch
>
>
> {code:java}
> 2018-01-27 22:51:43,794 INFO [hconnection-0x3291b443-shared-pool11-t6]
> client.AsyncRequestFutureImpl(778): id=5, table=testQuotaStatusFromMaster3,
> attempt=6/16 failed=20ops, last
> exception=org.apache.hadoop.hbase.client.WrongRowIOException:
> org.apache.hadoop.hbase.client.WrongRowIOException: The row in xxx doesn't
> match the original one aaa
> at org.apache.hadoop.hbase.client.Mutation.add(Mutation.java:776)
> at org.apache.hadoop.hbase.client.Put.add(Put.java:282)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toPut(ProtobufUtil.java:642)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:952)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:896)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2591)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304){code}
> I noticed this bug when testing the table space quota.
> When rs are converting pb mutation to hbase.mutation, the quota exception or
> cell exception may be thrown.
> {code}
> Unable to find source-code formatter for language:
> rsrpcservices#dobatchop.java. Available languages are: actionscript, ada,
> applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html,
> java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r,
> rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml for
> (ClientProtos.Action action: mutations) {
> MutationProto m = action.getMutation();
> Mutation mutation;
> if (m.getMutateType() == MutationType.PUT) {
> mutation = ProtobufUtil.toPut(m, cells);
> batchContainsPuts = true;
> } else {
> mutation = ProtobufUtil.toDelete(m, cells);
> batchContainsDelete = true;
> }
> mutationActionMap.put(mutation, action);
> mArray[i++] = mutation;
> checkCellSizeLimit(region, mutation);
> // Check if a space quota disallows this mutation
> spaceQuotaEnforcement.getPolicyEnforcement(region).check(mutation);
> quota.addMutation(mutation);
> }
> {code}
> rs has caught the exception but it doesn't have the cellscanner skip the
> failed cells.
> {code:java}
> } catch (IOException ie) {
> if (atomic) {
> throw ie;
> }
> for (Action mutation : mutations) {
> builder.addResultOrException(getResultOrException(ie,
> mutation.getIndex()));
> }
> }
> {code}
> The bug results in the WrongRowIOException to remaining mutations since they
> refer to invalid cells.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19990) Create remote wal directory when transitting to state S
[
https://issues.apache.org/jira/browse/HBASE-19990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362430#comment-16362430
]
Hadoop QA commented on HBASE-19990:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HBASE-19064 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
51s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
26s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} HBASE-19064 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch hbase-replication passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch hbase-procedure passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} hbase-server: The patch generated 0 new + 12
unchanged - 7 fixed = 12 total (was 19) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
11s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m
6s{color} | {color:red} The patch causes 10 errors with Hadoop v2.6.5. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m
3s{color} | {color:red} The patch causes 10 errors with Hadoop v2.7.4. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 10m
7s{color} | {color:red} The patch causes 10 errors with Hadoop v3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
19s{color} | {color:green} hbase-replication in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
40s{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 96m
42s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
0s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}130m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce
[jira] [Updated] (HBASE-19533) How to do controlled shutdown in branch-2?
[ https://issues.apache.org/jira/browse/HBASE-19533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19533: -- Fix Version/s: (was: 2.0.0-beta-2) 2.0.0 > How to do controlled shutdown in branch-2? > -- > > Key: HBASE-19533 > URL: https://issues.apache.org/jira/browse/HBASE-19533 > Project: HBase > Issue Type: Task >Reporter: stack >Priority: Critical > Fix For: 2.0.0 > > > Before HBASE-18946, setting shutdown of a cluster, the Master would exit > immediately. RegionServers would run region closes and then try and notify > the Master of the close and would spew exceptions that the Master was > unreachable. > This is different to how branch-1 used to do it. It used to keep Master up > and it would be like the captain of the ship, the last to go down. As of > HBASE-18946, this is again the case but there are still open issues. > # Usually Master does all open and close of regions. On cluster shutdown, it > is the one time where the Regions run the region close. Currently, the > regions report the close to the Master which disregards the message since it > did not start the region closes. Should we do different? Try and update state > in hbase:meta setting it to CLOSE? We might not be able to write CLOSE for > all regions since hbase:meta will be closing too (the RS that is hosting > hbase:meta will close it last but that may not be enough). > # Should the Master run the cluster shutdown sending out close for all > regions? What if cluster of 1M regions? Untenable? Send a message per server? > That might be better. > Anyways, this needs attention. Filing issue in meantime. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19945) Make TestRSGroups LargeTest to prevent timeout
[
https://issues.apache.org/jira/browse/HBASE-19945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362540#comment-16362540
]
Ted Yu commented on HBASE-19945:
Just checked HBaseClassTestRule - haven't followed recent changes in the rule
closely.
Let's see if the test fails in Jenkins any more.
> Make TestRSGroups LargeTest to prevent timeout
> --
>
> Key: HBASE-19945
> URL: https://issues.apache.org/jira/browse/HBASE-19945
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 19945.v1.txt, 19945.v2.txt
>
>
> TestRSGroups is annotated as MediumTests. It times out on Jenkins:
> https://builds.apache.org/job/HBase-Trunk_matrix/4537/jdk=JDK%201.8%20(latest),label=(Hadoop%20&&%20!H5)/testReport/junit/org.apache.hadoop.hbase.rsgroup/TestRSGroups/org_apache_hadoop_hbase_rsgroup_TestRSGroups/
> {code}
> org.junit.runners.model.TestTimedOutException: test timed out after 180
> seconds
> {code}
> The above is reproducible on Linux locally.
> TestRSGroups should be made LargeTest.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19979) ReplicationSyncUp tool may leak Zookeeper connection
[
https://issues.apache.org/jira/browse/HBASE-19979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362573#comment-16362573
]
Ted Yu commented on HBASE-19979:
Can you re-attach patch for master branch ?
{code}
Hunk #1 FAILED at 114.
1 out of 1 hunk FAILED -- saving rejects to file
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSyncUp.java.rej
{code}
> ReplicationSyncUp tool may leak Zookeeper connection
>
>
> Key: HBASE-19979
> URL: https://issues.apache.org/jira/browse/HBASE-19979
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 1.5.0, 2.0.0-beta-2
>
> Attachments: HBASE-19979-branch-1.001.patch, HBASE-19979.patch
>
>
> ReplicationSyncUp tool may leak Zookeeper connection in the following code
> snippet,
> {code}
> try {
> int numberOfOldSource = 1; // default wait once
> while (numberOfOldSource > 0) {
> Thread.sleep(SLEEP_TIME);
> numberOfOldSource = manager.getOldSources().size();
> }
> } catch (InterruptedException e) {
> System.err.println("didn't wait long enough:" + e);
> return (-1);
> }
> manager.join();
> zkw.close();
> {code}
> ZooKeeperWatcher will not be closed in case of InterruptedException.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19533) How to do controlled shutdown in branch-2?
[ https://issues.apache.org/jira/browse/HBASE-19533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362571#comment-16362571 ] stack commented on HBASE-19533: --- Moving this out of beta-2. The big question on whether Master runs the shutdown closing Regions will not be addressed in time for hbase-2.0.0. Punting on it. Particular shutdown problems have been handled in subissues and elsewhere. > How to do controlled shutdown in branch-2? > -- > > Key: HBASE-19533 > URL: https://issues.apache.org/jira/browse/HBASE-19533 > Project: HBase > Issue Type: Task >Reporter: stack >Priority: Critical > Fix For: 2.0.0 > > > Before HBASE-18946, setting shutdown of a cluster, the Master would exit > immediately. RegionServers would run region closes and then try and notify > the Master of the close and would spew exceptions that the Master was > unreachable. > This is different to how branch-1 used to do it. It used to keep Master up > and it would be like the captain of the ship, the last to go down. As of > HBASE-18946, this is again the case but there are still open issues. > # Usually Master does all open and close of regions. On cluster shutdown, it > is the one time where the Regions run the region close. Currently, the > regions report the close to the Master which disregards the message since it > did not start the region closes. Should we do different? Try and update state > in hbase:meta setting it to CLOSE? We might not be able to write CLOSE for > all regions since hbase:meta will be closing too (the RS that is hosting > hbase:meta will close it last but that may not be enough). > # Should the Master run the cluster shutdown sending out close for all > regions? What if cluster of 1M regions? Untenable? Send a message per server? > That might be better. > Anyways, this needs attention. Filing issue in meantime. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19945) Make TestRSGroups LargeTest to prevent timeout
[
https://issues.apache.org/jira/browse/HBASE-19945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-19945:
---
Resolution: Won't Fix
Assignee: (was: Ted Yu)
Status: Resolved (was: Patch Available)
Timeout is 10 minutes which should be long enough.
> Make TestRSGroups LargeTest to prevent timeout
> --
>
> Key: HBASE-19945
> URL: https://issues.apache.org/jira/browse/HBASE-19945
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Priority: Major
> Attachments: 19945.v1.txt, 19945.v2.txt
>
>
> TestRSGroups is annotated as MediumTests. It times out on Jenkins:
> https://builds.apache.org/job/HBase-Trunk_matrix/4537/jdk=JDK%201.8%20(latest),label=(Hadoop%20&&%20!H5)/testReport/junit/org.apache.hadoop.hbase.rsgroup/TestRSGroups/org_apache_hadoop_hbase_rsgroup_TestRSGroups/
> {code}
> org.junit.runners.model.TestTimedOutException: test timed out after 180
> seconds
> {code}
> The above is reproducible on Linux locally.
> TestRSGroups should be made LargeTest.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19844) Shell should support flush by regionserver
[ https://issues.apache.org/jira/browse/HBASE-19844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-19844: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: (was: 2.0.0) 2.0.0-beta-2 Status: Resolved (was: Patch Available) Thanks for the patch, Reid > Shell should support flush by regionserver > -- > > Key: HBASE-19844 > URL: https://issues.apache.org/jira/browse/HBASE-19844 > Project: HBase > Issue Type: New Feature > Components: shell >Reporter: Chia-Ping Tsai >Assignee: Reid Chan >Priority: Minor > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19844.master.001.patch, > HBASE-19844.master.002.patch, HBASE-19844.master.003.patch, > HBASE-19844.master.004.patch > > > HBASE-4224 add a method to admin that can do the flush by regionserver. As > with other Admin methods, we should enable shell to use the flush method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19930) fix ImmutableMemStoreLAB#forceCopyOfBigCellInto
[ https://issues.apache.org/jira/browse/HBASE-19930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gali Sheffi updated HBASE-19930: Attachment: HBASE-19930-V05.patch > fix ImmutableMemStoreLAB#forceCopyOfBigCellInto > --- > > Key: HBASE-19930 > URL: https://issues.apache.org/jira/browse/HBASE-19930 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-1 >Reporter: Gali Sheffi >Assignee: Gali Sheffi >Priority: Major > Attachments: HBASE-19930-V01.patch, HBASE-19930-V02.patch, > HBASE-19930-V03.patch, HBASE-19930-V04.patch, HBASE-19930-V05.patch > > > This issue is about fixing ImmutableMemStoreLAB#forceCopyOfBigCellInto. > Following a comment in HBASE-19133 regarding a bug in > ImmutableMemStoreLAB#forceCopyOfBigCellInto (assuming this method is never > called for an ImmutableMemStoreLAB, and just throwing an > IllegalStateException whenever called), the forceCopyOfBigCellInto method now > performs the copy of big cells on the first MSLABImpl in its mslabs > linked-list. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19844) Shell should support flush by regionserver
[ https://issues.apache.org/jira/browse/HBASE-19844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-19844: --- Summary: Shell should support flush by regionserver (was: Shell should support to flush by regionserver) > Shell should support flush by regionserver > -- > > Key: HBASE-19844 > URL: https://issues.apache.org/jira/browse/HBASE-19844 > Project: HBase > Issue Type: New Feature > Components: shell >Reporter: Chia-Ping Tsai >Assignee: Reid Chan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-19844.master.001.patch, > HBASE-19844.master.002.patch, HBASE-19844.master.003.patch, > HBASE-19844.master.004.patch > > > HBASE-4224 add a method to admin that can do the flush by regionserver. As > with other Admin methods, we should enable shell to use the flush method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19970) Remove unused functions from TableAuthManager
[ https://issues.apache.org/jira/browse/HBASE-19970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362386#comment-16362386 ] stack commented on HBASE-19970: --- Thanks [~ashish singhi]. [~appy] Reverted for now. It messes up our dashboard. https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests-branch2.0/lastSuccessfulBuild/artifact/dashboard.html > Remove unused functions from TableAuthManager > - > > Key: HBASE-19970 > URL: https://issues.apache.org/jira/browse/HBASE-19970 > Project: HBase > Issue Type: Task >Reporter: Appy >Assignee: Appy >Priority: Minor > Fix For: 1.5.0, 2.0.0-beta-2 > > Attachments: HBASE-19970.master.001.patch > > > Functions deleted in TableAuthManager: > - setTableUserPermissions > - setTableGroupPermissions > - setNamespaceUserPermissions > - setNamespaceGroupPermissions > - writeTableToZooKeeper > - writeNamespaceToZooKeeper > To make sure it was not a bug, and that relevant functionality moved to some > alternate code path, tried to find out why and when these functions went out > of use. But just couldn't figure out...until i reached the patch which added > them. Looks like they were dead functions to start with :) > Jira which added them: HBASE-8409. Commit id: > ac10b3c13d6b66e12d0c9601204b01dfa525ed19 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19945) Make TestRSGroups LargeTest to prevent timeout
[
https://issues.apache.org/jira/browse/HBASE-19945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362453#comment-16362453
]
stack commented on HBASE-19945:
---
bq. TestRSGroups is marked MediumTests, hence the 10 minute bound didn't apply.
How you make out that? See #getTimeoutInSeconds in HBaseClassTestRule
> Make TestRSGroups LargeTest to prevent timeout
> --
>
> Key: HBASE-19945
> URL: https://issues.apache.org/jira/browse/HBASE-19945
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 19945.v1.txt, 19945.v2.txt
>
>
> TestRSGroups is annotated as MediumTests. It times out on Jenkins:
> https://builds.apache.org/job/HBase-Trunk_matrix/4537/jdk=JDK%201.8%20(latest),label=(Hadoop%20&&%20!H5)/testReport/junit/org.apache.hadoop.hbase.rsgroup/TestRSGroups/org_apache_hadoop_hbase_rsgroup_TestRSGroups/
> {code}
> org.junit.runners.model.TestTimedOutException: test timed out after 180
> seconds
> {code}
> The above is reproducible on Linux locally.
> TestRSGroups should be made LargeTest.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19844) Shell should support to flush by regionserver
[ https://issues.apache.org/jira/browse/HBASE-19844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362408#comment-16362408 ] Reid Chan commented on HBASE-19844: --- :D, Thank you, Ted! Happy New Year~! > Shell should support to flush by regionserver > - > > Key: HBASE-19844 > URL: https://issues.apache.org/jira/browse/HBASE-19844 > Project: HBase > Issue Type: New Feature > Components: shell >Reporter: Chia-Ping Tsai >Assignee: Reid Chan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-19844.master.001.patch, > HBASE-19844.master.002.patch, HBASE-19844.master.003.patch, > HBASE-19844.master.004.patch > > > HBASE-4224 add a method to admin that can do the flush by regionserver. As > with other Admin methods, we should enable shell to use the flush method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19979) ReplicationSyncUp tool may leak Zookeeper connection
[
https://issues.apache.org/jira/browse/HBASE-19979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-19979:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Nice one [~pankaj2461] Good find. Pushed to branch-2 and master. Looks like
[~yuzhih...@gmail.com] pushed to branch-1.4 and branch-1 (Again, please use
--author param so you can accredit the patch properly Ted Yu).
> ReplicationSyncUp tool may leak Zookeeper connection
>
>
> Key: HBASE-19979
> URL: https://issues.apache.org/jira/browse/HBASE-19979
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 1.5.0, 2.0.0-beta-2, 1.4.2
>
> Attachments: HBASE-19979-branch-1.001.patch, HBASE-19979.patch
>
>
> ReplicationSyncUp tool may leak Zookeeper connection in the following code
> snippet,
> {code}
> try {
> int numberOfOldSource = 1; // default wait once
> while (numberOfOldSource > 0) {
> Thread.sleep(SLEEP_TIME);
> numberOfOldSource = manager.getOldSources().size();
> }
> } catch (InterruptedException e) {
> System.err.println("didn't wait long enough:" + e);
> return (-1);
> }
> manager.join();
> zkw.close();
> {code}
> ZooKeeperWatcher will not be closed in case of InterruptedException.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19953) Avoid calling post* hook when procedure fails
[
https://issues.apache.org/jira/browse/HBASE-19953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362622#comment-16362622
]
Josh Elser commented on HBASE-19953:
Let me take a look at that. Thanks for the pointer, sir.
> Avoid calling post* hook when procedure fails
> -
>
> Key: HBASE-19953
> URL: https://issues.apache.org/jira/browse/HBASE-19953
> Project: HBase
> Issue Type: Bug
> Components: master, proc-v2
>Reporter: Ramesh Mani
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-2
>
>
> Ramesh pointed out a case where I think we're mishandling some post\*
> MasterObserver hooks. Specifically, I'm looking at the deleteNamespace.
> We synchronously execute the DeleteNamespace procedure. When the user
> provides a namespace that isn't empty, the procedure does a rollback (which
> is just a no-op), but this doesn't propagate an exception up to the
> NonceProcedureRunnable in {{HMaster#deleteNamespace}}. It took Ramesh
> pointing it out a bit better to me that the code executes a bit differently
> than we actually expect.
> I think we need to double-check our post hooks and make sure we aren't
> invoking them when the procedure actually failed. cc/ [~Apache9], [~stack].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19930) fix ImmutableMemStoreLAB#forceCopyOfBigCellInto
[
https://issues.apache.org/jira/browse/HBASE-19930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362743#comment-16362743
]
Hadoop QA commented on HBASE-19930:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
2s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
5s{color} | {color:red} hbase-server: The patch generated 1 new + 16 unchanged
- 0 fixed = 17 total (was 16) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
1s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
20m 29s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}106m
36s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}146m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19930 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910401/HBASE-19930-V05.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 8ed3c1587fd0 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / ba402b1e7b |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/11510/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/11510/testReport/ |
| Max. process+thread count | 4974 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/11510/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was
[jira] [Created] (HBASE-19991) lots of hbase-rest test failures against hadoop 3
Mike Drob created HBASE-19991: - Summary: lots of hbase-rest test failures against hadoop 3 Key: HBASE-19991 URL: https://issues.apache.org/jira/browse/HBASE-19991 Project: HBase Issue Type: Bug Components: REST, test Reporter: Mike Drob Assignee: Mike Drob Fix For: 2.0.0 mvn clean test -pl hbase-rest -Dhadoop.profile=3.0 [ERROR] Tests run: 106, Failures: 95, Errors: 8, Skipped: 1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19116) Currently the tail of hfiles with CellComparator* classname makes it so hbase1 can't open hbase2 written hfiles; fix
[ https://issues.apache.org/jira/browse/HBASE-19116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19116: -- Status: Patch Available (was: Open) > Currently the tail of hfiles with CellComparator* classname makes it so > hbase1 can't open hbase2 written hfiles; fix > > > Key: HBASE-19116 > URL: https://issues.apache.org/jira/browse/HBASE-19116 > Project: HBase > Issue Type: Sub-task > Components: HFile, migration >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19116.branch-2.001.patch > > > See tail of HBASE-19052 for discussion which concludes we should try and make > it so operators do not have to go to latest hbase version before they > upgrade, at least if we can avoid it. > The necessary change of our default comparator from KV to Cell naming has > hfiles with tails that have the classname CellComparator in them in place of > KeyValueComparator. If an hbase1 tries to open them, it will fail not having > a CellComparator in its classpath (We have name of comparator in tail because > different files require different comparators... perhaps we write an alias > instead of a class one day... TODO). HBASE-16189 and HBASE-19052 are about > trying to carry knowledge of hbase2 back to hbase1, a brittle approach making > it so operators will have to upgrade to the latest branch-1 before they can > go to hbase2. > This issue is about undoing our writing of an incompatible (to hbase1) tail, > not unless we really have to (and it sounds like we could do without writing > an incompatible tail) to see if we can avoid requiring operators go to > lastest branch-1 (we may end up needing this but lets a have a really good > reason for it if we do). > Oh, let this filing be an answer to our [~anoop.hbase]'s old high-level > question over in HBASE-16189: > bq. ...means when rolling upgrade done to 2.0, first users have to upgrade to > some 1.x versions which is having this fix and then to 2.0.. What do you guys > think Whether we should avoid this kind of indirection? cc Enis Soztutar, > Stack, Ted Yu, Matteo Bertozzi > Yeah, lets try to avoid this if we can... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19977) FileMmapEngine allocation of byte buffers should be synchronized
[ https://issues.apache.org/jira/browse/HBASE-19977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362606#comment-16362606 ] Hudson commented on HBASE-19977: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4578 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4578/]) HBASE-19977 FileMmapEngine allocation of byte buffers should be (ramkrishna.s.vasudevan: rev 16f1f5b49424fcabc9b5c10882dab4f5bf7fa84b) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/FileMmapEngine.java > FileMmapEngine allocation of byte buffers should be synchronized > > > Key: HBASE-19977 > URL: https://issues.apache.org/jira/browse/HBASE-19977 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-alpha-4 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19977.patch > > > Recently we have been testing bucket cache with mmap mode. We found that > after the multi threading way of allocating the byte buffers for offheap > bucket cache, the creation of the mmapped byte buffers needs to be > synchronized for allocating the right sized mmapped buffers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19979) ReplicationSyncUp tool may leak Zookeeper connection
[
https://issues.apache.org/jira/browse/HBASE-19979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-19979:
--
Fix Version/s: 1.4.2
> ReplicationSyncUp tool may leak Zookeeper connection
>
>
> Key: HBASE-19979
> URL: https://issues.apache.org/jira/browse/HBASE-19979
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 1.5.0, 2.0.0-beta-2, 1.4.2
>
> Attachments: HBASE-19979-branch-1.001.patch, HBASE-19979.patch
>
>
> ReplicationSyncUp tool may leak Zookeeper connection in the following code
> snippet,
> {code}
> try {
> int numberOfOldSource = 1; // default wait once
> while (numberOfOldSource > 0) {
> Thread.sleep(SLEEP_TIME);
> numberOfOldSource = manager.getOldSources().size();
> }
> } catch (InterruptedException e) {
> System.err.println("didn't wait long enough:" + e);
> return (-1);
> }
> manager.join();
> zkw.close();
> {code}
> ZooKeeperWatcher will not be closed in case of InterruptedException.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19953) Avoid calling post* hook when procedure fails
[
https://issues.apache.org/jira/browse/HBASE-19953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362607#comment-16362607
]
stack commented on HBASE-19953:
---
Looking at related Master-side Operations, I see them take a latch in the
NonceProcedureRunnable implementation. When latch is thrown, they call the post
op. See enableTable, createTable, etc. This delete namespace should do similar?
Later we should come back and get rid of all these latches (and then we'll have
to figure how Observer can monitor Procedure).
> Avoid calling post* hook when procedure fails
> -
>
> Key: HBASE-19953
> URL: https://issues.apache.org/jira/browse/HBASE-19953
> Project: HBase
> Issue Type: Bug
> Components: master, proc-v2
>Reporter: Ramesh Mani
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-2
>
>
> Ramesh pointed out a case where I think we're mishandling some post\*
> MasterObserver hooks. Specifically, I'm looking at the deleteNamespace.
> We synchronously execute the DeleteNamespace procedure. When the user
> provides a namespace that isn't empty, the procedure does a rollback (which
> is just a no-op), but this doesn't propagate an exception up to the
> NonceProcedureRunnable in {{HMaster#deleteNamespace}}. It took Ramesh
> pointing it out a bit better to me that the code executes a bit differently
> than we actually expect.
> I think we need to double-check our post hooks and make sure we aren't
> invoking them when the procedure actually failed. cc/ [~Apache9], [~stack].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19116) Currently the tail of hfiles with CellComparator* classname makes it so hbase1 can't open hbase2 written hfiles; fix
[ https://issues.apache.org/jira/browse/HBASE-19116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19116: -- Attachment: HBASE-19116.branch-2.001.patch > Currently the tail of hfiles with CellComparator* classname makes it so > hbase1 can't open hbase2 written hfiles; fix > > > Key: HBASE-19116 > URL: https://issues.apache.org/jira/browse/HBASE-19116 > Project: HBase > Issue Type: Sub-task > Components: HFile, migration >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19116.branch-2.001.patch > > > See tail of HBASE-19052 for discussion which concludes we should try and make > it so operators do not have to go to latest hbase version before they > upgrade, at least if we can avoid it. > The necessary change of our default comparator from KV to Cell naming has > hfiles with tails that have the classname CellComparator in them in place of > KeyValueComparator. If an hbase1 tries to open them, it will fail not having > a CellComparator in its classpath (We have name of comparator in tail because > different files require different comparators... perhaps we write an alias > instead of a class one day... TODO). HBASE-16189 and HBASE-19052 are about > trying to carry knowledge of hbase2 back to hbase1, a brittle approach making > it so operators will have to upgrade to the latest branch-1 before they can > go to hbase2. > This issue is about undoing our writing of an incompatible (to hbase1) tail, > not unless we really have to (and it sounds like we could do without writing > an incompatible tail) to see if we can avoid requiring operators go to > lastest branch-1 (we may end up needing this but lets a have a really good > reason for it if we do). > Oh, let this filing be an answer to our [~anoop.hbase]'s old high-level > question over in HBASE-16189: > bq. ...means when rolling upgrade done to 2.0, first users have to upgrade to > some 1.x versions which is having this fix and then to 2.0.. What do you guys > think Whether we should avoid this kind of indirection? cc Enis Soztutar, > Stack, Ted Yu, Matteo Bertozzi > Yeah, lets try to avoid this if we can... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19988) HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while waiting for a row lock
[
https://issues.apache.org/jira/browse/HBASE-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362989#comment-16362989
]
Umesh Agashe commented on HBASE-19988:
--
It was logging following exception... several times!
{code:java}
2018-02-10 04:24:25,503 WARN [PutThread] regionserver.HRegion(5636): Thread
interrupted waiting for lock on row: row0
2018-02-10 04:24:25,503 WARN [PutThread]
regionserver.HRegion$BatchOperation(3173): Failed getting lock, row=row0
java.io.InterruptedIOException
at
org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5637)
at
org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.lockRowsAndBuildMiniBatch(HRegion.java:3168)
at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3837)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3810)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3741)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3732)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3746)
at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4074)
at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:2925)
at
org.apache.hadoop.hbase.regionserver.TestHRegion$PutThread.run(TestHRegion.java:3891)
Caused by: java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1326)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.tryLock(ReentrantReadWriteLock.java:871)
at
org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5621)
... 9 more{code}
There is a loop in the write batch path:
{code:java}
while (!batchOp.isDone()) {
doMiniBatchMutate(batchOp);
}{code}
This loop essentially, tries to acquire locks on as many rows in a batch as
possible and creates a mini-batch of those rows to write. Next time, locks are
acquired from last row (row for which previous iteration failed to acquire a
lock) on till the entire batch is written.
The operation was aborted/ stopped only on Timeout exception. All other
exceptions were logged and ignored to resume creating and writing mini-batches
for an input batch.
In this particular case, getRowLockInternal() used to fail with exception
InterruptedIOException caused by surefire (possibly due to test timeout). This
exception was ignored to proceed with write operation containing locked rows so
far. This was causing continuous calls to doMinibatchMutate() in a loop,
filling up the logs.
> HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while
> waiting for a row lock
> ---
>
> Key: HBASE-19988
> URL: https://issues.apache.org/jira/browse/HBASE-19988
> Project: HBase
> Issue Type: Improvement
> Components: amv2
>Affects Versions: 2.0.0-beta-1
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
>Priority: Minor
> Fix For: 2.0.0-beta-2
>
> Attachments: hbase-19988.master.001.patch
>
>
> See HBASE-19970, TestHRegionWithInMemoryFlush created 4.2g log file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19989) READY_TO_MERGE and READY_TO_SPLIT do not update region state correctly
[ https://issues.apache.org/jira/browse/HBASE-19989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362878#comment-16362878 ] Francis Liu commented on HBASE-19989: - [~yuzhih...@gmail.com] This is a bug in zkless assignment there used to be tests but they were removed. We'll include the zkless split tests in this patch. We've already been running the tests and this patch in prod. We'll work on adding back the rest of the zkless tests as part of HBASE-14626. > READY_TO_MERGE and READY_TO_SPLIT do not update region state correctly > -- > > Key: HBASE-19989 > URL: https://issues.apache.org/jira/browse/HBASE-19989 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1, 1.4.1 >Reporter: Ben Lau >Assignee: Ben Lau >Priority: Major > Attachments: HBASE-19989.patch > > > Region state transitions do not work correctly for READY_TO_MERGE/SPLIT. > [~thiruvel] and I noticed this is due to break statements being in the wrong > place in AssignmentManager. This allows a race condition for example in > which one of the regions being merged could be moved concurrently, resulting > in the merge transaction failing and then double assignment and/or dataloss. > This bug appears to only affect branch-1 (for example 1.3 and 1.4) and not > branch-2 as the relevant code in AM has since been rewritten. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19166) Add translation for handling hbase.regionserver.wal.WALEdit
[
https://issues.apache.org/jira/browse/HBASE-19166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362929#comment-16362929
]
stack commented on HBASE-19166:
---
Regards the description, a WALPlayer from hbase1 trying to read a hbase2 WAL,
just use an hbase2 WALPlayer to do the job.
On an hbase1 splitting hbase2 logs and failing as per the above, that might be
ok; it just means we need to add more RegionServers to the cluster of
hbase2-type that can split the logs. Need to plan rolling upgrade. That'll tell
us if we need this facility or not. Meantime moving out of beta-2.
> Add translation for handling hbase.regionserver.wal.WALEdit
> ---
>
> Key: HBASE-19166
> URL: https://issues.apache.org/jira/browse/HBASE-19166
> Project: HBase
> Issue Type: Bug
> Components: wal
>Reporter: Ted Yu
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
>
> For hlog generated by 1.x, using WALPlayer from hbase2 would result in:
> {code}
> 2017-11-02 21:22:40,907 INFO [main] mapreduce.Job: Task Id :
> attempt_1509641483571_0003_m_00_0, Status : FAILED
> Error: java.lang.ClassCastException:
> org.apache.hadoop.hbase.regionserver.wal.WALEdit cannot be cast to
> org.apache.hadoop.hbase.wal.WALEdit
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALCellMapper.map(WALPlayer.java:143)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
> {code}
> HBASE-16479 relocated WALEdit.
> Chatting with Enis, he mentioned adding translation for handling
> hbase.regionserver.wal.WALEdit
> This way, WAL from 1.x can be recognized by hbase-2
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-19767) Master web UI shows negative values for Remaining KVs
[ https://issues.apache.org/jira/browse/HBASE-19767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Umesh Agashe reassigned HBASE-19767: Assignee: Umesh Agashe > Master web UI shows negative values for Remaining KVs > - > > Key: HBASE-19767 > URL: https://issues.apache.org/jira/browse/HBASE-19767 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-alpha-4 >Reporter: Jean-Marc Spaggiari >Assignee: Umesh Agashe >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: Screen Shot 2018-01-12 at 12.18.41 PM.png > > > In the Master Web UI, under the compaction tab, the Remaining KVs sometimes > shows negative values. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19116) Currently the tail of hfiles with CellComparator* classname makes it so hbase1 can't open hbase2 written hfiles; fix
[
https://issues.apache.org/jira/browse/HBASE-19116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363010#comment-16363010
]
Hadoop QA commented on HBASE-19116:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
37s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
8s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
7s{color} | {color:red} hbase-server: The patch generated 4 new + 18 unchanged
- 2 fixed = 22 total (was 20) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
10s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
14m 41s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}103m
11s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}134m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:9f2f2db |
| JIRA Issue | HBASE-19116 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910426/HBASE-19116.branch-2.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux be4b69301fc8 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | branch-2 / 4594f7156d |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/11511/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/11511/testReport/ |
| Max. process+thread count | 4974 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/11511/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This
[jira] [Commented] (HBASE-19989) READY_TO_MERGE and READY_TO_SPLIT do not update region state correctly
[ https://issues.apache.org/jira/browse/HBASE-19989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362886#comment-16362886 ] Ted Yu commented on HBASE-19989: Thanks for the update. Happy New Year, Francis and Ben. > READY_TO_MERGE and READY_TO_SPLIT do not update region state correctly > -- > > Key: HBASE-19989 > URL: https://issues.apache.org/jira/browse/HBASE-19989 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1, 1.4.1 >Reporter: Ben Lau >Assignee: Ben Lau >Priority: Major > Attachments: HBASE-19989.patch > > > Region state transitions do not work correctly for READY_TO_MERGE/SPLIT. > [~thiruvel] and I noticed this is due to break statements being in the wrong > place in AssignmentManager. This allows a race condition for example in > which one of the regions being merged could be moved concurrently, resulting > in the merge transaction failing and then double assignment and/or dataloss. > This bug appears to only affect branch-1 (for example 1.3 and 1.4) and not > branch-2 as the relevant code in AM has since been rewritten. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19989) READY_TO_MERGE and READY_TO_SPLIT do not update region state correctly
[ https://issues.apache.org/jira/browse/HBASE-19989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362896#comment-16362896 ] Ted Yu commented on HBASE-19989: In the next patch, please add comment in the place of the previous break, explaining why the break is absent. > READY_TO_MERGE and READY_TO_SPLIT do not update region state correctly > -- > > Key: HBASE-19989 > URL: https://issues.apache.org/jira/browse/HBASE-19989 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1, 1.4.1 >Reporter: Ben Lau >Assignee: Ben Lau >Priority: Major > Attachments: HBASE-19989.patch > > > Region state transitions do not work correctly for READY_TO_MERGE/SPLIT. > [~thiruvel] and I noticed this is due to break statements being in the wrong > place in AssignmentManager. This allows a race condition for example in > which one of the regions being merged could be moved concurrently, resulting > in the merge transaction failing and then double assignment and/or dataloss. > This bug appears to only affect branch-1 (for example 1.3 and 1.4) and not > branch-2 as the relevant code in AM has since been rewritten. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19166) Add translation for handling hbase.regionserver.wal.WALEdit
[
https://issues.apache.org/jira/browse/HBASE-19166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362897#comment-16362897
]
stack commented on HBASE-19166:
---
hbase1 complaint is now:
{code}
1134718 2018-02-13 10:43:57,589 DEBUG [RS_LOG_REPLAY_OPS-ve0530:16020-0]
wal.WALSplitter: Finishing writing output logs and closing down.
1134719 2018-02-13 10:43:57,589 INFO [RS_LOG_REPLAY_OPS-ve0530:16020-0]
wal.WALSplitter: Processed 0 edits across 0 regions; edits skipped=0; log
file=hdfs://ve0524.halxg.cloudera.com:8020/hbase/WALs/ve0534.halxg.cloudera.com,16020,1518546984742-splitting/ve0534.halxg.cloudera.com%2C1
6020%2C1518546984742.meta.1518546993545.meta, length=23982,
corrupted=false, progress failed=false
1134720 2018-02-13 10:43:57,590 WARN [RS_LOG_REPLAY_OPS-ve0530:16020-0]
regionserver.SplitLogWorker: log splitting of
WALs/ve0534.halxg.cloudera.com,16020,1518546984742-splitting/ve0534.halxg.cloudera.com%2C16020%2C1518546984742.meta.1518546993545.meta
failed, returning error
1134721 java.io.IOException: Got unknown writer class: AsyncProtobufLogWriter
1134722 at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initInternal(ProtobufLogReader.java:220)
1134723 at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initReader(ProtobufLogReader.java:169)
1134724 at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:66)
1134725 at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:164)
1134726 at
org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:303)
1134727 at
org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:267)
1134728 at
org.apache.hadoop.hbase.wal.WALSplitter.getReader(WALSplitter.java:853)
1134729 at
org.apache.hadoop.hbase.wal.WALSplitter.getReader(WALSplitter.java:777)
1134730 at
org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:298)
1134731 at
org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:236)
1134732 at
org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:104)
1134733 at
org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:72)
1134734 at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
1134735 at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
1134736 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
1134737 at java.lang.Thread.run(Thread.java:748)
{code}
> Add translation for handling hbase.regionserver.wal.WALEdit
> ---
>
> Key: HBASE-19166
> URL: https://issues.apache.org/jira/browse/HBASE-19166
> Project: HBase
> Issue Type: Bug
> Components: wal
>Reporter: Ted Yu
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0-beta-2
>
>
> For hlog generated by 1.x, using WALPlayer from hbase2 would result in:
> {code}
> 2017-11-02 21:22:40,907 INFO [main] mapreduce.Job: Task Id :
> attempt_1509641483571_0003_m_00_0, Status : FAILED
> Error: java.lang.ClassCastException:
> org.apache.hadoop.hbase.regionserver.wal.WALEdit cannot be cast to
> org.apache.hadoop.hbase.wal.WALEdit
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALCellMapper.map(WALPlayer.java:143)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
> {code}
> HBASE-16479 relocated WALEdit.
> Chatting with Enis, he mentioned adding translation for handling
> hbase.regionserver.wal.WALEdit
> This way, WAL from 1.x can be recognized by hbase-2
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19166) Add translation for handling hbase.regionserver.wal.WALEdit
[
https://issues.apache.org/jira/browse/HBASE-19166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-19166:
--
Fix Version/s: (was: 2.0.0-beta-2)
2.0.0
> Add translation for handling hbase.regionserver.wal.WALEdit
> ---
>
> Key: HBASE-19166
> URL: https://issues.apache.org/jira/browse/HBASE-19166
> Project: HBase
> Issue Type: Bug
> Components: wal
>Reporter: Ted Yu
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
>
> For hlog generated by 1.x, using WALPlayer from hbase2 would result in:
> {code}
> 2017-11-02 21:22:40,907 INFO [main] mapreduce.Job: Task Id :
> attempt_1509641483571_0003_m_00_0, Status : FAILED
> Error: java.lang.ClassCastException:
> org.apache.hadoop.hbase.regionserver.wal.WALEdit cannot be cast to
> org.apache.hadoop.hbase.wal.WALEdit
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALCellMapper.map(WALPlayer.java:143)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
> {code}
> HBASE-16479 relocated WALEdit.
> Chatting with Enis, he mentioned adding translation for handling
> hbase.regionserver.wal.WALEdit
> This way, WAL from 1.x can be recognized by hbase-2
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19767) Master web UI shows negative values for Remaining KVs
[ https://issues.apache.org/jira/browse/HBASE-19767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362932#comment-16362932 ] Umesh Agashe commented on HBASE-19767: -- [~stack], I will pick this up. > Master web UI shows negative values for Remaining KVs > - > > Key: HBASE-19767 > URL: https://issues.apache.org/jira/browse/HBASE-19767 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-alpha-4 >Reporter: Jean-Marc Spaggiari >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: Screen Shot 2018-01-12 at 12.18.41 PM.png > > > In the Master Web UI, under the compaction tab, the Remaining KVs sometimes > shows negative values. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19970) Remove unused functions from TableAuthManager
[ https://issues.apache.org/jira/browse/HBASE-19970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363040#comment-16363040 ] Hudson commented on HBASE-19970: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4579 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4579/]) Revert "HBASE-19970 Remove unused functions from TableAuthManager." (stack: rev ba402b1e7b446144d4d20f90cb71e6aa19ecce3c) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestZKPermissionWatcher.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestTablePermissions.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java > Remove unused functions from TableAuthManager > - > > Key: HBASE-19970 > URL: https://issues.apache.org/jira/browse/HBASE-19970 > Project: HBase > Issue Type: Task >Reporter: Appy >Assignee: Appy >Priority: Minor > Fix For: 1.5.0, 2.0.0-beta-2 > > Attachments: HBASE-19970.master.001.patch > > > Functions deleted in TableAuthManager: > - setTableUserPermissions > - setTableGroupPermissions > - setNamespaceUserPermissions > - setNamespaceGroupPermissions > - writeTableToZooKeeper > - writeNamespaceToZooKeeper > To make sure it was not a bug, and that relevant functionality moved to some > alternate code path, tried to find out why and when these functions went out > of use. But just couldn't figure out...until i reached the patch which added > them. Looks like they were dead functions to start with :) > Jira which added them: HBASE-8409. Commit id: > ac10b3c13d6b66e12d0c9601204b01dfa525ed19 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19979) ReplicationSyncUp tool may leak Zookeeper connection
[
https://issues.apache.org/jira/browse/HBASE-19979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363042#comment-16363042
]
Hudson commented on HBASE-19979:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4579 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/4579/])
HBASE-19979 ReplicationSyncUp tool may leak Zookeeper connection (stack: rev
39e191e5598529c68007c96e69acdd923a294d33)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSyncUp.java
> ReplicationSyncUp tool may leak Zookeeper connection
>
>
> Key: HBASE-19979
> URL: https://issues.apache.org/jira/browse/HBASE-19979
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 1.5.0, 2.0.0-beta-2, 1.4.2
>
> Attachments: HBASE-19979-branch-1.001.patch, HBASE-19979.patch
>
>
> ReplicationSyncUp tool may leak Zookeeper connection in the following code
> snippet,
> {code}
> try {
> int numberOfOldSource = 1; // default wait once
> while (numberOfOldSource > 0) {
> Thread.sleep(SLEEP_TIME);
> numberOfOldSource = manager.getOldSources().size();
> }
> } catch (InterruptedException e) {
> System.err.println("didn't wait long enough:" + e);
> return (-1);
> }
> manager.join();
> zkw.close();
> {code}
> ZooKeeperWatcher will not be closed in case of InterruptedException.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19844) Shell should support flush by regionserver
[ https://issues.apache.org/jira/browse/HBASE-19844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363041#comment-16363041 ] Hudson commented on HBASE-19844: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4579 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4579/]) HBASE-19844 Shell should support to flush by regionserver (tedyu: rev 8e8e1e5a1bbb240a6f4a71bc8b0271d31da633b3) * (edit) hbase-shell/src/main/ruby/shell/commands/flush.rb * (edit) hbase-shell/src/test/ruby/hbase/admin_test.rb * (edit) hbase-shell/src/main/ruby/hbase/admin.rb > Shell should support flush by regionserver > -- > > Key: HBASE-19844 > URL: https://issues.apache.org/jira/browse/HBASE-19844 > Project: HBase > Issue Type: New Feature > Components: shell >Reporter: Chia-Ping Tsai >Assignee: Reid Chan >Priority: Minor > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19844.master.001.patch, > HBASE-19844.master.002.patch, HBASE-19844.master.003.patch, > HBASE-19844.master.004.patch > > > HBASE-4224 add a method to admin that can do the flush by regionserver. As > with other Admin methods, we should enable shell to use the flush method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19876) The exception happening in converting pb mutation to hbase.mutation messes up the CellScanner
[
https://issues.apache.org/jira/browse/HBASE-19876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363039#comment-16363039
]
Hudson commented on HBASE-19876:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4579 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/4579/])
HBASE-19876 The exception happening in converting pb mutation to (chia7712: rev
2f48fdbb26ff555485b4aa3393d835b7dd8797a0)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMalformedCellFromClient.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/RequestConverter.java
> The exception happening in converting pb mutation to hbase.mutation messes up
> the CellScanner
> -
>
> Key: HBASE-19876
> URL: https://issues.apache.org/jira/browse/HBASE-19876
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 1.3.2, 1.5.0, 1.2.7, 2.0.0-beta-2, 1.4.2
>
> Attachments: HBASE-19876.branch-1.2.v0.patch,
> HBASE-19876.master.001.patch, HBASE-19876.v0.patch, HBASE-19876.v1.patch,
> HBASE-19876.v2.patch, HBASE-19876.v3.patch, HBASE-19876.v3.patch,
> HBASE-19876.v3.patch, HBASE-19876.v3.patch, HBASE-19876.v4.patch,
> HBASE-19876.v5.patch, HBASE-19876.v6.patch
>
>
> {code:java}
> 2018-01-27 22:51:43,794 INFO [hconnection-0x3291b443-shared-pool11-t6]
> client.AsyncRequestFutureImpl(778): id=5, table=testQuotaStatusFromMaster3,
> attempt=6/16 failed=20ops, last
> exception=org.apache.hadoop.hbase.client.WrongRowIOException:
> org.apache.hadoop.hbase.client.WrongRowIOException: The row in xxx doesn't
> match the original one aaa
> at org.apache.hadoop.hbase.client.Mutation.add(Mutation.java:776)
> at org.apache.hadoop.hbase.client.Put.add(Put.java:282)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toPut(ProtobufUtil.java:642)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:952)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:896)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2591)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304){code}
> I noticed this bug when testing the table space quota.
> When rs are converting pb mutation to hbase.mutation, the quota exception or
> cell exception may be thrown.
> {code}
> Unable to find source-code formatter for language:
> rsrpcservices#dobatchop.java. Available languages are: actionscript, ada,
> applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html,
> java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r,
> rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml for
> (ClientProtos.Action action: mutations) {
> MutationProto m = action.getMutation();
> Mutation mutation;
> if (m.getMutateType() == MutationType.PUT) {
> mutation = ProtobufUtil.toPut(m, cells);
> batchContainsPuts = true;
> } else {
> mutation = ProtobufUtil.toDelete(m, cells);
> batchContainsDelete = true;
> }
> mutationActionMap.put(mutation, action);
> mArray[i++] = mutation;
> checkCellSizeLimit(region, mutation);
> // Check if a space quota disallows this mutation
> spaceQuotaEnforcement.getPolicyEnforcement(region).check(mutation);
> quota.addMutation(mutation);
> }
> {code}
> rs has caught the exception but it doesn't have the cellscanner skip the
> failed cells.
> {code:java}
> } catch (IOException ie) {
> if (atomic) {
> throw ie;
> }
> for (Action mutation : mutations) {
> builder.addResultOrException(getResultOrException(ie,
> mutation.getIndex()));
> }
> }
> {code}
> The bug results in the WrongRowIOException to remaining mutations since they
> refer to invalid cells.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19988) HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while waiting for a row lock
[ https://issues.apache.org/jira/browse/HBASE-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19988: -- Attachment: hbase-19988.master.001.patch > HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while > waiting for a row lock > --- > > Key: HBASE-19988 > URL: https://issues.apache.org/jira/browse/HBASE-19988 > Project: HBase > Issue Type: Improvement > Components: amv2 >Affects Versions: 2.0.0-beta-1 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Minor > Fix For: 2.0.0-beta-2 > > Attachments: hbase-19988.master.001.patch, > hbase-19988.master.001.patch > > > See HBASE-19970, TestHRegionWithInMemoryFlush created 4.2g log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19988) HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while waiting for a row lock
[ https://issues.apache.org/jira/browse/HBASE-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363051#comment-16363051 ] Umesh Agashe commented on HBASE-19988: -- Thanks [~stack]! Lets wait for what [~appy] has to say on this. > HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while > waiting for a row lock > --- > > Key: HBASE-19988 > URL: https://issues.apache.org/jira/browse/HBASE-19988 > Project: HBase > Issue Type: Improvement > Components: amv2 >Affects Versions: 2.0.0-beta-1 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Minor > Fix For: 2.0.0-beta-2 > > Attachments: hbase-19988.master.001.patch, > hbase-19988.master.001.patch > > > See HBASE-19970, TestHRegionWithInMemoryFlush created 4.2g log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19988) HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while waiting for a row lock
[ https://issues.apache.org/jira/browse/HBASE-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363052#comment-16363052 ] stack commented on HBASE-19988: --- Retry > HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while > waiting for a row lock > --- > > Key: HBASE-19988 > URL: https://issues.apache.org/jira/browse/HBASE-19988 > Project: HBase > Issue Type: Improvement > Components: amv2 >Affects Versions: 2.0.0-beta-1 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Minor > Fix For: 2.0.0-beta-2 > > Attachments: hbase-19988.master.001.patch, > hbase-19988.master.001.patch > > > See HBASE-19970, TestHRegionWithInMemoryFlush created 4.2g log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-19992) Hole in namespace table assign
stack created HBASE-19992:
-
Summary: Hole in namespace table assign
Key: HBASE-19992
URL: https://issues.apache.org/jira/browse/HBASE-19992
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: stack
If the assign fails before it comes up in a Master initialization, the table
will have been created and may even be marked ENABLED successfully, but on
restart, we don't assign the table.
Manifest is:
{code}
2018-02-13 11:45:24,504 ERROR [master/ve0524:16000] master.HMaster: Failed to
become active master
java.lang.IllegalStateException: Expected the service ClusterSchemaServiceImpl
[FAILED] to be RUNNING, but the service has FAILED
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:345)
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:291)
at
org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1052)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:916)
at
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2026)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:555)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Timedout 30ms waiting for namespace table
to be assigned and enabled: ENABLED
at
org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:107)
at
org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:62)
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:226)
at
org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1050)
... 4 more
2018-02-13 11:45:24,506 ERROR [master/ve0524:16000] master.HMaster: Master
server abort: loaded coprocessors are:
[org.apache.hadoop.hbase.security.access.AccessController]
2018-02-13 11:45:24,506 ERROR [master/ve0524:16000] master.HMaster: *
ABORTING master ve0524.halxg.cloudera.com,16000,1518550812400: Unhandled
exception. Starting shutdown. *
java.lang.IllegalStateException: Expected the service ClusterSchemaServiceImpl
[FAILED] to be RUNNING, but the service has FAILED
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:345)
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:291)
at
org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1052)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:916)
at
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2026)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:555)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException:
Timedout 30ms waiting for namespace table to be assigned and enabled:
ENABLED
at
org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:107)
at
org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:62)
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:226)
at
org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1050)
... 4 more
{code}
Last thing in log before Master crash was:
2018-02-13 11:34:17,084 INFO [master/ve0524:16000] hbase.MetaTableAccessor:
Updated table hbase:namespace state to ENABLED in META
There is no one doing an assign.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-19994) Create a new class for RPC throttling exception, make it retryable.
[
https://issues.apache.org/jira/browse/HBASE-19994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huaxiang sun reassigned HBASE-19994:
Assignee: huaxiang sun
> Create a new class for RPC throttling exception, make it retryable.
>
>
> Key: HBASE-19994
> URL: https://issues.apache.org/jira/browse/HBASE-19994
> Project: HBase
> Issue Type: Improvement
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
>
> Based on a discussion at dev mailing list.
>
> {code:java}
> Thanks Andrew.
> +1 for the second option, I will create a jira for this change.
> Huaxiang
> On Feb 9, 2018, at 1:09 PM, Andrew Purtell wrote:
> We have
> public class ThrottlingException extends QuotaExceededException
> public class QuotaExceededException extends DoNotRetryIOException
> Let the storage quota limits throw QuotaExceededException directly (based
> on DNRIOE). That seems fine.
> However, ThrottlingException is thrown as a result of a temporal quota,
> so it is inappropriate for this to inherit from DNRIOE, it should inherit
> IOException instead so the client is allowed to retry until successful, or
> until the retry policy is exhausted.
> We are in a bit of a pickle because we've released with this inheritance
> hierarchy, so to change it we will need a new minor, or we will want to
> deprecate ThrottlingException and use a new exception class instead, one
> which does not inherit from DNRIOE.
> On Feb 7, 2018, at 9:25 AM, Huaxiang Sun wrote:
> Hi Mike,
> You are right. For rpc throttling, definitely it is retryable. For storage
> quota, I think it will be fail faster (non-retryable).
> We probably need to separate these two types of exceptions, I will do some
> more research and follow up.
> Thanks,
> Huaxiang
> On Feb 7, 2018, at 9:16 AM, Mike Drob wrote:
> I think, philosophically, there can be two kinds of QEE -
> For throttling, we can retry. The quota is a temporal quota - you have done
> too many operations this minute, please try again next minute and
> everything will work.
> For storage, we shouldn't retry. The quota is a fixed quote - you have
> exceeded your allotted disk space, please do not try again until you have
> remedied the situation.
> Our current usage conflates the two, sometimes it is correct, sometimes not.
> On Wed, Feb 7, 2018 at 11:00 AM, Huaxiang Sun wrote:
> Hi Stack,
> I run into a case that a mapreduce job in hive cannot finish because
> it runs into a QEE.
> I need to look into the hive mr task to see if QEE is not handled
> correctly in hbase code or in hive code.
> I am thinking that if QEE is a retryable exception, then it should be
> taken care of by the hbase code.
> I will check more and report back.
> Thanks,
> Huaxiang
> On Feb 7, 2018, at 8:23 AM, Stack wrote:
> QEE being a DNRIOE seems right on the face of it.
> But if throttling, a DNRIOE is inappropriate. Where you seeing a QEE in a
> throttling scenario Huaxiang?
> Thanks,
> S
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19993) Publish tests jar for hbase-zookeeper in bin tarball
[
https://issues.apache.org/jira/browse/HBASE-19993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363155#comment-16363155
]
Appy commented on HBASE-19993:
--
eh? it's there in beta-1 bin tarball. How? Even though we are not copying it
explicitly like other tests jar
([https://github.com/apache/hbase/blob/branch-2/hbase-assembly/src/main/assembly/components.xml#L110])
> Publish tests jar for hbase-zookeeper in bin tarball
>
>
> Key: HBASE-19993
> URL: https://issues.apache.org/jira/browse/HBASE-19993
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
>Priority: Major
>
> Since {{HBTU extends HBZKTU}} (such short forms! i know!), we need to publish
> hbase-zookeeper's tests jar too. Many IT tests use HBTU.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19991) lots of hbase-rest test failures against hadoop 3
[ https://issues.apache.org/jira/browse/HBASE-19991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-19991: -- Attachment: HBASE-19991.WIP.patch > lots of hbase-rest test failures against hadoop 3 > - > > Key: HBASE-19991 > URL: https://issues.apache.org/jira/browse/HBASE-19991 > Project: HBase > Issue Type: Bug > Components: REST, test >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-19991.WIP.patch > > > mvn clean test -pl hbase-rest -Dhadoop.profile=3.0 > [ERROR] Tests run: 106, Failures: 95, Errors: 8, Skipped: 1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19991) lots of hbase-rest test failures against hadoop 3
[ https://issues.apache.org/jira/browse/HBASE-19991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363164#comment-16363164 ] Mike Drob commented on HBASE-19991: --- This is failing due to loading jersey-1 classes via hadoop in the hadoop-3 configuration. This patch is my WIP, but I don't see anything jersey-1 left in dependency:tree report. > lots of hbase-rest test failures against hadoop 3 > - > > Key: HBASE-19991 > URL: https://issues.apache.org/jira/browse/HBASE-19991 > Project: HBase > Issue Type: Bug > Components: REST, test >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Fix For: 2.0.0 > > > mvn clean test -pl hbase-rest -Dhadoop.profile=3.0 > [ERROR] Tests run: 106, Failures: 95, Errors: 8, Skipped: 1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
