[jira] [Commented] (HBASE-18038) Rename StoreFile to HStoreFile and add a StoreFile interface for CP
[ https://issues.apache.org/jira/browse/HBASE-18038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011785#comment-16011785 ] Duo Zhang commented on HBASE-18038: --- Let me check the failed UTs. > Rename StoreFile to HStoreFile and add a StoreFile interface for CP > --- > > Key: HBASE-18038 > URL: https://issues.apache.org/jira/browse/HBASE-18038 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors, regionserver >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18038.patch, HBASE-18038-v1.patch, > HBASE-18038-v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-11013) Clone Snapshots on Secure Cluster Should provide option to apply Retained User Permissions
[
https://issues.apache.org/jira/browse/HBASE-11013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011782#comment-16011782
]
Hadoop QA commented on HBASE-11013:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m 26s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m 0s
{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m 0s
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 16 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 44s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 40s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 16m
2s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
57s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 55s
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 16m
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 58s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 28s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 29s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 110m 39s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 8s
{color} | {color:green} hbase-rsgroup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m 17s
{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
0s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 222m 23s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
[jira] [Commented] (HBASE-18049) It is not necessary to re-open the region when MOB files cannot be found
[
https://issues.apache.org/jira/browse/HBASE-18049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011778#comment-16011778
]
Anoop Sam John commented on HBASE-18049:
bq.Right, this is a simple solution which can probably fix the issue. But I
think users need to investigate the cases a little before running such
commands. Given such info in logs might mislead users to directly run the
compaction? I think we'd better not add this for now?
Ok. Just asked as recently some users faced this.
> It is not necessary to re-open the region when MOB files cannot be found
>
>
> Key: HBASE-18049
> URL: https://issues.apache.org/jira/browse/HBASE-18049
> Project: HBase
> Issue Type: Bug
> Components: Scanners
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Fix For: 2.0.0
>
> Attachments: HBASE-18049.patch
>
>
> In HBASE-17712, we try to re-open the region when store files cannot be
> found. This is useful for store files in a region, but is not necessary when
> the MOB files cannot be found, because the store files in a region only
> contain the references to the MOB files and a re-open of a region doesn't
> help the lost MOB files.
> In this JIRA, we will directly throw DNRIOE only when the MOB files are not
> found in {{MobStoreScanner}} and {{ReversedMobStoreScanner}}. Other logics
> keep the same.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18038) Rename StoreFile to HStoreFile and add a StoreFile interface for CP
[
https://issues.apache.org/jira/browse/HBASE-18038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011774#comment-16011774
]
Hadoop QA commented on HBASE-18038:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 30s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 38 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
57s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 27s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
60m 0s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 162m 13s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
54s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 252m 56s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestRegionReplicaFailover |
| | hadoop.hbase.security.access.TestAccessController |
| | hadoop.hbase.mob.compactions.TestMobCompactor |
| | hadoop.hbase.security.token.TestZKSecretWatcher |
| | hadoop.hbase.regionserver.TestMobStoreCompaction |
| | hadoop.hbase.regionserver.TestEncryptionKeyRotation |
| | hadoop.hbase.regionserver.TestPerColumnFamilyFlush |
| | hadoop.hbase.mob.compactions.TestPartitionedMobCompactor |
| | hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster |
| | hadoop.hbase.regionserver.TestCorruptedRegionStoreFile |
| | hadoop.hbase.regionserver.TestReversibleScanners |
| Timed out junit tests |
org.apache.hadoop.hbase.snapshot.TestMobSecureExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestExportSnapshot |
| | org.apache.hadoop.hbase.TestIOFencing |
| | org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12868202/HBASE-18038-v1.patch |
| JIRA Issue | HBASE-18038 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux ee9db86a07eb 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016
x86_64
[jira] [Commented] (HBASE-18049) It is not necessary to re-open the region when MOB files cannot be found
[
https://issues.apache.org/jira/browse/HBASE-18049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011767#comment-16011767
]
Jingcheng Du commented on HBASE-18049:
--
Thanks [~yuzhih...@gmail.com] and [~anoop.hbase] for the comments!
bq. So here in new change as well, we need getCause() check?
Right, we should add this.
bq. A solution to restore the state and data is to do a MOB major compaction
right?
Right, this is a simple solution which can probably fix the issue. But I think
users need to investigate the cases a little before running such commands.
Given such info in logs might mislead users to directly run the compaction? I
think we'd better not add this for now?
> It is not necessary to re-open the region when MOB files cannot be found
>
>
> Key: HBASE-18049
> URL: https://issues.apache.org/jira/browse/HBASE-18049
> Project: HBase
> Issue Type: Bug
> Components: Scanners
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Fix For: 2.0.0
>
> Attachments: HBASE-18049.patch
>
>
> In HBASE-17712, we try to re-open the region when store files cannot be
> found. This is useful for store files in a region, but is not necessary when
> the MOB files cannot be found, because the store files in a region only
> contain the references to the MOB files and a re-open of a region doesn't
> help the lost MOB files.
> In this JIRA, we will directly throw DNRIOE only when the MOB files are not
> found in {{MobStoreScanner}} and {{ReversedMobStoreScanner}}. Other logics
> keep the same.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18040) TestBlockEvictionFromClient fails in master branch
[
https://issues.apache.org/jira/browse/HBASE-18040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011766#comment-16011766
]
ramkrishna.s.vasudevan commented on HBASE-18040:
Found the issue. Will raise a new JIRA to fix it. It needs some discussion.
Will link this failure to the other new JIRA.
> TestBlockEvictionFromClient fails in master branch
> --
>
> Key: HBASE-18040
> URL: https://issues.apache.org/jira/browse/HBASE-18040
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: ramkrishna.s.vasudevan
>
> According to
> https://builds.apache.org/job/HBASE-Flaky-Tests/16000/#showFailuresLink , the
> test first failed with commit 0ae0edcd630aa1dcb6c47ea11fa4367ae0a5baa8
> On Linux, I got the following failures:
> {code}
> testParallelGetsAndScans(org.apache.hadoop.hbase.client.TestBlockEvictionFromClient)
> Time elapsed: 3.016 sec <<< FAILURE!
> java.lang.AssertionError: expected:<3> but was:<6>
> at
> org.apache.hadoop.hbase.client.TestBlockEvictionFromClient.checkForBlockEviction(TestBlockEvictionFromClient.java:1308)
> at
> org.apache.hadoop.hbase.client.TestBlockEvictionFromClient.testParallelGetsAndScans(TestBlockEvictionFromClient.java:293)
> testBlockRefCountAfterSplits(org.apache.hadoop.hbase.client.TestBlockEvictionFromClient)
> Time elapsed: 7.786 sec <<< FAILURE!
> java.lang.AssertionError: expected:<0> but was:<1>
> at
> org.apache.hadoop.hbase.client.TestBlockEvictionFromClient.iterateBlockCache(TestBlockEvictionFromClient.java:1215)
> at
> org.apache.hadoop.hbase.client.TestBlockEvictionFromClient.testBlockRefCountAfterSplits(TestBlockEvictionFromClient.java:607)
> testParallelGetsAndScanWithWrappedRegionScanner(org.apache.hadoop.hbase.client.TestBlockEvictionFromClient)
> Time elapsed: 2.631 sec <<< FAILURE!
> java.lang.AssertionError: expected:<3> but was:<6>
> at
> org.apache.hadoop.hbase.client.TestBlockEvictionFromClient.checkForBlockEviction(TestBlockEvictionFromClient.java:1322)
> at
> org.apache.hadoop.hbase.client.TestBlockEvictionFromClient.testParallelGetsAndScanWithWrappedRegionScanner(TestBlockEvictionFromClient.java:839)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18053) AsyncTableResultScanner will hang when scan wrong column family
[
https://issues.apache.org/jira/browse/HBASE-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011760#comment-16011760
]
Hadoop QA commented on HBASE-18053:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 37s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 49s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 41m 25s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 95m 12s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.10.1 Server=1.10.1 Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12868221/HBASE-18053.patch |
| JIRA Issue | HBASE-18053 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux c59267d594b2 3.19.0-25-generic #26~14.04.1-Ubuntu SMP Fri Jul
24 21:16:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 3765077 |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6801/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/6801/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
[jira] [Commented] (HBASE-18026) ProtobufUtil seems to do extra array copying
[ https://issues.apache.org/jira/browse/HBASE-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011757#comment-16011757 ] Hudson commented on HBASE-18026: SUCCESS: Integrated in Jenkins build HBase-1.3-JDK8 #178 (See [https://builds.apache.org/job/HBase-1.3-JDK8/178/]) Amend HBASE-18026 ProtobufUtil seems to do extra array copying (apurtell: rev d18eb62d8d0f02ea822090f409c2ded6f5279c3d) * (edit) hbase-protocol/src/main/java/com/google/protobuf/HBaseZeroCopyByteString.java > ProtobufUtil seems to do extra array copying > > > Key: HBASE-18026 > URL: https://issues.apache.org/jira/browse/HBASE-18026 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Minor > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-18026.addendum.patch, > HBASE-18026.branch-1.v1.patch, HBASE-18026.master.v1.patch > > > In ProtobufUtil, the protobuf fields are copied into an array using > toByteArray(). These are then passed into the KeyValue constructor which > does another copy. > It seems like we can avoid a copy here by using > HBaseZeroCopyByteString#zeroCopyGetBytes() ? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18051) balance_rsgroup still run when the Load Balancer is not enabled.
[ https://issues.apache.org/jira/browse/HBASE-18051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated HBASE-18051: -- Attachment: HBASE-18051-v3.patch > balance_rsgroup still run when the Load Balancer is not enabled. > > > Key: HBASE-18051 > URL: https://issues.apache.org/jira/browse/HBASE-18051 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 2.0.0 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Fix For: 2.0.0 > > Attachments: HBASE-18051-v1.patch, HBASE-18051-v2.patch, > HBASE-18051-v3.patch > > > When the Load Balancer is not enabled, the command "balancer" will not be > executed , but the command "balance_rsgroup" can still run, which is not > reasonable.So we need to fix it -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18051) balance_rsgroup still run when the Load Balancer is not enabled.
[ https://issues.apache.org/jira/browse/HBASE-18051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011747#comment-16011747 ] Guangxu Cheng commented on HBASE-18051: --- v3: Add a unit test and fix the failed tests. > balance_rsgroup still run when the Load Balancer is not enabled. > > > Key: HBASE-18051 > URL: https://issues.apache.org/jira/browse/HBASE-18051 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 2.0.0 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Fix For: 2.0.0 > > Attachments: HBASE-18051-v1.patch, HBASE-18051-v2.patch, > HBASE-18051-v3.patch > > > When the Load Balancer is not enabled, the command "balancer" will not be > executed , but the command "balance_rsgroup" can still run, which is not > reasonable.So we need to fix it -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18027) Replication should respect RPC size limits when batching edits
[
https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011745#comment-16011745
]
Hadoop QA commented on HBASE-18027:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 31m 44s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 15s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
58m 28s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 52s
{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 176m 44s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
35s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 293m 53s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Inconsistent synchronization of
org.apache.hadoop.hbase.replication.regionserver.ReplicationSinkManager.sinks;
locked 72% of time Unsynchronized access at ReplicationSinkManager.java:72% of
time Unsynchronized access at ReplicationSinkManager.java:[line 175] |
| Failed junit tests | hadoop.hbase.client.TestAsyncBalancerAdminApi |
| | hadoop.hbase.client.TestMobSnapshotCloneIndependence |
| | hadoop.hbase.client.TestAsyncTableAdminApi |
| | hadoop.hbase.security.access.TestCoprocessorWhitelistMasterObserver |
| Timed out junit tests |
org.apache.hadoop.hbase.replication.regionserver.TestWALEntryStream |
| | org.apache.hadoop.hbase.client.TestFromClientSide3 |
| | org.apache.hadoop.hbase.snapshot.TestExportSnapshot |
| | org.apache.hadoop.hbase.client.TestHCM |
| | org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClient |
| |
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas |
| | org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12868182/HBASE-18027.patch |
| JIRA Issue | HBASE-18027 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 565cb4f8f76d 4.8.3-std-1 #1 SMP Fri Oct
[jira] [Commented] (HBASE-18026) ProtobufUtil seems to do extra array copying
[ https://issues.apache.org/jira/browse/HBASE-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011739#comment-16011739 ] Hudson commented on HBASE-18026: SUCCESS: Integrated in Jenkins build HBase-1.3-JDK7 #164 (See [https://builds.apache.org/job/HBase-1.3-JDK7/164/]) Amend HBASE-18026 ProtobufUtil seems to do extra array copying (apurtell: rev d18eb62d8d0f02ea822090f409c2ded6f5279c3d) * (edit) hbase-protocol/src/main/java/com/google/protobuf/HBaseZeroCopyByteString.java > ProtobufUtil seems to do extra array copying > > > Key: HBASE-18026 > URL: https://issues.apache.org/jira/browse/HBASE-18026 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Minor > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-18026.addendum.patch, > HBASE-18026.branch-1.v1.patch, HBASE-18026.master.v1.patch > > > In ProtobufUtil, the protobuf fields are copied into an array using > toByteArray(). These are then passed into the KeyValue constructor which > does another copy. > It seems like we can avoid a copy here by using > HBaseZeroCopyByteString#zeroCopyGetBytes() ? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18035) Meta replica does not give any primaryOperationTimeout to primary meta region
[
https://issues.apache.org/jira/browse/HBASE-18035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011737#comment-16011737
]
Hadoop QA commented on HBASE-18035:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m 52s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 47s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 22s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
32m 32s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 56s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 30s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 123m 54s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
0s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 199m 51s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestMobStoreScanner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12868201/HBASE-18035-master-v001.patch
|
| JIRA Issue | HBASE-18035 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux af9a95d24e3e 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 3765077 |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6798/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
[jira] [Commented] (HBASE-18026) ProtobufUtil seems to do extra array copying
[ https://issues.apache.org/jira/browse/HBASE-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011732#comment-16011732 ] Hudson commented on HBASE-18026: SUCCESS: Integrated in Jenkins build HBase-1.2-JDK7 #135 (See [https://builds.apache.org/job/HBase-1.2-JDK7/135/]) Amend HBASE-18026 ProtobufUtil seems to do extra array copying (apurtell: rev 719ab32cc6910cf393ce235315190ffd8dd7a3c8) * (edit) hbase-protocol/src/main/java/com/google/protobuf/HBaseZeroCopyByteString.java > ProtobufUtil seems to do extra array copying > > > Key: HBASE-18026 > URL: https://issues.apache.org/jira/browse/HBASE-18026 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Minor > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-18026.addendum.patch, > HBASE-18026.branch-1.v1.patch, HBASE-18026.master.v1.patch > > > In ProtobufUtil, the protobuf fields are copied into an array using > toByteArray(). These are then passed into the KeyValue constructor which > does another copy. > It seems like we can avoid a copy here by using > HBaseZeroCopyByteString#zeroCopyGetBytes() ? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18051) balance_rsgroup still run when the Load Balancer is not enabled.
[
https://issues.apache.org/jira/browse/HBASE-18051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011729#comment-16011729
]
Hadoop QA commented on HBASE-18051:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 40s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 19s {color}
| {color:red} hbase-rsgroup in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 25s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.rsgroup.TestRSGroups |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12868215/HBASE-18051-v2.patch |
| JIRA Issue | HBASE-18051 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux d7190214c3fa 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 3765077 |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6800/artifact/patchprocess/patch-unit-hbase-rsgroup.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/6800/artifact/patchprocess/patch-unit-hbase-rsgroup.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6800/testReport/ |
| modules | C: hbase-rsgroup U: hbase-rsgroup |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6800/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> balance_rsgroup still run
[jira] [Updated] (HBASE-11013) Clone Snapshots on Secure Cluster Should provide option to apply Retained User Permissions
[
https://issues.apache.org/jira/browse/HBASE-11013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-11013:
-
Attachment: HBASE-11013.branch-1.v1.patch
> Clone Snapshots on Secure Cluster Should provide option to apply Retained
> User Permissions
> --
>
> Key: HBASE-11013
> URL: https://issues.apache.org/jira/browse/HBASE-11013
> Project: HBase
> Issue Type: Improvement
> Components: snapshots
>Reporter: Ted Yu
>Assignee: Zheng Hu
> Fix For: 2.0.0
>
> Attachments: HBASE-11013.branch-1.v1.patch,
> HBASE-11013.branch-1.v1.patch, HBASE-11013.master.addendum.patch,
> HBASE-11013.v1.patch, HBASE-11013.v2.patch, HBASE-11013.v3.patch,
> HBASE-11013.v3.patch, HBASE-11013.v4.patch, HBASE-11013.v4.patch
>
>
> Currently,
> {code}
> sudo su - test_user
> create 't1', 'f1'
> sudo su - hbase
> snapshot 't1', 'snap_one'
> clone_snapshot 'snap_one', 't2'
> {code}
> In this scenario the user - test_user would not have permissions for the
> clone table t2.
> We need to add improvement feature such that the permissions of the original
> table are recorded in snapshot metadata and an option is provided for
> applying them to the new table as part of the clone process.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18053) AsyncTableResultScanner will hang when scan wrong column family
[ https://issues.apache.org/jira/browse/HBASE-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011724#comment-16011724 ] Ted Yu commented on HBASE-18053: lgtm, pending QA > AsyncTableResultScanner will hang when scan wrong column family > --- > > Key: HBASE-18053 > URL: https://issues.apache.org/jira/browse/HBASE-18053 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18053.patch, HBASE-18053-v1.patch > > > AsynTableResultScanner did not call notify(). So the next will hang on > wait(). It is easy to fix. And will add a ut for this. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-11013) Clone Snapshots on Secure Cluster Should provide option to apply Retained User Permissions
[
https://issues.apache.org/jira/browse/HBASE-11013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011725#comment-16011725
]
Zheng Hu commented on HBASE-11013:
--
[~tedyu], The main differences is: branch-1 has no hbase-protocol-shaded
sub-module, so we need not to move AccessControl.proto to
hbase-protocol-shaded directory, it saves lots of generated code.
> Clone Snapshots on Secure Cluster Should provide option to apply Retained
> User Permissions
> --
>
> Key: HBASE-11013
> URL: https://issues.apache.org/jira/browse/HBASE-11013
> Project: HBase
> Issue Type: Improvement
> Components: snapshots
>Reporter: Ted Yu
>Assignee: Zheng Hu
> Fix For: 2.0.0
>
> Attachments: HBASE-11013.branch-1.v1.patch,
> HBASE-11013.master.addendum.patch, HBASE-11013.v1.patch,
> HBASE-11013.v2.patch, HBASE-11013.v3.patch, HBASE-11013.v3.patch,
> HBASE-11013.v4.patch, HBASE-11013.v4.patch
>
>
> Currently,
> {code}
> sudo su - test_user
> create 't1', 'f1'
> sudo su - hbase
> snapshot 't1', 'snap_one'
> clone_snapshot 'snap_one', 't2'
> {code}
> In this scenario the user - test_user would not have permissions for the
> clone table t2.
> We need to add improvement feature such that the permissions of the original
> table are recorded in snapshot metadata and an option is provided for
> applying them to the new table as part of the clone process.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18026) ProtobufUtil seems to do extra array copying
[ https://issues.apache.org/jira/browse/HBASE-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011722#comment-16011722 ] Hudson commented on HBASE-18026: SUCCESS: Integrated in Jenkins build HBase-1.1-JDK8 #1953 (See [https://builds.apache.org/job/HBase-1.1-JDK8/1953/]) Amend HBASE-18026 ProtobufUtil seems to do extra array copying (apurtell: rev 2cff94c40933ce4f431f733b2de6d44e3e19c6f1) * (edit) hbase-protocol/src/main/java/com/google/protobuf/HBaseZeroCopyByteString.java > ProtobufUtil seems to do extra array copying > > > Key: HBASE-18026 > URL: https://issues.apache.org/jira/browse/HBASE-18026 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Minor > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-18026.addendum.patch, > HBASE-18026.branch-1.v1.patch, HBASE-18026.master.v1.patch > > > In ProtobufUtil, the protobuf fields are copied into an array using > toByteArray(). These are then passed into the KeyValue constructor which > does another copy. > It seems like we can avoid a copy here by using > HBaseZeroCopyByteString#zeroCopyGetBytes() ? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18053) AsyncTableResultScanner will hang when scan wrong column family
[ https://issues.apache.org/jira/browse/HBASE-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011721#comment-16011721 ] Guanghao Zhang commented on HBASE-18053: I add the ut in AbstractTestAsyncTableScan. And TestAsyncTableScanAll extends AbstractTestAsyncTableScan, too. The scanAll method return a CompletableFuture. So the exception is a ExecutionExceotion and caused by NoSuchColumnFamilyException. So I changed the assert condition in v1 patch. Thanks. > AsyncTableResultScanner will hang when scan wrong column family > --- > > Key: HBASE-18053 > URL: https://issues.apache.org/jira/browse/HBASE-18053 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18053.patch, HBASE-18053-v1.patch > > > AsynTableResultScanner did not call notify(). So the next will hang on > wait(). It is easy to fix. And will add a ut for this. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-11013) Clone Snapshots on Secure Cluster Should provide option to apply Retained User Permissions
[
https://issues.apache.org/jira/browse/HBASE-11013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011719#comment-16011719
]
Ted Yu commented on HBASE-11013:
Add test category:
{code}
+public class TestSnapshotWithAcl extends SecureTestUtil {
{code}
> Clone Snapshots on Secure Cluster Should provide option to apply Retained
> User Permissions
> --
>
> Key: HBASE-11013
> URL: https://issues.apache.org/jira/browse/HBASE-11013
> Project: HBase
> Issue Type: Improvement
> Components: snapshots
>Reporter: Ted Yu
>Assignee: Zheng Hu
> Fix For: 2.0.0
>
> Attachments: HBASE-11013.branch-1.v1.patch,
> HBASE-11013.master.addendum.patch, HBASE-11013.v1.patch,
> HBASE-11013.v2.patch, HBASE-11013.v3.patch, HBASE-11013.v3.patch,
> HBASE-11013.v4.patch, HBASE-11013.v4.patch
>
>
> Currently,
> {code}
> sudo su - test_user
> create 't1', 'f1'
> sudo su - hbase
> snapshot 't1', 'snap_one'
> clone_snapshot 'snap_one', 't2'
> {code}
> In this scenario the user - test_user would not have permissions for the
> clone table t2.
> We need to add improvement feature such that the permissions of the original
> table are recorded in snapshot metadata and an option is provided for
> applying them to the new table as part of the clone process.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-11013) Clone Snapshots on Secure Cluster Should provide option to apply Retained User Permissions
[
https://issues.apache.org/jira/browse/HBASE-11013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011718#comment-16011718
]
Ted Yu commented on HBASE-11013:
Zheng:
Can you highlight anything unique to the branch-1 patch (compared to the master
branch patch) ?
> Clone Snapshots on Secure Cluster Should provide option to apply Retained
> User Permissions
> --
>
> Key: HBASE-11013
> URL: https://issues.apache.org/jira/browse/HBASE-11013
> Project: HBase
> Issue Type: Improvement
> Components: snapshots
>Reporter: Ted Yu
>Assignee: Zheng Hu
> Fix For: 2.0.0
>
> Attachments: HBASE-11013.branch-1.v1.patch,
> HBASE-11013.master.addendum.patch, HBASE-11013.v1.patch,
> HBASE-11013.v2.patch, HBASE-11013.v3.patch, HBASE-11013.v3.patch,
> HBASE-11013.v4.patch, HBASE-11013.v4.patch
>
>
> Currently,
> {code}
> sudo su - test_user
> create 't1', 'f1'
> sudo su - hbase
> snapshot 't1', 'snap_one'
> clone_snapshot 'snap_one', 't2'
> {code}
> In this scenario the user - test_user would not have permissions for the
> clone table t2.
> We need to add improvement feature such that the permissions of the original
> table are recorded in snapshot metadata and an option is provided for
> applying them to the new table as part of the clone process.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-18053) AsyncTableResultScanner will hang when scan wrong column family
[ https://issues.apache.org/jira/browse/HBASE-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-18053: --- Description: AsynTableResultScanner did not call notify(). So the next will hang on wait(). It is easy to fix. And will add a ut for this. > AsyncTableResultScanner will hang when scan wrong column family > --- > > Key: HBASE-18053 > URL: https://issues.apache.org/jira/browse/HBASE-18053 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18053.patch, HBASE-18053-v1.patch > > > AsynTableResultScanner did not call notify(). So the next will hang on > wait(). It is easy to fix. And will add a ut for this. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17997) jruby-complete-1.6.8.jar is in cached_classpath.txt
[
https://issues.apache.org/jira/browse/HBASE-17997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011717#comment-16011717
]
Ted Yu commented on HBASE-17997:
[~waterlx]:
What do you think of the above ?
> jruby-complete-1.6.8.jar is in cached_classpath.txt
> ---
>
> Key: HBASE-17997
> URL: https://issues.apache.org/jira/browse/HBASE-17997
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>
> HBASE-15199 moves jruby-complete-1.6.8.jar to lib/ruby directory.
> However, jruby-complete-1.6.8.jar still appears in cached_classpath.txt
> This means that user would see exception similar to the following when
> starting hbase in standalone mode with s3a as rootdir :
> {code}
> 2017-05-04 16:41:32,854 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=38659]
> internal.S3MetadataResponseHandler: Unable to parse last modified date: Thu,
> 04 May 2017 16:27:09 GMT
> java.lang.IllegalStateException: Joda-time 2.2 or later version is required,
> but found version: null
> at com.amazonaws.util.DateUtils.handleException(DateUtils.java:149)
> at com.amazonaws.util.DateUtils.parseRFC822Date(DateUtils.java:195)
> at
> com.amazonaws.services.s3.internal.ServiceUtils.parseRfc822Date(ServiceUtils.java:78)
> at
> com.amazonaws.services.s3.internal.AbstractS3ResponseHandler.populateObjectMetadata(AbstractS3ResponseHandler.java:115)
> at
> com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:52)
> at
> com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:30)
> at
> com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:1072)
> at
> com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:746)
> at
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
> at
> com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1191)
> at org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:148)
> at org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:281)
> at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:364)
> at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:75)
> at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.positionalReadWithExtra(HFileBlock.java:722)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1420)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1677)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1504)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:439)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:904)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:267)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:169)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:363)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:217)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:2132)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2122)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5687)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2679)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2665)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2647)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6906)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6885)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2007)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-11013) Clone Snapshots on Secure Cluster Should provide option to apply Retained User Permissions
[
https://issues.apache.org/jira/browse/HBASE-11013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-11013:
-
Attachment: HBASE-11013.branch-1.v1.patch
> Clone Snapshots on Secure Cluster Should provide option to apply Retained
> User Permissions
> --
>
> Key: HBASE-11013
> URL: https://issues.apache.org/jira/browse/HBASE-11013
> Project: HBase
> Issue Type: Improvement
> Components: snapshots
>Reporter: Ted Yu
>Assignee: Zheng Hu
> Fix For: 2.0.0
>
> Attachments: HBASE-11013.branch-1.v1.patch,
> HBASE-11013.master.addendum.patch, HBASE-11013.v1.patch,
> HBASE-11013.v2.patch, HBASE-11013.v3.patch, HBASE-11013.v3.patch,
> HBASE-11013.v4.patch, HBASE-11013.v4.patch
>
>
> Currently,
> {code}
> sudo su - test_user
> create 't1', 'f1'
> sudo su - hbase
> snapshot 't1', 'snap_one'
> clone_snapshot 'snap_one', 't2'
> {code}
> In this scenario the user - test_user would not have permissions for the
> clone table t2.
> We need to add improvement feature such that the permissions of the original
> table are recorded in snapshot metadata and an option is provided for
> applying them to the new table as part of the clone process.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-18053) AsyncTableResultScanner will hang when scan wrong column family
[ https://issues.apache.org/jira/browse/HBASE-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-18053: --- Attachment: HBASE-18053-v1.patch > AsyncTableResultScanner will hang when scan wrong column family > --- > > Key: HBASE-18053 > URL: https://issues.apache.org/jira/browse/HBASE-18053 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18053.patch, HBASE-18053-v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18026) ProtobufUtil seems to do extra array copying
[ https://issues.apache.org/jira/browse/HBASE-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011702#comment-16011702 ] Hudson commented on HBASE-18026: SUCCESS: Integrated in Jenkins build HBase-1.2-JDK8 #130 (See [https://builds.apache.org/job/HBase-1.2-JDK8/130/]) Amend HBASE-18026 ProtobufUtil seems to do extra array copying (apurtell: rev 719ab32cc6910cf393ce235315190ffd8dd7a3c8) * (edit) hbase-protocol/src/main/java/com/google/protobuf/HBaseZeroCopyByteString.java > ProtobufUtil seems to do extra array copying > > > Key: HBASE-18026 > URL: https://issues.apache.org/jira/browse/HBASE-18026 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Minor > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-18026.addendum.patch, > HBASE-18026.branch-1.v1.patch, HBASE-18026.master.v1.patch > > > In ProtobufUtil, the protobuf fields are copied into an array using > toByteArray(). These are then passed into the KeyValue constructor which > does another copy. > It seems like we can avoid a copy here by using > HBaseZeroCopyByteString#zeroCopyGetBytes() ? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18053) AsyncTableResultScanner will hang when scan wrong column family
[ https://issues.apache.org/jira/browse/HBASE-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011697#comment-16011697 ] Ted Yu commented on HBASE-18053: You can use the following in the new test: @Test(expected = X.class) Please fill out description. > AsyncTableResultScanner will hang when scan wrong column family > --- > > Key: HBASE-18053 > URL: https://issues.apache.org/jira/browse/HBASE-18053 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18053.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18051) balance_rsgroup still run when the Load Balancer is not enabled.
[ https://issues.apache.org/jira/browse/HBASE-18051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011693#comment-16011693 ] Guangxu Cheng commented on HBASE-18051: --- sure,I will do work for it > balance_rsgroup still run when the Load Balancer is not enabled. > > > Key: HBASE-18051 > URL: https://issues.apache.org/jira/browse/HBASE-18051 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 2.0.0 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Fix For: 2.0.0 > > Attachments: HBASE-18051-v1.patch, HBASE-18051-v2.patch > > > When the Load Balancer is not enabled, the command "balancer" will not be > executed , but the command "balance_rsgroup" can still run, which is not > reasonable.So we need to fix it -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18053) AsyncTableResultScanner will hang when scan wrong column family
[ https://issues.apache.org/jira/browse/HBASE-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-18053: --- Assignee: Guanghao Zhang Status: Patch Available (was: Open) > AsyncTableResultScanner will hang when scan wrong column family > --- > > Key: HBASE-18053 > URL: https://issues.apache.org/jira/browse/HBASE-18053 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18053.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18053) AsyncTableResultScanner will hang when scan wrong column family
[ https://issues.apache.org/jira/browse/HBASE-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-18053: --- Attachment: HBASE-18053.patch > AsyncTableResultScanner will hang when scan wrong column family > --- > > Key: HBASE-18053 > URL: https://issues.apache.org/jira/browse/HBASE-18053 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Guanghao Zhang > Attachments: HBASE-18053.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18026) ProtobufUtil seems to do extra array copying
[ https://issues.apache.org/jira/browse/HBASE-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011689#comment-16011689 ] Hudson commented on HBASE-18026: SUCCESS: Integrated in Jenkins build HBase-1.1-JDK7 #1870 (See [https://builds.apache.org/job/HBase-1.1-JDK7/1870/]) Amend HBASE-18026 ProtobufUtil seems to do extra array copying (apurtell: rev 2cff94c40933ce4f431f733b2de6d44e3e19c6f1) * (edit) hbase-protocol/src/main/java/com/google/protobuf/HBaseZeroCopyByteString.java > ProtobufUtil seems to do extra array copying > > > Key: HBASE-18026 > URL: https://issues.apache.org/jira/browse/HBASE-18026 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Minor > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-18026.addendum.patch, > HBASE-18026.branch-1.v1.patch, HBASE-18026.master.v1.patch > > > In ProtobufUtil, the protobuf fields are copied into an array using > toByteArray(). These are then passed into the KeyValue constructor which > does another copy. > It seems like we can avoid a copy here by using > HBaseZeroCopyByteString#zeroCopyGetBytes() ? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-18054) log when we add/remove failed servers in client
Sean Busbey created HBASE-18054:
---
Summary: log when we add/remove failed servers in client
Key: HBASE-18054
URL: https://issues.apache.org/jira/browse/HBASE-18054
Project: HBase
Issue Type: Bug
Components: Client, Operability
Affects Versions: 1.3.0, 1.2.0, 1.1.0, 2.0.0, 1.4.0
Reporter: Sean Busbey
Currently we log if a server is in the failed server list when we go to connect
to it, but we don't log anything about when the server got into the list.
This means we have to search the log for errors involving the same server name
that (hopefully) managed to get into the log within
{{FAILED_SERVER_EXPIRY_KEY}} milliseconds earlier (default 2 seconds).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18054) log when we add/remove failed servers in client
[
https://issues.apache.org/jira/browse/HBASE-18054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011690#comment-16011690
]
Sean Busbey commented on HBASE-18054:
-
log at DEBUG, I guess?
> log when we add/remove failed servers in client
> ---
>
> Key: HBASE-18054
> URL: https://issues.apache.org/jira/browse/HBASE-18054
> Project: HBase
> Issue Type: Bug
> Components: Client, Operability
>Affects Versions: 2.0.0, 1.1.0, 1.2.0, 1.3.0, 1.4.0
>Reporter: Sean Busbey
>
> Currently we log if a server is in the failed server list when we go to
> connect to it, but we don't log anything about when the server got into the
> list.
> This means we have to search the log for errors involving the same server
> name that (hopefully) managed to get into the log within
> {{FAILED_SERVER_EXPIRY_KEY}} milliseconds earlier (default 2 seconds).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18051) balance_rsgroup still run when the Load Balancer is not enabled.
[ https://issues.apache.org/jira/browse/HBASE-18051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011688#comment-16011688 ] Ted Yu commented on HBASE-18051: Is it possible to add a unit test (to guard against regression) ? > balance_rsgroup still run when the Load Balancer is not enabled. > > > Key: HBASE-18051 > URL: https://issues.apache.org/jira/browse/HBASE-18051 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 2.0.0 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Fix For: 2.0.0 > > Attachments: HBASE-18051-v1.patch, HBASE-18051-v2.patch > > > When the Load Balancer is not enabled, the command "balancer" will not be > executed , but the command "balance_rsgroup" can still run, which is not > reasonable.So we need to fix it -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-18051) balance_rsgroup still run when the Load Balancer is not enabled.
[ https://issues.apache.org/jira/browse/HBASE-18051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011682#comment-16011682 ] Guangxu Cheng edited comment on HBASE-18051 at 5/16/17 3:34 AM: v2 takes the suggestion from [~yuzhih...@gmail.com]. Thanks was (Author: andrewcheng): v2 takes the suggestion from [~yuzhih...@gmail.com] > balance_rsgroup still run when the Load Balancer is not enabled. > > > Key: HBASE-18051 > URL: https://issues.apache.org/jira/browse/HBASE-18051 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 2.0.0 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Fix For: 2.0.0 > > Attachments: HBASE-18051-v1.patch, HBASE-18051-v2.patch > > > When the Load Balancer is not enabled, the command "balancer" will not be > executed , but the command "balance_rsgroup" can still run, which is not > reasonable.So we need to fix it -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18051) balance_rsgroup still run when the Load Balancer is not enabled.
[ https://issues.apache.org/jira/browse/HBASE-18051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated HBASE-18051: -- Attachment: HBASE-18051-v2.patch > balance_rsgroup still run when the Load Balancer is not enabled. > > > Key: HBASE-18051 > URL: https://issues.apache.org/jira/browse/HBASE-18051 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 2.0.0 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Fix For: 2.0.0 > > Attachments: HBASE-18051-v1.patch, HBASE-18051-v2.patch > > > When the Load Balancer is not enabled, the command "balancer" will not be > executed , but the command "balance_rsgroup" can still run, which is not > reasonable.So we need to fix it -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18051) balance_rsgroup still run when the Load Balancer is not enabled.
[ https://issues.apache.org/jira/browse/HBASE-18051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011682#comment-16011682 ] Guangxu Cheng commented on HBASE-18051: --- v2 takes the suggestion from [~yuzhih...@gmail.com] > balance_rsgroup still run when the Load Balancer is not enabled. > > > Key: HBASE-18051 > URL: https://issues.apache.org/jira/browse/HBASE-18051 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 2.0.0 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Fix For: 2.0.0 > > Attachments: HBASE-18051-v1.patch, HBASE-18051-v2.patch > > > When the Load Balancer is not enabled, the command "balancer" will not be > executed , but the command "balance_rsgroup" can still run, which is not > reasonable.So we need to fix it -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18051) balance_rsgroup still run when the Load Balancer is not enabled.
[ https://issues.apache.org/jira/browse/HBASE-18051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated HBASE-18051: -- Status: Patch Available (was: Open) > balance_rsgroup still run when the Load Balancer is not enabled. > > > Key: HBASE-18051 > URL: https://issues.apache.org/jira/browse/HBASE-18051 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 2.0.0 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Fix For: 2.0.0 > > Attachments: HBASE-18051-v1.patch, HBASE-18051-v2.patch > > > When the Load Balancer is not enabled, the command "balancer" will not be > executed , but the command "balance_rsgroup" can still run, which is not > reasonable.So we need to fix it -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-18053) AsyncTableResultScanner will hang when scan wrong column family
Guanghao Zhang created HBASE-18053: -- Summary: AsyncTableResultScanner will hang when scan wrong column family Key: HBASE-18053 URL: https://issues.apache.org/jira/browse/HBASE-18053 Project: HBase Issue Type: Bug Reporter: Guanghao Zhang -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18053) AsyncTableResultScanner will hang when scan wrong column family
[ https://issues.apache.org/jira/browse/HBASE-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-18053: --- Component/s: Client > AsyncTableResultScanner will hang when scan wrong column family > --- > > Key: HBASE-18053 > URL: https://issues.apache.org/jira/browse/HBASE-18053 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Guanghao Zhang > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18051) balance_rsgroup still run when the Load Balancer is not enabled.
[ https://issues.apache.org/jira/browse/HBASE-18051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011662#comment-16011662 ] Ted Yu commented on HBASE-18051: If there is no other way, the above is fine by me. > balance_rsgroup still run when the Load Balancer is not enabled. > > > Key: HBASE-18051 > URL: https://issues.apache.org/jira/browse/HBASE-18051 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 2.0.0 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Fix For: 2.0.0 > > Attachments: HBASE-18051-v1.patch > > > When the Load Balancer is not enabled, the command "balancer" will not be > executed , but the command "balance_rsgroup" can still run, which is not > reasonable.So we need to fix it -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18051) balance_rsgroup still run when the Load Balancer is not enabled.
[ https://issues.apache.org/jira/browse/HBASE-18051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011658#comment-16011658 ] Guangxu Cheng commented on HBASE-18051: --- bq.See if you can utilize existing method(s) and not add isBalancerOn() to MasterServices yep, isBalancerEnabled () calls the method isBalancerOn() in HMaster to get the state of the load balancer. In RSGroupAdminServer, the type of master is MasterServices, but isBalancerOn() is not defined in MasterServices, so I add isBalancerOn() to MasterServices. Can I use (HMaster) master).isBalancerOn() instead of adding isBalancerOn() to MasterServices? > balance_rsgroup still run when the Load Balancer is not enabled. > > > Key: HBASE-18051 > URL: https://issues.apache.org/jira/browse/HBASE-18051 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 2.0.0 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Fix For: 2.0.0 > > Attachments: HBASE-18051-v1.patch > > > When the Load Balancer is not enabled, the command "balancer" will not be > executed , but the command "balance_rsgroup" can still run, which is not > reasonable.So we need to fix it -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18026) ProtobufUtil seems to do extra array copying
[ https://issues.apache.org/jira/browse/HBASE-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011654#comment-16011654 ] Hudson commented on HBASE-18026: SUCCESS: Integrated in Jenkins build HBase-1.2-IT #869 (See [https://builds.apache.org/job/HBase-1.2-IT/869/]) Amend HBASE-18026 ProtobufUtil seems to do extra array copying (apurtell: rev 719ab32cc6910cf393ce235315190ffd8dd7a3c8) * (edit) hbase-protocol/src/main/java/com/google/protobuf/HBaseZeroCopyByteString.java > ProtobufUtil seems to do extra array copying > > > Key: HBASE-18026 > URL: https://issues.apache.org/jira/browse/HBASE-18026 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Minor > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-18026.addendum.patch, > HBASE-18026.branch-1.v1.patch, HBASE-18026.master.v1.patch > > > In ProtobufUtil, the protobuf fields are copied into an array using > toByteArray(). These are then passed into the KeyValue constructor which > does another copy. > It seems like we can avoid a copy here by using > HBaseZeroCopyByteString#zeroCopyGetBytes() ? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18026) ProtobufUtil seems to do extra array copying
[ https://issues.apache.org/jira/browse/HBASE-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011645#comment-16011645 ] Hudson commented on HBASE-18026: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #46 (See [https://builds.apache.org/job/HBase-1.3-IT/46/]) Amend HBASE-18026 ProtobufUtil seems to do extra array copying (apurtell: rev d18eb62d8d0f02ea822090f409c2ded6f5279c3d) * (edit) hbase-protocol/src/main/java/com/google/protobuf/HBaseZeroCopyByteString.java > ProtobufUtil seems to do extra array copying > > > Key: HBASE-18026 > URL: https://issues.apache.org/jira/browse/HBASE-18026 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Minor > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-18026.addendum.patch, > HBASE-18026.branch-1.v1.patch, HBASE-18026.master.v1.patch > > > In ProtobufUtil, the protobuf fields are copied into an array using > toByteArray(). These are then passed into the KeyValue constructor which > does another copy. > It seems like we can avoid a copy here by using > HBaseZeroCopyByteString#zeroCopyGetBytes() ? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-14247) Separate the old WALs into different regionserver directories
[
https://issues.apache.org/jira/browse/HBASE-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011643#comment-16011643
]
Enis Soztutar commented on HBASE-14247:
---
Please feel free to go ahead.
> Separate the old WALs into different regionserver directories
> -
>
> Key: HBASE-14247
> URL: https://issues.apache.org/jira/browse/HBASE-14247
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-14247-v001.diff, HBASE-14247-v002.diff,
> HBASE-14247-v003.diff
>
>
> Currently all old WALs of regionservers are achieved into the single
> directory of oldWALs. In big clusters, because of long TTL of WAL or disabled
> replications, the number of files under oldWALs may reach the
> max-directory-items limit of HDFS, which will make the hbase cluster crashed.
> {quote}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
> The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded:
> limit=1048576 items=1048576
> {quote}
> A simple solution is to separate the old WALs into different directories
> according to the server name of the WAL.
> Suggestions are welcomed~ Thanks
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17359) Implement async admin
[ https://issues.apache.org/jira/browse/HBASE-17359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011642#comment-16011642 ] Zheng Hu commented on HBASE-17359: -- Assigned myself with HBASE-18003 and HBASE-18052. > Implement async admin > - > > Key: HBASE-17359 > URL: https://issues.apache.org/jira/browse/HBASE-17359 > Project: HBase > Issue Type: Umbrella > Components: Client >Reporter: Duo Zhang >Assignee: Guanghao Zhang > Labels: asynchronous > Fix For: 2.0.0 > > > And as we will return a CompletableFuture, I think we can just remove the > XXXAsync methods, and make all the methods blocking which means we will only > finish the CompletableFuture when the operation is done. User can choose > whether to wait on the returned CompletableFuture. > Convert this to a umbrella task. There maybe some sub-tasks. > 1. Table admin operations. > 2. Region admin operations. > 3. Namespace admin operations. > 4. Snapshot admin operations. > 5. Replication admin operations. > 6. Other operations, like quota, balance.. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HBASE-18003) Fix flaky test TestAsyncTableAdminApi and TestAsyncSnapshotAdminApi
[ https://issues.apache.org/jira/browse/HBASE-18003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu reassigned HBASE-18003: Assignee: Zheng Hu > Fix flaky test TestAsyncTableAdminApi and TestAsyncSnapshotAdminApi > --- > > Key: HBASE-18003 > URL: https://issues.apache.org/jira/browse/HBASE-18003 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Guanghao Zhang >Assignee: Zheng Hu > Fix For: 2.0.0 > > > See > https://builds.apache.org/job/HBASE-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HBASE-18052) Add doc, examples for async admin
[ https://issues.apache.org/jira/browse/HBASE-18052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu reassigned HBASE-18052: Assignee: Zheng Hu > Add doc, examples for async admin > - > > Key: HBASE-18052 > URL: https://issues.apache.org/jira/browse/HBASE-18052 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Guanghao Zhang >Assignee: Zheng Hu > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18026) ProtobufUtil seems to do extra array copying
[ https://issues.apache.org/jira/browse/HBASE-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011638#comment-16011638 ] Hudson commented on HBASE-18026: FAILURE: Integrated in Jenkins build HBase-1.4 #735 (See [https://builds.apache.org/job/HBase-1.4/735/]) Amend HBASE-18026 ProtobufUtil seems to do extra array copying (apurtell: rev d8ef495063ebf794164603d0739fcb27dce106fa) * (edit) hbase-protocol/src/main/java/com/google/protobuf/HBaseZeroCopyByteString.java > ProtobufUtil seems to do extra array copying > > > Key: HBASE-18026 > URL: https://issues.apache.org/jira/browse/HBASE-18026 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Minor > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-18026.addendum.patch, > HBASE-18026.branch-1.v1.patch, HBASE-18026.master.v1.patch > > > In ProtobufUtil, the protobuf fields are copied into an array using > toByteArray(). These are then passed into the KeyValue constructor which > does another copy. > It seems like we can avoid a copy here by using > HBaseZeroCopyByteString#zeroCopyGetBytes() ? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18043) Institute a hard limit for individual cell size that cannot be overridden by clients
[ https://issues.apache.org/jira/browse/HBASE-18043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011637#comment-16011637 ] Hudson commented on HBASE-18043: FAILURE: Integrated in Jenkins build HBase-1.4 #735 (See [https://builds.apache.org/job/HBase-1.4/735/]) HBASE-18043 Institute a hard limit for individual cell size that cannot (apurtell: rev 292226690ac842be4513366633f31bba4fa62d34) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java * (edit) hbase-common/src/main/resources/hbase-default.xml * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java > Institute a hard limit for individual cell size that cannot be overridden by > clients > > > Key: HBASE-18043 > URL: https://issues.apache.org/jira/browse/HBASE-18043 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, regionserver >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-18043-branch-1.patch, HBASE-18043-branch-1.patch, > HBASE-18043-branch-1.patch, HBASE-18043-branch-1.patch, HBASE-18043.patch, > HBASE-18043.patch, HBASE-18043.patch, HBASE-18043.patch, HBASE-18043.patch, > HBASE-18043.patch > > > For sake of service protection we should not give absolute trust to clients > regarding resource limits that can impact stability, like cell size limits. > We should add a server side configuration that sets a hard limit for > individual cell size that cannot be overridden by the client. We can keep the > client side check, because it's expensive to reject a RPC that has already > come in. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18049) It is not necessary to re-open the region when MOB files cannot be found
[
https://issues.apache.org/jira/browse/HBASE-18049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011627#comment-16011627
]
Anoop Sam John commented on HBASE-18049:
A bug some where in the code makes this MOB data not found. The cell may be in
some other MOB file and we have an issue in the MOB ref cell read. A solution
to restore the state and data is to do a MOB major compaction right? May be we
should add that info the error log clearly.
> It is not necessary to re-open the region when MOB files cannot be found
>
>
> Key: HBASE-18049
> URL: https://issues.apache.org/jira/browse/HBASE-18049
> Project: HBase
> Issue Type: Bug
> Components: Scanners
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Fix For: 2.0.0
>
> Attachments: HBASE-18049.patch
>
>
> In HBASE-17712, we try to re-open the region when store files cannot be
> found. This is useful for store files in a region, but is not necessary when
> the MOB files cannot be found, because the store files in a region only
> contain the references to the MOB files and a re-open of a region doesn't
> help the lost MOB files.
> In this JIRA, we will directly throw DNRIOE only when the MOB files are not
> found in {{MobStoreScanner}} and {{ReversedMobStoreScanner}}. Other logics
> keep the same.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-11013) Clone Snapshots on Secure Cluster Should provide option to apply Retained User Permissions
[
https://issues.apache.org/jira/browse/HBASE-11013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-11013:
-
Attachment: HBASE-11013.v4.patch
Trigger QA again.
> Clone Snapshots on Secure Cluster Should provide option to apply Retained
> User Permissions
> --
>
> Key: HBASE-11013
> URL: https://issues.apache.org/jira/browse/HBASE-11013
> Project: HBase
> Issue Type: Improvement
> Components: snapshots
>Reporter: Ted Yu
>Assignee: Zheng Hu
> Fix For: 2.0.0
>
> Attachments: HBASE-11013.master.addendum.patch, HBASE-11013.v1.patch,
> HBASE-11013.v2.patch, HBASE-11013.v3.patch, HBASE-11013.v3.patch,
> HBASE-11013.v4.patch, HBASE-11013.v4.patch
>
>
> Currently,
> {code}
> sudo su - test_user
> create 't1', 'f1'
> sudo su - hbase
> snapshot 't1', 'snap_one'
> clone_snapshot 'snap_one', 't2'
> {code}
> In this scenario the user - test_user would not have permissions for the
> clone table t2.
> We need to add improvement feature such that the permissions of the original
> table are recorded in snapshot metadata and an option is provided for
> applying them to the new table as part of the clone process.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18049) It is not necessary to re-open the region when MOB files cannot be found
[
https://issues.apache.org/jira/browse/HBASE-18049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011621#comment-16011621
]
Anoop Sam John commented on HBASE-18049:
In above lines we have
{code}
if ((e instanceof FileNotFoundException) ||
(e.getCause() instanceof FileNotFoundException)) {
{code}
So here in new change as well, we need getCause() check? Just asking. I did
not check the flow to see how the Exception stack will be possibly be. Else
looks ok.
> It is not necessary to re-open the region when MOB files cannot be found
>
>
> Key: HBASE-18049
> URL: https://issues.apache.org/jira/browse/HBASE-18049
> Project: HBase
> Issue Type: Bug
> Components: Scanners
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Fix For: 2.0.0
>
> Attachments: HBASE-18049.patch
>
>
> In HBASE-17712, we try to re-open the region when store files cannot be
> found. This is useful for store files in a region, but is not necessary when
> the MOB files cannot be found, because the store files in a region only
> contain the references to the MOB files and a re-open of a region doesn't
> help the lost MOB files.
> In this JIRA, we will directly throw DNRIOE only when the MOB files are not
> found in {{MobStoreScanner}} and {{ReversedMobStoreScanner}}. Other logics
> keep the same.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-14247) Separate the old WALs into different regionserver directories
[
https://issues.apache.org/jira/browse/HBASE-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011613#comment-16011613
]
Guanghao Zhang commented on HBASE-14247:
[~enis] Do you work on this issue? I have a patch for our 0.98 branch. If you
not work on this, I can help to prepare a patch for master branch. Thanks.
> Separate the old WALs into different regionserver directories
> -
>
> Key: HBASE-14247
> URL: https://issues.apache.org/jira/browse/HBASE-14247
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-14247-v001.diff, HBASE-14247-v002.diff,
> HBASE-14247-v003.diff
>
>
> Currently all old WALs of regionservers are achieved into the single
> directory of oldWALs. In big clusters, because of long TTL of WAL or disabled
> replications, the number of files under oldWALs may reach the
> max-directory-items limit of HDFS, which will make the hbase cluster crashed.
> {quote}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
> The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded:
> limit=1048576 items=1048576
> {quote}
> A simple solution is to separate the old WALs into different directories
> according to the server name of the WAL.
> Suggestions are welcomed~ Thanks
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17994) Add async client test to Performance Evaluation tool
[ https://issues.apache.org/jira/browse/HBASE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-17994: --- Issue Type: Sub-task (was: New Feature) Parent: HBASE-16833 > Add async client test to Performance Evaluation tool > > > Key: HBASE-17994 > URL: https://issues.apache.org/jira/browse/HBASE-17994 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17359) Implement async admin
[ https://issues.apache.org/jira/browse/HBASE-17359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011605#comment-16011605 ] Guanghao Zhang commented on HBASE-17359: There are two sub-task need to resolve. HBASE-18003 is about the flaky ut. And HBASE-18052 is about doc, examples for async admin. Thanks. > Implement async admin > - > > Key: HBASE-17359 > URL: https://issues.apache.org/jira/browse/HBASE-17359 > Project: HBase > Issue Type: Umbrella > Components: Client >Reporter: Duo Zhang >Assignee: Guanghao Zhang > Labels: asynchronous > Fix For: 2.0.0 > > > And as we will return a CompletableFuture, I think we can just remove the > XXXAsync methods, and make all the methods blocking which means we will only > finish the CompletableFuture when the operation is done. User can choose > whether to wait on the returned CompletableFuture. > Convert this to a umbrella task. There maybe some sub-tasks. > 1. Table admin operations. > 2. Region admin operations. > 3. Namespace admin operations. > 4. Snapshot admin operations. > 5. Replication admin operations. > 6. Other operations, like quota, balance.. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-18052) Add doc, examples for async admin
Guanghao Zhang created HBASE-18052: -- Summary: Add doc, examples for async admin Key: HBASE-18052 URL: https://issues.apache.org/jira/browse/HBASE-18052 Project: HBase Issue Type: Sub-task Reporter: Guanghao Zhang -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18043) Institute a hard limit for individual cell size that cannot be overridden by clients
[ https://issues.apache.org/jira/browse/HBASE-18043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18043: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Institute a hard limit for individual cell size that cannot be overridden by > clients > > > Key: HBASE-18043 > URL: https://issues.apache.org/jira/browse/HBASE-18043 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, regionserver >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-18043-branch-1.patch, HBASE-18043-branch-1.patch, > HBASE-18043-branch-1.patch, HBASE-18043-branch-1.patch, HBASE-18043.patch, > HBASE-18043.patch, HBASE-18043.patch, HBASE-18043.patch, HBASE-18043.patch, > HBASE-18043.patch > > > For sake of service protection we should not give absolute trust to clients > regarding resource limits that can impact stability, like cell size limits. > We should add a server side configuration that sets a hard limit for > individual cell size that cannot be overridden by the client. We can keep the > client side check, because it's expensive to reject a RPC that has already > come in. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18026) ProtobufUtil seems to do extra array copying
[ https://issues.apache.org/jira/browse/HBASE-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18026: --- Affects Version/s: (was: 2.0.0) Fix Version/s: (was: 2.0.0) > ProtobufUtil seems to do extra array copying > > > Key: HBASE-18026 > URL: https://issues.apache.org/jira/browse/HBASE-18026 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Minor > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-18026.addendum.patch, > HBASE-18026.branch-1.v1.patch, HBASE-18026.master.v1.patch > > > In ProtobufUtil, the protobuf fields are copied into an array using > toByteArray(). These are then passed into the KeyValue constructor which > does another copy. > It seems like we can avoid a copy here by using > HBaseZeroCopyByteString#zeroCopyGetBytes() ? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17359) Implement async admin
[ https://issues.apache.org/jira/browse/HBASE-17359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011595#comment-16011595 ] Duo Zhang commented on HBASE-17359: --- So what's the status of this issue? Have we done? [~zghaobac] [~openinx]. > Implement async admin > - > > Key: HBASE-17359 > URL: https://issues.apache.org/jira/browse/HBASE-17359 > Project: HBase > Issue Type: Umbrella > Components: Client >Reporter: Duo Zhang >Assignee: Guanghao Zhang > Labels: asynchronous > Fix For: 2.0.0 > > > And as we will return a CompletableFuture, I think we can just remove the > XXXAsync methods, and make all the methods blocking which means we will only > finish the CompletableFuture when the operation is done. User can choose > whether to wait on the returned CompletableFuture. > Convert this to a umbrella task. There maybe some sub-tasks. > 1. Table admin operations. > 2. Region admin operations. > 3. Namespace admin operations. > 4. Snapshot admin operations. > 5. Replication admin operations. > 6. Other operations, like quota, balance.. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18038) Rename StoreFile to HStoreFile and add a StoreFile interface for CP
[ https://issues.apache.org/jira/browse/HBASE-18038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-18038: -- Attachment: HBASE-18038-v1.patch Retry. > Rename StoreFile to HStoreFile and add a StoreFile interface for CP > --- > > Key: HBASE-18038 > URL: https://issues.apache.org/jira/browse/HBASE-18038 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors, regionserver >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18038.patch, HBASE-18038-v1.patch, > HBASE-18038-v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18035) Meta replica does not give any primaryOperationTimeout to primary meta region
[ https://issues.apache.org/jira/browse/HBASE-18035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011587#comment-16011587 ] huaxiang sun commented on HBASE-18035: -- As a side note, why is memstore replication not turned on for meta replica? Probably will create a jira to track it. > Meta replica does not give any primaryOperationTimeout to primary meta region > - > > Key: HBASE-18035 > URL: https://issues.apache.org/jira/browse/HBASE-18035 > Project: HBase > Issue Type: Bug >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Critical > Attachments: 18035-unittest.patch, HBASE-18035-master-v001.patch > > > I was working on my unittest and it failed with TableNotFoundException. I > debugged a bit and found out that for meta scan, it does not give any > primaryOperationTimeout to primary meta region. This will be an issue as the > meta replica will contain stale data and it is possible that the meta replica > will return back first than primary. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java#L823 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18035) Meta replica does not give any primaryOperationTimeout to primary meta region
[ https://issues.apache.org/jira/browse/HBASE-18035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-18035: - Attachment: HBASE-18035-master-v001.patch Upload v1 patch. > Meta replica does not give any primaryOperationTimeout to primary meta region > - > > Key: HBASE-18035 > URL: https://issues.apache.org/jira/browse/HBASE-18035 > Project: HBase > Issue Type: Bug >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Critical > Attachments: 18035-unittest.patch, HBASE-18035-master-v001.patch > > > I was working on my unittest and it failed with TableNotFoundException. I > debugged a bit and found out that for meta scan, it does not give any > primaryOperationTimeout to primary meta region. This will be an issue as the > meta replica will contain stale data and it is possible that the meta replica > will return back first than primary. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java#L823 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18035) Meta replica does not give any primaryOperationTimeout to primary meta region
[ https://issues.apache.org/jira/browse/HBASE-18035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-18035: - Status: Patch Available (was: Open) > Meta replica does not give any primaryOperationTimeout to primary meta region > - > > Key: HBASE-18035 > URL: https://issues.apache.org/jira/browse/HBASE-18035 > Project: HBase > Issue Type: Bug >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Critical > Attachments: 18035-unittest.patch, HBASE-18035-master-v001.patch > > > I was working on my unittest and it failed with TableNotFoundException. I > debugged a bit and found out that for meta scan, it does not give any > primaryOperationTimeout to primary meta region. This will be an issue as the > meta replica will contain stale data and it is possible that the meta replica > will return back first than primary. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java#L823 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18037) Do not expose implementation classes to CP
[ https://issues.apache.org/jira/browse/HBASE-18037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011574#comment-16011574 ] Andrew Purtell commented on HBASE-18037: bq. you are free to hold a reference to a class which is declared as IA.Private, but you can only use it as a monolithic object. You can use it as a parameter to call a method, or use it as a return value, but you should never try to call its methods to change its state of do something else. Sounds good [~Apache9]. I see you filed HBASE-18050 for followup > Do not expose implementation classes to CP > -- > > Key: HBASE-18037 > URL: https://issues.apache.org/jira/browse/HBASE-18037 > Project: HBase > Issue Type: Umbrella > Components: Coprocessors >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Priority: Blocker > Fix For: 2.0.0 > > > For example, StoreFile. Expose the implementation classes to CP will make it > harder to implement new features or improve the old implementation. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-18026) ProtobufUtil seems to do extra array copying
[ https://issues.apache.org/jira/browse/HBASE-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011570#comment-16011570 ] Andrew Purtell edited comment on HBASE-18026 at 5/16/17 1:10 AM: - bq. I suggest revering the change in master branch as any way it is not useful and needed. For other branches, I also suggest adding the addendum Ok [~anoop.hbase] - doing that now was (Author: apurtell): Ok [~anoop.hbase] - doing that now > ProtobufUtil seems to do extra array copying > > > Key: HBASE-18026 > URL: https://issues.apache.org/jira/browse/HBASE-18026 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.2 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Minor > Fix For: 2.0.0, 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-18026.addendum.patch, > HBASE-18026.branch-1.v1.patch, HBASE-18026.master.v1.patch > > > In ProtobufUtil, the protobuf fields are copied into an array using > toByteArray(). These are then passed into the KeyValue constructor which > does another copy. > It seems like we can avoid a copy here by using > HBaseZeroCopyByteString#zeroCopyGetBytes() ? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18026) ProtobufUtil seems to do extra array copying
[ https://issues.apache.org/jira/browse/HBASE-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011570#comment-16011570 ] Andrew Purtell commented on HBASE-18026: Ok [~anoop.hbase] - doing that now > ProtobufUtil seems to do extra array copying > > > Key: HBASE-18026 > URL: https://issues.apache.org/jira/browse/HBASE-18026 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.2 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Minor > Fix For: 2.0.0, 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-18026.addendum.patch, > HBASE-18026.branch-1.v1.patch, HBASE-18026.master.v1.patch > > > In ProtobufUtil, the protobuf fields are copied into an array using > toByteArray(). These are then passed into the KeyValue constructor which > does another copy. > It seems like we can avoid a copy here by using > HBaseZeroCopyByteString#zeroCopyGetBytes() ? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18027) Replication should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18027: --- Status: Patch Available (was: Open) > Replication should respect RPC size limits when batching edits > -- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.3.1, 2.0.0, 1.4.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027.patch, HBASE-18027.patch, HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum of configured replicator > threads, number of 100-waledit batches, or number of current sinks. Every > pending entry in the replication context is then placed in order by hash of > encoded region name into one of these N lists. Each of the N lists is then > sent all at once in one replication RPC. We do not test if the sum of data in > each N list will exceed RPC size limits. This code presumes each individual > edit is reasonably small. Not checking for aggregate size while assembling > the lists into RPCs is an oversight and can lead to replication failure when > that assumption is violated. > We can fix this by generating as many replication RPC calls as we need to > drain a list, keeping each RPC under limit, instead of assuming the whole > list will fit in one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18044) Bug fix in flaky-dashboard-template
[ https://issues.apache.org/jira/browse/HBASE-18044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011521#comment-16011521 ] Hudson commented on HBASE-18044: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #3017 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3017/]) HBASE-18044 Fix bug in report-flakies.py where if counter is set outside (appy: rev 841bb00655b3eb58e7d76d945e24cfeadfdb11de) * (edit) dev-support/flaky-dashboard-template.html > Bug fix in flaky-dashboard-template > --- > > Key: HBASE-18044 > URL: https://issues.apache.org/jira/browse/HBASE-18044 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-18044.master.001.patch > > > Due to some scoping issues, counters don't work in jinja templates like we > expect them to (in java/c++,etc). > (http://stackoverflow.com/questions/7537439/how-to-increment-a-variable-on-a-for-loop-in-jinja-template). > Due to this, clicking show/hide button for tests which fail in multiple urls > will not work sometime. Fixing it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18035) Meta replica does not give any primaryOperationTimeout to primary meta region
[ https://issues.apache.org/jira/browse/HBASE-18035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011517#comment-16011517 ] huaxiang sun commented on HBASE-18035: -- I am planning to propose add "hbase.client.metaReplicaCallTimeout.scan" for primary meta region scan. > Meta replica does not give any primaryOperationTimeout to primary meta region > - > > Key: HBASE-18035 > URL: https://issues.apache.org/jira/browse/HBASE-18035 > Project: HBase > Issue Type: Bug >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Critical > Attachments: 18035-unittest.patch > > > I was working on my unittest and it failed with TableNotFoundException. I > debugged a bit and found out that for meta scan, it does not give any > primaryOperationTimeout to primary meta region. This will be an issue as the > meta replica will contain stale data and it is possible that the meta replica > will return back first than primary. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java#L823 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18043) Institute a hard limit for individual cell size that cannot be overridden by clients
[
https://issues.apache.org/jira/browse/HBASE-18043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011510#comment-16011510
]
Hadoop QA commented on HBASE-18043:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m 23s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 30s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
32m 21s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 45s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 99m 52s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
30s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 171m 43s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestMobStoreScanner |
| | hadoop.hbase.mapreduce.TestImportTSVWithVisibilityLabels |
| | hadoop.hbase.master.TestMasterMetrics |
| | hadoop.hbase.snapshot.TestMobFlushSnapshotFromClient |
| | hadoop.hbase.quotas.TestQuotaThrottle |
| Timed out junit tests |
org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure |
| | org.apache.hadoop.hbase.master.TestMasterFailover |
| | org.apache.hadoop.hbase.regionserver.TestTags |
| | org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster |
| | org.apache.hadoop.hbase.master.balancer.TestFavoredStochasticLoadBalancer
|
| | org.apache.hadoop.hbase.master.balancer.TestFavoredNodeTableImport |
| | org.apache.hadoop.hbase.mapreduce.TestTableMapReduce |
| | org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestExportSnapshot |
| |
[jira] [Updated] (HBASE-18035) Meta replica does not give any primaryOperationTimeout to primary meta region
[ https://issues.apache.org/jira/browse/HBASE-18035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-18035: - Attachment: (was: 18035-unittest.diff) > Meta replica does not give any primaryOperationTimeout to primary meta region > - > > Key: HBASE-18035 > URL: https://issues.apache.org/jira/browse/HBASE-18035 > Project: HBase > Issue Type: Bug >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Critical > Attachments: 18035-unittest.patch > > > I was working on my unittest and it failed with TableNotFoundException. I > debugged a bit and found out that for meta scan, it does not give any > primaryOperationTimeout to primary meta region. This will be an issue as the > meta replica will contain stale data and it is possible that the meta replica > will return back first than primary. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java#L823 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18035) Meta replica does not give any primaryOperationTimeout to primary meta region
[ https://issues.apache.org/jira/browse/HBASE-18035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-18035: - Attachment: 18035-unittest.patch > Meta replica does not give any primaryOperationTimeout to primary meta region > - > > Key: HBASE-18035 > URL: https://issues.apache.org/jira/browse/HBASE-18035 > Project: HBase > Issue Type: Bug >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Critical > Attachments: 18035-unittest.patch > > > I was working on my unittest and it failed with TableNotFoundException. I > debugged a bit and found out that for meta scan, it does not give any > primaryOperationTimeout to primary meta region. This will be an issue as the > meta replica will contain stale data and it is possible that the meta replica > will return back first than primary. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java#L823 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18035) Meta replica does not give any primaryOperationTimeout to primary meta region
[ https://issues.apache.org/jira/browse/HBASE-18035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-18035: - Attachment: 18035-unittest.diff Upload a unittest case to demo the issue. > Meta replica does not give any primaryOperationTimeout to primary meta region > - > > Key: HBASE-18035 > URL: https://issues.apache.org/jira/browse/HBASE-18035 > Project: HBase > Issue Type: Bug >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Critical > Attachments: 18035-unittest.diff > > > I was working on my unittest and it failed with TableNotFoundException. I > debugged a bit and found out that for meta scan, it does not give any > primaryOperationTimeout to primary meta region. This will be an issue as the > meta replica will contain stale data and it is possible that the meta replica > will return back first than primary. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java#L823 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-18027) Replication should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011484#comment-16011484 ] Andrew Purtell edited comment on HBASE-18027 at 5/15/17 11:13 PM: -- Ok, here is a patch that moves the discussed concerns up into the caller(s). This is in response to feedback from [~lhofhansl]. Although inside HBaseInterClusterReplicationEndpoint the batch of edits may be broken up into smaller batches for parallelization, up in the callers we can't see how the sub-batches might be partitioned. So, we have to consider the RPC request limit with respect to the whole batch, which does simplify the change: If the replication batch capacity will exceed the RPC request limit we should simply use the RPC request limit as the replication batch capacity. The downside is this is more pessimistic than if we check limits right where we are building the sub-batches and know the size of each. On the other hand it is easier to understand how the respective configuration parameters will interact. Changes: - Where we set replicationBatchSizeCapacity in ReplicationSourceWALReaderThread, check if replicationBatchSizeCapacity will exceed the RPC request size limit, and if so use the request size limit instead and warn about it with mention of relevant configuration keys. - When building the current replication batch, check if we will exceed the batch size capacity _before_ adding an entry to the batch. Ensure at least one entry is added to the batch even if it means we violate quota or batch size capacity or otherwise replication would become stuck. - WALEntryStream needs a putBack() method to undo next() when we've decided the entry we just iterated to will cause the batch to exceed size limits. and minor changes to logging: - New debug logging in HBaseInterClusterReplicationEndpoint. One new DEBUG level line per replication batch. - Fix trace level log that conflated total amount of data in replication context with size of data in the worklist submitted as Replicator runnable. - Add details to the warning logged if the number of edits replicated is different from the number received. All replication unit tests pass with these changes applied. was (Author: apurtell): Ok, here is a patch that moves the discussed concerns up into the caller(s). This is in response to feedback from [~lhofhansl]. Although inside HBaseInterClusterReplicationEndpoint the batch of edits may be broken up into smaller batches for parallelization, up in the callers we can't see how the sub-batches might be partitioned. So, we have to consider the RPC request limit with respect to the whole batch, which does simplify the change: If the replication batch capacity will exceed the RPC request limit we should simply use the RPC request limit as the replication batch capacity. The downside is this is more pessimistic than if we check limits right where we are building the sub-batches and know the size of each. Changes: - Where we set replicationBatchSizeCapacity in ReplicationSourceWALReaderThread, check if replicationBatchSizeCapacity will exceed the RPC request size limit, and if so use the request size limit instead and warn about it with mention of relevant configuration keys. - When building the current replication batch, check if we will exceed the batch size capacity _before_ adding an entry to the batch. Ensure at least one entry is added to the batch even if it means we violate quota or batch size capacity or otherwise replication would become stuck. - WALEntryStream needs a putBack() method to undo next() when we've decided the entry we just iterated to will cause the batch to exceed size limits. and minor changes to logging: - New debug logging in HBaseInterClusterReplicationEndpoint. One new DEBUG level line per replication batch. - Fix trace level log that conflated total amount of data in replication context with size of data in the worklist submitted as Replicator runnable. - Add details to the warning logged if the number of edits replicated is different from the number received. All replication unit tests pass with these changes applied. > Replication should respect RPC size limits when batching edits > -- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0, 1.3.1 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027.patch, HBASE-18027.patch, HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum of configured
[jira] [Updated] (HBASE-18027) Replication should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18027: --- Attachment: HBASE-18027.patch Ok, here is a patch that moves the discussed concerns up into the caller(s). This is in response to feedback from [~lhofhansl]. Although inside HBaseInterClusterReplicationEndpoint the batch of edits may be broken up into smaller batches for parallelization, up in the callers we can't see how the sub-batches might be partitioned. So, we have to consider the RPC request limit with respect to the whole batch, which does simplify the change: If the replication batch capacity will exceed the RPC request limit we should simply use the RPC request limit as the replication batch capacity. The downside is this is more pessimistic than if we check limits right where we are building the sub-batches and know the size of each. Changes: - Where we set replicationBatchSizeCapacity in ReplicationSourceWALReaderThread, check if replicationBatchSizeCapacity will exceed the RPC request size limit, and if so use the request size limit instead and warn about it with mention of relevant configuration keys. - When building the current replication batch, check if we will exceed the batch size capacity _before_ adding an entry to the batch. Ensure at least one entry is added to the batch even if it means we violate quota or batch size capacity or otherwise replication would become stuck. - WALEntryStream needs a putBack() method to undo next() when we've decided the entry we just iterated to will cause the batch to exceed size limits. and minor changes to logging: - New debug logging in HBaseInterClusterReplicationEndpoint. One new DEBUG level line per replication batch. - Fix trace level log that conflated total amount of data in replication context with size of data in the worklist submitted as Replicator runnable. - Add details to the warning logged if the number of edits replicated is different from the number received. All *Replication* unit tests pass with these changes applied. > Replication should respect RPC size limits when batching edits > -- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0, 1.3.1 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027.patch, HBASE-18027.patch, HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum of configured replicator > threads, number of 100-waledit batches, or number of current sinks. Every > pending entry in the replication context is then placed in order by hash of > encoded region name into one of these N lists. Each of the N lists is then > sent all at once in one replication RPC. We do not test if the sum of data in > each N list will exceed RPC size limits. This code presumes each individual > edit is reasonably small. Not checking for aggregate size while assembling > the lists into RPCs is an oversight and can lead to replication failure when > that assumption is violated. > We can fix this by generating as many replication RPC calls as we need to > drain a list, keeping each RPC under limit, instead of assuming the whole > list will fit in one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-18027) Replication should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011484#comment-16011484 ] Andrew Purtell edited comment on HBASE-18027 at 5/15/17 11:12 PM: -- Ok, here is a patch that moves the discussed concerns up into the caller(s). This is in response to feedback from [~lhofhansl]. Although inside HBaseInterClusterReplicationEndpoint the batch of edits may be broken up into smaller batches for parallelization, up in the callers we can't see how the sub-batches might be partitioned. So, we have to consider the RPC request limit with respect to the whole batch, which does simplify the change: If the replication batch capacity will exceed the RPC request limit we should simply use the RPC request limit as the replication batch capacity. The downside is this is more pessimistic than if we check limits right where we are building the sub-batches and know the size of each. Changes: - Where we set replicationBatchSizeCapacity in ReplicationSourceWALReaderThread, check if replicationBatchSizeCapacity will exceed the RPC request size limit, and if so use the request size limit instead and warn about it with mention of relevant configuration keys. - When building the current replication batch, check if we will exceed the batch size capacity _before_ adding an entry to the batch. Ensure at least one entry is added to the batch even if it means we violate quota or batch size capacity or otherwise replication would become stuck. - WALEntryStream needs a putBack() method to undo next() when we've decided the entry we just iterated to will cause the batch to exceed size limits. and minor changes to logging: - New debug logging in HBaseInterClusterReplicationEndpoint. One new DEBUG level line per replication batch. - Fix trace level log that conflated total amount of data in replication context with size of data in the worklist submitted as Replicator runnable. - Add details to the warning logged if the number of edits replicated is different from the number received. All replication unit tests pass with these changes applied. was (Author: apurtell): Ok, here is a patch that moves the discussed concerns up into the caller(s). This is in response to feedback from [~lhofhansl]. Although inside HBaseInterClusterReplicationEndpoint the batch of edits may be broken up into smaller batches for parallelization, up in the callers we can't see how the sub-batches might be partitioned. So, we have to consider the RPC request limit with respect to the whole batch, which does simplify the change: If the replication batch capacity will exceed the RPC request limit we should simply use the RPC request limit as the replication batch capacity. The downside is this is more pessimistic than if we check limits right where we are building the sub-batches and know the size of each. Changes: - Where we set replicationBatchSizeCapacity in ReplicationSourceWALReaderThread, check if replicationBatchSizeCapacity will exceed the RPC request size limit, and if so use the request size limit instead and warn about it with mention of relevant configuration keys. - When building the current replication batch, check if we will exceed the batch size capacity _before_ adding an entry to the batch. Ensure at least one entry is added to the batch even if it means we violate quota or batch size capacity or otherwise replication would become stuck. - WALEntryStream needs a putBack() method to undo next() when we've decided the entry we just iterated to will cause the batch to exceed size limits. and minor changes to logging: - New debug logging in HBaseInterClusterReplicationEndpoint. One new DEBUG level line per replication batch. - Fix trace level log that conflated total amount of data in replication context with size of data in the worklist submitted as Replicator runnable. - Add details to the warning logged if the number of edits replicated is different from the number received. All *Replication* unit tests pass with these changes applied. > Replication should respect RPC size limits when batching edits > -- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0, 1.3.1 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027.patch, HBASE-18027.patch, HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum of configured replicator > threads, number of 100-waledit batches, or number of current sinks. Every > pending entry
[jira] [Commented] (HBASE-18042) Client Compatibility breaks between versions 1.2 and 1.3
[
https://issues.apache.org/jira/browse/HBASE-18042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011477#comment-16011477
]
Duo Zhang commented on HBASE-18042:
---
The field has been introduced in all branches, even for 0.98. So I think the
fix for AsyncHBase is straightforward? If the field is present then depend on
the value of this field to decide if we need to go to the next region. If not,
use the old logic.
And we also need a fix for all branches below 1.3 I think. If this field is
present then we should not issue a new next request then. We can do it here or
open a new issue.
Thanks.
> Client Compatibility breaks between versions 1.2 and 1.3
>
>
> Key: HBASE-18042
> URL: https://issues.apache.org/jira/browse/HBASE-18042
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.1
>Reporter: Karan Mehta
>Assignee: Karan Mehta
>
> OpenTSDB uses AsyncHBase as its client, rather than using the traditional
> HBase Client. From version 1.2 to 1.3, the {{ClientProtos}} have been
> changed. Newer fields are added to {{ScanResponse}} proto.
> For a typical Scan request in 1.2, would require caller to make an
> OpenScanner Request, GetNextRows Request and a CloseScanner Request, based on
> {{more_rows}} boolean field in the {{ScanResponse}} proto.
> However, from 1.3, new parameter {{more_results_in_region}} was added, which
> limits the results per region. Therefore the client has to now manage sending
> all the requests for each region. Further more, if the results are exhausted
> from a particular region, the {{ScanResponse}} will set
> {{more_results_in_region}} to false, but {{more_results}} can still be true.
> Whenever the former is set to false, the {{RegionScanner}} will also be
> closed.
> OpenTSDB makes an OpenScanner Request and receives all its results in the
> first {{ScanResponse}} itself, thus creating a condition as described in
> above paragraph. Since {{more_rows}} is true, it will proceed to send next
> request at which point the {{RSRpcServices}} will throw
> {{UnknownScannerException}}. The protobuf client compatibility is maintained
> but expected behavior is modified.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18005) read replica: handle the case that region server hosting both primary replica and meta region is down
[
https://issues.apache.org/jira/browse/HBASE-18005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011469#comment-16011469
]
Hadoop QA commented on HBASE-18005:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m 2s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
34m 20s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
33s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 20s
{color} | {color:red} hbase-client generated 1 new + 1 unchanged - 0 fixed = 2
total (was 1) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 41s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 135m 39s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 215m 19s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12868131/HBASE-18005-master-002.patch
|
| JIRA Issue | HBASE-18005 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux ab45fba779fb 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 341223d |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/6793/artifact/patchprocess/whitespace-eol.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-HBASE-Build/6793/artifact/patchprocess/diff-javadoc-javadoc-hbase-client.txt
|
| Test Results |
[jira] [Commented] (HBASE-15370) Backport Moderate Object Storage (MOB) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-15370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011412#comment-16011412 ] Sean Busbey commented on HBASE-15370: - FWIW, I wouldn't say that thread has run its course yet. I agree that it looks like things are bending towards no backport. > Backport Moderate Object Storage (MOB) to branch-1 > -- > > Key: HBASE-15370 > URL: https://issues.apache.org/jira/browse/HBASE-15370 > Project: HBase > Issue Type: Task >Reporter: Ted Yu > Fix For: 1.4.0 > > Attachments: 15370-test.out, HBASE-15370-branch-1.v1.patch, > HBASE-15370-branch-1.v2.patch, merge-conflict.list, mob-cmmits.txt, > mob-commits-v2.txt > > > MOB feature was integrated to master branch half a year ago. > Since then there have been bug fixes which stabilize the feature. > Some customers have been using it at PB scale. > Here is discussion thread on dev mailing list: > http://search-hadoop.com/m/YGbbDSqxD1PYXK62/hbase+MOB+in+branch-1=Re+MOB+in+branch+1+Re+RESULT+VOTE+Merge+branch+hbase+11339+HBase+MOB+to+trunk+ > This issue is to backport MOB feature to branch-1. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15370) Backport Moderate Object Storage (MOB) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-15370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011402#comment-16011402 ] Ted Yu commented on HBASE-15370: This is recent thread on backporting where consensus is not to backport: http://search-hadoop.com/m/HBase/YGbb2NdxC1okisI1?subj=Re+HBASE+and+MOB > Backport Moderate Object Storage (MOB) to branch-1 > -- > > Key: HBASE-15370 > URL: https://issues.apache.org/jira/browse/HBASE-15370 > Project: HBase > Issue Type: Task >Reporter: Ted Yu > Fix For: 1.4.0 > > Attachments: 15370-test.out, HBASE-15370-branch-1.v1.patch, > HBASE-15370-branch-1.v2.patch, merge-conflict.list, mob-cmmits.txt, > mob-commits-v2.txt > > > MOB feature was integrated to master branch half a year ago. > Since then there have been bug fixes which stabilize the feature. > Some customers have been using it at PB scale. > Here is discussion thread on dev mailing list: > http://search-hadoop.com/m/YGbbDSqxD1PYXK62/hbase+MOB+in+branch-1=Re+MOB+in+branch+1+Re+RESULT+VOTE+Merge+branch+hbase+11339+HBase+MOB+to+trunk+ > This issue is to backport MOB feature to branch-1. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-18042) Client Compatibility breaks between versions 1.2 and 1.3
[
https://issues.apache.org/jira/browse/HBASE-18042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011284#comment-16011284
]
Andrew Purtell edited comment on HBASE-18042 at 5/15/17 9:49 PM:
-
We are not supposed to break wire compatibility in minor versions. This is
arguable. The protobuf change is compatible in the strictest sense in that
mismatched versions can still parse messages. The fact that semantics for
handling a given field has changed between older and newer versions, such that
mishandling causes exceptions and scanner failure, means not really.
was (Author: apurtell):
We are not supposed to break wire compatibility in minor versions. This is
arguable. The protobuf change is compatible in the strictest sense in that
mismatched versions can still parse messages. The fact that semantics rely on a
new field not present in older versions means not really.
> Client Compatibility breaks between versions 1.2 and 1.3
>
>
> Key: HBASE-18042
> URL: https://issues.apache.org/jira/browse/HBASE-18042
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.1
>Reporter: Karan Mehta
>Assignee: Karan Mehta
>
> OpenTSDB uses AsyncHBase as its client, rather than using the traditional
> HBase Client. From version 1.2 to 1.3, the {{ClientProtos}} have been
> changed. Newer fields are added to {{ScanResponse}} proto.
> For a typical Scan request in 1.2, would require caller to make an
> OpenScanner Request, GetNextRows Request and a CloseScanner Request, based on
> {{more_rows}} boolean field in the {{ScanResponse}} proto.
> However, from 1.3, new parameter {{more_results_in_region}} was added, which
> limits the results per region. Therefore the client has to now manage sending
> all the requests for each region. Further more, if the results are exhausted
> from a particular region, the {{ScanResponse}} will set
> {{more_results_in_region}} to false, but {{more_results}} can still be true.
> Whenever the former is set to false, the {{RegionScanner}} will also be
> closed.
> OpenTSDB makes an OpenScanner Request and receives all its results in the
> first {{ScanResponse}} itself, thus creating a condition as described in
> above paragraph. Since {{more_rows}} is true, it will proceed to send next
> request at which point the {{RSRpcServices}} will throw
> {{UnknownScannerException}}. The protobuf client compatibility is maintained
> but expected behavior is modified.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-18027) Replication should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18027: --- Summary: Replication should respect RPC size limits when batching edits (was: HBaseInterClusterReplicationEndpoint should respect RPC size limits when batching edits) > Replication should respect RPC size limits when batching edits > -- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0, 1.3.1 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027.patch, HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum of configured replicator > threads, number of 100-waledit batches, or number of current sinks. Every > pending entry in the replication context is then placed in order by hash of > encoded region name into one of these N lists. Each of the N lists is then > sent all at once in one replication RPC. We do not test if the sum of data in > each N list will exceed RPC size limits. This code presumes each individual > edit is reasonably small. Not checking for aggregate size while assembling > the lists into RPCs is an oversight and can lead to replication failure when > that assumption is violated. > We can fix this by generating as many replication RPC calls as we need to > drain a list, keeping each RPC under limit, instead of assuming the whole > list will fit in one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18044) Bug fix in flaky-dashboard-template
[ https://issues.apache.org/jira/browse/HBASE-18044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011344#comment-16011344 ] Appy commented on HBASE-18044: -- Thanks [~stack]. Your suggestion would surely make it easier for me to fix small things in our dev-support. > Bug fix in flaky-dashboard-template > --- > > Key: HBASE-18044 > URL: https://issues.apache.org/jira/browse/HBASE-18044 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-18044.master.001.patch > > > Due to some scoping issues, counters don't work in jinja templates like we > expect them to (in java/c++,etc). > (http://stackoverflow.com/questions/7537439/how-to-increment-a-variable-on-a-for-loop-in-jinja-template). > Due to this, clicking show/hide button for tests which fail in multiple urls > will not work sometime. Fixing it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18044) Bug fix in flaky-dashboard-template
[ https://issues.apache.org/jira/browse/HBASE-18044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-18044: - Resolution: Fixed Status: Resolved (was: Patch Available) > Bug fix in flaky-dashboard-template > --- > > Key: HBASE-18044 > URL: https://issues.apache.org/jira/browse/HBASE-18044 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-18044.master.001.patch > > > Due to some scoping issues, counters don't work in jinja templates like we > expect them to (in java/c++,etc). > (http://stackoverflow.com/questions/7537439/how-to-increment-a-variable-on-a-for-loop-in-jinja-template). > Due to this, clicking show/hide button for tests which fail in multiple urls > will not work sometime. Fixing it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18042) Client Compatibility breaks between versions 1.2 and 1.3
[
https://issues.apache.org/jira/browse/HBASE-18042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011327#comment-16011327
]
Benoit Sigoure commented on HBASE-18042:
We can easily update AsyncHBase to accommodate to the change however I would
like to voice disagreement with this statement:
{quote}
It is an unfortunate thing that we have broken the semantics, but in general
this is "allowed".
{quote}
Such semantic changes are like breaking API changes, they are, well, breaking
changes. Not cool.
One of the challenges with AsyncHBase is that it has to work with all versions
of HBase. Since {{more_results_in_region}} was already there in 1.2 but needs
to be handled differently in 1.3, that makes it kinda hard for AsyncHBase to
know how, exactly, to deal with this flag being set, right?
> Client Compatibility breaks between versions 1.2 and 1.3
>
>
> Key: HBASE-18042
> URL: https://issues.apache.org/jira/browse/HBASE-18042
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.1
>Reporter: Karan Mehta
>Assignee: Karan Mehta
>
> OpenTSDB uses AsyncHBase as its client, rather than using the traditional
> HBase Client. From version 1.2 to 1.3, the {{ClientProtos}} have been
> changed. Newer fields are added to {{ScanResponse}} proto.
> For a typical Scan request in 1.2, would require caller to make an
> OpenScanner Request, GetNextRows Request and a CloseScanner Request, based on
> {{more_rows}} boolean field in the {{ScanResponse}} proto.
> However, from 1.3, new parameter {{more_results_in_region}} was added, which
> limits the results per region. Therefore the client has to now manage sending
> all the requests for each region. Further more, if the results are exhausted
> from a particular region, the {{ScanResponse}} will set
> {{more_results_in_region}} to false, but {{more_results}} can still be true.
> Whenever the former is set to false, the {{RegionScanner}} will also be
> closed.
> OpenTSDB makes an OpenScanner Request and receives all its results in the
> first {{ScanResponse}} itself, thus creating a condition as described in
> above paragraph. Since {{more_rows}} is true, it will proceed to send next
> request at which point the {{RSRpcServices}} will throw
> {{UnknownScannerException}}. The protobuf client compatibility is maintained
> but expected behavior is modified.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-18043) Institute a hard limit for individual cell size that cannot be overridden by clients
[ https://issues.apache.org/jira/browse/HBASE-18043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18043: --- Attachment: HBASE-18043.patch > Institute a hard limit for individual cell size that cannot be overridden by > clients > > > Key: HBASE-18043 > URL: https://issues.apache.org/jira/browse/HBASE-18043 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, regionserver >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-18043-branch-1.patch, HBASE-18043-branch-1.patch, > HBASE-18043-branch-1.patch, HBASE-18043-branch-1.patch, HBASE-18043.patch, > HBASE-18043.patch, HBASE-18043.patch, HBASE-18043.patch, HBASE-18043.patch, > HBASE-18043.patch > > > For sake of service protection we should not give absolute trust to clients > regarding resource limits that can impact stability, like cell size limits. > We should add a server side configuration that sets a hard limit for > individual cell size that cannot be overridden by the client. We can keep the > client side check, because it's expensive to reject a RPC that has already > come in. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18043) Institute a hard limit for individual cell size that cannot be overridden by clients
[ https://issues.apache.org/jira/browse/HBASE-18043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011301#comment-16011301 ] Andrew Purtell commented on HBASE-18043: Thanks for the reminder. I thought I had removed it. > Institute a hard limit for individual cell size that cannot be overridden by > clients > > > Key: HBASE-18043 > URL: https://issues.apache.org/jira/browse/HBASE-18043 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, regionserver >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-18043-branch-1.patch, HBASE-18043-branch-1.patch, > HBASE-18043-branch-1.patch, HBASE-18043-branch-1.patch, HBASE-18043.patch, > HBASE-18043.patch, HBASE-18043.patch, HBASE-18043.patch, HBASE-18043.patch, > HBASE-18043.patch > > > For sake of service protection we should not give absolute trust to clients > regarding resource limits that can impact stability, like cell size limits. > We should add a server side configuration that sets a hard limit for > individual cell size that cannot be overridden by the client. We can keep the > client side check, because it's expensive to reject a RPC that has already > come in. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17850) Backup system restore /repair utility
[
https://issues.apache.org/jira/browse/HBASE-17850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011285#comment-16011285
]
Vladimir Rodionov commented on HBASE-17850:
---
Here is RB:
https://reviews.apache.org/r/59292/
> Backup system restore /repair utility
> -
>
> Key: HBASE-17850
> URL: https://issues.apache.org/jira/browse/HBASE-17850
> Project: HBase
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-17850-v2.patch
>
>
> Backup repair tool restores integrity of a backup system table and removes
> artefacts of a failed backup session from a file system(s)
> This is command-line tool. To run the tool:
> {code}
> hbase backup repair
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18042) Client Compatibility breaks between versions 1.2 and 1.3
[
https://issues.apache.org/jira/browse/HBASE-18042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011284#comment-16011284
]
Andrew Purtell commented on HBASE-18042:
We are not supposed to break wire compatibility in minor versions. This is
arguable. The protobuf change is compatible in the strictest sense in that
mismatched versions can still parse messages. The fact that semantics rely on a
new field not present in older versions means not really.
> Client Compatibility breaks between versions 1.2 and 1.3
>
>
> Key: HBASE-18042
> URL: https://issues.apache.org/jira/browse/HBASE-18042
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.1
>Reporter: Karan Mehta
>Assignee: Karan Mehta
>
> OpenTSDB uses AsyncHBase as its client, rather than using the traditional
> HBase Client. From version 1.2 to 1.3, the {{ClientProtos}} have been
> changed. Newer fields are added to {{ScanResponse}} proto.
> For a typical Scan request in 1.2, would require caller to make an
> OpenScanner Request, GetNextRows Request and a CloseScanner Request, based on
> {{more_rows}} boolean field in the {{ScanResponse}} proto.
> However, from 1.3, new parameter {{more_results_in_region}} was added, which
> limits the results per region. Therefore the client has to now manage sending
> all the requests for each region. Further more, if the results are exhausted
> from a particular region, the {{ScanResponse}} will set
> {{more_results_in_region}} to false, but {{more_results}} can still be true.
> Whenever the former is set to false, the {{RegionScanner}} will also be
> closed.
> OpenTSDB makes an OpenScanner Request and receives all its results in the
> first {{ScanResponse}} itself, thus creating a condition as described in
> above paragraph. Since {{more_rows}} is true, it will proceed to send next
> request at which point the {{RSRpcServices}} will throw
> {{UnknownScannerException}}. The protobuf client compatibility is maintained
> but expected behavior is modified.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17850) Backup system restore /repair utility
[
https://issues.apache.org/jira/browse/HBASE-17850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011283#comment-16011283
]
Vladimir Rodionov commented on HBASE-17850:
---
{quote}
If the test fails in one run, how do we reproduce the failure ? Next run may
inject error at different stage.
{quote}
I can loop over all failure stages 0 ..4 sequentially, but it will increase
execution time.
> Backup system restore /repair utility
> -
>
> Key: HBASE-17850
> URL: https://issues.apache.org/jira/browse/HBASE-17850
> Project: HBase
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-17850-v2.patch
>
>
> Backup repair tool restores integrity of a backup system table and removes
> artefacts of a failed backup session from a file system(s)
> This is command-line tool. To run the tool:
> {code}
> hbase backup repair
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17850) Backup system restore /repair utility
[
https://issues.apache.org/jira/browse/HBASE-17850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011281#comment-16011281
]
Vladimir Rodionov commented on HBASE-17850:
---
{quote}
Seems the current FullTableBackupClientForTest is closer to real client.
{quote}
Not really, client handles failures automatically. The only case that client
does not handle is when client application exits abnormally.
> Backup system restore /repair utility
> -
>
> Key: HBASE-17850
> URL: https://issues.apache.org/jira/browse/HBASE-17850
> Project: HBase
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-17850-v2.patch
>
>
> Backup repair tool restores integrity of a backup system table and removes
> artefacts of a failed backup session from a file system(s)
> This is command-line tool. To run the tool:
> {code}
> hbase backup repair
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18042) Client Compatibility breaks between versions 1.2 and 1.3
[
https://issues.apache.org/jira/browse/HBASE-18042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011277#comment-16011277
]
Lars Hofhansl commented on HBASE-18042:
---
So just to be clear: The HBase 1.2.x client will successfully work with a 1.3.x
cluster, right? Including no leaked scanners that have to wait for their leases
to expire... (Just making sure)
[~enis] can you confirm?
If 1.2.x clients work fine then it seems we should update asynchbase with this
new behavior. [~tsuna], in case you have an opinion.
> Client Compatibility breaks between versions 1.2 and 1.3
>
>
> Key: HBASE-18042
> URL: https://issues.apache.org/jira/browse/HBASE-18042
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.1
>Reporter: Karan Mehta
>Assignee: Karan Mehta
>
> OpenTSDB uses AsyncHBase as its client, rather than using the traditional
> HBase Client. From version 1.2 to 1.3, the {{ClientProtos}} have been
> changed. Newer fields are added to {{ScanResponse}} proto.
> For a typical Scan request in 1.2, would require caller to make an
> OpenScanner Request, GetNextRows Request and a CloseScanner Request, based on
> {{more_rows}} boolean field in the {{ScanResponse}} proto.
> However, from 1.3, new parameter {{more_results_in_region}} was added, which
> limits the results per region. Therefore the client has to now manage sending
> all the requests for each region. Further more, if the results are exhausted
> from a particular region, the {{ScanResponse}} will set
> {{more_results_in_region}} to false, but {{more_results}} can still be true.
> Whenever the former is set to false, the {{RegionScanner}} will also be
> closed.
> OpenTSDB makes an OpenScanner Request and receives all its results in the
> first {{ScanResponse}} itself, thus creating a condition as described in
> above paragraph. Since {{more_rows}} is true, it will proceed to send next
> request at which point the {{RSRpcServices}} will throw
> {{UnknownScannerException}}. The protobuf client compatibility is maintained
> but expected behavior is modified.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17489) ClientScanner may send a next request to a RegionScanner which has been exhausted
[ https://issues.apache.org/jira/browse/HBASE-17489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011270#comment-16011270 ] Lars Hofhansl commented on HBASE-17489: --- Thanks [~tsuna] (has been a while) :) All good now. For correct asynchronous behavior it seems generally we do not want to impede our ability to link different callbacks back to their originators - even when this is not needed in this particular case. (We fixed it in HBase ... but ... if you had an opinion on HBASE-18042 that would be great!) > ClientScanner may send a next request to a RegionScanner which has been > exhausted > - > > Key: HBASE-17489 > URL: https://issues.apache.org/jira/browse/HBASE-17489 > Project: HBase > Issue Type: Bug > Components: Client, scan >Affects Versions: 2.0.0, 1.3.0, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0, 1.4.0, 1.3.1 > > Attachments: HBASE-17489-branch-1.3.patch, > HBASE-17489-branch-1.patch, HBASE-17489.patch, HBASE-17489-v1.patch, > HBASE-17489-v2.patch, HBASE-17489-v3.patch, HBASE-17489-v4.patch, > HBASE-17489-v4.patch, HBASE-17489-v5.patch, HBASE-17489-v6.patch > > > Found it when implementing HBASE-17045. Seems the final result of the scan is > correct but no doubt the logic is broken. We need to fix it to stop things > get worse in the future. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18043) Institute a hard limit for individual cell size that cannot be overridden by clients
[
https://issues.apache.org/jira/browse/HBASE-18043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011241#comment-16011241
]
Chia-Ping Tsai commented on HBASE-18043:
{code}
- * Licensed to the Apache Software Foundation (ASF) under one
+hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * Licensed
to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
{code}
just remind you about the paste-o.
> Institute a hard limit for individual cell size that cannot be overridden by
> clients
>
>
> Key: HBASE-18043
> URL: https://issues.apache.org/jira/browse/HBASE-18043
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, regionserver
>Affects Versions: 2.0.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-18043-branch-1.patch, HBASE-18043-branch-1.patch,
> HBASE-18043-branch-1.patch, HBASE-18043-branch-1.patch, HBASE-18043.patch,
> HBASE-18043.patch, HBASE-18043.patch, HBASE-18043.patch, HBASE-18043.patch
>
>
> For sake of service protection we should not give absolute trust to clients
> regarding resource limits that can impact stability, like cell size limits.
> We should add a server side configuration that sets a hard limit for
> individual cell size that cannot be overridden by the client. We can keep the
> client side check, because it's expensive to reject a RPC that has already
> come in.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18049) It is not necessary to re-open the region when MOB files cannot be found
[
https://issues.apache.org/jira/browse/HBASE-18049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011208#comment-16011208
]
Ted Yu commented on HBASE-18049:
lgtm
> It is not necessary to re-open the region when MOB files cannot be found
>
>
> Key: HBASE-18049
> URL: https://issues.apache.org/jira/browse/HBASE-18049
> Project: HBase
> Issue Type: Bug
> Components: Scanners
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Fix For: 2.0.0
>
> Attachments: HBASE-18049.patch
>
>
> In HBASE-17712, we try to re-open the region when store files cannot be
> found. This is useful for store files in a region, but is not necessary when
> the MOB files cannot be found, because the store files in a region only
> contain the references to the MOB files and a re-open of a region doesn't
> help the lost MOB files.
> In this JIRA, we will directly throw DNRIOE only when the MOB files are not
> found in {{MobStoreScanner}} and {{ReversedMobStoreScanner}}. Other logics
> keep the same.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-18027) HBaseInterClusterReplicationEndpoint should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18027: --- Attachment: (was: HBASE-18027-branch-1.patch) > HBaseInterClusterReplicationEndpoint should respect RPC size limits when > batching edits > --- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0, 1.3.1 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027.patch, HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum of configured replicator > threads, number of 100-waledit batches, or number of current sinks. Every > pending entry in the replication context is then placed in order by hash of > encoded region name into one of these N lists. Each of the N lists is then > sent all at once in one replication RPC. We do not test if the sum of data in > each N list will exceed RPC size limits. This code presumes each individual > edit is reasonably small. Not checking for aggregate size while assembling > the lists into RPCs is an oversight and can lead to replication failure when > that assumption is violated. > We can fix this by generating as many replication RPC calls as we need to > drain a list, keeping each RPC under limit, instead of assuming the whole > list will fit in one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18027) HBaseInterClusterReplicationEndpoint should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18027: --- Attachment: (was: HBASE-18027.patch) > HBaseInterClusterReplicationEndpoint should respect RPC size limits when > batching edits > --- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0, 1.3.1 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027.patch, HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum of configured replicator > threads, number of 100-waledit batches, or number of current sinks. Every > pending entry in the replication context is then placed in order by hash of > encoded region name into one of these N lists. Each of the N lists is then > sent all at once in one replication RPC. We do not test if the sum of data in > each N list will exceed RPC size limits. This code presumes each individual > edit is reasonably small. Not checking for aggregate size while assembling > the lists into RPCs is an oversight and can lead to replication failure when > that assumption is violated. > We can fix this by generating as many replication RPC calls as we need to > drain a list, keeping each RPC under limit, instead of assuming the whole > list will fit in one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18027) HBaseInterClusterReplicationEndpoint should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18027: --- Attachment: (was: HBASE-18027-branch-1.patch) > HBaseInterClusterReplicationEndpoint should respect RPC size limits when > batching edits > --- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0, 1.3.1 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027.patch, HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum of configured replicator > threads, number of 100-waledit batches, or number of current sinks. Every > pending entry in the replication context is then placed in order by hash of > encoded region name into one of these N lists. Each of the N lists is then > sent all at once in one replication RPC. We do not test if the sum of data in > each N list will exceed RPC size limits. This code presumes each individual > edit is reasonably small. Not checking for aggregate size while assembling > the lists into RPCs is an oversight and can lead to replication failure when > that assumption is violated. > We can fix this by generating as many replication RPC calls as we need to > drain a list, keeping each RPC under limit, instead of assuming the whole > list will fit in one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
