[jira] [Commented] (HBASE-18532) Improve cache related stats rendered on RS UI
[ https://issues.apache.org/jira/browse/HBASE-18532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124476#comment-16124476 ] Anoop Sam John commented on HBASE-18532: Ya it make sense to remove rendering the large list of bucket info. The bucket info can be presented with its own link and in a paging fashion (that might be the best). Here the fix you would do will be proper size accounting right? We were doing it based on the 100K limited list but that is wrong. provide a patch? > Improve cache related stats rendered on RS UI > - > > Key: HBASE-18532 > URL: https://issues.apache.org/jira/browse/HBASE-18532 > Project: HBase > Issue Type: Improvement > Components: regionserver, UI >Affects Versions: 1.1.2 >Reporter: Biju Nair > Attachments: COMBINED-STATS.PNG, L1-STATS.PNG, L2-STATS.PNG > > > The stats currently rendered for L1 and L2 cache are incorrect. Refer to the > attached screenshots of stats from a cluster showing the combined cache > stats, L1 stats and L2 stats. For e.g. the combined stats shows 38 GB used > for cache while if we sum size of L1 and L2 cache the value is way less. One > way we can improve this is to use the same stats used to populate the > combined stats to render the values of L1 & L2 cache. Also for usability we > can remove the table with details with BucketCache buckets from the L2 cache > stats since this is going to be long for any installation using L2 cache. > This will help in understanding the cache usage better. Thoughts? If there > are no concerns will submit a patch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18526) FIFOCompactionPolicy pre-check uses wrong scope
[
https://issues.apache.org/jira/browse/HBASE-18526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124454#comment-16124454
]
Hudson commented on HBASE-18526:
SUCCESS: Integrated in Jenkins build HBase-2.0 #313 (See
[https://builds.apache.org/job/HBase-2.0/313/])
HBASE-18526 FIFOCompactionPolicy pre-check uses wrong scope (Vladimir (tedyu:
rev b5d4a0aa412c5cd086d03c99f35c9bf66958cc79)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
> FIFOCompactionPolicy pre-check uses wrong scope
> ---
>
> Key: HBASE-18526
> URL: https://issues.apache.org/jira/browse/HBASE-18526
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 1.3.1
>Reporter: Lars George
>Assignee: Vladimir Rodionov
> Attachments: HBASE-18526-v1.patch
>
>
> See https://issues.apache.org/jira/browse/HBASE-14468
> It adds this check to {{HMaster.checkCompactionPolicy()}}:
> {code}
> // 1. Check TTL
> if (hcd.getTimeToLive() == HColumnDescriptor.DEFAULT_TTL) {
> message = "Default TTL is not supported for FIFO compaction";
> throw new IOException(message);
> }
> // 2. Check min versions
> if (hcd.getMinVersions() > 0) {
> message = "MIN_VERSION > 0 is not supported for FIFO compaction";
> throw new IOException(message);
> }
> // 3. blocking file count
> String sbfc = htd.getConfigurationValue(HStore.BLOCKING_STOREFILES_KEY);
> if (sbfc != null) {
> blockingFileCount = Integer.parseInt(sbfc);
> }
> if (blockingFileCount < 1000) {
> message =
> "blocking file count '" + HStore.BLOCKING_STOREFILES_KEY + "' "
> + blockingFileCount
> + " is below recommended minimum of 1000";
> throw new IOException(message);
> }
> {code}
> Why does it only check the blocking file count on the HTD level, while
> others are check on the HCD level? Doing this for example fails
> because of it:
> {noformat}
> hbase(main):008:0> create 'ttltable', { NAME => 'cf1', TTL => 300,
> CONFIGURATION => { 'hbase.hstore.defaultengine.compactionpolicy.class'
> => 'org.apache.hadoop.hbase.regionserver.compactions.FIFOCompactionPolicy',
> 'hbase.hstore.blockingStoreFiles' => 2000 } }
> ERROR: org.apache.hadoop.hbase.DoNotRetryIOException: blocking file
> count 'hbase.hstore.blockingStoreFiles' 10 is below recommended
> minimum of 1000 Set hbase.table.sanity.checks to false at conf or
> table descriptor if you want to bypass sanity checks
> at
> org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:1782)
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1663)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1545)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:469)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58549)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
> Caused by: java.io.IOException: blocking file count
> 'hbase.hstore.blockingStoreFiles' 10 is below recommended minimum of
> 1000
> at
> org.apache.hadoop.hbase.master.HMaster.checkCompactionPolicy(HMaster.java:1773)
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1661)
> ... 7 more
> {noformat}
> The check should be performed on the column family level instead.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18555) Remove redundant familyMap.put() from addxxx() of sub-classes of Mutation and Query
[
https://issues.apache.org/jira/browse/HBASE-18555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124453#comment-16124453
]
Hadoop QA commented on HBASE-18555:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
34m 48s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
39s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}119m
28s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
49s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}181m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:bdc94b1 |
| JIRA Issue | HBASE-18555 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881590/HBASE-18555.master.002.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 04d87ab36481 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / aa8f67a |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/8054/testReport/ |
| modules | C: hbase-client hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/8054/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was automatically generated.
> Remove redundant familyMap.put()
[jira] [Commented] (HBASE-15511) ClusterStatus should be able to return responses by scope
[
https://issues.apache.org/jira/browse/HBASE-15511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124448#comment-16124448
]
Hadoop QA commented on HBASE-15511:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
32m 49s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
1m 20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
18s{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
32s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}109m
35s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
59s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}179m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:bdc94b1 |
| JIRA Issue | HBASE-15511 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881588/HBASE-15511.master.014.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile cc hbaseprotoc |
| uname | Linux bd244671ee48 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
|
[jira] [Commented] (HBASE-18409) Migrate Client Metrics from codahale to hbase-metrics
[ https://issues.apache.org/jira/browse/HBASE-18409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124431#comment-16124431 ] Ronald Macmaster commented on HBASE-18409: -- What do you all think? Can we apply this patch? > Migrate Client Metrics from codahale to hbase-metrics > - > > Key: HBASE-18409 > URL: https://issues.apache.org/jira/browse/HBASE-18409 > Project: HBase > Issue Type: Improvement > Components: Client, java, metrics >Affects Versions: 3.0.0 >Reporter: Ronald Macmaster > Labels: newbie > Fix For: 3.0.0 > > Attachments: > 0001-HBASE-18409-MetricsConnection-client-metrics-migration.patch, > 0002-HBASE-18409-MetricsConnection-client-metrics-migrati.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, the metrics for hbase-client are tailored for reporting via a > client-side JMX server. > The MetricsConnection handles the metrics management and reporting via the > metrics platform from codahale. > This approach worked well for hbase-1.3.1 when the metrics platform was still > relatively young, but it could be improved by using the new > hbase-metrics-api. > Now that we have an actual hbase-metrics-api that master, regionserver, > zookeeper, and other daemons use, it would be good to also allow the client > to leverage the metrics-api. > Then, the client could also report its metrics via Hadoop's metrics2 if > desired or through another platform that utilizes the hbase-metrics-api. > If left alone, client metrics will continue to be only barely visible through > a client-side JMX server. > The migration to the new metrics-api could be done by simply changing the > Metrics data types from codahale types to hbase-metrics types without > changing the metrics signatures of MetricsConnection unless completely > necessary. > The codahale MetricsRegistry would also have to be exchanged for a > hbase-metrics MetricsRegistry. > I found this to be a necessary change after attempting to implement my own > Reporter to use within the MetricsConnection class. > I was attempting to create a HadoopMetrics2Reporter that extends the codahale > ScheduledReporter and reports the MetricsConnection metrics to Hadoop's > metrics2 system. > The already existing infrastructure in the hbase-metrics and > hbase-metrics-api projects could be easily leveraged for a cleaner solution. > If completed successfully, users could instead access their client-side > metrics through the hbase-metrics-api. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18511) Default no regions on master
[
https://issues.apache.org/jira/browse/HBASE-18511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124427#comment-16124427
]
Hadoop QA commented on HBASE-18511:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 20 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 40s{color}
| {color:red} hbase-server generated 2 new + 6 unchanged - 0 fixed = 8 total
(was 6) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 39s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}136m 45s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}183m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.regionserver.TestRegionServerReadRequestMetrics |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:bdc94b1 |
| JIRA Issue | HBASE-18511 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881579/HBASE-18511.master.006.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 32b8683a74de 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / aa8f67a |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC3 |
| javac |
https://builds.apache.org/job/PreCommit-HBASE-Build/8052/artifact/patchprocess/diff-compile-javac-hbase-server.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/8052/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/8052/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/8052/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was automatically generated.
> Default no regions on master
>
>
>
[jira] [Commented] (HBASE-18526) FIFOCompactionPolicy pre-check uses wrong scope
[
https://issues.apache.org/jira/browse/HBASE-18526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124407#comment-16124407
]
Hudson commented on HBASE-18526:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3516 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/3516/])
HBASE-18526 FIFOCompactionPolicy pre-check uses wrong scope (Vladimir (tedyu:
rev aa8f67a148cbefbfc4bfdc25b2dc48c7ed947212)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
> FIFOCompactionPolicy pre-check uses wrong scope
> ---
>
> Key: HBASE-18526
> URL: https://issues.apache.org/jira/browse/HBASE-18526
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 1.3.1
>Reporter: Lars George
>Assignee: Vladimir Rodionov
> Attachments: HBASE-18526-v1.patch
>
>
> See https://issues.apache.org/jira/browse/HBASE-14468
> It adds this check to {{HMaster.checkCompactionPolicy()}}:
> {code}
> // 1. Check TTL
> if (hcd.getTimeToLive() == HColumnDescriptor.DEFAULT_TTL) {
> message = "Default TTL is not supported for FIFO compaction";
> throw new IOException(message);
> }
> // 2. Check min versions
> if (hcd.getMinVersions() > 0) {
> message = "MIN_VERSION > 0 is not supported for FIFO compaction";
> throw new IOException(message);
> }
> // 3. blocking file count
> String sbfc = htd.getConfigurationValue(HStore.BLOCKING_STOREFILES_KEY);
> if (sbfc != null) {
> blockingFileCount = Integer.parseInt(sbfc);
> }
> if (blockingFileCount < 1000) {
> message =
> "blocking file count '" + HStore.BLOCKING_STOREFILES_KEY + "' "
> + blockingFileCount
> + " is below recommended minimum of 1000";
> throw new IOException(message);
> }
> {code}
> Why does it only check the blocking file count on the HTD level, while
> others are check on the HCD level? Doing this for example fails
> because of it:
> {noformat}
> hbase(main):008:0> create 'ttltable', { NAME => 'cf1', TTL => 300,
> CONFIGURATION => { 'hbase.hstore.defaultengine.compactionpolicy.class'
> => 'org.apache.hadoop.hbase.regionserver.compactions.FIFOCompactionPolicy',
> 'hbase.hstore.blockingStoreFiles' => 2000 } }
> ERROR: org.apache.hadoop.hbase.DoNotRetryIOException: blocking file
> count 'hbase.hstore.blockingStoreFiles' 10 is below recommended
> minimum of 1000 Set hbase.table.sanity.checks to false at conf or
> table descriptor if you want to bypass sanity checks
> at
> org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:1782)
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1663)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1545)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:469)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58549)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
> Caused by: java.io.IOException: blocking file count
> 'hbase.hstore.blockingStoreFiles' 10 is below recommended minimum of
> 1000
> at
> org.apache.hadoop.hbase.master.HMaster.checkCompactionPolicy(HMaster.java:1773)
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1661)
> ... 7 more
> {noformat}
> The check should be performed on the column family level instead.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18271) Shade netty
[ https://issues.apache.org/jira/browse/HBASE-18271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124406#comment-16124406 ] stack commented on HBASE-18271: --- Rebase and retry. > Shade netty > --- > > Key: HBASE-18271 > URL: https://issues.apache.org/jira/browse/HBASE-18271 > Project: HBase > Issue Type: Sub-task > Components: rpc >Affects Versions: 2.0.0 >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-18271.master.001.patch, > HBASE-18271.master.002.patch, HBASE-18271.master.003.patch, > HBASE-18271-Shade-netty-Purge-mention-of-netty-all.003.patch > > > Our new prefatory project, hbase-thirdparty, includes a relocated netty > 4.1.12. This issue is about changing references in hbase to make use of this > shaded netty. This way we will take ourselves out of the clashing library > saga and change netty as we see fit. > One kink is the inclusion inside the netty-all jar of an .so. We need to make > sure it gets loaded and that on linux we are doing native epoll; all should > be in place to do this but need to add a system property to bin/hbase for > netty to pick up. TODO as part of this issue. See tail of upgrade guava > sibling issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18271) Shade netty
[ https://issues.apache.org/jira/browse/HBASE-18271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-18271: -- Attachment: HBASE-18271.master.003.patch > Shade netty > --- > > Key: HBASE-18271 > URL: https://issues.apache.org/jira/browse/HBASE-18271 > Project: HBase > Issue Type: Sub-task > Components: rpc >Affects Versions: 2.0.0 >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-18271.master.001.patch, > HBASE-18271.master.002.patch, HBASE-18271.master.003.patch, > HBASE-18271-Shade-netty-Purge-mention-of-netty-all.003.patch > > > Our new prefatory project, hbase-thirdparty, includes a relocated netty > 4.1.12. This issue is about changing references in hbase to make use of this > shaded netty. This way we will take ourselves out of the clashing library > saga and change netty as we see fit. > One kink is the inclusion inside the netty-all jar of an .so. We need to make > sure it gets loaded and that on linux we are doing native epoll; all should > be in place to do this but need to add a system property to bin/hbase for > netty to pick up. TODO as part of this issue. See tail of upgrade guava > sibling issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18581) Remove dead code and some tidy up in BaseLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-18581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124403#comment-16124403
]
Hadoop QA commented on HBASE-18581:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
32m 10s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}117m
6s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}165m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:bdc94b1 |
| JIRA Issue | HBASE-18581 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881569/hbase-18581.master.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux f801b935eaa8 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 1070888 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/8051/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/8051/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was automatically generated.
> Remove dead code and some tidy up in BaseLoadBalancer
> -
>
> Key: HBASE-18581
> URL: https://issues.apache.org/jira/browse/HBASE-18581
> Project: HBase
> Issue Type: Improvement
> Components: Balancer
>Reporter: Umesh
[jira] [Commented] (HBASE-18424) Fix TestAsyncTableGetMultiThreaded
[
https://issues.apache.org/jira/browse/HBASE-18424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124393#comment-16124393
]
Hadoop QA commented on HBASE-18424:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m
36s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
32m 37s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}122m
11s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}191m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.13.1 Server=1.13.1 Image:yetus/hbase:bdc94b1 |
| JIRA Issue | HBASE-18424 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881557/HBASE-18424-v2.patch |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 460a03511e2a 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 1070888 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/8050/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/8050/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was automatically generated.
> Fix TestAsyncTableGetMultiThreaded
> --
>
> Key: HBASE-18424
> URL: https://issues.apache.org/jira/browse/HBASE-18424
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-18424-v1.patch, HBASE-18424-v2.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18025) CatalogJanitor should collect outdated RegionStates from the AM
[ https://issues.apache.org/jira/browse/HBASE-18025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124390#comment-16124390 ] Hudson commented on HBASE-18025: FAILURE: Integrated in Jenkins build HBase-2.0 #312 (See [https://builds.apache.org/job/HBase-2.0/312/]) HBASE-18025 CatalogJanitor should collect outdated RegionStates from the (esteban: rev b627cfad35bb7d925506d043f62ff69b0d57869d) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitorInMemoryStates.java > CatalogJanitor should collect outdated RegionStates from the AM > --- > > Key: HBASE-18025 > URL: https://issues.apache.org/jira/browse/HBASE-18025 > Project: HBase > Issue Type: Bug >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez > Fix For: 2.0.0, 3.0.0, 1.4.0, 1.5.0 > > Attachments: HBASE-18025.001.patch, HBASE-18025.002.patch, > HBASE-18025.003.patch, HBASE-18025.004.patch, HBASE-18025.005.patch, > HBASE-18025-branch-1.005.patch, HBASE-18025-branch-1.006.patch > > > I don't think this will matter on the long run for HBase 2, but at least in > branch-1 and the current master we keep in multiple places copies of the > region states in the master and this copies include information like the HRI. > A problem that we have observed is when region replicas are being used and > there is a split, the region replica from parent doesn't get collected from > the region states and when the balancer tries to assign the old parent region > replica, this will cause the RegionServer to create a new HRI with the > details of the parent causing an inconstancy (see HBASE-18024). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18551) [AMv2] UnassignProcedure and crashed regionservers
[ https://issues.apache.org/jira/browse/HBASE-18551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124392#comment-16124392 ] Hudson commented on HBASE-18551: FAILURE: Integrated in Jenkins build HBase-2.0 #312 (See [https://builds.apache.org/job/HBase-2.0/312/]) HBASE-18551 [AMv2] UnassignProcedure and crashed regionservers; (stack: rev 7197b40cbfe0599fa792b8152ed94761377e75e3) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestAssignmentManager.java > [AMv2] UnassignProcedure and crashed regionservers > -- > > Key: HBASE-18551 > URL: https://issues.apache.org/jira/browse/HBASE-18551 > Project: HBase > Issue Type: Bug > Components: amv2 >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: HBASE-18551.master.001.patch, > HBASE-18551.master.002.patch, HBASE-18551.master.003.patch > > > This has been [~uagashe] and my obsession over the last few days, what should > an UnassignProcedure do when it dispatches a CLOSE but the CLOSE fails > because of ConnectException or SocketTimeout. > + We used to let UnassignProcedure continue presuming the Region would be > closed since the server is dead. BUT, if the unassign was part of a > MoveProcedure, the unassign would proceed and the Move would then run WITHOUT > first splitting logs. Bad. > + So, we made it so UnassignProcedure failed; let the upper layers take care > of the failure. See HBASE-18491 that enabled this behavior. BUT, we are since > figuring that even if the UP completes as a failure, since it gives up the > Region lock on completion, another procedure -- say an AssignProcedure -- > could cut in before the ServerCrashProcedure had finished and again there > could be dataloss. > + Now we are thinking the UP should hold on to the Region lock until we are > signalled by a ServerCrashProcedure; only then let go of the region. The UP > has context that is hard to pass another. Waiting on a SCP has the UP living > on for what could be a good amount of time. It might be ok if we can suspend > the procedure. > There is a good sample scenario that came up doing the no-regions-on-master > issue, HBASE-18511. When meta is not on master, TestSplitTransactionOnCluster > is failing. It fails because though the test completes, the tests commonly > kill a RegionServer. The teardown for the test runs before we've noticed the > aborted RS. So, the disable of the table in the teardown prepartory to our > deleting the test table as part of clean up, goes to unassign regions but the > unassign fails against the aborted server. > Good stuff. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18557) change splitable to mergeable in MergeTableRegionsProcedure
[
https://issues.apache.org/jira/browse/HBASE-18557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124391#comment-16124391
]
Hudson commented on HBASE-18557:
FAILURE: Integrated in Jenkins build HBase-2.0 #312 (See
[https://builds.apache.org/job/HBase-2.0/312/])
HBASE-18557: Change splitable to mergeable in MergeTableRegionsProcedure
(jerryjch: rev b5a4e07c6a521b13146daccb7a9d502317fde427)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
> change splitable to mergeable in MergeTableRegionsProcedure
> ---
>
> Key: HBASE-18557
> URL: https://issues.apache.org/jira/browse/HBASE-18557
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-18557-master-v1.patch, HBASE-18557-master-v1.patch
>
>
> {code}
> private boolean isMergeable(final MasterProcedureEnv env, final RegionState
> rs)
> throws IOException {
> GetRegionInfoResponse response =
> Util.getRegionInfoResponse(env, rs.getServerName(), rs.getRegion());
> return response.hasSplittable() && response.getSplittable();
> }
> {code}
> should be mergeable
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18555) Remove redundant familyMap.put() from addxxx() of sub-classes of Mutation and Query
[
https://issues.apache.org/jira/browse/HBASE-18555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiang Li updated HBASE-18555:
-
Status: Patch Available (was: Open)
> Remove redundant familyMap.put() from addxxx() of sub-classes of Mutation and
> Query
> ---
>
> Key: HBASE-18555
> URL: https://issues.apache.org/jira/browse/HBASE-18555
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Attachments: HBASE-18555.master.000.patch,
> HBASE-18555.master.001.patch, HBASE-18555.master.002.patch
>
>
> In addxxx() functions of Mutation(Append, Delete, Increment and Put) and
> Query(Get and Scan), there are redundant Map#put() calls which could be
> removed to improve the performance.
> For example, in Put#addColumn() and addImmutable(), after getting the cell
> list of the given family and add the cell into the list, the code puts
> (key=family, value=list) into familyMap.
> In addColumn(), it is like
> {code}
> List list = getCellList(family);
> KeyValue kv = createPutKeyValue(family, qualifier, ts, value);
> list.add(kv);
> familyMap.put(CellUtil.cloneFamily(kv), list); // <-- here
> return this;
> {code}
> In addImmutable(), it is like
> {code}
> List list = getCellList(family);
> KeyValue kv = createPutKeyValue(family, qualifier, ts, value, tag);
> list.add(kv);
> familyMap.put(family, list); // <-- here
> return this;
> {code}
> I think those put() for Map only take effect when getCellList(family) returns
> a new allocated ArrayList. When the list for a family already exist, put()
> for Map will update the value to the reference of the list, but actually, the
> reference of the list is not changed.
> Those put() do not do any harm in terms of the correctness when they are
> here. But it could be removed to improve the performance. familyMap searches
> for key and set the new value and return the old value. Those operation take
> some time but actually are not needed.
> The put() could be moved into Mutation#getCellList(family)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18555) Remove redundant familyMap.put() from addxxx() of sub-classes of Mutation and Query
[
https://issues.apache.org/jira/browse/HBASE-18555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiang Li updated HBASE-18555:
-
Attachment: HBASE-18555.master.002.patch
Thanks [~chia7712] and [~jerryhe] for the comments. I updated a test case
(TestScannerHeartbeatMessages) in hbase-server and upload patch 002. Hadoop QA
is triggered.
> Remove redundant familyMap.put() from addxxx() of sub-classes of Mutation and
> Query
> ---
>
> Key: HBASE-18555
> URL: https://issues.apache.org/jira/browse/HBASE-18555
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Attachments: HBASE-18555.master.000.patch,
> HBASE-18555.master.001.patch, HBASE-18555.master.002.patch
>
>
> In addxxx() functions of Mutation(Append, Delete, Increment and Put) and
> Query(Get and Scan), there are redundant Map#put() calls which could be
> removed to improve the performance.
> For example, in Put#addColumn() and addImmutable(), after getting the cell
> list of the given family and add the cell into the list, the code puts
> (key=family, value=list) into familyMap.
> In addColumn(), it is like
> {code}
> List list = getCellList(family);
> KeyValue kv = createPutKeyValue(family, qualifier, ts, value);
> list.add(kv);
> familyMap.put(CellUtil.cloneFamily(kv), list); // <-- here
> return this;
> {code}
> In addImmutable(), it is like
> {code}
> List list = getCellList(family);
> KeyValue kv = createPutKeyValue(family, qualifier, ts, value, tag);
> list.add(kv);
> familyMap.put(family, list); // <-- here
> return this;
> {code}
> I think those put() for Map only take effect when getCellList(family) returns
> a new allocated ArrayList. When the list for a family already exist, put()
> for Map will update the value to the reference of the list, but actually, the
> reference of the list is not changed.
> Those put() do not do any harm in terms of the correctness when they are
> here. But it could be removed to improve the performance. familyMap searches
> for key and set the new value and return the old value. Those operation take
> some time but actually are not needed.
> The put() could be moved into Mutation#getCellList(family)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-15511) ClusterStatus should be able to return responses by scope
[ https://issues.apache.org/jira/browse/HBASE-15511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-15511: -- Attachment: HBASE-15511.master.014.patch Sorry for not updating the last few patches on reiview board, and not again. Fix as suggestions > ClusterStatus should be able to return responses by scope > - > > Key: HBASE-15511 > URL: https://issues.apache.org/jira/browse/HBASE-15511 > Project: HBase > Issue Type: Improvement >Reporter: Esteban Gutierrez >Assignee: Reid Chan > Fix For: 2.0.0 > > Attachments: HBASE-15511.master.001.patch, > HBASE-15511.master.002.patch, HBASE-15511.master.003.patch, > HBASE-15511.master.004.patch, HBASE-15511.master.005.patch, > HBASE-15511.master.006.patch, HBASE-15511.master.007.patch, > HBASE-15511.master.008.patch, HBASE-15511.master.009.patch, > HBASE-15511.master.010.patch, HBASE-15511.master.011.patch, > HBASE-15511.master.012.patch, HBASE-15511.master.013.patch, > HBASE-15511.master.014.patch > > > The current ClusterStatus response returns too much information about the > load per region and replication cluster wide. Sometimes that response can be > quite large (10s or 100s of MBs) and methods like getServerSize() or > getRegionsCount() don't really need the full response. One possibility is to > provide a scope (or filter) for the ClusterStatus requests to limit the > response back to the client. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-15511) ClusterStatus should be able to return responses by scope
[ https://issues.apache.org/jira/browse/HBASE-15511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-15511: -- Status: Patch Available (was: Open) > ClusterStatus should be able to return responses by scope > - > > Key: HBASE-15511 > URL: https://issues.apache.org/jira/browse/HBASE-15511 > Project: HBase > Issue Type: Improvement >Reporter: Esteban Gutierrez >Assignee: Reid Chan > Fix For: 2.0.0 > > Attachments: HBASE-15511.master.001.patch, > HBASE-15511.master.002.patch, HBASE-15511.master.003.patch, > HBASE-15511.master.004.patch, HBASE-15511.master.005.patch, > HBASE-15511.master.006.patch, HBASE-15511.master.007.patch, > HBASE-15511.master.008.patch, HBASE-15511.master.009.patch, > HBASE-15511.master.010.patch, HBASE-15511.master.011.patch, > HBASE-15511.master.012.patch, HBASE-15511.master.013.patch, > HBASE-15511.master.014.patch > > > The current ClusterStatus response returns too much information about the > load per region and replication cluster wide. Sometimes that response can be > quite large (10s or 100s of MBs) and methods like getServerSize() or > getRegionsCount() don't really need the full response. One possibility is to > provide a scope (or filter) for the ClusterStatus requests to limit the > response back to the client. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-15511) ClusterStatus should be able to return responses by scope
[ https://issues.apache.org/jira/browse/HBASE-15511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-15511: -- Status: Open (was: Patch Available) > ClusterStatus should be able to return responses by scope > - > > Key: HBASE-15511 > URL: https://issues.apache.org/jira/browse/HBASE-15511 > Project: HBase > Issue Type: Improvement >Reporter: Esteban Gutierrez >Assignee: Reid Chan > Fix For: 2.0.0 > > Attachments: HBASE-15511.master.001.patch, > HBASE-15511.master.002.patch, HBASE-15511.master.003.patch, > HBASE-15511.master.004.patch, HBASE-15511.master.005.patch, > HBASE-15511.master.006.patch, HBASE-15511.master.007.patch, > HBASE-15511.master.008.patch, HBASE-15511.master.009.patch, > HBASE-15511.master.010.patch, HBASE-15511.master.011.patch, > HBASE-15511.master.012.patch, HBASE-15511.master.013.patch > > > The current ClusterStatus response returns too much information about the > load per region and replication cluster wide. Sometimes that response can be > quite large (10s or 100s of MBs) and methods like getServerSize() or > getRegionsCount() don't really need the full response. One possibility is to > provide a scope (or filter) for the ClusterStatus requests to limit the > response back to the client. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18555) Remove redundant familyMap.put() from addxxx() of sub-classes of Mutation and Query
[
https://issues.apache.org/jira/browse/HBASE-18555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiang Li updated HBASE-18555:
-
Status: Open (was: Patch Available)
> Remove redundant familyMap.put() from addxxx() of sub-classes of Mutation and
> Query
> ---
>
> Key: HBASE-18555
> URL: https://issues.apache.org/jira/browse/HBASE-18555
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Attachments: HBASE-18555.master.000.patch,
> HBASE-18555.master.001.patch
>
>
> In addxxx() functions of Mutation(Append, Delete, Increment and Put) and
> Query(Get and Scan), there are redundant Map#put() calls which could be
> removed to improve the performance.
> For example, in Put#addColumn() and addImmutable(), after getting the cell
> list of the given family and add the cell into the list, the code puts
> (key=family, value=list) into familyMap.
> In addColumn(), it is like
> {code}
> List list = getCellList(family);
> KeyValue kv = createPutKeyValue(family, qualifier, ts, value);
> list.add(kv);
> familyMap.put(CellUtil.cloneFamily(kv), list); // <-- here
> return this;
> {code}
> In addImmutable(), it is like
> {code}
> List list = getCellList(family);
> KeyValue kv = createPutKeyValue(family, qualifier, ts, value, tag);
> list.add(kv);
> familyMap.put(family, list); // <-- here
> return this;
> {code}
> I think those put() for Map only take effect when getCellList(family) returns
> a new allocated ArrayList. When the list for a family already exist, put()
> for Map will update the value to the reference of the list, but actually, the
> reference of the list is not changed.
> Those put() do not do any harm in terms of the correctness when they are
> here. But it could be removed to improve the performance. familyMap searches
> for key and set the new value and return the old value. Those operation take
> some time but actually are not needed.
> The put() could be moved into Mutation#getCellList(family)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18551) [AMv2] UnassignProcedure and crashed regionservers
[ https://issues.apache.org/jira/browse/HBASE-18551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124374#comment-16124374 ] Hudson commented on HBASE-18551: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #3515 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3515/]) HBASE-18551 [AMv2] UnassignProcedure and crashed regionservers; (stack: rev 1070888fff3a89d435018f11bfb2fd5609be8bab) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestAssignmentManager.java > [AMv2] UnassignProcedure and crashed regionservers > -- > > Key: HBASE-18551 > URL: https://issues.apache.org/jira/browse/HBASE-18551 > Project: HBase > Issue Type: Bug > Components: amv2 >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: HBASE-18551.master.001.patch, > HBASE-18551.master.002.patch, HBASE-18551.master.003.patch > > > This has been [~uagashe] and my obsession over the last few days, what should > an UnassignProcedure do when it dispatches a CLOSE but the CLOSE fails > because of ConnectException or SocketTimeout. > + We used to let UnassignProcedure continue presuming the Region would be > closed since the server is dead. BUT, if the unassign was part of a > MoveProcedure, the unassign would proceed and the Move would then run WITHOUT > first splitting logs. Bad. > + So, we made it so UnassignProcedure failed; let the upper layers take care > of the failure. See HBASE-18491 that enabled this behavior. BUT, we are since > figuring that even if the UP completes as a failure, since it gives up the > Region lock on completion, another procedure -- say an AssignProcedure -- > could cut in before the ServerCrashProcedure had finished and again there > could be dataloss. > + Now we are thinking the UP should hold on to the Region lock until we are > signalled by a ServerCrashProcedure; only then let go of the region. The UP > has context that is hard to pass another. Waiting on a SCP has the UP living > on for what could be a good amount of time. It might be ok if we can suspend > the procedure. > There is a good sample scenario that came up doing the no-regions-on-master > issue, HBASE-18511. When meta is not on master, TestSplitTransactionOnCluster > is failing. It fails because though the test completes, the tests commonly > kill a RegionServer. The teardown for the test runs before we've noticed the > aborted RS. So, the disable of the table in the teardown prepartory to our > deleting the test table as part of clean up, goes to unassign regions but the > unassign fails against the aborted server. > Good stuff. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18025) CatalogJanitor should collect outdated RegionStates from the AM
[ https://issues.apache.org/jira/browse/HBASE-18025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124373#comment-16124373 ] Hudson commented on HBASE-18025: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #3515 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3515/]) HBASE-18025 CatalogJanitor should collect outdated RegionStates from the (esteban: rev 71a9a9a9440c9f2e2e9dd301dd372197e38e70c5) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitorInMemoryStates.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java > CatalogJanitor should collect outdated RegionStates from the AM > --- > > Key: HBASE-18025 > URL: https://issues.apache.org/jira/browse/HBASE-18025 > Project: HBase > Issue Type: Bug >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez > Fix For: 2.0.0, 3.0.0, 1.4.0, 1.5.0 > > Attachments: HBASE-18025.001.patch, HBASE-18025.002.patch, > HBASE-18025.003.patch, HBASE-18025.004.patch, HBASE-18025.005.patch, > HBASE-18025-branch-1.005.patch, HBASE-18025-branch-1.006.patch > > > I don't think this will matter on the long run for HBase 2, but at least in > branch-1 and the current master we keep in multiple places copies of the > region states in the master and this copies include information like the HRI. > A problem that we have observed is when region replicas are being used and > there is a split, the region replica from parent doesn't get collected from > the region states and when the balancer tries to assign the old parent region > replica, this will cause the RegionServer to create a new HRI with the > details of the parent causing an inconstancy (see HBASE-18024). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-15511) ClusterStatus should be able to return responses by scope
[ https://issues.apache.org/jira/browse/HBASE-15511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124351#comment-16124351 ] Reid Chan commented on HBASE-15511: --- [~ajayjadhav] Thanks for the review, i'm fixing it. > ClusterStatus should be able to return responses by scope > - > > Key: HBASE-15511 > URL: https://issues.apache.org/jira/browse/HBASE-15511 > Project: HBase > Issue Type: Improvement >Reporter: Esteban Gutierrez >Assignee: Reid Chan > Fix For: 2.0.0 > > Attachments: HBASE-15511.master.001.patch, > HBASE-15511.master.002.patch, HBASE-15511.master.003.patch, > HBASE-15511.master.004.patch, HBASE-15511.master.005.patch, > HBASE-15511.master.006.patch, HBASE-15511.master.007.patch, > HBASE-15511.master.008.patch, HBASE-15511.master.009.patch, > HBASE-15511.master.010.patch, HBASE-15511.master.011.patch, > HBASE-15511.master.012.patch, HBASE-15511.master.013.patch > > > The current ClusterStatus response returns too much information about the > load per region and replication cluster wide. Sometimes that response can be > quite large (10s or 100s of MBs) and methods like getServerSize() or > getRegionsCount() don't really need the full response. One possibility is to > provide a scope (or filter) for the ClusterStatus requests to limit the > response back to the client. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18507) [C++] Support for MultiPuts in AsyncBatchRpcRetryingCaller class
[
https://issues.apache.org/jira/browse/HBASE-18507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124337#comment-16124337
]
Enis Soztutar commented on HBASE-18507:
---
Thanks for the updating the work with templates.
- We have updated the base folly version, these will not compile anymore:
{code}
-#include
#include
#include
+#include
{code}
also
{code}
-#include
-#include
+#include
{code}
(you can update the folly version using the same exact version, and commands
from {{docker-files/Dockerfile}}, see section for folly and wangle.
- The patch undo's the changes already committed from HBASE-18564 (especially
client-test.cc). Can you please only do changes related to the new patch.
- remove:
{code}
+ LOG(INFO) << "Does this work";
{code}
{code}
+ for (const auto& row : rows) {
+VLOG(4) << "row is " << row->row();
+ }
{code}
- Nice tests, we can add more like the ones in multi-retry-test.
- Are you gonna address this?
{code}
+ // TODO pass the below as parameter
+ // connection_conf_->read_rpc_timeout();
+ // connection_conf_->write_rpc_timeout();
{code}
- Let's throw an exception here:
{code}
+ //TODO runtime_error reqd ?
{code}
also this should not be caught, or should be rethrown:
{code}
+ } catch (const std::bad_typeid ) {
+std::cout << " caught " << e.what() << '\n';
+ }
{code}
- This does not look right:
{code}
if (strstr(typeid(*pget).name(), "Get") != nullptr)
{code}
why are you doing string comparison. It is very inefficient to do this. We
should just compare the typeids, or templatize this code path as well.
- Remove if not needed:
{code}
+// template class RequestConverter;
{code}
{code}
+// template class AsyncBatchRpcRetryingCaller;
{code}
- We have talked (offline) about Puts returning {{folly::Unit}} instead of
empty {{Result}} objects. Are you gonna address that?
{code}
+folly::Future>>
RawAsyncTable::Put(
{code}
- Remaining looks good.
> [C++] Support for MultiPuts in AsyncBatchRpcRetryingCaller class
>
>
> Key: HBASE-18507
> URL: https://issues.apache.org/jira/browse/HBASE-18507
> Project: HBase
> Issue Type: Sub-task
>Reporter: Sudeep Sunthankar
>Assignee: Sudeep Sunthankar
> Attachments: HBASE-18507.HBASE-14850.v1.patch,
> HBASE-18507.HBASE-14850.v2.patch
>
>
> We will be addressing Multi Puts changes to AsyncBatchRpcRetryingCaller class
> here
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15511) ClusterStatus should be able to return responses by scope
[ https://issues.apache.org/jira/browse/HBASE-15511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124332#comment-16124332 ] Ajay Jadhav commented on HBASE-15511: - I have provided some minor comments on the review board. > ClusterStatus should be able to return responses by scope > - > > Key: HBASE-15511 > URL: https://issues.apache.org/jira/browse/HBASE-15511 > Project: HBase > Issue Type: Improvement >Reporter: Esteban Gutierrez >Assignee: Reid Chan > Fix For: 2.0.0 > > Attachments: HBASE-15511.master.001.patch, > HBASE-15511.master.002.patch, HBASE-15511.master.003.patch, > HBASE-15511.master.004.patch, HBASE-15511.master.005.patch, > HBASE-15511.master.006.patch, HBASE-15511.master.007.patch, > HBASE-15511.master.008.patch, HBASE-15511.master.009.patch, > HBASE-15511.master.010.patch, HBASE-15511.master.011.patch, > HBASE-15511.master.012.patch, HBASE-15511.master.013.patch > > > The current ClusterStatus response returns too much information about the > load per region and replication cluster wide. Sometimes that response can be > quite large (10s or 100s of MBs) and methods like getServerSize() or > getRegionsCount() don't really need the full response. One possibility is to > provide a scope (or filter) for the ClusterStatus requests to limit the > response back to the client. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18271) Shade netty
[
https://issues.apache.org/jira/browse/HBASE-18271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124328#comment-16124328
]
Hadoop QA commented on HBASE-18271:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
5s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
52s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
53s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 0m
5s{color} | {color:red} The patch generated 6 new + 492 unchanged - 6 fixed =
498 total (was 498) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
6s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m
24s{color} | {color:red} The patch causes 14 errors with Hadoop v2.6.1. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m
42s{color} | {color:red} The patch causes 14 errors with Hadoop v2.6.2. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m
0s{color} | {color:red} The patch causes 14 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m
17s{color} | {color:red} The patch causes 14 errors with Hadoop v2.6.4. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m
36s{color} | {color:red} The patch causes 14 errors with Hadoop v2.6.5. {color}
|
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
22s{color} | {color:red} hbase-client generated 2 new + 0 unchanged - 0 fixed =
2 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 2m
55s{color} | {color:red} root generated 2 new + 23 unchanged - 0 fixed = 25
total (was 23) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
[jira] [Commented] (HBASE-18511) Default no regions on master
[ https://issues.apache.org/jira/browse/HBASE-18511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124320#comment-16124320 ] stack commented on HBASE-18511: --- .006 should fix most of the test failures. Its not finished yet though. I've put in place a boolean flag for tables on master, yes or no. I need to add tests for case where flag is true (currently it is false). There is then the case where systems tables were exclusively on master. That will be another flag rather than a table list. I need to do that too. Working on it. > Default no regions on master > > > Key: HBASE-18511 > URL: https://issues.apache.org/jira/browse/HBASE-18511 > Project: HBase > Issue Type: Task > Components: master >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-18511.master.001.patch, > HBASE-18511.master.002.patch, HBASE-18511.master.003.patch, > HBASE-18511.master.004.patch, HBASE-18511.master.005.patch, > HBASE-18511.master.006.patch > > > Let this be umbrella issue for no-regions-on-master as default deploy (as it > was in branch-1). > Also need to make sure we can run WITH regions on master; in particular > system tables with RPC short-circuit as it is now in hbase master. > Background is that master branch carried a change that allowed Master carry > regions. On top of this improvement on branch-1, Master defaulted to carry > system tables only. No release was made with this configuration. Now we are > going to cut the 2.0.0 release, the decision is that hbase-2 should have the > same layout as hbase-1 so this issue implements the undoing of Master > carrying system tables by default (though the capability remains). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18511) Default no regions on master
[ https://issues.apache.org/jira/browse/HBASE-18511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-18511: -- Attachment: HBASE-18511.master.006.patch > Default no regions on master > > > Key: HBASE-18511 > URL: https://issues.apache.org/jira/browse/HBASE-18511 > Project: HBase > Issue Type: Task > Components: master >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-18511.master.001.patch, > HBASE-18511.master.002.patch, HBASE-18511.master.003.patch, > HBASE-18511.master.004.patch, HBASE-18511.master.005.patch, > HBASE-18511.master.006.patch > > > Let this be umbrella issue for no-regions-on-master as default deploy (as it > was in branch-1). > Also need to make sure we can run WITH regions on master; in particular > system tables with RPC short-circuit as it is now in hbase master. > Background is that master branch carried a change that allowed Master carry > regions. On top of this improvement on branch-1, Master defaulted to carry > system tables only. No release was made with this configuration. Now we are > going to cut the 2.0.0 release, the decision is that hbase-2 should have the > same layout as hbase-1 so this issue implements the undoing of Master > carrying system tables by default (though the capability remains). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18579) Enable core dump by default for docker
[ https://issues.apache.org/jira/browse/HBASE-18579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-18579: --- Attachment: 18579.v1.txt > Enable core dump by default for docker > -- > > Key: HBASE-18579 > URL: https://issues.apache.org/jira/browse/HBASE-18579 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Priority: Minor > Attachments: 18579.v1.txt > > > In recent debugging experience, I found that by default the ulimit value > prohibits the generation of core dump. This makes debugging difficult. > We should enable core dump generation by default. > https://www.akadia.com/services/ora_enable_core.html -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HBASE-18579) Enable core dump by default for docker
[ https://issues.apache.org/jira/browse/HBASE-18579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reassigned HBASE-18579: -- Assignee: Ted Yu > Enable core dump by default for docker > -- > > Key: HBASE-18579 > URL: https://issues.apache.org/jira/browse/HBASE-18579 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Attachments: 18579.v1.txt > > > In recent debugging experience, I found that by default the ulimit value > prohibits the generation of core dump. This makes debugging difficult. > We should enable core dump generation by default. > https://www.akadia.com/services/ora_enable_core.html -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18526) FIFOCompactionPolicy pre-check uses wrong scope
[
https://issues.apache.org/jira/browse/HBASE-18526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124284#comment-16124284
]
Ted Yu commented on HBASE-18526:
Vlad:
Can you attach patch for branch-1 ?
> FIFOCompactionPolicy pre-check uses wrong scope
> ---
>
> Key: HBASE-18526
> URL: https://issues.apache.org/jira/browse/HBASE-18526
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 1.3.1
>Reporter: Lars George
>Assignee: Vladimir Rodionov
> Attachments: HBASE-18526-v1.patch
>
>
> See https://issues.apache.org/jira/browse/HBASE-14468
> It adds this check to {{HMaster.checkCompactionPolicy()}}:
> {code}
> // 1. Check TTL
> if (hcd.getTimeToLive() == HColumnDescriptor.DEFAULT_TTL) {
> message = "Default TTL is not supported for FIFO compaction";
> throw new IOException(message);
> }
> // 2. Check min versions
> if (hcd.getMinVersions() > 0) {
> message = "MIN_VERSION > 0 is not supported for FIFO compaction";
> throw new IOException(message);
> }
> // 3. blocking file count
> String sbfc = htd.getConfigurationValue(HStore.BLOCKING_STOREFILES_KEY);
> if (sbfc != null) {
> blockingFileCount = Integer.parseInt(sbfc);
> }
> if (blockingFileCount < 1000) {
> message =
> "blocking file count '" + HStore.BLOCKING_STOREFILES_KEY + "' "
> + blockingFileCount
> + " is below recommended minimum of 1000";
> throw new IOException(message);
> }
> {code}
> Why does it only check the blocking file count on the HTD level, while
> others are check on the HCD level? Doing this for example fails
> because of it:
> {noformat}
> hbase(main):008:0> create 'ttltable', { NAME => 'cf1', TTL => 300,
> CONFIGURATION => { 'hbase.hstore.defaultengine.compactionpolicy.class'
> => 'org.apache.hadoop.hbase.regionserver.compactions.FIFOCompactionPolicy',
> 'hbase.hstore.blockingStoreFiles' => 2000 } }
> ERROR: org.apache.hadoop.hbase.DoNotRetryIOException: blocking file
> count 'hbase.hstore.blockingStoreFiles' 10 is below recommended
> minimum of 1000 Set hbase.table.sanity.checks to false at conf or
> table descriptor if you want to bypass sanity checks
> at
> org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:1782)
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1663)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1545)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:469)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58549)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
> Caused by: java.io.IOException: blocking file count
> 'hbase.hstore.blockingStoreFiles' 10 is below recommended minimum of
> 1000
> at
> org.apache.hadoop.hbase.master.HMaster.checkCompactionPolicy(HMaster.java:1773)
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1661)
> ... 7 more
> {noformat}
> The check should be performed on the column family level instead.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18526) FIFOCompactionPolicy pre-check uses wrong scope
[
https://issues.apache.org/jira/browse/HBASE-18526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124278#comment-16124278
]
Ted Yu commented on HBASE-18526:
TestZKSecretWatcher failure was not related to the patch.
> FIFOCompactionPolicy pre-check uses wrong scope
> ---
>
> Key: HBASE-18526
> URL: https://issues.apache.org/jira/browse/HBASE-18526
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 1.3.1
>Reporter: Lars George
>Assignee: Vladimir Rodionov
> Attachments: HBASE-18526-v1.patch
>
>
> See https://issues.apache.org/jira/browse/HBASE-14468
> It adds this check to {{HMaster.checkCompactionPolicy()}}:
> {code}
> // 1. Check TTL
> if (hcd.getTimeToLive() == HColumnDescriptor.DEFAULT_TTL) {
> message = "Default TTL is not supported for FIFO compaction";
> throw new IOException(message);
> }
> // 2. Check min versions
> if (hcd.getMinVersions() > 0) {
> message = "MIN_VERSION > 0 is not supported for FIFO compaction";
> throw new IOException(message);
> }
> // 3. blocking file count
> String sbfc = htd.getConfigurationValue(HStore.BLOCKING_STOREFILES_KEY);
> if (sbfc != null) {
> blockingFileCount = Integer.parseInt(sbfc);
> }
> if (blockingFileCount < 1000) {
> message =
> "blocking file count '" + HStore.BLOCKING_STOREFILES_KEY + "' "
> + blockingFileCount
> + " is below recommended minimum of 1000";
> throw new IOException(message);
> }
> {code}
> Why does it only check the blocking file count on the HTD level, while
> others are check on the HCD level? Doing this for example fails
> because of it:
> {noformat}
> hbase(main):008:0> create 'ttltable', { NAME => 'cf1', TTL => 300,
> CONFIGURATION => { 'hbase.hstore.defaultengine.compactionpolicy.class'
> => 'org.apache.hadoop.hbase.regionserver.compactions.FIFOCompactionPolicy',
> 'hbase.hstore.blockingStoreFiles' => 2000 } }
> ERROR: org.apache.hadoop.hbase.DoNotRetryIOException: blocking file
> count 'hbase.hstore.blockingStoreFiles' 10 is below recommended
> minimum of 1000 Set hbase.table.sanity.checks to false at conf or
> table descriptor if you want to bypass sanity checks
> at
> org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:1782)
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1663)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1545)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:469)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58549)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
> Caused by: java.io.IOException: blocking file count
> 'hbase.hstore.blockingStoreFiles' 10 is below recommended minimum of
> 1000
> at
> org.apache.hadoop.hbase.master.HMaster.checkCompactionPolicy(HMaster.java:1773)
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1661)
> ... 7 more
> {noformat}
> The check should be performed on the column family level instead.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18576) [C++] Add ping for RPC test
[ https://issues.apache.org/jira/browse/HBASE-18576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-18576: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HBASE-14850 Status: Resolved (was: Patch Available) I've pushed this. Thanks [~xiaobingo] for the patch. > [C++] Add ping for RPC test > --- > > Key: HBASE-18576 > URL: https://issues.apache.org/jira/browse/HBASE-18576 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Fix For: HBASE-14850 > > Attachments: HBASE-18576.000.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18526) FIFOCompactionPolicy pre-check uses wrong scope
[
https://issues.apache.org/jira/browse/HBASE-18526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124266#comment-16124266
]
Hadoop QA commented on HBASE-18526:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
38m 49s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}112m 51s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}170m 3s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.security.token.TestZKSecretWatcher |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:bdc94b1 |
| JIRA Issue | HBASE-18526 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12880682/HBASE-18526-v1.patch |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 14ce30d5dfb3 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 71a9a9a |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC3 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/8048/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/8048/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/8048/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was automatically generated.
> FIFOCompactionPolicy pre-check uses wrong scope
> ---
>
> Key: HBASE-18526
>
[jira] [Commented] (HBASE-14135) HBase Backup/Restore Phase 3: Merge backup images
[
https://issues.apache.org/jira/browse/HBASE-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124262#comment-16124262
]
Vladimir Rodionov commented on HBASE-14135:
---
[~elserj], we had a clean QA run.
> HBase Backup/Restore Phase 3: Merge backup images
> -
>
> Key: HBASE-14135
> URL: https://issues.apache.org/jira/browse/HBASE-14135
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Critical
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-14135-v10.patch, HBASE-14135-v11.patch,
> HBASE-14135-v3.patch, HBASE-14135-v5.patch, HBASE-14135-v6.patch,
> HBASE-14135-v7.patch, HBASE-14135-v8.patch, HBASE-14135-v9.patch
>
>
> User can merge incremental backup images into single incremental backup image.
> # Merge supports only incremental images
> # Merge supports only images for the same backup destinations
> Command:
> {code}
> hbase backup merge image1,image2,..imageK
> {code}
> Example:
> {code}
> hbase backup merge backup_143126764557,backup_143126764456
> {code}
> When operation is complete, only the most recent backup image will be kept
> (in above example - backup_143126764557) as a merged backup image, all other
> images will be deleted from both: file system and backup system tables,
> corresponding backup manifest for the merged backup image will be updated to
> remove dependencies from deleted images. Merged backup image will contains
> all the data from original image and from deleted images.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18581) Remove dead code and some tidy up in BaseLoadBalancer
[ https://issues.apache.org/jira/browse/HBASE-18581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Umesh Agashe updated HBASE-18581: - Status: Patch Available (was: In Progress) > Remove dead code and some tidy up in BaseLoadBalancer > - > > Key: HBASE-18581 > URL: https://issues.apache.org/jira/browse/HBASE-18581 > Project: HBase > Issue Type: Improvement > Components: Balancer >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Minor > Attachments: hbase-18581.master.001.patch > > > * calls to methods getLowestLocalityRegionServer() & > getLeastLoadedTopServerForRegion() got removed in HBASE-18164 > * call to calculateRegionServerLocalities() got removed in HBASE-15486 > * Some other minor improvements -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18581) Remove dead code and some tidy up in BaseLoadBalancer
[ https://issues.apache.org/jira/browse/HBASE-18581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Umesh Agashe updated HBASE-18581: - Attachment: hbase-18581.master.001.patch > Remove dead code and some tidy up in BaseLoadBalancer > - > > Key: HBASE-18581 > URL: https://issues.apache.org/jira/browse/HBASE-18581 > Project: HBase > Issue Type: Improvement > Components: Balancer >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Minor > Attachments: hbase-18581.master.001.patch > > > * calls to methods getLowestLocalityRegionServer() & > getLeastLoadedTopServerForRegion() got removed in HBASE-18164 > * call to calculateRegionServerLocalities() got removed in HBASE-15486 > * Some other minor improvements -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-14135) HBase Backup/Restore Phase 3: Merge backup images
[
https://issues.apache.org/jira/browse/HBASE-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124250#comment-16124250
]
Hadoop QA commented on HBASE-14135:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 51s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}120m
55s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}169m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:bdc94b1 |
| JIRA Issue | HBASE-14135 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881536/HBASE-14135-v11.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux e2ed64238b29 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 71a9a9a |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/8047/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/8047/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was automatically generated.
> HBase Backup/Restore Phase 3: Merge backup images
> -
>
> Key: HBASE-14135
> URL: https://issues.apache.org/jira/browse/HBASE-14135
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Critical
> Labels: backup
> Fix For: 2.0.0
>
> Attachments:
[jira] [Work started] (HBASE-18581) Remove dead code and some tidy up in BaseLoadBalancer
[ https://issues.apache.org/jira/browse/HBASE-18581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-18581 started by Umesh Agashe. > Remove dead code and some tidy up in BaseLoadBalancer > - > > Key: HBASE-18581 > URL: https://issues.apache.org/jira/browse/HBASE-18581 > Project: HBase > Issue Type: Improvement > Components: Balancer >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Minor > > * calls to methods getLowestLocalityRegionServer() & > getLeastLoadedTopServerForRegion() got removed in HBASE-18164 > * call to calculateRegionServerLocalities() got removed in HBASE-15486 > * Some other minor improvements -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18581) Remove dead code and some tidy up in BaseLoadBalancer
Umesh Agashe created HBASE-18581: Summary: Remove dead code and some tidy up in BaseLoadBalancer Key: HBASE-18581 URL: https://issues.apache.org/jira/browse/HBASE-18581 Project: HBase Issue Type: Improvement Components: Balancer Reporter: Umesh Agashe Assignee: Umesh Agashe Priority: Minor * calls to methods getLowestLocalityRegionServer() & getLeastLoadedTopServerForRegion() got removed in HBASE-18164 * call to calculateRegionServerLocalities() got removed in HBASE-15486 * Some other minor improvements -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18578) [C++] Add pause for RPC test
[
https://issues.apache.org/jira/browse/HBASE-18578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124247#comment-16124247
]
Xiaobing Zhou commented on HBASE-18578:
---
It runs into broken promise exception, e.g.
{noformat}
I0811 23:05:53.956799 402 rpc-client.cc:86] RpcClient Exception:
N5folly13BrokenPromiseE: Broken promise for type name
`St10unique_ptrIN5hbase8ResponseESt14default_deleteIS1_EE`
I0811 23:05:53.956811 402 rpc-test.cc:240] RPC pause returned:
N5folly13BrokenPromiseE: Broken promise for type name
`St10unique_ptrIN5hbase8ResponseESt14default_deleteIS1_EE`.
{noformat}
Steps to reproduce it:
# buck test --no-cache --no-results-cache //connection:rpc-test
# GLOG_v=3 ./buck-out/gen/connection/rpc-test
> [C++] Add pause for RPC test
>
>
> Key: HBASE-18578
> URL: https://issues.apache.org/jira/browse/HBASE-18578
> Project: HBase
> Issue Type: Sub-task
>Reporter: Xiaobing Zhou
> Attachments: HBASE-18578.000.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HBASE-18350) RSGroups are broken under AMv2
[ https://issues.apache.org/jira/browse/HBASE-18350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang reassigned HBASE-18350: -- Assignee: Thiruvel Thirumoolan (was: Stephen Yuan Jiang) > RSGroups are broken under AMv2 > -- > > Key: HBASE-18350 > URL: https://issues.apache.org/jira/browse/HBASE-18350 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 2.0.0-alpha-1 >Reporter: Stephen Yuan Jiang >Assignee: Thiruvel Thirumoolan >Priority: Blocker > Fix For: 2.0.0-beta-2 > > > The following RSGroups tests were disabled by Core Proc-V2 AM in HBASE-14614: > - Disabled/Ignore TestRSGroupsOfflineMode#testOffline; need to dig in on what > offline is. > - Disabled/Ignore TestRSGroups. > This JIRA tracks the work to enable them (or remove/modify if not applicable). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18578) [C++] Add pause for RPC test
[ https://issues.apache.org/jira/browse/HBASE-18578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124246#comment-16124246 ] Xiaobing Zhou commented on HBASE-18578: --- posted v0 patch. > [C++] Add pause for RPC test > > > Key: HBASE-18578 > URL: https://issues.apache.org/jira/browse/HBASE-18578 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou > Attachments: HBASE-18578.000.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18578) [C++] Add pause for RPC test
[ https://issues.apache.org/jira/browse/HBASE-18578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HBASE-18578: -- Attachment: HBASE-18578.000.patch > [C++] Add pause for RPC test > > > Key: HBASE-18578 > URL: https://issues.apache.org/jira/browse/HBASE-18578 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou > Attachments: HBASE-18578.000.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18025) CatalogJanitor should collect outdated RegionStates from the AM
[ https://issues.apache.org/jira/browse/HBASE-18025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124237#comment-16124237 ] Hudson commented on HBASE-18025: FAILURE: Integrated in Jenkins build HBase-1.4 #850 (See [https://builds.apache.org/job/HBase-1.4/850/]) HBASE-18025 CatalogJanitor should collect outdated RegionStates from the (esteban: rev 578e29f96b37875cd8092f7c6b3baf2b511148d0) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitorInMemoryStates.java > CatalogJanitor should collect outdated RegionStates from the AM > --- > > Key: HBASE-18025 > URL: https://issues.apache.org/jira/browse/HBASE-18025 > Project: HBase > Issue Type: Bug >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez > Fix For: 2.0.0, 3.0.0, 1.4.0, 1.5.0 > > Attachments: HBASE-18025.001.patch, HBASE-18025.002.patch, > HBASE-18025.003.patch, HBASE-18025.004.patch, HBASE-18025.005.patch, > HBASE-18025-branch-1.005.patch, HBASE-18025-branch-1.006.patch > > > I don't think this will matter on the long run for HBase 2, but at least in > branch-1 and the current master we keep in multiple places copies of the > region states in the master and this copies include information like the HRI. > A problem that we have observed is when region replicas are being used and > there is a split, the region replica from parent doesn't get collected from > the region states and when the balancer tries to assign the old parent region > replica, this will cause the RegionServer to create a new HRI with the > details of the parent causing an inconstancy (see HBASE-18024). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18580) Allow gdb to attach to selected process in docker
Ted Yu created HBASE-18580: -- Summary: Allow gdb to attach to selected process in docker Key: HBASE-18580 URL: https://issues.apache.org/jira/browse/HBASE-18580 Project: HBase Issue Type: Sub-task Reporter: Ted Yu In the current docker image, if you want to attach gdb to a process, you would see: bq. ptrace: Operation not permitted We should provide better support for gdb in docker. This article is a start: https://thirld.com/blog/2016/08/15/war-stories-debugging-julia-gdb-docker/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18555) Remove redundant familyMap.put() from addxxx() of sub-classes of Mutation and Query
[
https://issues.apache.org/jira/browse/HBASE-18555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124215#comment-16124215
]
Jerry He commented on HBASE-18555:
--
Yup. Modify trivial test case value will do then.
> Remove redundant familyMap.put() from addxxx() of sub-classes of Mutation and
> Query
> ---
>
> Key: HBASE-18555
> URL: https://issues.apache.org/jira/browse/HBASE-18555
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Attachments: HBASE-18555.master.000.patch,
> HBASE-18555.master.001.patch
>
>
> In addxxx() functions of Mutation(Append, Delete, Increment and Put) and
> Query(Get and Scan), there are redundant Map#put() calls which could be
> removed to improve the performance.
> For example, in Put#addColumn() and addImmutable(), after getting the cell
> list of the given family and add the cell into the list, the code puts
> (key=family, value=list) into familyMap.
> In addColumn(), it is like
> {code}
> List list = getCellList(family);
> KeyValue kv = createPutKeyValue(family, qualifier, ts, value);
> list.add(kv);
> familyMap.put(CellUtil.cloneFamily(kv), list); // <-- here
> return this;
> {code}
> In addImmutable(), it is like
> {code}
> List list = getCellList(family);
> KeyValue kv = createPutKeyValue(family, qualifier, ts, value, tag);
> list.add(kv);
> familyMap.put(family, list); // <-- here
> return this;
> {code}
> I think those put() for Map only take effect when getCellList(family) returns
> a new allocated ArrayList. When the list for a family already exist, put()
> for Map will update the value to the reference of the list, but actually, the
> reference of the list is not changed.
> Those put() do not do any harm in terms of the correctness when they are
> here. But it could be removed to improve the performance. familyMap searches
> for key and set the new value and return the old value. Those operation take
> some time but actually are not needed.
> The put() could be moved into Mutation#getCellList(family)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18507) [C++] Support for MultiPuts in AsyncBatchRpcRetryingCaller class
[ https://issues.apache.org/jira/browse/HBASE-18507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudeep Sunthankar updated HBASE-18507: -- Attachment: HBASE-18507.HBASE-14850.v2.patch - Addressed feedback from the last patch. - Templatized request in AsyncBatchRpc. As of now we are persisting with hbase::Row, which can be made more cleaner by passing the actual derived classes like hbase::Get and hbase::Put. Thanks. > [C++] Support for MultiPuts in AsyncBatchRpcRetryingCaller class > > > Key: HBASE-18507 > URL: https://issues.apache.org/jira/browse/HBASE-18507 > Project: HBase > Issue Type: Sub-task >Reporter: Sudeep Sunthankar >Assignee: Sudeep Sunthankar > Attachments: HBASE-18507.HBASE-14850.v1.patch, > HBASE-18507.HBASE-14850.v2.patch > > > We will be addressing Multi Puts changes to AsyncBatchRpcRetryingCaller class > here -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18579) Enable core dump by default for docker
Ted Yu created HBASE-18579: -- Summary: Enable core dump by default for docker Key: HBASE-18579 URL: https://issues.apache.org/jira/browse/HBASE-18579 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Priority: Minor In recent debugging experience, I found that by default the ulimit value prohibits the generation of core dump. This makes debugging difficult. We should enable core dump generation by default. https://www.akadia.com/services/ora_enable_core.html -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18576) [C++] Add ping for RPC test
[
https://issues.apache.org/jira/browse/HBASE-18576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124190#comment-16124190
]
Hadoop QA commented on HBASE-18576:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HBASE-18576 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.4.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-18576 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881553/HBASE-18576.000.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/8049/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was automatically generated.
> [C++] Add ping for RPC test
> ---
>
> Key: HBASE-18576
> URL: https://issues.apache.org/jira/browse/HBASE-18576
> Project: HBase
> Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HBASE-18576.000.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18154) Verify Master carrying system-tables only with short-circuit RPC
[ https://issues.apache.org/jira/browse/HBASE-18154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124189#comment-16124189 ] stack commented on HBASE-18154: --- Tests in TestMasterNoCluster have been disabled because there is as yet no specification for what master-carrying-tables looks like; i.e. how to specify master carries system tables. TODO. FIX. > Verify Master carrying system-tables only with short-circuit RPC > > > Key: HBASE-18154 > URL: https://issues.apache.org/jira/browse/HBASE-18154 > Project: HBase > Issue Type: Sub-task > Components: Region Assignment >Reporter: stack >Priority: Blocker > Fix For: 2.0.0 > > > This sort-of-works in new AMv2 given this the current default. Spend some > time on it. Ensure that when Master goes down, on relaunch, that it gets the > system tables back. Ensure we are doing the short-circuit RPC. > RegionServer Groups might be the right conduit for this sort of deploy type. > Review. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18577) shaded client includes several non-relocated third party dependencies
[ https://issues.apache.org/jira/browse/HBASE-18577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-18577: Attachment: HBASE-18577.WIP.-1.patch attaching a WIP patch that outputs things that look off. started working through the list to see if we can avoid entries or relocate them. > shaded client includes several non-relocated third party dependencies > - > > Key: HBASE-18577 > URL: https://issues.apache.org/jira/browse/HBASE-18577 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Attachments: HBASE-18577.WIP.-1.patch > > > we have some unexpected unrelocated third party dependencies in our shaded > artifacts. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Work started] (HBASE-18577) shaded client includes several non-relocated third party dependencies
[ https://issues.apache.org/jira/browse/HBASE-18577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-18577 started by Sean Busbey. --- > shaded client includes several non-relocated third party dependencies > - > > Key: HBASE-18577 > URL: https://issues.apache.org/jira/browse/HBASE-18577 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > > we have some unexpected unrelocated third party dependencies in our shaded > artifacts. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-18424) Fix TestAsyncTableGetMultiThreaded
[ https://issues.apache.org/jira/browse/HBASE-18424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124177#comment-16124177 ] Vladimir Rodionov edited comment on HBASE-18424 at 8/11/17 10:08 PM: - Test works OK now. I just enabled it (very small fix :)) cc: [~syuanjiang], [~Apache9] was (Author: vrodionov): Test works OK now. I just enabled it (very small fix :)) cc: [~syuanjiang] > Fix TestAsyncTableGetMultiThreaded > -- > > Key: HBASE-18424 > URL: https://issues.apache.org/jira/browse/HBASE-18424 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-18424-v1.patch, HBASE-18424-v2.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18424) Fix TestAsyncTableGetMultiThreaded
[ https://issues.apache.org/jira/browse/HBASE-18424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-18424: -- Attachment: HBASE-18424-v2.patch Test works OK now. I just enabled it (very small fix :)) cc: [~syuanjiang] > Fix TestAsyncTableGetMultiThreaded > -- > > Key: HBASE-18424 > URL: https://issues.apache.org/jira/browse/HBASE-18424 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-18424-v1.patch, HBASE-18424-v2.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18577) shaded client includes several non-relocated third party dependencies
[ https://issues.apache.org/jira/browse/HBASE-18577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124173#comment-16124173 ] Sean Busbey commented on HBASE-18577: - See the discussion in HBASE-13517 about why we don't expect to relocated Hadoop classes. Everything else either needs to be relocated or excluded (possibly while staying as a visible dependency) > shaded client includes several non-relocated third party dependencies > - > > Key: HBASE-18577 > URL: https://issues.apache.org/jira/browse/HBASE-18577 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > > we have some unexpected unrelocated third party dependencies in our shaded > artifacts. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18576) [C++] Add ping for RPC test
[ https://issues.apache.org/jira/browse/HBASE-18576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HBASE-18576: -- Attachment: HBASE-18576.000.patch > [C++] Add ping for RPC test > --- > > Key: HBASE-18576 > URL: https://issues.apache.org/jira/browse/HBASE-18576 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HBASE-18576.000.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18577) shaded client includes several non-relocated third party dependencies
Sean Busbey created HBASE-18577: --- Summary: shaded client includes several non-relocated third party dependencies Key: HBASE-18577 URL: https://issues.apache.org/jira/browse/HBASE-18577 Project: HBase Issue Type: Bug Components: Client Reporter: Sean Busbey Assignee: Sean Busbey Priority: Critical we have some unexpected unrelocated third party dependencies in our shaded artifacts. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18578) [C++] Add pause for RPC test
Xiaobing Zhou created HBASE-18578: - Summary: [C++] Add pause for RPC test Key: HBASE-18578 URL: https://issues.apache.org/jira/browse/HBASE-18578 Project: HBase Issue Type: Sub-task Reporter: Xiaobing Zhou -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18576) [C++] Add ping for RPC test
[ https://issues.apache.org/jira/browse/HBASE-18576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124165#comment-16124165 ] Xiaobing Zhou commented on HBASE-18576: --- posted v0 for reviews. > [C++] Add ping for RPC test > --- > > Key: HBASE-18576 > URL: https://issues.apache.org/jira/browse/HBASE-18576 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HBASE-18576.000.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18576) [C++] Add ping for RPC test
[ https://issues.apache.org/jira/browse/HBASE-18576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HBASE-18576: -- Status: Patch Available (was: Open) > [C++] Add ping for RPC test > --- > > Key: HBASE-18576 > URL: https://issues.apache.org/jira/browse/HBASE-18576 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HBASE-18576.000.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18576) [C++] Add ping for RPC test
Xiaobing Zhou created HBASE-18576: - Summary: [C++] Add ping for RPC test Key: HBASE-18576 URL: https://issues.apache.org/jira/browse/HBASE-18576 Project: HBase Issue Type: Sub-task Reporter: Xiaobing Zhou Assignee: Xiaobing Zhou -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18271) Shade netty
[ https://issues.apache.org/jira/browse/HBASE-18271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124151#comment-16124151 ] stack commented on HBASE-18271: --- bq. I don't understand how we derive the property name/value. Understood. Its cryptic stuff. You'd have to dig in the links I added to the release notes. My favorite is the trailing '.'. Silent fail w/o it. But at least we are not the first to shade netty. There is a well-worn path. > Shade netty > --- > > Key: HBASE-18271 > URL: https://issues.apache.org/jira/browse/HBASE-18271 > Project: HBase > Issue Type: Sub-task > Components: rpc >Affects Versions: 2.0.0 >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-18271.master.001.patch, > HBASE-18271.master.002.patch, > HBASE-18271-Shade-netty-Purge-mention-of-netty-all.003.patch > > > Our new prefatory project, hbase-thirdparty, includes a relocated netty > 4.1.12. This issue is about changing references in hbase to make use of this > shaded netty. This way we will take ourselves out of the clashing library > saga and change netty as we see fit. > One kink is the inclusion inside the netty-all jar of an .so. We need to make > sure it gets loaded and that on linux we are doing native epoll; all should > be in place to do this but need to add a system property to bin/hbase for > netty to pick up. TODO as part of this issue. See tail of upgrade guava > sibling issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18575) [AMv2] Enable and fix TestRestartCluster#testRetainAssignmentOnRestart on master
[ https://issues.apache.org/jira/browse/HBASE-18575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124143#comment-16124143 ] stack commented on HBASE-18575: --- Lets see if HBASE-18511 helps here. > [AMv2] Enable and fix TestRestartCluster#testRetainAssignmentOnRestart on > master > > > Key: HBASE-18575 > URL: https://issues.apache.org/jira/browse/HBASE-18575 > Project: HBase > Issue Type: Bug > Components: amv2 >Affects Versions: 2.0.0 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Critical > Fix For: 2.0.0 > > > * Test creates 3 node cluster (master + 2 RS) and creates 3 tables with 1 > region each. > * It then takes a snapshot of region assignments > * Shuts down the cluster and restarts with 4 nodes (2 RS on same ports, 1 RS > on port where master was running previously and master on new port) > * Takes snapshot of region assignments > * Expected behavior is regions will be retained on RS with appropriate ports > Debugging done so far shows that, when cluster is restarted meta is loaded > but meta has old entries (though the ports are same for RSs, start timestamps > are different). AssignmentManager#processofflineServersWithOnlineRegions() > finds that old RS with different timestamps are not online even though > regions assigned to them are online; triggering submitting SCP for each those > RSs. Subprocedure AssignProcedure of SCP has forceNewPlan set to true. Which > triggers re-assignment of regions. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-18541) [C++] Segfaults from JNI
[
https://issues.apache.org/jira/browse/HBASE-18541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124037#comment-16124037
]
Ted Yu edited comment on HBASE-18541 at 8/11/17 9:46 PM:
-
Looped the multi-retry-test 10 times which passed.
Previously I encountered deadlock which was resolved earlier today
(HBASE-18565).
was (Author: yuzhih...@gmail.com):
Looped the test 10 times which passed.
Previously I encountered deadlock which was resolved earlier today
(HBASE-18565).
> [C++] Segfaults from JNI
>
>
> Key: HBASE-18541
> URL: https://issues.apache.org/jira/browse/HBASE-18541
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Ted Yu
>
> retry-test and multi-retry-test fails flakily when run with
> {code}
> buck test --all --no-results-cache
> {code}
> or when run in a loop:
> {code}
> for i in `seq 1 10`; do buck test --no-results-cache core:retry-test || break
> 1; done
> {code}
> The problem seems to be within the JNI internals and usually happens at the
> create table method call. I was not able to inspect much, but the comments in
> our mini-cluster indicate that we may need to use global references instead
> of local ones. I suspect the problem happens when there is a GC run for the
> test since the failure happens usually after some time (but almost always in
> create table method).
> [~ted_yu] do you mind taking a look at this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18575) [AMv2] Enable and fix TestRestartCluster#testRetainAssignmentOnRestart on master
[ https://issues.apache.org/jira/browse/HBASE-18575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Umesh Agashe updated HBASE-18575: - Description: * Test creates 3 node cluster (master + 2 RS) and creates 3 tables with 1 region each. * It then takes a snapshot of region assignments * Shuts down the cluster and restarts with 4 nodes (2 RS on same ports, 1 RS on port where master was running previously and master on new port) * Takes snapshot of region assignments * Expected behavior is regions will be retained on RS with appropriate ports Debugging done so far shows that, when cluster is restarted meta is loaded but meta has old entries (though the ports are same for RSs, start timestamps are different). AssignmentManager#processofflineServersWithOnlineRegions() finds that old RS with different timestamps are not online even though regions assigned to them are online; triggering submitting SCP for each those RSs. Subprocedure AssignProcedure of SCP has forceNewPlan set to true. Which triggers re-assignment of regions. was: * Test creates 3 node cluster (master + 2 RS) and creates 3 tables with 1 region each. * It then takes a snapshot of region assignments * Shuts down the cluster and restarts with 4 nodes (2 RS on same ports, 1 RS on port where master was running previously and master on new port) * Takes snapshot of region assignments * Expected behavior is regions will be retained on RS with appropriate ports Debugging done so far shows that, when cluster is restarted meta is loaded but meta has old entries (though the ports are same for RSs, start timestamps are different). AssignmentManager#processofflineServersWithOnlineRegions() finds that old RS with different timestamps are not online even though regions assigned to them are online. This trigger submitting SCP for each those RSs. Subprocedure AssignProcedure of SCP has forceNewPlan set to true. Which triggers re-assignment of regions. > [AMv2] Enable and fix TestRestartCluster#testRetainAssignmentOnRestart on > master > > > Key: HBASE-18575 > URL: https://issues.apache.org/jira/browse/HBASE-18575 > Project: HBase > Issue Type: Bug > Components: amv2 >Affects Versions: 2.0.0 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Critical > Fix For: 2.0.0 > > > * Test creates 3 node cluster (master + 2 RS) and creates 3 tables with 1 > region each. > * It then takes a snapshot of region assignments > * Shuts down the cluster and restarts with 4 nodes (2 RS on same ports, 1 RS > on port where master was running previously and master on new port) > * Takes snapshot of region assignments > * Expected behavior is regions will be retained on RS with appropriate ports > Debugging done so far shows that, when cluster is restarted meta is loaded > but meta has old entries (though the ports are same for RSs, start timestamps > are different). AssignmentManager#processofflineServersWithOnlineRegions() > finds that old RS with different timestamps are not online even though > regions assigned to them are online; triggering submitting SCP for each those > RSs. Subprocedure AssignProcedure of SCP has forceNewPlan set to true. Which > triggers re-assignment of regions. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18197) Avoided to call job.waitForCompletion(true) two times
[
https://issues.apache.org/jira/browse/HBASE-18197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124129#comment-16124129
]
Hudson commented on HBASE-18197:
SUCCESS: Integrated in Jenkins build HBase-1.3-JDK8 #243 (See
[https://builds.apache.org/job/HBase-1.3-JDK8/243/])
HBASE-18197 Avoided to call job.waitForCompletion(true) two times (chia7712:
rev 18726b370e0f9c178ae2b7a82803f3c53b6fd2eb)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
> Avoided to call job.waitForCompletion(true) two times
> -
>
> Key: HBASE-18197
> URL: https://issues.apache.org/jira/browse/HBASE-18197
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 1.0.2, 1.2.0, 1.4.0
>Reporter: Chandra Sekhar
>Assignee: Jan Hentschel
>Priority: Trivial
> Fix For: 1.4.0, 1.3.2, 1.5.0, 1.2.7
>
> Attachments: HBASE-18197.branch-1.0.001.patch,
> HBASE-18197.branch-1.0.001.patch, HBASE-18197.branch-1.2.001.patch,
> HBASE-18197.branch-1.2.001.patch, HBASE-18197.branch-1.2.002.patch
>
>
> import.java, job output is printing two times.
> {quote}
> after job completed, job.waitForCompletion(true) is calling two times.
> {quote}
> {code}
> Job job = createSubmittableJob(conf, otherArgs);
> boolean isJobSuccessful = job.waitForCompletion(true);
> if(isJobSuccessful){
> // Flush all the regions of the table
> flushRegionsIfNecessary(conf);
> }
> long inputRecords =
> job.getCounters().findCounter(TaskCounter.MAP_INPUT_RECORDS).getValue();
> long outputRecords =
> job.getCounters().findCounter(TaskCounter.MAP_OUTPUT_RECORDS).getValue();
> if (outputRecords < inputRecords) {
> System.err.println("Warning, not all records were imported (maybe
> filtered out).");
> if (outputRecords == 0) {
> System.err.println("If the data was exported from HBase 0.94 "+
> "consider using -Dhbase.import.version=0.94.");
> }
> }
> System.exit(job.waitForCompletion(true) ? 0 : 1);
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18574) [C++] flakey test failure of TestFailWithException and FailWithExceptionFromRegionLocationLookupSplitRegions
[
https://issues.apache.org/jira/browse/HBASE-18574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaobing Zhou updated HBASE-18574:
--
Description:
It's been noticed TestFailWithException (i.e. in async-rpc-retrying-test.cc)
and/or FailWithExceptionFromRegionLocationLookupSplitRegions(i.e. in
async-batch-rpc-retrying-test.cc) fails occasionally if run in a batch or
individually, e.g.
{noformat}
buck test --no-cache --no-results-cache --all
{noformat}
{noformat}
buck test --no-cache --no-results-cache //core:retry-test
{noformat}
{noformat}
buck test --no-cache --no-results-cache //core:multi-retry-test
{noformat}
was:
It's been noticed TestFailWithException (i.e. in async-rpc-retrying-test.cc)
and/or FailWithExceptionFromRegionLocationLookupSplitRegions(i.e. in
async-batch-rpc-retrying-test.cc) fails occasionally if run in a batch or
individually, e.g.
{noformat}
buck test --no-cache --no-results-cache --all
{noformat}
{noformat}
buck test --no-cache --no-results-cache //core:retry-test
{noformat}
> [C++] flakey test failure of TestFailWithException and
> FailWithExceptionFromRegionLocationLookupSplitRegions
>
>
> Key: HBASE-18574
> URL: https://issues.apache.org/jira/browse/HBASE-18574
> Project: HBase
> Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>
> It's been noticed TestFailWithException (i.e. in async-rpc-retrying-test.cc)
> and/or FailWithExceptionFromRegionLocationLookupSplitRegions(i.e. in
> async-batch-rpc-retrying-test.cc) fails occasionally if run in a batch or
> individually, e.g.
> {noformat}
> buck test --no-cache --no-results-cache --all
> {noformat}
> {noformat}
> buck test --no-cache --no-results-cache //core:retry-test
> {noformat}
> {noformat}
> buck test --no-cache --no-results-cache //core:multi-retry-test
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18574) [C++] flakey test failure of TestFailWithException and FailWithExceptionFromRegionLocationLookupSplitRegions
[
https://issues.apache.org/jira/browse/HBASE-18574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaobing Zhou updated HBASE-18574:
--
Description:
It's been noticed TestFailWithException (in async-rpc-retrying-test.cc) and/or
fails occasionally if run in a batch or individually, e.g.
{noformat}
buck test --no-cache --no-results-cache --all
{noformat}
{noformat}
buck test --no-cache --no-results-cache //core:retry-test
{noformat}
was:
It's been noticed TestFailWithException (in async-rpc-retrying-test.cc) fails
occasionally if run in a batch or individually, e.g.
{noformat}
buck test --no-cache --no-results-cache --all
{noformat}
{noformat}
buck test --no-cache --no-results-cache //core:retry-test
{noformat}
> [C++] flakey test failure of TestFailWithException and
> FailWithExceptionFromRegionLocationLookupSplitRegions
>
>
> Key: HBASE-18574
> URL: https://issues.apache.org/jira/browse/HBASE-18574
> Project: HBase
> Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>
> It's been noticed TestFailWithException (in async-rpc-retrying-test.cc)
> and/or fails occasionally if run in a batch or individually, e.g.
> {noformat}
> buck test --no-cache --no-results-cache --all
> {noformat}
> {noformat}
> buck test --no-cache --no-results-cache //core:retry-test
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18574) [C++] flakey test failure of TestFailWithException and FailWithExceptionFromRegionLocationLookupSplitRegions
[
https://issues.apache.org/jira/browse/HBASE-18574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaobing Zhou updated HBASE-18574:
--
Description:
It's been noticed TestFailWithException (i.e. in async-rpc-retrying-test.cc)
and/or FailWithExceptionFromRegionLocationLookupSplitRegions(i.e. in
async-batch-rpc-retrying-test.cc) fails occasionally if run in a batch or
individually, e.g.
{noformat}
buck test --no-cache --no-results-cache --all
{noformat}
{noformat}
buck test --no-cache --no-results-cache //core:retry-test
{noformat}
was:
It's been noticed TestFailWithException (in async-rpc-retrying-test.cc) and/or
fails occasionally if run in a batch or individually, e.g.
{noformat}
buck test --no-cache --no-results-cache --all
{noformat}
{noformat}
buck test --no-cache --no-results-cache //core:retry-test
{noformat}
> [C++] flakey test failure of TestFailWithException and
> FailWithExceptionFromRegionLocationLookupSplitRegions
>
>
> Key: HBASE-18574
> URL: https://issues.apache.org/jira/browse/HBASE-18574
> Project: HBase
> Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>
> It's been noticed TestFailWithException (i.e. in async-rpc-retrying-test.cc)
> and/or FailWithExceptionFromRegionLocationLookupSplitRegions(i.e. in
> async-batch-rpc-retrying-test.cc) fails occasionally if run in a batch or
> individually, e.g.
> {noformat}
> buck test --no-cache --no-results-cache --all
> {noformat}
> {noformat}
> buck test --no-cache --no-results-cache //core:retry-test
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18574) [C++] flakey test failure of TestFailWithException and FailWithExceptionFromRegionLocationLookupSplitRegions
[
https://issues.apache.org/jira/browse/HBASE-18574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaobing Zhou updated HBASE-18574:
--
Summary: [C++] flakey test failure of TestFailWithException and
FailWithExceptionFromRegionLocationLookupSplitRegions (was: [C++] flakey test
failure of AsyncRpcRetryTest::TestFailWithException )
> [C++] flakey test failure of TestFailWithException and
> FailWithExceptionFromRegionLocationLookupSplitRegions
>
>
> Key: HBASE-18574
> URL: https://issues.apache.org/jira/browse/HBASE-18574
> Project: HBase
> Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>
> It's been noticed TestFailWithException (in async-rpc-retrying-test.cc) fails
> occasionally if run in a batch or individually, e.g.
> {noformat}
> buck test --no-cache --no-results-cache --all
> {noformat}
> {noformat}
> buck test --no-cache --no-results-cache //core:retry-test
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HBASE-18575) [AMv2] Enable and fix TestRestartCluster#testRetainAssignmentOnRestart on master
Umesh Agashe created HBASE-18575: Summary: [AMv2] Enable and fix TestRestartCluster#testRetainAssignmentOnRestart on master Key: HBASE-18575 URL: https://issues.apache.org/jira/browse/HBASE-18575 Project: HBase Issue Type: Bug Components: amv2 Affects Versions: 2.0.0 Reporter: Umesh Agashe Assignee: Umesh Agashe Priority: Critical Fix For: 2.0.0 * Test creates 3 node cluster (master + 2 RS) and creates 3 tables with 1 region each. * It then takes a snapshot of region assignments * Shuts down the cluster and restarts with 4 nodes (2 RS on same ports, 1 RS on port where master was running previously and master on new port) * Takes snapshot of region assignments * Expected behavior is regions will be retained on RS with appropriate ports Debugging done so far shows that, when cluster is restarted meta is loaded but meta has old entries (though the ports are same for RSs, start timestamps are different). AssignmentManager#processofflineServersWithOnlineRegions() finds that old RS with different timestamps are not online even though regions assigned to them are online. This trigger submitting SCP for each those RSs. Subprocedure AssignProcedure of SCP has forceNewPlan set to true. Which triggers re-assignment of regions. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HBASE-18144) Forward-port the old exclusive row lock; there are scenarios where it performs better
[ https://issues.apache.org/jira/browse/HBASE-18144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-18144. --- Resolution: Won't Fix Resolving as 'wont fix'; HBASE-17924) plus HBASE-18233 is a better solution to the regression and they do not require configuration. > Forward-port the old exclusive row lock; there are scenarios where it > performs better > - > > Key: HBASE-18144 > URL: https://issues.apache.org/jira/browse/HBASE-18144 > Project: HBase > Issue Type: Bug > Components: Increment >Affects Versions: 1.2.5 >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0, 1.3.2, 1.2.7 > > Attachments: DisorderedBatchAndIncrementUT.patch, > HBASE-18144.master.001.patch > > > Description to follow. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18197) Avoided to call job.waitForCompletion(true) two times
[
https://issues.apache.org/jira/browse/HBASE-18197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124105#comment-16124105
]
Hudson commented on HBASE-18197:
SUCCESS: Integrated in Jenkins build HBase-1.2-JDK8 #187 (See
[https://builds.apache.org/job/HBase-1.2-JDK8/187/])
HBASE-18197 Avoided to call job.waitForCompletion(true) two times (chia7712:
rev b9a57f45578f0d21aab04d95c241d7c7e80e2552)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
> Avoided to call job.waitForCompletion(true) two times
> -
>
> Key: HBASE-18197
> URL: https://issues.apache.org/jira/browse/HBASE-18197
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 1.0.2, 1.2.0, 1.4.0
>Reporter: Chandra Sekhar
>Assignee: Jan Hentschel
>Priority: Trivial
> Fix For: 1.4.0, 1.3.2, 1.5.0, 1.2.7
>
> Attachments: HBASE-18197.branch-1.0.001.patch,
> HBASE-18197.branch-1.0.001.patch, HBASE-18197.branch-1.2.001.patch,
> HBASE-18197.branch-1.2.001.patch, HBASE-18197.branch-1.2.002.patch
>
>
> import.java, job output is printing two times.
> {quote}
> after job completed, job.waitForCompletion(true) is calling two times.
> {quote}
> {code}
> Job job = createSubmittableJob(conf, otherArgs);
> boolean isJobSuccessful = job.waitForCompletion(true);
> if(isJobSuccessful){
> // Flush all the regions of the table
> flushRegionsIfNecessary(conf);
> }
> long inputRecords =
> job.getCounters().findCounter(TaskCounter.MAP_INPUT_RECORDS).getValue();
> long outputRecords =
> job.getCounters().findCounter(TaskCounter.MAP_OUTPUT_RECORDS).getValue();
> if (outputRecords < inputRecords) {
> System.err.println("Warning, not all records were imported (maybe
> filtered out).");
> if (outputRecords == 0) {
> System.err.println("If the data was exported from HBase 0.94 "+
> "consider using -Dhbase.import.version=0.94.");
> }
> }
> System.exit(job.waitForCompletion(true) ? 0 : 1);
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18574) [C++] flakey test failure of AsyncRpcRetryTest::TestFailWithException
[
https://issues.apache.org/jira/browse/HBASE-18574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaobing Zhou updated HBASE-18574:
--
Description:
It's been noticed TestFailWithException (in async-rpc-retrying-test.cc) fails
occasionally if run in a batch or individually, e.g.
{noformat}
buck test --no-cache --no-results-cache --all
{noformat}
{noformat}
buck test --no-cache --no-results-cache //core:retry-test
{noformat}
was:It's been noticed TestFailWithException (in async-rpc-retrying-test.cc)
fails occasionally.
> [C++] flakey test failure of AsyncRpcRetryTest::TestFailWithException
> --
>
> Key: HBASE-18574
> URL: https://issues.apache.org/jira/browse/HBASE-18574
> Project: HBase
> Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>
> It's been noticed TestFailWithException (in async-rpc-retrying-test.cc) fails
> occasionally if run in a batch or individually, e.g.
> {noformat}
> buck test --no-cache --no-results-cache --all
> {noformat}
> {noformat}
> buck test --no-cache --no-results-cache //core:retry-test
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18233) We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock
[ https://issues.apache.org/jira/browse/HBASE-18233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-18233: -- Release Note: This patch plus the sort of mutations done in HBASE-17924 fixes a regression doing increments and/or checkAndPut-style operations. > We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock > - > > Key: HBASE-18233 > URL: https://issues.apache.org/jira/browse/HBASE-18233 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.7 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0, 1.3.2, 1.2.7 > > Attachments: HBASE-18233-branch-1.2.patch, > HBASE-18233-branch-1.2.v2.patch, HBASE-18233-branch-1.2.v3.patch > > > Please refer to the discuss in HBASE-18144 > https://issues.apache.org/jira/browse/HBASE-18144?focusedCommentId=16051701=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16051701 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18574) [C++] flakey test failure of AsyncRpcRetryTest::TestFailWithException
[ https://issues.apache.org/jira/browse/HBASE-18574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HBASE-18574: -- Description: It's been noticed TestFailWithException (in async-rpc-retrying-test.cc) fails occasionally. > [C++] flakey test failure of AsyncRpcRetryTest::TestFailWithException > -- > > Key: HBASE-18574 > URL: https://issues.apache.org/jira/browse/HBASE-18574 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou > > It's been noticed TestFailWithException (in async-rpc-retrying-test.cc) fails > occasionally. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18233) We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock
[ https://issues.apache.org/jira/browse/HBASE-18233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-18233: -- Priority: Blocker (was: Major) > We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock > - > > Key: HBASE-18233 > URL: https://issues.apache.org/jira/browse/HBASE-18233 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.7 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0, 1.3.2, 1.2.7 > > Attachments: HBASE-18233-branch-1.2.patch, > HBASE-18233-branch-1.2.v2.patch, HBASE-18233-branch-1.2.v3.patch > > > Please refer to the discuss in HBASE-18144 > https://issues.apache.org/jira/browse/HBASE-18144?focusedCommentId=16051701=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16051701 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18233) We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock
[ https://issues.apache.org/jira/browse/HBASE-18233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-18233: -- Fix Version/s: 1.2.7 1.3.2 1.4.0 2.0.0 > We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock > - > > Key: HBASE-18233 > URL: https://issues.apache.org/jira/browse/HBASE-18233 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.7 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0, 1.3.2, 1.2.7 > > Attachments: HBASE-18233-branch-1.2.patch, > HBASE-18233-branch-1.2.v2.patch, HBASE-18233-branch-1.2.v3.patch > > > Please refer to the discuss in HBASE-18144 > https://issues.apache.org/jira/browse/HBASE-18144?focusedCommentId=16051701=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16051701 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18144) Forward-port the old exclusive row lock; there are scenarios where it performs better
[ https://issues.apache.org/jira/browse/HBASE-18144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124095#comment-16124095 ] stack commented on HBASE-18144: --- Update. Sort (HBASE-17924) plus HBASE-18233 is as good as and probably better than this backport according to a production deploy test so am going to resolve this as won't do in favor of HBASE-18233. > Forward-port the old exclusive row lock; there are scenarios where it > performs better > - > > Key: HBASE-18144 > URL: https://issues.apache.org/jira/browse/HBASE-18144 > Project: HBase > Issue Type: Bug > Components: Increment >Affects Versions: 1.2.5 >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0, 1.3.2, 1.2.7 > > Attachments: DisorderedBatchAndIncrementUT.patch, > HBASE-18144.master.001.patch > > > Description to follow. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18574) [C++] flakey test failure of AsyncRpcRetryTest::TestFailWithException
Xiaobing Zhou created HBASE-18574: - Summary: [C++] flakey test failure of AsyncRpcRetryTest::TestFailWithException Key: HBASE-18574 URL: https://issues.apache.org/jira/browse/HBASE-18574 Project: HBase Issue Type: Sub-task Reporter: Xiaobing Zhou -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18555) Remove redundant familyMap.put() from addxxx() of sub-classes of Mutation and Query
[
https://issues.apache.org/jira/browse/HBASE-18555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124070#comment-16124070
]
Chia-Ping Tsai commented on HBASE-18555:
bq. I wonder how to trigger the hbase-server test too.
Add a trivial change to hbase-server
> Remove redundant familyMap.put() from addxxx() of sub-classes of Mutation and
> Query
> ---
>
> Key: HBASE-18555
> URL: https://issues.apache.org/jira/browse/HBASE-18555
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Attachments: HBASE-18555.master.000.patch,
> HBASE-18555.master.001.patch
>
>
> In addxxx() functions of Mutation(Append, Delete, Increment and Put) and
> Query(Get and Scan), there are redundant Map#put() calls which could be
> removed to improve the performance.
> For example, in Put#addColumn() and addImmutable(), after getting the cell
> list of the given family and add the cell into the list, the code puts
> (key=family, value=list) into familyMap.
> In addColumn(), it is like
> {code}
> List list = getCellList(family);
> KeyValue kv = createPutKeyValue(family, qualifier, ts, value);
> list.add(kv);
> familyMap.put(CellUtil.cloneFamily(kv), list); // <-- here
> return this;
> {code}
> In addImmutable(), it is like
> {code}
> List list = getCellList(family);
> KeyValue kv = createPutKeyValue(family, qualifier, ts, value, tag);
> list.add(kv);
> familyMap.put(family, list); // <-- here
> return this;
> {code}
> I think those put() for Map only take effect when getCellList(family) returns
> a new allocated ArrayList. When the list for a family already exist, put()
> for Map will update the value to the reference of the list, but actually, the
> reference of the list is not changed.
> Those put() do not do any harm in terms of the correctness when they are
> here. But it could be removed to improve the performance. familyMap searches
> for key and set the new value and return the old value. Those operation take
> some time but actually are not needed.
> The put() could be moved into Mutation#getCellList(family)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18555) Remove redundant familyMap.put() from addxxx() of sub-classes of Mutation and Query
[
https://issues.apache.org/jira/browse/HBASE-18555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124067#comment-16124067
]
Jerry He commented on HBASE-18555:
--
+1
I wonder how to trigger the hbase-server test too.
> Remove redundant familyMap.put() from addxxx() of sub-classes of Mutation and
> Query
> ---
>
> Key: HBASE-18555
> URL: https://issues.apache.org/jira/browse/HBASE-18555
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Attachments: HBASE-18555.master.000.patch,
> HBASE-18555.master.001.patch
>
>
> In addxxx() functions of Mutation(Append, Delete, Increment and Put) and
> Query(Get and Scan), there are redundant Map#put() calls which could be
> removed to improve the performance.
> For example, in Put#addColumn() and addImmutable(), after getting the cell
> list of the given family and add the cell into the list, the code puts
> (key=family, value=list) into familyMap.
> In addColumn(), it is like
> {code}
> List list = getCellList(family);
> KeyValue kv = createPutKeyValue(family, qualifier, ts, value);
> list.add(kv);
> familyMap.put(CellUtil.cloneFamily(kv), list); // <-- here
> return this;
> {code}
> In addImmutable(), it is like
> {code}
> List list = getCellList(family);
> KeyValue kv = createPutKeyValue(family, qualifier, ts, value, tag);
> list.add(kv);
> familyMap.put(family, list); // <-- here
> return this;
> {code}
> I think those put() for Map only take effect when getCellList(family) returns
> a new allocated ArrayList. When the list for a family already exist, put()
> for Map will update the value to the reference of the list, but actually, the
> reference of the list is not changed.
> Those put() do not do any harm in terms of the correctness when they are
> here. But it could be removed to improve the performance. familyMap searches
> for key and set the new value and return the old value. Those operation take
> some time but actually are not needed.
> The put() could be moved into Mutation#getCellList(family)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18532) Improve cache related stats rendered on RS UI
[ https://issues.apache.org/jira/browse/HBASE-18532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124063#comment-16124063 ] Biju Nair commented on HBASE-18532: --- Looks like HBASE-14582 removes the long L2 block list rendered on the UI when L2 cache is selected. > Improve cache related stats rendered on RS UI > - > > Key: HBASE-18532 > URL: https://issues.apache.org/jira/browse/HBASE-18532 > Project: HBase > Issue Type: Improvement > Components: regionserver, UI >Affects Versions: 1.1.2 >Reporter: Biju Nair > Attachments: COMBINED-STATS.PNG, L1-STATS.PNG, L2-STATS.PNG > > > The stats currently rendered for L1 and L2 cache are incorrect. Refer to the > attached screenshots of stats from a cluster showing the combined cache > stats, L1 stats and L2 stats. For e.g. the combined stats shows 38 GB used > for cache while if we sum size of L1 and L2 cache the value is way less. One > way we can improve this is to use the same stats used to populate the > combined stats to render the values of L1 & L2 cache. Also for usability we > can remove the table with details with BucketCache buckets from the L2 cache > stats since this is going to be long for any installation using L2 cache. > This will help in understanding the cache usage better. Thoughts? If there > are no concerns will submit a patch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18522) Add RowMutations support to Batch
[ https://issues.apache.org/jira/browse/HBASE-18522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124055#comment-16124055 ] Jerry He commented on HBASE-18522: -- Hello, more review please. If no more, I will commit soon. Thanks. > Add RowMutations support to Batch > - > > Key: HBASE-18522 > URL: https://issues.apache.org/jira/browse/HBASE-18522 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.6 >Reporter: Jerry He >Assignee: Jerry He > Fix For: 2.0.0, 3.0.0, 1.4.0, 1.5.0 > > Attachments: HBASE-18522-branch-1.patch, > HBASE-18522-branch-1-v2.patch, HBASE-18522-master-v2.patch > > > RowMutations is multiple Puts and/or Deletes atomically on a single row. > Current Batch call does not support RowMutations as part of the batch. > We should add this missing part. We should be able to batch RowMutations. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-18541) [C++] Segfaults from JNI
[
https://issues.apache.org/jira/browse/HBASE-18541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124037#comment-16124037
]
Ted Yu edited comment on HBASE-18541 at 8/11/17 8:59 PM:
-
Looped the test 10 times which passed.
Previously I encountered deadlock which was resolved earlier today
(HBASE-18565).
was (Author: yuzhih...@gmail.com):
Looped the test 10 times which passed.
Previously I encountered deadlock which was resolved earlier today.
> [C++] Segfaults from JNI
>
>
> Key: HBASE-18541
> URL: https://issues.apache.org/jira/browse/HBASE-18541
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Ted Yu
>
> retry-test and multi-retry-test fails flakily when run with
> {code}
> buck test --all --no-results-cache
> {code}
> or when run in a loop:
> {code}
> for i in `seq 1 10`; do buck test --no-results-cache core:retry-test || break
> 1; done
> {code}
> The problem seems to be within the JNI internals and usually happens at the
> create table method call. I was not able to inspect much, but the comments in
> our mini-cluster indicate that we may need to use global references instead
> of local ones. I suspect the problem happens when there is a GC run for the
> test since the failure happens usually after some time (but almost always in
> create table method).
> [~ted_yu] do you mind taking a look at this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18541) [C++] Segfaults from JNI
[
https://issues.apache.org/jira/browse/HBASE-18541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124037#comment-16124037
]
Ted Yu commented on HBASE-18541:
Looped the test 10 times which passed.
Previously I encountered deadlock which was resolved earlier today.
> [C++] Segfaults from JNI
>
>
> Key: HBASE-18541
> URL: https://issues.apache.org/jira/browse/HBASE-18541
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Ted Yu
>
> retry-test and multi-retry-test fails flakily when run with
> {code}
> buck test --all --no-results-cache
> {code}
> or when run in a loop:
> {code}
> for i in `seq 1 10`; do buck test --no-results-cache core:retry-test || break
> 1; done
> {code}
> The problem seems to be within the JNI internals and usually happens at the
> create table method call. I was not able to inspect much, but the comments in
> our mini-cluster indicate that we may need to use global references instead
> of local ones. I suspect the problem happens when there is a GC run for the
> test since the failure happens usually after some time (but almost always in
> create table method).
> [~ted_yu] do you mind taking a look at this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18197) Avoided to call job.waitForCompletion(true) two times
[
https://issues.apache.org/jira/browse/HBASE-18197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124029#comment-16124029
]
Hudson commented on HBASE-18197:
FAILURE: Integrated in Jenkins build HBase-1.4 #849 (See
[https://builds.apache.org/job/HBase-1.4/849/])
HBASE-18197 Avoided to call job.waitForCompletion(true) two times (chia7712:
rev b181f172ef92ff1fb3bf2a2907c7c143bcd5ac75)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
> Avoided to call job.waitForCompletion(true) two times
> -
>
> Key: HBASE-18197
> URL: https://issues.apache.org/jira/browse/HBASE-18197
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 1.0.2, 1.2.0, 1.4.0
>Reporter: Chandra Sekhar
>Assignee: Jan Hentschel
>Priority: Trivial
> Fix For: 1.4.0, 1.3.2, 1.5.0, 1.2.7
>
> Attachments: HBASE-18197.branch-1.0.001.patch,
> HBASE-18197.branch-1.0.001.patch, HBASE-18197.branch-1.2.001.patch,
> HBASE-18197.branch-1.2.001.patch, HBASE-18197.branch-1.2.002.patch
>
>
> import.java, job output is printing two times.
> {quote}
> after job completed, job.waitForCompletion(true) is calling two times.
> {quote}
> {code}
> Job job = createSubmittableJob(conf, otherArgs);
> boolean isJobSuccessful = job.waitForCompletion(true);
> if(isJobSuccessful){
> // Flush all the regions of the table
> flushRegionsIfNecessary(conf);
> }
> long inputRecords =
> job.getCounters().findCounter(TaskCounter.MAP_INPUT_RECORDS).getValue();
> long outputRecords =
> job.getCounters().findCounter(TaskCounter.MAP_OUTPUT_RECORDS).getValue();
> if (outputRecords < inputRecords) {
> System.err.println("Warning, not all records were imported (maybe
> filtered out).");
> if (outputRecords == 0) {
> System.err.println("If the data was exported from HBase 0.94 "+
> "consider using -Dhbase.import.version=0.94.");
> }
> }
> System.exit(job.waitForCompletion(true) ? 0 : 1);
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18557) change splitable to mergeable in MergeTableRegionsProcedure
[
https://issues.apache.org/jira/browse/HBASE-18557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124011#comment-16124011
]
Jerry He commented on HBASE-18557:
--
Pushed to master and branch-2.
> change splitable to mergeable in MergeTableRegionsProcedure
> ---
>
> Key: HBASE-18557
> URL: https://issues.apache.org/jira/browse/HBASE-18557
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-18557-master-v1.patch, HBASE-18557-master-v1.patch
>
>
> {code}
> private boolean isMergeable(final MasterProcedureEnv env, final RegionState
> rs)
> throws IOException {
> GetRegionInfoResponse response =
> Util.getRegionInfoResponse(env, rs.getServerName(), rs.getRegion());
> return response.hasSplittable() && response.getSplittable();
> }
> {code}
> should be mergeable
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18557) change splitable to mergeable in MergeTableRegionsProcedure
[
https://issues.apache.org/jira/browse/HBASE-18557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry He updated HBASE-18557:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: (was: 2.0.0-alpha-3)
2.0.0-alpha-2
Status: Resolved (was: Patch Available)
> change splitable to mergeable in MergeTableRegionsProcedure
> ---
>
> Key: HBASE-18557
> URL: https://issues.apache.org/jira/browse/HBASE-18557
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-18557-master-v1.patch, HBASE-18557-master-v1.patch
>
>
> {code}
> private boolean isMergeable(final MasterProcedureEnv env, final RegionState
> rs)
> throws IOException {
> GetRegionInfoResponse response =
> Util.getRegionInfoResponse(env, rs.getServerName(), rs.getRegion());
> return response.hasSplittable() && response.getSplittable();
> }
> {code}
> should be mergeable
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18557) change splitable to mergeable in MergeTableRegionsProcedure
[
https://issues.apache.org/jira/browse/HBASE-18557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124004#comment-16124004
]
Jerry He commented on HBASE-18557:
--
+1
> change splitable to mergeable in MergeTableRegionsProcedure
> ---
>
> Key: HBASE-18557
> URL: https://issues.apache.org/jira/browse/HBASE-18557
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Fix For: 3.0.0, 2.0.0-alpha-3
>
> Attachments: HBASE-18557-master-v1.patch, HBASE-18557-master-v1.patch
>
>
> {code}
> private boolean isMergeable(final MasterProcedureEnv env, final RegionState
> rs)
> throws IOException {
> GetRegionInfoResponse response =
> Util.getRegionInfoResponse(env, rs.getServerName(), rs.getRegion());
> return response.hasSplittable() && response.getSplittable();
> }
> {code}
> should be mergeable
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18558) clean up duplicate dependency management entries for hbase-shaded-miscellaneous
[
https://issues.apache.org/jira/browse/HBASE-18558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124002#comment-16124002
]
Hudson commented on HBASE-18558:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3514 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/3514/])
HBASE-18558 clean up duplicate dependency management entries for (stack: rev
043ec9b37e43328e8784f88e3d6867b007a31d1d)
* (edit) pom.xml
> clean up duplicate dependency management entries for
> hbase-shaded-miscellaneous
> ---
>
> Key: HBASE-18558
> URL: https://issues.apache.org/jira/browse/HBASE-18558
> Project: HBase
> Issue Type: Task
> Components: dependencies, pom
>Affects Versions: 3.0.0
>Reporter: Sean Busbey
>Assignee: ChunHao
>Priority: Minor
> Labels: beginner
> Fix For: 3.0.0
>
>
> our root pom's dependency management section has two entries like:
> {code}
>
> org.apache.hbase.thirdparty
> hbase-shaded-miscellaneous
> ${hbase-thirdparty.version}
>
> {code}
> which causes a bunch of warnings at build time. remove one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18551) [AMv2] UnassignProcedure and crashed regionservers
[ https://issues.apache.org/jira/browse/HBASE-18551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124001#comment-16124001 ] Hudson commented on HBASE-18551: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3514 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3514/]) HBASE-18551 [AMv2] UnassignProcedure and crashed regionservers (stack: rev 6f44b24860192d81dbf88ffd834d4b998a6fe636) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableNamespaceManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashException.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionTransitionProcedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignProcedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/UnassignProcedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java > [AMv2] UnassignProcedure and crashed regionservers > -- > > Key: HBASE-18551 > URL: https://issues.apache.org/jira/browse/HBASE-18551 > Project: HBase > Issue Type: Bug > Components: amv2 >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: HBASE-18551.master.001.patch, > HBASE-18551.master.002.patch, HBASE-18551.master.003.patch > > > This has been [~uagashe] and my obsession over the last few days, what should > an UnassignProcedure do when it dispatches a CLOSE but the CLOSE fails > because of ConnectException or SocketTimeout. > + We used to let UnassignProcedure continue presuming the Region would be > closed since the server is dead. BUT, if the unassign was part of a > MoveProcedure, the unassign would proceed and the Move would then run WITHOUT > first splitting logs. Bad. > + So, we made it so UnassignProcedure failed; let the upper layers take care > of the failure. See HBASE-18491 that enabled this behavior. BUT, we are since > figuring that even if the UP completes as a failure, since it gives up the > Region lock on completion, another procedure -- say an AssignProcedure -- > could cut in before the ServerCrashProcedure had finished and again there > could be dataloss. > + Now we are thinking the UP should hold on to the Region lock until we are > signalled by a ServerCrashProcedure; only then let go of the region. The UP > has context that is hard to pass another. Waiting on a SCP has the UP living > on for what could be a good amount of time. It might be ok if we can suspend > the procedure. > There is a good sample scenario that came up doing the no-regions-on-master > issue, HBASE-18511. When meta is not on master, TestSplitTransactionOnCluster > is failing. It fails because though the test completes, the tests commonly > kill a RegionServer. The teardown for the test runs before we've noticed the > aborted RS. So, the disable of the table in the teardown prepartory to our > deleting the test table as part of clean up, goes to unassign regions but the > unassign fails against the aborted server. > Good stuff. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18526) FIFOCompactionPolicy pre-check uses wrong scope
[
https://issues.apache.org/jira/browse/HBASE-18526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123998#comment-16123998
]
Ted Yu commented on HBASE-18526:
lgtm, pending QA.
> FIFOCompactionPolicy pre-check uses wrong scope
> ---
>
> Key: HBASE-18526
> URL: https://issues.apache.org/jira/browse/HBASE-18526
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 1.3.1
>Reporter: Lars George
>Assignee: Vladimir Rodionov
> Attachments: HBASE-18526-v1.patch
>
>
> See https://issues.apache.org/jira/browse/HBASE-14468
> It adds this check to {{HMaster.checkCompactionPolicy()}}:
> {code}
> // 1. Check TTL
> if (hcd.getTimeToLive() == HColumnDescriptor.DEFAULT_TTL) {
> message = "Default TTL is not supported for FIFO compaction";
> throw new IOException(message);
> }
> // 2. Check min versions
> if (hcd.getMinVersions() > 0) {
> message = "MIN_VERSION > 0 is not supported for FIFO compaction";
> throw new IOException(message);
> }
> // 3. blocking file count
> String sbfc = htd.getConfigurationValue(HStore.BLOCKING_STOREFILES_KEY);
> if (sbfc != null) {
> blockingFileCount = Integer.parseInt(sbfc);
> }
> if (blockingFileCount < 1000) {
> message =
> "blocking file count '" + HStore.BLOCKING_STOREFILES_KEY + "' "
> + blockingFileCount
> + " is below recommended minimum of 1000";
> throw new IOException(message);
> }
> {code}
> Why does it only check the blocking file count on the HTD level, while
> others are check on the HCD level? Doing this for example fails
> because of it:
> {noformat}
> hbase(main):008:0> create 'ttltable', { NAME => 'cf1', TTL => 300,
> CONFIGURATION => { 'hbase.hstore.defaultengine.compactionpolicy.class'
> => 'org.apache.hadoop.hbase.regionserver.compactions.FIFOCompactionPolicy',
> 'hbase.hstore.blockingStoreFiles' => 2000 } }
> ERROR: org.apache.hadoop.hbase.DoNotRetryIOException: blocking file
> count 'hbase.hstore.blockingStoreFiles' 10 is below recommended
> minimum of 1000 Set hbase.table.sanity.checks to false at conf or
> table descriptor if you want to bypass sanity checks
> at
> org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:1782)
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1663)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1545)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:469)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58549)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
> Caused by: java.io.IOException: blocking file count
> 'hbase.hstore.blockingStoreFiles' 10 is below recommended minimum of
> 1000
> at
> org.apache.hadoop.hbase.master.HMaster.checkCompactionPolicy(HMaster.java:1773)
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1661)
> ... 7 more
> {noformat}
> The check should be performed on the column family level instead.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18233) We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock
[ https://issues.apache.org/jira/browse/HBASE-18233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123999#comment-16123999 ] stack commented on HBASE-18233: --- [~allan163] Got notice from the user who was having trouble w/ increments being slow. They said this patch plus sort makes a big difference. Lets commit. I tried to but it has rotted. I'll have a look in next week or so unless you beat me too it. Good stuff. > We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock > - > > Key: HBASE-18233 > URL: https://issues.apache.org/jira/browse/HBASE-18233 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.7 >Reporter: Allan Yang >Assignee: Allan Yang > Attachments: HBASE-18233-branch-1.2.patch, > HBASE-18233-branch-1.2.v2.patch, HBASE-18233-branch-1.2.v3.patch > > > Please refer to the discuss in HBASE-18144 > https://issues.apache.org/jira/browse/HBASE-18144?focusedCommentId=16051701=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16051701 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18526) FIFOCompactionPolicy pre-check uses wrong scope
[
https://issues.apache.org/jira/browse/HBASE-18526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-18526:
---
Status: Patch Available (was: Open)
> FIFOCompactionPolicy pre-check uses wrong scope
> ---
>
> Key: HBASE-18526
> URL: https://issues.apache.org/jira/browse/HBASE-18526
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 1.3.1
>Reporter: Lars George
>Assignee: Vladimir Rodionov
> Attachments: HBASE-18526-v1.patch
>
>
> See https://issues.apache.org/jira/browse/HBASE-14468
> It adds this check to {{HMaster.checkCompactionPolicy()}}:
> {code}
> // 1. Check TTL
> if (hcd.getTimeToLive() == HColumnDescriptor.DEFAULT_TTL) {
> message = "Default TTL is not supported for FIFO compaction";
> throw new IOException(message);
> }
> // 2. Check min versions
> if (hcd.getMinVersions() > 0) {
> message = "MIN_VERSION > 0 is not supported for FIFO compaction";
> throw new IOException(message);
> }
> // 3. blocking file count
> String sbfc = htd.getConfigurationValue(HStore.BLOCKING_STOREFILES_KEY);
> if (sbfc != null) {
> blockingFileCount = Integer.parseInt(sbfc);
> }
> if (blockingFileCount < 1000) {
> message =
> "blocking file count '" + HStore.BLOCKING_STOREFILES_KEY + "' "
> + blockingFileCount
> + " is below recommended minimum of 1000";
> throw new IOException(message);
> }
> {code}
> Why does it only check the blocking file count on the HTD level, while
> others are check on the HCD level? Doing this for example fails
> because of it:
> {noformat}
> hbase(main):008:0> create 'ttltable', { NAME => 'cf1', TTL => 300,
> CONFIGURATION => { 'hbase.hstore.defaultengine.compactionpolicy.class'
> => 'org.apache.hadoop.hbase.regionserver.compactions.FIFOCompactionPolicy',
> 'hbase.hstore.blockingStoreFiles' => 2000 } }
> ERROR: org.apache.hadoop.hbase.DoNotRetryIOException: blocking file
> count 'hbase.hstore.blockingStoreFiles' 10 is below recommended
> minimum of 1000 Set hbase.table.sanity.checks to false at conf or
> table descriptor if you want to bypass sanity checks
> at
> org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:1782)
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1663)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1545)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:469)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58549)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
> Caused by: java.io.IOException: blocking file count
> 'hbase.hstore.blockingStoreFiles' 10 is below recommended minimum of
> 1000
> at
> org.apache.hadoop.hbase.master.HMaster.checkCompactionPolicy(HMaster.java:1773)
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1661)
> ... 7 more
> {noformat}
> The check should be performed on the column family level instead.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18526) FIFOCompactionPolicy pre-check uses wrong scope
[
https://issues.apache.org/jira/browse/HBASE-18526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123989#comment-16123989
]
Vladimir Rodionov commented on HBASE-18526:
---
[~te...@apache.org] can you take a look? This is a small fix.
> FIFOCompactionPolicy pre-check uses wrong scope
> ---
>
> Key: HBASE-18526
> URL: https://issues.apache.org/jira/browse/HBASE-18526
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 1.3.1
>Reporter: Lars George
>Assignee: Vladimir Rodionov
> Attachments: HBASE-18526-v1.patch
>
>
> See https://issues.apache.org/jira/browse/HBASE-14468
> It adds this check to {{HMaster.checkCompactionPolicy()}}:
> {code}
> // 1. Check TTL
> if (hcd.getTimeToLive() == HColumnDescriptor.DEFAULT_TTL) {
> message = "Default TTL is not supported for FIFO compaction";
> throw new IOException(message);
> }
> // 2. Check min versions
> if (hcd.getMinVersions() > 0) {
> message = "MIN_VERSION > 0 is not supported for FIFO compaction";
> throw new IOException(message);
> }
> // 3. blocking file count
> String sbfc = htd.getConfigurationValue(HStore.BLOCKING_STOREFILES_KEY);
> if (sbfc != null) {
> blockingFileCount = Integer.parseInt(sbfc);
> }
> if (blockingFileCount < 1000) {
> message =
> "blocking file count '" + HStore.BLOCKING_STOREFILES_KEY + "' "
> + blockingFileCount
> + " is below recommended minimum of 1000";
> throw new IOException(message);
> }
> {code}
> Why does it only check the blocking file count on the HTD level, while
> others are check on the HCD level? Doing this for example fails
> because of it:
> {noformat}
> hbase(main):008:0> create 'ttltable', { NAME => 'cf1', TTL => 300,
> CONFIGURATION => { 'hbase.hstore.defaultengine.compactionpolicy.class'
> => 'org.apache.hadoop.hbase.regionserver.compactions.FIFOCompactionPolicy',
> 'hbase.hstore.blockingStoreFiles' => 2000 } }
> ERROR: org.apache.hadoop.hbase.DoNotRetryIOException: blocking file
> count 'hbase.hstore.blockingStoreFiles' 10 is below recommended
> minimum of 1000 Set hbase.table.sanity.checks to false at conf or
> table descriptor if you want to bypass sanity checks
> at
> org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:1782)
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1663)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1545)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:469)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58549)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
> Caused by: java.io.IOException: blocking file count
> 'hbase.hstore.blockingStoreFiles' 10 is below recommended minimum of
> 1000
> at
> org.apache.hadoop.hbase.master.HMaster.checkCompactionPolicy(HMaster.java:1773)
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1661)
> ... 7 more
> {noformat}
> The check should be performed on the column family level instead.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-14135) HBase Backup/Restore Phase 3: Merge backup images
[
https://issues.apache.org/jira/browse/HBASE-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Rodionov updated HBASE-14135:
--
Attachment: HBASE-14135-v11.patch
We need a clean build run. Renamed v10 to v11
> HBase Backup/Restore Phase 3: Merge backup images
> -
>
> Key: HBASE-14135
> URL: https://issues.apache.org/jira/browse/HBASE-14135
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Critical
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-14135-v10.patch, HBASE-14135-v11.patch,
> HBASE-14135-v3.patch, HBASE-14135-v5.patch, HBASE-14135-v6.patch,
> HBASE-14135-v7.patch, HBASE-14135-v8.patch, HBASE-14135-v9.patch
>
>
> User can merge incremental backup images into single incremental backup image.
> # Merge supports only incremental images
> # Merge supports only images for the same backup destinations
> Command:
> {code}
> hbase backup merge image1,image2,..imageK
> {code}
> Example:
> {code}
> hbase backup merge backup_143126764557,backup_143126764456
> {code}
> When operation is complete, only the most recent backup image will be kept
> (in above example - backup_143126764557) as a merged backup image, all other
> images will be deleted from both: file system and backup system tables,
> corresponding backup manifest for the merged backup image will be updated to
> remove dependencies from deleted images. Merged backup image will contains
> all the data from original image and from deleted images.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18025) CatalogJanitor should collect outdated RegionStates from the AM
[ https://issues.apache.org/jira/browse/HBASE-18025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18025: --- Fix Version/s: 1.4.0 > CatalogJanitor should collect outdated RegionStates from the AM > --- > > Key: HBASE-18025 > URL: https://issues.apache.org/jira/browse/HBASE-18025 > Project: HBase > Issue Type: Bug >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez > Fix For: 2.0.0, 3.0.0, 1.4.0, 1.5.0 > > Attachments: HBASE-18025.001.patch, HBASE-18025.002.patch, > HBASE-18025.003.patch, HBASE-18025.004.patch, HBASE-18025.005.patch, > HBASE-18025-branch-1.005.patch, HBASE-18025-branch-1.006.patch > > > I don't think this will matter on the long run for HBase 2, but at least in > branch-1 and the current master we keep in multiple places copies of the > region states in the master and this copies include information like the HRI. > A problem that we have observed is when region replicas are being used and > there is a split, the region replica from parent doesn't get collected from > the region states and when the balancer tries to assign the old parent region > replica, this will cause the RegionServer to create a new HRI with the > details of the parent causing an inconstancy (see HBASE-18024). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HBASE-18399) Files in a snapshot can go missing even after the snapshot is taken successfully
[ https://issues.apache.org/jira/browse/HBASE-18399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashu Pachauri resolved HBASE-18399. --- Resolution: Duplicate Assignee: Ashu Pachauri Fix Version/s: (was: 1.3.2) Since this has been fixed as part of HBASE-18398, closing it as a duplicate. > Files in a snapshot can go missing even after the snapshot is taken > successfully > > > Key: HBASE-18399 > URL: https://issues.apache.org/jira/browse/HBASE-18399 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > > Files missing after the snapshot is taken (only applicable when the TTL for > the TimeToLiveHFileCleaner is small, like the default 5 mins) > * SnapshotManifest#addRegion visits store_file_A, but is yet to write it > to the manifest. > * store_file_A is marked as compacted away and HFileArchiver moves the > file to archive. > * HFileCleaner comes in and sees the store_file_A in archive. It adds the > file to the list of files that might need to be cleaned up. > * HFileCleaner's SnapshotHFileCleaner plugin is kicked in. > * SnapshotFileCache#getUnreferencedFiles also says that store_file_A is > unreferenced and should be cleaned up (It has not yet been written to the > manifest). > * SnapshotHFileCleaner is still going through rest of the files in > archive. > * store_file_A reference is created and written to snapshot manifest. > * Snapshot verification runs and sees the store_file_A is present in > archive, and thus the verification passes. > * Now, the SnapshotHFileCleaner finishes and TimeToLiveHFileCleaner is > triggered. If TTL has passed since the store_file_A was moved to archive > (SnapshotHFileCleaner could take easily several minutes to go through rest of > the files), the TimeToLiveHFileCleaner also marks the file as deletable. > * Since all cleaner plugins marked file as deletable, the store_file_A is > deleted. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
