[jira] [Commented] (HBASE-20908) Infinite loop on regionserver if region replica are reduced
[
https://issues.apache.org/jira/browse/HBASE-20908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551539#comment-16551539
]
Hudson commented on HBASE-20908:
Results for branch branch-2
[build #1007 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1007/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1007//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1007//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1007//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Infinite loop on regionserver if region replica are reduced
>
>
> Key: HBASE-20908
> URL: https://issues.apache.org/jira/browse/HBASE-20908
> Project: HBase
> Issue Type: Bug
> Components: read replicas
>Affects Versions: 1.2.0, 2.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: 20908_v3.patch, HBASE-20908.patch, HBASE-20908_v1.patch,
> HBASE-20908_v3-branch-1.patch, HBASE-20908_v3.patch
>
>
> Steps to reproduce
> {code}
> hbase(main):003:0> create 'myTable','cf',{REGION_REPLICATION=>3}
> hbase(main):003:0> put 'myTable','r1','cf:col1','1'
> 0 row(s) in 0.1230 seconds
> hbase(main):004:0> disable 'myTable'
> alter '0 row(s) in 2.3040 seconds
> hbase(main):005:0> alter 'myTable',{REGION_REPLICATION=>1}
> Updating all regions with the new schema...
> 1/1 regions updated.
> Done.
> 0 row(s) in 11.9550 seconds
> hbase(main):006:0> enable 'myTable'
> 0 row(s) in 1.2620 seconds
> hbase(main):007:0> put 'myTable1','r2','cf:col1','1'
> 0 row(s) in 0.0060 seconds
> {code}
> This is the replica region request which will not be present now in Meta but
> was there in cache. Server will say that he is not serving this region.
> {code}
> com.google.protobuf.ServiceException:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.NotServingRegionException):
> org.apache.hadoop.hbase.NotServingRegionException: Region
> d997d9b47a106216b9b117617ec09015 is not online on
> 10.22.9.76,16020,1531341039091
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3124)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3106)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.replay(RSRpcServices.java:1714)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22773)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
> {code}
> Eventually, when we will update our cache after looking into meta , we will
> get into an infinite loop as this event will not be replicated because the
> location of the replica will not appear again.
> {code}
> java.net.SocketTimeoutException: callTimeout=120, callDuration=2181316:
> Can't get the location null
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:170)
> at
> org.apache.hadoop.hbase.replication.regionserver.RegionReplicaReplicationEndpoint$RetryingRpcCallable.call(RegionReplicaReplicationEndpoint.java:606)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't
> get the location
> at
> org.apache.hadoop.hbase.client.RegionAdminServiceCallable.getRegionLocations(RegionAdminServiceCallable.java:178)
> at
> org.apache.hadoop.hbase.client.RegionAdminServiceCallable.getLocation(RegionAdminServiceCallable.java:105)
> at
> org.apache.hadoop.hbase.client.RegionAdminServiceCallable.prepare(RegionAdminServiceCallable.java:89)
> at
>
[jira] [Updated] (HBASE-20598) Upgrade to JRuby 9.2
[ https://issues.apache.org/jira/browse/HBASE-20598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jack Bearden updated HBASE-20598: - Attachment: HBASE-20598.001.patch Status: Patch Available (was: Open) Hey folks! I hope you don't mind that I ignorantly kick the tires here with patch 001. I'm curious to see the test report from Yetus > Upgrade to JRuby 9.2 > > > Key: HBASE-20598 > URL: https://issues.apache.org/jira/browse/HBASE-20598 > Project: HBase > Issue Type: Bug > Components: dependencies, shell >Reporter: Josh Elser >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20598.001.patch > > > [~mdrob] pointed out that there's a JRuby 9.2 release. We should see if we > can get ourselves onto that from our current 9.1 release line. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20559) Backport HBASE-18083 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-20559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551532#comment-16551532 ] Tak Lon (Stephen) Wu commented on HBASE-20559: -- fixed checkstyle in second patch. The problem was that one line had a extra semicolon lol, and I found the original HBASE-18083 and [commit|https://github.com/apache/hbase/commit/4fe73857679ecba89a7edd3c17d9f92e4c0e2164#diff-1364a76d149e28ad87911c3941da17a6R280] also had it. > Backport HBASE-18083 to branch-1 > > > Key: HBASE-20559 > URL: https://issues.apache.org/jira/browse/HBASE-20559 > Project: HBase > Issue Type: Sub-task > Components: HFile >Affects Versions: 1.4.4, 1.4.5 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Major > Attachments: HBASE-20559.branch-1.001.patch, > HBASE-20559.branch-1.002.patch > > > As part of HBASE-20555, HBASE-18083 is the fourth/last patch that is needed > for backporting. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20559) Backport HBASE-18083 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-20559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak Lon (Stephen) Wu updated HBASE-20559: - Attachment: HBASE-20559.branch-1.002.patch > Backport HBASE-18083 to branch-1 > > > Key: HBASE-20559 > URL: https://issues.apache.org/jira/browse/HBASE-20559 > Project: HBase > Issue Type: Sub-task > Components: HFile >Affects Versions: 1.4.4, 1.4.5 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Major > Attachments: HBASE-20559.branch-1.001.patch, > HBASE-20559.branch-1.002.patch > > > As part of HBASE-20555, HBASE-18083 is the fourth/last patch that is needed > for backporting. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20914) Trim Master memory usage
[
https://issues.apache.org/jira/browse/HBASE-20914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551528#comment-16551528
]
Hudson commented on HBASE-20914:
Results for branch branch-2.0
[build #573 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/573/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/573//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/573//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/573//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Trim Master memory usage
>
>
> Key: HBASE-20914
> URL: https://issues.apache.org/jira/browse/HBASE-20914
> Project: HBase
> Issue Type: Sub-task
> Components: master
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 2.0.2, 2.2.0, 2.1.1
>
> Attachments: HBASE-20914.branch-2.0.001.patch, Screen Shot 2018-07-19
> at 1.20.23 PM.png, Screen Shot 2018-07-19 at 1.20.23 PM.png, Screen Shot
> 2018-07-19 at 1.38.56 PM.png, Screen Shot 2018-07-19 at 1.38.56 PM.png,
> Screen Shot 2018-07-19 at 2.22.42 PM.png, Screen Shot 2018-07-19 at 2.22.42
> PM.png, Screen Shot 2018-07-19 at 2.43.42 PM.png, Screen Shot 2018-07-19 at
> 2.43.42 PM.png
>
>
> While working on the parent issue, looking at a heap from a Master tha was
> running ~650 servers and > 300k regions, I tripped over some silly items in
> the heap:
> 1. Balancer has a regions x server matrix which takes up 18% of the Master
> heap according to jxray and 40% according to eclipse. Looks like the matrix
> should be regions x racks which would be much smaller (Issue came in with
> HBASE-18164 Fast locality computation in balancer -Added new
> LocalityCostFunction and LocalityCandidateGenerator ..). See !Screen Shot
> 2018-07-19 at 1.20.23 PM.png! !Screen Shot 2018-07-19 at 1.38.56 PM.png!
> 2. Duplicate Strings make up ~5% of the Master heap. Of these, ServerName
> seems to be the font. Interesting is report that there 54k instances of
> ServerName in this heap though there are only 650 Servers. See !Screen Shot
> 2018-07-19 at 2.22.42 PM.png!
> 3. ArrayDequeue initializes its internal elements array with 16 elements. We
> use this in a few places. In Procedures, of which there are many in this
> heap, we near never make use of this array. See !Screen Shot 2018-07-19 at
> 2.43.42 PM.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20893) Data loss if splitting region while ServerCrashProcedure executing
[
https://issues.apache.org/jira/browse/HBASE-20893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551525#comment-16551525
]
Hadoop QA commented on HBASE-20893:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 3m
34s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.0 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
41s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
26s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
19s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
53s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
10s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} branch-2.0 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 11m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
52s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 54s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
1m 5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
30s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}147m
12s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}215m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:6f01af0 |
| JIRA Issue | HBASE-20893 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12932522/HBASE-20893.branch-2.0.004.patch
|
| Optional Tests | asflicense cc unit hbaseprotoc javac javadoc findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 28d67fbec402 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14
08:53:28 UTC 2018 x86_64 GNU/Linux |
| Build
[jira] [Commented] (HBASE-19893) restore_snapshot is broken in master branch when region splits
[
https://issues.apache.org/jira/browse/HBASE-19893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551503#comment-16551503
]
Ted Yu commented on HBASE-19893:
Sounds good.
> restore_snapshot is broken in master branch when region splits
> --
>
> Key: HBASE-19893
> URL: https://issues.apache.org/jira/browse/HBASE-19893
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Critical
> Attachments: 19893.master.004.patch, 19893.master.004.patch,
> 19893.master.004.patch, HBASE-19893.master.001.patch,
> HBASE-19893.master.002.patch, HBASE-19893.master.003.patch,
> HBASE-19893.master.003.patch, HBASE-19893.master.004.patch,
> HBASE-19893.master.005.patch, HBASE-19893.master.005.patch,
> HBASE-19893.master.005.patch,
> org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas-output.txt
>
>
> When I was investigating HBASE-19850, I found restore_snapshot didn't work in
> master branch.
>
> Steps to reproduce are as follows:
> 1. Create a table
> {code:java}
> create "test", "cf"
> {code}
> 2. Load data (2000 rows) to the table
> {code:java}
> (0...2000).each{|i| put "test", "row#{i}", "cf:col", "val"}
> {code}
> 3. Split the table
> {code:java}
> split "test"
> {code}
> 4. Take a snapshot
> {code:java}
> snapshot "test", "snap"
> {code}
> 5. Load more data (2000 rows) to the table and split the table agin
> {code:java}
> (2000...4000).each{|i| put "test", "row#{i}", "cf:col", "val"}
> split "test"
> {code}
> 6. Restore the table from the snapshot
> {code:java}
> disable "test"
> restore_snapshot "snap"
> enable "test"
> {code}
> 7. Scan the table
> {code:java}
> scan "test"
> {code}
> However, this scan returns only 244 rows (it should return 2000 rows) like
> the following:
> {code:java}
> hbase(main):038:0> scan "test"
> ROW COLUMN+CELL
> row78 column=cf:col, timestamp=1517298307049, value=val
>
> row999 column=cf:col, timestamp=1517298307608, value=val
> 244 row(s)
> Took 0.1500 seconds
> {code}
>
> Also, the restored table should have 2 online regions but it has 3 online
> regions.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19893) restore_snapshot is broken in master branch when region splits
[
https://issues.apache.org/jira/browse/HBASE-19893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551501#comment-16551501
]
Toshihiro Suzuki commented on HBASE-19893:
--
[~yuzhih...@gmail.com]
{quote}
The above code occurs twice in deleteRegionsFromInMemoryStates.
Can you put them in private method to reduce code duplication ?
{quote}
The code actually isn't twice in deleteRegionsFromInMemoryStates(). The second
part uses deleteRegion(), not deleteRegions().
{quote}
Why do you need the new way of retrieving regionInfo ?
{quote}
We need it because we should restore the regionInfo from the snapshot manifests
file. The exiting regionInfo instance is from the current state from HDFS. It
can be different from a restored state we want.
{quote}
The above method is only used locally - it doesn't have to be public.
{quote}
Yes, I can modify it.
If you agree with the above, I'll attatch a new patch.
Thanks.
> restore_snapshot is broken in master branch when region splits
> --
>
> Key: HBASE-19893
> URL: https://issues.apache.org/jira/browse/HBASE-19893
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Critical
> Attachments: 19893.master.004.patch, 19893.master.004.patch,
> 19893.master.004.patch, HBASE-19893.master.001.patch,
> HBASE-19893.master.002.patch, HBASE-19893.master.003.patch,
> HBASE-19893.master.003.patch, HBASE-19893.master.004.patch,
> HBASE-19893.master.005.patch, HBASE-19893.master.005.patch,
> HBASE-19893.master.005.patch,
> org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas-output.txt
>
>
> When I was investigating HBASE-19850, I found restore_snapshot didn't work in
> master branch.
>
> Steps to reproduce are as follows:
> 1. Create a table
> {code:java}
> create "test", "cf"
> {code}
> 2. Load data (2000 rows) to the table
> {code:java}
> (0...2000).each{|i| put "test", "row#{i}", "cf:col", "val"}
> {code}
> 3. Split the table
> {code:java}
> split "test"
> {code}
> 4. Take a snapshot
> {code:java}
> snapshot "test", "snap"
> {code}
> 5. Load more data (2000 rows) to the table and split the table agin
> {code:java}
> (2000...4000).each{|i| put "test", "row#{i}", "cf:col", "val"}
> split "test"
> {code}
> 6. Restore the table from the snapshot
> {code:java}
> disable "test"
> restore_snapshot "snap"
> enable "test"
> {code}
> 7. Scan the table
> {code:java}
> scan "test"
> {code}
> However, this scan returns only 244 rows (it should return 2000 rows) like
> the following:
> {code:java}
> hbase(main):038:0> scan "test"
> ROW COLUMN+CELL
> row78 column=cf:col, timestamp=1517298307049, value=val
>
> row999 column=cf:col, timestamp=1517298307608, value=val
> 244 row(s)
> Took 0.1500 seconds
> {code}
>
> Also, the restored table should have 2 online regions but it has 3 online
> regions.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-19893) restore_snapshot is broken in master branch when region splits
[
https://issues.apache.org/jira/browse/HBASE-19893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551491#comment-16551491
]
Toshihiro Suzuki edited comment on HBASE-19893 at 7/21/18 2:00 AM:
---
Thank you for reviewing [~yuzhih...@gmail.com]. I'll fix the patch and attatch
a new patch.
{quote}
BTW you used double negation in your above comment - probably not what you
intended.
{quote}
Yeah, I wanted to say: "I don't think the test failure of
TestDLSFSHLog.testRecoveredEdits() is related to the patch."
was (Author: brfrn169):
Thank you for reviewing [~yuzhih...@gmail.com]. I'll fix the patch and attache
a new patch.
{quote}
BTW you used double negation in your above comment - probably not what you
intended.
{quote}
Yeah, I wanted to say: "I don't think the test failure of
TestDLSFSHLog.testRecoveredEdits() is related to the patch."
> restore_snapshot is broken in master branch when region splits
> --
>
> Key: HBASE-19893
> URL: https://issues.apache.org/jira/browse/HBASE-19893
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Critical
> Attachments: 19893.master.004.patch, 19893.master.004.patch,
> 19893.master.004.patch, HBASE-19893.master.001.patch,
> HBASE-19893.master.002.patch, HBASE-19893.master.003.patch,
> HBASE-19893.master.003.patch, HBASE-19893.master.004.patch,
> HBASE-19893.master.005.patch, HBASE-19893.master.005.patch,
> HBASE-19893.master.005.patch,
> org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas-output.txt
>
>
> When I was investigating HBASE-19850, I found restore_snapshot didn't work in
> master branch.
>
> Steps to reproduce are as follows:
> 1. Create a table
> {code:java}
> create "test", "cf"
> {code}
> 2. Load data (2000 rows) to the table
> {code:java}
> (0...2000).each{|i| put "test", "row#{i}", "cf:col", "val"}
> {code}
> 3. Split the table
> {code:java}
> split "test"
> {code}
> 4. Take a snapshot
> {code:java}
> snapshot "test", "snap"
> {code}
> 5. Load more data (2000 rows) to the table and split the table agin
> {code:java}
> (2000...4000).each{|i| put "test", "row#{i}", "cf:col", "val"}
> split "test"
> {code}
> 6. Restore the table from the snapshot
> {code:java}
> disable "test"
> restore_snapshot "snap"
> enable "test"
> {code}
> 7. Scan the table
> {code:java}
> scan "test"
> {code}
> However, this scan returns only 244 rows (it should return 2000 rows) like
> the following:
> {code:java}
> hbase(main):038:0> scan "test"
> ROW COLUMN+CELL
> row78 column=cf:col, timestamp=1517298307049, value=val
>
> row999 column=cf:col, timestamp=1517298307608, value=val
> 244 row(s)
> Took 0.1500 seconds
> {code}
>
> Also, the restored table should have 2 online regions but it has 3 online
> regions.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20559) Backport HBASE-18083 to branch-1
[
https://issues.apache.org/jira/browse/HBASE-20559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551492#comment-16551492
]
Hadoop QA commented on HBASE-20559:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
45s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
24s{color} | {color:red} hbase-server: The patch generated 1 new + 5 unchanged
- 0 fixed = 6 total (was 5) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
49s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 41s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}104m 25s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}123m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:61288f8 |
| JIRA Issue | HBASE-20559 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12932519/HBASE-20559.branch-1.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 988aa3694ca1 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
|
[jira] [Commented] (HBASE-19893) restore_snapshot is broken in master branch when region splits
[
https://issues.apache.org/jira/browse/HBASE-19893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551491#comment-16551491
]
Toshihiro Suzuki commented on HBASE-19893:
--
Thank you for reviewing [~yuzhih...@gmail.com]. I'll fix the patch and attache
a new patch.
{quote}
BTW you used double negation in your above comment - probably not what you
intended.
{quote}
Yeah, I wanted to say: "I don't think the test failure of
TestDLSFSHLog.testRecoveredEdits() is related to the patch."
> restore_snapshot is broken in master branch when region splits
> --
>
> Key: HBASE-19893
> URL: https://issues.apache.org/jira/browse/HBASE-19893
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Critical
> Attachments: 19893.master.004.patch, 19893.master.004.patch,
> 19893.master.004.patch, HBASE-19893.master.001.patch,
> HBASE-19893.master.002.patch, HBASE-19893.master.003.patch,
> HBASE-19893.master.003.patch, HBASE-19893.master.004.patch,
> HBASE-19893.master.005.patch, HBASE-19893.master.005.patch,
> HBASE-19893.master.005.patch,
> org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas-output.txt
>
>
> When I was investigating HBASE-19850, I found restore_snapshot didn't work in
> master branch.
>
> Steps to reproduce are as follows:
> 1. Create a table
> {code:java}
> create "test", "cf"
> {code}
> 2. Load data (2000 rows) to the table
> {code:java}
> (0...2000).each{|i| put "test", "row#{i}", "cf:col", "val"}
> {code}
> 3. Split the table
> {code:java}
> split "test"
> {code}
> 4. Take a snapshot
> {code:java}
> snapshot "test", "snap"
> {code}
> 5. Load more data (2000 rows) to the table and split the table agin
> {code:java}
> (2000...4000).each{|i| put "test", "row#{i}", "cf:col", "val"}
> split "test"
> {code}
> 6. Restore the table from the snapshot
> {code:java}
> disable "test"
> restore_snapshot "snap"
> enable "test"
> {code}
> 7. Scan the table
> {code:java}
> scan "test"
> {code}
> However, this scan returns only 244 rows (it should return 2000 rows) like
> the following:
> {code:java}
> hbase(main):038:0> scan "test"
> ROW COLUMN+CELL
> row78 column=cf:col, timestamp=1517298307049, value=val
>
> row999 column=cf:col, timestamp=1517298307608, value=val
> 244 row(s)
> Took 0.1500 seconds
> {code}
>
> Also, the restored table should have 2 online regions but it has 3 online
> regions.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20827) Add pause when retrying after CallQueueTooBigException for reportRegionStateTransition
[
https://issues.apache.org/jira/browse/HBASE-20827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551482#comment-16551482
]
Ted Yu commented on HBASE-20827:
After addressing the one checkstyle warning, should be good to go.
> Add pause when retrying after CallQueueTooBigException for
> reportRegionStateTransition
> ---
>
> Key: HBASE-20827
> URL: https://issues.apache.org/jira/browse/HBASE-20827
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HBASE-20827.patch
>
>
> We should be adding a pause during retry of CallQueueTooBigException to avoid
> flooding master.
> {code}
> org.apache.hadoop.hbase.CallQueueTooBigException: Call queue is full on
> ctr-e138-1518143905142-387782-01-03.hwx.site,2,1530317245246, too
> many items queued ?
> at sun.reflect.GeneratedConstructorAccessor108.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:100)
> at
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:90)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:359)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:336)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportRegionStateTransition(HRegionServer.java:2259)
> at
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:120)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.CallQueueTooBigException):
> Call queue is full on
> ctr-e138-1518143905142-387782-01-03.hwx.site,2,1530317245246, too
> many items queued ?
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:387)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406)
> at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103)
> at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:161)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:191)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19893) restore_snapshot is broken in master branch when region splits
[
https://issues.apache.org/jira/browse/HBASE-19893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551481#comment-16551481
]
Ted Yu commented on HBASE-19893:
{code}
+env.getAssignmentManager().getRegionStates().deleteRegions(regionInfos);
+env.getMasterServices().getServerManager().removeRegions(regionInfos);
+if (fnm != null) {
+ fnm.deleteFavoredNodesForRegions(regionInfos);
+}
{code}
The above code occurs twice in {{deleteRegionsFromInMemoryStates}}.
Can you put them in private method to reduce code duplication ?
{code}
- metaChanges.addRegionToRestore(regionInfo);
+
metaChanges.addRegionToRestore(ProtobufUtil.toRegionInfo(regionManifests.get(regionName)
+ .getRegionInfo()));
{code}
Why do you need the new way of retrieving regionInfo ?
{code}
+ public List getRegions(TableName tableName) throws IOException {
{code}
The above method is only used locally - it doesn't have to be public.
BTW you used double negation in your above comment - probably not what you
intended.
> restore_snapshot is broken in master branch when region splits
> --
>
> Key: HBASE-19893
> URL: https://issues.apache.org/jira/browse/HBASE-19893
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Critical
> Attachments: 19893.master.004.patch, 19893.master.004.patch,
> 19893.master.004.patch, HBASE-19893.master.001.patch,
> HBASE-19893.master.002.patch, HBASE-19893.master.003.patch,
> HBASE-19893.master.003.patch, HBASE-19893.master.004.patch,
> HBASE-19893.master.005.patch, HBASE-19893.master.005.patch,
> HBASE-19893.master.005.patch,
> org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas-output.txt
>
>
> When I was investigating HBASE-19850, I found restore_snapshot didn't work in
> master branch.
>
> Steps to reproduce are as follows:
> 1. Create a table
> {code:java}
> create "test", "cf"
> {code}
> 2. Load data (2000 rows) to the table
> {code:java}
> (0...2000).each{|i| put "test", "row#{i}", "cf:col", "val"}
> {code}
> 3. Split the table
> {code:java}
> split "test"
> {code}
> 4. Take a snapshot
> {code:java}
> snapshot "test", "snap"
> {code}
> 5. Load more data (2000 rows) to the table and split the table agin
> {code:java}
> (2000...4000).each{|i| put "test", "row#{i}", "cf:col", "val"}
> split "test"
> {code}
> 6. Restore the table from the snapshot
> {code:java}
> disable "test"
> restore_snapshot "snap"
> enable "test"
> {code}
> 7. Scan the table
> {code:java}
> scan "test"
> {code}
> However, this scan returns only 244 rows (it should return 2000 rows) like
> the following:
> {code:java}
> hbase(main):038:0> scan "test"
> ROW COLUMN+CELL
> row78 column=cf:col, timestamp=1517298307049, value=val
>
> row999 column=cf:col, timestamp=1517298307608, value=val
> 244 row(s)
> Took 0.1500 seconds
> {code}
>
> Also, the restored table should have 2 online regions but it has 3 online
> regions.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20827) Add pause when retrying after CallQueueTooBigException for reportRegionStateTransition
[
https://issues.apache.org/jira/browse/HBASE-20827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551475#comment-16551475
]
Hadoop QA commented on HBASE-20827:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
59s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
0s{color} | {color:red} hbase-server: The patch generated 1 new + 75 unchanged
- 0 fixed = 76 total (was 75) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
0s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 54s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}113m
5s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}152m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20827 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12929832/HBASE-20827.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux c0eaf7d0d79d 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / eb906e20ee |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/13705/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13705/testReport/ |
| Max. process+thread count |
[jira] [Updated] (HBASE-20893) Data loss if splitting region while ServerCrashProcedure executing
[ https://issues.apache.org/jira/browse/HBASE-20893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allan Yang updated HBASE-20893: --- Attachment: HBASE-20893.branch-2.0.004.patch > Data loss if splitting region while ServerCrashProcedure executing > -- > > Key: HBASE-20893 > URL: https://issues.apache.org/jira/browse/HBASE-20893 > Project: HBase > Issue Type: Sub-task >Affects Versions: 3.0.0, 2.1.0, 2.0.1 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Attachments: HBASE-20893.branch-2.0.001.patch, > HBASE-20893.branch-2.0.002.patch, HBASE-20893.branch-2.0.003.patch, > HBASE-20893.branch-2.0.004.patch > > > Similar case as HBASE-20878. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20558) Backport HBASE-17854 to branch-1
[
https://issues.apache.org/jira/browse/HBASE-20558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551466#comment-16551466
]
Hadoop QA commented on HBASE-20558:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
52s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
37s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} hbase-server: The patch generated 0 new + 5
unchanged - 5 fixed = 5 total (was 10) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
38s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 34s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 98m 18s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}116m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:61288f8 |
| JIRA Issue | HBASE-20558 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12930929/HBASE-20558.branch-1.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 736eb26b0356 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64
[jira] [Commented] (HBASE-20908) Infinite loop on regionserver if region replica are reduced
[
https://issues.apache.org/jira/browse/HBASE-20908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551458#comment-16551458
]
Hadoop QA commented on HBASE-20908:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m
41s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
40s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
25s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
15s{color} | {color:red} hbase-server: The patch generated 4 new + 3 unchanged
- 5 fixed = 7 total (was 8) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
24s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 30s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}135m 34s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}171m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:61288f8 |
| JIRA Issue | HBASE-20908 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12932501/HBASE-20908_v3-branch-1.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux c8b40ddc6453 4.4.0-130-generic
[jira] [Commented] (HBASE-19893) restore_snapshot is broken in master branch when region splits
[
https://issues.apache.org/jira/browse/HBASE-19893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551454#comment-16551454
]
Toshihiro Suzuki commented on HBASE-19893:
--
I don't think the test failure of TestDLSFSHLog. testRecoveredEdits() is not
related to the patch and I ran the test successfully in my local env. Could you
please take a look at the patch? [~yuzhih...@gmail.com] If you are ok, could
you please push it?
> restore_snapshot is broken in master branch when region splits
> --
>
> Key: HBASE-19893
> URL: https://issues.apache.org/jira/browse/HBASE-19893
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Critical
> Attachments: 19893.master.004.patch, 19893.master.004.patch,
> 19893.master.004.patch, HBASE-19893.master.001.patch,
> HBASE-19893.master.002.patch, HBASE-19893.master.003.patch,
> HBASE-19893.master.003.patch, HBASE-19893.master.004.patch,
> HBASE-19893.master.005.patch, HBASE-19893.master.005.patch,
> HBASE-19893.master.005.patch,
> org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas-output.txt
>
>
> When I was investigating HBASE-19850, I found restore_snapshot didn't work in
> master branch.
>
> Steps to reproduce are as follows:
> 1. Create a table
> {code:java}
> create "test", "cf"
> {code}
> 2. Load data (2000 rows) to the table
> {code:java}
> (0...2000).each{|i| put "test", "row#{i}", "cf:col", "val"}
> {code}
> 3. Split the table
> {code:java}
> split "test"
> {code}
> 4. Take a snapshot
> {code:java}
> snapshot "test", "snap"
> {code}
> 5. Load more data (2000 rows) to the table and split the table agin
> {code:java}
> (2000...4000).each{|i| put "test", "row#{i}", "cf:col", "val"}
> split "test"
> {code}
> 6. Restore the table from the snapshot
> {code:java}
> disable "test"
> restore_snapshot "snap"
> enable "test"
> {code}
> 7. Scan the table
> {code:java}
> scan "test"
> {code}
> However, this scan returns only 244 rows (it should return 2000 rows) like
> the following:
> {code:java}
> hbase(main):038:0> scan "test"
> ROW COLUMN+CELL
> row78 column=cf:col, timestamp=1517298307049, value=val
>
> row999 column=cf:col, timestamp=1517298307608, value=val
> 244 row(s)
> Took 0.1500 seconds
> {code}
>
> Also, the restored table should have 2 online regions but it has 3 online
> regions.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20914) Trim Master memory usage
[
https://issues.apache.org/jira/browse/HBASE-20914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551450#comment-16551450
]
Hudson commented on HBASE-20914:
Results for branch branch-2
[build #1006 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1006/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1006//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1006//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1006//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Trim Master memory usage
>
>
> Key: HBASE-20914
> URL: https://issues.apache.org/jira/browse/HBASE-20914
> Project: HBase
> Issue Type: Sub-task
> Components: master
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 2.0.2, 2.2.0, 2.1.1
>
> Attachments: HBASE-20914.branch-2.0.001.patch, Screen Shot 2018-07-19
> at 1.20.23 PM.png, Screen Shot 2018-07-19 at 1.20.23 PM.png, Screen Shot
> 2018-07-19 at 1.38.56 PM.png, Screen Shot 2018-07-19 at 1.38.56 PM.png,
> Screen Shot 2018-07-19 at 2.22.42 PM.png, Screen Shot 2018-07-19 at 2.22.42
> PM.png, Screen Shot 2018-07-19 at 2.43.42 PM.png, Screen Shot 2018-07-19 at
> 2.43.42 PM.png
>
>
> While working on the parent issue, looking at a heap from a Master tha was
> running ~650 servers and > 300k regions, I tripped over some silly items in
> the heap:
> 1. Balancer has a regions x server matrix which takes up 18% of the Master
> heap according to jxray and 40% according to eclipse. Looks like the matrix
> should be regions x racks which would be much smaller (Issue came in with
> HBASE-18164 Fast locality computation in balancer -Added new
> LocalityCostFunction and LocalityCandidateGenerator ..). See !Screen Shot
> 2018-07-19 at 1.20.23 PM.png! !Screen Shot 2018-07-19 at 1.38.56 PM.png!
> 2. Duplicate Strings make up ~5% of the Master heap. Of these, ServerName
> seems to be the font. Interesting is report that there 54k instances of
> ServerName in this heap though there are only 650 Servers. See !Screen Shot
> 2018-07-19 at 2.22.42 PM.png!
> 3. ArrayDequeue initializes its internal elements array with 16 elements. We
> use this in a few places. In Procedures, of which there are many in this
> heap, we near never make use of this array. See !Screen Shot 2018-07-19 at
> 2.43.42 PM.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20559) Backport HBASE-18083 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-20559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak Lon (Stephen) Wu updated HBASE-20559: - Description: As part of HBASE-20555, HBASE-18083 is the fourth/last patch that is needed for backporting. (was: As part of HBASE-20555, HBASE-18083 --is the fourth/last patch that is needed for backporting.--) > Backport HBASE-18083 to branch-1 > > > Key: HBASE-20559 > URL: https://issues.apache.org/jira/browse/HBASE-20559 > Project: HBase > Issue Type: Sub-task > Components: HFile >Affects Versions: 1.4.4, 1.4.5 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Major > Attachments: HBASE-20559.branch-1.001.patch > > > As part of HBASE-20555, HBASE-18083 is the fourth/last patch that is needed > for backporting. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20401) Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
[ https://issues.apache.org/jira/browse/HBASE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551433#comment-16551433 ] Tak Lon (Stephen) Wu edited comment on HBASE-20401 at 7/20/18 11:39 PM: [~reidchan] for branch-1 patch, can we wait till HBASE-20559 get in first? because the dynamic configurations checking need to be added in this branch-1's patch (then we don't need another JIRA, I think it could be done on Monday if [~zyork] reviews HBASE-20559 ;P ) meanwhile, for branch-2 and master, it's safe to commit. was (Author: taklwu): [~reidchan] for branch-1 patch, can we wait till HBASE-20559 get it first? because the dynamic configuration checking need to be added in this branch-1 patch (then we don't need another JIRA, I think it could be done on Monday if [~zyork] reviews HBASE-20559 ;P ) meanwhile, for branch-2 and master, it's safe to commit. > Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable > -- > > Key: HBASE-20401 > URL: https://issues.apache.org/jira/browse/HBASE-20401 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 3.0.0, 1.5.0, 2.0.0-beta-1, 1.4.4, 2.0.0 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Minor > Labels: beginner > Attachments: HBASE-20401.branch-1.001.patch, > HBASE-20401.master.001.patch, HBASE-20401.master.002.patch, > HBASE-20401.master.003.patch, HBASE-20401.master.004.patch, > HBASE-20401.master.005.patch, HBASE-20401.master.006.patch > > > When backporting HBASE-18309 in HBASE-20352, the deleteFiles calls > CleanerContext.java#getResult with a waitIfNotFinished timeout to wait for > notification (notify) from the fs.delete file thread. there might be two > situation need to tune the MAX_WAIT in CleanerContext or waitIfNotFinished > when LogClearner call getResult. > # fs.delete never complete (strange but possible), then we need to wait for > a max of 60 seconds. here, 60 seconds might be too long > # getResult is waiting in the period of 500 milliseconds, but the fs.delete > has completed and setFromClear is set but yet notify(). one might want to > tune this 500 milliseconds to 200 or less . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20559) Backport HBASE-18083 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-20559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551434#comment-16551434 ] Tak Lon (Stephen) Wu commented on HBASE-20559: -- [PR #84|https://github.com/apache/hbase/pull/84] created, included build, test and compat-check. Also, I comment on HBASE-20401 about waiting this patch get into branch-1 first. > Backport HBASE-18083 to branch-1 > > > Key: HBASE-20559 > URL: https://issues.apache.org/jira/browse/HBASE-20559 > Project: HBase > Issue Type: Sub-task > Components: HFile >Affects Versions: 1.4.4, 1.4.5 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Major > Attachments: HBASE-20559.branch-1.001.patch > > > As part of HBASE-20555, HBASE-18083 --is the fourth/last patch that is needed > for backporting.-- -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20401) Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
[ https://issues.apache.org/jira/browse/HBASE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551433#comment-16551433 ] Tak Lon (Stephen) Wu commented on HBASE-20401: -- [~reidchan] for branch-1 patch, can we wait till HBASE-20559 get it first? because the dynamic configuration checking need to be added in this branch-1 patch (then we don't need another JIRA, I think it could be done on Monday if [~zyork] reviews HBASE-20559 ;P ) meanwhile, for branch-2 and master, it's safe to commit. > Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable > -- > > Key: HBASE-20401 > URL: https://issues.apache.org/jira/browse/HBASE-20401 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 3.0.0, 1.5.0, 2.0.0-beta-1, 1.4.4, 2.0.0 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Minor > Labels: beginner > Attachments: HBASE-20401.branch-1.001.patch, > HBASE-20401.master.001.patch, HBASE-20401.master.002.patch, > HBASE-20401.master.003.patch, HBASE-20401.master.004.patch, > HBASE-20401.master.005.patch, HBASE-20401.master.006.patch > > > When backporting HBASE-18309 in HBASE-20352, the deleteFiles calls > CleanerContext.java#getResult with a waitIfNotFinished timeout to wait for > notification (notify) from the fs.delete file thread. there might be two > situation need to tune the MAX_WAIT in CleanerContext or waitIfNotFinished > when LogClearner call getResult. > # fs.delete never complete (strange but possible), then we need to wait for > a max of 60 seconds. here, 60 seconds might be too long > # getResult is waiting in the period of 500 milliseconds, but the fs.delete > has completed and setFromClear is set but yet notify(). one might want to > tune this 500 milliseconds to 200 or less . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20558) Backport HBASE-17854 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-20558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-20558: -- Resolution: Fixed Fix Version/s: 1.4.6 1.5.0 Status: Resolved (was: Patch Available) > Backport HBASE-17854 to branch-1 > > > Key: HBASE-20558 > URL: https://issues.apache.org/jira/browse/HBASE-20558 > Project: HBase > Issue Type: Sub-task > Components: HFile >Affects Versions: 1.4.4, 1.4.5 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Major > Fix For: 1.5.0, 1.4.6 > > Attachments: HBASE-20558.branch-1.001.patch, report.html > > > As part of HBASE-20555, HBASE-17854 is the third patch that is needed for > backporting HBASE-18083 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20559) Backport HBASE-18083 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-20559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak Lon (Stephen) Wu updated HBASE-20559: - Status: Patch Available (was: Open) > Backport HBASE-18083 to branch-1 > > > Key: HBASE-20559 > URL: https://issues.apache.org/jira/browse/HBASE-20559 > Project: HBase > Issue Type: Sub-task > Components: HFile >Affects Versions: 1.4.5, 1.4.4 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Major > Attachments: HBASE-20559.branch-1.001.patch > > > As part of HBASE-20555, HBASE-18083 --is the fourth/last patch that is needed > for backporting.-- -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20559) Backport HBASE-18083 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-20559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak Lon (Stephen) Wu updated HBASE-20559: - Attachment: HBASE-20559.branch-1.001.patch > Backport HBASE-18083 to branch-1 > > > Key: HBASE-20559 > URL: https://issues.apache.org/jira/browse/HBASE-20559 > Project: HBase > Issue Type: Sub-task > Components: HFile >Affects Versions: 1.4.4, 1.4.5 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Major > Attachments: HBASE-20559.branch-1.001.patch > > > As part of HBASE-20555, HBASE-18083 --is the fourth/last patch that is needed > for backporting.-- -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20558) Backport HBASE-17854 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-20558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551421#comment-16551421 ] Zach York commented on HBASE-20558: --- Ah! Perfect, thanks! > Backport HBASE-17854 to branch-1 > > > Key: HBASE-20558 > URL: https://issues.apache.org/jira/browse/HBASE-20558 > Project: HBase > Issue Type: Sub-task > Components: HFile >Affects Versions: 1.4.4, 1.4.5 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Major > Attachments: HBASE-20558.branch-1.001.patch, report.html > > > As part of HBASE-20555, HBASE-17854 is the third patch that is needed for > backporting HBASE-18083 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17885) Backport HBASE-15871 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-17885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551397#comment-16551397 ] Andrew Purtell commented on HBASE-17885: Let me try this out locally. > Backport HBASE-15871 to branch-1 > > > Key: HBASE-17885 > URL: https://issues.apache.org/jira/browse/HBASE-17885 > Project: HBase > Issue Type: Bug > Components: Scanners >Affects Versions: 1.3.1, 1.2.5, 1.1.8 >Reporter: ramkrishna.s.vasudevan >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 1.5.0 > > Attachments: HBASE-17885.branch-1.001.patch, > HBASE-17885.branch-1.001.patch, HBASE-17885.branch-1.002.patch > > > Will try to rebase the branch-1 patch at the earliest. Hope the fix versions > are correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20558) Backport HBASE-17854 to branch-1
[
https://issues.apache.org/jira/browse/HBASE-20558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551393#comment-16551393
]
Andrew Purtell commented on HBASE-20558:
[~zyork] Run the compat checker like this:
{noformat}
./dev-support/checkcompatibility.py \
--annotation org.apache.hadoop.hbase.classification.InterfaceAudience.Public \
--annotation
org.apache.hadoop.hbase.classification.InterfaceAudience.LimitedPrivate \
--include-file "hbase-*" \
\
{noformat}
to make sure things that aren't tagged Public or LP are filtered out. For what
remains, you have to review the compatibility guidelines in the book
(http://hbase.apache.org/book.html#hbase.versioning) and make a judgement.
In this particular report I don't see anything of concern.
> Backport HBASE-17854 to branch-1
>
>
> Key: HBASE-20558
> URL: https://issues.apache.org/jira/browse/HBASE-20558
> Project: HBase
> Issue Type: Sub-task
> Components: HFile
>Affects Versions: 1.4.4, 1.4.5
>Reporter: Tak Lon (Stephen) Wu
>Assignee: Tak Lon (Stephen) Wu
>Priority: Major
> Attachments: HBASE-20558.branch-1.001.patch, report.html
>
>
> As part of HBASE-20555, HBASE-17854 is the third patch that is needed for
> backporting HBASE-18083
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20558) Backport HBASE-17854 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-20558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551385#comment-16551385 ] Zach York commented on HBASE-20558: --- Note, this was done on branch-1. I was planning on doing the same for branch-1.4 > Backport HBASE-17854 to branch-1 > > > Key: HBASE-20558 > URL: https://issues.apache.org/jira/browse/HBASE-20558 > Project: HBase > Issue Type: Sub-task > Components: HFile >Affects Versions: 1.4.4, 1.4.5 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Major > Attachments: HBASE-20558.branch-1.001.patch, report.html > > > As part of HBASE-20555, HBASE-17854 is the third patch that is needed for > backporting HBASE-18083 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20558) Backport HBASE-17854 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-20558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-20558: -- Attachment: report.html > Backport HBASE-17854 to branch-1 > > > Key: HBASE-20558 > URL: https://issues.apache.org/jira/browse/HBASE-20558 > Project: HBase > Issue Type: Sub-task > Components: HFile >Affects Versions: 1.4.4, 1.4.5 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Major > Attachments: HBASE-20558.branch-1.001.patch, report.html > > > As part of HBASE-20555, HBASE-17854 is the third patch that is needed for > backporting HBASE-18083 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20558) Backport HBASE-17854 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-20558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551383#comment-16551383 ] Zach York commented on HBASE-20558: --- [~apurtell] I ran the compat-checker and it generated errors, most of them are renamed methods from the previous patches, but this also added a constructor which seems okay to add within a version. How do you judge whether something is okay? I'll attach the report.html > Backport HBASE-17854 to branch-1 > > > Key: HBASE-20558 > URL: https://issues.apache.org/jira/browse/HBASE-20558 > Project: HBase > Issue Type: Sub-task > Components: HFile >Affects Versions: 1.4.4, 1.4.5 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Major > Attachments: HBASE-20558.branch-1.001.patch, report.html > > > As part of HBASE-20555, HBASE-17854 is the third patch that is needed for > backporting HBASE-18083 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20910) Fix dev-support/submit-patch.py by opening file with 'b' mode
[
https://issues.apache.org/jira/browse/HBASE-20910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551381#comment-16551381
]
Ted Yu commented on HBASE-20910:
Thanks for the patch, Ankit
Thanks for the review, Mike
> Fix dev-support/submit-patch.py by opening file with 'b' mode
> -
>
> Key: HBASE-20910
> URL: https://issues.apache.org/jira/browse/HBASE-20910
> Project: HBase
> Issue Type: Bug
> Components: scripts
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HBASE-20910.patch, HBASE-20910_v1.patch
>
>
> {code:python}
> hbase$ python3.6 dev-support/submit-patch.py
> INFO:submit-patch: Active branch: master
> INFO:submit-patch: Using tracking branch as base branch
> INFO:submit-patch: Base branch: origin/master
> INFO:submit-patch: Patch directory: /Users/asinghal/patches
> INFO:submit-patch: Patch name: master.patch
> Traceback (most recent call last):
> File "dev-support/submit-patch.py", line 253, in
> f.write(diff.encode('utf8'))
> TypeError: write() argument must be str, not bytes
> {code}
> [~te...@apache.org], FYI
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20910) Fix dev-support/submit-patch.py by opening file with 'b' mode
[
https://issues.apache.org/jira/browse/HBASE-20910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20910:
---
Summary: Fix dev-support/submit-patch.py by opening file with 'b' mode
(was: Fix dev-support/submit-patch.py)
> Fix dev-support/submit-patch.py by opening file with 'b' mode
> -
>
> Key: HBASE-20910
> URL: https://issues.apache.org/jira/browse/HBASE-20910
> Project: HBase
> Issue Type: Bug
> Components: scripts
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HBASE-20910.patch, HBASE-20910_v1.patch
>
>
> {code:python}
> hbase$ python3.6 dev-support/submit-patch.py
> INFO:submit-patch: Active branch: master
> INFO:submit-patch: Using tracking branch as base branch
> INFO:submit-patch: Base branch: origin/master
> INFO:submit-patch: Patch directory: /Users/asinghal/patches
> INFO:submit-patch: Patch name: master.patch
> Traceback (most recent call last):
> File "dev-support/submit-patch.py", line 253, in
> f.write(diff.encode('utf8'))
> TypeError: write() argument must be str, not bytes
> {code}
> [~te...@apache.org], FYI
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20910) Fix dev-support/submit-patch.py
[
https://issues.apache.org/jira/browse/HBASE-20910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ankit Singhal updated HBASE-20910:
--
Attachment: HBASE-20910_v1.patch
> Fix dev-support/submit-patch.py
> ---
>
> Key: HBASE-20910
> URL: https://issues.apache.org/jira/browse/HBASE-20910
> Project: HBase
> Issue Type: Bug
> Components: scripts
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HBASE-20910.patch, HBASE-20910_v1.patch
>
>
> {code:python}
> hbase$ python3.6 dev-support/submit-patch.py
> INFO:submit-patch: Active branch: master
> INFO:submit-patch: Using tracking branch as base branch
> INFO:submit-patch: Base branch: origin/master
> INFO:submit-patch: Patch directory: /Users/asinghal/patches
> INFO:submit-patch: Patch name: master.patch
> Traceback (most recent call last):
> File "dev-support/submit-patch.py", line 253, in
> f.write(diff.encode('utf8'))
> TypeError: write() argument must be str, not bytes
> {code}
> [~te...@apache.org], FYI
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20914) Trim Master memory usage
[
https://issues.apache.org/jira/browse/HBASE-20914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551375#comment-16551375
]
Hudson commented on HBASE-20914:
Results for branch branch-2.1
[build #84 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/84/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/84//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/84//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/84//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Trim Master memory usage
>
>
> Key: HBASE-20914
> URL: https://issues.apache.org/jira/browse/HBASE-20914
> Project: HBase
> Issue Type: Sub-task
> Components: master
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 2.0.2, 2.2.0, 2.1.1
>
> Attachments: HBASE-20914.branch-2.0.001.patch, Screen Shot 2018-07-19
> at 1.20.23 PM.png, Screen Shot 2018-07-19 at 1.20.23 PM.png, Screen Shot
> 2018-07-19 at 1.38.56 PM.png, Screen Shot 2018-07-19 at 1.38.56 PM.png,
> Screen Shot 2018-07-19 at 2.22.42 PM.png, Screen Shot 2018-07-19 at 2.22.42
> PM.png, Screen Shot 2018-07-19 at 2.43.42 PM.png, Screen Shot 2018-07-19 at
> 2.43.42 PM.png
>
>
> While working on the parent issue, looking at a heap from a Master tha was
> running ~650 servers and > 300k regions, I tripped over some silly items in
> the heap:
> 1. Balancer has a regions x server matrix which takes up 18% of the Master
> heap according to jxray and 40% according to eclipse. Looks like the matrix
> should be regions x racks which would be much smaller (Issue came in with
> HBASE-18164 Fast locality computation in balancer -Added new
> LocalityCostFunction and LocalityCandidateGenerator ..). See !Screen Shot
> 2018-07-19 at 1.20.23 PM.png! !Screen Shot 2018-07-19 at 1.38.56 PM.png!
> 2. Duplicate Strings make up ~5% of the Master heap. Of these, ServerName
> seems to be the font. Interesting is report that there 54k instances of
> ServerName in this heap though there are only 650 Servers. See !Screen Shot
> 2018-07-19 at 2.22.42 PM.png!
> 3. ArrayDequeue initializes its internal elements array with 16 elements. We
> use this in a few places. In Procedures, of which there are many in this
> heap, we near never make use of this array. See !Screen Shot 2018-07-19 at
> 2.43.42 PM.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20911) correct Swtich/case indentation in formatter template for eclipse
[
https://issues.apache.org/jira/browse/HBASE-20911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ankit Singhal updated HBASE-20911:
--
Description:
Making it consistent with our checkstyle requirments.
{code}
**
{code}
> correct Swtich/case indentation in formatter template for eclipse
> -
>
> Key: HBASE-20911
> URL: https://issues.apache.org/jira/browse/HBASE-20911
> Project: HBase
> Issue Type: Bug
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HBASE-20911.patch, HBASE-20911_v1.patch
>
>
> Making it consistent with our checkstyle requirments.
> {code}
>
>
> **
>
>
>
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20911) correct Swtich/case indentation in formatter template for eclipse
[
https://issues.apache.org/jira/browse/HBASE-20911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551373#comment-16551373
]
Ankit Singhal commented on HBASE-20911:
---
Is the new indentation consistent with checkstyle ?
Yes, this is done to make it consistent with checkstyle otherwise eclipse user
has to manually update the formatter or format the code for switch/case.
{code}
{code}
Updated a new patch with license header.
> correct Swtich/case indentation in formatter template for eclipse
> -
>
> Key: HBASE-20911
> URL: https://issues.apache.org/jira/browse/HBASE-20911
> Project: HBase
> Issue Type: Bug
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HBASE-20911.patch, HBASE-20911_v1.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20911) correct Swtich/case indentation in formatter template for eclipse
[ https://issues.apache.org/jira/browse/HBASE-20911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Singhal updated HBASE-20911: -- Attachment: HBASE-20911_v1.patch > correct Swtich/case indentation in formatter template for eclipse > - > > Key: HBASE-20911 > URL: https://issues.apache.org/jira/browse/HBASE-20911 > Project: HBase > Issue Type: Bug >Reporter: Ankit Singhal >Assignee: Ankit Singhal >Priority: Major > Attachments: HBASE-20911.patch, HBASE-20911_v1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20911) correct Swtich/case indentation in formatter template for eclipse
[ https://issues.apache.org/jira/browse/HBASE-20911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551363#comment-16551363 ] Ted Yu commented on HBASE-20911: Is the new indentation consistent with checkstyle ? Please fill out description. > correct Swtich/case indentation in formatter template for eclipse > - > > Key: HBASE-20911 > URL: https://issues.apache.org/jira/browse/HBASE-20911 > Project: HBase > Issue Type: Bug >Reporter: Ankit Singhal >Assignee: Ankit Singhal >Priority: Major > Attachments: HBASE-20911.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20894) Move BucketCache from java serialization to protobuf
[
https://issues.apache.org/jira/browse/HBASE-20894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551358#comment-16551358
]
Ted Yu commented on HBASE-20894:
I was running related unit tests and got the following ?
{code}
testRetrieveFromFile[0: blockSize=8,192,
bucketSizes=null](org.apache.hadoop.hbase.io.hfile.bucket.TestBucketCache)
Time elapsed: 1.15 sec <<< ERROR!
org.apache.hbase.thirdparty.com.google.protobuf.UninitializedMessageException:
Message missing required fields: access_counter
at
org.apache.hadoop.hbase.io.hfile.bucket.TestBucketCache.testRetrieveFromFile(TestBucketCache.java:266)
testRetrieveFromFile[1: blockSize=16,384,
bucketSizes=[I@2ef5e5e3](org.apache.hadoop.hbase.io.hfile.bucket.TestBucketCache)
Time elapsed: 0.262 sec <<< ERROR!
org.apache.hbase.thirdparty.com.google.protobuf.UninitializedMessageException:
Message missing required fields: access_counter
at
org.apache.hadoop.hbase.io.hfile.bucket.TestBucketCache.testRetrieveFromFile(TestBucketCache.java:266)
{code}
Can you check whether access_counter field can be optional ?
Thanks
> Move BucketCache from java serialization to protobuf
>
>
> Key: HBASE-20894
> URL: https://issues.apache.org/jira/browse/HBASE-20894
> Project: HBase
> Issue Type: Task
> Components: BucketCache
>Affects Versions: 2.0.0
>Reporter: Mike Drob
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20894.WIP-2.patch, HBASE-20894.WIP.patch,
> HBASE-20894.master.001.patch
>
>
> We should use a better serialization format instead of Java Serialization for
> the BucketCache entry persistence.
> Suggested by Chris McCown, who does not appear to have a JIRA account.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20827) Add pause when retrying after CallQueueTooBigException for reportRegionStateTransition
[
https://issues.apache.org/jira/browse/HBASE-20827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20827:
---
Status: Patch Available (was: Open)
> Add pause when retrying after CallQueueTooBigException for
> reportRegionStateTransition
> ---
>
> Key: HBASE-20827
> URL: https://issues.apache.org/jira/browse/HBASE-20827
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HBASE-20827.patch
>
>
> We should be adding a pause during retry of CallQueueTooBigException to avoid
> flooding master.
> {code}
> org.apache.hadoop.hbase.CallQueueTooBigException: Call queue is full on
> ctr-e138-1518143905142-387782-01-03.hwx.site,2,1530317245246, too
> many items queued ?
> at sun.reflect.GeneratedConstructorAccessor108.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:100)
> at
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:90)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:359)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:336)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportRegionStateTransition(HRegionServer.java:2259)
> at
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:120)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.CallQueueTooBigException):
> Call queue is full on
> ctr-e138-1518143905142-387782-01-03.hwx.site,2,1530317245246, too
> many items queued ?
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:387)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406)
> at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103)
> at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:161)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:191)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20827) Add pause when retrying after CallQueueTooBigException for reportRegionStateTransition
[
https://issues.apache.org/jira/browse/HBASE-20827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551355#comment-16551355
]
Ted Yu commented on HBASE-20827:
Makes sense.
Let's see what QA says.
> Add pause when retrying after CallQueueTooBigException for
> reportRegionStateTransition
> ---
>
> Key: HBASE-20827
> URL: https://issues.apache.org/jira/browse/HBASE-20827
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HBASE-20827.patch
>
>
> We should be adding a pause during retry of CallQueueTooBigException to avoid
> flooding master.
> {code}
> org.apache.hadoop.hbase.CallQueueTooBigException: Call queue is full on
> ctr-e138-1518143905142-387782-01-03.hwx.site,2,1530317245246, too
> many items queued ?
> at sun.reflect.GeneratedConstructorAccessor108.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:100)
> at
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:90)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:359)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:336)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportRegionStateTransition(HRegionServer.java:2259)
> at
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:120)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.CallQueueTooBigException):
> Call queue is full on
> ctr-e138-1518143905142-387782-01-03.hwx.site,2,1530317245246, too
> many items queued ?
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:387)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406)
> at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103)
> at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:161)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:191)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18152) [AMv2] Corrupt Procedure WAL file; procedure data stored out of order
[
https://issues.apache.org/jira/browse/HBASE-18152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551344#comment-16551344
]
stack commented on HBASE-18152:
---
Saw this again. Here is current example:
{code}
2018-07-17 22:33:03,678 WARN [Thread-18] wal.ProcedureWALFormatReader: NOT
INCREASING! current=class_name:
"org.apache.hadoop.hbase.master.assignment.AssignProcedure"
parent_id: 3902
proc_id: 3906
submitted_time: 1531890545613
owner: "stack"
state: RUNNABLE
stack_id: 6
stack_id: 16
last_update: 1531890545980
state_message {
type_url: "type.googleapis.com/hbase.pb.AssignRegionStateData"
value:
"\b\002\022F\b\200\336\354\331\312,\022\'\n\adefault\022\034IntegrationTestBigLinkedList\032\004\230\344(~\"\b\234q\307\034q\307\034h(\\0008\000"
}
, candidate=class_name:
"org.apache.hadoop.hbase.master.assignment.AssignProcedure"
parent_id: 3902
proc_id: 3906
submitted_time: 1531890545613
owner: "stack"
state: RUNNABLE
stack_id: 6
last_update: 1531890545812
state_message {
type_url: "type.googleapis.com/hbase.pb.AssignRegionStateData"
value:
"\b\002\022F\b\200\336\354\331\312,\022\'\n\adefault\022\034IntegrationTestBigLinkedList\032\004\230\344(~\"\b\234q\307\034q\307\034h(\\0008\000"
}
...
{code}
We have written the entries backwards indeed.
Looking at the logging for this particular procedure, I see the below entries
which correspond to the last_update times:
{code}
2018-07-17 22:09:05,812 INFO [PEWorker-8] assignment.AssignProcedure: Starting
pid=3906, ppid=3902, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure
table=IntegrationTestBigLinkedList, region=26d8fa5172ffdf3e89319e4b73fbf164;
rit=OFFLINE, location=ve0538.halxg.cloudera.com,16020,1531890104902;
forceNewPlan=false, retain=true
2018-07-17 22:09:05,980 INFO [PEWorker-10]
assignment.RegionTransitionProcedure: Dispatch pid=3906, ppid=3902,
state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure
table=IntegrationTestBigLinkedList, region=26d8fa5172ffdf3e89319e4b73fbf164;
rit=OPENING, location=ve0536.halxg.cloudera.com,16020,1531889066114
{code}
Notice how they are on different worker threads. The procedures "execute" in
the correct order but the edits land in the store in another order, something
quiet possible given the distance between execute, post-execution cleanup, and
store.
> [AMv2] Corrupt Procedure WAL file; procedure data stored out of order
> -
>
> Key: HBASE-18152
> URL: https://issues.apache.org/jira/browse/HBASE-18152
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Affects Versions: 2.0.0
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HBASE-18152.master.001.patch,
> hbase-hbase-master-ctr-e138-1518143905142-221855-01-02.hwx.site.log.gz,
> pv2-0036.log, pv2-0047.log,
> reading_bad_wal.patch
>
>
> I've seen corruption from time-to-time testing. Its rare enough. Often we
> can get over it but sometimes we can't. It took me a while to capture an
> instance of corruption. Turns out we are write to the WAL out-of-order which
> undoes a basic tenet; that WAL content is ordered in line w/ execution.
> Below I'll post a corrupt WAL.
> Looking at the write-side, there is a lot going on. I'm not clear on how we
> could write out of order. Will try and get more insight. Meantime parking
> this issue here to fill data into.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20908) Infinite loop on regionserver if region replica are reduced
[
https://issues.apache.org/jira/browse/HBASE-20908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551343#comment-16551343
]
Ted Yu commented on HBASE-20908:
Ran TestRegionReplicaReplicationEndpoint with branch-1 patch which passed.
lgtm
> Infinite loop on regionserver if region replica are reduced
>
>
> Key: HBASE-20908
> URL: https://issues.apache.org/jira/browse/HBASE-20908
> Project: HBase
> Issue Type: Bug
> Components: read replicas
>Affects Versions: 1.2.0, 2.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: 20908_v3.patch, HBASE-20908.patch, HBASE-20908_v1.patch,
> HBASE-20908_v3-branch-1.patch, HBASE-20908_v3.patch
>
>
> Steps to reproduce
> {code}
> hbase(main):003:0> create 'myTable','cf',{REGION_REPLICATION=>3}
> hbase(main):003:0> put 'myTable','r1','cf:col1','1'
> 0 row(s) in 0.1230 seconds
> hbase(main):004:0> disable 'myTable'
> alter '0 row(s) in 2.3040 seconds
> hbase(main):005:0> alter 'myTable',{REGION_REPLICATION=>1}
> Updating all regions with the new schema...
> 1/1 regions updated.
> Done.
> 0 row(s) in 11.9550 seconds
> hbase(main):006:0> enable 'myTable'
> 0 row(s) in 1.2620 seconds
> hbase(main):007:0> put 'myTable1','r2','cf:col1','1'
> 0 row(s) in 0.0060 seconds
> {code}
> This is the replica region request which will not be present now in Meta but
> was there in cache. Server will say that he is not serving this region.
> {code}
> com.google.protobuf.ServiceException:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.NotServingRegionException):
> org.apache.hadoop.hbase.NotServingRegionException: Region
> d997d9b47a106216b9b117617ec09015 is not online on
> 10.22.9.76,16020,1531341039091
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3124)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3106)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.replay(RSRpcServices.java:1714)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22773)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
> {code}
> Eventually, when we will update our cache after looking into meta , we will
> get into an infinite loop as this event will not be replicated because the
> location of the replica will not appear again.
> {code}
> java.net.SocketTimeoutException: callTimeout=120, callDuration=2181316:
> Can't get the location null
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:170)
> at
> org.apache.hadoop.hbase.replication.regionserver.RegionReplicaReplicationEndpoint$RetryingRpcCallable.call(RegionReplicaReplicationEndpoint.java:606)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't
> get the location
> at
> org.apache.hadoop.hbase.client.RegionAdminServiceCallable.getRegionLocations(RegionAdminServiceCallable.java:178)
> at
> org.apache.hadoop.hbase.client.RegionAdminServiceCallable.getLocation(RegionAdminServiceCallable.java:105)
> at
> org.apache.hadoop.hbase.client.RegionAdminServiceCallable.prepare(RegionAdminServiceCallable.java:89)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
> ... 5 more
> Caused by: java.io.IOException: HRegionInfo was null in myTable,
> row=keyvalues={myTable,,1531262022075.f2b68622cfd5851023be29d5599db6c9./info:regioninfo/1531262022425/Put/vlen=41/seqid=0,
>
> myTable,,1531262022075.f2b68622cfd5851023be29d5599db6c9./info:seqnumDuringOpen/1531341209944/Put/vlen=8/seqid=0,
>
> myTable,,1531262022075.f2b68622cfd5851023be29d5599db6c9./info:server/1531341209944/Put/vlen=16/seqid=0,
>
> myTable,,1531262022075.f2b68622cfd5851023be29d5599db6c9./info:serverstartcode/1531341209944/Put/vlen=8/seqid=0}
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1289)
> at
>
[jira] [Commented] (HBASE-20555) Backport HBASE-18083 and related changes in branch-1
[ https://issues.apache.org/jira/browse/HBASE-20555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551341#comment-16551341 ] Andrew Purtell commented on HBASE-20555: Sounds good [~zyork], will wait until they are in > Backport HBASE-18083 and related changes in branch-1 > > > Key: HBASE-20555 > URL: https://issues.apache.org/jira/browse/HBASE-20555 > Project: HBase > Issue Type: Umbrella > Components: HFile, snapshots >Affects Versions: 1.4.4, 1.4.5 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Major > > This will be the umbrella JIRA for backporting HBASE-18083 of `Make > large/small file clean thread number configurable in HFileCleaner` from > HBase's branch-2 to HBase's branch-1 that will needs a total of 4 sub-tasks > that backport HBASE-16490, HBASE-17215, HBASE-17854 and then HBASE-18083 > The goal is to improve HFile cleaning performance that has been introduced in > branch-2 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20711) Save on a Cell iteration when writing
[ https://issues.apache.org/jira/browse/HBASE-20711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551315#comment-16551315 ] stack commented on HBASE-20711: --- That would be sweet [~mdrob]. Thanks. > Save on a Cell iteration when writing > - > > Key: HBASE-20711 > URL: https://issues.apache.org/jira/browse/HBASE-20711 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack >Priority: Minor > Attachments: HBASE-20711.branch-2.0.001.patch > > > This is a minor savings. We were doing a spin through all Cells on receipt > just to check their size when subsequently, we were doing an iteration of all > Cells to insert. It manifest as a little spike in perf output. This change > removes the purposed spin through Cells and just does size check as part of > the general Cell insert (perf spike no longer shows but the cost of the size > check still remains). > There is also a minor bug fix where on receipt we were using the Puts row > rather than the Cells row; client may have succeeded in submitting a Cell > that disagreed with the hosting Mutation and it would have been written as > something else altogether -- with the Puts row -- rather than being rejected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20908) Infinite loop on regionserver if region replica are reduced
[
https://issues.apache.org/jira/browse/HBASE-20908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ankit Singhal updated HBASE-20908:
--
Attachment: HBASE-20908_v3-branch-1.patch
> Infinite loop on regionserver if region replica are reduced
>
>
> Key: HBASE-20908
> URL: https://issues.apache.org/jira/browse/HBASE-20908
> Project: HBase
> Issue Type: Bug
> Components: read replicas
>Affects Versions: 1.2.0, 2.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: 20908_v3.patch, HBASE-20908.patch, HBASE-20908_v1.patch,
> HBASE-20908_v3-branch-1.patch, HBASE-20908_v3.patch
>
>
> Steps to reproduce
> {code}
> hbase(main):003:0> create 'myTable','cf',{REGION_REPLICATION=>3}
> hbase(main):003:0> put 'myTable','r1','cf:col1','1'
> 0 row(s) in 0.1230 seconds
> hbase(main):004:0> disable 'myTable'
> alter '0 row(s) in 2.3040 seconds
> hbase(main):005:0> alter 'myTable',{REGION_REPLICATION=>1}
> Updating all regions with the new schema...
> 1/1 regions updated.
> Done.
> 0 row(s) in 11.9550 seconds
> hbase(main):006:0> enable 'myTable'
> 0 row(s) in 1.2620 seconds
> hbase(main):007:0> put 'myTable1','r2','cf:col1','1'
> 0 row(s) in 0.0060 seconds
> {code}
> This is the replica region request which will not be present now in Meta but
> was there in cache. Server will say that he is not serving this region.
> {code}
> com.google.protobuf.ServiceException:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.NotServingRegionException):
> org.apache.hadoop.hbase.NotServingRegionException: Region
> d997d9b47a106216b9b117617ec09015 is not online on
> 10.22.9.76,16020,1531341039091
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3124)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3106)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.replay(RSRpcServices.java:1714)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22773)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
> {code}
> Eventually, when we will update our cache after looking into meta , we will
> get into an infinite loop as this event will not be replicated because the
> location of the replica will not appear again.
> {code}
> java.net.SocketTimeoutException: callTimeout=120, callDuration=2181316:
> Can't get the location null
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:170)
> at
> org.apache.hadoop.hbase.replication.regionserver.RegionReplicaReplicationEndpoint$RetryingRpcCallable.call(RegionReplicaReplicationEndpoint.java:606)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't
> get the location
> at
> org.apache.hadoop.hbase.client.RegionAdminServiceCallable.getRegionLocations(RegionAdminServiceCallable.java:178)
> at
> org.apache.hadoop.hbase.client.RegionAdminServiceCallable.getLocation(RegionAdminServiceCallable.java:105)
> at
> org.apache.hadoop.hbase.client.RegionAdminServiceCallable.prepare(RegionAdminServiceCallable.java:89)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
> ... 5 more
> Caused by: java.io.IOException: HRegionInfo was null in myTable,
> row=keyvalues={myTable,,1531262022075.f2b68622cfd5851023be29d5599db6c9./info:regioninfo/1531262022425/Put/vlen=41/seqid=0,
>
> myTable,,1531262022075.f2b68622cfd5851023be29d5599db6c9./info:seqnumDuringOpen/1531341209944/Put/vlen=8/seqid=0,
>
> myTable,,1531262022075.f2b68622cfd5851023be29d5599db6c9./info:server/1531341209944/Put/vlen=16/seqid=0,
>
> myTable,,1531262022075.f2b68622cfd5851023be29d5599db6c9./info:serverstartcode/1531341209944/Put/vlen=8/seqid=0}
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1289)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1179)
> at
>
[jira] [Commented] (HBASE-20555) Backport HBASE-18083 and related changes in branch-1
[ https://issues.apache.org/jira/browse/HBASE-20555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551268#comment-16551268 ] Zach York commented on HBASE-20555: --- [~apurtell] FYI, I'd like to get the remaining two backports in for 1.4.6 if possible since the first 2 are there. I plan to push #3 soon and will review #4 soon after that. We should be able to get this in today or Monday at the latest. > Backport HBASE-18083 and related changes in branch-1 > > > Key: HBASE-20555 > URL: https://issues.apache.org/jira/browse/HBASE-20555 > Project: HBase > Issue Type: Umbrella > Components: HFile, snapshots >Affects Versions: 1.4.4, 1.4.5 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Major > > This will be the umbrella JIRA for backporting HBASE-18083 of `Make > large/small file clean thread number configurable in HFileCleaner` from > HBase's branch-2 to HBase's branch-1 that will needs a total of 4 sub-tasks > that backport HBASE-16490, HBASE-17215, HBASE-17854 and then HBASE-18083 > The goal is to improve HFile cleaning performance that has been introduced in > branch-2 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20558) Backport HBASE-17854 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-20558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551260#comment-16551260 ] Zach York commented on HBASE-20558: --- +1 I will commit in an hour if no objections. > Backport HBASE-17854 to branch-1 > > > Key: HBASE-20558 > URL: https://issues.apache.org/jira/browse/HBASE-20558 > Project: HBase > Issue Type: Sub-task > Components: HFile >Affects Versions: 1.4.4, 1.4.5 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Major > Attachments: HBASE-20558.branch-1.001.patch > > > As part of HBASE-20555, HBASE-17854 is the third patch that is needed for > backporting HBASE-18083 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20711) Save on a Cell iteration when writing
[ https://issues.apache.org/jira/browse/HBASE-20711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551234#comment-16551234 ] Mike Drob commented on HBASE-20711: --- I may have time to pick it up next week. > Save on a Cell iteration when writing > - > > Key: HBASE-20711 > URL: https://issues.apache.org/jira/browse/HBASE-20711 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack >Priority: Minor > Attachments: HBASE-20711.branch-2.0.001.patch > > > This is a minor savings. We were doing a spin through all Cells on receipt > just to check their size when subsequently, we were doing an iteration of all > Cells to insert. It manifest as a little spike in perf output. This change > removes the purposed spin through Cells and just does size check as part of > the general Cell insert (perf spike no longer shows but the cost of the size > check still remains). > There is also a minor bug fix where on receipt we were using the Puts row > rather than the Cells row; client may have succeeded in submitting a Cell > that disagreed with the hosting Mutation and it would have been written as > something else altogether -- with the Puts row -- rather than being rejected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20711) Save on a Cell iteration when writing
[ https://issues.apache.org/jira/browse/HBASE-20711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551220#comment-16551220 ] stack commented on HBASE-20711: --- No. I should be though. You want to take it [~mdrob]? > Save on a Cell iteration when writing > - > > Key: HBASE-20711 > URL: https://issues.apache.org/jira/browse/HBASE-20711 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack >Priority: Minor > Attachments: HBASE-20711.branch-2.0.001.patch > > > This is a minor savings. We were doing a spin through all Cells on receipt > just to check their size when subsequently, we were doing an iteration of all > Cells to insert. It manifest as a little spike in perf output. This change > removes the purposed spin through Cells and just does size check as part of > the general Cell insert (perf spike no longer shows but the cost of the size > check still remains). > There is also a minor bug fix where on receipt we were using the Puts row > rather than the Cells row; client may have succeeded in submitting a Cell > that disagreed with the hosting Mutation and it would have been written as > something else altogether -- with the Puts row -- rather than being rejected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20711) Save on a Cell iteration when writing
[ https://issues.apache.org/jira/browse/HBASE-20711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551190#comment-16551190 ] Mike Drob commented on HBASE-20711: --- [~stack] - are you still working on this? > Save on a Cell iteration when writing > - > > Key: HBASE-20711 > URL: https://issues.apache.org/jira/browse/HBASE-20711 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack >Priority: Minor > Attachments: HBASE-20711.branch-2.0.001.patch > > > This is a minor savings. We were doing a spin through all Cells on receipt > just to check their size when subsequently, we were doing an iteration of all > Cells to insert. It manifest as a little spike in perf output. This change > removes the purposed spin through Cells and just does size check as part of > the general Cell insert (perf spike no longer shows but the cost of the size > check still remains). > There is also a minor bug fix where on receipt we were using the Puts row > rather than the Cells row; client may have succeeded in submitting a Cell > that disagreed with the hosting Mutation and it would have been written as > something else altogether -- with the Puts row -- rather than being rejected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20749) Upgrade our use of checkstyle to 8.6+
[ https://issues.apache.org/jira/browse/HBASE-20749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551187#comment-16551187 ] Mike Drob commented on HBASE-20749: --- Pushed to a branch so that we can compare the checkstyle output against nightly master. > Upgrade our use of checkstyle to 8.6+ > - > > Key: HBASE-20749 > URL: https://issues.apache.org/jira/browse/HBASE-20749 > Project: HBase > Issue Type: Improvement > Components: build, community >Reporter: Sean Busbey >Assignee: Mike Drob >Priority: Minor > Attachments: HBASE-20749.master.001.patch > > > We should upgrade our checkstyle version to 8.6 or later so we can use the > "match violation message to this regex" feature for suppression. That will > allow us to make sure we don't regress on HTrace v3 vs v4 APIs (came up in > HBASE-20332). > We're currently blocked on upgrading to 8.3+ by [checkstyle > #5279|https://github.com/checkstyle/checkstyle/issues/5279], a regression > that flags our use of both the "separate import groups" and "put static > imports over here" configs as an error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18477) Umbrella JIRA for HBase Read Replica clusters
[
https://issues.apache.org/jira/browse/HBASE-18477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551183#comment-16551183
]
Hudson commented on HBASE-18477:
Results for branch HBASE-18477
[build #269 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/269/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/269//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/269//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/269//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/269//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Umbrella JIRA for HBase Read Replica clusters
> -
>
> Key: HBASE-18477
> URL: https://issues.apache.org/jira/browse/HBASE-18477
> Project: HBase
> Issue Type: New Feature
>Reporter: Zach York
>Assignee: Zach York
>Priority: Major
> Attachments: HBase Read-Replica Clusters Scope doc.docx, HBase
> Read-Replica Clusters Scope doc.pdf, HBase Read-Replica Clusters Scope
> doc_v2.docx, HBase Read-Replica Clusters Scope doc_v2.pdf
>
>
> Recently, changes (such as HBASE-17437) have unblocked HBase to run with a
> root directory external to the cluster (such as in Amazon S3). This means
> that the data is stored outside of the cluster and can be accessible after
> the cluster has been terminated. One use case that is often asked about is
> pointing multiple clusters to one root directory (sharing the data) to have
> read resiliency in the case of a cluster failure.
>
> This JIRA is an umbrella JIRA to contain all the tasks necessary to create a
> read-replica HBase cluster that is pointed at the same root directory.
>
> This requires making the Read-Replica cluster Read-Only (no metadata
> operation or data operations).
> Separating the hbase:meta table for each cluster (Otherwise HBase gets
> confused with multiple clusters trying to update the meta table with their ip
> addresses)
> Adding refresh functionality for the meta table to ensure new metadata is
> picked up on the read replica cluster.
> Adding refresh functionality for HFiles for a given table to ensure new data
> is picked up on the read replica cluster.
>
> This can be used with any existing cluster that is backed by an external
> filesystem.
>
> Please note that this feature is still quite manual (with the potential for
> automation later).
>
> More information on this particular feature can be found here:
> https://aws.amazon.com/blogs/big-data/setting-up-read-replica-clusters-with-hbase-on-amazon-s3/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20908) Infinite loop on regionserver if region replica are reduced
[
https://issues.apache.org/jira/browse/HBASE-20908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551167#comment-16551167
]
Hadoop QA commented on HBASE-20908:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
37s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
30s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 9s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}115m
12s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}156m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20908 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12932438/20908_v3.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 46dace0aaf22 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 03e596c669 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13703/testReport/ |
| Max. process+thread count | (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13703/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Infinite loop on regionserver if region
[jira] [Commented] (HBASE-20894) Move BucketCache from java serialization to protobuf
[ https://issues.apache.org/jira/browse/HBASE-20894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551155#comment-16551155 ] Mike Drob commented on HBASE-20894: --- Conversion is almost done, still needs some more testing, a configuration option to enable backwards compatibility, and to finish up the deserializer mapping on reading from file which I don't completely understand. > Move BucketCache from java serialization to protobuf > > > Key: HBASE-20894 > URL: https://issues.apache.org/jira/browse/HBASE-20894 > Project: HBase > Issue Type: Task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: Mike Drob >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20894.WIP-2.patch, HBASE-20894.WIP.patch, > HBASE-20894.master.001.patch > > > We should use a better serialization format instead of Java Serialization for > the BucketCache entry persistence. > Suggested by Chris McCown, who does not appear to have a JIRA account. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20894) Move BucketCache from java serialization to protobuf
[ https://issues.apache.org/jira/browse/HBASE-20894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-20894: -- Attachment: HBASE-20894.master.001.patch > Move BucketCache from java serialization to protobuf > > > Key: HBASE-20894 > URL: https://issues.apache.org/jira/browse/HBASE-20894 > Project: HBase > Issue Type: Task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: Mike Drob >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20894.WIP-2.patch, HBASE-20894.WIP.patch, > HBASE-20894.master.001.patch > > > We should use a better serialization format instead of Java Serialization for > the BucketCache entry persistence. > Suggested by Chris McCown, who does not appear to have a JIRA account. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20893) Data loss if splitting region while ServerCrashProcedure executing
[
https://issues.apache.org/jira/browse/HBASE-20893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551106#comment-16551106
]
Hadoop QA commented on HBASE-20893:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.0 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
26s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
52s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
22s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
15s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
57s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} branch-2.0 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 11m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
2s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 32s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
1m 6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
30s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}192m 10s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
48s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}258m 55s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestAsyncRegionAdminApi2 |
| | hadoop.hbase.client.TestTableFavoredNodes |
| | hadoop.hbase.regionserver.wal.TestLogRolling |
| | hadoop.hbase.client.TestMultiRespectsLimits |
| | hadoop.hbase.tool.TestLoadIncrementalHFilesSplitRecovery |
| | hadoop.hbase.master.TestCatalogJanitorInMemoryStates |
| | hadoop.hbase.regionserver.TestEndToEndSplitTransaction |
| | hadoop.hbase.master.TestAssignmentListener |
| |
[jira] [Updated] (HBASE-20151) Bug with SingleColumnValueFilter and FamilyFilter
[
https://issues.apache.org/jira/browse/HBASE-20151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-20151:
---
Fix Version/s: (was: 2.1.1)
(was: 2.0.2)
(was: 1.4.6)
2.2.0
1.5.0
> Bug with SingleColumnValueFilter and FamilyFilter
> -
>
> Key: HBASE-20151
> URL: https://issues.apache.org/jira/browse/HBASE-20151
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.0.1, 1.4.5
> Environment: MacOS 10.13.3
> HBase 1.3.1
>Reporter: Steven Sadowski
>Assignee: Reid Chan
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-20151.master.001.patch,
> HBASE-20151.master.002.patch, HBASE-20151.master.003.patch,
> HBASE-20151.master.004.patch, HBASE-20151.master.004.patch,
> HBASE-20151.master.005.patch
>
>
> When running the following queries, the result is sometimes return correctly
> and other times incorrectly based on the qualifier queried.
> Setup:
> {code:java}
> create 'test', 'a', 'b'
> test = get_table 'test'
> test.put '1', 'a:1', nil
> test.put '1', 'a:10', nil
> test.put '1', 'b:2', nil
> {code}
>
> This query works fine when the SCVF's qualifier has length 1 (i.e. '1') :
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','1',=,'binary:',true,true) AND
> FamilyFilter(=,'binary:b') )"})
> ROW COLUMN+CELL
> 1column=b:2,
> timestamp=1520455888059, value=
> 1 row(s) in 0.0060 seconds
> {code}
>
> The query should return the same result when passed a qualifier of length 2
> (i.e. '10') :
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','10',=,'binary:',true,true) AND
> FamilyFilter(=,'binary:b') )"})
> ROW COLUMN+CELL
> 0 row(s) in 0.0110 seconds
> {code}
> However, in this case, it does not return any row (expected result would be
> to return the same result as the first query).
>
> Removing the family filter while the qualifier is '10' yields expected
> results:
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','10',=,'binary:',true,true) )"})
> ROW COLUMN+CELL
> 1column=a:1,
> timestamp=1520455887954, value=
> 1column=a:10,
> timestamp=1520455888024, value=
> 1column=b:2,
> timestamp=1520455888059, value=
> 1 row(s) in 0.0140 seconds
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20741) Split of a region with replicas creates all daughter regions and its replica in same server
[
https://issues.apache.org/jira/browse/HBASE-20741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551032#comment-16551032
]
Ted Yu edited comment on HBASE-20741 at 7/20/18 5:42 PM:
-
Thanks for the patch, Ram.
{code}
final RegionInfo hri =
RegionReplicaUtil.getRegionInfoForReplica(daughter_2_RI, i);
- procs[procsIdx++] =
env.getAssignmentManager().createAssignProcedure(hri, targetServer);
+ if (RegionReplicaUtil.isDefaultReplica(hri)) {
+procs[procsIdx++] =
env.getAssignmentManager().createAssignProcedure(hri, targetServer);
+ } else {
+serverIdx = serverIdx % (onlineServers.size());
+procs[procsIdx++] =
+env.getAssignmentManager().createAssignProcedure(hri,
onlineServers.get(serverIdx));
+serverIdx++;
+ }
{code}
Previously there was only two method calls for each iteration of the loop. Now
there are several lines of code which are similar for both daughter regions.
Please consider extracting such code into a method.
was (Author: yuzhih...@gmail.com):
Thanks for the patch, Ram.
{code}
final RegionInfo hri =
RegionReplicaUtil.getRegionInfoForReplica(daughter_2_RI, i);
- procs[procsIdx++] =
env.getAssignmentManager().createAssignProcedure(hri, targetServer);
+ if (RegionReplicaUtil.isDefaultReplica(hri)) {
+procs[procsIdx++] =
env.getAssignmentManager().createAssignProcedure(hri, targetServer);
+ } else {
+serverIdx = serverIdx % (onlineServers.size());
+procs[procsIdx++] =
+env.getAssignmentManager().createAssignProcedure(hri,
onlineServers.get(serverIdx));
+serverIdx++;
+ }
{code}
Previously there was only one call inside the if block. Now there are several
lines of code which are similar for both daughter regions.
Please consider extracting such code into a method.
> Split of a region with replicas creates all daughter regions and its replica
> in same server
> ---
>
> Key: HBASE-20741
> URL: https://issues.apache.org/jira/browse/HBASE-20741
> Project: HBase
> Issue Type: Bug
> Components: read replicas
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Major
> Fix For: 2.2.0
>
> Attachments: HBASE-20741.patch
>
>
> Generally it is better that the parent region when split creates the daughter
> region in the same target server.
> But for replicas also we do the same and all the replica regions are created
> in the same target server. We should ideally be doing a round robin and only
> the primary daughter region should be opened in the intended target server
> (where the parent was previously opened).
> [~huaxiang] FYI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20741) Split of a region with replicas creates all daughter regions and its replica in same server
[
https://issues.apache.org/jira/browse/HBASE-20741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551032#comment-16551032
]
Ted Yu commented on HBASE-20741:
Thanks for the patch, Ram.
{code}
final RegionInfo hri =
RegionReplicaUtil.getRegionInfoForReplica(daughter_2_RI, i);
- procs[procsIdx++] =
env.getAssignmentManager().createAssignProcedure(hri, targetServer);
+ if (RegionReplicaUtil.isDefaultReplica(hri)) {
+procs[procsIdx++] =
env.getAssignmentManager().createAssignProcedure(hri, targetServer);
+ } else {
+serverIdx = serverIdx % (onlineServers.size());
+procs[procsIdx++] =
+env.getAssignmentManager().createAssignProcedure(hri,
onlineServers.get(serverIdx));
+serverIdx++;
+ }
{code}
Previously there was only one call inside the if block. Now there are several
lines of code which are similar for both daughter regions.
Please consider extracting such code into a method.
> Split of a region with replicas creates all daughter regions and its replica
> in same server
> ---
>
> Key: HBASE-20741
> URL: https://issues.apache.org/jira/browse/HBASE-20741
> Project: HBase
> Issue Type: Bug
> Components: read replicas
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Major
> Fix For: 2.2.0
>
> Attachments: HBASE-20741.patch
>
>
> Generally it is better that the parent region when split creates the daughter
> region in the same target server.
> But for replicas also we do the same and all the replica regions are created
> in the same target server. We should ideally be doing a round robin and only
> the primary daughter region should be opened in the intended target server
> (where the parent was previously opened).
> [~huaxiang] FYI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20914) Trim Master memory usage
[ https://issues.apache.org/jira/browse/HBASE-20914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551013#comment-16551013 ] stack commented on HBASE-20914: --- Hmm... it looks like there are 300k instances of SN in this heap though only 650 actual instances. > Trim Master memory usage > > > Key: HBASE-20914 > URL: https://issues.apache.org/jira/browse/HBASE-20914 > Project: HBase > Issue Type: Sub-task > Components: master >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 3.0.0, 2.0.2, 2.2.0, 2.1.1 > > Attachments: HBASE-20914.branch-2.0.001.patch, Screen Shot 2018-07-19 > at 1.20.23 PM.png, Screen Shot 2018-07-19 at 1.20.23 PM.png, Screen Shot > 2018-07-19 at 1.38.56 PM.png, Screen Shot 2018-07-19 at 1.38.56 PM.png, > Screen Shot 2018-07-19 at 2.22.42 PM.png, Screen Shot 2018-07-19 at 2.22.42 > PM.png, Screen Shot 2018-07-19 at 2.43.42 PM.png, Screen Shot 2018-07-19 at > 2.43.42 PM.png > > > While working on the parent issue, looking at a heap from a Master tha was > running ~650 servers and > 300k regions, I tripped over some silly items in > the heap: > 1. Balancer has a regions x server matrix which takes up 18% of the Master > heap according to jxray and 40% according to eclipse. Looks like the matrix > should be regions x racks which would be much smaller (Issue came in with > HBASE-18164 Fast locality computation in balancer -Added new > LocalityCostFunction and LocalityCandidateGenerator ..). See !Screen Shot > 2018-07-19 at 1.20.23 PM.png! !Screen Shot 2018-07-19 at 1.38.56 PM.png! > 2. Duplicate Strings make up ~5% of the Master heap. Of these, ServerName > seems to be the font. Interesting is report that there 54k instances of > ServerName in this heap though there are only 650 Servers. See !Screen Shot > 2018-07-19 at 2.22.42 PM.png! > 3. ArrayDequeue initializes its internal elements array with 16 elements. We > use this in a few places. In Procedures, of which there are many in this > heap, we near never make use of this array. See !Screen Shot 2018-07-19 at > 2.43.42 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20741) Split of a region with replicas creates all daughter regions and its replica in same server
[ https://issues.apache.org/jira/browse/HBASE-20741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-20741: --- Attachment: HBASE-20741.patch > Split of a region with replicas creates all daughter regions and its replica > in same server > --- > > Key: HBASE-20741 > URL: https://issues.apache.org/jira/browse/HBASE-20741 > Project: HBase > Issue Type: Bug > Components: read replicas >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > Fix For: 2.2.0 > > Attachments: HBASE-20741.patch > > > Generally it is better that the parent region when split creates the daughter > region in the same target server. > But for replicas also we do the same and all the replica regions are created > in the same target server. We should ideally be doing a round robin and only > the primary daughter region should be opened in the intended target server > (where the parent was previously opened). > [~huaxiang] FYI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20741) Split of a region with replicas creates all daughter regions and its replica in same server
[ https://issues.apache.org/jira/browse/HBASE-20741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551009#comment-16551009 ] ramkrishna.s.vasudevan commented on HBASE-20741: Initial patch. Just doing a round robin when there is a split to assign the daughter regions. Probably still there are some corner cases. > Split of a region with replicas creates all daughter regions and its replica > in same server > --- > > Key: HBASE-20741 > URL: https://issues.apache.org/jira/browse/HBASE-20741 > Project: HBase > Issue Type: Bug > Components: read replicas >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > Fix For: 2.2.0 > > Attachments: HBASE-20741.patch > > > Generally it is better that the parent region when split creates the daughter > region in the same target server. > But for replicas also we do the same and all the replica regions are created > in the same target server. We should ideally be doing a round robin and only > the primary daughter region should be opened in the intended target server > (where the parent was previously opened). > [~huaxiang] FYI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20914) Trim Master memory usage
[ https://issues.apache.org/jira/browse/HBASE-20914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20914: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to branch-2.0. Thanks for taking a look [~yuzhih...@gmail.com] > Trim Master memory usage > > > Key: HBASE-20914 > URL: https://issues.apache.org/jira/browse/HBASE-20914 > Project: HBase > Issue Type: Sub-task > Components: master >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 3.0.0, 2.0.2, 2.2.0, 2.1.1 > > Attachments: HBASE-20914.branch-2.0.001.patch, Screen Shot 2018-07-19 > at 1.20.23 PM.png, Screen Shot 2018-07-19 at 1.20.23 PM.png, Screen Shot > 2018-07-19 at 1.38.56 PM.png, Screen Shot 2018-07-19 at 1.38.56 PM.png, > Screen Shot 2018-07-19 at 2.22.42 PM.png, Screen Shot 2018-07-19 at 2.22.42 > PM.png, Screen Shot 2018-07-19 at 2.43.42 PM.png, Screen Shot 2018-07-19 at > 2.43.42 PM.png > > > While working on the parent issue, looking at a heap from a Master tha was > running ~650 servers and > 300k regions, I tripped over some silly items in > the heap: > 1. Balancer has a regions x server matrix which takes up 18% of the Master > heap according to jxray and 40% according to eclipse. Looks like the matrix > should be regions x racks which would be much smaller (Issue came in with > HBASE-18164 Fast locality computation in balancer -Added new > LocalityCostFunction and LocalityCandidateGenerator ..). See !Screen Shot > 2018-07-19 at 1.20.23 PM.png! !Screen Shot 2018-07-19 at 1.38.56 PM.png! > 2. Duplicate Strings make up ~5% of the Master heap. Of these, ServerName > seems to be the font. Interesting is report that there 54k instances of > ServerName in this heap though there are only 650 Servers. See !Screen Shot > 2018-07-19 at 2.22.42 PM.png! > 3. ArrayDequeue initializes its internal elements array with 16 elements. We > use this in a few places. In Procedures, of which there are many in this > heap, we near never make use of this array. See !Screen Shot 2018-07-19 at > 2.43.42 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20908) Infinite loop on regionserver if region replica are reduced
[
https://issues.apache.org/jira/browse/HBASE-20908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20908:
---
Attachment: 20908_v3.patch
> Infinite loop on regionserver if region replica are reduced
>
>
> Key: HBASE-20908
> URL: https://issues.apache.org/jira/browse/HBASE-20908
> Project: HBase
> Issue Type: Bug
> Components: read replicas
>Affects Versions: 1.2.0, 2.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: 20908_v3.patch, HBASE-20908.patch, HBASE-20908_v1.patch,
> HBASE-20908_v3.patch
>
>
> Steps to reproduce
> {code}
> hbase(main):003:0> create 'myTable','cf',{REGION_REPLICATION=>3}
> hbase(main):003:0> put 'myTable','r1','cf:col1','1'
> 0 row(s) in 0.1230 seconds
> hbase(main):004:0> disable 'myTable'
> alter '0 row(s) in 2.3040 seconds
> hbase(main):005:0> alter 'myTable',{REGION_REPLICATION=>1}
> Updating all regions with the new schema...
> 1/1 regions updated.
> Done.
> 0 row(s) in 11.9550 seconds
> hbase(main):006:0> enable 'myTable'
> 0 row(s) in 1.2620 seconds
> hbase(main):007:0> put 'myTable1','r2','cf:col1','1'
> 0 row(s) in 0.0060 seconds
> {code}
> This is the replica region request which will not be present now in Meta but
> was there in cache. Server will say that he is not serving this region.
> {code}
> com.google.protobuf.ServiceException:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.NotServingRegionException):
> org.apache.hadoop.hbase.NotServingRegionException: Region
> d997d9b47a106216b9b117617ec09015 is not online on
> 10.22.9.76,16020,1531341039091
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3124)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3106)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.replay(RSRpcServices.java:1714)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22773)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
> {code}
> Eventually, when we will update our cache after looking into meta , we will
> get into an infinite loop as this event will not be replicated because the
> location of the replica will not appear again.
> {code}
> java.net.SocketTimeoutException: callTimeout=120, callDuration=2181316:
> Can't get the location null
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:170)
> at
> org.apache.hadoop.hbase.replication.regionserver.RegionReplicaReplicationEndpoint$RetryingRpcCallable.call(RegionReplicaReplicationEndpoint.java:606)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't
> get the location
> at
> org.apache.hadoop.hbase.client.RegionAdminServiceCallable.getRegionLocations(RegionAdminServiceCallable.java:178)
> at
> org.apache.hadoop.hbase.client.RegionAdminServiceCallable.getLocation(RegionAdminServiceCallable.java:105)
> at
> org.apache.hadoop.hbase.client.RegionAdminServiceCallable.prepare(RegionAdminServiceCallable.java:89)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
> ... 5 more
> Caused by: java.io.IOException: HRegionInfo was null in myTable,
> row=keyvalues={myTable,,1531262022075.f2b68622cfd5851023be29d5599db6c9./info:regioninfo/1531262022425/Put/vlen=41/seqid=0,
>
> myTable,,1531262022075.f2b68622cfd5851023be29d5599db6c9./info:seqnumDuringOpen/1531341209944/Put/vlen=8/seqid=0,
>
> myTable,,1531262022075.f2b68622cfd5851023be29d5599db6c9./info:server/1531341209944/Put/vlen=16/seqid=0,
>
> myTable,,1531262022075.f2b68622cfd5851023be29d5599db6c9./info:serverstartcode/1531341209944/Put/vlen=8/seqid=0}
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1289)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1179)
> at
>
[jira] [Commented] (HBASE-17885) Backport HBASE-15871 to branch-1
[
https://issues.apache.org/jira/browse/HBASE-17885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550919#comment-16550919
]
Hadoop QA commented on HBASE-17885:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
3s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
0s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
53s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
53s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 48s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}102m 19s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}123m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:61288f8 |
| JIRA Issue | HBASE-17885 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12932419/HBASE-17885.branch-1.002.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 8e2ca9ab6077 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-20893) Data loss if splitting region while ServerCrashProcedure executing
[ https://issues.apache.org/jira/browse/HBASE-20893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550878#comment-16550878 ] stack commented on HBASE-20893: --- Thanks. > Data loss if splitting region while ServerCrashProcedure executing > -- > > Key: HBASE-20893 > URL: https://issues.apache.org/jira/browse/HBASE-20893 > Project: HBase > Issue Type: Sub-task >Affects Versions: 3.0.0, 2.1.0, 2.0.1 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Attachments: HBASE-20893.branch-2.0.001.patch, > HBASE-20893.branch-2.0.002.patch, HBASE-20893.branch-2.0.003.patch > > > Similar case as HBASE-20878. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20886) [Auth] Support keytab login in hbase client
[
https://issues.apache.org/jira/browse/HBASE-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550851#comment-16550851
]
Hadoop QA commented on HBASE-20886:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
29s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} hbase-common: The patch generated 0 new + 7
unchanged - 1 fixed = 7 total (was 8) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} The patch hbase-client passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} The patch hbase-server passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
39s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
15m 24s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
48s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 5m 18s{color}
| {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}152m 48s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
4s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}226m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestAsyncProcess |
| | hadoop.hbase.replication.TestSyncReplicationRemoveRemoteWAL |
| | hadoop.hbase.TestMetaTableAccessorNoCluster |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce
[jira] [Updated] (HBASE-20893) Data loss if splitting region while ServerCrashProcedure executing
[ https://issues.apache.org/jira/browse/HBASE-20893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allan Yang updated HBASE-20893: --- Attachment: (was: HBASE-20893.branch-2.0.003.patch) > Data loss if splitting region while ServerCrashProcedure executing > -- > > Key: HBASE-20893 > URL: https://issues.apache.org/jira/browse/HBASE-20893 > Project: HBase > Issue Type: Sub-task >Affects Versions: 3.0.0, 2.1.0, 2.0.1 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Attachments: HBASE-20893.branch-2.0.001.patch, > HBASE-20893.branch-2.0.002.patch, HBASE-20893.branch-2.0.003.patch > > > Similar case as HBASE-20878. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20893) Data loss if splitting region while ServerCrashProcedure executing
[ https://issues.apache.org/jira/browse/HBASE-20893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allan Yang updated HBASE-20893: --- Attachment: HBASE-20893.branch-2.0.003.patch > Data loss if splitting region while ServerCrashProcedure executing > -- > > Key: HBASE-20893 > URL: https://issues.apache.org/jira/browse/HBASE-20893 > Project: HBase > Issue Type: Sub-task >Affects Versions: 3.0.0, 2.1.0, 2.0.1 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Attachments: HBASE-20893.branch-2.0.001.patch, > HBASE-20893.branch-2.0.002.patch, HBASE-20893.branch-2.0.003.patch > > > Similar case as HBASE-20878. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20909) Add 2.1.0 to the download page
[
https://issues.apache.org/jira/browse/HBASE-20909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550791#comment-16550791
]
Hudson commented on HBASE-20909:
Results for branch master
[build #402 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/402/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/402//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/402//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/402//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add 2.1.0 to the download page
> --
>
> Key: HBASE-20909
> URL: https://issues.apache.org/jira/browse/HBASE-20909
> Project: HBase
> Issue Type: Sub-task
> Components: website
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20909.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20853) Polish "Add defaults to Table Interface so Implementors don't have to"
[
https://issues.apache.org/jira/browse/HBASE-20853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550788#comment-16550788
]
Hudson commented on HBASE-20853:
Results for branch master
[build #402 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/402/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/402//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/402//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/402//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Polish "Add defaults to Table Interface so Implementors don't have to"
> --
>
> Key: HBASE-20853
> URL: https://issues.apache.org/jira/browse/HBASE-20853
> Project: HBase
> Issue Type: Sub-task
> Components: API
>Reporter: stack
>Assignee: Balazs Meszaros
>Priority: Major
> Labels: beginner, beginners
> Fix For: 3.0.0, 2.0.2, 2.2.0, 2.1.1
>
> Attachments: HBASE-20853.master.001.patch,
> HBASE-20853.master.002.patch, HBASE-20853.master.003.patch,
> HBASE-20853.master.004.patch, HBASE-20853.master.005.patch
>
>
> This issue is to address feedback that came in after commit on the parent
> (FYI [~chia7712]). See tail of parent issue and amendment attached to parent
> adding better defaults to the Table Interface.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20870) Wrong HBase root dir in ITBLL's Search Tool
[
https://issues.apache.org/jira/browse/HBASE-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550794#comment-16550794
]
Hudson commented on HBASE-20870:
Results for branch master
[build #402 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/402/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/402//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/402//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/402//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Wrong HBase root dir in ITBLL's Search Tool
> ---
>
> Key: HBASE-20870
> URL: https://issues.apache.org/jira/browse/HBASE-20870
> Project: HBase
> Issue Type: Bug
> Components: integration tests
>Affects Versions: 3.0.0, 2.1.0, 2.0.1
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Minor
> Fix For: 3.0.0, 2.0.2, 2.2.0, 2.1.1
>
> Attachments: HBASE-20870.branch-2.0.001.patch,
> HBASE-20870.branch-2.0.002.patch, HBASE-20870.branch-2.0.003.patch
>
>
> When using IntegrationTestBigLinkedList's Search tools, it always fails since
> it tries to read WALs in a wrong HBase root dir. Turned out that when
> initializing IntegrationTestingUtility in IntegrationTestBigLinkedList, its
> super class HBaseTestingUtility will change hbase.rootdir to a local random
> dir. It is not wrong since HBaseTestingUtility is mostly used by Minicluster.
> But for IntegrationTest runs on distributed clusters, we should change it
> back.
> Here is the error info.
> {code:java}
> 2018-07-11 16:35:49,679 DEBUG [main] hbase.HBaseCommonTestingUtility: Setting
> hbase.rootdir to
> /home/hadoop/target/test-data/deb67611-2737-4696-abe9-32a7783df7bb
> 2018-07-11 16:35:50,736 ERROR [main] util.AbstractHBaseTool: Error running
> command-line tool java.io.FileNotFoundException: File
> file:/home/hadoop/target/test-data/deb67611-2737-4696-abe9-32a7783df7bb/WALs
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:431)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1517)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20869) Endpoint-based Export use incorrect user to write to destination
[
https://issues.apache.org/jira/browse/HBASE-20869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550789#comment-16550789
]
Hudson commented on HBASE-20869:
Results for branch master
[build #402 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/402/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/402//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/402//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/402//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Endpoint-based Export use incorrect user to write to destination
>
>
> Key: HBASE-20869
> URL: https://issues.apache.org/jira/browse/HBASE-20869
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Affects Versions: 2.0.0
> Environment: Hadoop 3.0.0 + HBase 2.0.0, Kerberos.
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 3.0.0, 2.0.2, 2.2.0, 2.1.1
>
> Attachments: HBASE-20869.master.001.patch,
> HBASE-20869.master.002.patch
>
>
> HBASE-15806 implemented an endpoint based export. It gets caller's HDFS
> delegation token, and RegionServer is supposed to write out exported files as
> the caller.
> Everything works fine if you use run export as hbase user. However, once you
> use a different user to export, it fails.
> To reproduce,
> Add to configuration key hbase.coprocessor.region.classes the coprocessor
> class org.apache.hadoop.hbase.coprocessor.Export.
> create a table t1, assign permission to a user foo:
>
> {noformat}
> hbase(main):004:0> user_permission 't1'
> User Namespace,Table,Family,Qualifier:Permission
> hbase default,t1,,: [Permission: actions=READ,WRITE,EXEC,CREATE,ADMIN]
> foo default,t1,,: [Permission: actions=READ,WRITE,EXEC,CREATE,ADMIN]{noformat}
>
> As user foo, execute the following command:
>
> {noformat}
> $ hdfs dfs -mkdir /tmp/export_hbase2
> $ hbase org.apache.hadoop.hbase.coprocessor.Export t1 /tmp/export_hbase2/t2/
>
> 18/07/10 14:03:59 INFO client.RpcRetryingCallerImpl: Call exception, tries=6,
> retries=6, started=4457 ms ago, cancelled=false,
> msg=org.apache.hadoop.security.AccessControlException: Permission denied:
> user=hbase, access=WRITE,
> inode="/tmp/export_hbase2/t2":foo:supergroup:drwxr-xr-x
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:400)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:256)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:194)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1846)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1830)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1789)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.resolvePathForStartFile(FSDirWriteFileOp.java:316)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2411)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2343)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:764)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:451)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> at
[jira] [Commented] (HBASE-6028) Implement a cancel for in-progress compactions
[
https://issues.apache.org/jira/browse/HBASE-6028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550790#comment-16550790
]
Hudson commented on HBASE-6028:
---
Results for branch master
[build #402 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/402/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/402//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/402//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/402//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Implement a cancel for in-progress compactions
> --
>
> Key: HBASE-6028
> URL: https://issues.apache.org/jira/browse/HBASE-6028
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Derek Wollenstein
>Assignee: Mohit Goel
>Priority: Minor
> Labels: beginner
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-6028.master.007.patch,
> HBASE-6028.master.008.patch, HBASE-6028.master.008.patch,
> HBASE-6028.master.009.patch
>
>
> Depending on current server load, it can be extremely expensive to run
> periodic minor / major compactions. It would be helpful to have a feature
> where a user could use the shell or a client tool to explicitly cancel an
> in-progress compactions. This would allow a system to recover when too many
> regions became eligible for compactions at once
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20901) Reducing region replica has no effect
[
https://issues.apache.org/jira/browse/HBASE-20901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550792#comment-16550792
]
Hudson commented on HBASE-20901:
Results for branch master
[build #402 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/402/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/402//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/402//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/402//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Reducing region replica has no effect
> -
>
> Key: HBASE-20901
> URL: https://issues.apache.org/jira/browse/HBASE-20901
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Labels: replica
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-20901.patch, HBASE-20901_v1.patch,
> HBASE-20901_v2.patch
>
>
> While reducing the region replica, server name(sn) and state column of the
> replica are not getting deleted, resulting in assignment manager to think
> that these regions are CLOSED and assign them again.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20672) New metrics ReadRequestRate and WriteRequestRate
[
https://issues.apache.org/jira/browse/HBASE-20672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550793#comment-16550793
]
Hudson commented on HBASE-20672:
Results for branch master
[build #402 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/402/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/402//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/402//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/402//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> New metrics ReadRequestRate and WriteRequestRate
>
>
> Key: HBASE-20672
> URL: https://issues.apache.org/jira/browse/HBASE-20672
> Project: HBase
> Issue Type: Improvement
> Components: metrics
>Reporter: Ankit Jain
>Assignee: Ankit Jain
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-20672.branch-1.001.patch,
> HBASE-20672.branch-1.002.patch, HBASE-20672.branch-2.001.patch,
> HBASE-20672.master.001.patch, HBASE-20672.master.002.patch,
> HBASE-20672.master.003.patch, hits1vs2.4.40.400.png
>
>
> Hbase currently provides counter read/write requests (ReadRequestCount,
> WriteRequestCount). That said it is not easy to use counter that reset only
> after a restart of the service, we would like to expose 2 new metrics in
> HBase to provide ReadRequestRate and WriteRequestRate at region server level.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20823) Wrong param name in javadoc for HRegionServer#buildRegionSpaceUseReportRequest
[
https://issues.apache.org/jira/browse/HBASE-20823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550787#comment-16550787
]
Hudson commented on HBASE-20823:
Results for branch master
[build #402 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/402/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/402//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/402//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/402//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Wrong param name in javadoc for HRegionServer#buildRegionSpaceUseReportRequest
> --
>
> Key: HBASE-20823
> URL: https://issues.apache.org/jira/browse/HBASE-20823
> Project: HBase
> Issue Type: Improvement
>Reporter: Reid Chan
>Assignee: Jack Bearden
>Priority: Trivial
> Labels: beginner
> Fix For: 3.0.0
>
> Attachments: HBASE-20823.001.patch
>
>
> {code}
> /**
>* Builds a {@link RegionSpaceUseReportRequest} protobuf message from the
> region size map.
>*
>* @param regionSizeStore The size in bytes of regions
>* @return The corresponding protocol buffer message.
>*/
> RegionSpaceUseReportRequest
> buildRegionSpaceUseReportRequest(RegionSizeStore regionSizes)
> {code}
> {{@param regionSizeStore}}, but actual name is regionSizes
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17885) Backport HBASE-15871 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-17885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550778#comment-16550778 ] Toshihiro Suzuki commented on HBASE-17885: -- I just attached v2 patch. I increased the timeout of the test TestLoadIncrementalHFiles.testRegionCrossingHFileSplit() as it failed due to timeout. > Backport HBASE-15871 to branch-1 > > > Key: HBASE-17885 > URL: https://issues.apache.org/jira/browse/HBASE-17885 > Project: HBase > Issue Type: Bug > Components: Scanners >Affects Versions: 1.3.1, 1.2.5, 1.1.8 >Reporter: ramkrishna.s.vasudevan >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 1.5.0 > > Attachments: HBASE-17885.branch-1.001.patch, > HBASE-17885.branch-1.001.patch, HBASE-17885.branch-1.002.patch > > > Will try to rebase the branch-1 patch at the earliest. Hope the fix versions > are correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-17885) Backport HBASE-15871 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-17885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Toshihiro Suzuki updated HBASE-17885: - Attachment: HBASE-17885.branch-1.002.patch > Backport HBASE-15871 to branch-1 > > > Key: HBASE-17885 > URL: https://issues.apache.org/jira/browse/HBASE-17885 > Project: HBase > Issue Type: Bug > Components: Scanners >Affects Versions: 1.3.1, 1.2.5, 1.1.8 >Reporter: ramkrishna.s.vasudevan >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 1.5.0 > > Attachments: HBASE-17885.branch-1.001.patch, > HBASE-17885.branch-1.001.patch, HBASE-17885.branch-1.002.patch > > > Will try to rebase the branch-1 patch at the earliest. Hope the fix versions > are correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17885) Backport HBASE-15871 to branch-1
[
https://issues.apache.org/jira/browse/HBASE-17885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550707#comment-16550707
]
Hadoop QA commented on HBASE-17885:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
27s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
24s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 34s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}131m 43s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}149m 34s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFiles |
| | hadoop.hbase.mapreduce.TestLoadIncrementalHFilesUseSecurityEndPoint |
| | hadoop.hbase.mapreduce.TestLoadIncrementalHFiles |
| | hadoop.hbase.regionserver.throttle.TestFlushWithThroughputController |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:61288f8 |
| JIRA Issue | HBASE-17885 |
| JIRA Patch URL |
[jira] [Commented] (HBASE-20870) Wrong HBase root dir in ITBLL's Search Tool
[
https://issues.apache.org/jira/browse/HBASE-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550674#comment-16550674
]
Hudson commented on HBASE-20870:
Results for branch branch-2
[build #1004 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1004/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1004//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1004//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1004//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Wrong HBase root dir in ITBLL's Search Tool
> ---
>
> Key: HBASE-20870
> URL: https://issues.apache.org/jira/browse/HBASE-20870
> Project: HBase
> Issue Type: Bug
> Components: integration tests
>Affects Versions: 3.0.0, 2.1.0, 2.0.1
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Minor
> Fix For: 3.0.0, 2.0.2, 2.2.0, 2.1.1
>
> Attachments: HBASE-20870.branch-2.0.001.patch,
> HBASE-20870.branch-2.0.002.patch, HBASE-20870.branch-2.0.003.patch
>
>
> When using IntegrationTestBigLinkedList's Search tools, it always fails since
> it tries to read WALs in a wrong HBase root dir. Turned out that when
> initializing IntegrationTestingUtility in IntegrationTestBigLinkedList, its
> super class HBaseTestingUtility will change hbase.rootdir to a local random
> dir. It is not wrong since HBaseTestingUtility is mostly used by Minicluster.
> But for IntegrationTest runs on distributed clusters, we should change it
> back.
> Here is the error info.
> {code:java}
> 2018-07-11 16:35:49,679 DEBUG [main] hbase.HBaseCommonTestingUtility: Setting
> hbase.rootdir to
> /home/hadoop/target/test-data/deb67611-2737-4696-abe9-32a7783df7bb
> 2018-07-11 16:35:50,736 ERROR [main] util.AbstractHBaseTool: Error running
> command-line tool java.io.FileNotFoundException: File
> file:/home/hadoop/target/test-data/deb67611-2737-4696-abe9-32a7783df7bb/WALs
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:431)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1517)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20672) New metrics ReadRequestRate and WriteRequestRate
[
https://issues.apache.org/jira/browse/HBASE-20672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550659#comment-16550659
]
Hudson commented on HBASE-20672:
Results for branch branch-1
[build #386 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/386/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/386//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/386//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/386//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> New metrics ReadRequestRate and WriteRequestRate
>
>
> Key: HBASE-20672
> URL: https://issues.apache.org/jira/browse/HBASE-20672
> Project: HBase
> Issue Type: Improvement
> Components: metrics
>Reporter: Ankit Jain
>Assignee: Ankit Jain
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-20672.branch-1.001.patch,
> HBASE-20672.branch-1.002.patch, HBASE-20672.branch-2.001.patch,
> HBASE-20672.master.001.patch, HBASE-20672.master.002.patch,
> HBASE-20672.master.003.patch, hits1vs2.4.40.400.png
>
>
> Hbase currently provides counter read/write requests (ReadRequestCount,
> WriteRequestCount). That said it is not easy to use counter that reset only
> after a restart of the service, we would like to expose 2 new metrics in
> HBase to provide ReadRequestRate and WriteRequestRate at region server level.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20855) PeerConfigTracker only supporting one listener will cause problem when there is a recovered replication queue
[
https://issues.apache.org/jira/browse/HBASE-20855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550658#comment-16550658
]
Hudson commented on HBASE-20855:
Results for branch branch-1
[build #386 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/386/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/386//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/386//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/386//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> PeerConfigTracker only supporting one listener will cause problem when there
> is a recovered replication queue
> -
>
> Key: HBASE-20855
> URL: https://issues.apache.org/jira/browse/HBASE-20855
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0, 1.5.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
> Fix For: 1.5.0, 1.4.6
>
> Attachments: HBASE-20855.branch-1.001.patch,
> HBASE-20855.branch-1.002.patch, HBASE-20855.branch-1.003.patch,
> HBASE-20855.branch-1.004.patch, HBASE-20855.branch-1.005.patch,
> HBASE-20855.branch-1.006.patch, HBASE-20855.branch-1.007.patch
>
>
> {code}
> public void init(Context context) throws IOException {
> this.ctx = context;
> if (this.ctx != null){
> ReplicationPeer peer = this.ctx.getReplicationPeer();
> if (peer != null){
> peer.trackPeerConfigChanges(this);
> } else {
> LOG.warn("Not tracking replication peer config changes for Peer Id " +
> this.ctx.getPeerId() +
> " because there's no such peer");
> }
> }
> }
> {code}
> As we know, replication source will set itself to the PeerConfigTracker in
> ReplicationPeer. When there is one or more recovered queue, each queue will
> generate a new replication source, But they share the same ReplicationPeer.
> Then when it calls setListener, the new generated one will cover the older
> one. Thus there will only has one ReplicationPeer that receive the peer
> config change notify.
> {code}
> public synchronized void setListener(ReplicationPeerConfigListener listener){
> this.listener = listener;
> }
> {code}
>
> To solve this, PeerConfigTracker need to support multiple listener and
> listener should be removed when the replication endpoint terminated.
> I will upload a patch later with fix and UT.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20870) Wrong HBase root dir in ITBLL's Search Tool
[
https://issues.apache.org/jira/browse/HBASE-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550656#comment-16550656
]
Hudson commented on HBASE-20870:
Results for branch branch-2.1
[build #82 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/82/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/82//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/82//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/82//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Wrong HBase root dir in ITBLL's Search Tool
> ---
>
> Key: HBASE-20870
> URL: https://issues.apache.org/jira/browse/HBASE-20870
> Project: HBase
> Issue Type: Bug
> Components: integration tests
>Affects Versions: 3.0.0, 2.1.0, 2.0.1
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Minor
> Fix For: 3.0.0, 2.0.2, 2.2.0, 2.1.1
>
> Attachments: HBASE-20870.branch-2.0.001.patch,
> HBASE-20870.branch-2.0.002.patch, HBASE-20870.branch-2.0.003.patch
>
>
> When using IntegrationTestBigLinkedList's Search tools, it always fails since
> it tries to read WALs in a wrong HBase root dir. Turned out that when
> initializing IntegrationTestingUtility in IntegrationTestBigLinkedList, its
> super class HBaseTestingUtility will change hbase.rootdir to a local random
> dir. It is not wrong since HBaseTestingUtility is mostly used by Minicluster.
> But for IntegrationTest runs on distributed clusters, we should change it
> back.
> Here is the error info.
> {code:java}
> 2018-07-11 16:35:49,679 DEBUG [main] hbase.HBaseCommonTestingUtility: Setting
> hbase.rootdir to
> /home/hadoop/target/test-data/deb67611-2737-4696-abe9-32a7783df7bb
> 2018-07-11 16:35:50,736 ERROR [main] util.AbstractHBaseTool: Error running
> command-line tool java.io.FileNotFoundException: File
> file:/home/hadoop/target/test-data/deb67611-2737-4696-abe9-32a7783df7bb/WALs
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:431)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1517)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20870) Wrong HBase root dir in ITBLL's Search Tool
[
https://issues.apache.org/jira/browse/HBASE-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550652#comment-16550652
]
Hudson commented on HBASE-20870:
Results for branch branch-2.0
[build #570 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/570/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/570//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/570//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/570//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Wrong HBase root dir in ITBLL's Search Tool
> ---
>
> Key: HBASE-20870
> URL: https://issues.apache.org/jira/browse/HBASE-20870
> Project: HBase
> Issue Type: Bug
> Components: integration tests
>Affects Versions: 3.0.0, 2.1.0, 2.0.1
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Minor
> Fix For: 3.0.0, 2.0.2, 2.2.0, 2.1.1
>
> Attachments: HBASE-20870.branch-2.0.001.patch,
> HBASE-20870.branch-2.0.002.patch, HBASE-20870.branch-2.0.003.patch
>
>
> When using IntegrationTestBigLinkedList's Search tools, it always fails since
> it tries to read WALs in a wrong HBase root dir. Turned out that when
> initializing IntegrationTestingUtility in IntegrationTestBigLinkedList, its
> super class HBaseTestingUtility will change hbase.rootdir to a local random
> dir. It is not wrong since HBaseTestingUtility is mostly used by Minicluster.
> But for IntegrationTest runs on distributed clusters, we should change it
> back.
> Here is the error info.
> {code:java}
> 2018-07-11 16:35:49,679 DEBUG [main] hbase.HBaseCommonTestingUtility: Setting
> hbase.rootdir to
> /home/hadoop/target/test-data/deb67611-2737-4696-abe9-32a7783df7bb
> 2018-07-11 16:35:50,736 ERROR [main] util.AbstractHBaseTool: Error running
> command-line tool java.io.FileNotFoundException: File
> file:/home/hadoop/target/test-data/deb67611-2737-4696-abe9-32a7783df7bb/WALs
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:431)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1517)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20886) [Auth] Support keytab login in hbase client
[
https://issues.apache.org/jira/browse/HBASE-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reid Chan updated HBASE-20886:
--
Attachment: HBASE-20886.master.003.patch
> [Auth] Support keytab login in hbase client
> ---
>
> Key: HBASE-20886
> URL: https://issues.apache.org/jira/browse/HBASE-20886
> Project: HBase
> Issue Type: Improvement
> Components: asyncclient, Client, security
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Critical
> Attachments: HBASE-20886.master.001.patch,
> HBASE-20886.master.002.patch, HBASE-20886.master.003.patch
>
>
> There're lots of questions about how to connect to kerberized hbase cluster
> through hbase-client api from user-mail and slack channel.
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> already existed in code base, but they are only used in {{Canary}}.
> This issue is to make use of two configs to support client-side keytab based
> login, after this issue resolved, hbase-client should directly connect to
> kerberized cluster without changing any code as long as
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> specified.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-4361) Certain filter expressions fail in the shell
[
https://issues.apache.org/jira/browse/HBASE-4361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sreeram Venkatasubramanian reassigned HBASE-4361:
-
Assignee: Sreeram Venkatasubramanian
> Certain filter expressions fail in the shell
>
>
> Key: HBASE-4361
> URL: https://issues.apache.org/jira/browse/HBASE-4361
> Project: HBase
> Issue Type: Bug
> Components: Filters, shell
>Affects Versions: 2.0.0
>Reporter: Todd Lipcon
>Assignee: Sreeram Venkatasubramanian
>Priority: Major
> Labels: beginner
> Attachments: Filter Language.docx, small-improvements.txt
>
>
> Running the following in the shell hangs and then fails:
> {noformat}
> scan 't1', { FILTER => "SingleColumnValueFilter(">", '1', 'f1', 'col_a')" }
> {noformat}
> The error seems to be: org.jruby.exceptions.RaiseException: (NoMethodError)
> undefined method `write' for true:TrueClass
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20886) [Auth] Support keytab login in hbase client
[
https://issues.apache.org/jira/browse/HBASE-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reid Chan updated HBASE-20886:
--
Attachment: HBASE-20886.master.003.patch
> [Auth] Support keytab login in hbase client
> ---
>
> Key: HBASE-20886
> URL: https://issues.apache.org/jira/browse/HBASE-20886
> Project: HBase
> Issue Type: Improvement
> Components: asyncclient, Client, security
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Critical
> Attachments: HBASE-20886.master.001.patch,
> HBASE-20886.master.002.patch
>
>
> There're lots of questions about how to connect to kerberized hbase cluster
> through hbase-client api from user-mail and slack channel.
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> already existed in code base, but they are only used in {{Canary}}.
> This issue is to make use of two configs to support client-side keytab based
> login, after this issue resolved, hbase-client should directly connect to
> kerberized cluster without changing any code as long as
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> specified.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20886) [Auth] Support keytab login in hbase client
[
https://issues.apache.org/jira/browse/HBASE-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reid Chan updated HBASE-20886:
--
Attachment: (was: HBASE-20886.master.003.patch)
> [Auth] Support keytab login in hbase client
> ---
>
> Key: HBASE-20886
> URL: https://issues.apache.org/jira/browse/HBASE-20886
> Project: HBase
> Issue Type: Improvement
> Components: asyncclient, Client, security
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Critical
> Attachments: HBASE-20886.master.001.patch,
> HBASE-20886.master.002.patch
>
>
> There're lots of questions about how to connect to kerberized hbase cluster
> through hbase-client api from user-mail and slack channel.
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> already existed in code base, but they are only used in {{Canary}}.
> This issue is to make use of two configs to support client-side keytab based
> login, after this issue resolved, hbase-client should directly connect to
> kerberized cluster without changing any code as long as
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> specified.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20401) Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
[ https://issues.apache.org/jira/browse/HBASE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550633#comment-16550633 ] Reid Chan commented on HBASE-20401: --- ping [~yuzhih...@gmail.com], [~mdrob], have more comments? > Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable > -- > > Key: HBASE-20401 > URL: https://issues.apache.org/jira/browse/HBASE-20401 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 3.0.0, 1.5.0, 2.0.0-beta-1, 1.4.4, 2.0.0 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Minor > Labels: beginner > Attachments: HBASE-20401.branch-1.001.patch, > HBASE-20401.master.001.patch, HBASE-20401.master.002.patch, > HBASE-20401.master.003.patch, HBASE-20401.master.004.patch, > HBASE-20401.master.005.patch, HBASE-20401.master.006.patch > > > When backporting HBASE-18309 in HBASE-20352, the deleteFiles calls > CleanerContext.java#getResult with a waitIfNotFinished timeout to wait for > notification (notify) from the fs.delete file thread. there might be two > situation need to tune the MAX_WAIT in CleanerContext or waitIfNotFinished > when LogClearner call getResult. > # fs.delete never complete (strange but possible), then we need to wait for > a max of 60 seconds. here, 60 seconds might be too long > # getResult is waiting in the period of 500 milliseconds, but the fs.delete > has completed and setFromClear is set but yet notify(). one might want to > tune this 500 milliseconds to 200 or less . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20916) Backport HBASE-19818 to branch-1
Minwoo Kang created HBASE-20916: --- Summary: Backport HBASE-19818 to branch-1 Key: HBASE-20916 URL: https://issues.apache.org/jira/browse/HBASE-20916 Project: HBase Issue Type: Bug Affects Versions: 1.2.6 Reporter: Minwoo Kang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-17885) Backport HBASE-15871 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-17885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Toshihiro Suzuki updated HBASE-17885: - Attachment: HBASE-17885.branch-1.001.patch > Backport HBASE-15871 to branch-1 > > > Key: HBASE-17885 > URL: https://issues.apache.org/jira/browse/HBASE-17885 > Project: HBase > Issue Type: Bug > Components: Scanners >Affects Versions: 1.3.1, 1.2.5, 1.1.8 >Reporter: ramkrishna.s.vasudevan >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 1.5.0 > > Attachments: HBASE-17885.branch-1.001.patch, > HBASE-17885.branch-1.001.patch > > > Will try to rebase the branch-1 patch at the earliest. Hope the fix versions > are correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17885) Backport HBASE-15871 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-17885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550561#comment-16550561 ] Toshihiro Suzuki commented on HBASE-17885: -- I don't think the test failure in the last QA is related to the patch. I just reattached the same patch to rerun QA. > Backport HBASE-15871 to branch-1 > > > Key: HBASE-17885 > URL: https://issues.apache.org/jira/browse/HBASE-17885 > Project: HBase > Issue Type: Bug > Components: Scanners >Affects Versions: 1.3.1, 1.2.5, 1.1.8 >Reporter: ramkrishna.s.vasudevan >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 1.5.0 > > Attachments: HBASE-17885.branch-1.001.patch, > HBASE-17885.branch-1.001.patch > > > Will try to rebase the branch-1 patch at the earliest. Hope the fix versions > are correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20401) Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
[
https://issues.apache.org/jira/browse/HBASE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550549#comment-16550549
]
Hadoop QA commented on HBASE-20401:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
1s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 5m
9s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
26s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 4m
50s{color} | {color:blue} patch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
21s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 43s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}245m
5s{color} | {color:green} root in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
2s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}314m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20401 |
| JIRA Patch URL |
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[
https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550535#comment-16550535
]
Ted Yu commented on HBASE-20565:
Zheng:
Can you attach patch for branch-1 ?
{code}
+result.listCells().forEach(results::add);
{code}
The above wouldn't compile by Java 1.7
> ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect
> result since 1.4
> -
>
> Key: HBASE-20565
> URL: https://issues.apache.org/jira/browse/HBASE-20565
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 2.1.0, 1.4.4, 2.0.1
>Reporter: Jerry He
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 2.1.0, 1.5.0, 1.4.6, 2.0.2
>
> Attachments: HBASE-20565.v1.patch, HBASE-20565.v2.patch, debug.diff,
> debug.log, test-branch-1.4.patch
>
>
> When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see
> incorrect result.
> Here is a simple example.
> One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for
> range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset
> 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter,
> ColumnPaginationFilter).
> We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns
> are returned.
> In 1.2.x, the correct 5 columns are returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20904) Prometheus /metrics http endpoint for monitoring integration
[ https://issues.apache.org/jira/browse/HBASE-20904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated HBASE-20904: Component/s: metrics > Prometheus /metrics http endpoint for monitoring integration > > > Key: HBASE-20904 > URL: https://issues.apache.org/jira/browse/HBASE-20904 > Project: HBase > Issue Type: New Feature > Components: metrics, monitoring >Reporter: Hari Sekhon >Priority: Major > > Feature Request to add Prometheus /metrics http endpoint for monitoring > integration: > [https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E] > Prometheus metrics format for that endpoint: > [https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20915) Remove the commit column on our download page
Duo Zhang created HBASE-20915:
-
Summary: Remove the commit column on our download page
Key: HBASE-20915
URL: https://issues.apache.org/jira/browse/HBASE-20915
Project: HBase
Issue Type: Task
Reporter: Duo Zhang
As required by the ASF moderator.
{quote}
However first please fix the download page so it does not link to the Git
repo.
Please drop the commit id and the link; they are not appropriate for a
download page.
Only approved releases should be linked from the download page.
Whilst the commit itself may have been included in the VOTE email as part
of the approval process, it is not a formal release artefact. And links to
repos give access to code that has not been approved.
Information on commit ids and git repos etc should appear on pages intended
for the HBase developer community only.
{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
