[jira] [Commented] (HBASE-23743) Release 1.4.13
[ https://issues.apache.org/jira/browse/HBASE-23743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17037716#comment-17037716 ] Sakthi commented on HBASE-23743: Have moved few jiras to 1.4.14. Rest looks ok. Let me start testing out. > Release 1.4.13 > -- > > Key: HBASE-23743 > URL: https://issues.apache.org/jira/browse/HBASE-23743 > Project: HBase > Issue Type: Task > Components: build, community >Affects Versions: 1.4.13 >Reporter: Sean Busbey >Assignee: Sakthi >Priority: Major > Fix For: 1.4.13 > > > over due for 1.4.13. I'll try to use the release manager scripts this time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23656) [MERGETOOL] HBASE Support Merge region by pattern
[ https://issues.apache.org/jira/browse/HBASE-23656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sakthi updated HBASE-23656: --- Fix Version/s: (was: 1.4.13) 1.4.14 > [MERGETOOL] HBASE Support Merge region by pattern > - > > Key: HBASE-23656 > URL: https://issues.apache.org/jira/browse/HBASE-23656 > Project: HBase > Issue Type: New Feature > Components: master >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Fix For: 3.0.0, 1.4.14 > > Attachments: HBASE-23656.branch-1.4.v1.patch, > HBASE-23656.master.v1.patch > > > Design Objective: > # Merge empty region > # Neat region > # merge expired region > Usage: bin/hbase onlinemerge [--tableName=] [--startRegion=] [--stopRegion=] > [--maxRegionSize=] [--maxRegionCreateTime=] [--numMaxMergePlans=] > [--targetRegionCount=] [--printExecutionPlan=] [--configMergePauseTime=] > Options: > --h or --h print help > --tableName table name must be not null > --startRegion start region > --stopRegion stop region > --maxRegionSize max region size Unit GB > --maxRegionCreateTime max Region Create Time /MM/dd HH:mm:ss > --numMaxMergePlans num MaxMerge Plans > --targetRegionCount target Region Count > --configMergePauseTime config Merge Pause Time In milliseconds > --printExecutionPlan Value default is true print execution plans false is > execution merge > Examples: > bin/hbase onlinemerge --tableName=test:test1 > --startRegion=test:test1,,1576835912332.01d0d6c2b41e204104524d9aec6074fb. > --stopRegion=test:test1,,1573044786980.0c9b5bd93f3b19eb9bd1a1011ddff66f. > --maxRegionSize=0 --maxRegionCreateTime=/MM/dd HH:mm:ss > --numMaxMergePlans=2 --targetRegionCount=4 --printExecutionPlan=false -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-21903) Backport major compaction tool HBASE-19528 from to 1.4 and 1.3
[ https://issues.apache.org/jira/browse/HBASE-21903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sakthi updated HBASE-21903: --- Fix Version/s: 1.4.14 > Backport major compaction tool HBASE-19528 from to 1.4 and 1.3 > -- > > Key: HBASE-21903 > URL: https://issues.apache.org/jira/browse/HBASE-21903 > Project: HBase > Issue Type: Task > Components: Client, Compaction, tooling >Affects Versions: 1.3.3, 1.4.9 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Fix For: 1.3.7, 1.4.14 > > Attachments: HBASE-21903-branch-1.3-addendum.patch > > > Our internal deployments are based on branch-1.3. We will be using the major > compaction tool HBASE-19528 from [~churromorales] and the enhancements on top > of it HBASE-21883 on our 1.3 clusters. I would like to backport HBASE-19528 > to 1.3 and hence 1.4 as well. Since its a standalone tool without any other > dependency or code changes, I believe that should be ok. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-21903) Backport major compaction tool HBASE-19528 from to 1.4 and 1.3
[ https://issues.apache.org/jira/browse/HBASE-21903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sakthi updated HBASE-21903: --- Fix Version/s: (was: 1.4.1) > Backport major compaction tool HBASE-19528 from to 1.4 and 1.3 > -- > > Key: HBASE-21903 > URL: https://issues.apache.org/jira/browse/HBASE-21903 > Project: HBase > Issue Type: Task > Components: Client, Compaction, tooling >Affects Versions: 1.3.3, 1.4.9 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Fix For: 1.3.7 > > Attachments: HBASE-21903-branch-1.3-addendum.patch > > > Our internal deployments are based on branch-1.3. We will be using the major > compaction tool HBASE-19528 from [~churromorales] and the enhancements on top > of it HBASE-21883 on our 1.3 clusters. I would like to backport HBASE-19528 > to 1.3 and hence 1.4 as well. Since its a standalone tool without any other > dependency or code changes, I believe that should be ok. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-21903) Backport major compaction tool HBASE-19528 from to 1.4 and 1.3
[ https://issues.apache.org/jira/browse/HBASE-21903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sakthi updated HBASE-21903: --- Fix Version/s: (was: 1.4.13) 1.4.1 > Backport major compaction tool HBASE-19528 from to 1.4 and 1.3 > -- > > Key: HBASE-21903 > URL: https://issues.apache.org/jira/browse/HBASE-21903 > Project: HBase > Issue Type: Task > Components: Client, Compaction, tooling >Affects Versions: 1.3.3, 1.4.9 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Fix For: 1.4.1, 1.3.7 > > Attachments: HBASE-21903-branch-1.3-addendum.patch > > > Our internal deployments are based on branch-1.3. We will be using the major > compaction tool HBASE-19528 from [~churromorales] and the enhancements on top > of it HBASE-21883 on our 1.3 clusters. I would like to backport HBASE-19528 > to 1.3 and hence 1.4 as well. Since its a standalone tool without any other > dependency or code changes, I believe that should be ok. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] jatsakthi opened a new pull request #1174: HBASE-23827 [HBCK2] Add mergeRegions method to hbck service
jatsakthi opened a new pull request #1174: HBASE-23827 [HBCK2] Add mergeRegions method to hbck service URL: https://github.com/apache/hbase/pull/1174 Have also included little bit of pending cleanups from HBASE-22673 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-23811) [OpenTracing] Add shaded JaegerTracing tracer to hbase-thirdparty
[ https://issues.apache.org/jira/browse/HBASE-23811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HBASE-23811: Status: Patch Available (was: Open) > [OpenTracing] Add shaded JaegerTracing tracer to hbase-thirdparty > - > > Key: HBASE-23811 > URL: https://issues.apache.org/jira/browse/HBASE-23811 > Project: HBase > Issue Type: Sub-task > Components: thirdparty >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > > JaegerTracing pulls in lots of dependencies. Some, like libthrift (0.13.0) > conflicts the one ships in HBase (0.12.0). > Additionally, not everyone may want to use Jaeger. > I propose to shade JaegerTracing and its dependencies into an uber jar, place > it as a hbase-thirdparty artifact. As an added benefit, this makes the > management of tracers in the HBase's dependency tree much easier. Finally, we > can follow the same suit and provide Zipkin tracer support in the future. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22514) Move rsgroup feature into core of HBase
[
https://issues.apache.org/jira/browse/HBASE-22514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17037710#comment-17037710
]
Hudson commented on HBASE-22514:
Results for branch HBASE-22514
[build #275 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/275/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/275//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/275//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/275//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Move rsgroup feature into core of HBase
> ---
>
> Key: HBASE-22514
> URL: https://issues.apache.org/jira/browse/HBASE-22514
> Project: HBase
> Issue Type: Umbrella
> Components: Admin, Client, rsgroup
>Reporter: Yechao Chen
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HBASE-22514.master.001.patch,
> image-2019-05-31-18-25-38-217.png
>
>
> The class RSGroupAdminClient is not public
> we need to use java api RSGroupAdminClient to manager RSG
> so RSGroupAdminClient should be public
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22749) Distributed MOB compactions
[
https://issues.apache.org/jira/browse/HBASE-22749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17037703#comment-17037703
]
Hudson commented on HBASE-22749:
Results for branch HBASE-22749
[build #4 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22749/4/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22749/4//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22749/4//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22749/4//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Distributed MOB compactions
>
>
> Key: HBASE-22749
> URL: https://issues.apache.org/jira/browse/HBASE-22749
> Project: HBase
> Issue Type: New Feature
> Components: mob
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Major
> Attachments: HBASE-22749-branch-2.2-v4.patch,
> HBASE-22749-master-v1.patch, HBASE-22749-master-v2.patch,
> HBASE-22749-master-v3.patch, HBASE-22749-master-v4.patch,
> HBase-MOB-2.0-v3.0.pdf
>
>
> There are several drawbacks in the original MOB 1.0 (Moderate Object
> Storage) implementation, which can limit the adoption of the MOB feature:
> # MOB compactions are executed in a Master as a chore, which limits
> scalability because all I/O goes through a single HBase Master server.
> # Yarn/Mapreduce framework is required to run MOB compactions in a scalable
> way, but this won’t work in a stand-alone HBase cluster.
> # Two separate compactors for MOB and for regular store files and their
> interactions can result in a data loss (see HBASE-22075)
> The design goals for MOB 2.0 were to provide 100% MOB 1.0 - compatible
> implementation, which is free of the above drawbacks and can be used as a
> drop in replacement in existing MOB deployments. So, these are design goals
> of a MOB 2.0:
> # Make MOB compactions scalable without relying on Yarn/Mapreduce framework
> # Provide unified compactor for both MOB and regular store files
> # Make it more robust especially w.r.t. to data losses.
> # Simplify and reduce the overall MOB code.
> # Provide 100% compatible implementation with MOB 1.0.
> # No migration of data should be required between MOB 1.0 and MOB 2.0 - just
> software upgrade.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22749) Distributed MOB compactions
[
https://issues.apache.org/jira/browse/HBASE-22749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17037662#comment-17037662
]
Hudson commented on HBASE-22749:
Results for branch HBASE-22749
[build #3 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22749/3/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22749/3//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22749/3//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22749/3//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Distributed MOB compactions
>
>
> Key: HBASE-22749
> URL: https://issues.apache.org/jira/browse/HBASE-22749
> Project: HBase
> Issue Type: New Feature
> Components: mob
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Major
> Attachments: HBASE-22749-branch-2.2-v4.patch,
> HBASE-22749-master-v1.patch, HBASE-22749-master-v2.patch,
> HBASE-22749-master-v3.patch, HBASE-22749-master-v4.patch,
> HBase-MOB-2.0-v3.0.pdf
>
>
> There are several drawbacks in the original MOB 1.0 (Moderate Object
> Storage) implementation, which can limit the adoption of the MOB feature:
> # MOB compactions are executed in a Master as a chore, which limits
> scalability because all I/O goes through a single HBase Master server.
> # Yarn/Mapreduce framework is required to run MOB compactions in a scalable
> way, but this won’t work in a stand-alone HBase cluster.
> # Two separate compactors for MOB and for regular store files and their
> interactions can result in a data loss (see HBASE-22075)
> The design goals for MOB 2.0 were to provide 100% MOB 1.0 - compatible
> implementation, which is free of the above drawbacks and can be used as a
> drop in replacement in existing MOB deployments. So, these are design goals
> of a MOB 2.0:
> # Make MOB compactions scalable without relying on Yarn/Mapreduce framework
> # Provide unified compactor for both MOB and regular store files
> # Make it more robust especially w.r.t. to data losses.
> # Simplify and reduce the overall MOB code.
> # Provide 100% compatible implementation with MOB 1.0.
> # No migration of data should be required between MOB 1.0 and MOB 2.0 - just
> software upgrade.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22749) Distributed MOB compactions
[
https://issues.apache.org/jira/browse/HBASE-22749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17037661#comment-17037661
]
HBase QA commented on HBASE-22749:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
43s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 23 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
0s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 6m
23s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
1s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 5m
7s{color} | {color:blue} Used deprecated FindBugs config; considering switching
to SpotBugs. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
17s{color} | {color:blue} branch/hbase-checkstyle no findbugs output file
(findbugsXml.xml) {color} |
| {color:orange}-0{color} | {color:orange} patch {color} | {color:orange} 5m
38s{color} | {color:orange} Used diff version of patch file. Binary files and
potentially other changes not applied. Please rebase and squash commits if
necessary. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
59s{color} | {color:green} root: The patch generated 0 new + 427 unchanged - 58
fixed = 427 total (was 485) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 6m
25s{color} | {color:blue} patch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
1s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
20m 10s{color} | {color:green} Patch does not cause any errors with Hadoop
2.8.5 2.9.2 or 3.1.2. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
14s{color} | {color:blue} hbase-checkstyle has no data from findbugs {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit
[GitHub] [hbase] Apache-HBase commented on issue #921: HBASE-22749: Distributed MOB compactions
Apache-HBase commented on issue #921: HBASE-22749: Distributed MOB compactions URL: https://github.com/apache/hbase/pull/921#issuecomment-586650488 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 43s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 23 new or modified test files. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 41s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 7m 5s | master passed | | +1 :green_heart: | compile | 2m 25s | master passed | | +1 :green_heart: | checkstyle | 3m 0s | master passed | | +0 :ok: | refguide | 6m 23s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | shadedjars | 6m 1s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 37s | master passed | | +0 :ok: | spotbugs | 5m 7s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +0 :ok: | findbugs | 0m 17s | branch/hbase-checkstyle no findbugs output file (findbugsXml.xml) | | -0 :warning: | patch | 5m 38s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 6m 33s | the patch passed | | +1 :green_heart: | compile | 2m 24s | the patch passed | | +1 :green_heart: | javac | 2m 24s | the patch passed | | +1 :green_heart: | checkstyle | 2m 59s | root: The patch generated 0 new + 427 unchanged - 58 fixed = 427 total (was 485) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 4s | The patch has no ill-formed XML file. | | +0 :ok: | refguide | 6m 25s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | shadedjars | 6m 1s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 20m 10s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 1m 26s | the patch passed | | +0 :ok: | findbugs | 0m 14s | hbase-checkstyle has no data from findbugs | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 14s | hbase-checkstyle in the patch passed. | | +1 :green_heart: | unit | 1m 3s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 66m 16s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 1m 16s | hbase-it in the patch passed. | | +1 :green_heart: | asflicense | 2m 15s | The patch does not generate ASF License warnings. | | | | 165m 14s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-921/14/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/921 | | JIRA Issue | HBASE-22749 | | Optional Tests | dupname asflicense checkstyle javac javadoc unit xml refguide spotbugs findbugs shadedjars hadoopcheck hbaseanti compile | | uname | Linux 514b8f7656b0 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-921/out/precommit/personality/provided.sh | | git revision | master / ae781e569c | | Default Java | 1.8.0_181 | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-921/14/artifact/out/branch-site/book.html | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-921/14/artifact/out/patch-site/book.html | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-921/14/testReport/ | | Max. process+thread count | 8774 (vs. ulimit of 1) | | modules | C: hbase-checkstyle hbase-common hbase-server hbase-it U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-921/14/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
[jira] [Commented] (HBASE-23853) [Flakey Test] TestBlockEvictionFromClient#testBlockRefCountAfterSplits
[ https://issues.apache.org/jira/browse/HBASE-23853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17037653#comment-17037653 ] Michael Stack commented on HBASE-23853: --- I pushed a bit of debug and a wait on compactions to complete. Lets see if this helps. Leaving issue open for now. > [Flakey Test] TestBlockEvictionFromClient#testBlockRefCountAfterSplits > -- > > Key: HBASE-23853 > URL: https://issues.apache.org/jira/browse/HBASE-23853 > Project: HBase > Issue Type: Bug > Components: flakies >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-23853-Flakey-Test-TestBlockEvictionFromClient-.patch, Screen Shot > 2020-02-15 at 2.36.47 PM.png > > > Fails 12-25% of the time in the GCE build. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23853) [Flakey Test] TestBlockEvictionFromClient#testBlockRefCountAfterSplits
[ https://issues.apache.org/jira/browse/HBASE-23853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-23853: -- Fix Version/s: 2.3.0 3.0.0 > [Flakey Test] TestBlockEvictionFromClient#testBlockRefCountAfterSplits > -- > > Key: HBASE-23853 > URL: https://issues.apache.org/jira/browse/HBASE-23853 > Project: HBase > Issue Type: Bug > Components: flakies >Reporter: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: > 0001-HBASE-23853-Flakey-Test-TestBlockEvictionFromClient-.patch, Screen Shot > 2020-02-15 at 2.36.47 PM.png > > > Fails 12-25% of the time in the GCE build. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23853) [Flakey Test] TestBlockEvictionFromClient#testBlockRefCountAfterSplits
[ https://issues.apache.org/jira/browse/HBASE-23853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-23853: -- Attachment: 0001-HBASE-23853-Flakey-Test-TestBlockEvictionFromClient-.patch > [Flakey Test] TestBlockEvictionFromClient#testBlockRefCountAfterSplits > -- > > Key: HBASE-23853 > URL: https://issues.apache.org/jira/browse/HBASE-23853 > Project: HBase > Issue Type: Bug > Components: flakies >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-23853-Flakey-Test-TestBlockEvictionFromClient-.patch, Screen Shot > 2020-02-15 at 2.36.47 PM.png > > > Fails 12-25% of the time in the GCE build. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23853) [Flakey Test] TestBlockEvictionFromClient#testBlockRefCountAfterSplits
[ https://issues.apache.org/jira/browse/HBASE-23853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17037651#comment-17037651 ] Michael Stack commented on HBASE-23853: --- !Screen Shot 2020-02-15 at 2.36.47 PM.png! > [Flakey Test] TestBlockEvictionFromClient#testBlockRefCountAfterSplits > -- > > Key: HBASE-23853 > URL: https://issues.apache.org/jira/browse/HBASE-23853 > Project: HBase > Issue Type: Bug > Components: flakies >Reporter: Michael Stack >Priority: Major > Attachments: Screen Shot 2020-02-15 at 2.36.47 PM.png > > > Fails 12-25% of the time in the GCE build. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23853) [Flakey Test] TestBlockEvictionFromClient#testBlockRefCountAfterSplits
[ https://issues.apache.org/jira/browse/HBASE-23853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-23853: -- Attachment: Screen Shot 2020-02-15 at 2.36.47 PM.png > [Flakey Test] TestBlockEvictionFromClient#testBlockRefCountAfterSplits > -- > > Key: HBASE-23853 > URL: https://issues.apache.org/jira/browse/HBASE-23853 > Project: HBase > Issue Type: Bug > Components: flakies >Reporter: Michael Stack >Priority: Major > Attachments: Screen Shot 2020-02-15 at 2.36.47 PM.png > > > Fails 12-25% of the time in the GCE build. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23853) [Flakey Test] TestBlockEvictionFromClient#testBlockRefCountAfterSplits
Michael Stack created HBASE-23853: - Summary: [Flakey Test] TestBlockEvictionFromClient#testBlockRefCountAfterSplits Key: HBASE-23853 URL: https://issues.apache.org/jira/browse/HBASE-23853 Project: HBase Issue Type: Bug Components: flakies Reporter: Michael Stack Fails 12-25% of the time in the GCE build. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23282) HBCKServerCrashProcedure for 'Unknown Servers'
[
https://issues.apache.org/jira/browse/HBASE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17037637#comment-17037637
]
Michael Stack commented on HBASE-23282:
---
[~jfrabaute] ok.
FYI, edit of hbase:meta directly is problematic because master does not 'learn'
of the edit made. It writes all state to hbase:meta so doesn't expect it has to
read it.
On running scheduleProcedureRecoveries, you should at least see the procedure
registered in the master log.
Good you try serveral times. HBCKSCP will only do extra repair if straight-SCP
finds nothing to be done.
Yeah, recover w/ the bulk load. It should show in the HBCK Report as orphan.
Need to figure how this condition comes about. It was happening in tests here
because i was overdriving the cluster but had fixed most problem-causing
conditions.
Why it happens, need to figure.
The Region is in the OPENING state but the Region it is OPENING against is gone
for whatever reason (Restart of a server will clear the old instance from the
dead servers list making it a no-longer 'known' server). The HBCKSCP is
explicitly for this case where hbase:meta has a reference but the server does
not exist anymore. You didn't see... any logs with:
" LOG.info("Found {} mentions of {} in hbase:meta of OPEN/OPENING Regions:
{}","
odd that CatalogJanitor could see the server but HBCKSCP could not.
You did regenerate the HBCK Report after running HBCKSCP a few times?
Otherwise, HBCKSCP fixes the issue but the report is stale until CatalogJanitor
is re-run.
Thanks.
> HBCKServerCrashProcedure for 'Unknown Servers'
> --
>
> Key: HBASE-23282
> URL: https://issues.apache.org/jira/browse/HBASE-23282
> Project: HBase
> Issue Type: Bug
> Components: hbck2, proc-v2
>Affects Versions: 2.2.2
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.2.3
>
>
> With an overdriving, sustained load, I can fairly easily manufacture an
> hbase:meta table that references servers that are no longer in the live list
> nor are members of deadservers; i.e. 'Unknown Servers'. The new 'HBCK
> Report' UI in Master has a section where it lists 'Unknown Servers' if any in
> hbase:meta.
> Once in this state, the repair is awkward. Our assign/unassign Procedure is
> particularly dogged about insisting that we confirm close/open of Regions
> when it is going about its business which is well and good if server is in
> live/dead sets but when an 'Unknown Server', we invariably end up trying to
> confirm against a non-longer present server (More on this in follow-on
> issues).
> What is wanted is queuing of a ServerCrashProcedure for each 'Unknown
> Server'. It would split any WALs (there shouldn't be any if server was
> restarted) and ideally it would cancel out any assigns and reassign regions
> off the 'Unknown Server'. But the 'normal' SCP consults the in-memory
> cluster state figuring what Regions were on the crashed server... And
> 'Unknown Servers' don't have state in in-master memory Maps of Servers to
> Regions or in DeadServers list which works fine for the usual case.
> Suggestion here is that hbck2 be able to drive in a special SCP, one which
> would get list of Regions by scanning hbase:meta rather than asking Master
> memory; an HBCKSCP.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] HorizonNet merged pull request #1168: HBASE-23837 Removed deprecated isLegalFamilyName(byte[]) from HColumnDescriptor
HorizonNet merged pull request #1168: HBASE-23837 Removed deprecated isLegalFamilyName(byte[]) from HColumnDescriptor URL: https://github.com/apache/hbase/pull/1168 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-18095) Provide an option for clients to find the server hosting META that does not involve the ZooKeeper client
[
https://issues.apache.org/jira/browse/HBASE-18095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17037582#comment-17037582
]
Hudson commented on HBASE-18095:
Results for branch HBASE-18095/client-locate-meta-no-zookeeper
[build #77 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18095%252Fclient-locate-meta-no-zookeeper/77/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18095%252Fclient-locate-meta-no-zookeeper/77//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18095%252Fclient-locate-meta-no-zookeeper/77//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18095%252Fclient-locate-meta-no-zookeeper/77//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Provide an option for clients to find the server hosting META that does not
> involve the ZooKeeper client
>
>
> Key: HBASE-18095
> URL: https://issues.apache.org/jira/browse/HBASE-18095
> Project: HBase
> Issue Type: New Feature
> Components: Client
>Reporter: Andrew Kyle Purtell
>Assignee: Bharath Vissapragada
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 1.7.0
>
> Attachments: HBASE-18095.master-v1.patch, HBASE-18095.master-v2.patch
>
>
> Clients are required to connect to ZooKeeper to find the location of the
> regionserver hosting the meta table region. Site configuration provides the
> client a list of ZK quorum peers and the client uses an embedded ZK client to
> query meta location. Timeouts and retry behavior of this embedded ZK client
> are managed orthogonally to HBase layer settings and in some cases the ZK
> cannot manage what in theory the HBase client can, i.e. fail fast upon outage
> or network partition.
> We should consider new configuration settings that provide a list of
> well-known master and backup master locations, and with this information the
> client can contact any of the master processes directly. Any master in either
> active or passive state will track meta location and respond to requests for
> it with its cached last known location. If this location is stale, the client
> can ask again with a flag set that requests the master refresh its location
> cache and return the up-to-date location. Every client interaction with the
> cluster thus uses only HBase RPC as transport, with appropriate settings
> applied to the connection. The configuration toggle that enables this
> alternative meta location lookup should be false by default.
> This removes the requirement that HBase clients embed the ZK client and
> contact the ZK service directly at the beginning of the connection lifecycle.
> This has several benefits. ZK service need not be exposed to clients, and
> their potential abuse, yet no benefit ZK provides the HBase server cluster is
> compromised. Normalizing HBase client and ZK client timeout settings and
> retry behavior - in some cases, impossible, i.e. for fail-fast - is no longer
> necessary.
> And, from [~ghelmling]: There is an additional complication here for
> token-based authentication. When a delegation token is used for SASL
> authentication, the client uses the cluster ID obtained from Zookeeper to
> select the token identifier to use. So there would also need to be some
> Zookeeper-less, unauthenticated way to obtain the cluster ID as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #1173: HBASE-23851 Log networks and bind addresses when multicast publisher/…
Apache-HBase commented on issue #1173: HBASE-23851 Log networks and bind addresses when multicast publisher/… URL: https://github.com/apache/hbase/pull/1173#issuecomment-586634738 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 12s | branch-2 passed | | +1 :green_heart: | compile | 1m 49s | branch-2 passed | | +1 :green_heart: | checkstyle | 2m 11s | branch-2 passed | | +1 :green_heart: | shadedjars | 4m 16s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 23s | branch-2 passed | | +0 :ok: | spotbugs | 3m 27s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 5m 25s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 56s | the patch passed | | +1 :green_heart: | compile | 1m 49s | the patch passed | | +1 :green_heart: | javac | 1m 49s | the patch passed | | -1 :x: | checkstyle | 0m 34s | hbase-client: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | -1 :x: | checkstyle | 1m 7s | hbase-server: The patch generated 1 new + 104 unchanged - 0 fixed = 105 total (was 104) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 4m 19s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 15m 30s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 1m 23s | the patch passed | | +1 :green_heart: | findbugs | 5m 26s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 8s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 2m 12s | hbase-client in the patch passed. | | -1 :x: | unit | 65m 55s | hbase-server in the patch failed. | | +1 :green_heart: | asflicense | 1m 44s | The patch does not generate ASF License warnings. | | | | 134m 50s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.client.replication.TestReplicationAdminWithTwoDifferentZKClusters | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1173/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1173 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 5fda13476a0b 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1173/out/precommit/personality/provided.sh | | git revision | branch-2 / bcbeb8ec6d | | Default Java | 1.8.0_181 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1173/1/artifact/out/diff-checkstyle-hbase-client.txt | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1173/1/artifact/out/diff-checkstyle-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1173/1/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1173/1/testReport/ | | Max. process+thread count | 7873 (vs. ulimit of 1) | | modules | C: hbase-common hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1173/1/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated.
[jira] [Created] (HBASE-23852) [Documentation] Doc the dead servers multicast feature
Michael Stack created HBASE-23852: - Summary: [Documentation] Doc the dead servers multicast feature Key: HBASE-23852 URL: https://issues.apache.org/jira/browse/HBASE-23852 Project: HBase Issue Type: Bug Components: documentation Reporter: Michael Stack Setting a config on master enables the multicast feature where master will udp broadcast to all listeners -- clients -- when it learns of dead servers. Clients that get notification react by redoing their region locations cache which helps them be more response when servers die. This feature has been around for years but probably goes unknown since no mention in documentation. This issue is about adding a basic paragraph on what it does and how to use it. The original issue has a nice release note that could serve as the basis for this paragraph. See HBASE-7590 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack opened a new pull request #1173: HBASE-23851 Log networks and bind addresses when multicast publisher/…
saintstack opened a new pull request #1173: HBASE-23851 Log networks and bind addresses when multicast publisher/… URL: https://github.com/apache/hbase/pull/1173 …listener enabled This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (HBASE-23851) Log networks and bind addresses when multicast publisher/listener enabled
Michael Stack created HBASE-23851: - Summary: Log networks and bind addresses when multicast publisher/listener enabled Key: HBASE-23851 URL: https://issues.apache.org/jira/browse/HBASE-23851 Project: HBase Issue Type: Sub-task Reporter: Michael Stack Log details on publisher and listeners at DEBUG level so can figure issues when client and server do not match. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23850) [Flakey Test] TestAsyncTableRSCrashPublish times out VPN enabled
Michael Stack created HBASE-23850: - Summary: [Flakey Test] TestAsyncTableRSCrashPublish times out VPN enabled Key: HBASE-23850 URL: https://issues.apache.org/jira/browse/HBASE-23850 Project: HBase Issue Type: Bug Reporter: Michael Stack This is an odd one. If I put up VPN on my local machine, this TestAsyncTableRSCrashPublish test times out. If I take down the VPN, the test passes. This is actually an awkward issue to trace since the udp multicast stuff doesn't log much about its operation. Digging, the VPN adds a utun8 network interface to my box. When this eighth interface comes on line, the TestAsyncTableRSCrashPublish test will select this new interface, which is ipv4, for the master to publish events on, but the clients will choose the seventh utun interface, which is ipv6, for listening. If no vpn, both pick an en?? interface and all just works. I've messed around with the test and tried having the master add the networkinterface its chosen to the configuration, and then had the clients use same network interface, but somehow clients insist on listening on ipv6. No matter how I mess -- even insisting on ipv4 on client and server (see coming attached code), the clients don't get the master broadcast. Attached code is my hacking trying to make sense of this. Let me make a trivial subtask to add the logging at least in case someone runs into this in prod, they'll have a chance for figuring out mismatched networking (currently there is no way to figure w/o changing code). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #754: HBASE-22978 : Online slow response log
Apache-HBase commented on issue #754: HBASE-22978 : Online slow response log URL: https://github.com/apache/hbase/pull/754#issuecomment-586593501 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 37s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 10s | master passed | | +1 :green_heart: | compile | 3m 7s | master passed | | +1 :green_heart: | checkstyle | 2m 25s | master passed | | +0 :ok: | refguide | 4m 50s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | shadedjars | 4m 38s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 5m 7s | master passed | | +0 :ok: | spotbugs | 1m 23s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 23m 25s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 5s | the patch passed | | +1 :green_heart: | compile | 3m 4s | the patch passed | | +1 :green_heart: | cc | 3m 4s | the patch passed | | +1 :green_heart: | javac | 3m 4s | the patch passed | | +1 :green_heart: | checkstyle | 2m 23s | root: The patch generated 0 new + 448 unchanged - 2 fixed = 448 total (was 450) | | +1 :green_heart: | rubocop | 0m 9s | There were no new rubocop issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +0 :ok: | refguide | 4m 40s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | shadedjars | 4m 41s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 15m 52s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | hbaseprotoc | 8m 29s | the patch passed | | +1 :green_heart: | javadoc | 5m 17s | the patch passed | | +1 :green_heart: | findbugs | 24m 39s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 99m 7s | root in the patch failed. | | +1 :green_heart: | asflicense | 4m 8s | The patch does not generate ASF License warnings. | | | | 235m 56s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-754/14/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/754 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile refguide xml cc hbaseprotoc prototool rubocop | | uname | Linux eb6c9ef5e8be 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-754/out/precommit/personality/provided.sh | | git revision | master / 0e96638898 | | Default Java | 1.8.0_181 | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-754/14/artifact/out/branch-site/book.html | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-754/14/artifact/out/patch-site/book.html | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-754/14/artifact/out/patch-unit-root.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-754/14/testReport/ | | Max. process+thread count | 8828 (vs. ulimit of 1) | | modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-server hbase-thrift hbase-shell . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-754/14/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 rubocop=0.79.0 | | Powered by | Apache Yetus 0.11.1
[GitHub] [hbase] virajjasani commented on issue #754: HBASE-22978 : Online slow response log
virajjasani commented on issue #754: HBASE-22978 : Online slow response log URL: https://github.com/apache/hbase/pull/754#issuecomment-586575972 Using thread-safe wrapper over EvictingQueue makes life simpler, however it does not have any perf improvement because earlier I was using lock to write data in array in thread safe manner. However, using it has good advantages over not losing any writes specifically if manual locks time out or get InterruptedException, and of course less manual code to maintain. Thanks @bharathv for the suggestion. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log
virajjasani commented on a change in pull request #754: HBASE-22978 : Online
slow response log
URL: https://github.com/apache/hbase/pull/754#discussion_r379822605
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
##
@@ -496,11 +518,65 @@ void logResponse(Message param, String methodName,
String call, String tag,
}
}
}
- responseInfo.put("multi.gets", numGets);
- responseInfo.put("multi.mutations", numMutations);
- responseInfo.put("multi.servicecalls", numServiceCalls);
+ responseInfo.put(MULTI_GETS, numGets);
+ responseInfo.put(MULTI_MUTATIONS, numMutations);
+ responseInfo.put(MULTI_SERVICE_CALLS, numServiceCalls);
}
+final String tag = (tooLarge && tooSlow) ? "TooLarge & TooSlow"
+ : (tooSlow ? "TooSlow" : "TooLarge");
LOG.warn("(response" + tag + "): " + GSON.toJson(responseInfo));
+if (tooSlow && this.slowLogRecorder != null) {
+ logOnlineSlowResponse(param, methodName, call, clientAddress, startTime,
+processingTime, qTime, responseSize, userName, className,
responseInfo);
+}
+ }
+
+ /**
+ * Add too slow log to ringbuffer for retrieval of latest n slow logs
+ *
+ * @param param The parameters received in the call
+ * @param methodName The name of the method invoked
+ * @param call The string representation of the call
+ * @param clientAddress The address of the client who made this call
+ * @param startTime The time that the call was initiated, in ms
+ * @param processingTime The duration that the call took to run, in ms
+ * @param qTime The duration that the call spent on the queue
+ * prior to being initiated, in ms
+ * @param responseSize The size in bytes of the response buffer
+ * @param userName UserName of the current RPC Call
+ * @param className ClassName of the SlowLog call
+ * @param responseInfo Base information map that is reported regardless of
type of call
+ */
+ private void logOnlineSlowResponse(Message param, String methodName, String
call,
Review comment:
I thought of this some time back and somehow lost it, Thanks for the
reminder. Updated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log
virajjasani commented on a change in pull request #754: HBASE-22978 : Online
slow response log
URL: https://github.com/apache/hbase/pull/754#discussion_r379820239
##
File path:
hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/client/ThriftAdmin.java
##
@@ -1157,6 +1158,18 @@ public boolean isSnapshotCleanupEnabled() {
throw new NotImplementedException("isSnapshotCleanupEnabled not supported
in ThriftAdmin");
}
+ @Override
+ public List getSlowLogResponses(final List
serverNames)
+ throws IOException {
+throw new NotImplementedException("getSlowLogResponses not supported in
ThriftAdmin");
Review comment:
Sure, I don't mind taking this up, but in a follow up task because we are
not promising thrift support in main Jira, will take this up in follow up task.
Thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log
virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log URL: https://github.com/apache/hbase/pull/754#discussion_r379769648 ## File path: hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java ## @@ -2262,4 +2262,30 @@ boolean snapshotCleanupSwitch(final boolean on, final boolean synchronous) */ boolean isSnapshotCleanupEnabled() throws IOException; + + /** + * Retrieves online slowlog responses from in memory ringbuffer maintained by + * RegionServers + * + * @param serverNames Server names to get slowlog responses from + * @param slowLogQueryFilter filter to be used if provided + * @return online slowlog response list + * @throws IOException if a remote or network exception occurs + */ + List getSlowLogResponses(final List serverNames, Review comment: Actually, these names were different in my first patch, then I changed it based on PR review and also based on suggestion on Jira(~4 months back), I think more time was spent with this and now it feels like good to maintain parity between shell command and Admin API naming. Hence, I don't feel comfortable changing it now :) Let me update JavaDocs accordingly and mention about slow RPC logs. Hope it is fine with you @bharathv This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log
virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log URL: https://github.com/apache/hbase/pull/754#discussion_r379774633 ## File path: hbase-common/src/main/resources/hbase-default.xml ## @@ -1956,4 +1956,27 @@ possible configurations would overwhelm and obscure the important. enable this feature. + +hbase.regionserver.slowlog.ringbuffer.size +256 + + Default size of ringbuffer to be maintained by each RegionServer in order + to store online slowlog responses. This is an in-memory ring buffer of + requests that were judged to be too slow in addition to the responseTooSlow + logging. The in-memory representation would be complete. + For more details, please look into Doc Section: + Get Slow Response Log from shell + + + +hbase.regionserver.slowlog.buffer.enabled Review comment: On one hand I agree with this and this is something I have implemented in past too but on the other hand, user would not want to change `hbase.regionserver.slowlog.ringbuffer.size` that frequently. The default value for the config is not too high, not too low either. Hence, the reason to have enable/disable config is to ensure we provide reasonable default value to ringbuffer size, something user doesn't have to spend much time with and let user enable this config without touching ringbuffer size. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log
virajjasani commented on a change in pull request #754: HBASE-22978 : Online
slow response log
URL: https://github.com/apache/hbase/pull/754#discussion_r379771274
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncAdmin.java
##
@@ -1507,4 +1507,24 @@
*/
CompletableFuture isSnapshotCleanupEnabled();
+ /**
+ * Retrieves online slowlog responses from in memory ringbuffer maintained by
+ * RegionServers
+ *
+ * @param serverNames Server names to get slowlog responses from
+ * @param slowLogQueryFilter filter to be used if provided
+ * @return Online slowlog response list. The return value wrapped by a
{@link CompletableFuture}
+ */
+ CompletableFuture> getSlowLogResponses(final
List serverNames,
Review comment:
Since we are maintaining ringbuffer, we are not really much interested with
duplicates and even some missing events. Eventually, for any slower RPC calls,
these APIs should provide hints good enough for user to perform immediate
remediations specifically when system is really slow. Duplicates are fine but
we are less likely to get duplicates unless RpcServer itself sends duplicate
events. The reason why we should not prefer Set<> is because we want to
maintain order also, better to display latest records first.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log
virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log URL: https://github.com/apache/hbase/pull/754#discussion_r379774953 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java ## @@ -85,6 +90,12 @@ protected static final CallQueueTooBigException CALL_QUEUE_TOO_BIG_EXCEPTION = new CallQueueTooBigException(); + private static final String MULTI_GETS = "multi.gets"; Review comment: but these are not being referred to anywhere else, let's keep them here only :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log
virajjasani commented on a change in pull request #754: HBASE-22978 : Online
slow response log
URL: https://github.com/apache/hbase/pull/754#discussion_r379772397
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/SlowLogParams.java
##
@@ -0,0 +1,89 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.client;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.builder.EqualsBuilder;
+import org.apache.commons.lang3.builder.HashCodeBuilder;
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * SlowLog params object that contains detailed info as params and region name
: to be used
+ * for filter purpose
+ */
+@InterfaceAudience.Private
+public class SlowLogParams {
Review comment:
`log` part is to remind developer that this is RPC slow logs, as per my
other comment else where, even this was discussed somewhere in my earlier
patches. I really intend to keep it as is but let me update javadoc with `RPC
log` info.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log
virajjasani commented on a change in pull request #754: HBASE-22978 : Online
slow response log
URL: https://github.com/apache/hbase/pull/754#discussion_r379772997
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/SlowLogRecord.java
##
@@ -0,0 +1,317 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.client;
+
+import org.apache.commons.lang3.builder.EqualsBuilder;
+import org.apache.commons.lang3.builder.HashCodeBuilder;
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.hadoop.hbase.util.GsonUtil;
+import org.apache.yetus.audience.InterfaceAudience;
+
+import org.apache.hbase.thirdparty.com.google.gson.Gson;
+import org.apache.hbase.thirdparty.com.google.gson.JsonObject;
+import org.apache.hbase.thirdparty.com.google.gson.JsonSerializer;
+
+/**
+ * SlowLog payload for client
+ */
+@InterfaceAudience.Private
+final public class SlowLogRecord {
+
+ private static final Gson GSON = GsonUtil.createGson()
+.setPrettyPrinting()
+.registerTypeAdapter(SlowLogRecord.class, (JsonSerializer)
+ (slowLogPayload, type, jsonSerializationContext) -> {
+Gson gson = new Gson();
+JsonObject jsonObj = (JsonObject) gson.toJsonTree(slowLogPayload);
+if (slowLogPayload.getMultiGetsCount() == 0) {
+ jsonObj.remove("multiGetsCount");
+}
+if (slowLogPayload.getMultiMutationsCount() == 0) {
+ jsonObj.remove("multiMutationsCount");
+}
+if (slowLogPayload.getMultiServiceCalls() == 0) {
+ jsonObj.remove("multiServiceCalls");
+}
+return jsonObj;
+ }).create();
+
+ private long startTime;
+ private int processingTime;
+ private int queueTime;
+ private long responseSize;
+ private String clientAddress;
+ private String serverClass;
+ private String methodName;
+ private String callDetails;
+ private String param;
+ // we don't want to serialize region name, it is just for the filter purpose
+ // hence avoiding deserialization
+ private transient String regionName;
+ private String userName;
+ private int multiGetsCount;
+ private int multiMutationsCount;
+ private int multiServiceCalls;
+
+ public long getStartTime() {
+return startTime;
+ }
+
+ public int getProcessingTime() {
+return processingTime;
+ }
+
+ public int getQueueTime() {
+return queueTime;
+ }
+
+ public long getResponseSize() {
+return responseSize;
+ }
+
+ public String getClientAddress() {
+return clientAddress;
+ }
+
+ public String getServerClass() {
+return serverClass;
+ }
+
+ public String getMethodName() {
+return methodName;
+ }
+
+ public String getCallDetails() {
+return callDetails;
+ }
+
+ public String getParam() {
+return param;
+ }
+
+ public String getRegionName() {
+return regionName;
+ }
+
+ public String getUserName() {
+return userName;
+ }
+
+ public int getMultiGetsCount() {
+return multiGetsCount;
+ }
+
+ public int getMultiMutationsCount() {
+return multiMutationsCount;
+ }
+
+ public int getMultiServiceCalls() {
+return multiServiceCalls;
+ }
+
+ private SlowLogRecord(final long startTime, final int processingTime, final
int queueTime,
Review comment:
This is used by builder only, private constructor being used in `build()`:
```
public SlowLogRecord build() {
return new SlowLogRecord(startTime, processingTime, queueTime,
responseSize,
clientAddress, serverClass, methodName, callDetails, param,
regionName,
userName, multiGetsCount, multiMutationsCount, multiServiceCalls);
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
