[GitHub] [hbase] Apache-HBase commented on pull request #1724: HBASE-24386 TableSnapshotScanner support scan limit
Apache-HBase commented on pull request #1724: URL: https://github.com/apache/hbase/pull/1724#issuecomment-629747495 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 20s | master passed | | +1 :green_heart: | checkstyle | 1m 4s | master passed | | +1 :green_heart: | spotbugs | 1m 55s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 19s | the patch passed | | +1 :green_heart: | checkstyle | 1m 4s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 3s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 8s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 31m 46s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1724 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 10025e107d9c 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1724: HBASE-24386 TableSnapshotScanner support scan limit
Apache-HBase commented on pull request #1724: URL: https://github.com/apache/hbase/pull/1724#issuecomment-629744988 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 42s | master passed | | +1 :green_heart: | compile | 0m 56s | master passed | | +1 :green_heart: | shadedjars | 5m 32s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 26s | the patch passed | | +1 :green_heart: | compile | 0m 56s | the patch passed | | +1 :green_heart: | javac | 0m 56s | the patch passed | | +1 :green_heart: | shadedjars | 5m 32s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 133m 21s | hbase-server in the patch passed. | | | | 157m 18s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1724 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 6a5a6d6399b8 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/1/testReport/ | | Max. process+thread count | 4681 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1724: HBASE-24386 TableSnapshotScanner support scan limit
Apache-HBase commented on pull request #1724: URL: https://github.com/apache/hbase/pull/1724#issuecomment-629744652 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 11s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 49s | master passed | | +1 :green_heart: | compile | 1m 21s | master passed | | +1 :green_heart: | shadedjars | 6m 33s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 42s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 6s | the patch passed | | +1 :green_heart: | compile | 1m 4s | the patch passed | | +1 :green_heart: | javac | 1m 4s | the patch passed | | +1 :green_heart: | shadedjars | 5m 56s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 40s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 123m 55s | hbase-server in the patch passed. | | | | 153m 29s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1724 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 3c75d6dfd955 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/1/testReport/ | | Max. process+thread count | 4588 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24387) TableSnapshotInputFormatImpl support scan limit on each InputSplit
[ https://issues.apache.org/jira/browse/HBASE-24387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin updated HBASE-24387: - Description: In some scenario , We want to scan limited rows on each InputSplit for sampling verification > TableSnapshotInputFormatImpl support scan limit on each InputSplit > -- > > Key: HBASE-24387 > URL: https://issues.apache.org/jira/browse/HBASE-24387 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: niuyulin >Assignee: niuyulin >Priority: Major > > In some scenario , We want to scan limited rows on each InputSplit for > sampling verification -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24387) TableSnapshotInputFormatImpl support scan limit on each InputSplit
niuyulin created HBASE-24387: Summary: TableSnapshotInputFormatImpl support scan limit on each InputSplit Key: HBASE-24387 URL: https://issues.apache.org/jira/browse/HBASE-24387 Project: HBase Issue Type: Improvement Components: mapreduce Reporter: niuyulin Assignee: niuyulin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1724: HBASE-24386 TableSnapshotScanner support scan limit
Apache-HBase commented on pull request #1724: URL: https://github.com/apache/hbase/pull/1724#issuecomment-629736234 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 39s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 44s | master passed | | +1 :green_heart: | checkstyle | 1m 7s | master passed | | +1 :green_heart: | spotbugs | 1m 58s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 24s | the patch passed | | -0 :warning: | checkstyle | 1m 3s | hbase-server: The patch generated 4 new + 1 unchanged - 0 fixed = 5 total (was 1) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 8s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 5s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 33m 31s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1724 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 26ffc0c0e299 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] nyl3532016 opened a new pull request #1724: HBASE-24386 TableSnapshotScanner support scan limit
nyl3532016 opened a new pull request #1724: URL: https://github.com/apache/hbase/pull/1724 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-24386) TableSnapshotScanner support scan limit
niuyulin created HBASE-24386: Summary: TableSnapshotScanner support scan limit Key: HBASE-24386 URL: https://issues.apache.org/jira/browse/HBASE-24386 Project: HBase Issue Type: Improvement Components: Scanners, snapshots Reporter: niuyulin Assignee: niuyulin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24271) Set values in `conf/hbase-site.xml` that enable running on `LocalFileSystem` out of the box
[
https://issues.apache.org/jira/browse/HBASE-24271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109272#comment-17109272
]
Hudson commented on HBASE-24271:
Results for branch branch-2
[build #2664 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2664/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2664/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2664/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2664/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2664/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Set values in `conf/hbase-site.xml` that enable running on `LocalFileSystem`
> out of the box
> ---
>
> Key: HBASE-24271
> URL: https://issues.apache.org/jira/browse/HBASE-24271
> Project: HBase
> Issue Type: Task
>Affects Versions: 3.0.0-alpha-1, 2.3.0, 1.7.0, 2.2.4
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 1.7.0, 2.2.5
>
>
> This ticket is to implement the changes as described on the [discussion on
> dev|https://lists.apache.org/thread.html/r089de243a9bc9d923fa07c81e6bc825b82be68f567b892a342a0c61f%40%3Cdev.hbase.apache.org%3E].
> It reverts and supersedes changes made on HBASE-24086 and HBASE-24106.
> {quote}
> The conclusion I understand from this thread looks something like this:
> 1. revert HBASE-24086, make it so that running on `LocalFileSystem` is a
> fatal condition with default configs.
> 2. ship a conf/hbase-site.xml that contains
> hbase.unsafe.stream.capability.enforce=false, along with a big comment saying
> this is not safe for production.
> 3. ship a conf/hbase-site.xml that contains hbase.tmp.dir=./tmp, along with a
> comment saying herein you'll find temporary and persistent data, reconfigure
> for production with hbase.rootdir pointed to a durable filesystem that
> supports our required stream capabilities (see above).
> 4. update HBASE-24106 as appropriate.
> Neither 2 nor 3 are suitable for production deployments, thus the changes do
> not go into hbase-default.xml. Anyone standing up a production deploy must
> edit hbase-site.xml anyway, so this doesn't change anything. It also restores
> our "simple" first-time user experience of not needing to run anything
> besides `bin/start-hbase.sh` (or `bin/hbase master start`, or whatever it is
> we're telling people these days).
> We can reassess this once more when a durable equivalent to LocalFileSystem
> comes along.
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24368) Let HBCKSCP clear 'Unknown Servers', even if RegionStateNode has RegionLocation == null
[
https://issues.apache.org/jira/browse/HBASE-24368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109269#comment-17109269
]
Hudson commented on HBASE-24368:
Results for branch branch-2
[build #2664 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2664/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2664/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2664/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2664/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2664/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Let HBCKSCP clear 'Unknown Servers', even if RegionStateNode has
> RegionLocation == null
> ---
>
> Key: HBASE-24368
> URL: https://issues.apache.org/jira/browse/HBASE-24368
> Project: HBase
> Issue Type: Bug
> Components: hbck2
>Affects Versions: 2.3.0
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
>
> This is an incidental noticed when in a hole trying to fix up a cluster. The
> 'obvious' remediation didn't work. This issue is about addressing this.
> HBASE-23594 added a filtering of Regions on the crashed server to handle the
> case where an Assign may be concurrent to the ServerCrashProcedure. To avoid
> double assign, the SCP will skip assign if the RegionStateNode RegionLocation
> is not that of the crashed server.
> This is good.
> Where it is an obstacle is when a Region is stuck in OPENING state, it
> references an 'Unknown Server' -- a server no longer tracked by the Master --
> and there is no assign currently in flight. In this case, scheduling a
> ServerCrashProcedure to clean up the reference to the Unknown Server and to
> get the Region reassigned skips out when RegionStateNode in Master has a
> RegionLocation that does not match that of the ServerCrashProcedure, even
> when it is set to null (we set the RegionLocation to null when we fail an
> assign as we might if the server no longer is part of the cluster).
> For background, cluster had a RIT. The RIT was a Region failing to open
> because of a missing Reference (Another issue). The Region open would fail
> with a FileNotFoundException. The master would attempt assign and then would
> fail when it went to confirm OPEN, logging the complaint about FNFE asking
> for operator intervention in master logs.
> This state was in place for weeks on this particular cluster (a dev cluster
> not under close observation). The cluster had been restarted once or twice so
> the server the Region had once been on was no longer 'known' but it still had
> an entry in the hbase:meta table as last location assigned (The now 'Unknown
> Server').
> To fix, I went about the task in the wrong order. I bypassed the long-running
> stuck procedure to terminate it and cleanup 'Procedures and Locks'. Mistake.
> Now there was no longer an assign Procedure for this Region. But I now had a
> Region in OPENING state with a reference to an unknown server with an
> in-memory RegionStateNode whose RegionLocation was null (set null on each
> failed assign). Running catalogjanitor_run and hbck_chore_report had the
> unknown server show in the 'HBCK Report' in the 'Unknown Servers' list.
> Attempts at assign fail because Region is in OPENING state -- you can't
> assign a Region in OPENING state. Scheduling an HBCKSCP via hbck2
> scheduleRecoveries always generated the below in the logs.
> {code}
> org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure: pid=157217,
> state=RUNNABLE:SERVER_CRASH_ASSIGN, locked=true; HBCKServerCrashProcedure
> server=unknown_server.example.com,16020,1587577972683, splitWal=true,
> meta=false found a region state=OPENING, location=null,
> table=bobby_analytics, region=1501ea3bd822c1a3e4e6216ea48733bd which is no
> longer on us unknown_server.example.com,16020,1587577972683, give up
> assigning...
> {code}
> My workaround was setting region state to CLOSED with hbck2 and then doing an
>
[jira] [Commented] (HBASE-23702) `hbase.hstore.flusher.count` setting to 0 breaks HMaster
[
https://issues.apache.org/jira/browse/HBASE-23702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109273#comment-17109273
]
Hudson commented on HBASE-23702:
Results for branch branch-2
[build #2664 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2664/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2664/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2664/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2664/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2664/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> `hbase.hstore.flusher.count` setting to 0 breaks HMaster
>
>
> Key: HBASE-23702
> URL: https://issues.apache.org/jira/browse/HBASE-23702
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Environment: HBase-2.2.3
> Java openjdk 1.8.0_222
> Ubuntu 18.04
>Reporter: Ctest
>Assignee: Ctest
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 2.2.5
>
> Attachments: HBASE-23702-000.patch, HBASE-23702-000.patch,
> HBASE-23702-000.patch, HBASE-23702-000.patch, HBASE-23702-001.patch
>
>
> Summary
> The configuration parameter `hbase.hstore.flusher.count` allows `0` (which
> turns out to be a bad value after I understand the source code).
> When we set `hbase.hstore.flusher.count` to 0, HMaster will fail in 200,000
> ms without any error message, but
> ```
> java.lang.RuntimeException: Master not initialized after 20ms
> ```
>
> There’s no messages pinpointing `hbase.hstore.flusher.count` being the root
> cause.
>
> 0 means `MemStoreFlusher` is not alive. During the initialization time of
> HMaster, `isHealthy()` will be called and it will try to check if
> `MemStoreFlusher` is alive or not. However, since we set
> `hbase.hstore.flusher.count` to 0, there will be no flusher created at the
> first place, which cause`isHealthy()` to return false. As a consequence,
> HRegionServer does not enter its main loop.
>
> *How To Reproduce*
>
> The version of HBase: 2.2.3
>
> # Set the value of configuration parameter `hbase.hstore.flusher.count` to
> `0` in “conf/hbase-site.xml”
> # Start hbase by running “bin/start-hbase.sh”
> # Wait for 3-4 minutes
> # Open the log file “logs/hbase-- master -.log”
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24350) HBase table level replication metrics for shippedBytes are always 0
[
https://issues.apache.org/jira/browse/HBASE-24350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109270#comment-17109270
]
Hudson commented on HBASE-24350:
Results for branch branch-2
[build #2664 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2664/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2664/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2664/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2664/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2664/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> HBase table level replication metrics for shippedBytes are always 0
> ---
>
> Key: HBASE-24350
> URL: https://issues.apache.org/jira/browse/HBASE-24350
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Affects Versions: 3.0.0-alpha-1, master, 1.7.0, 2.4.0
>Reporter: Sandeep Pal
>Assignee: Sandeep Pal
>Priority: Major
> Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0
>
>
> It was observed during some investigations that table level metrics for
> shippedBytes are always 0 consistently even though data is getting shipped.
> There are two problems with table-level metrics:
> # There are no table-level metrics for shipped bytes.
> # Another problem is that it's using `MetricsReplicationSourceSourceImpl`
> which is creating all source-level metrics at table level as well but updated
> only ageOfLastShippedOp. This reports lot of false/incorrect replication
> metrics at table level.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24086) Disable output stream capability enforcement when running in standalone mode
[
https://issues.apache.org/jira/browse/HBASE-24086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109271#comment-17109271
]
Hudson commented on HBASE-24086:
Results for branch branch-2
[build #2664 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2664/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2664/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2664/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2664/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2664/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Disable output stream capability enforcement when running in standalone mode
>
>
> Key: HBASE-24086
> URL: https://issues.apache.org/jira/browse/HBASE-24086
> Project: HBase
> Issue Type: Task
> Components: master, Operability
>Affects Versions: 3.0.0-alpha-1, 2.3.0
>Reporter: Nick Dimiduk
>Priority: Critical
>
> {noformat}
> $
> JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home
> mvn clean install -DskipTests
> $
> JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home
> ./bin/hbase master start
> {noformat}
> gives
> {noformat}
> 2020-03-30 17:12:43,857 ERROR
> [master/192.168.111.13:16000:becomeActiveMaster] master.HMaster: Failed to
> become active master
>
> java.io.IOException: cannot get log writer
>
>
> at
> org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:118)
>
>
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createAsyncWriter(AsyncFSWAL.java:704)
>
>
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:710)
>
>
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:128)
>
>
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:839)
>
>
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:549)
>
>
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.init(AbstractFSWAL.java:490)
>
>
> at
> org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:156)
>
>
> at
> org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:61)
>
>
> at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:297)
>
>
> at
> org.apache.hadoop.hbase.procedure2.store.region.RegionProcedureStore.createWAL(RegionProcedureStore.java:256)
>
>
> at
>
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Apache-HBase commented on pull request #1620: URL: https://github.com/apache/hbase/pull/1620#issuecomment-629727207 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 54s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 8s | Maven dependency ordering for patch | ||| _ Other Tests _ | | | | 3m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1620 | | Optional Tests | | | uname | Linux 9be806ee7c4b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | Max. process+thread count | 52 (vs. ulimit of 12500) | | modules | C: U: | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Apache-HBase commented on pull request #1620: URL: https://github.com/apache/hbase/pull/1620#issuecomment-629727114 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 8s | Maven dependency ordering for patch | | +1 :green_heart: | hadolint | 0m 3s | There were no new hadolint issues. | | -0 :warning: | shellcheck | 0m 2s | The patch generated 6 new + 5 unchanged - 0 fixed = 11 total (was 5) | | -0 :warning: | whitespace | 0m 0s | The patch 6 line(s) with tabs. | ||| _ Other Tests _ | | +0 :ok: | asflicense | 0m 0s | ASF License check generated no output? | | | | 2m 0s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1620 | | Optional Tests | dupname asflicense shellcheck shelldocs hadolint | | uname | Linux f1a0a30f2407 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | shellcheck | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/4/artifact/yetus-general-check/output/diff-patch-shellcheck.txt | | whitespace | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/4/artifact/yetus-general-check/output/whitespace-tabs.txt | | Max. process+thread count | 47 (vs. ulimit of 12500) | | modules | C: U: | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) shellcheck=0.4.6 hadolint=1.17.5-0-g443423c | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Apache-HBase commented on pull request #1620: URL: https://github.com/apache/hbase/pull/1620#issuecomment-629726953 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | docker | 0m 0s | Docker command '/usr/bin/docker' is too old ( < 17.0). | | Subsystem | Report/Notes | |--:|:-| | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/4/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1681: HBASE-23938 : System table hbase:slowlog to store complete slow/large…
Apache-HBase commented on pull request #1681: URL: https://github.com/apache/hbase/pull/1681#issuecomment-629722569 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 40s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 58s | master passed | | +1 :green_heart: | compile | 1m 23s | master passed | | +1 :green_heart: | shadedjars | 6m 12s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 56s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 5s | the patch passed | | +1 :green_heart: | compile | 1m 21s | the patch passed | | +1 :green_heart: | javac | 1m 21s | the patch passed | | +1 :green_heart: | shadedjars | 6m 8s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 58s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 32s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 227m 56s | hbase-server in the patch passed. | | | | 258m 40s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1681 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux e785d39e0917 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/6/testReport/ | | Max. process+thread count | 2587 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/6/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1681: HBASE-23938 : System table hbase:slowlog to store complete slow/large…
Apache-HBase commented on pull request #1681: URL: https://github.com/apache/hbase/pull/1681#issuecomment-629721606 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 39s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 48s | master passed | | +1 :green_heart: | compile | 1m 45s | master passed | | +1 :green_heart: | shadedjars | 6m 35s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 19s | hbase-common in master failed. | | -0 :warning: | javadoc | 0m 46s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 47s | the patch passed | | +1 :green_heart: | compile | 1m 41s | the patch passed | | +1 :green_heart: | javac | 1m 41s | the patch passed | | +1 :green_heart: | shadedjars | 6m 37s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 16s | hbase-common in the patch failed. | | -0 :warning: | javadoc | 0m 51s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 12s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 213m 51s | hbase-server in the patch passed. | | | | 248m 37s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1681 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 2751c15da3a8 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/6/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-common.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/6/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/6/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-common.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/6/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/6/testReport/ | | Max. process+thread count | 3351 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/6/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24165) maxPoolSize is logged incorrectly in ByteBufferPool
[ https://issues.apache.org/jira/browse/HBASE-24165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109242#comment-17109242 ] Guanghao Zhang commented on HBASE-24165: [~stack] No need to revert, sir. The findbugs warning was fixed by the new addendum patch. Thanks. > maxPoolSize is logged incorrectly in ByteBufferPool > --- > > Key: HBASE-24165 > URL: https://issues.apache.org/jira/browse/HBASE-24165 > Project: HBase > Issue Type: Bug >Affects Versions: 2.2.4 >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Minor > Fix For: 2.2.5 > > > In ByteBufferPool _maxPoolSize_ is converted into byte format, > https://github.com/apache/hbase/blob/a521a80c4b9a8b0749c368d1ff66fea2ed2d77a2/hbase-common/src/main/java/org/apache/hadoop/hbase/io/ByteBufferPool.java#L85 > > Currently maxPoolSize is logged as below, > 2020-04-10 14:20:56,000 INFO [Time-limited test] io.ByteBufferPool(83): > Created with bufferSize=64 KB and maxPoolSize=320 B -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24165) maxPoolSize is logged incorrectly in ByteBufferPool
[ https://issues.apache.org/jira/browse/HBASE-24165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang resolved HBASE-24165. Resolution: Fixed > maxPoolSize is logged incorrectly in ByteBufferPool > --- > > Key: HBASE-24165 > URL: https://issues.apache.org/jira/browse/HBASE-24165 > Project: HBase > Issue Type: Bug >Affects Versions: 2.2.4 >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Minor > Fix For: 2.2.5 > > > In ByteBufferPool _maxPoolSize_ is converted into byte format, > https://github.com/apache/hbase/blob/a521a80c4b9a8b0749c368d1ff66fea2ed2d77a2/hbase-common/src/main/java/org/apache/hadoop/hbase/io/ByteBufferPool.java#L85 > > Currently maxPoolSize is logged as below, > 2020-04-10 14:20:56,000 INFO [Time-limited test] io.ByteBufferPool(83): > Created with bufferSize=64 KB and maxPoolSize=320 B -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1685: Backport "HBASE-24305 Prepare deprecations in ServerName (#1666)" to branch-2
Apache-HBase commented on pull request #1685: URL: https://github.com/apache/hbase/pull/1685#issuecomment-629711022 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 43s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 12s | branch-2 passed | | +1 :green_heart: | compile | 2m 28s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 44s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 56s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 41s | the patch passed | | +1 :green_heart: | compile | 2m 29s | the patch passed | | +1 :green_heart: | javac | 2m 29s | the patch passed | | +1 :green_heart: | shadedjars | 5m 29s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 47s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 42s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 2m 29s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 194m 17s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 4m 28s | hbase-rest in the patch passed. | | | | 235m 19s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1685/3/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1685 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 2f9043607be5 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 692a597b63 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1685/3/testReport/ | | Max. process+thread count | 2696 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-client hbase-server hbase-rest U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1685/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24165) maxPoolSize is logged incorrectly in ByteBufferPool
[ https://issues.apache.org/jira/browse/HBASE-24165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109236#comment-17109236 ] Michael Stack commented on HBASE-24165: --- I can revert unless you want to fix it [~pankajkumar] ? Thanks. > maxPoolSize is logged incorrectly in ByteBufferPool > --- > > Key: HBASE-24165 > URL: https://issues.apache.org/jira/browse/HBASE-24165 > Project: HBase > Issue Type: Bug >Affects Versions: 2.2.4 >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Minor > Fix For: 2.2.5 > > > In ByteBufferPool _maxPoolSize_ is converted into byte format, > https://github.com/apache/hbase/blob/a521a80c4b9a8b0749c368d1ff66fea2ed2d77a2/hbase-common/src/main/java/org/apache/hadoop/hbase/io/ByteBufferPool.java#L85 > > Currently maxPoolSize is logged as below, > 2020-04-10 14:20:56,000 INFO [Time-limited test] io.ByteBufferPool(83): > Created with bufferSize=64 KB and maxPoolSize=320 B -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1685: Backport "HBASE-24305 Prepare deprecations in ServerName (#1666)" to branch-2
Apache-HBase commented on pull request #1685: URL: https://github.com/apache/hbase/pull/1685#issuecomment-629703546 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 21s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 57s | branch-2 passed | | +1 :green_heart: | compile | 2m 32s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 56s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 30s | hbase-client in branch-2 failed. | | -0 :warning: | javadoc | 0m 17s | hbase-common in branch-2 failed. | | -0 :warning: | javadoc | 0m 23s | hbase-rest in branch-2 failed. | | -0 :warning: | javadoc | 0m 40s | hbase-server in branch-2 failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 3s | the patch passed | | +1 :green_heart: | compile | 2m 26s | the patch passed | | +1 :green_heart: | javac | 2m 26s | the patch passed | | +1 :green_heart: | shadedjars | 5m 48s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 19s | hbase-common in the patch failed. | | -0 :warning: | javadoc | 0m 27s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 37s | hbase-server in the patch failed. | | -0 :warning: | javadoc | 0m 23s | hbase-rest in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 43s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 2m 18s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 123m 37s | hbase-server in the patch passed. | | -1 :x: | unit | 2m 24s | hbase-rest in the patch failed. | | | | 165m 22s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1685/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1685 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux cd8083948897 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 692a597b63 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1685/3/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1685/3/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-common.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1685/3/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-rest.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1685/3/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1685/3/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-common.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1685/3/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1685/3/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1685/3/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-rest.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1685/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-rest.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1685/3/testReport/ | | Max. process+thread count | 3679 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-client hbase-server hbase-rest U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1685/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was
[GitHub] [hbase] Apache-HBase commented on pull request #1681: HBASE-23938 : System table hbase:slowlog to store complete slow/large…
Apache-HBase commented on pull request #1681: URL: https://github.com/apache/hbase/pull/1681#issuecomment-629701564 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 28s | master passed | | +1 :green_heart: | checkstyle | 1m 31s | master passed | | +1 :green_heart: | spotbugs | 2m 39s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 22s | the patch passed | | -0 :warning: | checkstyle | 1m 9s | hbase-server: The patch generated 6 new + 150 unchanged - 0 fixed = 156 total (was 150) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 7s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 58s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 35m 38s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/6/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1681 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 9e895c872c28 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/6/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/6/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1681: HBASE-23938 : System table hbase:slowlog to store complete slow/large…
Apache-HBase commented on pull request #1681: URL: https://github.com/apache/hbase/pull/1681#issuecomment-629697348 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 32s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 11s | master passed | | +1 :green_heart: | compile | 1m 24s | master passed | | +1 :green_heart: | shadedjars | 6m 14s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 57s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 0s | the patch passed | | +1 :green_heart: | compile | 1m 23s | the patch passed | | +1 :green_heart: | javac | 1m 23s | the patch passed | | +1 :green_heart: | shadedjars | 6m 20s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 58s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 33s | hbase-common in the patch passed. | | -1 :x: | unit | 226m 47s | hbase-server in the patch failed. | | | | 257m 51s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1681 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d2a356e42dd9 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/5/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/5/testReport/ | | Max. process+thread count | 2876 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/5/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1681: HBASE-23938 : System table hbase:slowlog to store complete slow/large…
Apache-HBase commented on pull request #1681: URL: https://github.com/apache/hbase/pull/1681#issuecomment-629696664 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 1s | master passed | | +1 :green_heart: | compile | 1m 44s | master passed | | +1 :green_heart: | shadedjars | 6m 33s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 20s | hbase-common in master failed. | | -0 :warning: | javadoc | 0m 46s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 40s | the patch passed | | +1 :green_heart: | compile | 1m 45s | the patch passed | | +1 :green_heart: | javac | 1m 45s | the patch passed | | +1 :green_heart: | shadedjars | 6m 49s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 17s | hbase-common in the patch failed. | | -0 :warning: | javadoc | 0m 50s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 10s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 216m 44s | hbase-server in the patch passed. | | | | 251m 46s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1681 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ba2350f20220 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/5/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-common.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/5/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/5/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-common.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/5/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/5/testReport/ | | Max. process+thread count | 3341 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/5/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1685: Backport "HBASE-24305 Prepare deprecations in ServerName (#1666)" to branch-2
Apache-HBase commented on pull request #1685: URL: https://github.com/apache/hbase/pull/1685#issuecomment-629690731 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 42s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 12s | branch-2 passed | | +1 :green_heart: | checkstyle | 2m 43s | branch-2 passed | | +1 :green_heart: | spotbugs | 4m 56s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 48s | the patch passed | | +1 :green_heart: | checkstyle | 0m 22s | hbase-common: The patch generated 0 new + 0 unchanged - 25 fixed = 0 total (was 25) | | +1 :green_heart: | checkstyle | 0m 35s | The patch passed checkstyle in hbase-client | | +1 :green_heart: | checkstyle | 1m 20s | hbase-server: The patch generated 0 new + 275 unchanged - 3 fixed = 275 total (was 278) | | +1 :green_heart: | checkstyle | 0m 18s | The patch passed checkstyle in hbase-rest | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 13m 26s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 6m 40s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 48s | The patch does not generate ASF License warnings. | | | | 51m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1685/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1685 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 8d72ecf60bbb 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 692a597b63 | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-client hbase-server hbase-rest U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1685/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24165) maxPoolSize is logged incorrectly in ByteBufferPool
[
https://issues.apache.org/jira/browse/HBASE-24165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109167#comment-17109167
]
Hudson commented on HBASE-24165:
Results for branch branch-2.2
[build #864 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/864/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/864//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/864//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/864//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> maxPoolSize is logged incorrectly in ByteBufferPool
> ---
>
> Key: HBASE-24165
> URL: https://issues.apache.org/jira/browse/HBASE-24165
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.2.4
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Minor
> Fix For: 2.2.5
>
>
> In ByteBufferPool _maxPoolSize_ is converted into byte format,
> https://github.com/apache/hbase/blob/a521a80c4b9a8b0749c368d1ff66fea2ed2d77a2/hbase-common/src/main/java/org/apache/hadoop/hbase/io/ByteBufferPool.java#L85
>
> Currently maxPoolSize is logged as below,
> 2020-04-10 14:20:56,000 INFO [Time-limited test] io.ByteBufferPool(83):
> Created with bufferSize=64 KB and maxPoolSize=320 B
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] virajjasani commented on a change in pull request #1681: HBASE-23938 : System table hbase:slowlog to store complete slow/large…
virajjasani commented on a change in pull request #1681:
URL: https://github.com/apache/hbase/pull/1681#discussion_r426177730
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/slowlog/SlowLogRecorder.java
##
@@ -86,7 +87,9 @@ public SlowLogRecorder(Configuration conf) {
this.disruptor.setDefaultExceptionHandler(new DisruptorExceptionHandler());
// initialize ringbuffer event handler
-this.logEventHandler = new LogEventHandler(this.eventCount);
+final boolean isSlowLogTableEnabled =
conf.getBoolean(HConstants.SLOW_LOG_SYS_TABLE_ENABLED_KEY,
Review comment:
Yes, we will create that handler because that is primarily used for
in-memory ring buffer. That part is fine, nothing to worry
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/slowlog/SlowLogTableAccessor.java
##
@@ -0,0 +1,140 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.regionserver.slowlog;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Durability;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.TooSlowLog;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Slowlog Accessor to record slow/large RPC log identified at each
RegionServer RpcServer level.
+ * This can be done only optionally to record the entire history of slow/large
rpc calls
+ * since RingBuffer can handle only limited latest records.
+ */
+@InterfaceAudience.Private
+public class SlowLogTableAccessor {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(SlowLogTableAccessor.class);
+
+ private static final Random RANDOM = new Random();
+
+ private static Connection connection;
+
+ private static void doPut(final Connection connection, final List puts)
+ throws IOException {
+try (Table table = connection.getTable(TableName.SLOW_LOG_TABLE_NAME)) {
+ table.put(puts);
+}
+ }
+
+ /**
+ * Add slow/large log records to hbase:slowlog table
+ * @param slowLogPayloads List of SlowLogPayload to process
+ * @param configuration Configuration to use for connection
+ */
+ public static void addSlowLogRecords(final List
slowLogPayloads,
+ final Configuration configuration) {
+List puts = new ArrayList<>(slowLogPayloads.size());
+for (TooSlowLog.SlowLogPayload slowLogPayload : slowLogPayloads) {
+ final byte[] rowKey = getRowKey(slowLogPayload);
+ final Put put = new Put(rowKey).setDurability(Durability.SKIP_WAL)
+.addColumn(HConstants.SLOWLOG_INFO_FAMILY,
Bytes.toBytes("call_details"),
+ Bytes.toBytes(slowLogPayload.getCallDetails()))
+.addColumn(HConstants.SLOWLOG_INFO_FAMILY,
Bytes.toBytes("client_address"),
+ Bytes.toBytes(slowLogPayload.getClientAddress()))
+.addColumn(HConstants.SLOWLOG_INFO_FAMILY,
Bytes.toBytes("method_name"),
+ Bytes.toBytes(slowLogPayload.getMethodName()))
+.addColumn(HConstants.SLOWLOG_INFO_FAMILY, Bytes.toBytes("param"),
+ Bytes.toBytes(slowLogPayload.getParam()))
+.addColumn(HConstants.SLOWLOG_INFO_FAMILY,
Bytes.toBytes("processing_time"),
+ Bytes.toBytes(Integer.toString(slowLogPayload.getProcessingTime(
+.addColumn(HConstants.SLOWLOG_INFO_FAMILY, Bytes.toBytes("queue_time"),
+ Bytes.toBytes(Integer.toString(slowLogPayload.getQueueTime(
+.addColumn(HConstants.SLOWLOG_INFO_FAMILY,
Bytes.toBytes("region_name"),
+ Bytes.toBytes(slowLogPayload.getRegionName()))
+.addColumn(HConstants.SLOWLOG_INFO_FAMILY,
[jira] [Updated] (HBASE-24385) release scripts should do compat report that includes LimitedPrivate APIs for maintenance releases
[ https://issues.apache.org/jira/browse/HBASE-24385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-24385: Component/s: community > release scripts should do compat report that includes LimitedPrivate APIs for > maintenance releases > -- > > Key: HBASE-24385 > URL: https://issues.apache.org/jira/browse/HBASE-24385 > Project: HBase > Issue Type: Bug > Components: community >Reporter: Sean Busbey >Priority: Major > > currently the release script does a compatibility report that just covers > IA.Public apis. That's covers our promises for new minor versions. When doing > maintenance versions (e.g. 2.2.5) we ought to be reporting on any breaking > changes in IA.LimitedPrivate APIs as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24385) release scripts should do compat report that includes LimitedPrivate APIs for maintenance releases
Sean Busbey created HBASE-24385: --- Summary: release scripts should do compat report that includes LimitedPrivate APIs for maintenance releases Key: HBASE-24385 URL: https://issues.apache.org/jira/browse/HBASE-24385 Project: HBase Issue Type: Bug Reporter: Sean Busbey currently the release script does a compatibility report that just covers IA.Public apis. That's covers our promises for new minor versions. When doing maintenance versions (e.g. 2.2.5) we ought to be reporting on any breaking changes in IA.LimitedPrivate APIs as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24384) new major version in release script should default compat report to previous latest minor release
Sean Busbey created HBASE-24384:
---
Summary: new major version in release script should default compat
report to previous latest minor release
Key: HBASE-24384
URL: https://issues.apache.org/jira/browse/HBASE-24384
Project: HBase
Issue Type: Improvement
Components: community
Reporter: Sean Busbey
currently if you are starting a new major version, the default calculated
comparison version is invalid:
{code}
=== Gathering release details.
PROJECT [hbase]: hbase
GIT_BRANCH [branch-2.3]: master
Current branch VERSION is 3.0.0-SNAPSHOT.
RELEASE_VERSION [3.0.0]: 3.0.0-alpha-1
NEXT_VERSION [3.0.1-SNAPSHOT]: 3.0.0-alpha-2-SNAPSHOT
RC_COUNT [0]:
RELEASE_TAG [3.0.0-alpha-1RC0]:
This is a dry run. If tag does not actually exist, please confirm the ref that
will be built for testing.
GIT_REF [3.0.0-alpha-1RC0]:
api_diff_tag [rel/3.0.-2]: rel/2.2.4
{code}
when the new version is a new major we should either default to the prior
initial major release (e.g. 2.0.0) or we should scan the release metadata for
the most recent release from the prior major release (e.g. 2.2.4).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-24383) compatibility report in release process should use existing binaries
Sean Busbey created HBASE-24383: --- Summary: compatibility report in release process should use existing binaries Key: HBASE-24383 URL: https://issues.apache.org/jira/browse/HBASE-24383 Project: HBase Issue Type: Improvement Components: community Reporter: Sean Busbey right now the compatibility report generation in the release scripts does an additional build of both the current RC and the comparison release. Instead, we should have an option to grab artifacts for the previous release from the mirroring / archive system. Additionally, we should always use the binary bits we just created for the binary distribution for the RC. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ramkrish86 commented on a change in pull request #1681: HBASE-23938 : System table hbase:slowlog to store complete slow/large…
ramkrish86 commented on a change in pull request #1681:
URL: https://github.com/apache/hbase/pull/1681#discussion_r426174968
##
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
##
@@ -1532,6 +1532,16 @@
"hbase.regionserver.slowlog.buffer.enabled";
public static final boolean DEFAULT_ONLINE_LOG_PROVIDER_ENABLED = false;
+ /** The slowlog info family as a string*/
+ private static final String SLOWLOG_INFO_FAMILY_STR = "info";
+
+ /** The slowlog info family */
+ public static final byte [] SLOWLOG_INFO_FAMILY =
Bytes.toBytes(SLOWLOG_INFO_FAMILY_STR);
+
+ public static final String SLOW_LOG_SYS_TABLE_ENABLED_KEY =
+"hbase.regionserver.slowlog.systable.enabled";
+ public static final boolean DEFAULT_SLOW_LOG_SYS_TABLE_ENABLED_KEY = false;
Review comment:
Good. Generally if we have a log system which will store the logs then
this may be redundant in such systems.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/slowlog/SlowLogTableAccessor.java
##
@@ -0,0 +1,140 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.regionserver.slowlog;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Durability;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.TooSlowLog;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Slowlog Accessor to record slow/large RPC log identified at each
RegionServer RpcServer level.
+ * This can be done only optionally to record the entire history of slow/large
rpc calls
+ * since RingBuffer can handle only limited latest records.
+ */
+@InterfaceAudience.Private
+public class SlowLogTableAccessor {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(SlowLogTableAccessor.class);
+
+ private static final Random RANDOM = new Random();
+
+ private static Connection connection;
+
+ private static void doPut(final Connection connection, final List puts)
+ throws IOException {
+try (Table table = connection.getTable(TableName.SLOW_LOG_TABLE_NAME)) {
+ table.put(puts);
+}
+ }
+
+ /**
+ * Add slow/large log records to hbase:slowlog table
+ * @param slowLogPayloads List of SlowLogPayload to process
+ * @param configuration Configuration to use for connection
+ */
+ public static void addSlowLogRecords(final List
slowLogPayloads,
+ final Configuration configuration) {
+List puts = new ArrayList<>(slowLogPayloads.size());
+for (TooSlowLog.SlowLogPayload slowLogPayload : slowLogPayloads) {
+ final byte[] rowKey = getRowKey(slowLogPayload);
+ final Put put = new Put(rowKey).setDurability(Durability.SKIP_WAL)
+.addColumn(HConstants.SLOWLOG_INFO_FAMILY,
Bytes.toBytes("call_details"),
+ Bytes.toBytes(slowLogPayload.getCallDetails()))
+.addColumn(HConstants.SLOWLOG_INFO_FAMILY,
Bytes.toBytes("client_address"),
+ Bytes.toBytes(slowLogPayload.getClientAddress()))
+.addColumn(HConstants.SLOWLOG_INFO_FAMILY,
Bytes.toBytes("method_name"),
+ Bytes.toBytes(slowLogPayload.getMethodName()))
+.addColumn(HConstants.SLOWLOG_INFO_FAMILY, Bytes.toBytes("param"),
+ Bytes.toBytes(slowLogPayload.getParam()))
+.addColumn(HConstants.SLOWLOG_INFO_FAMILY,
Bytes.toBytes("processing_time"),
+ Bytes.toBytes(Integer.toString(slowLogPayload.getProcessingTime(
+.addColumn(HConstants.SLOWLOG_INFO_FAMILY, Bytes.toBytes("queue_time"),
+ Bytes.toBytes(Integer.toString(slowLogPayload.getQueueTime(
+
[GitHub] [hbase] HorizonNet commented on a change in pull request #1723: HBASE-24381 The Size metrics in Master Webui is wrong if the size is 0
HorizonNet commented on a change in pull request #1723:
URL: https://github.com/apache/hbase/pull/1723#discussion_r426174769
##
File path:
hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/RegionServerListTmpl.jamon
##
@@ -162,10 +162,10 @@ Arrays.sort(serverNames);
<%java>
final String ZEROMB = "0 MB";
-String usedHeapStr = ZEROMB;
-String maxHeapStr = ZEROMB;
-String memStoreSizeMBStr = ZEROMB;
for (ServerName serverName: serverNames) {
+String usedHeapStr = ZEROMB;
Review comment:
Indentation seems to be different here compared to the other changes.
##
File path:
hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/RegionServerListTmpl.jamon
##
@@ -276,12 +276,12 @@ if (sl != null) {
<%java>
final String ZEROKB = "0 KB";
final String ZEROMB = "0 MB";
+for (ServerName serverName: serverNames) {
+
String storeUncompressedSizeMBStr = ZEROMB;
Review comment:
Ditto.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1681: HBASE-23938 : System table hbase:slowlog to store complete slow/large…
Apache-HBase commented on pull request #1681: URL: https://github.com/apache/hbase/pull/1681#issuecomment-629668344 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 26s | master passed | | +1 :green_heart: | checkstyle | 1m 32s | master passed | | +1 :green_heart: | spotbugs | 2m 38s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 20s | the patch passed | | -0 :warning: | checkstyle | 1m 8s | hbase-server: The patch generated 6 new + 150 unchanged - 0 fixed = 156 total (was 150) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 10s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | -1 :x: | spotbugs | 2m 11s | hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 35m 35s | | | Reason | Tests | |---:|:--| | FindBugs | module:hbase-server | | | Possible doublecheck on org.apache.hadoop.hbase.regionserver.slowlog.SlowLogTableAccessor.connection in org.apache.hadoop.hbase.regionserver.slowlog.SlowLogTableAccessor.addSlowLogRecords(List, Configuration) At SlowLogTableAccessor.java:org.apache.hadoop.hbase.regionserver.slowlog.SlowLogTableAccessor.addSlowLogRecords(List, Configuration) At SlowLogTableAccessor.java:[lines 100-102] | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1681 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 47259b58b106 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/5/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | spotbugs | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/5/artifact/yetus-general-check/output/new-spotbugs-hbase-server.html | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/5/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-23296) Add CompositeBucketCache to support tiered BC
[ https://issues.apache.org/jira/browse/HBASE-23296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109108#comment-17109108 ] Anoop Sam John commented on HBASE-23296: [~javaman_chen] I dont have access to the Google doc. Can u make it public pls? > Add CompositeBucketCache to support tiered BC > - > > Key: HBASE-23296 > URL: https://issues.apache.org/jira/browse/HBASE-23296 > Project: HBase > Issue Type: New Feature > Components: BlockCache >Reporter: chenxu >Assignee: chenxu >Priority: Major > > LruBlockCache is not suitable in the following scenarios: > (1) cache size too large (will take too much heap memory, and > evictBlocksByHfileName is not so efficient, as HBASE-23277 mentioned) > (2) block evicted frequently, especially cacheOnWrite & prefetchOnOpen are > enabled. > Since block‘s data is reclaimed by GC, this may affect GC performance. > So how about enabling a Bucket based L1 Cache. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] virajjasani commented on a change in pull request #1681: HBASE-23938 : System table hbase:slowlog to store complete slow/large…
virajjasani commented on a change in pull request #1681: URL: https://github.com/apache/hbase/pull/1681#discussion_r426163100 ## File path: hbase-common/src/main/java/org/apache/hadoop/hbase/TableName.java ## @@ -83,6 +83,12 @@ public static final TableName META_TABLE_NAME = valueOf(NamespaceDescriptor.SYSTEM_NAMESPACE_NAME_STR, "meta"); + /** hbase:slowlog table name - can be enabled + * with config - hbase.regionserver.slowlog.systable.enabled + */ + public static final TableName SLOW_LOG_TABLE_NAME = Review comment: Yes, I tried disable and enable of 'hbase:slowlog' locally and both worked fine. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani commented on a change in pull request #1681: HBASE-23938 : System table hbase:slowlog to store complete slow/large…
virajjasani commented on a change in pull request #1681:
URL: https://github.com/apache/hbase/pull/1681#discussion_r426139520
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/slowlog/LogEventHandler.java
##
@@ -256,4 +278,32 @@ private boolean
isFilterProvided(AdminProtos.SlowLogResponseRequest request) {
return filteredSlowLogPayloads;
}
+ /**
+ * Poll from queueForSysTable and insert 100 records in hbase:slowlog table
in single batch
+ */
+ void addAllLogsToSysTable() {
+if (queueForSysTable == null) {
+ LOG.warn("hbase.regionserver.slowlog.systable.enabled is turned off.
Exiting.");
+ return;
+}
+if (this.connection == null) {
+ LOG.warn("LogEventHandler has null connection. Exiting.");
+ return;
+}
+List slowLogPayloads = new ArrayList<>();
+int i = 0;
+while (!queueForSysTable.isEmpty()) {
+ slowLogPayloads.add(queueForSysTable.poll());
+ i++;
+ if (i == 100) {
Review comment:
I believe if region name is quite long, in the worst case, each request
might be of ~ 3-4k bytes.
Also, mutate() allows by default 5000 rows in single request and hence, this
is reasonably lower no of requests. Size is also not much higher.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/slowlog/SlowLogTableAccessor.java
##
@@ -0,0 +1,124 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.regionserver.slowlog;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.Durability;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.TooSlowLog;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Slowlog Accessor to record slow/large RPC log identified at each
RegionServer RpcServer level.
+ * This can be done only optionally to record the entire history of slow/large
rpc calls
+ * since RingBuffer can handle only limited latest records.
+ */
+@InterfaceAudience.Private
+public class SlowLogTableAccessor {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(SlowLogTableAccessor.class);
+
+ private static final Random RANDOM = new Random();
+
+ private static void doPut(final Connection connection, final List puts)
+ throws IOException {
+try (Table table = connection.getTable(TableName.SLOW_LOG_TABLE_NAME)) {
+ table.put(puts);
Review comment:
@anoopsjohn on master branch, this retry count is 45?
```
public static void setServerSideHConnectionRetriesConfig(final
Configuration c, final String sn,
final Logger log) {
// TODO: Fix this. Not all connections from server side should have 10
times the retries.
int hcRetries = c.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
// Go big. Multiply by 10. If we can't get to meta after this many
retries
// then something seriously wrong.
int serversideMultiplier =
c.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
int retries = hcRetries * serversideMultiplier;
c.setInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER, retries);
log.info(sn + " server-side Connection retries=" + retries);
}
```
Or I am missing some calculation?
Let me get these configs:
```
conf.setInt(HConstants.HBASE_RPC_TIMEOUT_KEY, 2);
conf.setInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER, 5);
conf.setInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
1);
```
[jira] [Commented] (HBASE-24327) TestMasterShutdown#testMasterShutdownBeforeStartingAnyRegionServer can fail with retries exhausted on an admin call.
[
https://issues.apache.org/jira/browse/HBASE-24327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109080#comment-17109080
]
Hudson commented on HBASE-24327:
Results for branch master
[build #1729 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1729/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1729/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1475//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1729/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1729/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> TestMasterShutdown#testMasterShutdownBeforeStartingAnyRegionServer can fail
> with retries exhausted on an admin call.
>
>
> Key: HBASE-24327
> URL: https://issues.apache.org/jira/browse/HBASE-24327
> Project: HBase
> Issue Type: Test
> Components: test
>Reporter: Mark Robert Miller
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danil Lipovoy updated HBASE-23887:
--
Attachment: BlockCacheEvictionProcess.gif
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, cmp.png, evict_BC100_vs_BC23.png,
> read_requests_100pBC_vs_23pBC.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109077#comment-17109077
]
Danil Lipovoy commented on HBASE-23887:
---
Just made some visualisation:
!BlockCacheEvictionProcess.gif!
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, cmp.png, evict_BC100_vs_BC23.png,
> read_requests_100pBC_vs_23pBC.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HBASE-24382) Flush partial stores of region filtered by seqId when archive wal due to too many wals
[ https://issues.apache.org/jira/browse/HBASE-24382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-24382 started by Zheng Wang. -- > Flush partial stores of region filtered by seqId when archive wal due to too > many wals > -- > > Key: HBASE-24382 > URL: https://issues.apache.org/jira/browse/HBASE-24382 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > > When the logRoller archive the oldest wal due to too many wals, if a region > should be flushed, we flush all stores of it, but it is not necessary, maybe > we can use unflushedSeqId of store to filter them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24382) Flush partial stores of region filtered by seqId when archive wal due to too many wals
[ https://issues.apache.org/jira/browse/HBASE-24382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Wang updated HBASE-24382: --- Description: When the logRoller archive the oldest wal due to too many wals, if a region should be flushed, we flush all stores of it, but it is not necessary, maybe we can use unflushedSeqId of store to filter them. (was: When the logRoller archive the oldest wal due to too many wals, if a region should be flushed, we flush all stores of it, but it is not necessary, maybe we can use unflushedSeqId of store to do a filter.) > Flush partial stores of region filtered by seqId when archive wal due to too > many wals > -- > > Key: HBASE-24382 > URL: https://issues.apache.org/jira/browse/HBASE-24382 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > > When the logRoller archive the oldest wal due to too many wals, if a region > should be flushed, we flush all stores of it, but it is not necessary, maybe > we can use unflushedSeqId of store to filter them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24382) Flush partial stores of region filtered by seqId when archive wal due to too many wals
Zheng Wang created HBASE-24382: -- Summary: Flush partial stores of region filtered by seqId when archive wal due to too many wals Key: HBASE-24382 URL: https://issues.apache.org/jira/browse/HBASE-24382 Project: HBase Issue Type: Improvement Components: wal Reporter: Zheng Wang Assignee: Zheng Wang When the logRoller archive the oldest wal due to too many wals, if a region should be flushed, we flush all stores of it, but it is not necessary, maybe we can use unflushedSeqId of store to do a filter. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24327) TestMasterShutdown#testMasterShutdownBeforeStartingAnyRegionServer can fail with retries exhausted on an admin call.
[ https://issues.apache.org/jira/browse/HBASE-24327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani resolved HBASE-24327. -- Resolution: Fixed > TestMasterShutdown#testMasterShutdownBeforeStartingAnyRegionServer can fail > with retries exhausted on an admin call. > > > Key: HBASE-24327 > URL: https://issues.apache.org/jira/browse/HBASE-24327 > Project: HBase > Issue Type: Test > Components: test >Reporter: Mark Robert Miller >Assignee: Viraj Jasani >Priority: Major > Fix For: 3.0.0-alpha-1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24327) TestMasterShutdown#testMasterShutdownBeforeStartingAnyRegionServer can fail with retries exhausted on an admin call.
[ https://issues.apache.org/jira/browse/HBASE-24327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108889#comment-17108889 ] Viraj Jasani commented on HBASE-24327: -- Got it, Thanks [~stack] Since we are going to continue with ZKConnectionRegistry in branch-2+ and release 3.0 will be the first to have MasterRegistry as default, we ideally need this patch only for master branch. We are all good with synchronized call to shutdown() RPC in branch-2 and 2.3. > TestMasterShutdown#testMasterShutdownBeforeStartingAnyRegionServer can fail > with retries exhausted on an admin call. > > > Key: HBASE-24327 > URL: https://issues.apache.org/jira/browse/HBASE-24327 > Project: HBase > Issue Type: Test > Components: test >Reporter: Mark Robert Miller >Assignee: Viraj Jasani >Priority: Major > Fix For: 3.0.0-alpha-1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
