[jira] [Updated] (HBASE-22760) Stop/Resume Snapshot Auto-Cleanup activity with shell command
[ https://issues.apache.org/jira/browse/HBASE-22760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani updated HBASE-22760: - Attachment: HBASE-22760.branch-1.001.patch > Stop/Resume Snapshot Auto-Cleanup activity with shell command > - > > Key: HBASE-22760 > URL: https://issues.apache.org/jira/browse/HBASE-22760 > Project: HBase > Issue Type: Improvement > Components: Admin, shell, snapshots >Affects Versions: 3.0.0, 1.5.0, 2.3.0 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.3.0 > > Attachments: HBASE-22760.branch-1.000.patch, > HBASE-22760.branch-1.001.patch, HBASE-22760.branch-2.000.patch, > HBASE-22760.master.003.patch, HBASE-22760.master.004.patch, > HBASE-22760.master.005.patch, HBASE-22760.master.008.patch, > HBASE-22760.master.009.patch > > > For any scheduled snapshot backup activity, we would like to disable > auto-cleaner for snapshot based on TTL. However, as per HBASE-22648 we have a > config to disable snapshot auto-cleaner: > hbase.master.cleaner.snapshot.disable, which would take effect only upon > HMaster restart just similar to any other hbase-site configs. > For any running cluster, we should be able to stop/resume auto-cleanup > activity for snapshot based on shell command. Something similar to below > command should be able to stop/start cleanup chore: > hbase(main):001:0> snapshot_auto_cleanup_switch false (disable > auto-cleaner) > hbase(main):001:0> snapshot_auto_cleanup_switch true (enable auto-cleaner) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [hbase-operator-tools] jfrabaute commented on issue #23: Fix invalid function call (from usage to showErrorMessage)
jfrabaute commented on issue #23: Fix invalid function call (from usage to showErrorMessage) URL: https://github.com/apache/hbase-operator-tools/pull/23#issuecomment-528709477 Thanks for the quick merge! (Yes, I checked the error log of the jenkins build and it was complaining about the code without the PR applied. Weird.) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase-operator-tools] busbey merged pull request #23: Fix invalid function call (from usage to showErrorMessage)
busbey merged pull request #23: Fix invalid function call (from usage to showErrorMessage) URL: https://github.com/apache/hbase-operator-tools/pull/23 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase-operator-tools] asf-ci commented on issue #23: Fix invalid function call (from usage to showErrorMessage)
asf-ci commented on issue #23: Fix invalid function call (from usage to showErrorMessage) URL: https://github.com/apache/hbase-operator-tools/pull/23#issuecomment-528706542 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-HBASE-OPERATOR-TOOLS-Build/86/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-22888) Use separate classe to deal with streaming read and pread
[
https://issues.apache.org/jira/browse/HBASE-22888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923925#comment-16923925
]
Anoop Sam John commented on HBASE-22888:
A nice find. So now if there is a stream reader is created, it will create new
HFileReader which in turn do loading of the FFT, Meta blocks, load on startup
blocks etc. Also if prefect cache is needed, it will again do the prefetch
read into cache !.
On closing the stream reader, it would have cancelled the cache prefetch if it
was still in progress
Checking the E2E reader creation flow, I can see it will again call the CP
hooks around the StoreFileReader creation! May be even that is not needed
really?
I like the way we have 2 HFileReader impls now. You can clearly do the extra
steps for Stream reader (ie. While creation, create a new FSReaderImpl and on
close, close that alone.
{code}
dataBlockIndexReader.copyFields(reader.getDataBlockIndexReader());
metaBlockIndexReader.copyFields(reader.getMetaBlockIndexReader());
{code}
Do we really need to do this copy? The stream reader's block index reader can
directly refer to the pread HFileReader's block index reader? More over can we
do this clone stuff within the HFileReader itself? If HFileReader having an
API like cloneForNewStreamRead() (Pardon the name) , we can do all the cone
stuff etc within itself. Same way would have been best for StoreFileReader
also so that we can avoid the CP hooks calls (?)
Again this is a nice catch
> Use separate classe to deal with streaming read and pread
> -
>
> Key: HBASE-22888
> URL: https://issues.apache.org/jira/browse/HBASE-22888
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: chenxu
>Assignee: chenxu
>Priority: Major
>
> When switching pread to stream read, new HFileReaderImpl will be create, but
> the two different readers do not share informations with each other. maybe we
> can divide HFileReaderImpl into two different class, such as HFilePreadReader
> and HFileStreamReader. When constructing HFileStreamReader, it will copy some
> stuffs (fileInfo, index, etc) from an already existing Reader, and no need to
> do prefetch operations.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [hbase-operator-tools] jfrabaute opened a new pull request #23: Fix invalid function call (from usage to showErrorMessage)
jfrabaute opened a new pull request #23: Fix invalid function call (from usage to showErrorMessage) URL: https://github.com/apache/hbase-operator-tools/pull/23 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-22930) Set unique name to longCompactions/shortCompactions threads
[ https://issues.apache.org/jira/browse/HBASE-22930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923914#comment-16923914 ] Anoop Sam John commented on HBASE-22930: Sorry for being later in replying to u [~pankaj2461].. Yes my comment was on the usage of the local Atomic variable usage. The pull request #585 looks good. You want fix this in branch-1 also? There seems we are not even using google common ThreadFactoryBuilder so own Atomic variable usage might be needed. Its ok to fix this in branch-2 + also. > Set unique name to longCompactions/shortCompactions threads > --- > > Key: HBASE-22930 > URL: https://issues.apache.org/jira/browse/HBASE-22930 > Project: HBase > Issue Type: Improvement >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Minor > Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2 > > > Sometime observed multiple compaction threads have the same name which is > misleading while analysing the log. > Currently we just append the current timestamp which can be same for multiple > threads, > [https://github.com/apache/hbase/blob/ec68bf30ba91ea7484d3b56b0809ff5434f9406e/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplit.java#L144] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [hbase] Apache-HBase commented on issue #585: HBASE-22930 Set unique name to longCompactions/shortCompactions/split threads
Apache-HBase commented on issue #585: HBASE-22930 Set unique name to longCompactions/shortCompactions/split threads URL: https://github.com/apache/hbase/pull/585#issuecomment-528690354 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 0m 39s | Docker mode activated. | ||| _ Prechecks _ | | :green_heart: | dupname | 0m 0s | No case conflicting files found. | | :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | :yellow_heart: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | :green_heart: | mvninstall | 5m 49s | master passed | | :green_heart: | compile | 0m 59s | master passed | | :green_heart: | checkstyle | 1m 18s | master passed | | :green_heart: | shadedjars | 4m 41s | branch has no errors when building our shaded downstream artifacts. | | :green_heart: | javadoc | 0m 38s | master passed | | :blue_heart: | spotbugs | 3m 59s | Used deprecated FindBugs config; considering switching to SpotBugs. | | :green_heart: | findbugs | 3m 58s | master passed | ||| _ Patch Compile Tests _ | | :green_heart: | mvninstall | 4m 52s | the patch passed | | :green_heart: | compile | 1m 2s | the patch passed | | :green_heart: | javac | 1m 2s | the patch passed | | :green_heart: | checkstyle | 1m 17s | the patch passed | | :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | :green_heart: | shadedjars | 4m 30s | patch has no errors when building our shaded downstream artifacts. | | :green_heart: | hadoopcheck | 15m 30s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | :green_heart: | javadoc | 0m 35s | the patch passed | | :green_heart: | findbugs | 4m 8s | the patch passed | ||| _ Other Tests _ | | :green_heart: | unit | 160m 18s | hbase-server in the patch passed. | | :green_heart: | asflicense | 0m 35s | The patch does not generate ASF License warnings. | | | | 217m 5s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-585/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/585 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 7c4969a43fd0 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-585/out/precommit/personality/provided.sh | | git revision | master / fbd5b5e327 | | Default Java | 1.8.0_181 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-585/2/testReport/ | | Max. process+thread count | 4835 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-585/2/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-22859) [HBCK2] Fix the orphan regions on filesystem
[
https://issues.apache.org/jira/browse/HBASE-22859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923893#comment-16923893
]
HBase QA commented on HBASE-22859:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 6m
17s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
51s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
33s{color} | {color:red} root: The patch generated 1 new + 0 unchanged - 0
fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 6m
8s{color} | {color:blue} patch has no errors when building the reference guide.
See footer for rendered docs, which you should manually inspect. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
51s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
16m 46s{color} | {color:green} Patch does not cause any errors with Hadoop
2.8.5 2.9.2 or 3.1.2. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}244m 9s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
47s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} |
[jira] [Comment Edited] (HBASE-22978) Online slow response log
[
https://issues.apache.org/jira/browse/HBASE-22978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923852#comment-16923852
]
Andrew Purtell edited comment on HBASE-22978 at 9/6/19 1:19 AM:
You can invoke the shell with a command and pipe the output to a file, like
{{./bin/hbase shell 'command' > output.txt 2>&1}} . Or the output can be piped
to something else. Nothing special need be done there.
I like the idea of persisting the complete slow log with best effort, though.
Reminds me of the [MySQL slow log dump
tool|https://dev.mysql.com/doc/refman/8.0/en/mysqldumpslow.html]. Initially I
was thinking this could be part of general performance surveillance, where
sampling is good enough, but maybe there could be a tough to debug case that's
rare so would also be hard to catch in the ring buffers. For that we'd
configure a slow log directory in site configuration, presumably in HDFS, into
which regionservers would each append to a file, rolling at some configured
bound. A tool that decodes and prints to stdout, like HFilePrettyPrinter and
such, can mostly share common code with what we put in the regionserver to do
the same out to RPC for the shell.
was (Author: apurtell):
You can invoke the shell with a command and pipe the output to a file, like
{{./bin/hbase shell 'command' > output.txt 2>&1}} . Or the output can be piped
to something else. Nothing special need be done there.
I like the idea of persisting the complete slow log with best effort, though.
Reminds me of the [MySQL slow
log|https://dev.mysql.com/doc/refman/8.0/en/slow-query-log.html]. Initially I
was thinking this could be part of general performance surveillance, where
sampling is good enough, but maybe there could be a tough to debug case that's
rare so would also be hard to catch in the ring buffers. For that we'd
configure a slow log directory in site configuration, presumably in HDFS, into
which regionservers would each append to a file, rolling at some configured
bound. A tool that decodes and prints to stdout, like HFilePrettyPrinter and
such, can mostly share common code with what we put in the regionserver to do
the same out to RPC for the shell.
> Online slow response log
>
>
> Key: HBASE-22978
> URL: https://issues.apache.org/jira/browse/HBASE-22978
> Project: HBase
> Issue Type: New Feature
> Components: Admin, Operability, regionserver, shell
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
>
> Today when an individual RPC exceeds a configurable time bound we log a
> complaint by way of the logging subsystem. These log lines look like:
> {noformat}
> 2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
> (responseTooSlow):
> {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
> "starttimems":1567203007549,
> "responsesize":6819737,
> "method":"Scan",
> "param":"region { type: REGION_NAME value:
> \"tsdb,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
> "processingtimems":28646,
> "client":"10.253.196.215:41116",
> "queuetimems":22453,
> "class":"HRegionServer"}
> {noformat}
> Unfortunately we often truncate the request parameters, like in the above
> example. We do this because the human readable representation is verbose, the
> rate of too slow warnings may be high, and the combination of these things
> can overwhelm the log capture system. The truncation is unfortunate because
> it eliminates much of the utility of the warnings. For example, the region
> name, the start and end keys, and the filter hierarchy are all important
> clues for debugging performance problems caused by moderate to low
> selectivity queries or queries made at a high rate.
> We can maintain an in-memory ring buffer of requests that were judged to be
> too slow in addition to the responseTooSlow logging. The in-memory
> representation can be complete and compressed. A new admin API and shell
> command can provide access to the ring buffer for online performance
> debugging. A modest sizing of the ring buffer will prevent excessive memory
> utilization for a minor performance debugging feature by limiting the total
> number of retained records. There is some chance a high rate of requests will
> cause information on other interesting requests to be overwritten before it
> can be read. This is the nature of a ring buffer and an acceptable trade off.
> The write request types do not require us to retain all information submitted
> in the request. We don't need to retain all key-values in the mutation, which
> may be too large to comfortably retain. We only need a unique set of row
> keys, or even a min/max range, and total
[jira] [Commented] (HBASE-22760) Stop/Resume Snapshot Auto-Cleanup activity with shell command
[
https://issues.apache.org/jira/browse/HBASE-22760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923854#comment-16923854
]
Andrew Purtell commented on HBASE-22760:
[~vjasani] I can't commit this yet. TestShell fails on branch-1
{noformat}
1) Error:
test_snapshot_auto_cleanup_should_work(Hbase::AdminMethodsTest):
NoMethodError: undefined method `command' for
#
./src/test/ruby/hbase/admin_test.rb:163:in
`test_snapshot_auto_cleanup_should_work'
org/jruby/RubyProc.java:270:in `call'
org/jruby/RubyKernel.java:2105:in `send'
org/jruby/RubyArray.java:1620:in `each'
org/jruby/RubyArray.java:1620:in `each'
422 tests, 593 assertions, 0 failures, 1 errors
{noformat}
> Stop/Resume Snapshot Auto-Cleanup activity with shell command
> -
>
> Key: HBASE-22760
> URL: https://issues.apache.org/jira/browse/HBASE-22760
> Project: HBase
> Issue Type: Improvement
> Components: Admin, shell, snapshots
>Affects Versions: 3.0.0, 1.5.0, 2.3.0
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0
>
> Attachments: HBASE-22760.branch-1.000.patch,
> HBASE-22760.branch-2.000.patch, HBASE-22760.master.003.patch,
> HBASE-22760.master.004.patch, HBASE-22760.master.005.patch,
> HBASE-22760.master.008.patch, HBASE-22760.master.009.patch
>
>
> For any scheduled snapshot backup activity, we would like to disable
> auto-cleaner for snapshot based on TTL. However, as per HBASE-22648 we have a
> config to disable snapshot auto-cleaner:
> hbase.master.cleaner.snapshot.disable, which would take effect only upon
> HMaster restart just similar to any other hbase-site configs.
> For any running cluster, we should be able to stop/resume auto-cleanup
> activity for snapshot based on shell command. Something similar to below
> command should be able to stop/start cleanup chore:
> hbase(main):001:0> snapshot_auto_cleanup_switch false (disable
> auto-cleaner)
> hbase(main):001:0> snapshot_auto_cleanup_switch true (enable auto-cleaner)
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22978) Online slow response log
[
https://issues.apache.org/jira/browse/HBASE-22978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923853#comment-16923853
]
Sean Busbey commented on HBASE-22978:
-
bq. You can invoke the shell with a command and pipe the output to a file, like
{{./bin/hbase shell 'command' > output.txt 2>&1}} . Or the output can be piped
to something else. Nothing special need be done there.
Sure, but each RS writing to HDFS and then using tools to pull all or some
binary blobs down to a local machine (or inspecting them on HDFS itself) is
going to be less error prone then making things run through a jruby shell.
Agree that a general purpose "dump everything over here at best effort" is
going to be better for the kind of problems I'm thinking about though.
> Online slow response log
>
>
> Key: HBASE-22978
> URL: https://issues.apache.org/jira/browse/HBASE-22978
> Project: HBase
> Issue Type: New Feature
> Components: Admin, Operability, regionserver, shell
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
>
> Today when an individual RPC exceeds a configurable time bound we log a
> complaint by way of the logging subsystem. These log lines look like:
> {noformat}
> 2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
> (responseTooSlow):
> {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
> "starttimems":1567203007549,
> "responsesize":6819737,
> "method":"Scan",
> "param":"region { type: REGION_NAME value:
> \"tsdb,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
> "processingtimems":28646,
> "client":"10.253.196.215:41116",
> "queuetimems":22453,
> "class":"HRegionServer"}
> {noformat}
> Unfortunately we often truncate the request parameters, like in the above
> example. We do this because the human readable representation is verbose, the
> rate of too slow warnings may be high, and the combination of these things
> can overwhelm the log capture system. The truncation is unfortunate because
> it eliminates much of the utility of the warnings. For example, the region
> name, the start and end keys, and the filter hierarchy are all important
> clues for debugging performance problems caused by moderate to low
> selectivity queries or queries made at a high rate.
> We can maintain an in-memory ring buffer of requests that were judged to be
> too slow in addition to the responseTooSlow logging. The in-memory
> representation can be complete and compressed. A new admin API and shell
> command can provide access to the ring buffer for online performance
> debugging. A modest sizing of the ring buffer will prevent excessive memory
> utilization for a minor performance debugging feature by limiting the total
> number of retained records. There is some chance a high rate of requests will
> cause information on other interesting requests to be overwritten before it
> can be read. This is the nature of a ring buffer and an acceptable trade off.
> The write request types do not require us to retain all information submitted
> in the request. We don't need to retain all key-values in the mutation, which
> may be too large to comfortably retain. We only need a unique set of row
> keys, or even a min/max range, and total counts.
> The consumers of this information will be debugging tools. We can afford to
> apply fast compression to ring buffer entries (if codec support is
> available), something like snappy or zstandard, and decompress on the fly
> when servicing the retrieval API request. This will minimize the impact of
> retaining more information about slow requests than we do today.
> This proposal is for retention of request information only, the same
> information provided by responseTooSlow warnings. Total size of response
> serialization, possibly also total cell or row counts, should be sufficient
> to characterize the response.
> Optionally persist new entries added to the ring buffer into one or more
> files in HDFS in a write-behind manner. If the HDFS writer blocks or falls
> behind and we are unable to persist an entry before it is overwritten, that
> is fine. Response too slow logging is best effort. If we can detect this make
> a note of it in the log file. Provide a tool for parsing, dumping, filtering,
> and pretty printing the slow logs written to HDFS. The tool and the shell can
> share and reuse some utility classes and methods for accomplishing that.
> —
> New shell commands:
> {{get_slow_responses [ ... , ] [ , \{
> } ]}}
> Retrieve, decode, and pretty print the contents of the too slow response ring
> buffer maintained by the given list of servers; or all servers in the cluster
> if no list is provided.
[jira] [Updated] (HBASE-22760) Stop/Resume Snapshot Auto-Cleanup activity with shell command
[ https://issues.apache.org/jira/browse/HBASE-22760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-22760: --- Fix Version/s: (was: 2.2.2) (was: 2.1.7) (was: 1.4.11) > Stop/Resume Snapshot Auto-Cleanup activity with shell command > - > > Key: HBASE-22760 > URL: https://issues.apache.org/jira/browse/HBASE-22760 > Project: HBase > Issue Type: Improvement > Components: Admin, shell, snapshots >Affects Versions: 3.0.0, 1.5.0, 2.3.0 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.3.0 > > Attachments: HBASE-22760.branch-1.000.patch, > HBASE-22760.branch-2.000.patch, HBASE-22760.master.003.patch, > HBASE-22760.master.004.patch, HBASE-22760.master.005.patch, > HBASE-22760.master.008.patch, HBASE-22760.master.009.patch > > > For any scheduled snapshot backup activity, we would like to disable > auto-cleaner for snapshot based on TTL. However, as per HBASE-22648 we have a > config to disable snapshot auto-cleaner: > hbase.master.cleaner.snapshot.disable, which would take effect only upon > HMaster restart just similar to any other hbase-site configs. > For any running cluster, we should be able to stop/resume auto-cleanup > activity for snapshot based on shell command. Something similar to below > command should be able to stop/start cleanup chore: > hbase(main):001:0> snapshot_auto_cleanup_switch false (disable > auto-cleaner) > hbase(main):001:0> snapshot_auto_cleanup_switch true (enable auto-cleaner) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HBASE-22978) Online slow response log
[
https://issues.apache.org/jira/browse/HBASE-22978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923852#comment-16923852
]
Andrew Purtell commented on HBASE-22978:
You can invoke the shell with a command and pipe the output to a file, like
{{./bin/hbase shell 'command' > output.txt 2>&1}} . Or the output can be piped
to something else. Nothing special need be done there.
I like the idea of persisting the complete slow log with best effort, though.
Reminds me of the [MySQL slow
log|https://dev.mysql.com/doc/refman/8.0/en/slow-query-log.html]. Initially I
was thinking this could be part of general performance surveillance, where
sampling is good enough, but maybe there could be a tough to debug case that's
rare so would also be hard to catch in the ring buffers. For that we'd
configure a slow log directory in site configuration, presumably in HDFS, into
which regionservers would each append to a file, rolling at some configured
bound. A tool that decodes and prints to stdout, like HFilePrettyPrinter and
such, can mostly share common code with what we put in the regionserver to do
the same out to RPC for the shell.
> Online slow response log
>
>
> Key: HBASE-22978
> URL: https://issues.apache.org/jira/browse/HBASE-22978
> Project: HBase
> Issue Type: New Feature
> Components: Admin, Operability, regionserver, shell
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
>
> Today when an individual RPC exceeds a configurable time bound we log a
> complaint by way of the logging subsystem. These log lines look like:
> {noformat}
> 2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
> (responseTooSlow):
> {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
> "starttimems":1567203007549,
> "responsesize":6819737,
> "method":"Scan",
> "param":"region { type: REGION_NAME value:
> \"tsdb,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
> "processingtimems":28646,
> "client":"10.253.196.215:41116",
> "queuetimems":22453,
> "class":"HRegionServer"}
> {noformat}
> Unfortunately we often truncate the request parameters, like in the above
> example. We do this because the human readable representation is verbose, the
> rate of too slow warnings may be high, and the combination of these things
> can overwhelm the log capture system. The truncation is unfortunate because
> it eliminates much of the utility of the warnings. For example, the region
> name, the start and end keys, and the filter hierarchy are all important
> clues for debugging performance problems caused by moderate to low
> selectivity queries or queries made at a high rate.
> We can maintain an in-memory ring buffer of requests that were judged to be
> too slow in addition to the responseTooSlow logging. The in-memory
> representation can be complete and compressed. A new admin API and shell
> command can provide access to the ring buffer for online performance
> debugging. A modest sizing of the ring buffer will prevent excessive memory
> utilization for a minor performance debugging feature by limiting the total
> number of retained records. There is some chance a high rate of requests will
> cause information on other interesting requests to be overwritten before it
> can be read. This is the nature of a ring buffer and an acceptable trade off.
> The write request types do not require us to retain all information submitted
> in the request. We don't need to retain all key-values in the mutation, which
> may be too large to comfortably retain. We only need a unique set of row
> keys, or even a min/max range, and total counts.
> The consumers of this information will be debugging tools. We can afford to
> apply fast compression to ring buffer entries (if codec support is
> available), something like snappy or zstandard, and decompress on the fly
> when servicing the retrieval API request. This will minimize the impact of
> retaining more information about slow requests than we do today.
> This proposal is for retention of request information only, the same
> information provided by responseTooSlow warnings. Total size of response
> serialization, possibly also total cell or row counts, should be sufficient
> to characterize the response.
> Optionally persist new entries added to the ring buffer into one or more
> files in HDFS in a write-behind manner. If the HDFS writer blocks or falls
> behind and we are unable to persist an entry before it is overwritten, that
> is fine. Response too slow logging is best effort. If we can detect this make
> a note of it in the log file. Provide a tool for parsing, dumping, filtering,
> and pretty printing the slow logs
[jira] [Commented] (HBASE-22034) Backport HBASE-21401 and HBASE-22032 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923838#comment-16923838 ] Hudson commented on HBASE-22034: FAILURE: Integrated in Jenkins build PreCommit-PHOENIX-Build #2945 (See [https://builds.apache.org/job/PreCommit-PHOENIX-Build/2945/]) PHOENIX-5455 IndexedKeyValue creation fails after HBASE-22034 (gjacoby: rev 50a6aeb47a1d5a4f97ad48bb298292becfe5776e) * (edit) phoenix-core/src/main/java/org/apache/phoenix/hbase/index/wal/IndexedKeyValue.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/hbase/index/Indexer.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/hbase/index/wal/KeyValueCodec.java * (edit) phoenix-core/src/test/java/org/apache/hadoop/hbase/regionserver/wal/IndexedKeyValueTest.java * (edit) phoenix-core/src/test/java/org/apache/hadoop/hbase/regionserver/wal/ReadWriteKeyValuesWithCodecTest.java > Backport HBASE-21401 and HBASE-22032 to branch-1 > > > Key: HBASE-22034 > URL: https://issues.apache.org/jira/browse/HBASE-22034 > Project: HBase > Issue Type: Improvement >Reporter: Geoffrey Jacoby >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0 > > Attachments: HBASE-22034-branch-1.patch, HBASE-22034-branch-1.patch > > > Branch-2 and master have good validation checks when constructing KeyValues. > We should also have them on branch-1. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HBASE-22978) Online slow response log
[
https://issues.apache.org/jira/browse/HBASE-22978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-22978:
Component/s: Operability
> Online slow response log
>
>
> Key: HBASE-22978
> URL: https://issues.apache.org/jira/browse/HBASE-22978
> Project: HBase
> Issue Type: New Feature
> Components: Admin, Operability, regionserver, shell
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
>
> Today when an individual RPC exceeds a configurable time bound we log a
> complaint by way of the logging subsystem. These log lines look like:
> {noformat}
> 2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
> (responseTooSlow):
> {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
> "starttimems":1567203007549,
> "responsesize":6819737,
> "method":"Scan",
> "param":"region { type: REGION_NAME value:
> \"tsdb,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
> "processingtimems":28646,
> "client":"10.253.196.215:41116",
> "queuetimems":22453,
> "class":"HRegionServer"}
> {noformat}
> Unfortunately we often truncate the request parameters, like in the above
> example. We do this because the human readable representation is verbose, the
> rate of too slow warnings may be high, and the combination of these things
> can overwhelm the log capture system. The truncation is unfortunate because
> it eliminates much of the utility of the warnings. For example, the region
> name, the start and end keys, and the filter hierarchy are all important
> clues for debugging performance problems caused by moderate to low
> selectivity queries or queries made at a high rate.
> We can maintain an in-memory ring buffer of requests that were judged to be
> too slow in addition to the responseTooSlow logging. The in-memory
> representation can be complete and compressed. A new admin API and shell
> command can provide access to the ring buffer for online performance
> debugging. A modest sizing of the ring buffer will prevent excessive memory
> utilization for a minor performance debugging feature by limiting the total
> number of retained records. There is some chance a high rate of requests will
> cause information on other interesting requests to be overwritten before it
> can be read. This is the nature of a ring buffer and an acceptable trade off.
> The write request types do not require us to retain all information submitted
> in the request. We don't need to retain all key-values in the mutation, which
> may be too large to comfortably retain. We only need a unique set of row
> keys, or even a min/max range, and total counts.
> The consumers of this information will be debugging tools. We can afford to
> apply fast compression to ring buffer entries (if codec support is
> available), something like snappy or zstandard, and decompress on the fly
> when servicing the retrieval API request. This will minimize the impact of
> retaining more information about slow requests than we do today.
> This proposal is for retention of request information only, the same
> information provided by responseTooSlow warnings. Total size of response
> serialization, possibly also total cell or row counts, should be sufficient
> to characterize the response.
> Optionally persist new entries added to the ring buffer into one or more
> files in HDFS in a write-behind manner. If the HDFS writer blocks or falls
> behind and we are unable to persist an entry before it is overwritten, that
> is fine. Response too slow logging is best effort. If we can detect this make
> a note of it in the log file. Provide a tool for parsing, dumping, filtering,
> and pretty printing the slow logs written to HDFS. The tool and the shell can
> share and reuse some utility classes and methods for accomplishing that.
> —
> New shell commands:
> {{get_slow_responses [ ... , ] [ , \{
> } ]}}
> Retrieve, decode, and pretty print the contents of the too slow response ring
> buffer maintained by the given list of servers; or all servers in the cluster
> if no list is provided. Optionally provide a map of parameters for filtering
> as additional argument. The TABLE filter, which expects a string containing a
> table name, will include only entries pertaining to that table. The REGION
> filter, which expects a string containing an encoded region name, will
> include only entries pertaining to that region. The CLIENT_IP filter, which
> expects a string containing an IP address, will include only entries
> pertaining to that client. The USER filter, which expects a string containing
> a user name, will include only entries pertaining to that user. Filters are
> additive,
[jira] [Commented] (HBASE-22978) Online slow response log
[
https://issues.apache.org/jira/browse/HBASE-22978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923836#comment-16923836
]
Sean Busbey commented on HBASE-22978:
-
bq. I'll update the description to include write-behind of the ring buffer to a
directory in HDFS, but it shouldn't block, so if we stall during writing some
ring buffer entries may have been lost. If we can detect that we can log that
it happened in the file.
oh that's really nice. I just meant like "send the ones that match to this HDFS
directory instead of to my hbase shell here" on the assumption that I will
eventually run into a case where the buffer is eventually big enough that I
will want to look at the results with $EDITOR or other shell tools instead of
our shell and I'd rather not have to write a java tool to do it then.
> Online slow response log
>
>
> Key: HBASE-22978
> URL: https://issues.apache.org/jira/browse/HBASE-22978
> Project: HBase
> Issue Type: New Feature
> Components: Admin, regionserver, shell
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
>
> Today when an individual RPC exceeds a configurable time bound we log a
> complaint by way of the logging subsystem. These log lines look like:
> {noformat}
> 2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
> (responseTooSlow):
> {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
> "starttimems":1567203007549,
> "responsesize":6819737,
> "method":"Scan",
> "param":"region { type: REGION_NAME value:
> \"tsdb,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
> "processingtimems":28646,
> "client":"10.253.196.215:41116",
> "queuetimems":22453,
> "class":"HRegionServer"}
> {noformat}
> Unfortunately we often truncate the request parameters, like in the above
> example. We do this because the human readable representation is verbose, the
> rate of too slow warnings may be high, and the combination of these things
> can overwhelm the log capture system. The truncation is unfortunate because
> it eliminates much of the utility of the warnings. For example, the region
> name, the start and end keys, and the filter hierarchy are all important
> clues for debugging performance problems caused by moderate to low
> selectivity queries or queries made at a high rate.
> We can maintain an in-memory ring buffer of requests that were judged to be
> too slow in addition to the responseTooSlow logging. The in-memory
> representation can be complete and compressed. A new admin API and shell
> command can provide access to the ring buffer for online performance
> debugging. A modest sizing of the ring buffer will prevent excessive memory
> utilization for a minor performance debugging feature by limiting the total
> number of retained records. There is some chance a high rate of requests will
> cause information on other interesting requests to be overwritten before it
> can be read. This is the nature of a ring buffer and an acceptable trade off.
> The write request types do not require us to retain all information submitted
> in the request. We don't need to retain all key-values in the mutation, which
> may be too large to comfortably retain. We only need a unique set of row
> keys, or even a min/max range, and total counts.
> The consumers of this information will be debugging tools. We can afford to
> apply fast compression to ring buffer entries (if codec support is
> available), something like snappy or zstandard, and decompress on the fly
> when servicing the retrieval API request. This will minimize the impact of
> retaining more information about slow requests than we do today.
> This proposal is for retention of request information only, the same
> information provided by responseTooSlow warnings. Total size of response
> serialization, possibly also total cell or row counts, should be sufficient
> to characterize the response.
> Optionally persist new entries added to the ring buffer into one or more
> files in HDFS in a write-behind manner. If the HDFS writer blocks or falls
> behind and we are unable to persist an entry before it is overwritten, that
> is fine. Response too slow logging is best effort. If we can detect this make
> a note of it in the log file. Provide a tool for parsing, dumping, filtering,
> and pretty printing the slow logs written to HDFS. The tool and the shell can
> share and reuse some utility classes and methods for accomplishing that.
> —
> New shell commands:
> {{get_slow_responses [ ... , ] [ , \{
> } ]}}
> Retrieve, decode, and pretty print the contents of the too slow response ring
> buffer maintained by the given list of servers; or all servers in the
[jira] [Commented] (HBASE-22937) The RawBytesComparator in branch-1 have wrong comparison order
[
https://issues.apache.org/jira/browse/HBASE-22937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923829#comment-16923829
]
Hudson commented on HBASE-22937:
Results for branch branch-1.3
[build #957 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/957/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/957//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/957//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/957//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> The RawBytesComparator in branch-1 have wrong comparison order
> --
>
> Key: HBASE-22937
> URL: https://issues.apache.org/jira/browse/HBASE-22937
> Project: HBase
> Issue Type: Bug
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 1.5.0, 1.3.6, 1.4.11
>
>
> When digging the HBASE-22862, we found a bug in
> RawBytesComparator#compareOnlyKeyPortion (although it's unrelated to the
> corruption in HBASE-22862).
> {code}
> @Override
> @VisibleForTesting
> public int compareOnlyKeyPortion(Cell left, Cell right) {
> // ...
> return (0xff & left.getTypeByte()) - (0xff & right.getTypeByte());
> }
> {code}
> Here should be (0xff & right.getTypeByte()) - (0xff & left.getTypeByte()) I
> think.
> I can see the BloomFilter or HFile v2 are still using the comparator in
> branch-1 (but not in branch-2). Maybe we can just remove the class (if some
> HFile encoded with this comparator, then mapping to the correct KVComparator
> just like 2.x), or fix the bug in current RawBytesComparator.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22937) The RawBytesComparator in branch-1 have wrong comparison order
[
https://issues.apache.org/jira/browse/HBASE-22937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923820#comment-16923820
]
Hudson commented on HBASE-22937:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #635 (See
[https://builds.apache.org/job/HBase-1.3-IT/635/])
Amend HBASE-22937 The RawBytesComparator in branch-1 have wrong (apurtell: rev
4abfc11a722884629f7c6d1c526b1df873cb2785)
* (edit) hbase-common/src/test/java/org/apache/hadoop/hbase/TestKeyValue.java
> The RawBytesComparator in branch-1 have wrong comparison order
> --
>
> Key: HBASE-22937
> URL: https://issues.apache.org/jira/browse/HBASE-22937
> Project: HBase
> Issue Type: Bug
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 1.5.0, 1.3.6, 1.4.11
>
>
> When digging the HBASE-22862, we found a bug in
> RawBytesComparator#compareOnlyKeyPortion (although it's unrelated to the
> corruption in HBASE-22862).
> {code}
> @Override
> @VisibleForTesting
> public int compareOnlyKeyPortion(Cell left, Cell right) {
> // ...
> return (0xff & left.getTypeByte()) - (0xff & right.getTypeByte());
> }
> {code}
> Here should be (0xff & right.getTypeByte()) - (0xff & left.getTypeByte()) I
> think.
> I can see the BloomFilter or HFile v2 are still using the comparator in
> branch-1 (but not in branch-2). Maybe we can just remove the class (if some
> HFile encoded with this comparator, then mapping to the correct KVComparator
> just like 2.x), or fix the bug in current RawBytesComparator.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22701) Better handle invalid local directory for DynamicClassLoader
[
https://issues.apache.org/jira/browse/HBASE-22701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923819#comment-16923819
]
Hudson commented on HBASE-22701:
Results for branch branch-2.2
[build #579 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/579/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/579//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/579//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/579//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Better handle invalid local directory for DynamicClassLoader
>
>
> Key: HBASE-22701
> URL: https://issues.apache.org/jira/browse/HBASE-22701
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 2.3.0, 2.1.7, 2.2.2
>
> Attachments: HBASE-22701.001.patch, HBASE-22701.002.patch,
> HBASE-22701.003.patch
>
>
> If you give HBase an {{hbase.local.dir}} (usually, "{{hbase.tmp.dir}}/local")
> which is not writable to it, you will get some weird errors on the scan path.
> I just saw this (again?) with Phoenix.
> Specifically, the first attempt to reference DynamicClassLoader (via
> ProtobufUtil), will result in an ExceptionInInitializationError because the
> unchecked exception coming out of DynamicClassLoader's constructor interrupts
> the loading of {{DynamicClassLoader.class}}.
> {noformat}
> 2019-07-14 06:25:34,284 ERROR
> [RpcServer.Metadata.Fifo.handler=12,queue=0,port=16020]
> coprocessor.MetaDataEndpointImpl: dropTable failed
> org.apache.hadoop.hbase.DoNotRetryIOException:
> java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.translateException(RpcRetryingCallerImpl.java:221)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:194)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:387)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:361)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1598)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toScan(ProtobufUtil.java:1152)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2967)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3301)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:332)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:242)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58)
> at
> org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:127)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192)
> ... 7 more
> Caused by: java.lang.RuntimeException: Failed to create local dir
> /hadoopfs/fs1/hbase/local/jars, DynamicClassLoader failed to init
> at
> org.apache.hadoop.hbase.util.DynamicClassLoader.initTempDir(DynamicClassLoader.java:110)
> at
> org.apache.hadoop.hbase.util.DynamicClassLoader.(DynamicClassLoader.java:98)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder.lambda$static$0(ProtobufUtil.java:261)
> at java.security.AccessController.doPrivileged(Native Method)
>
[jira] [Commented] (HBASE-22937) The RawBytesComparator in branch-1 have wrong comparison order
[
https://issues.apache.org/jira/browse/HBASE-22937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923815#comment-16923815
]
Hudson commented on HBASE-22937:
Results for branch branch-1.4
[build #999 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/999/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/999//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/999//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/999//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> The RawBytesComparator in branch-1 have wrong comparison order
> --
>
> Key: HBASE-22937
> URL: https://issues.apache.org/jira/browse/HBASE-22937
> Project: HBase
> Issue Type: Bug
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 1.5.0, 1.3.6, 1.4.11
>
>
> When digging the HBASE-22862, we found a bug in
> RawBytesComparator#compareOnlyKeyPortion (although it's unrelated to the
> corruption in HBASE-22862).
> {code}
> @Override
> @VisibleForTesting
> public int compareOnlyKeyPortion(Cell left, Cell right) {
> // ...
> return (0xff & left.getTypeByte()) - (0xff & right.getTypeByte());
> }
> {code}
> Here should be (0xff & right.getTypeByte()) - (0xff & left.getTypeByte()) I
> think.
> I can see the BloomFilter or HFile v2 are still using the comparator in
> branch-1 (but not in branch-2). Maybe we can just remove the class (if some
> HFile encoded with this comparator, then mapping to the correct KVComparator
> just like 2.x), or fix the bug in current RawBytesComparator.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22701) Better handle invalid local directory for DynamicClassLoader
[
https://issues.apache.org/jira/browse/HBASE-22701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923816#comment-16923816
]
Hudson commented on HBASE-22701:
Results for branch branch-2.1
[build #1565 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1565/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1565//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1565//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1565//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Better handle invalid local directory for DynamicClassLoader
>
>
> Key: HBASE-22701
> URL: https://issues.apache.org/jira/browse/HBASE-22701
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 2.3.0, 2.1.7, 2.2.2
>
> Attachments: HBASE-22701.001.patch, HBASE-22701.002.patch,
> HBASE-22701.003.patch
>
>
> If you give HBase an {{hbase.local.dir}} (usually, "{{hbase.tmp.dir}}/local")
> which is not writable to it, you will get some weird errors on the scan path.
> I just saw this (again?) with Phoenix.
> Specifically, the first attempt to reference DynamicClassLoader (via
> ProtobufUtil), will result in an ExceptionInInitializationError because the
> unchecked exception coming out of DynamicClassLoader's constructor interrupts
> the loading of {{DynamicClassLoader.class}}.
> {noformat}
> 2019-07-14 06:25:34,284 ERROR
> [RpcServer.Metadata.Fifo.handler=12,queue=0,port=16020]
> coprocessor.MetaDataEndpointImpl: dropTable failed
> org.apache.hadoop.hbase.DoNotRetryIOException:
> java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.translateException(RpcRetryingCallerImpl.java:221)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:194)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:387)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:361)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1598)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toScan(ProtobufUtil.java:1152)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2967)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3301)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:332)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:242)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58)
> at
> org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:127)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192)
> ... 7 more
> Caused by: java.lang.RuntimeException: Failed to create local dir
> /hadoopfs/fs1/hbase/local/jars, DynamicClassLoader failed to init
> at
> org.apache.hadoop.hbase.util.DynamicClassLoader.initTempDir(DynamicClassLoader.java:110)
> at
> org.apache.hadoop.hbase.util.DynamicClassLoader.(DynamicClassLoader.java:98)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder.lambda$static$0(ProtobufUtil.java:261)
> at java.security.AccessController.doPrivileged(Native Method)
>
[GitHub] [hbase] saintstack commented on issue #585: HBASE-22930 Set unique name to longCompactions/shortCompactions/split threads
saintstack commented on issue #585: HBASE-22930 Set unique name to longCompactions/shortCompactions/split threads URL: https://github.com/apache/hbase/pull/585#issuecomment-528646559 Running new build. Bunch of test failures. See if related (doubt it). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-22701) Better handle invalid local directory for DynamicClassLoader
[
https://issues.apache.org/jira/browse/HBASE-22701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923814#comment-16923814
]
Hudson commented on HBASE-22701:
Results for branch master
[build #1405 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1405/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1405//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1405//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1405//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Better handle invalid local directory for DynamicClassLoader
>
>
> Key: HBASE-22701
> URL: https://issues.apache.org/jira/browse/HBASE-22701
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 2.3.0, 2.1.7, 2.2.2
>
> Attachments: HBASE-22701.001.patch, HBASE-22701.002.patch,
> HBASE-22701.003.patch
>
>
> If you give HBase an {{hbase.local.dir}} (usually, "{{hbase.tmp.dir}}/local")
> which is not writable to it, you will get some weird errors on the scan path.
> I just saw this (again?) with Phoenix.
> Specifically, the first attempt to reference DynamicClassLoader (via
> ProtobufUtil), will result in an ExceptionInInitializationError because the
> unchecked exception coming out of DynamicClassLoader's constructor interrupts
> the loading of {{DynamicClassLoader.class}}.
> {noformat}
> 2019-07-14 06:25:34,284 ERROR
> [RpcServer.Metadata.Fifo.handler=12,queue=0,port=16020]
> coprocessor.MetaDataEndpointImpl: dropTable failed
> org.apache.hadoop.hbase.DoNotRetryIOException:
> java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.translateException(RpcRetryingCallerImpl.java:221)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:194)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:387)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:361)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1598)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toScan(ProtobufUtil.java:1152)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2967)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3301)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:332)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:242)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58)
> at
> org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:127)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192)
> ... 7 more
> Caused by: java.lang.RuntimeException: Failed to create local dir
> /hadoopfs/fs1/hbase/local/jars, DynamicClassLoader failed to init
> at
> org.apache.hadoop.hbase.util.DynamicClassLoader.initTempDir(DynamicClassLoader.java:110)
> at
> org.apache.hadoop.hbase.util.DynamicClassLoader.(DynamicClassLoader.java:98)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder.lambda$static$0(ProtobufUtil.java:261)
> at java.security.AccessController.doPrivileged(Native Method)
> at
>
[GitHub] [hbase-operator-tools] busbey commented on issue #16: HBASE-22825 [HBCK2] Add a client-side to hbase-operator-tools that ca…
busbey commented on issue #16: HBASE-22825 [HBCK2] Add a client-side to hbase-operator-tools that ca… URL: https://github.com/apache/hbase-operator-tools/pull/16#issuecomment-528645702 Sounds good. Next time you should be able to push a commit into this PR with the amendment you want, fyi. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-22701) Better handle invalid local directory for DynamicClassLoader
[
https://issues.apache.org/jira/browse/HBASE-22701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923812#comment-16923812
]
Hudson commented on HBASE-22701:
Results for branch branch-2
[build #2231 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2231/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2231//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2231//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2231//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Better handle invalid local directory for DynamicClassLoader
>
>
> Key: HBASE-22701
> URL: https://issues.apache.org/jira/browse/HBASE-22701
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 2.3.0, 2.1.7, 2.2.2
>
> Attachments: HBASE-22701.001.patch, HBASE-22701.002.patch,
> HBASE-22701.003.patch
>
>
> If you give HBase an {{hbase.local.dir}} (usually, "{{hbase.tmp.dir}}/local")
> which is not writable to it, you will get some weird errors on the scan path.
> I just saw this (again?) with Phoenix.
> Specifically, the first attempt to reference DynamicClassLoader (via
> ProtobufUtil), will result in an ExceptionInInitializationError because the
> unchecked exception coming out of DynamicClassLoader's constructor interrupts
> the loading of {{DynamicClassLoader.class}}.
> {noformat}
> 2019-07-14 06:25:34,284 ERROR
> [RpcServer.Metadata.Fifo.handler=12,queue=0,port=16020]
> coprocessor.MetaDataEndpointImpl: dropTable failed
> org.apache.hadoop.hbase.DoNotRetryIOException:
> java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.translateException(RpcRetryingCallerImpl.java:221)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:194)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:387)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:361)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1598)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toScan(ProtobufUtil.java:1152)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2967)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3301)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:332)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:242)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58)
> at
> org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:127)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192)
> ... 7 more
> Caused by: java.lang.RuntimeException: Failed to create local dir
> /hadoopfs/fs1/hbase/local/jars, DynamicClassLoader failed to init
> at
> org.apache.hadoop.hbase.util.DynamicClassLoader.initTempDir(DynamicClassLoader.java:110)
> at
> org.apache.hadoop.hbase.util.DynamicClassLoader.(DynamicClassLoader.java:98)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder.lambda$static$0(ProtobufUtil.java:261)
> at java.security.AccessController.doPrivileged(Native Method)
> at
[GitHub] [hbase-operator-tools] saintstack commented on issue #18: HBASE-22567 - HBCK2 addMissingRegionsToMeta
saintstack commented on issue #18: HBASE-22567 - HBCK2 addMissingRegionsToMeta URL: https://github.com/apache/hbase-operator-tools/pull/18#issuecomment-528643918 Lets try and get this in @wchevreuil I want to make a release of the hbase-operator-tools. Need this. Once in, I'll some text to the fixit section in hbck2 README talking up this facility. Thanks boss. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase-operator-tools] saintstack commented on a change in pull request #18: HBASE-22567 - HBCK2 addMissingRegionsToMeta
saintstack commented on a change in pull request #18: HBASE-22567 - HBCK2
addMissingRegionsToMeta
URL:
https://github.com/apache/hbase-operator-tools/pull/18#discussion_r321523746
##
File path: hbase-hbck2/src/main/java/org/apache/hbase/HBCK2.java
##
@@ -164,6 +178,102 @@ int setRegionState(String region, RegionState.State
newState)
return EXIT_FAILURE;
}
+ Map> reportTablesWithMissingRegionsInMeta(String...
nameSpaceOrTable)
+ throws Exception {
+Map> report;
+try(final MetaFixer metaFixer = new MetaFixer(this.conf)){
+ List names = nameSpaceOrTable != null ?
Arrays.asList(nameSpaceOrTable) : null;
+ report = metaFixer.reportTablesMissingRegions(names);
+} catch (Exception e) {
+ LOG.error("Error reporting missing regions: ", e);
+ throw e;
+}
+if(LOG.isDebugEnabled()) {
+ LOG.debug(formatMissingRegionsInMetaReport(report));
+}
+return report;
+ }
+
+ List addMissingRegionsInMeta(List regionsPath) throws
IOException {
+List reAddedRegionsEncodedNames = new ArrayList<>();
+try(final MetaFixer metaFixer = new MetaFixer(this.conf)){
+ for(Path regionPath : regionsPath){
+metaFixer.putRegionInfoFromHdfsInMeta(regionPath);
+reAddedRegionsEncodedNames.add(regionPath.getName());
+ }
+}
+return reAddedRegionsEncodedNames;
+ }
+
+ Pair, List>
addMissingRegionsInMetaForTables(String...
+ nameSpaceOrTable) {
+ExecutorService executorService = Executors.newFixedThreadPool(
+ (nameSpaceOrTable == null ||
+nameSpaceOrTable.length > Runtime.getRuntime().availableProcessors()) ?
+ Runtime.getRuntime().availableProcessors() :
+ nameSpaceOrTable.length);
+List>> futures = new ArrayList<>( nameSpaceOrTable ==
null ? 1 :
+ nameSpaceOrTable.length);
+final List readdedRegionNames = new ArrayList<>();
+List executionErrors = new ArrayList<>();
+try {
+ //reducing number of retries in case disable fails due to namespace
table region also missing
+ this.conf.setInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER, 1);
+ try(Connection conn = ConnectionFactory.createConnection(this.conf);
+final Admin admin = conn.getAdmin()) {
+Map> report =
reportTablesWithMissingRegionsInMeta(nameSpaceOrTable);
+for (TableName tableName : report.keySet()) {
+ if(admin.tableExists(tableName)) {
+futures.add(executorService.submit(new Callable>() {
+ @Override
+ public List call() throws Exception {
+LOG.debug("running thread for {}",
tableName.getNameWithNamespaceInclAsString());
+try {
+ admin.disableTable(tableName);
Review comment:
I think so. Could make things worse?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (HBASE-22796) [HBCK2] Add fix of overlaps to fixMeta hbck Service
[ https://issues.apache.org/jira/browse/HBASE-22796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923809#comment-16923809 ] stack commented on HBASE-22796: --- Ok. Let me give this a go [~jatsakthi]. I think you on holiday (smile). > [HBCK2] Add fix of overlaps to fixMeta hbck Service > --- > > Key: HBASE-22796 > URL: https://issues.apache.org/jira/browse/HBASE-22796 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Sakthi >Priority: Major > > fixMeta currently does holes in meta only courtesy of HBASE-22771 which added > fixMeta to hbck Service; missing was fix of overlaps too. This JIRA is about > adding fix of overlaps to general fixMeta call. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HBASE-22701) Better handle invalid local directory for DynamicClassLoader
[
https://issues.apache.org/jira/browse/HBASE-22701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-22701:
---
Hadoop Flags: Reviewed
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks for the reviews, all.
> Better handle invalid local directory for DynamicClassLoader
>
>
> Key: HBASE-22701
> URL: https://issues.apache.org/jira/browse/HBASE-22701
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 2.3.0, 2.1.7, 2.2.2
>
> Attachments: HBASE-22701.001.patch, HBASE-22701.002.patch,
> HBASE-22701.003.patch
>
>
> If you give HBase an {{hbase.local.dir}} (usually, "{{hbase.tmp.dir}}/local")
> which is not writable to it, you will get some weird errors on the scan path.
> I just saw this (again?) with Phoenix.
> Specifically, the first attempt to reference DynamicClassLoader (via
> ProtobufUtil), will result in an ExceptionInInitializationError because the
> unchecked exception coming out of DynamicClassLoader's constructor interrupts
> the loading of {{DynamicClassLoader.class}}.
> {noformat}
> 2019-07-14 06:25:34,284 ERROR
> [RpcServer.Metadata.Fifo.handler=12,queue=0,port=16020]
> coprocessor.MetaDataEndpointImpl: dropTable failed
> org.apache.hadoop.hbase.DoNotRetryIOException:
> java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.translateException(RpcRetryingCallerImpl.java:221)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:194)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:387)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:361)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1598)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toScan(ProtobufUtil.java:1152)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2967)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3301)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:332)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:242)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58)
> at
> org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:127)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192)
> ... 7 more
> Caused by: java.lang.RuntimeException: Failed to create local dir
> /hadoopfs/fs1/hbase/local/jars, DynamicClassLoader failed to init
> at
> org.apache.hadoop.hbase.util.DynamicClassLoader.initTempDir(DynamicClassLoader.java:110)
> at
> org.apache.hadoop.hbase.util.DynamicClassLoader.(DynamicClassLoader.java:98)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder.lambda$static$0(ProtobufUtil.java:261)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder.(ProtobufUtil.java:260)
> ... 16 more
> {noformat}
> Every subsequent call will result in a NoClassDefFoundError, because we
> already tried to load DynamicClassLoader.class once and failed.
> {noformat}
> 2019-07-14 06:25:34,380 ERROR
> [RpcServer.Metadata.Fifo.handler=2,queue=2,port=16020]
> coprocessor.MetaDataEndpointImpl: dropTable failed
> org.apache.hadoop.hbase.DoNotRetryIOException:
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.translateException(RpcRetryingCallerImpl.java:221)
> at
>
[jira] [Resolved] (HBASE-22825) [HBCK2] Add a client-side to hbase-operator-tools that can exploit fixMeta added in server side
[ https://issues.apache.org/jira/browse/HBASE-22825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-22825. --- Fix Version/s: hbase-operator-tools-1.0.0 Hadoop Flags: Reviewed Resolution: Fixed Merged after adjusting hbase version to refer to released 2.1.6. Thanks for patch [~jatsakthi] (and [~busbey] for review) > [HBCK2] Add a client-side to hbase-operator-tools that can exploit fixMeta > added in server side > --- > > Key: HBASE-22825 > URL: https://issues.apache.org/jira/browse/HBASE-22825 > Project: HBase > Issue Type: Sub-task > Components: hbck2 >Reporter: Sakthi >Assignee: Sakthi >Priority: Major > Fix For: hbase-operator-tools-1.0.0 > > > fixMeta was added in server side in HBASE-22771. Let's add the client side to > the hbase-operator-tools to utilize that. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HBASE-22937) The RawBytesComparator in branch-1 have wrong comparison order
[
https://issues.apache.org/jira/browse/HBASE-22937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923783#comment-16923783
]
Hudson commented on HBASE-22937:
Results for branch branch-1.3
[build #956 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/956/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/956//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/956//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/956//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> The RawBytesComparator in branch-1 have wrong comparison order
> --
>
> Key: HBASE-22937
> URL: https://issues.apache.org/jira/browse/HBASE-22937
> Project: HBase
> Issue Type: Bug
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 1.5.0, 1.3.6, 1.4.11
>
>
> When digging the HBASE-22862, we found a bug in
> RawBytesComparator#compareOnlyKeyPortion (although it's unrelated to the
> corruption in HBASE-22862).
> {code}
> @Override
> @VisibleForTesting
> public int compareOnlyKeyPortion(Cell left, Cell right) {
> // ...
> return (0xff & left.getTypeByte()) - (0xff & right.getTypeByte());
> }
> {code}
> Here should be (0xff & right.getTypeByte()) - (0xff & left.getTypeByte()) I
> think.
> I can see the BloomFilter or HFile v2 are still using the comparator in
> branch-1 (but not in branch-2). Maybe we can just remove the class (if some
> HFile encoded with this comparator, then mapping to the correct KVComparator
> just like 2.x), or fix the bug in current RawBytesComparator.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Comment Edited] (HBASE-22978) Online slow response log
[
https://issues.apache.org/jira/browse/HBASE-22978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923772#comment-16923772
]
Andrew Purtell edited comment on HBASE-22978 at 9/5/19 10:37 PM:
-
Description updated. Admin API simplified. The {{get_slow_responses}} shell
command is reconsidered as a general query tool.
Pseudocode of this command would be like:
{noformat}
servers = nil
filters = ...
if filters['REGION']
servers = locateRegion( filters['REGION'] )
elsif filters['TABLE']
servers = locateRegions( filters['TABLE'] )
results = @admin.getSlowResponses(servers).map {
// if a returned slow response entry matches any filter, return the
value, else nil
}.compact // remove any nils
// pretty print 'results'
{noformat}
was (Author: apurtell):
Description updated. Admin API simplified. The {{get_slow_responses}} shell
command is reconsidered as a general query tool.
Pseudocode of this command would be like:
{noformat}
servers = []
filters = ...
if filters['REGION']
servers = locateRegion( filters['REGION'] )
elsif filters['TABLE']
servers = locateRegions( filters['TABLE'] )
results = @admin.getSlowResponses(servers.empty? ? nil : servers).map {
// if a returned slow response entry matches any filter, return the
value, else nil
}.compact // remove any nils
// pretty print 'results'
{noformat}
> Online slow response log
>
>
> Key: HBASE-22978
> URL: https://issues.apache.org/jira/browse/HBASE-22978
> Project: HBase
> Issue Type: New Feature
> Components: Admin, regionserver, shell
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
>
> Today when an individual RPC exceeds a configurable time bound we log a
> complaint by way of the logging subsystem. These log lines look like:
> {noformat}
> 2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
> (responseTooSlow):
> {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
> "starttimems":1567203007549,
> "responsesize":6819737,
> "method":"Scan",
> "param":"region { type: REGION_NAME value:
> \"tsdb,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
> "processingtimems":28646,
> "client":"10.253.196.215:41116",
> "queuetimems":22453,
> "class":"HRegionServer"}
> {noformat}
> Unfortunately we often truncate the request parameters, like in the above
> example. We do this because the human readable representation is verbose, the
> rate of too slow warnings may be high, and the combination of these things
> can overwhelm the log capture system. The truncation is unfortunate because
> it eliminates much of the utility of the warnings. For example, the region
> name, the start and end keys, and the filter hierarchy are all important
> clues for debugging performance problems caused by moderate to low
> selectivity queries or queries made at a high rate.
> We can maintain an in-memory ring buffer of requests that were judged to be
> too slow in addition to the responseTooSlow logging. The in-memory
> representation can be complete and compressed. A new admin API and shell
> command can provide access to the ring buffer for online performance
> debugging. A modest sizing of the ring buffer will prevent excessive memory
> utilization for a minor performance debugging feature by limiting the total
> number of retained records. There is some chance a high rate of requests will
> cause information on other interesting requests to be overwritten before it
> can be read. This is the nature of a ring buffer and an acceptable trade off.
> The write request types do not require us to retain all information submitted
> in the request. We don't need to retain all key-values in the mutation, which
> may be too large to comfortably retain. We only need a unique set of row
> keys, or even a min/max range, and total counts.
> The consumers of this information will be debugging tools. We can afford to
> apply fast compression to ring buffer entries (if codec support is
> available), something like snappy or zstandard, and decompress on the fly
> when servicing the retrieval API request. This will minimize the impact of
> retaining more information about slow requests than we do today.
> This proposal is for retention of request information only, the same
> information provided by responseTooSlow warnings. Total size of response
> serialization, possibly also total cell or row counts, should be sufficient
> to characterize the response.
> Optionally persist new entries added to the ring buffer into one or more
> files in HDFS in a write-behind manner. If the HDFS writer blocks or falls
> behind and we are unable to persist an entry
[GitHub] [hbase-operator-tools] saintstack closed pull request #16: HBASE-22825 [HBCK2] Add a client-side to hbase-operator-tools that ca…
saintstack closed pull request #16: HBASE-22825 [HBCK2] Add a client-side to hbase-operator-tools that ca… URL: https://github.com/apache/hbase-operator-tools/pull/16 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase-operator-tools] saintstack commented on issue #16: HBASE-22825 [HBCK2] Add a client-side to hbase-operator-tools that ca…
saintstack commented on issue #16: HBASE-22825 [HBCK2] Add a client-side to hbase-operator-tools that ca… URL: https://github.com/apache/hbase-operator-tools/pull/16#issuecomment-528618119 Let me close this one and make a new one w/ @jatsakthi 's patch amended referring to released 2.1.6. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-22736) Add text to show how to fix the HBCK checking result
[ https://issues.apache.org/jira/browse/HBASE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923781#comment-16923781 ] stack commented on HBASE-22736: --- Moved to be subtask of HBCK2 version 2.0. > Add text to show how to fix the HBCK checking result > > > Key: HBASE-22736 > URL: https://issues.apache.org/jira/browse/HBASE-22736 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Priority: Major > > A follow-on issue after HBASE-22709. > Copied the comment here. > bq. In follow-on, we can add text on how to fix including commands. Also may > want a json version of this page. Lets see. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HBASE-22736) Add text to show how to fix the HBCK checking result
[ https://issues.apache.org/jira/browse/HBASE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-22736: -- Parent: HBASE-22977 Issue Type: Sub-task (was: Bug) > Add text to show how to fix the HBCK checking result > > > Key: HBASE-22736 > URL: https://issues.apache.org/jira/browse/HBASE-22736 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Priority: Major > > A follow-on issue after HBASE-22709. > Copied the comment here. > bq. In follow-on, we can add text on how to fix including commands. Also may > want a json version of this page. Lets see. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HBASE-22978) Online slow response log
[
https://issues.apache.org/jira/browse/HBASE-22978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-22978:
---
Description:
Today when an individual RPC exceeds a configurable time bound we log a
complaint by way of the logging subsystem. These log lines look like:
{noformat}
2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
(responseTooSlow):
{"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
"starttimems":1567203007549,
"responsesize":6819737,
"method":"Scan",
"param":"region { type: REGION_NAME value:
\"tsdb,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
"processingtimems":28646,
"client":"10.253.196.215:41116",
"queuetimems":22453,
"class":"HRegionServer"}
{noformat}
Unfortunately we often truncate the request parameters, like in the above
example. We do this because the human readable representation is verbose, the
rate of too slow warnings may be high, and the combination of these things can
overwhelm the log capture system. The truncation is unfortunate because it
eliminates much of the utility of the warnings. For example, the region name,
the start and end keys, and the filter hierarchy are all important clues for
debugging performance problems caused by moderate to low selectivity queries or
queries made at a high rate.
We can maintain an in-memory ring buffer of requests that were judged to be too
slow in addition to the responseTooSlow logging. The in-memory representation
can be complete and compressed. A new admin API and shell command can provide
access to the ring buffer for online performance debugging. A modest sizing of
the ring buffer will prevent excessive memory utilization for a minor
performance debugging feature by limiting the total number of retained records.
There is some chance a high rate of requests will cause information on other
interesting requests to be overwritten before it can be read. This is the
nature of a ring buffer and an acceptable trade off.
The write request types do not require us to retain all information submitted
in the request. We don't need to retain all key-values in the mutation, which
may be too large to comfortably retain. We only need a unique set of row keys,
or even a min/max range, and total counts.
The consumers of this information will be debugging tools. We can afford to
apply fast compression to ring buffer entries (if codec support is available),
something like snappy or zstandard, and decompress on the fly when servicing
the retrieval API request. This will minimize the impact of retaining more
information about slow requests than we do today.
This proposal is for retention of request information only, the same
information provided by responseTooSlow warnings. Total size of response
serialization, possibly also total cell or row counts, should be sufficient to
characterize the response.
Optionally persist new entries added to the ring buffer into one or more files
in HDFS in a write-behind manner. If the HDFS writer blocks or falls behind and
we are unable to persist an entry before it is overwritten, that is fine.
Response too slow logging is best effort. If we can detect this make a note of
it in the log file. Provide a tool for parsing, dumping, filtering, and pretty
printing the slow logs written to HDFS. The tool and the shell can share and
reuse some utility classes and methods for accomplishing that.
—
New shell commands:
{{get_slow_responses [ ... , ] [ , \{ }
]}}
Retrieve, decode, and pretty print the contents of the too slow response ring
buffer maintained by the given list of servers; or all servers in the cluster
if no list is provided. Optionally provide a map of parameters for filtering as
additional argument. The TABLE filter, which expects a string containing a
table name, will include only entries pertaining to that table. The REGION
filter, which expects a string containing an encoded region name, will include
only entries pertaining to that region. The CLIENT_IP filter, which expects a
string containing an IP address, will include only entries pertaining to that
client. The USER filter, which expects a string containing a user name, will
include only entries pertaining to that user. Filters are additive, for example
if both CLIENT_IP and USER filters are given, entries matching either or both
conditions will be included. The exception to this is if both TABLE and REGION
filters are provided, REGION will be preferred and TABLE will be ignored. A
server name is its host, port, and start code, e.g.
"host187.example.com,60020,1289493121758".
{{clear_slow_responses [ ... , ]}}
Clear the too slow response ring buffer maintained by the given list of
servers; or all servers on the cluster if no argument is provided. A
[jira] [Updated] (HBASE-22736) Add text to show how to fix the HBCK checking result
[ https://issues.apache.org/jira/browse/HBASE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-22736: -- Parent: (was: HBASE-21745) Issue Type: Bug (was: Sub-task) > Add text to show how to fix the HBCK checking result > > > Key: HBASE-22736 > URL: https://issues.apache.org/jira/browse/HBASE-22736 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Priority: Major > > A follow-on issue after HBASE-22709. > Copied the comment here. > bq. In follow-on, we can add text on how to fix including commands. Also may > want a json version of this page. Lets see. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (HBASE-22978) Online slow response log
[
https://issues.apache.org/jira/browse/HBASE-22978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell reassigned HBASE-22978:
--
Assignee: Andrew Purtell
> Online slow response log
>
>
> Key: HBASE-22978
> URL: https://issues.apache.org/jira/browse/HBASE-22978
> Project: HBase
> Issue Type: New Feature
> Components: Admin, regionserver, shell
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
>
> Today when an individual RPC exceeds a configurable time bound we log a
> complaint by way of the logging subsystem. These log lines look like:
> {noformat}
> 2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
> (responseTooSlow):
> {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
> "starttimems":1567203007549,
> "responsesize":6819737,
> "method":"Scan",
> "param":"region { type: REGION_NAME value:
> \"tsdb,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
> "processingtimems":28646,
> "client":"10.253.196.215:41116",
> "queuetimems":22453,
> "class":"HRegionServer"}
> {noformat}
> Unfortunately we often truncate the request parameters, like in the above
> example. We do this because the human readable representation is verbose, the
> rate of too slow warnings may be high, and the combination of these things
> can overwhelm the log capture system. The truncation is unfortunate because
> it eliminates much of the utility of the warnings. For example, the region
> name, the start and end keys, and the filter hierarchy are all important
> clues for debugging performance problems caused by moderate to low
> selectivity queries or queries made at a high rate.
> We can maintain an in-memory ring buffer of requests that were judged to be
> too slow in addition to the responseTooSlow logging. The in-memory
> representation can be complete and compressed. A new admin API and shell
> command can provide access to the ring buffer for online performance
> debugging. A modest sizing of the ring buffer will prevent excessive memory
> utilization for a minor performance debugging feature by limiting the total
> number of retained records. There is some chance a high rate of requests will
> cause information on other interesting requests to be overwritten before it
> can be read. This is the nature of a ring buffer and an acceptable trade off.
> The write request types do not require us to retain all information submitted
> in the request. We don't need to retain all key-values in the mutation, which
> may be too large to comfortably retain. We only need a unique set of row
> keys, or even a min/max range, and total counts.
> The consumers of this information will be debugging tools. We can afford to
> apply fast compression to ring buffer entries (if codec support is
> available), something like snappy or zstandard, and decompress on the fly
> when servicing the retrieval API request. This will minimize the impact of
> retaining more information about slow requests than we do today.
> This proposal is for retention of request information only, the same
> information provided by responseTooSlow warnings. Total size of response
> serialization, possibly also total cell or row counts, should be sufficient
> to characterize the response.
> Optionally persist new entries added to the ring buffer into one or more
> files in HDFS in a write-behind manner. If the HDFS writer blocks or falls
> behind and we are unable to persist an entry before it is overwritten, that
> is fine. Response too slow logging is best effort. If we can detect this make
> a note of it in the log file.
> —
> New shell commands:
> {{get_slow_responses [ ... , ] [ , \{
> } ]}}
> Retrieve, decode, and pretty print the contents of the too slow response ring
> buffer maintained by the given list of servers; or all servers in the cluster
> if no list is provided. Optionally provide a map of parameters for filtering
> as additional argument. The TABLE filter, which expects a string containing a
> table name, will include only entries pertaining to that table. The REGION
> filter, which expects a string containing an encoded region name, will
> include only entries pertaining to that region. The CLIENT_IP filter, which
> expects a string containing an IP address, will include only entries
> pertaining to that client. The USER filter, which expects a string containing
> a user name, will include only entries pertaining to that user. Filters are
> additive, for example if both CLIENT_IP and USER filters are given, entries
> matching either or both conditions will be included. The exception to this is
> if both TABLE and REGION filters are provided, REGION
[jira] [Commented] (HBASE-22859) [HBCK2] Fix the orphan regions on filesystem
[ https://issues.apache.org/jira/browse/HBASE-22859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923779#comment-16923779 ] stack commented on HBASE-22859: --- Also pushed a small edit on hbase-operator-tools under rubric of this issue... Its an edit that points to the refguide section added here and undoes a recommendation I've found subsequently doesn't work (--internal-classpath) but for which the alternative does. > [HBCK2] Fix the orphan regions on filesystem > > > Key: HBASE-22859 > URL: https://issues.apache.org/jira/browse/HBASE-22859 > Project: HBase > Issue Type: Sub-task > Components: documentation, hbck2 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2 > > Attachments: HBASE-22859.master.001.patch, > HBASE-22859.master.002.patch > > > Plan to add this feature to HBCK2 tool firstly. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HBASE-22701) Better handle invalid local directory for DynamicClassLoader
[
https://issues.apache.org/jira/browse/HBASE-22701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923777#comment-16923777
]
Josh Elser commented on HBASE-22701:
{noformat}
2019-09-05 18:24:02,100 ERROR [regionserver/localhost:16020]
util.DynamicClassLoader: Disabling the DynamicClassLoader as it failed to
initialize its temp directory. Check your configuration and filesystem
permissions. Custom coprocessor code may not be loaded as a result of this
failure.
java.lang.RuntimeException: Failed to create local dir /tmp/hbase/local/jars,
DynamicClassLoader failed to init
at
org.apache.hadoop.hbase.util.DynamicClassLoader.initTempDir(DynamicClassLoader.java:124)
at
org.apache.hadoop.hbase.util.DynamicClassLoader.(DynamicClassLoader.java:104)
at
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException$ClassLoaderHolder.lambda$static$0(RemoteWithExtrasException.java:55)
at java.security.AccessController.doPrivileged(Native Method)
at
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException$ClassLoaderHolder.(RemoteWithExtrasException.java:54)
at
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:78)
at
org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:362)
at
org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:340)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2685)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:981)
at java.lang.Thread.run(Thread.java:748)
{noformat}
This is what you'd see now by default. Much nicer looking.
> Better handle invalid local directory for DynamicClassLoader
>
>
> Key: HBASE-22701
> URL: https://issues.apache.org/jira/browse/HBASE-22701
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 2.3.0, 2.1.7, 2.2.2
>
> Attachments: HBASE-22701.001.patch, HBASE-22701.002.patch,
> HBASE-22701.003.patch
>
>
> If you give HBase an {{hbase.local.dir}} (usually, "{{hbase.tmp.dir}}/local")
> which is not writable to it, you will get some weird errors on the scan path.
> I just saw this (again?) with Phoenix.
> Specifically, the first attempt to reference DynamicClassLoader (via
> ProtobufUtil), will result in an ExceptionInInitializationError because the
> unchecked exception coming out of DynamicClassLoader's constructor interrupts
> the loading of {{DynamicClassLoader.class}}.
> {noformat}
> 2019-07-14 06:25:34,284 ERROR
> [RpcServer.Metadata.Fifo.handler=12,queue=0,port=16020]
> coprocessor.MetaDataEndpointImpl: dropTable failed
> org.apache.hadoop.hbase.DoNotRetryIOException:
> java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.translateException(RpcRetryingCallerImpl.java:221)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:194)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:387)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:361)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1598)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toScan(ProtobufUtil.java:1152)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2967)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3301)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:332)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:242)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58)
> at
> org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:127)
>
[jira] [Comment Edited] (HBASE-22978) Online slow response log
[
https://issues.apache.org/jira/browse/HBASE-22978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923772#comment-16923772
]
Andrew Purtell edited comment on HBASE-22978 at 9/5/19 10:25 PM:
-
Description updated. Admin API simplified. The {{get_slow_responses}} shell
command is reconsidered as a general query tool.
Pseudocode of this command would be like:
{noformat}
servers = []
filters = ...
if filters['REGION']
servers = locateRegion( filters['REGION'] )
elsif filters['TABLE']
servers = locateRegions( filters['TABLE'] )
results = @admin.getSlowResponses(servers.empty? ? nil : servers).map {
// if a returned slow response entry matches any filter, return the
value, else nil
}.compact // remove any nils
// pretty print 'results'
{noformat}
was (Author: apurtell):
Description updated. Admin API simplified. The {{get_slow_responses}} shell
command is reconsidered as a general query tool.
> Online slow response log
>
>
> Key: HBASE-22978
> URL: https://issues.apache.org/jira/browse/HBASE-22978
> Project: HBase
> Issue Type: New Feature
> Components: Admin, regionserver, shell
>Reporter: Andrew Purtell
>Priority: Minor
>
> Today when an individual RPC exceeds a configurable time bound we log a
> complaint by way of the logging subsystem. These log lines look like:
> {noformat}
> 2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
> (responseTooSlow):
> {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
> "starttimems":1567203007549,
> "responsesize":6819737,
> "method":"Scan",
> "param":"region { type: REGION_NAME value:
> \"tsdb,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
> "processingtimems":28646,
> "client":"10.253.196.215:41116",
> "queuetimems":22453,
> "class":"HRegionServer"}
> {noformat}
> Unfortunately we often truncate the request parameters, like in the above
> example. We do this because the human readable representation is verbose, the
> rate of too slow warnings may be high, and the combination of these things
> can overwhelm the log capture system. The truncation is unfortunate because
> it eliminates much of the utility of the warnings. For example, the region
> name, the start and end keys, and the filter hierarchy are all important
> clues for debugging performance problems caused by moderate to low
> selectivity queries or queries made at a high rate.
> We can maintain an in-memory ring buffer of requests that were judged to be
> too slow in addition to the responseTooSlow logging. The in-memory
> representation can be complete and compressed. A new admin API and shell
> command can provide access to the ring buffer for online performance
> debugging. A modest sizing of the ring buffer will prevent excessive memory
> utilization for a minor performance debugging feature by limiting the total
> number of retained records. There is some chance a high rate of requests will
> cause information on other interesting requests to be overwritten before it
> can be read. This is the nature of a ring buffer and an acceptable trade off.
> The write request types do not require us to retain all information submitted
> in the request. We don't need to retain all key-values in the mutation, which
> may be too large to comfortably retain. We only need a unique set of row
> keys, or even a min/max range, and total counts.
> The consumers of this information will be debugging tools. We can afford to
> apply fast compression to ring buffer entries (if codec support is
> available), something like snappy or zstandard, and decompress on the fly
> when servicing the retrieval API request. This will minimize the impact of
> retaining more information about slow requests than we do today.
> This proposal is for retention of request information only, the same
> information provided by responseTooSlow warnings. Total size of response
> serialization, possibly also total cell or row counts, should be sufficient
> to characterize the response.
> Optionally persist new entries added to the ring buffer into one or more
> files in HDFS in a write-behind manner. If the HDFS writer blocks or falls
> behind and we are unable to persist an entry before it is overwritten, that
> is fine. Response too slow logging is best effort. If we can detect this make
> a note of it in the log file.
> —
> New shell commands:
> {{get_slow_responses [ ... , ] [ , \{
> } ]}}
> Retrieve, decode, and pretty print the contents of the too slow response ring
> buffer maintained by the given list of servers; or all servers in the cluster
> if no list is provided. Optionally provide a map of parameters for filtering
> as
[jira] [Commented] (HBASE-22978) Online slow response log
[
https://issues.apache.org/jira/browse/HBASE-22978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923772#comment-16923772
]
Andrew Purtell commented on HBASE-22978:
Description updated. Admin API simplified. The {{get_slow_responses}} shell
command is reconsidered as a general query tool.
> Online slow response log
>
>
> Key: HBASE-22978
> URL: https://issues.apache.org/jira/browse/HBASE-22978
> Project: HBase
> Issue Type: New Feature
> Components: Admin, regionserver, shell
>Reporter: Andrew Purtell
>Priority: Minor
>
> Today when an individual RPC exceeds a configurable time bound we log a
> complaint by way of the logging subsystem. These log lines look like:
> {noformat}
> 2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
> (responseTooSlow):
> {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
> "starttimems":1567203007549,
> "responsesize":6819737,
> "method":"Scan",
> "param":"region { type: REGION_NAME value:
> \"tsdb,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
> "processingtimems":28646,
> "client":"10.253.196.215:41116",
> "queuetimems":22453,
> "class":"HRegionServer"}
> {noformat}
> Unfortunately we often truncate the request parameters, like in the above
> example. We do this because the human readable representation is verbose, the
> rate of too slow warnings may be high, and the combination of these things
> can overwhelm the log capture system. The truncation is unfortunate because
> it eliminates much of the utility of the warnings. For example, the region
> name, the start and end keys, and the filter hierarchy are all important
> clues for debugging performance problems caused by moderate to low
> selectivity queries or queries made at a high rate.
> We can maintain an in-memory ring buffer of requests that were judged to be
> too slow in addition to the responseTooSlow logging. The in-memory
> representation can be complete and compressed. A new admin API and shell
> command can provide access to the ring buffer for online performance
> debugging. A modest sizing of the ring buffer will prevent excessive memory
> utilization for a minor performance debugging feature by limiting the total
> number of retained records. There is some chance a high rate of requests will
> cause information on other interesting requests to be overwritten before it
> can be read. This is the nature of a ring buffer and an acceptable trade off.
> The write request types do not require us to retain all information submitted
> in the request. We don't need to retain all key-values in the mutation, which
> may be too large to comfortably retain. We only need a unique set of row
> keys, or even a min/max range, and total counts.
> The consumers of this information will be debugging tools. We can afford to
> apply fast compression to ring buffer entries (if codec support is
> available), something like snappy or zstandard, and decompress on the fly
> when servicing the retrieval API request. This will minimize the impact of
> retaining more information about slow requests than we do today.
> This proposal is for retention of request information only, the same
> information provided by responseTooSlow warnings. Total size of response
> serialization, possibly also total cell or row counts, should be sufficient
> to characterize the response.
> Optionally persist new entries added to the ring buffer into one or more
> files in HDFS in a write-behind manner. If the HDFS writer blocks or falls
> behind and we are unable to persist an entry before it is overwritten, that
> is fine. Response too slow logging is best effort. If we can detect this make
> a note of it in the log file.
> —
> New shell commands:
> {{get_slow_responses [ ... , ] [ , \{
> } ]}}
> Retrieve, decode, and pretty print the contents of the too slow response ring
> buffer maintained by the given list of servers; or all servers in the cluster
> if no list is provided. Optionally provide a map of parameters for filtering
> as additional argument. The TABLE filter, which expects a string containing a
> table name, will include only entries pertaining to that table. The REGION
> filter, which expects a string containing an encoded region name, will
> include only entries pertaining to that region. The CLIENT_IP filter, which
> expects a string containing an IP address, will include only entries
> pertaining to that client. The USER filter, which expects a string containing
> a user name, will include only entries pertaining to that user. Filters are
> additive, for example if both CLIENT_IP and USER filters are given, entries
> matching either or both conditions will be
[jira] [Updated] (HBASE-22978) Online slow response log
[
https://issues.apache.org/jira/browse/HBASE-22978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-22978:
---
Description:
Today when an individual RPC exceeds a configurable time bound we log a
complaint by way of the logging subsystem. These log lines look like:
{noformat}
2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
(responseTooSlow):
{"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
"starttimems":1567203007549,
"responsesize":6819737,
"method":"Scan",
"param":"region { type: REGION_NAME value:
\"tsdb,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
"processingtimems":28646,
"client":"10.253.196.215:41116",
"queuetimems":22453,
"class":"HRegionServer"}
{noformat}
Unfortunately we often truncate the request parameters, like in the above
example. We do this because the human readable representation is verbose, the
rate of too slow warnings may be high, and the combination of these things can
overwhelm the log capture system. The truncation is unfortunate because it
eliminates much of the utility of the warnings. For example, the region name,
the start and end keys, and the filter hierarchy are all important clues for
debugging performance problems caused by moderate to low selectivity queries or
queries made at a high rate.
We can maintain an in-memory ring buffer of requests that were judged to be too
slow in addition to the responseTooSlow logging. The in-memory representation
can be complete and compressed. A new admin API and shell command can provide
access to the ring buffer for online performance debugging. A modest sizing of
the ring buffer will prevent excessive memory utilization for a minor
performance debugging feature by limiting the total number of retained records.
There is some chance a high rate of requests will cause information on other
interesting requests to be overwritten before it can be read. This is the
nature of a ring buffer and an acceptable trade off.
The write request types do not require us to retain all information submitted
in the request. We don't need to retain all key-values in the mutation, which
may be too large to comfortably retain. We only need a unique set of row keys,
or even a min/max range, and total counts.
The consumers of this information will be debugging tools. We can afford to
apply fast compression to ring buffer entries (if codec support is available),
something like snappy or zstandard, and decompress on the fly when servicing
the retrieval API request. This will minimize the impact of retaining more
information about slow requests than we do today.
This proposal is for retention of request information only, the same
information provided by responseTooSlow warnings. Total size of response
serialization, possibly also total cell or row counts, should be sufficient to
characterize the response.
Optionally persist new entries added to the ring buffer into one or more files
in HDFS in a write-behind manner. If the HDFS writer blocks or falls behind and
we are unable to persist an entry before it is overwritten, that is fine.
Response too slow logging is best effort. If we can detect this make a note of
it in the log file.
—
New shell commands:
{{get_slow_responses [ ... , ] [ , \{ }
]}}
Retrieve, decode, and pretty print the contents of the too slow response ring
buffer maintained by the given list of servers; or all servers in the cluster
if no list is provided. Optionally provide a map of parameters for filtering as
additional argument. The TABLE filter, which expects a string containing a
table name, will include only entries pertaining to that table. The REGION
filter, which expects a string containing an encoded region name, will include
only entries pertaining to that region. The CLIENT_IP filter, which expects a
string containing an IP address, will include only entries pertaining to that
client. The USER filter, which expects a string containing a user name, will
include only entries pertaining to that user. Filters are additive, for example
if both CLIENT_IP and USER filters are given, entries matching either or both
conditions will be included. The exception to this is if both TABLE and REGION
filters are provided, REGION will be preferred and TABLE will be ignored. A
server name is its host, port, and start code, e.g.
"host187.example.com,60020,1289493121758".
{{clear_slow_responses [ ... , ]}}
Clear the too slow response ring buffer maintained by the given list of
servers; or all servers on the cluster if no argument is provided. A server
name is its host, port, and start code, e.g.
"host187.example.com,60020,1289493121758".
—
New Admin APIs:
{code:java}
List Admin#getSlowResponses(@Nullable List servers);
{code}
{code:java}
List
[jira] [Updated] (HBASE-22978) Online slow response log
[
https://issues.apache.org/jira/browse/HBASE-22978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-22978:
---
Description:
Today when an individual RPC exceeds a configurable time bound we log a
complaint by way of the logging subsystem. These log lines look like:
{noformat}
2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
(responseTooSlow):
{"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
"starttimems":1567203007549,
"responsesize":6819737,
"method":"Scan",
"param":"region { type: REGION_NAME value:
\"tsdb,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
"processingtimems":28646,
"client":"10.253.196.215:41116",
"queuetimems":22453,
"class":"HRegionServer"}
{noformat}
Unfortunately we often truncate the request parameters, like in the above
example. We do this because the human readable representation is verbose, the
rate of too slow warnings may be high, and the combination of these things can
overwhelm the log capture system. The truncation is unfortunate because it
eliminates much of the utility of the warnings. For example, the region name,
the start and end keys, and the filter hierarchy are all important clues for
debugging performance problems caused by moderate to low selectivity queries or
queries made at a high rate.
We can maintain an in-memory ring buffer of requests that were judged to be too
slow in addition to the responseTooSlow logging. The in-memory representation
can be complete and compressed. A new admin API and shell command can provide
access to the ring buffer for online performance debugging. A modest sizing of
the ring buffer will prevent excessive memory utilization for a minor
performance debugging feature by limiting the total number of retained records.
There is some chance a high rate of requests will cause information on other
interesting requests to be overwritten before it can be read. This is the
nature of a ring buffer and an acceptable trade off.
The write request types do not require us to retain all information submitted
in the request. We don't need to retain all key-values in the mutation, which
may be too large to comfortably retain. We only need a unique set of row keys,
or even a min/max range, and total counts.
The consumers of this information will be debugging tools. We can afford to
apply fast compression to ring buffer entries (if codec support is available),
something like snappy or zstandard, and decompress on the fly when servicing
the retrieval API request. This will minimize the impact of retaining more
information about slow requests than we do today.
This proposal is for retention of request information only, the same
information provided by responseTooSlow warnings. Total size of response
serialization, possibly also total cell or row counts, should be sufficient to
characterize the response.
Optionally persist new entries added to the ring buffer into one or more files
in HDFS in a write-behind manner. If the HDFS writer blocks or falls behind and
we are unable to persist an entry before it is overwritten, that is fine.
Response too slow logging is best effort. If we can detect this make a note of
it in the log file.
—
New shell commands:
{{get_slow_responses [ ... , ] [ , \{ }
]}}
Retrieve, decode, and pretty print the contents of the too slow response ring
buffer maintained by the given list of servers; or all servers in the cluster
if no list is provided. Optionally provide a map of parameters for filtering as
additional argument. The TABLE filter, which expects a string containing a
table name, will include only entries pertaining to that table. The REGION
filter, which expects a string containing an encoded region name, will include
only entries pertaining to that region. The CLIENT_IP filter, which expects a
string containing an IP address, will include only entries pertaining to that
client. The USER filter, which expects a string containing a user name, will
include only entries pertaining to that user. If both TABLE and REGION filters
are provided, REGION will be used. A server name is its host, port, and start
code, e.g. "host187.example.com,60020,1289493121758".
{{clear_slow_responses [ ... , ]}}
Clear the too slow response ring buffer maintained by the given list of
servers; or all servers on the cluster if no argument is provided. A server
name is its host, port, and start code, e.g.
"host187.example.com,60020,1289493121758".
—
New Admin APIs:
{code:java}
List Admin#getSlowResponses(@Nullable List servers);
{code}
{code:java}
List Admin#clearSlowResponses(@Nullable List servers);
{code}
was:
Today when an individual RPC exceeds a configurable time bound we log a
complaint by way of the logging subsystem. These log lines
[jira] [Updated] (HBASE-22978) Online slow response log
[
https://issues.apache.org/jira/browse/HBASE-22978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-22978:
---
Description:
Today when an individual RPC exceeds a configurable time bound we log a
complaint by way of the logging subsystem. These log lines look like:
{noformat}
2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
(responseTooSlow):
{"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
"starttimems":1567203007549,
"responsesize":6819737,
"method":"Scan",
"param":"region { type: REGION_NAME value:
\"tsdb,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
"processingtimems":28646,
"client":"10.253.196.215:41116",
"queuetimems":22453,
"class":"HRegionServer"}
{noformat}
Unfortunately we often truncate the request parameters, like in the above
example. We do this because the human readable representation is verbose, the
rate of too slow warnings may be high, and the combination of these things can
overwhelm the log capture system. The truncation is unfortunate because it
eliminates much of the utility of the warnings. For example, the region name,
the start and end keys, and the filter hierarchy are all important clues for
debugging performance problems caused by moderate to low selectivity queries or
queries made at a high rate.
We can maintain an in-memory ring buffer of requests that were judged to be too
slow in addition to the responseTooSlow logging. The in-memory representation
can be complete and compressed. A new admin API and shell command can provide
access to the ring buffer for online performance debugging. A modest sizing of
the ring buffer will prevent excessive memory utilization for a minor
performance debugging feature by limiting the total number of retained records.
There is some chance a high rate of requests will cause information on other
interesting requests to be overwritten before it can be read. This is the
nature of a ring buffer and an acceptable trade off.
The write request types do not require us to retain all information submitted
in the request. We don't need to retain all key-values in the mutation, which
may be too large to comfortably retain. We only need a unique set of row keys,
or even a min/max range, and total counts.
The consumers of this information will be debugging tools. We can afford to
apply fast compression to ring buffer entries (if codec support is available),
something like snappy or zstandard, and decompress on the fly when servicing
the retrieval API request. This will minimize the impact of retaining more
information about slow requests than we do today.
This proposal is for retention of request information only, the same
information provided by responseTooSlow warnings. Total size of response
serialization, possibly also total cell or row counts, should be sufficient to
characterize the response.
Optionally persist new entries added to the ring buffer into one or more files
in HDFS in a write-behind manner. If the HDFS writer blocks or falls behind and
we are unable to persist an entry before it is overwritten, that is fine.
Response too slow logging is best effort. If we can detect this make a note of
it in the log file.
—
New shell commands:
{{get_slow_responses [ ... , ] [ , \{ }
]}}
Retrieve, decode, and pretty print the contents of the too slow response ring
buffer maintained by the given list of servers; or all servers in the cluster
if no list is provided. Optionally provide a map of parameters for filtering as
additional argument. The TABLE filter, which expects a string containing a
table name, will include only entries pertaining to that table. The REGION
filter, which expects a string containing an encoded region name, will include
only entries pertaining to that region. The CLIENT_IP filter, which expects a
string containing an IP address, will include only entries pertaining to that
client. The USER filter, which expects a string containing a user name, will
include only entries pertaining to that user. A server name is its host, port,
and start code, e.g. "host187.example.com,60020,1289493121758".
{{clear_slow_responses [ ... , ]}}
Clear the too slow response ring buffer maintained by the given list of
servers; or all servers on the cluster if no argument is provided. A server
name is its host, port, and start code, e.g.
"host187.example.com,60020,1289493121758".
—
New Admin APIs:
{code:java}
List Admin#getSlowResponses(String tableOrRegion, @Nullable
List servers);
{code}
{code:java}
List Admin#getSlowResponses(@Nullable List servers);
{code}
{code:java}
List Admin#clearSlowResponses(@Nullable List servers);
{code}
was:
Today when an individual RPC exceeds a configurable time bound we log a
complaint by way of the logging
[jira] [Updated] (HBASE-22978) Online slow response log
[
https://issues.apache.org/jira/browse/HBASE-22978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-22978:
---
Description:
Today when an individual RPC exceeds a configurable time bound we log a
complaint by way of the logging subsystem. These log lines look like:
{noformat}
2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
(responseTooSlow):
{"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
"starttimems":1567203007549,
"responsesize":6819737,
"method":"Scan",
"param":"region { type: REGION_NAME value:
\"tsdb,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
"processingtimems":28646,
"client":"10.253.196.215:41116",
"queuetimems":22453,
"class":"HRegionServer"}
{noformat}
Unfortunately we often truncate the request parameters, like in the above
example. We do this because the human readable representation is verbose, the
rate of too slow warnings may be high, and the combination of these things can
overwhelm the log capture system. The truncation is unfortunate because it
eliminates much of the utility of the warnings. For example, the region name,
the start and end keys, and the filter hierarchy are all important clues for
debugging performance problems caused by moderate to low selectivity queries or
queries made at a high rate.
We can maintain an in-memory ring buffer of requests that were judged to be too
slow in addition to the responseTooSlow logging. The in-memory representation
can be complete and compressed. A new admin API and shell command can provide
access to the ring buffer for online performance debugging. A modest sizing of
the ring buffer will prevent excessive memory utilization for a minor
performance debugging feature by limiting the total number of retained records.
There is some chance a high rate of requests will cause information on other
interesting requests to be overwritten before it can be read. This is the
nature of a ring buffer and an acceptable trade off.
The write request types do not require us to retain all information submitted
in the request. We don't need to retain all key-values in the mutation, which
may be too large to comfortably retain. We only need a unique set of row keys,
or even a min/max range, and total counts.
The consumers of this information will be debugging tools. We can afford to
apply fast compression to ring buffer entries (if codec support is available),
something like snappy or zstandard, and decompress on the fly when servicing
the retrieval API request. This will minimize the impact of retaining more
information about slow requests than we do today.
This proposal is for retention of request information only, the same
information provided by responseTooSlow warnings. Total size of response
serialization, possibly also total cell or row counts, should be sufficient to
characterize the response.
—
New shell commands:
{{get_slow_responses [ ... , ] [ , \{ }
]}}
Retrieve, decode, and pretty print the contents of the too slow response ring
buffer maintained by the given list of servers; or all servers in the cluster
if no list is provided. Optionally provide a map of parameters for filtering as
additional argument. The TABLE filter, which expects a string containing a
table name, will include only entries pertaining to that table. The REGION
filter, which expects a string containing an encoded region name, will include
only entries pertaining to that region. The CLIENT_IP filter, which expects a
string containing an IP address, will include only entries pertaining to that
client. The USER filter, which expects a string containing a user name, will
include only entries pertaining to that user. A server name is its host, port,
and start code, e.g. "host187.example.com,60020,1289493121758".
{{clear_slow_responses [ ... , ]}}
Clear the too slow response ring buffer maintained by the given list of
servers; or all servers on the cluster if no argument is provided. A server
name is its host, port, and start code, e.g.
"host187.example.com,60020,1289493121758".
—
New Admin APIs:
{code:java}
List Admin#getSlowResponses(String tableOrRegion, @Nullable
List servers);
{code}
{code:java}
List Admin#getSlowResponses(@Nullable List servers);
{code}
{code:java}
List Admin#clearSlowResponses(@Nullable List servers);
{code}
was:
Today when an individual RPC exceeds a configurable time bound we log a
complaint by way of the logging subsystem. These log lines look like:
{noformat}
2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
(responseTooSlow):
{"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
"starttimems":1567203007549,
"responsesize":6819737,
"method":"Scan",
"param":"region { type: REGION_NAME value:
[jira] [Commented] (HBASE-22978) Online slow response log
[
https://issues.apache.org/jira/browse/HBASE-22978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923760#comment-16923760
]
Andrew Purtell commented on HBASE-22978:
I'll update the description to include write-behind of the ring buffer to a
directory in HDFS, but it shouldn't block, so if we stall during writing some
ring buffer entries may have been lost. If we can detect that we can log that
it happened in the file.
bq. Also saving the user and/or client IP of the request and being able to ask
for requests by those would be extra nice
Ok, will include user and client IP in the request details set aside.
As for query APIs, the admin API is sugar over fan out requests to
regionservers for whatever is currently sitting in the ring buffers. Where we
want to narrow the search by region or table we can get region locations and
prune the regionserver set. Filtering or sorting on other attributes would be
done locally in the client. I think it best to let the client index the list of
ResponseDetail however it likes.
The shell commands are one client of the admin APIs. This seems a good place to
put additional convenience filtering. Will update the description for this too.
> Online slow response log
>
>
> Key: HBASE-22978
> URL: https://issues.apache.org/jira/browse/HBASE-22978
> Project: HBase
> Issue Type: New Feature
> Components: Admin, regionserver, shell
>Reporter: Andrew Purtell
>Priority: Minor
>
> Today when an individual RPC exceeds a configurable time bound we log a
> complaint by way of the logging subsystem. These log lines look like:
> {noformat}
> 2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
> (responseTooSlow):
> {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
> "starttimems":1567203007549,
> "responsesize":6819737,
> "method":"Scan",
> "param":"region { type: REGION_NAME value:
> \"tsdb,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
> "processingtimems":28646,
> "client":"10.253.196.215:41116",
> "queuetimems":22453,
> "class":"HRegionServer"}
> {noformat}
> Unfortunately we often truncate the request parameters, like in the above
> example. We do this because the human readable representation is verbose, the
> rate of too slow warnings may be high, and the combination of these things
> can overwhelm the log capture system. The truncation is unfortunate because
> it eliminates much of the utility of the warnings. For example, the region
> name, the start and end keys, and the filter hierarchy are all important
> clues for debugging performance problems caused by moderate to low
> selectivity queries or queries made at a high rate.
> We can maintain an in-memory ring buffer of requests that were judged to be
> too slow in addition to the responseTooSlow logging. The in-memory
> representation can be complete and compressed. A new admin API and shell
> command can provide access to the ring buffer for online performance
> debugging. A modest sizing of the ring buffer will prevent excessive memory
> utilization for a minor performance debugging feature by limiting the total
> number of retained records. There is some chance a high rate of requests will
> cause information on other interesting requests to be overwritten before it
> can be read. This is the nature of a ring buffer and an acceptable trade off.
> The write request types do not require us to retain all information submitted
> in the request. We don't need to retain all key-values in the mutation, which
> may be too large to comfortably retain. We only need a unique set of row
> keys, or even a min/max range, and total counts.
> The consumers of this information will be debugging tools. We can afford to
> apply fast compression to ring buffer entries (if codec support is
> available), something like snappy or zstandard, and decompress on the fly
> when servicing the retrieval API request. This will minimize the impact of
> retaining more information about slow requests than we do today.
> This proposal is for retention of request information only, the same
> information provided by responseTooSlow warnings. Total size of response
> serialization, possibly also total cell or row counts, should be sufficient
> to characterize the response.
> —
> New shell commands:
> {{get_slow_responses [ , \{ SERVERS=> } ]}}
> Retrieve, decode, and pretty print the contents of the too slow response ring
> buffer. Provide a table name as first argument to find all regions and
> retrieve too slow response entries for the given table from all servers
> currently hosting it. Provide a region name as first argument to retrieve all
> too slow response entries for the
[GitHub] [hbase] Apache-HBase commented on issue #580: [HBASE-22874] Canary should not be IA.Public
Apache-HBase commented on issue #580: [HBASE-22874] Canary should not be IA.Public URL: https://github.com/apache/hbase/pull/580#issuecomment-528600152 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 0m 36s | Docker mode activated. | ||| _ Prechecks _ | | :green_heart: | dupname | 0m 1s | No case conflicting files found. | | :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ master Compile Tests _ | | :green_heart: | mvninstall | 5m 24s | master passed | | :green_heart: | compile | 0m 55s | master passed | | :green_heart: | checkstyle | 1m 18s | master passed | | :green_heart: | shadedjars | 4m 31s | branch has no errors when building our shaded downstream artifacts. | | :green_heart: | javadoc | 0m 37s | master passed | | :blue_heart: | spotbugs | 4m 0s | Used deprecated FindBugs config; considering switching to SpotBugs. | | :green_heart: | findbugs | 3m 58s | master passed | ||| _ Patch Compile Tests _ | | :green_heart: | mvninstall | 4m 49s | the patch passed | | :green_heart: | compile | 0m 54s | the patch passed | | :green_heart: | javac | 0m 54s | the patch passed | | :broken_heart: | checkstyle | 1m 19s | hbase-server: The patch generated 1 new + 21 unchanged - 2 fixed = 22 total (was 23) | | :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | :green_heart: | shadedjars | 4m 34s | patch has no errors when building our shaded downstream artifacts. | | :green_heart: | hadoopcheck | 15m 34s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | :green_heart: | javadoc | 0m 35s | the patch passed | | :green_heart: | findbugs | 4m 16s | the patch passed | ||| _ Other Tests _ | | :broken_heart: | unit | 158m 29s | hbase-server in the patch failed. | | :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. | | | | 215m 1s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-580/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/580 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 8bfe6c5d2891 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-580/out/precommit/personality/provided.sh | | git revision | master / 0d338a1820 | | Default Java | 1.8.0_181 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-580/2/artifact/out/diff-checkstyle-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-580/2/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-580/2/testReport/ | | Max. process+thread count | 4916 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-580/2/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-22859) [HBCK2] Fix the orphan regions on filesystem
[ https://issues.apache.org/jira/browse/HBASE-22859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-22859: -- Attachment: HBASE-22859.master.002.patch > [HBCK2] Fix the orphan regions on filesystem > > > Key: HBASE-22859 > URL: https://issues.apache.org/jira/browse/HBASE-22859 > Project: HBase > Issue Type: Sub-task > Components: documentation, hbck2 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Attachments: HBASE-22859.master.001.patch, > HBASE-22859.master.002.patch > > > Plan to add this feature to HBCK2 tool firstly. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HBASE-22859) [HBCK2] Fix the orphan regions on filesystem
[ https://issues.apache.org/jira/browse/HBASE-22859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-22859: -- Fix Version/s: 2.2.2 2.1.7 2.3.0 3.0.0 Status: Patch Available (was: Open) .002 Add section to the bulk load complete tool on how it can be used 'adopting' stray 'orphan' data turned up by hbck2 or the new reporting facility in the Master UI. Did a cleanup of BulkLoadHFileTool mostly around usage pointing back to this new documentation. > [HBCK2] Fix the orphan regions on filesystem > > > Key: HBASE-22859 > URL: https://issues.apache.org/jira/browse/HBASE-22859 > Project: HBase > Issue Type: Sub-task > Components: documentation, hbck2 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2 > > Attachments: HBASE-22859.master.001.patch, > HBASE-22859.master.002.patch > > > Plan to add this feature to HBCK2 tool firstly. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HBASE-22859) [HBCK2] Fix the orphan regions on filesystem
[ https://issues.apache.org/jira/browse/HBASE-22859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-22859: -- Component/s: hbck2 documentation > [HBCK2] Fix the orphan regions on filesystem > > > Key: HBASE-22859 > URL: https://issues.apache.org/jira/browse/HBASE-22859 > Project: HBase > Issue Type: Sub-task > Components: documentation, hbck2 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Attachments: HBASE-22859.master.001.patch > > > Plan to add this feature to HBCK2 tool firstly. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HBASE-22859) [HBCK2] Fix the orphan regions on filesystem
[ https://issues.apache.org/jira/browse/HBASE-22859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-22859: -- Attachment: HBASE-22859.master.001.patch > [HBCK2] Fix the orphan regions on filesystem > > > Key: HBASE-22859 > URL: https://issues.apache.org/jira/browse/HBASE-22859 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Attachments: HBASE-22859.master.001.patch > > > Plan to add this feature to HBCK2 tool firstly. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HBASE-22978) Online slow response log
[
https://issues.apache.org/jira/browse/HBASE-22978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923737#comment-16923737
]
Sean Busbey commented on HBASE-22978:
-
Can we include the ability to have entries from the buffer(s) dumped to a
FileSystem directory? I'd like to have something flexible available for when I
need to use this with an outlier case.
Also saving the user and/or client IP of the request and being able to ask for
requests by those would be extra nice
> Online slow response log
>
>
> Key: HBASE-22978
> URL: https://issues.apache.org/jira/browse/HBASE-22978
> Project: HBase
> Issue Type: New Feature
> Components: Admin, regionserver, shell
>Reporter: Andrew Purtell
>Priority: Minor
>
> Today when an individual RPC exceeds a configurable time bound we log a
> complaint by way of the logging subsystem. These log lines look like:
> {noformat}
> 2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
> (responseTooSlow):
> {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
> "starttimems":1567203007549,
> "responsesize":6819737,
> "method":"Scan",
> "param":"region { type: REGION_NAME value:
> \"tsdb,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
> "processingtimems":28646,
> "client":"10.253.196.215:41116",
> "queuetimems":22453,
> "class":"HRegionServer"}
> {noformat}
> Unfortunately we often truncate the request parameters, like in the above
> example. We do this because the human readable representation is verbose, the
> rate of too slow warnings may be high, and the combination of these things
> can overwhelm the log capture system. The truncation is unfortunate because
> it eliminates much of the utility of the warnings. For example, the region
> name, the start and end keys, and the filter hierarchy are all important
> clues for debugging performance problems caused by moderate to low
> selectivity queries or queries made at a high rate.
> We can maintain an in-memory ring buffer of requests that were judged to be
> too slow in addition to the responseTooSlow logging. The in-memory
> representation can be complete and compressed. A new admin API and shell
> command can provide access to the ring buffer for online performance
> debugging. A modest sizing of the ring buffer will prevent excessive memory
> utilization for a minor performance debugging feature by limiting the total
> number of retained records. There is some chance a high rate of requests will
> cause information on other interesting requests to be overwritten before it
> can be read. This is the nature of a ring buffer and an acceptable trade off.
> The write request types do not require us to retain all information submitted
> in the request. We don't need to retain all key-values in the mutation, which
> may be too large to comfortably retain. We only need a unique set of row
> keys, or even a min/max range, and total counts.
> The consumers of this information will be debugging tools. We can afford to
> apply fast compression to ring buffer entries (if codec support is
> available), something like snappy or zstandard, and decompress on the fly
> when servicing the retrieval API request. This will minimize the impact of
> retaining more information about slow requests than we do today.
> This proposal is for retention of request information only, the same
> information provided by responseTooSlow warnings. Total size of response
> serialization, possibly also total cell or row counts, should be sufficient
> to characterize the response.
> —
> New shell commands:
> {{get_slow_responses [ , \{ SERVERS=> } ]}}
> Retrieve, decode, and pretty print the contents of the too slow response ring
> buffer. Provide a table name as first argument to find all regions and
> retrieve too slow response entries for the given table from all servers
> currently hosting it. Provide a region name as first argument to retrieve all
> too slow response entries for the given region. Optionally provide a map of
> parameters as second argument. The SERVERS parameter, which expects a list of
> server names, will constrain the search to the given set of servers. A server
> name is its host, port, and start code, e.g.
> "host187.example.com,60020,1289493121758".
> {{get_slow_responses [ ... , ]}}
> Retrieve, decode, and pretty print the contents of the too slow response ring
> buffer maintained by the given list of servers; or all servers on the cluster
> if no argument is provided. A server name is its host, port, and start code,
> e.g. "host187.example.com,60020,1289493121758".
> {{clear_slow_responses [ ... , ]}}
> Clear the too slow response ring buffer maintained by the given
[jira] [Commented] (HBASE-22937) The RawBytesComparator in branch-1 have wrong comparison order
[
https://issues.apache.org/jira/browse/HBASE-22937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923734#comment-16923734
]
Hudson commented on HBASE-22937:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #634 (See
[https://builds.apache.org/job/HBase-1.3-IT/634/])
HBASE-22937 The RawBytesComparator in branch-1 have wrong comparison (apurtell:
rev 6dd891447c8456849c2389146e9ba9050f5a59b0)
* (edit) hbase-common/src/test/java/org/apache/hadoop/hbase/TestKeyValue.java
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java
> The RawBytesComparator in branch-1 have wrong comparison order
> --
>
> Key: HBASE-22937
> URL: https://issues.apache.org/jira/browse/HBASE-22937
> Project: HBase
> Issue Type: Bug
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 1.5.0, 1.3.6, 1.4.11
>
>
> When digging the HBASE-22862, we found a bug in
> RawBytesComparator#compareOnlyKeyPortion (although it's unrelated to the
> corruption in HBASE-22862).
> {code}
> @Override
> @VisibleForTesting
> public int compareOnlyKeyPortion(Cell left, Cell right) {
> // ...
> return (0xff & left.getTypeByte()) - (0xff & right.getTypeByte());
> }
> {code}
> Here should be (0xff & right.getTypeByte()) - (0xff & left.getTypeByte()) I
> think.
> I can see the BloomFilter or HFile v2 are still using the comparator in
> branch-1 (but not in branch-2). Maybe we can just remove the class (if some
> HFile encoded with this comparator, then mapping to the correct KVComparator
> just like 2.x), or fix the bug in current RawBytesComparator.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HBASE-22937) The RawBytesComparator in branch-1 have wrong comparison order
[
https://issues.apache.org/jira/browse/HBASE-22937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-22937:
---
Fix Version/s: 1.3.6
> The RawBytesComparator in branch-1 have wrong comparison order
> --
>
> Key: HBASE-22937
> URL: https://issues.apache.org/jira/browse/HBASE-22937
> Project: HBase
> Issue Type: Bug
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 1.5.0, 1.3.6, 1.4.11
>
>
> When digging the HBASE-22862, we found a bug in
> RawBytesComparator#compareOnlyKeyPortion (although it's unrelated to the
> corruption in HBASE-22862).
> {code}
> @Override
> @VisibleForTesting
> public int compareOnlyKeyPortion(Cell left, Cell right) {
> // ...
> return (0xff & left.getTypeByte()) - (0xff & right.getTypeByte());
> }
> {code}
> Here should be (0xff & right.getTypeByte()) - (0xff & left.getTypeByte()) I
> think.
> I can see the BloomFilter or HFile v2 are still using the comparator in
> branch-1 (but not in branch-2). Maybe we can just remove the class (if some
> HFile encoded with this comparator, then mapping to the correct KVComparator
> just like 2.x), or fix the bug in current RawBytesComparator.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HBASE-22978) Online slow response log
[
https://issues.apache.org/jira/browse/HBASE-22978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-22978:
---
Description:
Today when an individual RPC exceeds a configurable time bound we log a
complaint by way of the logging subsystem. These log lines look like:
{noformat}
2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
(responseTooSlow):
{"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
"starttimems":1567203007549,
"responsesize":6819737,
"method":"Scan",
"param":"region { type: REGION_NAME value:
\"tsdb,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
"processingtimems":28646,
"client":"10.253.196.215:41116",
"queuetimems":22453,
"class":"HRegionServer"}
{noformat}
Unfortunately we often truncate the request parameters, like in the above
example. We do this because the human readable representation is verbose, the
rate of too slow warnings may be high, and the combination of these things can
overwhelm the log capture system. The truncation is unfortunate because it
eliminates much of the utility of the warnings. For example, the region name,
the start and end keys, and the filter hierarchy are all important clues for
debugging performance problems caused by moderate to low selectivity queries or
queries made at a high rate.
We can maintain an in-memory ring buffer of requests that were judged to be too
slow in addition to the responseTooSlow logging. The in-memory representation
can be complete and compressed. A new admin API and shell command can provide
access to the ring buffer for online performance debugging. A modest sizing of
the ring buffer will prevent excessive memory utilization for a minor
performance debugging feature by limiting the total number of retained records.
There is some chance a high rate of requests will cause information on other
interesting requests to be overwritten before it can be read. This is the
nature of a ring buffer and an acceptable trade off.
The write request types do not require us to retain all information submitted
in the request. We don't need to retain all key-values in the mutation, which
may be too large to comfortably retain. We only need a unique set of row keys,
or even a min/max range, and total counts.
The consumers of this information will be debugging tools. We can afford to
apply fast compression to ring buffer entries (if codec support is available),
something like snappy or zstandard, and decompress on the fly when servicing
the retrieval API request. This will minimize the impact of retaining more
information about slow requests than we do today.
This proposal is for retention of request information only, the same
information provided by responseTooSlow warnings. Total size of response
serialization, possibly also total cell or row counts, should be sufficient to
characterize the response.
—
New shell commands:
{{get_slow_responses [ , \{ SERVERS=> } ]}}
Retrieve, decode, and pretty print the contents of the too slow response ring
buffer. Provide a table name as first argument to find all regions and retrieve
too slow response entries for the given table from all servers currently
hosting it. Provide a region name as first argument to retrieve all too slow
response entries for the given region. Optionally provide a map of parameters
as second argument. The SERVERS parameter, which expects a list of server
names, will constrain the search to the given set of servers. A server name is
its host, port, and start code, e.g. "host187.example.com,60020,1289493121758".
{{get_slow_responses [ ... , ]}}
Retrieve, decode, and pretty print the contents of the too slow response ring
buffer maintained by the given list of servers; or all servers on the cluster
if no argument is provided. A server name is its host, port, and start code,
e.g. "host187.example.com,60020,1289493121758".
{{clear_slow_responses [ ... , ]}}
Clear the too slow response ring buffer maintained by the given list of
servers; or all servers on the cluster if no argument is provided. A server
name is its host, port, and start code, e.g.
"host187.example.com,60020,1289493121758".
—
New Admin APIs:
{code:java}
List Admin#getSlowResponses(String tableOrRegion, @Nullable
List servers);
{code}
{code:java}
List Admin#getSlowResponses(@Nullable List servers);
{code}
{code:java}
List Admin#clearSlowResponses(@Nullable List servers);
{code}
was:
Today when an individual RPC exceeds a configurable time bound we log a
complaint by way of the logging subsystem. These log lines look like:
{noformat}
2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
(responseTooSlow):
{"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
[jira] [Updated] (HBASE-22978) Online slow response log
[
https://issues.apache.org/jira/browse/HBASE-22978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-22978:
---
Description:
Today when an individual RPC exceeds a configurable time bound we log a
complaint by way of the logging subsystem. These log lines look like:
{noformat}
2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
(responseTooSlow):
{"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
"starttimems":1567203007549,
"responsesize":6819737,
"method":"Scan","param":"region { type: REGION_NAME value:
\"tsdb,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
"processingtimems":28646,
"client":"10.253.196.215:41116",
"queuetimems":22453,
"class":"HRegionServer"}
{noformat}
Unfortunately we often truncate the request parameters, like in the above
example. We do this because the human readable representation is verbose, the
rate of too slow warnings may be high, and the combination of these things can
overwhelm the log capture system. (All of these things have been reported from
various production settings. Truncation was added after we crashed a user's log
capture system.) The truncation is unfortunate because it eliminates much of
the utility of the warnings. For example, the region name, the start and end
keys, and the filter hierarchy are all important clues for debugging
performance problems caused by moderate to low selectivity queries or queries
made at a high rate.
We can maintain an in-memory ring buffer of requests that were judged to be too
slow in addition to the responseTooSlow logging. The in-memory representation
can be complete and compressed. A new admin API and shell command can provide
access to the ring buffer for online performance debugging. A modest sizing of
the ring buffer will prevent excessive memory utilization for a minor
performance debugging feature by limiting the total number of retained records.
There is some chance a high rate of requests will cause information on other
interesting requests to be overwritten before it can be read. This is the
nature of a ring buffer and an acceptable trade off.
The write request types do not require us to retain all information submitted
in the request. We don't need to retain all key-values in the mutation, which
may be too large to comfortably retain. We only need a unique set of row keys,
or even a min/max range, and total counts.
The consumers of this information will be debugging tools. We can afford to
apply fast compression to ring buffer entries (if codec support is available),
something like snappy or zstandard, and decompress on the fly when servicing
the retrieval API request. This will minimize the impact of retaining more
information about slow requests than we do today.
This proposal is for retention of request information only, the same
information provided by responseTooSlow warnings. Total size of response
serialization, possibly also total cell or row counts, should be sufficient to
characterize the response.
—
New shell commands:
{{get_slow_responses [ , \{ SERVERS=> } ]}}
Retrieve, decode, and pretty print the contents of the too slow response ring
buffer. Provide a table name as first argument to find all regions and retrieve
too slow response entries for the given table from all servers currently
hosting it. Provide a region name as first argument to retrieve all too slow
response entries for the given region. Optionally provide a map of parameters
as second argument. The SERVERS parameter, which expects a list of server
names, will constrain the search to the given set of servers. A server name is
its host, port, and start code, e.g. "host187.example.com,60020,1289493121758".
{{get_slow_responses [ ... , ]}}
Retrieve, decode, and pretty print the contents of the too slow response ring
buffer maintained by the given list of servers; or all servers on the cluster
if no argument is provided. A server name is its host, port, and start code,
e.g. "host187.example.com,60020,1289493121758".
{{clear_slow_responses [ ... , ]}}
Clear the too slow response ring buffer maintained by the given list of
servers; or all servers on the cluster if no argument is provided. A server
name is its host, port, and start code, e.g.
"host187.example.com,60020,1289493121758".
—
New Admin APIs:
{code:java}
List Admin#getSlowResponses(String tableOrRegion, @Nullable
List servers);
{code}
{code:java}
List Admin#getSlowResponses(@Nullable List servers);
{code}
{code:java}
List Admin#clearSlowResponses(@Nullable List servers);
{code}
was:
Today when an individual RPC exceeds a configurable time bound we log a
complaint by way of the logging subsystem. These log lines look like:
{noformat}
2019-08-30 22:10:36,195 WARN [,queue=15,port=60020]
[jira] [Updated] (HBASE-22978) Online slow response log
[
https://issues.apache.org/jira/browse/HBASE-22978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-22978:
---
Description:
Today when an individual RPC exceeds a configurable time bound we log a
complaint by way of the logging subsystem. These log lines look like:
{noformat}
2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
(responseTooSlow):
{"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
"starttimems":1567203007549,
"responsesize":6819737,
"method":"Scan","param":"region { type: REGION_NAME value:
\"tsdb,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
"processingtimems":28646,
"client":"10.253.196.215:41116",
"queuetimems":22453,
"class":"HRegionServer"}
{noformat}
Unfortunately we often truncate the request parameters, like in the above
example. We do this because the human readable representation is verbose, the
rate of too slow warnings may be high, and the combination of these things can
overwhelm the log capture system. (All of these things have been reported from
various production settings. Truncation was added after we crashed a user's log
capture system.) The truncation is unfortunate because it eliminates much of
the utility of the warnings. For example, the region name, the start and end
keys, and the filter hierarchy are all important clues for debugging
performance problems caused by moderate to low selectivity queries or queries
made at a high rate.
We can maintain an in-memory ring buffer of requests that were judged to be too
slow in addition to the responseTooSlow logging. The in-memory representation
can be complete and compressed. A new admin API and shell command can provide
access to the ring buffer for online performance debugging. A modest sizing of
the ring buffer will prevent excessive memory utilization for a minor
performance debugging feature by limiting the total number of retained records.
There is some chance a high rate of requests will cause information on other
interesting requests to be overwritten before it can be read. This is the
nature of a ring buffer and an acceptable trade off.
The write request types do not require us to retain all information submitted
in the request. We don't need to retain all key-values in the mutation, which
may be too large to comfortably retain. We only need a unique set of row keys,
or even a min/max range, and total counts.
The consumers of this information will be debugging tools. We can afford to
apply fast compression to ring buffer entries (if codec support is available),
something like snappy or zstandard, and decompress on the fly when servicing
the retrieval API request. This will minimize the impact of retaining more
information about slow requests than we do today.
This proposal is for retention of request information only, the same
information provided by responseTooSlow warnings. Total size of response
serialization, possibly also total cell or row counts, should be sufficient to
characterize the response.
—
New shell commands:
{{get_slow_responses [ , \{ SERVERS=> } ]}}
Retrieve, decode, and pretty print the contents of the too slow response ring
buffer. Provide a table name as first argument to find all regions and retrieve
too slow response entries for the given table from all servers currently
hosting it. Provide a region name as first argument to retrieve all too slow
response entries for the given region. Optionally provide a map of parameters
as second argument. The SERVERS parameter, which expects a list of server
names, will constrain the search to the given set of servers. A server name is
its host, port, and start code, e.g. "host187.example.com,60020,1289493121758".
{{get_slow_responses [ ... , ]}}
Retrieve, decode, and pretty print the contents of the too slow response ring
buffer maintained by the given list of servers; or all servers on the cluster
if no argument is provided. A server name is its host, port, and start code,
e.g. "host187.example.com,60020,1289493121758".
{{clear_slow_responses [ ... , ]}}
Clear the too slow response ring buffer maintained by the given list of
servers; or all servers on the cluster if no argument is provided. A server
name is its host, port, and start code, e.g.
"host187.example.com,60020,1289493121758".
—
New Admin APIs:
{code:java}
List Admin#getSlowResponses(String tableOrRegion, @Nullable
List servers);
{code}
{code:java}
List Admin#getSlowResponses(@Nullable List servers);
{code}
{code:java}
List Admin#clearSlowResponses(@Nullable List servers);
{code}
was:
Today when an individual RPC exceeds a configurable time bound we log a
complaint by way of the logging subsystem. These log lines look like:
{noformat}
2019-08-30 22:10:36,195 WARN [,queue=15,port=60020]
[jira] [Created] (HBASE-22978) Online slow response log
Andrew Purtell created HBASE-22978:
--
Summary: Online slow response log
Key: HBASE-22978
URL: https://issues.apache.org/jira/browse/HBASE-22978
Project: HBase
Issue Type: New Feature
Components: Admin, regionserver, shell
Reporter: Andrew Purtell
Today when an individual RPC exceeds a configurable time bound we log a
complaint by way of the logging subsystem. These log lines look like:
{noformat}
2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
(responseTooSlow):
{"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)","starttimems":1567203007549,"responsesize":6819737,"method":"Scan","param":"region
{ type: REGION_NAME value:
\"tsdb,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000","processingtimems":28646,"client":"10.253.196.215:41116","queuetimems":22453,"class":"HRegionServer"}
{noformat}
Unfortunately we often truncate the request parameters, like in the above
example. We do this because the human readable representation is verbose, the
rate of too slow warnings may be high, and the combination of these things can
overwhelm the log capture system. (All of these things have been reported from
various production settings. Truncation was added after we crashed a user's log
capture system.) The truncation is unfortunate because it eliminates much of
the utility of the warnings. For example, the region name, the start and end
keys, and the filter hierarchy are all important clues for debugging
performance problems caused by moderate to low selectivity queries or queries
made at a high rate.
We can maintain an in-memory ring buffer of requests that were judged to be too
slow in addition to the responseTooSlow logging. The in-memory representation
can be complete and compressed. A new admin API and shell command can provide
access to the ring buffer for online performance debugging. A modest sizing of
the ring buffer will prevent excessive memory utilization for a minor
performance debugging feature by limiting the total number of retained records.
There is some chance a high rate of requests will cause information on other
interesting requests to be overwritten before it can be read. This is the
nature of a ring buffer and an acceptable trade off.
The write request types do not require us to retain all information submitted
in the request. We don't need to retain all key-values in the mutation, which
may be too large to comfortably retain. We only need a unique set of row keys,
or even a min/max range, and total counts.
The consumers of this information will be debugging tools. We can afford to
apply fast compression to ring buffer entries (if codec support is available),
something like snappy or zstandard, and decompress on the fly when servicing
the retrieval API request. This will minimize the impact of retaining more
information about slow requests than we do today.
This proposal is for retention of request information only, the same
information provided by responseTooSlow warnings. Total size of response
serialization, possibly also total cell or row counts, should be sufficient to
characterize the response.
—
New shell commands:
{{get_slow_responses [ , \{ SERVERS=> } ]}}
Retrieve, decode, and pretty print the contents of the too slow response ring
buffer. Provide a table name as first argument to find all regions and retrieve
too slow response entries for the given table from all servers currently
hosting it. Provide a region name as first argument to retrieve all too slow
response entries for the given region. Optionally provide a map of parameters
as second argument. The SERVERS parameter, which expects a list of server
names, will constrain the search to the given set of servers. A server name is
its host, port, and start code, e.g. "host187.example.com,60020,1289493121758".
{{get_slow_responses [ ... , ]}}
Retrieve, decode, and pretty print the contents of the too slow response ring
buffer maintained by the given list of servers; or all servers on the cluster
if no argument is provided. A server name is its host, port, and start code,
e.g. "host187.example.com,60020,1289493121758".
{{clear_slow_responses [ ... , ]}}
Clear the too slow response ring buffer maintained by the given list of
servers; or all servers on the cluster if no argument is provided. A server
name is its host, port, and start code, e.g.
"host187.example.com,60020,1289493121758".
—
New Admin APIs:
{code:java}
List Admin#getSlowResponses(String tableOrRegion, @Nullable
List servers);
{code}
{code:java}
List Admin#getSlowResponses(@Nullable List servers);
{code}
{code:java}
List Admin#clearSlowResponses(@Nullable List servers);
{code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22760) Stop/Resume Snapshot Auto-Cleanup activity with shell command
[ https://issues.apache.org/jira/browse/HBASE-22760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923719#comment-16923719 ] Andrew Purtell commented on HBASE-22760: Thanks. Will try to commit this today. If those checkstyle warns in the branch-1 report are related I'll fix on commit. > Stop/Resume Snapshot Auto-Cleanup activity with shell command > - > > Key: HBASE-22760 > URL: https://issues.apache.org/jira/browse/HBASE-22760 > Project: HBase > Issue Type: Improvement > Components: Admin, shell, snapshots >Affects Versions: 3.0.0, 1.5.0, 2.3.0 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.3.0, 1.4.11, 2.1.7, 2.2.2 > > Attachments: HBASE-22760.branch-1.000.patch, > HBASE-22760.branch-2.000.patch, HBASE-22760.master.003.patch, > HBASE-22760.master.004.patch, HBASE-22760.master.005.patch, > HBASE-22760.master.008.patch, HBASE-22760.master.009.patch > > > For any scheduled snapshot backup activity, we would like to disable > auto-cleaner for snapshot based on TTL. However, as per HBASE-22648 we have a > config to disable snapshot auto-cleaner: > hbase.master.cleaner.snapshot.disable, which would take effect only upon > HMaster restart just similar to any other hbase-site configs. > For any running cluster, we should be able to stop/resume auto-cleanup > activity for snapshot based on shell command. Something similar to below > command should be able to stop/start cleanup chore: > hbase(main):001:0> snapshot_auto_cleanup_switch false (disable > auto-cleaner) > hbase(main):001:0> snapshot_auto_cleanup_switch true (enable auto-cleaner) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [hbase] Apache-HBase commented on issue #584: HBASE-22927 Upgrade mockito version for Java 11 compatibility
Apache-HBase commented on issue #584: HBASE-22927 Upgrade mockito version for Java 11 compatibility URL: https://github.com/apache/hbase/pull/584#issuecomment-528569270 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 0m 41s | Docker mode activated. | ||| _ Prechecks _ | | :green_heart: | dupname | 0m 0s | No case conflicting files found. | | :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | :yellow_heart: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | :green_heart: | mvninstall | 5m 44s | master passed | | :green_heart: | compile | 3m 7s | master passed | | :green_heart: | shadedjars | 4m 52s | branch has no errors when building our shaded downstream artifacts. | | :green_heart: | javadoc | 2m 59s | master passed | | :yellow_heart: | patch | 8m 1s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | :green_heart: | mvninstall | 5m 4s | the patch passed | | :green_heart: | compile | 2m 55s | the patch passed | | :green_heart: | javac | 2m 55s | the patch passed | | :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | :broken_heart: | shadedjars | 0m 9s | patch has 7 errors when building our shaded downstream artifacts. | | :green_heart: | hadoopcheck | 15m 40s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | :green_heart: | javadoc | 2m 55s | the patch passed | ||| _ Other Tests _ | | :broken_heart: | unit | 240m 26s | root in the patch failed. | | :green_heart: | asflicense | 0m 49s | The patch does not generate ASF License warnings. | | | | 291m 54s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.rsgroup.TestRSGroupsAdmin2 | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-584/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/584 | | Optional Tests | dupname asflicense javac javadoc unit shadedjars hadoopcheck xml compile | | uname | Linux ff0541411fe9 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-584/out/precommit/personality/provided.sh | | git revision | master / 0d338a1820 | | Default Java | 1.8.0_181 | | shadedjars | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-584/2/artifact/out/patch-shadedjars.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-584/2/artifact/out/patch-unit-root.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-584/2/testReport/ | | Max. process+thread count | 5636 (vs. ulimit of 1) | | modules | C: . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-584/2/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] joshelser commented on a change in pull request #571: HBASE-20821 Re-creating a dropped namespace and contained table inherits previously set space quota settings
joshelser commented on a change in pull request #571: HBASE-20821 Re-creating a
dropped namespace and contained table inherits previously set space quota
settings
URL: https://github.com/apache/hbase/pull/571#discussion_r321426926
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/MasterQuotasObserver.java
##
@@ -78,6 +79,20 @@ public void postDeleteTable(
admin.setQuota(settings);
}
}
+} else if (quotasAtNamespace != null) {
+ // If quota present at namespace level remove the table entry from
'hbase:quota'
Review comment:
> Because even though table is dropped it's entry is still present in the
'hbase:quota' table because of the namespace quota. If we create another table
with same name inside that namespace it inherits the earlier quota settings.
That's by design, not a bug. If you set a quota on a namespace (as opposed
to a quota on the table), you would automatically get that quota for all tables
in that namespace.
This allows an organization to be given free-rule over some namespace while
still enforcing an upper-bound on the allowed resources in HBase. e.g. a
namespace space quota of 200G would allow 200G of data in any number of tables
in that namespace, not mattering which table uses that quota.
Now, if you drop a namespace and the quota still remains, that's a bug, but
I don't think that's what you're trying to solve.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (HBASE-21745) Make HBCK2 be able to fix issues other than region assignment
[ https://issues.apache.org/jira/browse/HBASE-21745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923691#comment-16923691 ] stack commented on HBASE-21745: --- Let what is in this issue make up HBCK2 1.0.0. I made HBASE-22977 for HBCK2 2.0.0 brainstorming. > Make HBCK2 be able to fix issues other than region assignment > - > > Key: HBASE-21745 > URL: https://issues.apache.org/jira/browse/HBASE-21745 > Project: HBase > Issue Type: Umbrella > Components: hbase-operator-tools, hbck2 >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > > This is what [~apurtell] posted on mailing-list, HBCK2 should support > * -Rebuild meta from region metadata in the filesystem, aka offline meta > rebuild.- > * -Fix assignment errors (undeployed regions, double assignments (yes, > should not be possible), etc)- (See > https://issues.apache.org/jira/browse/HBASE-21745?focusedCommentId=16888302=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16888302) > * Fix region holes, overlaps, and other errors in the region chain > * Fix failed split and merge transactions that have failed to roll back due > to some bug (related to previous) > * -Enumerate store files to determine file level corruption and sideline > corrupt files- > * -Fix hfile link problems (dangling / broken)- -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HBASE-22977) [HBCK2] hbck2 2.0
[ https://issues.apache.org/jira/browse/HBASE-22977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-22977: -- Description: Brainstorming issue for what to ship in an HBCK2 version 2.0 (HBASE-21745 describes HBCK2 1.0). Possibilities: * Integrate HBCK2 operations into core either by automating fix or by adding shell commands for fixup. * An 'adoption service' running in Master where you pass 'orphan data' -- a file or a directory -- and the running service figures what to do w/ the content (remove if empty, figure where the file should slot if an hfile, replay recovered edits if a recovered_edits dir, play WALs if wal dir, etc.). Could be smarter/interactive, a 'three-way-merge' as Sean and Josh have sketched. was:Brainstorming issue for what to ship in an HBCK2 version 2.0 (HBASE-21745 describes HBCK2 1.0). > [HBCK2] hbck2 2.0 > - > > Key: HBASE-22977 > URL: https://issues.apache.org/jira/browse/HBASE-22977 > Project: HBase > Issue Type: Brainstorming >Reporter: stack >Priority: Major > > Brainstorming issue for what to ship in an HBCK2 version 2.0 (HBASE-21745 > describes HBCK2 1.0). > Possibilities: > * Integrate HBCK2 operations into core either by automating fix or by adding > shell commands for fixup. > * An 'adoption service' running in Master where you pass 'orphan data' -- a > file or a directory -- and the running service figures what to do w/ the > content (remove if empty, figure where the file should slot if an hfile, > replay recovered edits if a recovered_edits dir, play WALs if wal dir, etc.). > Could be smarter/interactive, a 'three-way-merge' as Sean and Josh have > sketched. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HBASE-22976) [HBCK2] Add RecoveredEditsPlayer
[ https://issues.apache.org/jira/browse/HBASE-22976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-22976: -- Parent: HBASE-22977 Issue Type: Sub-task (was: Bug) > [HBCK2] Add RecoveredEditsPlayer > > > Key: HBASE-22976 > URL: https://issues.apache.org/jira/browse/HBASE-22976 > Project: HBase > Issue Type: Sub-task > Components: hbck2 >Reporter: stack >Priority: Major > > We need a recovered edits player. Messing w/ the 'adoption service' -- > tooling to adopt orphan regions and hfiles -- I've been manufacturing damaged > clusters by moving stuff around under the running cluster. No reason to think > that an hbase couldn't lose accounting of a whole region if a cataclysm. If > so, region will have stuff like the '.regioninfo', dirs per column family w/ > store files but it could too have a 'recovered_edits' directory with content > in it. We have a WALPlayer for errant WALs. We have the FSHLog tool which can > read recovered_edits content for debugging data loss. Missing is a > RecoveredEditsPlayer. > I took a look at extending the WALPlayer since it has a bunch of nice options > and it can run at bulk. Ideally, it would just digest recovered edits content > if passed an option or recovered edits directories. On first glance, it > didn't seem like an easy integration Would be worth taking a look again. > Would be good if we could avoid making a new, distinct tool, just for > Recovered Edits. > The bulkload tool expects hfiles in column family directories. Recovered > edits files are not hfiles and the files are x-columnfamily so this is not > the way to go though a bulkload-like tool that moved the recovered edits > files under the appropriate region dir and asked the region reopen would be a > possibility (Would need the bulk load complete trick of splitting input if > the region boundaries in the live cluster do not align w/ those of the errant > recovered edits files). -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (HBASE-22977) [HBCK2] hbck2 2.0
stack created HBASE-22977: - Summary: [HBCK2] hbck2 2.0 Key: HBASE-22977 URL: https://issues.apache.org/jira/browse/HBASE-22977 Project: HBase Issue Type: Brainstorming Reporter: stack Brainstorming issue for what to ship in an HBCK2 version 2.0 (HBASE-21745 describes HBCK2 1.0). -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HBASE-22976) [HBCK2] Add RecoveredEditsPlayer
[ https://issues.apache.org/jira/browse/HBASE-22976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-22976: -- Description: We need a recovered edits player. Messing w/ the 'adoption service' -- tooling to adopt orphan regions and hfiles -- I've been manufacturing damaged clusters by moving stuff around under the running cluster. No reason to think that an hbase couldn't lose accounting of a whole region if a cataclysm. If so, region will have stuff like the '.regioninfo', dirs per column family w/ store files but it could too have a 'recovered_edits' directory with content in it. We have a WALPlayer for errant WALs. We have the FSHLog tool which can read recovered_edits content for debugging data loss. Missing is a RecoveredEditsPlayer. I took a look at extending the WALPlayer since it has a bunch of nice options and it can run at bulk. Ideally, it would just digest recovered edits content if passed an option or recovered edits directories. On first glance, it didn't seem like an easy integration Would be worth taking a look again. Would be good if we could avoid making a new, distinct tool, just for Recovered Edits. The bulkload tool expects hfiles in column family directories. Recovered edits files are not hfiles and the files are x-columnfamily so this is not the way to go though a bulkload-like tool that moved the recovered edits files under the appropriate region dir and asked the region reopen would be a possibility (Would need the bulk load complete trick of splitting input if the region boundaries in the live cluster do not align w/ those of the errant recovered edits files). was: We need a recovered edits player. Messing w/ the 'adoption service' -- tooling to adopt orphan regions and hfiles -- I've been manufacturing damaged clusters by moving stuff around under the running cluster. No reason to think that an hbase couldn't lose accounting of a whole region if a cataclysm. If so, region will have stuff like the '.regioninfo', dirs per column family w/ store files but it could too have a 'recovered_edits' directory with content in it. We have a WALPlayer for errant WALs. We have the FSHLog tool which can read recovered_edits content for debugging data loss. Missing is a RecoveredEditsPlayer. I took a look at extending the WALPlayer since it has a bunch of nice options and it can run at bulk. Ideally, it would just digest recovered edits content if passed an option or recovered edits directories. On first glance, it didn't seem like an easy integration Would be worth taking a look again. Would be good if we could avoid making a new, distinct tool, just for Recovered Edits. > [HBCK2] Add RecoveredEditsPlayer > > > Key: HBASE-22976 > URL: https://issues.apache.org/jira/browse/HBASE-22976 > Project: HBase > Issue Type: Bug > Components: hbck2 >Reporter: stack >Priority: Major > > We need a recovered edits player. Messing w/ the 'adoption service' -- > tooling to adopt orphan regions and hfiles -- I've been manufacturing damaged > clusters by moving stuff around under the running cluster. No reason to think > that an hbase couldn't lose accounting of a whole region if a cataclysm. If > so, region will have stuff like the '.regioninfo', dirs per column family w/ > store files but it could too have a 'recovered_edits' directory with content > in it. We have a WALPlayer for errant WALs. We have the FSHLog tool which can > read recovered_edits content for debugging data loss. Missing is a > RecoveredEditsPlayer. > I took a look at extending the WALPlayer since it has a bunch of nice options > and it can run at bulk. Ideally, it would just digest recovered edits content > if passed an option or recovered edits directories. On first glance, it > didn't seem like an easy integration Would be worth taking a look again. > Would be good if we could avoid making a new, distinct tool, just for > Recovered Edits. > The bulkload tool expects hfiles in column family directories. Recovered > edits files are not hfiles and the files are x-columnfamily so this is not > the way to go though a bulkload-like tool that moved the recovered edits > files under the appropriate region dir and asked the region reopen would be a > possibility (Would need the bulk load complete trick of splitting input if > the region boundaries in the live cluster do not align w/ those of the errant > recovered edits files). -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (HBASE-22976) [HBCK2] Add RecoveredEditsPlayer
stack created HBASE-22976: - Summary: [HBCK2] Add RecoveredEditsPlayer Key: HBASE-22976 URL: https://issues.apache.org/jira/browse/HBASE-22976 Project: HBase Issue Type: Bug Components: hbck2 Reporter: stack We need a recovered edits player. Messing w/ the 'adoption service' -- tooling to adopt orphan regions and hfiles -- I've been manufacturing damaged clusters by moving stuff around under the running cluster. No reason to think that an hbase couldn't lose accounting of a whole region if a cataclysm. If so, region will have stuff like the '.regioninfo', dirs per column family w/ store files but it could too have a 'recovered_edits' directory with content in it. We have a WALPlayer for errant WALs. We have the FSHLog tool which can read recovered_edits content for debugging data loss. Missing is a RecoveredEditsPlayer. I took a look at extending the WALPlayer since it has a bunch of nice options and it can run at bulk. Ideally, it would just digest recovered edits content if passed an option or recovered edits directories. On first glance, it didn't seem like an easy integration Would be worth taking a look again. Would be good if we could avoid making a new, distinct tool, just for Recovered Edits. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work started] (HBASE-21521) Expose master startup status via JMX and web UI
[ https://issues.apache.org/jira/browse/HBASE-21521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-21521 started by Rushabh S Shah. -- > Expose master startup status via JMX and web UI > --- > > Key: HBASE-21521 > URL: https://issues.apache.org/jira/browse/HBASE-21521 > Project: HBase > Issue Type: Improvement > Components: master, UI >Reporter: Andrew Purtell >Assignee: Rushabh S Shah >Priority: Major > > Add an internal API to the master for tracking startup progress. Expose this > information via JMX. > Modify the master to bring the web UI up sooner. Will require tweaks to > various views to prevent attempts to retrieve state before the master fully > up (or else expect NPEs). Currently, before the master has fully initialized > an attempt to use the web UI will return a 500 error code and display an > error page. > Finally, update the web UI to display startup progress, like HDFS-4249. > Filing this for branch-1. Need to check what if anything is available or > improved in branch-2 and master. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HBASE-16450) Shell tool to dump replication queues
[ https://issues.apache.org/jira/browse/HBASE-16450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-16450: Release Note: New tool to dump existing replication peers, configurations and queues when using HBase Replication. hbase org.apache.hadoop.hbase.replication.regionserver.DumpReplicationQueues The tool provides two flags: --distributed This flag will poll each RS for information about the replication queues being processed on this RS. By default this is not enabled and the information about the replication queues and configuration will be obtained from ZooKeeper. --hdfs When --distributed is used, this flag will attempt to calculate the total size of the WAL files used by the replication queues. Since its possible that multiple peers can be configured this value can be overestimated. was: New tool to dump existing replication peers, configurations and queues when using HBase Replication. The tool provides two flags: --distributed This flag will poll each RS for information about the replication queues being processed on this RS. By default this is not enabled and the information about the replication queues and configuration will be obtained from ZooKeeper. --hdfs When --distributed is used, this flag will attempt to calculate the total size of the WAL files used by the replication queues. Since its possible that multiple peers can be configured this value can be overestimated. > Shell tool to dump replication queues > -- > > Key: HBASE-16450 > URL: https://issues.apache.org/jira/browse/HBASE-16450 > Project: HBase > Issue Type: Improvement > Components: Operability, Replication >Affects Versions: 1.3.0, 1.1.5, 1.2.2, 2.0.0 >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez >Priority: Major > Fix For: 1.3.0, 0.98.24, 2.0.0 > > Attachments: HBASE-16450.branch-1.001.patch, > HBASE-16450.branch-1.002.patch, HBASE-16450.master.001.patch, > HBASE-16450.master.002.patch, HBASE-16450.master.003.patch > > > Currently there is no way to dump list of the configured queues and the > replication queues when replication is enabled. Unfortunately the HBase > master only offers an option to dump the whole content of the znodes but not > details on the queues being processed on each RS. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [hbase] Apache-HBase commented on issue #586: HBASE-22975 Add read and write QPS metrics at server level and table level
Apache-HBase commented on issue #586: HBASE-22975 Add read and write QPS metrics at server level and table level URL: https://github.com/apache/hbase/pull/586#issuecomment-528480890 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 0m 40s | Docker mode activated. | ||| _ Prechecks _ | | :green_heart: | dupname | 0m 0s | No case conflicting files found. | | :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | :yellow_heart: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ branch-1 Compile Tests _ | | :blue_heart: | mvndep | 6m 42s | Maven dependency ordering for branch | | :green_heart: | mvninstall | 3m 16s | branch-1 passed | | :green_heart: | compile | 1m 6s | branch-1 passed | | :green_heart: | checkstyle | 2m 0s | branch-1 passed | | :broken_heart: | shadedjars | 0m 12s | branch has 10 errors when building our shaded downstream artifacts. | | :green_heart: | javadoc | 0m 59s | branch-1 passed | | :blue_heart: | spotbugs | 2m 2s | Used deprecated FindBugs config; considering switching to SpotBugs. | | :green_heart: | findbugs | 3m 0s | branch-1 passed | ||| _ Patch Compile Tests _ | | :blue_heart: | mvndep | 0m 11s | Maven dependency ordering for patch | | :green_heart: | mvninstall | 1m 47s | the patch passed | | :green_heart: | compile | 1m 7s | the patch passed | | :green_heart: | javac | 1m 7s | the patch passed | | :green_heart: | checkstyle | 1m 57s | the patch passed | | :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | :broken_heart: | shadedjars | 0m 12s | patch has 10 errors when building our shaded downstream artifacts. | | :green_heart: | hadoopcheck | 4m 32s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | :green_heart: | javadoc | 0m 58s | the patch passed | | :green_heart: | findbugs | 3m 14s | the patch passed | ||| _ Other Tests _ | | :green_heart: | unit | 0m 24s | hbase-hadoop-compat in the patch passed. | | :green_heart: | unit | 0m 30s | hbase-hadoop2-compat in the patch passed. | | :green_heart: | unit | 109m 47s | hbase-server in the patch passed. | | :broken_heart: | asflicense | 1m 13s | The patch generated 2 ASF License warnings. | | | | 147m 29s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-586/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/586 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux d1d8c05d7ccc 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-586/out/precommit/personality/provided.sh | | git revision | branch-1 / 352344c | | Default Java | 1.8.0_222 | | shadedjars | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-586/3/artifact/out/branch-shadedjars.txt | | shadedjars | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-586/3/artifact/out/patch-shadedjars.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-586/3/testReport/ | | asflicense | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-586/3/artifact/out/patch-asflicense-problems.txt | | Max. process+thread count | 4314 (vs. ulimit of 1) | | modules | C: hbase-hadoop-compat hbase-hadoop2-compat hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-586/3/console | | versions | git=1.9.1 maven=3.0.5 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] symat commented on a change in pull request #394: HBASE-22716:upgrade zookeeper version to 3.5.5
symat commented on a change in pull request #394: HBASE-22716:upgrade zookeeper version to 3.5.5 URL: https://github.com/apache/hbase/pull/394#discussion_r316576357 ## File path: pom.xml ## @@ -1471,7 +1471,7 @@ 0.5.0 thrift 0.12.0 -3.4.10 +3.5.5 0.9.94 Review comment: should we also update the jline version? I think it was added because of zookeeper, see https://issues.apache.org/jira/browse/HBASE-20108 zookeeper 3.5.5 is using jline 2.11 however, if we increment the JLine version, then the assembly:single maven target will fail, as the license part in JLine pom changed a bit (they named it 'The BSD License' instead of 'BSD License'). To make it work, we need to add a new supplement to `supplemental-models.xml` like: ``` jline jline JLine 2.11 BSD License http://www.opensource.org/licenses/bsd-license.php repo ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-21879) Read HFile's block to ByteBuffer directly instead of to byte for reducing young gc purpose
[
https://issues.apache.org/jira/browse/HBASE-21879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923616#comment-16923616
]
Hudson commented on HBASE-21879:
Results for branch HBASE-21879
[build #231 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-21879/231/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-21879/231//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-21879/231//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-21879/231//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Read HFile's block to ByteBuffer directly instead of to byte for reducing
> young gc purpose
> --
>
> Key: HBASE-21879
> URL: https://issues.apache.org/jira/browse/HBASE-21879
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HBASE-21879.v1.patch, HBASE-21879.v1.patch,
> QPS-latencies-before-HBASE-21879.png, gc-data-before-HBASE-21879.png
>
>
> In HFileBlock#readBlockDataInternal, we have the following:
> {code}
> @VisibleForTesting
> protected HFileBlock readBlockDataInternal(FSDataInputStream is, long offset,
> long onDiskSizeWithHeaderL, boolean pread, boolean verifyChecksum,
> boolean updateMetrics)
> throws IOException {
> // .
> // TODO: Make this ByteBuffer-based. Will make it easier to go to HDFS with
> BBPool (offheap).
> byte [] onDiskBlock = new byte[onDiskSizeWithHeader + hdrSize];
> int nextBlockOnDiskSize = readAtOffset(is, onDiskBlock, preReadHeaderSize,
> onDiskSizeWithHeader - preReadHeaderSize, true, offset +
> preReadHeaderSize, pread);
> if (headerBuf != null) {
> // ...
> }
> // ...
> }
> {code}
> In the read path, we still read the block from hfile to on-heap byte[], then
> copy the on-heap byte[] to offheap bucket cache asynchronously, and in my
> 100% get performance test, I also observed some frequent young gc, The
> largest memory footprint in the young gen should be the on-heap block byte[].
> In fact, we can read HFile's block to ByteBuffer directly instead of to
> byte[] for reducing young gc purpose. we did not implement this before,
> because no ByteBuffer reading interface in the older HDFS client, but 2.7+
> has supported this now, so we can fix this now. I think.
> Will provide an patch and some perf-comparison for this.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [hbase] saintstack commented on a change in pull request #476: HBASE-11062 hbtop
saintstack commented on a change in pull request #476: HBASE-11062 hbtop URL: https://github.com/apache/hbase/pull/476#discussion_r321372817 ## File path: hbase-hbtop/README.md ## @@ -0,0 +1,248 @@ + + +# hbtop + +## Overview + +`hbtop` is a real-time monitoring tool for HBase like Unix's top command. +It can display summary information as well as metrics per Region/Namespace/Table/RegionServer. +In this tool, you can see the metrics sorted by a selected field and filter the metrics to see only metrics you really want to see. +Also, with the drill-down feature, you can find hot regions easily in a top-down manner. + +## Usage + +You can run hbtop with the following command: + +``` +$ hbase hbtop +``` + +In this case, the values of `hbase.client.zookeeper.quorum` and `zookeeper.znode.parent` in `hbase-site.xml` in the classpath or the default values of them are used to connect. + +Or, you can specify your own zookeeper quorum and znode parent as follows: + +``` +$ hbase hbtop -Dhbase.client.zookeeper.quorum= -Dzookeeper.znode.parent= +``` + + + +The top screen consists of a summary part and of a metrics part. +In the summary part, you can see `HBase Version`, `Cluster ID`, `The number of region servers`, `Region count`, `Average Cluster Load` and `Aggregated Request/s`. +In the metrics part, you can see metrics per Region/Namespace/Table/RegionServer depending on the selected mode. +The top screen is refreshed in a certain period – 3 seconds by default. + +### Scrolling metric records + +You can scroll the metric records in the metrics part. + + + +### Command line arguments + +| Argument | Description | +|---|---| +| -delay arg | The refresh delay (in seconds); default is 3 seconds | +| -help | Print usage; for help while the tool is running press `h` key | +| -mode arg | The mode; `n` (Namespace)`t` (Table)r (Region)`s` (RegionServer), default is `r` (Region) | + +### Modes + +There are the following 4 modes in hbtop: + +| Mode | Description | +|---|---| +| Region | Showing metric records per region | +| Namespace | Showing metric records per namespace | +| Table | Showing metric records per table | +| RegionServer | Showing metric records per region server | + + Region mode + +In Region mode, the default sort field is `#REQ/S`. + +The fields in this mode are as follows: + +| Field | Description | Displayed by default | +|---|---|---| +| RNAME | Region Name | false | +| NAMESPACE | Namespace Name | true | +| TABLE | Table Name | true | +| SCODE | Start Code | false | +| REPID | Replica ID | false | +| REGION | Encoded Region Name | true | +| RS | Short Region Server Name | true | +| LRS | Long Region Server Name | false | +| #REQ/S | Request Count per second | true | +| #READ/S | Read Request Count per second | true | +| #FREAD/S | Filtered Read Request Count per second | true | +| #WRITE/S | Write Request Count per second | true | +| SF | StoreFile Size | true | +| USF | Uncompressed StoreFile Size | false | +| #SF | Number of StoreFiles | true | +| MEMSTORE | MemStore Size | true | +| LOCALITY | Block Locality | true | +| SKEY | Start Key | false | +| #COMPingCELL | Compacting Cell Count | false | +| #COMPedCELL | Compacted Cell Count | false | +| %COMP | Compaction Progress | false | +| LASTMCOMP | Last Major Compaction Time | false | + + Namespace mode + +In Namespace mode, the default sort field is `#REQ/S`. + +The fields in this mode are as follows: + +| Field | Description | Displayed by default | +|---|---|---| +| NAMESPACE | Namespace Name | true | +| #REGION | Region Count | true | +| #REQ/S | Request Count per second | true | +| #READ/S | Read Request Count per second | true | +| #FREAD/S | Filtered Read Request Count per second | true | +| #WRITE/S | Write Request Count per second | true | +| SF | StoreFile Size | true | +| USF | Uncompressed StoreFile Size | false | +| #SF | Number of StoreFiles | true | +| MEMSTORE | MemStore Size | true | + + Table mode + +In Table mode, the default sort field is `#REQ/S`. + +The fields in this mode are as follows: + +| Field | Description | Displayed by default | +|---|---|---| +| NAMESPACE | Namespace Name | true | +| TABLE | Table Name | true | +| #REGION | Region Count | true | +| #REQ/S | Request Count per second | true | +| #READ/S | Read Request Count per second | true | +| #FREAD/S | Filtered Read Request Count per second | true | +| #WRITE/S | Write Request Count per second | true | +| SF | StoreFile Size | true | +| USF | Uncompressed StoreFile Size | false | +| #SF | Number of StoreFiles | true | +| MEMSTORE | MemStore Size | true | + + RegionServer mode + +In RegionServer mode, the default sort field is `#REQ/S`. + +The fields in this mode are as follows: + +| Field | Description | Displayed by default | +|---|---|---| +| RS | Short Region Server Name | true |
[GitHub] [hbase] saintstack commented on a change in pull request #476: HBASE-11062 hbtop
saintstack commented on a change in pull request #476: HBASE-11062 hbtop
URL: https://github.com/apache/hbase/pull/476#discussion_r321374127
##
File path: hbase-hbtop/src/main/java/org/apache/hadoop/hbase/hbtop/HBTop.java
##
@@ -0,0 +1,141 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.hbtop;
+
+import java.util.Objects;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.conf.Configured;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.HBaseInterfaceAudience;
+import org.apache.hadoop.hbase.hbtop.mode.Mode;
+import org.apache.hadoop.hbase.hbtop.screen.Screen;
+import org.apache.hadoop.util.Tool;
+import org.apache.hadoop.util.ToolRunner;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine;
+import org.apache.hbase.thirdparty.org.apache.commons.cli.DefaultParser;
+import org.apache.hbase.thirdparty.org.apache.commons.cli.HelpFormatter;
+import org.apache.hbase.thirdparty.org.apache.commons.cli.Options;
+
+
+/**
+ * A real-time monitoring tool for HBase like Unix top command.
+ */
+@InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.TOOLS)
+public class HBTop extends Configured implements Tool {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(HBTop.class);
+
+ public HBTop() {
+this(HBaseConfiguration.create());
+ }
+
+ public HBTop(Configuration conf) {
+super(Objects.requireNonNull(conf));
+ }
+
+ @Override
+ public int run(String[] args) throws Exception {
+long initialRefreshDelay = 3 * 1000;
+Mode initialMode = Mode.REGION;
+try {
+ // Command line options
+ Options opts = new Options();
+ opts.addOption("help", false,
Review comment:
These do not use short forms? Are we doing the gnu '--help' rather than
wacky java/hadoop '-help'?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #476: HBASE-11062 hbtop
saintstack commented on a change in pull request #476: HBASE-11062 hbtop URL: https://github.com/apache/hbase/pull/476#discussion_r321373113 ## File path: hbase-hbtop/pom.xml ## @@ -0,0 +1,89 @@ + +http://maven.apache.org/POM/4.0.0; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> + + 4.0.0 + +hbase-build-configuration +org.apache.hbase +3.0.0-SNAPSHOT +../hbase-build-configuration + + hbase-hbtop + Apache HBase - HBTop + A real-time monitoring tool for HBase like Unix's top command + + + + +org.apache.maven.plugins +maven-source-plugin + + + + + + org.apache.hbase + hbase-client Review comment: It can't use shaded? (I think I asked that before). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ravowlga123 commented on issue #584: HBASE-22927 Upgrade mockito version for Java 11 compatibility
ravowlga123 commented on issue #584: HBASE-22927 Upgrade mockito version for Java 11 compatibility URL: https://github.com/apache/hbase/pull/584#issuecomment-528412349 Checked out both HBASE-20582 and MENFORCER-300. Will follow the example of MENFORCER-300 and add module-info in ignoreclass section. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #586: HBASE-22975 Add read and write QPS metrics at server level and table level
Apache-HBase commented on issue #586: HBASE-22975 Add read and write QPS metrics at server level and table level URL: https://github.com/apache/hbase/pull/586#issuecomment-528401886 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 0m 39s | Docker mode activated. | ||| _ Prechecks _ | | :green_heart: | dupname | 0m 1s | No case conflicting files found. | | :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | :yellow_heart: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ branch-1 Compile Tests _ | | :blue_heart: | mvndep | 5m 55s | Maven dependency ordering for branch | | :green_heart: | mvninstall | 3m 15s | branch-1 passed | | :green_heart: | compile | 1m 5s | branch-1 passed | | :green_heart: | checkstyle | 2m 2s | branch-1 passed | | :broken_heart: | shadedjars | 0m 13s | branch has 10 errors when building our shaded downstream artifacts. | | :green_heart: | javadoc | 1m 0s | branch-1 passed | | :blue_heart: | spotbugs | 2m 6s | Used deprecated FindBugs config; considering switching to SpotBugs. | | :green_heart: | findbugs | 3m 0s | branch-1 passed | ||| _ Patch Compile Tests _ | | :blue_heart: | mvndep | 0m 11s | Maven dependency ordering for patch | | :green_heart: | mvninstall | 1m 47s | the patch passed | | :green_heart: | compile | 1m 6s | the patch passed | | :green_heart: | javac | 1m 6s | the patch passed | | :broken_heart: | checkstyle | 0m 12s | hbase-hadoop2-compat: The patch generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1) | | :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | :broken_heart: | shadedjars | 0m 11s | patch has 10 errors when building our shaded downstream artifacts. | | :green_heart: | hadoopcheck | 4m 36s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | :green_heart: | javadoc | 0m 56s | the patch passed | | :broken_heart: | findbugs | 0m 36s | hbase-hadoop2-compat generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | :green_heart: | unit | 0m 25s | hbase-hadoop-compat in the patch passed. | | :green_heart: | unit | 0m 30s | hbase-hadoop2-compat in the patch passed. | | :green_heart: | unit | 110m 24s | hbase-server in the patch passed. | | :broken_heart: | asflicense | 1m 13s | The patch generated 2 ASF License warnings. | | | | 147m 33s | | | Reason | Tests | |---:|:--| | FindBugs | module:hbase-hadoop2-compat | | | Sequence of calls to java.util.concurrent.ConcurrentHashMap may not be atomic in org.apache.hadoop.hbase.regionserver.MetricsTableQPSImpl.getOrCreateTableMeter(String) At MetricsTableQPSImpl.java:may not be atomic in org.apache.hadoop.hbase.regionserver.MetricsTableQPSImpl.getOrCreateTableMeter(String) At MetricsTableQPSImpl.java:[line 60] | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-586/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/586 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 4b4d25d74244 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-586/out/precommit/personality/provided.sh | | git revision | branch-1 / 352344c | | Default Java | 1.8.0_222 | | shadedjars | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-586/2/artifact/out/branch-shadedjars.txt | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-586/2/artifact/out/diff-checkstyle-hbase-hadoop2-compat.txt | | shadedjars | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-586/2/artifact/out/patch-shadedjars.txt | | findbugs | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-586/2/artifact/out/new-findbugs-hbase-hadoop2-compat.html | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-586/2/testReport/ | | asflicense | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-586/2/artifact/out/patch-asflicense-problems.txt
[jira] [Commented] (HBASE-22338) LICENSE file only contains Apache 2.0
[ https://issues.apache.org/jira/browse/HBASE-22338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923490#comment-16923490 ] Rabi Kumar K C commented on HBASE-22338: Yes in hbase-connectors-assembly, all licenses for project have been aggregated. Regarding the dependencies other than ALv2 please take a look at the attached file [~busbey]. And please do let me know what would be the effective way to handle this. > LICENSE file only contains Apache 2.0 > - > > Key: HBASE-22338 > URL: https://issues.apache.org/jira/browse/HBASE-22338 > Project: HBase > Issue Type: Bug > Components: hbase-connectors >Affects Versions: connector-1.0.0 >Reporter: Peter Somogyi >Assignee: Rabi Kumar K C >Priority: Critical > Fix For: connector-1.0.1 > > Attachments: hbase-connectors-dependency.html > > > LICENSE.md file has only Apache 2.0 licenses but we package dependencies that > use different ones. For example jcodings uses MIT. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HBASE-22338) LICENSE file only contains Apache 2.0
[ https://issues.apache.org/jira/browse/HBASE-22338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rabi Kumar K C updated HBASE-22338: --- Attachment: hbase-connectors-dependency.html > LICENSE file only contains Apache 2.0 > - > > Key: HBASE-22338 > URL: https://issues.apache.org/jira/browse/HBASE-22338 > Project: HBase > Issue Type: Bug > Components: hbase-connectors >Affects Versions: connector-1.0.0 >Reporter: Peter Somogyi >Assignee: Rabi Kumar K C >Priority: Critical > Fix For: connector-1.0.1 > > Attachments: hbase-connectors-dependency.html > > > LICENSE.md file has only Apache 2.0 licenses but we package dependencies that > use different ones. For example jcodings uses MIT. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [hbase] busbey edited a comment on issue #584: HBASE-22927 Upgrade mockito version for Java 11 compatibility
busbey edited a comment on issue #584: HBASE-22927 Upgrade mockito version for Java 11 compatibility URL: https://github.com/apache/hbase/pull/584#issuecomment-528391822 > Seems hadoop has already updated the mockito version to 2.23.4? And seems the UTs are fine so this maybe a too strict enforcer rule? @busbey PTAL. This sounded familiar so I dug about. This came up with a JRuby update on [HBASE-20582](https://issues.apache.org/jira/browse/HBASE-20582) and is tracked in [MENFORCER-300](https://issues.apache.org/jira/browse/MENFORCER-300). On HBASE-20582 we pinned our version of JRuby to avoid the issue (which is what it looks like the byte-buddy developer suggested as a workaround). Since then MENFORCER-300 has been updated to include an example of how to ignore the `module-info` class. We should probably do that. It might also be worth reopening MENFORCER-300 to discuss what that plugin should be doing about the class. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-22338) LICENSE file only contains Apache 2.0
[ https://issues.apache.org/jira/browse/HBASE-22338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rabi Kumar K C updated HBASE-22338: --- Attachment: (was: hbase-connectors-dependency.html) > LICENSE file only contains Apache 2.0 > - > > Key: HBASE-22338 > URL: https://issues.apache.org/jira/browse/HBASE-22338 > Project: HBase > Issue Type: Bug > Components: hbase-connectors >Affects Versions: connector-1.0.0 >Reporter: Peter Somogyi >Assignee: Rabi Kumar K C >Priority: Critical > Fix For: connector-1.0.1 > > > LICENSE.md file has only Apache 2.0 licenses but we package dependencies that > use different ones. For example jcodings uses MIT. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HBASE-22338) LICENSE file only contains Apache 2.0
[ https://issues.apache.org/jira/browse/HBASE-22338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rabi Kumar K C updated HBASE-22338: --- Attachment: hbase-connectors-dependency.html > LICENSE file only contains Apache 2.0 > - > > Key: HBASE-22338 > URL: https://issues.apache.org/jira/browse/HBASE-22338 > Project: HBase > Issue Type: Bug > Components: hbase-connectors >Affects Versions: connector-1.0.0 >Reporter: Peter Somogyi >Assignee: Rabi Kumar K C >Priority: Critical > Fix For: connector-1.0.1 > > > LICENSE.md file has only Apache 2.0 licenses but we package dependencies that > use different ones. For example jcodings uses MIT. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [hbase] busbey commented on issue #584: HBASE-22927 Upgrade mockito version for Java 11 compatibility
busbey commented on issue #584: HBASE-22927 Upgrade mockito version for Java 11 compatibility URL: https://github.com/apache/hbase/pull/584#issuecomment-528393455 Hadoop updating shouldn't be taken as an indicator of anything, IMHO. They only ensure a certain minimum JDK is used at build time; they don't have any checks to make sure generated artifacts will work with any particular JDK version. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] busbey commented on issue #584: HBASE-22927 Upgrade mockito version for Java 11 compatibility
busbey commented on issue #584: HBASE-22927 Upgrade mockito version for Java 11
compatibility
URL: https://github.com/apache/hbase/pull/584#issuecomment-528391822
> Seems hadoop has already updated the mockito version to 2.23.4? And seems
the UTs are fine so this maybe a too strict enforcer rule? @busbey PTAL.
This sounded familiar so I dug about. This came up with a JRuby update on
[HBASE-20582](https://issues.apache.org/jira/browse/HBASE-20582) and is tracked
in [MENFORCER-300](https://issues.apache.org/jira/browse/MENFORCER-300).
On HBASE-20582 we pinned our version of JRuby to avoid the issue (which is
what it looks like the byte-buddy developer suggested as a workaround). Since
then MENFORCER-300 has been updated to include an example of how to ignore the
{{module-info}} class. We should probably do that. It might also be worth
reopening MENFORCER-300 to discuss what that plugin should be doing about the
class.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache9 commented on issue #584: HBASE-22927 Upgrade mockito version for Java 11 compatibility
Apache9 commented on issue #584: HBASE-22927 Upgrade mockito version for Java 11 compatibility URL: https://github.com/apache/hbase/pull/584#issuecomment-528382425 Seems hadoop has already updated the mockito version to 2.23.4? And seems the UTs are fine so this maybe a too strict enforcer rule? @busbey PTAL. And why not upgrade to 2.28.2? And also there is a 3.0.0 release now? What's is the difference 3.x and 2.x? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ravowlga123 edited a comment on issue #584: HBASE-22927 Upgrade mockito version for Java 11 compatibility
ravowlga123 edited a comment on issue #584: HBASE-22927 Upgrade mockito version for Java 11 compatibility URL: https://github.com/apache/hbase/pull/584#issuecomment-528378112 Comment on the similar issue https://github.com/raphw/byte-buddy/issues/541#issuecomment-429649501. Work around for above can be to add module-info.class in . What do you think @Apache9?? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ravowlga123 commented on issue #584: HBASE-22927 Upgrade mockito version for Java 11 compatibility
ravowlga123 commented on issue #584: HBASE-22927 Upgrade mockito version for Java 11 compatibility URL: https://github.com/apache/hbase/pull/584#issuecomment-528378112 Comment on the similar issue https://github.com/raphw/byte-buddy/issues/541#issuecomment-429649501. Work around for above can be to add module-info.class in . What do you think @Apache9 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #588: HBASE-22932 Add rs group management methods in Admin and AsyncAdmin
Apache-HBase commented on issue #588: HBASE-22932 Add rs group management methods in Admin and AsyncAdmin URL: https://github.com/apache/hbase/pull/588#issuecomment-528365186 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 1m 5s | Docker mode activated. | ||| _ Prechecks _ | | :green_heart: | dupname | 0m 0s | No case conflicting files found. | | :blue_heart: | prototool | 0m 1s | prototool was not available. | | :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ HBASE-22514 Compile Tests _ | | :blue_heart: | mvndep | 0m 33s | Maven dependency ordering for branch | | :green_heart: | mvninstall | 5m 46s | HBASE-22514 passed | | :green_heart: | compile | 2m 36s | HBASE-22514 passed | | :green_heart: | checkstyle | 2m 51s | HBASE-22514 passed | | :green_heart: | shadedjars | 4m 56s | branch has no errors when building our shaded downstream artifacts. | | :green_heart: | javadoc | 1m 49s | HBASE-22514 passed | | :blue_heart: | spotbugs | 1m 47s | Used deprecated FindBugs config; considering switching to SpotBugs. | | :green_heart: | findbugs | 11m 36s | HBASE-22514 passed | ||| _ Patch Compile Tests _ | | :blue_heart: | mvndep | 0m 14s | Maven dependency ordering for patch | | :broken_heart: | mvninstall | 3m 39s | root in the patch failed. | | :broken_heart: | compile | 1m 1s | hbase-server in the patch failed. | | :broken_heart: | cc | 1m 1s | hbase-server in the patch failed. | | :broken_heart: | javac | 1m 1s | hbase-server in the patch failed. | | :broken_heart: | checkstyle | 0m 40s | hbase-client: The patch generated 18 new + 162 unchanged - 0 fixed = 180 total (was 162) | | :broken_heart: | checkstyle | 1m 34s | hbase-server: The patch generated 26 new + 159 unchanged - 0 fixed = 185 total (was 159) | | :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | :broken_heart: | shadedjars | 4m 19s | patch has 98 errors when building our shaded downstream artifacts. | | :broken_heart: | hadoopcheck | 2m 18s | The patch causes 98 errors with Hadoop v2.8.5. | | :broken_heart: | hadoopcheck | 4m 31s | The patch causes 98 errors with Hadoop v2.9.2. | | :broken_heart: | hadoopcheck | 6m 46s | The patch causes 98 errors with Hadoop v3.1.2. | | :broken_heart: | hbaseprotoc | 0m 45s | hbase-server in the patch failed. | | :broken_heart: | javadoc | 0m 38s | hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | :broken_heart: | findbugs | 0m 56s | hbase-server in the patch failed. | ||| _ Other Tests _ | | :green_heart: | unit | 0m 41s | hbase-protocol-shaded in the patch passed. | | :green_heart: | unit | 1m 44s | hbase-client in the patch passed. | | :broken_heart: | unit | 0m 54s | hbase-server in the patch failed. | | :green_heart: | unit | 3m 41s | hbase-thrift in the patch passed. | | :green_heart: | asflicense | 0m 37s | The patch does not generate ASF License warnings. | | | | 73m 11s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-588/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/588 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile cc hbaseprotoc prototool | | uname | Linux a13a2f70f93a 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-588/out/precommit/personality/provided.sh | | git revision | HBASE-22514 / 855c0922f8 | | Default Java | 1.8.0_181 | | mvninstall | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-588/1/artifact/out/patch-mvninstall-root.txt | | compile | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-588/1/artifact/out/patch-compile-hbase-server.txt | | cc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-588/1/artifact/out/patch-compile-hbase-server.txt | | javac | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-588/1/artifact/out/patch-compile-hbase-server.txt | | checkstyle |
[GitHub] [hbase] Apache-HBase commented on issue #587: HBASE-22932 Add rs group management methods in Admin and AsyncAdmin
Apache-HBase commented on issue #587: HBASE-22932 Add rs group management methods in Admin and AsyncAdmin URL: https://github.com/apache/hbase/pull/587#issuecomment-528354779 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 0m 33s | Docker mode activated. | ||| _ Prechecks _ | | :green_heart: | dupname | 0m 0s | No case conflicting files found. | | :blue_heart: | prototool | 0m 0s | prototool was not available. | | :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ HBASE-22514 Compile Tests _ | | :blue_heart: | mvndep | 1m 1s | Maven dependency ordering for branch | | :green_heart: | mvninstall | 5m 45s | HBASE-22514 passed | | :green_heart: | compile | 2m 35s | HBASE-22514 passed | | :green_heart: | checkstyle | 2m 58s | HBASE-22514 passed | | :green_heart: | shadedjars | 4m 56s | branch has no errors when building our shaded downstream artifacts. | | :green_heart: | javadoc | 1m 41s | HBASE-22514 passed | | :blue_heart: | spotbugs | 1m 36s | Used deprecated FindBugs config; considering switching to SpotBugs. | | :green_heart: | findbugs | 10m 9s | HBASE-22514 passed | | :yellow_heart: | patch | 2m 2s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | :blue_heart: | mvndep | 0m 13s | Maven dependency ordering for patch | | :broken_heart: | mvninstall | 2m 55s | root in the patch failed. | | :broken_heart: | compile | 0m 52s | hbase-server in the patch failed. | | :broken_heart: | cc | 0m 52s | hbase-server in the patch failed. | | :broken_heart: | javac | 0m 52s | hbase-server in the patch failed. | | :broken_heart: | checkstyle | 0m 34s | hbase-client: The patch generated 18 new + 162 unchanged - 0 fixed = 180 total (was 162) | | :broken_heart: | checkstyle | 1m 30s | hbase-server: The patch generated 26 new + 159 unchanged - 0 fixed = 185 total (was 159) | | :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | :broken_heart: | shadedjars | 3m 48s | patch has 98 errors when building our shaded downstream artifacts. | | :broken_heart: | hadoopcheck | 2m 9s | The patch causes 98 errors with Hadoop v2.8.5. | | :broken_heart: | hadoopcheck | 4m 21s | The patch causes 98 errors with Hadoop v2.9.2. | | :broken_heart: | hadoopcheck | 6m 39s | The patch causes 98 errors with Hadoop v3.1.2. | | :broken_heart: | hbaseprotoc | 0m 44s | hbase-server in the patch failed. | | :broken_heart: | javadoc | 0m 33s | hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | :broken_heart: | findbugs | 0m 52s | hbase-server in the patch failed. | ||| _ Other Tests _ | | :green_heart: | unit | 0m 41s | hbase-protocol-shaded in the patch passed. | | :green_heart: | unit | 1m 45s | hbase-client in the patch passed. | | :broken_heart: | unit | 0m 54s | hbase-server in the patch failed. | | :green_heart: | unit | 2m 58s | hbase-thrift in the patch passed. | | :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. | | | | 67m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-587/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/587 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile cc hbaseprotoc prototool | | uname | Linux 193954262fc9 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-587/out/precommit/personality/provided.sh | | git revision | HBASE-22514 / 855c0922f8 | | Default Java | 1.8.0_181 | | mvninstall | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-587/1/artifact/out/patch-mvninstall-root.txt | | compile | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-587/1/artifact/out/patch-compile-hbase-server.txt | | cc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-587/1/artifact/out/patch-compile-hbase-server.txt | | javac |
[jira] [Commented] (HBASE-22649) FileNotFoundException shown in UI when tried to access HFILE URL of a column family name have special char (e.g #)
[
https://issues.apache.org/jira/browse/HBASE-22649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923377#comment-16923377
]
HBase QA commented on HBASE-22649:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
34s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
30s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
37s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
16m 23s{color} | {color:green} Patch does not cause any errors with Hadoop
2.8.5 2.9.2 or 3.1.2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}296m 29s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}352m 41s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.TestMasterShutdown |
| | hadoop.hbase.client.TestFromClientSide |
| | hadoop.hbase.client.TestAdmin1 |
| | hadoop.hbase.security.access.TestSnapshotScannerHDFSAclController |
| | hadoop.hbase.client.TestAsyncRegionAdminApi2 |
| | hadoop.hbase.client.TestSnapshotTemporaryDirectoryWithRegionReplicas |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 base:
https://builds.apache.org/job/PreCommit-HBASE-Build/864/artifact/patchprocess/Dockerfile
|
| JIRA Issue | HBASE-22649 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12979519/HBASE-22649_002.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux bc83c394bc63 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 0d338a1820 |
| maven | version: Apache Maven 3.5.4
[GitHub] [hbase] sunhelly commented on issue #588: HBASE-22932 Add rs group management methods in Admin and AsyncAdmin
sunhelly commented on issue #588: HBASE-22932 Add rs group management methods in Admin and AsyncAdmin URL: https://github.com/apache/hbase/pull/588#issuecomment-528337113 Not completed. Hasn't change UTs, and plan to add > rpc GetRSGroupInfoOfTable(GetRSGroupInfoOfTableRequest) > returns (GetRSGroupInfoOfTableResponse); and a method setRSGroup(TableName tableName). Pushed here for some suggestions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] sunhelly closed pull request #587: HBASE-22932 Add rs group management methods in Admin and AsyncAdmin
sunhelly closed pull request #587: HBASE-22932 Add rs group management methods in Admin and AsyncAdmin URL: https://github.com/apache/hbase/pull/587 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] sunhelly opened a new pull request #588: HBASE-22932 Add rs group management methods in Admin and AsyncAdmin
sunhelly opened a new pull request #588: HBASE-22932 Add rs group management methods in Admin and AsyncAdmin URL: https://github.com/apache/hbase/pull/588 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #586: HBASE-22975 Add read and write QPS metrics at server level and table level
Apache-HBase commented on issue #586: HBASE-22975 Add read and write QPS metrics at server level and table level URL: https://github.com/apache/hbase/pull/586#issuecomment-528335410 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 0m 44s | Docker mode activated. | ||| _ Prechecks _ | | :green_heart: | dupname | 0m 0s | No case conflicting files found. | | :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | :yellow_heart: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ branch-1 Compile Tests _ | | :blue_heart: | mvndep | 5m 53s | Maven dependency ordering for branch | | :green_heart: | mvninstall | 3m 22s | branch-1 passed | | :green_heart: | compile | 1m 11s | branch-1 passed | | :green_heart: | checkstyle | 2m 8s | branch-1 passed | | :broken_heart: | shadedjars | 0m 13s | branch has 10 errors when building our shaded downstream artifacts. | | :green_heart: | javadoc | 0m 59s | branch-1 passed | | :blue_heart: | spotbugs | 2m 8s | Used deprecated FindBugs config; considering switching to SpotBugs. | | :green_heart: | findbugs | 3m 9s | branch-1 passed | ||| _ Patch Compile Tests _ | | :blue_heart: | mvndep | 0m 11s | Maven dependency ordering for patch | | :green_heart: | mvninstall | 1m 54s | the patch passed | | :green_heart: | compile | 1m 9s | the patch passed | | :green_heart: | javac | 1m 9s | the patch passed | | :broken_heart: | checkstyle | 0m 11s | hbase-hadoop-compat: The patch generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0) | | :broken_heart: | checkstyle | 0m 13s | hbase-hadoop2-compat: The patch generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1) | | :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | :broken_heart: | shadedjars | 0m 12s | patch has 10 errors when building our shaded downstream artifacts. | | :green_heart: | hadoopcheck | 4m 47s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | :green_heart: | javadoc | 0m 58s | the patch passed | | :broken_heart: | findbugs | 0m 36s | hbase-hadoop2-compat generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | :green_heart: | unit | 0m 23s | hbase-hadoop-compat in the patch passed. | | :green_heart: | unit | 0m 31s | hbase-hadoop2-compat in the patch passed. | | :broken_heart: | unit | 116m 45s | hbase-server in the patch failed. | | :broken_heart: | asflicense | 1m 3s | The patch generated 2 ASF License warnings. | | | | 155m 7s | | | Reason | Tests | |---:|:--| | FindBugs | module:hbase-hadoop2-compat | | | Sequence of calls to java.util.concurrent.ConcurrentHashMap may not be atomic in org.apache.hadoop.hbase.regionserver.MetricsTableQPSImpl.getOrCreateTableMeter(String) At MetricsTableQPSImpl.java:may not be atomic in org.apache.hadoop.hbase.regionserver.MetricsTableQPSImpl.getOrCreateTableMeter(String) At MetricsTableQPSImpl.java:[line 60] | | Failed junit tests | hadoop.hbase.regionserver.TestCompactionWithCoprocessor | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-586/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/586 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux e0a56bcbf623 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-586/out/precommit/personality/provided.sh | | git revision | branch-1 / 157ffae | | Default Java | 1.8.0_222 | | shadedjars | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-586/1/artifact/out/branch-shadedjars.txt | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-586/1/artifact/out/diff-checkstyle-hbase-hadoop-compat.txt | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-586/1/artifact/out/diff-checkstyle-hbase-hadoop2-compat.txt | | shadedjars | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-586/1/artifact/out/patch-shadedjars.txt | |
[GitHub] [hbase] sunhelly opened a new pull request #587: HBASE-22932 Add rs group management methods in Admin and AsyncAdmin
sunhelly opened a new pull request #587: HBASE-22932 Add rs group management methods in Admin and AsyncAdmin URL: https://github.com/apache/hbase/pull/587 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-22912) [Backport] HBASE-22867 to branch-1 to avoid ForkJoinPool to spawn thousands of threads
[
https://issues.apache.org/jira/browse/HBASE-22912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923322#comment-16923322
]
Hudson commented on HBASE-22912:
Results for branch branch-1.4
[build #997 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/997/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/997//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/997//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/997//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> [Backport] HBASE-22867 to branch-1 to avoid ForkJoinPool to spawn thousands
> of threads
> --
>
> Key: HBASE-22912
> URL: https://issues.apache.org/jira/browse/HBASE-22912
> Project: HBase
> Issue Type: Improvement
>Reporter: Reid Chan
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 1.5.0, 1.4.11
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22867) The ForkJoinPool in CleanerChore will spawn thousands of threads in our cluster with thousands table
[
https://issues.apache.org/jira/browse/HBASE-22867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923323#comment-16923323
]
Hudson commented on HBASE-22867:
Results for branch branch-1.4
[build #997 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/997/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/997//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/997//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/997//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> The ForkJoinPool in CleanerChore will spawn thousands of threads in our
> cluster with thousands table
>
>
> Key: HBASE-22867
> URL: https://issues.apache.org/jira/browse/HBASE-22867
> Project: HBase
> Issue Type: Bug
> Components: master
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Critical
> Fix For: 3.0.0, 2.3.0, 2.2.1, 2.1.6
>
> Attachments: 191318.stack, 191318.stack.1, 31162.stack.1
>
>
> The thousands of spawned threads make the safepoint cost 80+s in our Master
> JVM processs.
> {code}
> 2019-08-15,19:35:35,861 INFO [main-SendThread(zjy-hadoop-prc-zk02.bj:11000)]
> org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard
> from server in 82260ms for sessionid 0x1691332e2d3aae5, closing socket
> connection and at
> tempting reconnect
> {code}
> The stdout from JVM (can see from here there're 9126 threads & sync cost 80+s)
> {code}
> vmop[threads: total initially_running wait_to_block]
> [time: spin block sync cleanup vmop] page_trap_count
> 32358.859: ForceAsyncSafepoint [9126 67
> 474] [ 128 8659687 101] 0
> {code}
> Also we got the jstack:
> {code}
> $ cat 31162.stack.1 | grep 'ForkJoinPool-1-worker' | wc -l
> 8648
> {code}
> It's a dangerous bug, make it as blocker.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
