[GitHub] [hbase] Apache-HBase commented on issue #1079: HBASE-23711 - Add test for MinVersions and KeepDeletedCells TTL
Apache-HBase commented on issue #1079: HBASE-23711 - Add test for MinVersions and KeepDeletedCells TTL URL: https://github.com/apache/hbase/pull/1079#issuecomment-577051743 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 14s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 59s | master passed | | +1 :green_heart: | compile | 1m 0s | master passed | | +1 :green_heart: | checkstyle | 1m 17s | master passed | | +1 :green_heart: | shadedjars | 5m 1s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | master passed | | +0 :ok: | spotbugs | 5m 3s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 5m 0s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 34s | the patch passed | | +1 :green_heart: | compile | 1m 7s | the patch passed | | +1 :green_heart: | javac | 1m 7s | the patch passed | | +1 :green_heart: | checkstyle | 1m 16s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 5m 1s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 18m 33s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | | +1 :green_heart: | findbugs | 5m 9s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 162m 33s | hbase-server in the patch passed. | | +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. | | | | 230m 15s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1079/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1079 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux e3c31cbc0b4b 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1079/out/precommit/personality/provided.sh | | git revision | master / 2ed81c645b | | Default Java | 1.8.0_181 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1079/2/testReport/ | | Max. process+thread count | 5253 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1079/2/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #1043: HBASE-23055 Alter hbase:meta
Apache9 commented on a change in pull request #1043: HBASE-23055 Alter hbase:meta URL: https://github.com/apache/hbase/pull/1043#discussion_r369405854 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyTableProcedure.java ## @@ -59,10 +62,19 @@ private TableDescriptor modifiedTableDescriptor; private boolean deleteColumnFamilyInModify; private boolean shouldCheckDescriptor; + /** + * List of column families that cannot be deleted from the hbase:meta table. + * They are critical to cluster operation. This is a bit of an odd place to + * keep this list but then this is the tooling that does add/remove. Keeping + * it local! + */ + private static final List UNDELETABLE_META_COLUMNFAMILIES = Review comment: There are other things we can not change. For example we should not set TTL on the columnf families, and also, you should not change the versions of REPLICATION_BARRIER_FAMILY... Can be a follow on. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (HBASE-23601) OutputSink.WriterThread exception gets stuck and repeated indefinietly
[
https://issues.apache.org/jira/browse/HBASE-23601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack resolved HBASE-23601.
---
Resolution: Fixed
Resolving. Can make subtasks to go back to other branches. Thanks for nice
patch [~bszabolcs]
> OutputSink.WriterThread exception gets stuck and repeated indefinietly
> --
>
> Key: HBASE-23601
> URL: https://issues.apache.org/jira/browse/HBASE-23601
> Project: HBase
> Issue Type: Bug
> Components: read replicas
>Affects Versions: 2.2.2
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
>
> When a WriterThread runs into an exception (ie: NotServingRegionException),
> the exception is stored in the controller. It is never removed and can not be
> overwritten either.
>
> {code:java}
> public void run() {
> try {
> doRun();
> } catch (Throwable t) {
> LOG.error("Exiting thread", t);
> controller.writerThreadError(t);
> }
> }{code}
> Thanks to this every time PipelineController.checkForErrors() is called the
> same old exception is rethrown.
>
> For example in RegionReplicaReplicationEndpoint.replicate there is a while
> loop that does the actual replicating. Every time it loops, it calls
> checkForErrors(), catches the rethrown exception, logs it but does nothing
> about it. This results in ~2GB log files in ~5min in my experience.
>
> My proposal would be to clean up the stored exception when it reaches
> RegionReplicaReplicationEndpoint.replicate and make sure we restart the
> WriterThread that died throwing it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23601) OutputSink.WriterThread exception gets stuck and repeated indefinietly
[
https://issues.apache.org/jira/browse/HBASE-23601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack updated HBASE-23601:
--
Fix Version/s: (was: 2.2.4)
(was: 2.1.9)
> OutputSink.WriterThread exception gets stuck and repeated indefinietly
> --
>
> Key: HBASE-23601
> URL: https://issues.apache.org/jira/browse/HBASE-23601
> Project: HBase
> Issue Type: Bug
> Components: read replicas
>Affects Versions: 2.2.2
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
>
> When a WriterThread runs into an exception (ie: NotServingRegionException),
> the exception is stored in the controller. It is never removed and can not be
> overwritten either.
>
> {code:java}
> public void run() {
> try {
> doRun();
> } catch (Throwable t) {
> LOG.error("Exiting thread", t);
> controller.writerThreadError(t);
> }
> }{code}
> Thanks to this every time PipelineController.checkForErrors() is called the
> same old exception is rethrown.
>
> For example in RegionReplicaReplicationEndpoint.replicate there is a while
> loop that does the actual replicating. Every time it loops, it calls
> checkForErrors(), catches the rethrown exception, logs it but does nothing
> about it. This results in ~2GB log files in ~5min in my experience.
>
> My proposal would be to clean up the stored exception when it reaches
> RegionReplicaReplicationEndpoint.replicate and make sure we restart the
> WriterThread that died throwing it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-17721) Provide streaming APIs with SSL/TLS
[ https://issues.apache.org/jira/browse/HBASE-17721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020853#comment-17020853 ] Michael Stack commented on HBASE-17721: --- Unscheduling feature from 2.3.0. > Provide streaming APIs with SSL/TLS > --- > > Key: HBASE-17721 > URL: https://issues.apache.org/jira/browse/HBASE-17721 > Project: HBase > Issue Type: Umbrella > Components: grpc, rpc, security >Reporter: Alex Araujo >Assignee: Alex Araujo >Priority: Major > Fix For: 3.0.0 > > Attachments: 0001-Initial-add-of-grpc.patch > > > Umbrella to add optional client/server streaming capabilities to HBase. > This would allow bandwidth to be used more efficiently for certain > operations, and allow clients to use SSL/TLS for authentication and > encryption. > Desired client/server scaffolding: > - HTTP/2 support > - Protocol negotiation (blocking vs streaming, auth, encryption, etc.) > - TLS/SSL support > - Streaming RPC support > Possibilities (and their tradeoffs): > - gRPC: Some initial work and discussion on HBASE-13467 (Prototype using GRPC > as IPC mechanism) > -- Has most or all of the desired scaffolding > -- Adds additional g* dependencies. Compat story for g* dependencies not > always ideal > - Custom HTTP/2 based client/server APIs > -- More control over compat story > -- Non-trivial to build scaffolding; might reinvent wheels along the way > - Others? > Related Jiras that might be rolled in as sub-tasks (or closed/replaced with > new ones): > HBASE-17708 (Expose config to set two-way auth over TLS in HttpServer and add > a test) > HBASE-8691 (High-Throughput Streaming Scan API) > HBASE-14899 (Create custom Streaming ReplicationEndpoint) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-17721) Provide streaming APIs with SSL/TLS
[ https://issues.apache.org/jira/browse/HBASE-17721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-17721: -- Component/s: grpc > Provide streaming APIs with SSL/TLS > --- > > Key: HBASE-17721 > URL: https://issues.apache.org/jira/browse/HBASE-17721 > Project: HBase > Issue Type: Umbrella > Components: grpc, rpc, security >Reporter: Alex Araujo >Assignee: Alex Araujo >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: 0001-Initial-add-of-grpc.patch > > > Umbrella to add optional client/server streaming capabilities to HBase. > This would allow bandwidth to be used more efficiently for certain > operations, and allow clients to use SSL/TLS for authentication and > encryption. > Desired client/server scaffolding: > - HTTP/2 support > - Protocol negotiation (blocking vs streaming, auth, encryption, etc.) > - TLS/SSL support > - Streaming RPC support > Possibilities (and their tradeoffs): > - gRPC: Some initial work and discussion on HBASE-13467 (Prototype using GRPC > as IPC mechanism) > -- Has most or all of the desired scaffolding > -- Adds additional g* dependencies. Compat story for g* dependencies not > always ideal > - Custom HTTP/2 based client/server APIs > -- More control over compat story > -- Non-trivial to build scaffolding; might reinvent wheels along the way > - Others? > Related Jiras that might be rolled in as sub-tasks (or closed/replaced with > new ones): > HBASE-17708 (Expose config to set two-way auth over TLS in HttpServer and add > a test) > HBASE-8691 (High-Throughput Streaming Scan API) > HBASE-14899 (Create custom Streaming ReplicationEndpoint) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-17721) Provide streaming APIs with SSL/TLS
[ https://issues.apache.org/jira/browse/HBASE-17721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-17721: -- Fix Version/s: (was: 2.3.0) > Provide streaming APIs with SSL/TLS > --- > > Key: HBASE-17721 > URL: https://issues.apache.org/jira/browse/HBASE-17721 > Project: HBase > Issue Type: Umbrella > Components: grpc, rpc, security >Reporter: Alex Araujo >Assignee: Alex Araujo >Priority: Major > Fix For: 3.0.0 > > Attachments: 0001-Initial-add-of-grpc.patch > > > Umbrella to add optional client/server streaming capabilities to HBase. > This would allow bandwidth to be used more efficiently for certain > operations, and allow clients to use SSL/TLS for authentication and > encryption. > Desired client/server scaffolding: > - HTTP/2 support > - Protocol negotiation (blocking vs streaming, auth, encryption, etc.) > - TLS/SSL support > - Streaming RPC support > Possibilities (and their tradeoffs): > - gRPC: Some initial work and discussion on HBASE-13467 (Prototype using GRPC > as IPC mechanism) > -- Has most or all of the desired scaffolding > -- Adds additional g* dependencies. Compat story for g* dependencies not > always ideal > - Custom HTTP/2 based client/server APIs > -- More control over compat story > -- Non-trivial to build scaffolding; might reinvent wheels along the way > - Others? > Related Jiras that might be rolled in as sub-tasks (or closed/replaced with > new ones): > HBASE-17708 (Expose config to set two-way auth over TLS in HttpServer and add > a test) > HBASE-8691 (High-Throughput Streaming Scan API) > HBASE-14899 (Create custom Streaming ReplicationEndpoint) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-18495) [AMv2] Make methods isServerOnline() and isServerDead() in ServerManager mutually exclusive
[ https://issues.apache.org/jira/browse/HBASE-18495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020852#comment-17020852 ] Michael Stack commented on HBASE-18495: --- Unscheduling issue not being worked on. May not be an issue going by last comment. > [AMv2] Make methods isServerOnline() and isServerDead() in ServerManager > mutually exclusive > --- > > Key: HBASE-18495 > URL: https://issues.apache.org/jira/browse/HBASE-18495 > Project: HBase > Issue Type: Bug > Components: amv2 >Affects Versions: 2.0.0 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Major > > During debugging HBASE-18366, it was found that > ServerManager.isServerOnline() and ServerManager.isServerDead() can return > true at the same time. This causes problems with the code depending on which > method is used. Making them mutually exclusive seems line right thing to do. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-18495) [AMv2] Make methods isServerOnline() and isServerDead() in ServerManager mutually exclusive
[ https://issues.apache.org/jira/browse/HBASE-18495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-18495: -- Fix Version/s: (was: 2.3.0) (was: 3.0.0) > [AMv2] Make methods isServerOnline() and isServerDead() in ServerManager > mutually exclusive > --- > > Key: HBASE-18495 > URL: https://issues.apache.org/jira/browse/HBASE-18495 > Project: HBase > Issue Type: Bug > Components: amv2 >Affects Versions: 2.0.0 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Major > > During debugging HBASE-18366, it was found that > ServerManager.isServerOnline() and ServerManager.isServerDead() can return > true at the same time. This causes problems with the code depending on which > method is used. Making them mutually exclusive seems line right thing to do. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-18300) Implement a Multi TieredBucketCache
[ https://issues.apache.org/jira/browse/HBASE-18300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-18300: -- Fix Version/s: (was: 2.3.0) > Implement a Multi TieredBucketCache > --- > > Key: HBASE-18300 > URL: https://issues.apache.org/jira/browse/HBASE-18300 > Project: HBase > Issue Type: New Feature > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > > We did an internal brainstorming to study the feasibility of this. Some of > our recent tests on SSDs like Optane shows that they are vastly faster in > randomreads and can act as effective caches. > In the current state we have a single tier of Bucket cache and the bucket > cache can either be offheap or configured to work with file mode. (The file > mode can have multiple files backing it). > So this model restricts us from using either the memory or the file and not > both. > With the advent of faster devices like Optane SSDs, NVMe based devices it is > better we try to utilize all those devices and try using them for the bucket > cache so that we can avoid the impact of slower devices where the actual data > resides on the HDFS data nodes. > Combined with this we can allow the user to configure the caching layer per > family/table so that one can effectively make use of the caching tiers. > Can upload a design doc here. Before that, would like to know the suggestions > here. Thoughts!!! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-18300) Implement a Multi TieredBucketCache
[ https://issues.apache.org/jira/browse/HBASE-18300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020850#comment-17020850 ] Michael Stack commented on HBASE-18300: --- Unscheduling fieature not being worked on. > Implement a Multi TieredBucketCache > --- > > Key: HBASE-18300 > URL: https://issues.apache.org/jira/browse/HBASE-18300 > Project: HBase > Issue Type: New Feature > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > > We did an internal brainstorming to study the feasibility of this. Some of > our recent tests on SSDs like Optane shows that they are vastly faster in > randomreads and can act as effective caches. > In the current state we have a single tier of Bucket cache and the bucket > cache can either be offheap or configured to work with file mode. (The file > mode can have multiple files backing it). > So this model restricts us from using either the memory or the file and not > both. > With the advent of faster devices like Optane SSDs, NVMe based devices it is > better we try to utilize all those devices and try using them for the bucket > cache so that we can avoid the impact of slower devices where the actual data > resides on the HDFS data nodes. > Combined with this we can allow the user to configure the caching layer per > family/table so that one can effectively make use of the caching tiers. > Can upload a design doc here. Before that, would like to know the suggestions > here. Thoughts!!! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-19952) Find tests which are declared with wrong category
[ https://issues.apache.org/jira/browse/HBASE-19952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-19952: -- Fix Version/s: (was: 2.3.0) > Find tests which are declared with wrong category > - > > Key: HBASE-19952 > URL: https://issues.apache.org/jira/browse/HBASE-19952 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: category-report > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-19952) Find tests which are declared with wrong category
[ https://issues.apache.org/jira/browse/HBASE-19952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020849#comment-17020849 ] Michael Stack commented on HBASE-19952: --- Unscheduling from 2.3.0. Pull it back in if get a chance to work on it. > Find tests which are declared with wrong category > - > > Key: HBASE-19952 > URL: https://issues.apache.org/jira/browse/HBASE-19952 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: category-report > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-20099) Need to support per procedure killAndToggleBeforeStoreUpdate
[ https://issues.apache.org/jira/browse/HBASE-20099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-20099: -- Fix Version/s: (was: 2.3.0) > Need to support per procedure killAndToggleBeforeStoreUpdate > > > Key: HBASE-20099 > URL: https://issues.apache.org/jira/browse/HBASE-20099 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Major > > Currently there is procedure framework wide killAndToggleBeforeStoreUpdate. > This affects all procedures being executed by procedure framework including > sub-procedures. Procedure specific toggle can be added as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-20099) Need to support per procedure killAndToggleBeforeStoreUpdate
[ https://issues.apache.org/jira/browse/HBASE-20099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-20099. --- Resolution: Later No progress. Resolving improvement. > Need to support per procedure killAndToggleBeforeStoreUpdate > > > Key: HBASE-20099 > URL: https://issues.apache.org/jira/browse/HBASE-20099 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Major > > Currently there is procedure framework wide killAndToggleBeforeStoreUpdate. > This affects all procedures being executed by procedure framework including > sub-procedures. Procedure specific toggle can be added as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-20383) [AMv2] AssignmentManager: Failed transition XYZ is not OPEN
[
https://issues.apache.org/jira/browse/HBASE-20383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack resolved HBASE-20383.
---
Fix Version/s: (was: 2.3.0)
(was: 3.0.0)
Resolution: Won't Fix
Resolving as 'wont fix'. Issues prepping for 2.0.0 which is a while ago now. If
see this again, will open new one in new context.
> [AMv2] AssignmentManager: Failed transition XYZ is not OPEN
> ---
>
> Key: HBASE-20383
> URL: https://issues.apache.org/jira/browse/HBASE-20383
> Project: HBase
> Issue Type: Bug
> Components: amv2
>Reporter: Michael Stack
>Priority: Major
> Attachments: HBASE-20383.master.001.patch
>
>
> Seeing a bunch of this testing 2.0.0:
> {code}
> 2018-04-10 13:57:09,430 WARN
> [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16000]
> assignment.AssignmentManager: Failed transition
>
>
> org.apache.hadoop.hbase.client.DoNotRetryRegionException:
> 19a2cd6f88abae0036415ee1ea041c2e is not OPEN
> at
> org.apache.hadoop.hbase.master.procedure.AbstractStateMachineTableProcedure.checkOnline(AbstractStateMachineTableProcedure.java:193)
> at
> org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.(SplitTableRegionProcedure.java:112)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createSplitProcedure(AssignmentManager.java:769)
>
>
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.updateRegionSplitTransition(AssignmentManager.java:911)
>
>
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportRegionStateTransition(AssignmentManager.java:819)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionStateTransition(MasterRpcServices.java:1538)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:11093)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>
>
>at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> {code}
> Looks like report back from Master OK'ing a split to go ahead but the split
> is already running. Figure how to shut these down. They are noisy at least.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] saintstack commented on issue #1062: HBASE-23705 Add CellComparator to HFileContext
saintstack commented on issue #1062: HBASE-23705 Add CellComparator to HFileContext URL: https://github.com/apache/hbase/pull/1062#issuecomment-577042490 The checkstyle complaints are two. One is an old method that is > 150 limit. It is 242 currently (after I tried cutting it down). Leaving it. Not related. Other is an import order I finally figured. On unit tests, I tried them locally and they fail; a checkstyle change suggesting class be final broke mockito. Put up new patch. Thanks for +1 @anoopsjohn This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (HBASE-22978) Online slow response log
[
https://issues.apache.org/jira/browse/HBASE-22978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020704#comment-17020704
]
Viraj Jasani edited comment on HBASE-22978 at 1/22/20 6:29 AM:
---
Thanks [~stack]
Sure let me try for 2.3.0 soon after this lands to master. Will address
comments.
Also, 2.3.0 backport might not take more time compared to 1.6.0 backport, will
create backport subtask for 1.6.0.
Looking forward to all backports :)
was (Author: vjasani):
Thanks [~stack]
Sure let me try for 2.3.0 soon after we have enough reviews and confidence.
Will address comments.
Also, 2.3.0 backport might not take more time after trunk commit compared to
1.6.0 backport, will create backport subtask for 1.6.0.
Looking forward to all backports :)
> Online slow response log
>
>
> Key: HBASE-22978
> URL: https://issues.apache.org/jira/browse/HBASE-22978
> Project: HBase
> Issue Type: New Feature
> Components: Admin, Operability, regionserver, shell
>Affects Versions: 3.0.0, 2.3.0, 1.5.1
>Reporter: Andrew Kyle Purtell
>Assignee: Viraj Jasani
>Priority: Minor
> Fix For: 3.0.0, 2.3.0, 1.6.0
>
> Attachments: Screen Shot 2019-10-19 at 2.31.59 AM.png, Screen Shot
> 2019-10-19 at 2.32.54 AM.png, Screen Shot 2019-10-19 at 2.34.11 AM.png,
> Screen Shot 2019-10-19 at 2.36.14 AM.png
>
>
> Today when an individual RPC exceeds a configurable time bound we log a
> complaint by way of the logging subsystem. These log lines look like:
> {noformat}
> 2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
> (responseTooSlow):
> {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
> "starttimems":1567203007549,
> "responsesize":6819737,
> "method":"Scan",
> "param":"region { type: REGION_NAME value:
> \"tsdb,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
> "processingtimems":28646,
> "client":"10.253.196.215:41116",
> "queuetimems":22453,
> "class":"HRegionServer"}
> {noformat}
> Unfortunately we often truncate the request parameters, like in the above
> example. We do this because the human readable representation is verbose, the
> rate of too slow warnings may be high, and the combination of these things
> can overwhelm the log capture system. The truncation is unfortunate because
> it eliminates much of the utility of the warnings. For example, the region
> name, the start and end keys, and the filter hierarchy are all important
> clues for debugging performance problems caused by moderate to low
> selectivity queries or queries made at a high rate.
> We can maintain an in-memory ring buffer of requests that were judged to be
> too slow in addition to the responseTooSlow logging. The in-memory
> representation can be complete and compressed. A new admin API and shell
> command can provide access to the ring buffer for online performance
> debugging. A modest sizing of the ring buffer will prevent excessive memory
> utilization for a minor performance debugging feature by limiting the total
> number of retained records. There is some chance a high rate of requests will
> cause information on other interesting requests to be overwritten before it
> can be read. This is the nature of a ring buffer and an acceptable trade off.
> The write request types do not require us to retain all information submitted
> in the request. We don't need to retain all key-values in the mutation, which
> may be too large to comfortably retain. We only need a unique set of row
> keys, or even a min/max range, and total counts.
> The consumers of this information will be debugging tools. We can afford to
> apply fast compression to ring buffer entries (if codec support is
> available), something like snappy or zstandard, and decompress on the fly
> when servicing the retrieval API request. This will minimize the impact of
> retaining more information about slow requests than we do today.
> This proposal is for retention of request information only, the same
> information provided by responseTooSlow warnings. Total size of response
> serialization, possibly also total cell or row counts, should be sufficient
> to characterize the response.
> Optionally persist new entries added to the ring buffer into one or more
> files in HDFS in a write-behind manner. If the HDFS writer blocks or falls
> behind and we are unable to persist an entry before it is overwritten, that
> is fine. Response too slow logging is best effort. If we can detect this make
> a note of it in the log file. Provide a tool for parsing, dumping, filtering,
> and pretty printing the slow logs written to HDFS. The tool and the shell can
> share and reuse some utility
[jira] [Commented] (HBASE-21649) Complete Thrift2
[ https://issues.apache.org/jira/browse/HBASE-21649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020812#comment-17020812 ] Michael Stack commented on HBASE-21649: --- Unscheduling parent issue that is not making recent progress. > Complete Thrift2 > > > Key: HBASE-21649 > URL: https://issues.apache.org/jira/browse/HBASE-21649 > Project: HBase > Issue Type: Umbrella >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > > Thrift1 and Thrift2 coexists in our project for a very long time. > Functionality is more complete in thrift1 but its interface design is bad for > adding new features（so we have get(), getVer()，getVerTs,getRowWithColumns() > and so many other methods for a single get request, this is bad). Thrift2 has > a more clean interface and structure definition, making our user more easy to > use. But, it has not been updated for a long time, lacking of DDL method is a > major weakness. > I think we should complete Thrift2 and supersede Thrift1, making Thrift2 as > the standard multi language definition. This is a umbrella issue to make it > happen. > The plan would be: > 1. Complete the DDL interface of thrift2 > 2. Making thrift2 server be able to handle thrift1 requests, user don't have > to choose which thrift server they need to start > 3. Deprecate thrift1(need to discuss and only go to 3.x) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-21649) Complete Thrift2
[ https://issues.apache.org/jira/browse/HBASE-21649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-21649: -- Fix Version/s: (was: 2.3.0) (was: 3.0.0) > Complete Thrift2 > > > Key: HBASE-21649 > URL: https://issues.apache.org/jira/browse/HBASE-21649 > Project: HBase > Issue Type: Umbrella >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > > Thrift1 and Thrift2 coexists in our project for a very long time. > Functionality is more complete in thrift1 but its interface design is bad for > adding new features（so we have get(), getVer()，getVerTs,getRowWithColumns() > and so many other methods for a single get request, this is bad). Thrift2 has > a more clean interface and structure definition, making our user more easy to > use. But, it has not been updated for a long time, lacking of DDL method is a > major weakness. > I think we should complete Thrift2 and supersede Thrift1, making Thrift2 as > the standard multi language definition. This is a umbrella issue to make it > happen. > The plan would be: > 1. Complete the DDL interface of thrift2 > 2. Making thrift2 server be able to handle thrift1 requests, user don't have > to choose which thrift server they need to start > 3. Deprecate thrift1(need to discuss and only go to 3.x) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-17267) Add OffheapMemoryTuner
[ https://issues.apache.org/jira/browse/HBASE-17267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020811#comment-17020811 ] Michael Stack commented on HBASE-17267: --- Unscheduled. No work being done here on this feature. > Add OffheapMemoryTuner > -- > > Key: HBASE-17267 > URL: https://issues.apache.org/jira/browse/HBASE-17267 > Project: HBase > Issue Type: New Feature > Components: regionserver >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > > This JIRA is aimed at tuning the offheap memory. It is not straight forward > as we should not cross the available Direct_memory configured. > Should include BC and offheap memstore configs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-17267) Add OffheapMemoryTuner
[ https://issues.apache.org/jira/browse/HBASE-17267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-17267: -- Fix Version/s: (was: 2.3.0) > Add OffheapMemoryTuner > -- > > Key: HBASE-17267 > URL: https://issues.apache.org/jira/browse/HBASE-17267 > Project: HBase > Issue Type: New Feature > Components: regionserver >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > > This JIRA is aimed at tuning the offheap memory. It is not straight forward > as we should not cross the available Direct_memory configured. > Should include BC and offheap memstore configs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-20249) Investigate behavior of hbase.client.operation.timeout
[ https://issues.apache.org/jira/browse/HBASE-20249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-20249. --- Fix Version/s: (was: 2.3.0) (was: 3.0.0) Resolution: Incomplete Resovling as incomplete. > Investigate behavior of hbase.client.operation.timeout > -- > > Key: HBASE-20249 > URL: https://issues.apache.org/jira/browse/HBASE-20249 > Project: HBase > Issue Type: Bug >Reporter: Apekshit Sharma >Priority: Critical > > hbase.client.operation.timeout should be hard limit on client operations, and > independent of number of retires. > Previous discussions here: > https://issues.apache.org/jira/browse/HBASE-17449?focusedCommentId=16390085=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16390085 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-16328) Reimplement web UI fixes without license problems
[ https://issues.apache.org/jira/browse/HBASE-16328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020808#comment-17020808 ] Michael Stack commented on HBASE-16328: --- Important but no assignee. Unscheduling. > Reimplement web UI fixes without license problems > - > > Key: HBASE-16328 > URL: https://issues.apache.org/jira/browse/HBASE-16328 > Project: HBase > Issue Type: Improvement > Components: dependencies, security, UI >Affects Versions: 1.3.0, 1.4.0, 1.1.6, 1.2.3, 2.0.0 >Reporter: Sean Busbey >Priority: Critical > > After HBASE-16317 we're missing some good improvements in our web ui. > This jira is to track re-implementing the reverted commits after either > # getting the ESAPI project to stop using cat-x dependencies > # reimplementing the functionality without the ESAPI project > For review, the category-x list is here: > https://www.apache.org/legal/resolved#category-x -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-16328) Reimplement web UI fixes without license problems
[ https://issues.apache.org/jira/browse/HBASE-16328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-16328: -- Fix Version/s: (was: 1.7.0) (was: 2.3.0) (was: 3.0.0) > Reimplement web UI fixes without license problems > - > > Key: HBASE-16328 > URL: https://issues.apache.org/jira/browse/HBASE-16328 > Project: HBase > Issue Type: Improvement > Components: dependencies, security, UI >Affects Versions: 1.3.0, 1.4.0, 1.1.6, 1.2.3, 2.0.0 >Reporter: Sean Busbey >Priority: Critical > > After HBASE-16317 we're missing some good improvements in our web ui. > This jira is to track re-implementing the reverted commits after either > # getting the ESAPI project to stop using cat-x dependencies > # reimplementing the functionality without the ESAPI project > For review, the category-x list is here: > https://www.apache.org/legal/resolved#category-x -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-18785) Blocker doc issues for 2.0.0
[ https://issues.apache.org/jira/browse/HBASE-18785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-18785. --- Resolution: Incomplete Resolving as incomplete. hbase 2 went out w/o the remaining linked issues. > Blocker doc issues for 2.0.0 > > > Key: HBASE-18785 > URL: https://issues.apache.org/jira/browse/HBASE-18785 > Project: HBase > Issue Type: Task >Reporter: Michael Stack >Priority: Critical > Fix For: 3.0.0, 2.3.0 > > > Hang blocker documentation issues off this one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23069) periodic dependency bump for Sep 2019
[ https://issues.apache.org/jira/browse/HBASE-23069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-23069: -- Release Note: caffeine: 2.6.2 => 2.8.1 commons-codec: 1.10 => 1.13 commons-io: 2.5 => 2.6 disrupter: 3.3.6 => 3.4.2 httpclient: 4.5.3 => 4.5.11 httpcore: 4.4.6 => 4.4.13 jackson: 2.9.10 => 2.10.1 jackson.databind: 2.9.10.1 => 2.10.1 jetty: 9.3.27.v20190418 => 9.3.28.v20191105 jruby: 9.1.17.0 => 9.2.9.0 protobuf.plugin: 0.5.0 => 0.6.1 zookeeper: 3.4.10 => 3.4.14 slf4j: 1.7.25 => 1.7.30 rat: 0.12 => 0.13 asciidoctor: 1.5.5 => 1.5.8.1 asciidoctor.pdf: 1.5.0-alpha.15 => 1.5.0-rc.2 error-prone: 2.3.3 => 2.3.4 > periodic dependency bump for Sep 2019 > - > > Key: HBASE-23069 > URL: https://issues.apache.org/jira/browse/HBASE-23069 > Project: HBase > Issue Type: Improvement > Components: dependencies, hbase-thirdparty >Reporter: Sean Busbey >Priority: Critical > Fix For: 3.0.0, 2.3.0, 1.6.0 > > > we should do a pass to see if there are any dependencies we can bump. (also > follow-on we should automate this check) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack opened a new pull request #1082: HBASE-23069 periodic dependency bump for Sep 2019
saintstack opened a new pull request #1082: HBASE-23069 periodic dependency bump for Sep 2019 URL: https://github.com/apache/hbase/pull/1082 caffeine: 2.6.2 => 2.8.1 commons-codec: 1.10 => 1.13 commons-io: 2.5 => 2.6 disrupter: 3.3.6 => 3.4.2 httpclient: 4.5.3 => 4.5.11 httpcore: 4.4.6 => 4.4.13 jackson: 2.9.10 => 2.10.1 jackson.databind: 2.9.10.1 => 2.10.1 jetty: 9.3.27.v20190418 => 9.3.28.v20191105 jruby: 9.1.17.0 => 9.2.9.0 protobuf.plugin: 0.5.0 => 0.6.1 zookeeper: 3.4.10 => 3.4.14 slf4j: 1.7.25 => 1.7.30 rat: 0.12 => 0.13 asciidoctor: 1.5.5 => 1.5.8.1 asciidoctor.pdf: 1.5.0-alpha.15 => 1.5.0-rc.2 error-prone: 2.3.3 => 2.3.4 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-23716) [hbase-thirdparty] Make release 3.2.0
[ https://issues.apache.org/jira/browse/HBASE-23716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-23716: -- Fix Version/s: thirdparty-3.2.0 > [hbase-thirdparty] Make release 3.2.0 > - > > Key: HBASE-23716 > URL: https://issues.apache.org/jira/browse/HBASE-23716 > Project: HBase > Issue Type: Bug > Components: thirdparty >Affects Versions: 3.0.0, 2.3.0 >Reporter: Michael Stack >Priority: Major > Fix For: thirdparty-3.2.0 > > > Update the dependencies in hbase-thirdparty. In particular, pb goes from 3.7 > to 3.11 with a few perf improvements. > Would like to update the hbase-thirdparty to include in the coming > hbase-2.3.0. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23717) [hbase-thirdparty] Change pom version from 3.1.2-SNAPSHOT to 3.2.0-SNAPSHOT
[ https://issues.apache.org/jira/browse/HBASE-23717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-23717: -- Fix Version/s: thirdparty-3.2.0 > [hbase-thirdparty] Change pom version from 3.1.2-SNAPSHOT to 3.2.0-SNAPSHOT > --- > > Key: HBASE-23717 > URL: https://issues.apache.org/jira/browse/HBASE-23717 > Project: HBase > Issue Type: Sub-task > Components: thirdparty >Affects Versions: thirdparty-3.2.0 >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: thirdparty-3.2.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase-thirdparty] saintstack opened a new pull request #10: HBASE-23718 [hbase-thirdparty] Update libs; pb from 3.9 to 3.11, etc.
saintstack opened a new pull request #10: HBASE-23718 [hbase-thirdparty] Update libs; pb from 3.9 to 3.11, etc. URL: https://github.com/apache/hbase-thirdparty/pull/10 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-23718) [hbase-thirdparty] Update libs; pb from 3.9 to 3.11, etc.
[ https://issues.apache.org/jira/browse/HBASE-23718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-23718: -- Release Note: gson: 2.8.5 => 2.8.6 guava: 28.1-jre => 28.2-jre error_prone: 2.3.3 => 2.3.4 netty: 4.1.42.Final => 4.1.44.Final protobuf: 3.9.2 => 3.11.1 maven-assembly-plugin: 3.1.1 => 3.2.0 Summary: [hbase-thirdparty] Update libs; pb from 3.9 to 3.11, etc. (was: [hbase-thirdparty] Update libs; pb from 2.7 to 2.11, etc.) > [hbase-thirdparty] Update libs; pb from 3.9 to 3.11, etc. > - > > Key: HBASE-23718 > URL: https://issues.apache.org/jira/browse/HBASE-23718 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23718) [hbase-thirdparty] Update libs; pb from 3.9 to 3.11, etc.
[ https://issues.apache.org/jira/browse/HBASE-23718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-23718: -- Fix Version/s: thirdparty-3.2.0 > [hbase-thirdparty] Update libs; pb from 3.9 to 3.11, etc. > - > > Key: HBASE-23718 > URL: https://issues.apache.org/jira/browse/HBASE-23718 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Priority: Major > Fix For: thirdparty-3.2.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23718) [hbase-thirdparty] Update libs; pb from 2.7 to 2.11, etc.
Michael Stack created HBASE-23718: - Summary: [hbase-thirdparty] Update libs; pb from 2.7 to 2.11, etc. Key: HBASE-23718 URL: https://issues.apache.org/jira/browse/HBASE-23718 Project: HBase Issue Type: Sub-task Reporter: Michael Stack -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23717) [hbase-thirdparty] Change pom version from 3.1.2-SNAPSHOT to 3.2.0-SNAPSHOT
[ https://issues.apache.org/jira/browse/HBASE-23717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-23717: -- Summary: [hbase-thirdparty] Change pom version from 3.1.2-SNAPSHOT to 3.2.0-SNAPSHOT (was: [hbase-thirdparty] Change pomp version from 3.1.2-SNAPSHOT to 3.2.0-SNAPSHOT) > [hbase-thirdparty] Change pom version from 3.1.2-SNAPSHOT to 3.2.0-SNAPSHOT > --- > > Key: HBASE-23717 > URL: https://issues.apache.org/jira/browse/HBASE-23717 > Project: HBase > Issue Type: Sub-task > Components: thirdparty >Affects Versions: thirdparty-3.2.0 >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23716) [hbase-thirdparty] Make release 3.2.0
[ https://issues.apache.org/jira/browse/HBASE-23716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-23716: -- Description: Update the dependencies in hbase-thirdparty. In particular, pb goes from 3.7 to 3.11 with a few perf improvements. Would like to update the hbase-thirdparty to include in the coming hbase-2.3.0. was:Update the dependencies in hbase-thirdparty. In particular, pb goes from 3.7 to 3.11 with a few perf improvements. > [hbase-thirdparty] Make release 3.2.0 > - > > Key: HBASE-23716 > URL: https://issues.apache.org/jira/browse/HBASE-23716 > Project: HBase > Issue Type: Bug > Components: thirdparty >Affects Versions: 3.0.0, 2.3.0 >Reporter: Michael Stack >Priority: Major > > Update the dependencies in hbase-thirdparty. In particular, pb goes from 3.7 > to 3.11 with a few perf improvements. > Would like to update the hbase-thirdparty to include in the coming > hbase-2.3.0. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase-thirdparty] saintstack opened a new pull request #9: HBASE-23717 [hbase-thirdparty] Change pomp version from 3.1.2-SNAPSHO…
saintstack opened a new pull request #9: HBASE-23717 [hbase-thirdparty] Change pomp version from 3.1.2-SNAPSHO… URL: https://github.com/apache/hbase-thirdparty/pull/9 …T to 3.2.0-SNAPSHOT This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (HBASE-23717) [hbase-thirdparty] Change pomp version from 3.1.2-SNAPSHOT to 3.2.0-SNAPSHOT
Michael Stack created HBASE-23717: - Summary: [hbase-thirdparty] Change pomp version from 3.1.2-SNAPSHOT to 3.2.0-SNAPSHOT Key: HBASE-23717 URL: https://issues.apache.org/jira/browse/HBASE-23717 Project: HBase Issue Type: Sub-task Components: thirdparty Affects Versions: thirdparty-3.2.0 Reporter: Michael Stack Assignee: Michael Stack -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23716) [hbase-thirdparty] Make release 3.2.0
Michael Stack created HBASE-23716: - Summary: [hbase-thirdparty] Make release 3.2.0 Key: HBASE-23716 URL: https://issues.apache.org/jira/browse/HBASE-23716 Project: HBase Issue Type: Bug Components: thirdparty Affects Versions: 3.0.0, 2.3.0 Reporter: Michael Stack Update the dependencies in hbase-thirdparty. In particular, pb goes from 3.7 to 3.11 with a few perf improvements. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] gjacoby126 commented on issue #1079: HBASE-23711 - Add test for MinVersions and KeepDeletedCells TTL
gjacoby126 commented on issue #1079: HBASE-23711 - Add test for MinVersions and KeepDeletedCells TTL URL: https://github.com/apache/hbase/pull/1079#issuecomment-576996302 Pushed up a fix to the one-line indent problem that checkstyle was complaining about. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-20188. --- Resolution: Fixed Resolving umbrella whose subtasks are done. > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: Michael Stack >Priority: Blocker > Fix For: 3.0.0, 2.3.0 > > Attachments: CAM-CONFIG-V01.patch, HBASE-20188-xac.sh, > HBASE-20188.sh, HBase 2.0 performance evaluation - 8GB(1).pdf, HBase 2.0 > performance evaluation - 8GB.pdf, HBase 2.0 performance evaluation - Basic vs > None_ system settings.pdf, HBase 2.0 performance evaluation - throughput > SSD_HDD.pdf, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, hbase-env.sh, hbase-site.xml, > hbase-site.xml, hits.png, hits_with_fp_scheduler.png, > lock.127.workloadc.20180402T200918Z.svg, > lock.2.memsize2.c.20180403T160257Z.svg, perregion.png, run_ycsb.sh, > total.png, tree.txt, workloadx, workloadx > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-21333) [amv2] large cluster startup is slow
[ https://issues.apache.org/jira/browse/HBASE-21333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-21333: -- Parent: (was: HBASE-20188) Issue Type: Bug (was: Sub-task) > [amv2] large cluster startup is slow > > > Key: HBASE-21333 > URL: https://issues.apache.org/jira/browse/HBASE-21333 > Project: HBase > Issue Type: Bug > Components: amv2 >Affects Versions: 2.1.0 >Reporter: Michael Stack >Priority: Major > Attachments: 2.1.1.129578.alloc.svg, 2.1.1.129578.cpu.svg, > 2.1.1.129578.lock.svg > > > Testing startup of cluster with 500+ nodes and .5M regions takes a few hours. > This is a 2.1.x cluster with batching disabled. > Looking at what the Master is doing, its mostly just parsing regionserver > reports. > Stats to follow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-21333) [amv2] large cluster startup is slow
[ https://issues.apache.org/jira/browse/HBASE-21333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020753#comment-17020753 ] Michael Stack commented on HBASE-21333: --- Unscheduling old issue that has had no work done on it of late. > [amv2] large cluster startup is slow > > > Key: HBASE-21333 > URL: https://issues.apache.org/jira/browse/HBASE-21333 > Project: HBase > Issue Type: Bug > Components: amv2 >Affects Versions: 2.1.0 >Reporter: Michael Stack >Priority: Major > Attachments: 2.1.1.129578.alloc.svg, 2.1.1.129578.cpu.svg, > 2.1.1.129578.lock.svg > > > Testing startup of cluster with 500+ nodes and .5M regions takes a few hours. > This is a 2.1.x cluster with batching disabled. > Looking at what the Master is doing, its mostly just parsing regionserver > reports. > Stats to follow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-20786) Table create with thousands of regions takes too long
[ https://issues.apache.org/jira/browse/HBASE-20786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020752#comment-17020752 ] Michael Stack commented on HBASE-20786: --- Moving issue out from under parent. It has had no work done on it of late and assigns seem fast now, faster than described here. > Table create with thousands of regions takes too long > - > > Key: HBASE-20786 > URL: https://issues.apache.org/jira/browse/HBASE-20786 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: Michael Stack >Priority: Major > Attachments: HBASE-20786.master.001.patch > > > Internal testing has create of a table with 33k regions taking 18 minutes. > Let me provide more info below. We have an executor with default ten threads > handling the creation of the regions in HDFS which helps distribute out the > load but its not enough. This cluster had >600 servers. Let me add detail. > Need to spend some time on speeding up create/assigns. Made this an umbrella > issue so can pick off pieces of the problem as subtasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-20786) Table create with thousands of regions takes too long
[ https://issues.apache.org/jira/browse/HBASE-20786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-20786: -- Parent: (was: HBASE-20188) Issue Type: Bug (was: Sub-task) > Table create with thousands of regions takes too long > - > > Key: HBASE-20786 > URL: https://issues.apache.org/jira/browse/HBASE-20786 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: Michael Stack >Priority: Major > Attachments: HBASE-20786.master.001.patch > > > Internal testing has create of a table with 33k regions taking 18 minutes. > Let me provide more info below. We have an executor with default ten threads > handling the creation of the regions in HDFS which helps distribute out the > load but its not enough. This cluster had >600 servers. Let me add detail. > Need to spend some time on speeding up create/assigns. Made this an umbrella > issue so can pick off pieces of the problem as subtasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-20711) Save on a Cell iteration when writing
[ https://issues.apache.org/jira/browse/HBASE-20711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020751#comment-17020751 ] Michael Stack commented on HBASE-20711: --- Move old issue that still seems legit but that is having no work being done out of its parent issue. > Save on a Cell iteration when writing > - > > Key: HBASE-20711 > URL: https://issues.apache.org/jira/browse/HBASE-20711 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Minor > Attachments: HBASE-20711.branch-2.0.001.patch > > > This is a minor savings. We were doing a spin through all Cells on receipt > just to check their size when subsequently, we were doing an iteration of all > Cells to insert. It manifest as a little spike in perf output. This change > removes the purposed spin through Cells and just does size check as part of > the general Cell insert (perf spike no longer shows but the cost of the size > check still remains). > There is also a minor bug fix where on receipt we were using the Puts row > rather than the Cells row; client may have succeeded in submitting a Cell > that disagreed with the hosting Mutation and it would have been written as > something else altogether -- with the Puts row -- rather than being rejected. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-20711) Save on a Cell iteration when writing
[ https://issues.apache.org/jira/browse/HBASE-20711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-20711: -- Parent: (was: HBASE-20188) Issue Type: Bug (was: Sub-task) > Save on a Cell iteration when writing > - > > Key: HBASE-20711 > URL: https://issues.apache.org/jira/browse/HBASE-20711 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Minor > Attachments: HBASE-20711.branch-2.0.001.patch > > > This is a minor savings. We were doing a spin through all Cells on receipt > just to check their size when subsequently, we were doing an iteration of all > Cells to insert. It manifest as a little spike in perf output. This change > removes the purposed spin through Cells and just does size check as part of > the general Cell insert (perf spike no longer shows but the cost of the size > check still remains). > There is also a minor bug fix where on receipt we were using the Puts row > rather than the Cells row; client may have succeeded in submitting a Cell > that disagreed with the hosting Mutation and it would have been written as > something else altogether -- with the Puts row -- rather than being rejected. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-20673) Reduce the number of Cell implementations; the profusion is distracting to users and JIT
[ https://issues.apache.org/jira/browse/HBASE-20673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020749#comment-17020749 ] Michael Stack commented on HBASE-20673: --- Converting subtask into issue. Still relevant but no work being done so unscheduling. > Reduce the number of Cell implementations; the profusion is distracting to > users and JIT > > > Key: HBASE-20673 > URL: https://issues.apache.org/jira/browse/HBASE-20673 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Attachments: 0001-current.patch, 0001-current.patch, hits.20673.png > > > We have a wild blossom of Cell implementations in hbase. Purge the bulk of > them. Make it so we do one type well. JIT gets confused if it has an abstract > or an interface and then the instantiated classes are myriad (megamorphic). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-20673) Reduce the number of Cell implementations; the profusion is distracting to users and JIT
[ https://issues.apache.org/jira/browse/HBASE-20673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-20673: -- Parent: (was: HBASE-20188) Issue Type: Bug (was: Sub-task) > Reduce the number of Cell implementations; the profusion is distracting to > users and JIT > > > Key: HBASE-20673 > URL: https://issues.apache.org/jira/browse/HBASE-20673 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Attachments: 0001-current.patch, 0001-current.patch, hits.20673.png > > > We have a wild blossom of Cell implementations in hbase. Purge the bulk of > them. Make it so we do one type well. JIT gets confused if it has an abstract > or an interface and then the instantiated classes are myriad (megamorphic). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-20455) [IHC] Workloadx
[
https://issues.apache.org/jira/browse/HBASE-20455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack resolved HBASE-20455.
---
Resolution: Incomplete
Resolving as incomplete.
> [IHC] Workloadx
> ---
>
> Key: HBASE-20455
> URL: https://issues.apache.org/jira/browse/HBASE-20455
> Project: HBase
> Issue Type: Sub-task
>Reporter: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
>
> I tried workloadx from parent issue, the ycsb workload that is meant to make
> IHC shine. I'm doing something wrong. I tried just plugging it in and
> doing this:
> {code}
> ycsb run hbase12 -P /home/stack/ycsb/workloads/workloadx -p table=ycsb
> -threads 48 -cp /home/stack/conf_hbase -p columnfamily=family -p
> clientbuffering=true -p writebuffersize=2097152 -s -p maxexecutiontime=1200
> -jvm-args=-Xmx8192
> m -Djava.security.egd=file:/dev/./urandom -p recordcount=2500 -p
> operationcount=2500 -p
> exportfile=logs/ycsb-workloadx-measurements-ve0524-20180419T02:58:14Z.json -p
> exporter=com.yahoo.ycsb.measurements.exporter.JSONArrayMeasurementsExporter
> {code}
> It completes in a minute and a half. Claims stuff like this for throughput:
> 236825 ops/second. In this case I have in-memory-compaction set to NONE. I
> was going to compare before and after.
> [~eshcar]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #1080: HBASE-23709 Unwrap the real user to properly dispatch proxy-user auth'n
Apache-HBase commented on issue #1080: HBASE-23709 Unwrap the real user to properly dispatch proxy-user auth'n URL: https://github.com/apache/hbase/pull/1080#issuecomment-576992359 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 45s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 36s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 6m 30s | master passed | | +1 :green_heart: | compile | 1m 40s | master passed | | +1 :green_heart: | checkstyle | 1m 30s | master passed | | +1 :green_heart: | shadedjars | 5m 38s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 23s | master passed | | +0 :ok: | spotbugs | 1m 50s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 4m 1s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 6m 14s | the patch passed | | +1 :green_heart: | compile | 1m 41s | the patch passed | | +1 :green_heart: | javac | 1m 41s | the patch passed | | +1 :green_heart: | checkstyle | 1m 35s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedjars | 5m 43s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 19m 50s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 1m 24s | the patch passed | | +1 :green_heart: | findbugs | 4m 40s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 9s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 2m 44s | hbase-thrift in the patch passed. | | +1 :green_heart: | unit | 4m 38s | hbase-rest in the patch passed. | | +1 :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. | | | | 83m 10s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1080/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1080 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile xml | | uname | Linux 7c50949b3453 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1080/out/precommit/personality/provided.sh | | git revision | master / ba3463d9de | | Default Java | 1.8.0_181 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1080/2/testReport/ | | Max. process+thread count | 2042 (vs. ulimit of 1) | | modules | C: hbase-client hbase-thrift hbase-rest U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1080/2/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-20236) [locking] Write-time worst offenders
[
https://issues.apache.org/jira/browse/HBASE-20236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack updated HBASE-20236:
--
Resolution: Incomplete
Status: Resolved (was: Patch Available)
Resolving as incomplete. Needs more work
> [locking] Write-time worst offenders
>
>
> Key: HBASE-20236
> URL: https://issues.apache.org/jira/browse/HBASE-20236
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Affects Versions: 2.0.0-beta-2
>Reporter: Michael Stack
>Priority: Major
> Attachments: 2.0623.111354.lock.svg, HBASE-20236.master.001.patch,
> no_semaphore_vs_semaphore.png
>
>
> Messing w/ my new toy, here are worst offenders locking; they must be bad if
> they show up in this sampling profiler:
> {code}
> 7 Total: 769321884622 (99.24%) samples: 2965
> 8 [ 0] java.util.concurrent.Semaphore$NonfairSync
> 9 [ 1] sun.misc.Unsafe.park
> 10 [ 2] java.util.concurrent.locks.LockSupport.park
> 11 [ 3]
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt
> 12 [ 4]
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly
> 13 [ 5]
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly
> 14 [ 6] java.util.concurrent.Semaphore.acquire
> 15 [ 7]
> org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner
> 16 [ 8] org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run
> 17
> 18 Total: 4284274263 (0.55%) samples: 23543
> 19 [ 0] org.apache.hadoop.hbase.regionserver.MutableSegment
> 20 [ 1] org.apache.hadoop.hbase.ByteBufferKeyValue.getSequenceId
> 21 [ 2] org.apache.hadoop.hbase.regionserver.Segment.updateMetaInfo
> 22 [ 3] org.apache.hadoop.hbase.regionserver.Segment.internalAdd
> 23 [ 4] org.apache.hadoop.hbase.regionserver.MutableSegment.add
> 24 [ 5] org.apache.hadoop.hbase.regionserver.AbstractMemStore.internalAdd
> 25 [ 6] org.apache.hadoop.hbase.regionserver.AbstractMemStore.add
> 26 [ 7] org.apache.hadoop.hbase.regionserver.AbstractMemStore.add
> 27 [ 8] org.apache.hadoop.hbase.regionserver.HStore.add
> 28 [ 9] org.apache.hadoop.hbase.regionserver.HRegion.applyToMemStore
> 29 [10] org.apache.hadoop.hbase.regionserver.HRegion.access$600
> 30 [11]
> org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.applyFamilyMapToMemStore
> 31 [12]
> org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.lambda$writeMiniBatchOperationsToMemStore$0
> 32 [13]
> org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation$$Lambda$442.1445825895.visit
> 33 [14]
> org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.visitBatchOperations
> 34 [15]
> org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.writeMiniBatchOperationsToMemStore
> 35 [16]
> org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.writeMiniBatchOperationsToMemStore
> 36 [17] org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate
> 37 [18] org.apache.hadoop.hbase.regionserver.HRegion.batchMutate
> 38 [19] org.apache.hadoop.hbase.regionserver.HRegion.batchMutate
> 39 [20] org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp
> 40 [21]
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp
> 41 [22]
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation
> 42 [23] org.apache.hadoop.hbase.regionserver.RSRpcServices.multi
> 43 [24]
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod
> 44 [25] org.apache.hadoop.hbase.ipc.RpcServer.call
> 45 [26] org.apache.hadoop.hbase.ipc.CallRunner.run
> 46 [27] org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run
> 47 [28] org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run
> 48
> 49 Total: 717708856 (0.09%) samples: 214
> 50 [ 0] java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync
> 51 [ 1] sun.misc.Unsafe.park
> 52 [ 2] java.util.concurrent.locks.LockSupport.park
> 53 [ 3]
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt
> 54 [ 4] java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued
> 55 [ 5] java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire
> 56 [ 6] java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock
> 57 [ 7] org.apache.hadoop.hbase.regionserver.HRegion.blockUpdates
> 58 [ 8]
> org.apache.hadoop.hbase.regionserver.RegionServicesForStores.blockUpdates
> 59 [ 9]
> org.apache.hadoop.hbase.regionserver.CompactingMemStore.flushInMemory
> 60 [10]
> org.apache.hadoop.hbase.regionserver.CompactingMemStore$InMemoryFlushRunnable.run
> 61 [11] java.util.concurrent.ThreadPoolExecutor.runWorker
> 62
[jira] [Commented] (HBASE-20234) Expose in-memory compaction metrics
[
https://issues.apache.org/jira/browse/HBASE-20234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020746#comment-17020746
]
Michael Stack commented on HBASE-20234:
---
Unscheduling a subtask since gone stale/unaddressed.
> Expose in-memory compaction metrics
> ---
>
> Key: HBASE-20234
> URL: https://issues.apache.org/jira/browse/HBASE-20234
> Project: HBase
> Issue Type: Bug
>Reporter: Michael Stack
>Assignee: Anastasia Braginsky
>Priority: Major
>
> Hard to glean insight from how well in-memory compaction is doing currently.
> It dumps stats into the logs but better if they were available to a
> dashboard. This issue is about exposing a couple of helpful counts. There are
> already by-region metrics. We can add a few for in-memory compaction (Help me
> out [~anastas]... what counts would be best to expose).
> Flush related metrics include
> {code}
> Namespace_default_table_tsdb-tree_region_cfbf23e7330a1a2bbde031f9583d3415_metric_flushesQueuedCount:
> {
> description: "Number flushes requested/queued for this region",
> value: 0
> {
> description: "The number of cells flushed to disk",
> value: 0
> },
> {
> description: "The total amount of data flushed to disk, in bytes",
> value: 0
> },
> ...
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-20234) Expose in-memory compaction metrics
[
https://issues.apache.org/jira/browse/HBASE-20234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack updated HBASE-20234:
--
Parent: (was: HBASE-20188)
Issue Type: Bug (was: Sub-task)
> Expose in-memory compaction metrics
> ---
>
> Key: HBASE-20234
> URL: https://issues.apache.org/jira/browse/HBASE-20234
> Project: HBase
> Issue Type: Bug
>Reporter: Michael Stack
>Assignee: Anastasia Braginsky
>Priority: Major
>
> Hard to glean insight from how well in-memory compaction is doing currently.
> It dumps stats into the logs but better if they were available to a
> dashboard. This issue is about exposing a couple of helpful counts. There are
> already by-region metrics. We can add a few for in-memory compaction (Help me
> out [~anastas]... what counts would be best to expose).
> Flush related metrics include
> {code}
> Namespace_default_table_tsdb-tree_region_cfbf23e7330a1a2bbde031f9583d3415_metric_flushesQueuedCount:
> {
> description: "Number flushes requested/queued for this region",
> value: 0
> {
> description: "The number of cells flushed to disk",
> value: 0
> },
> {
> description: "The total amount of data flushed to disk, in bytes",
> value: 0
> },
> ...
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-13428) [DOC] Migration to hbase-2.0.0
[ https://issues.apache.org/jira/browse/HBASE-13428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-13428. --- Resolution: Fixed Resolving umbrella issue as all subtasks done. > [DOC] Migration to hbase-2.0.0 > -- > > Key: HBASE-13428 > URL: https://issues.apache.org/jira/browse/HBASE-13428 > Project: HBase > Issue Type: Umbrella > Components: migration >Reporter: Michael Stack >Priority: Blocker > Fix For: 3.0.0, 2.3.0 > > > Opening a 2.0 umbrella migration issue. Lets hang off this one any tools and > expectations migrating from 1.0 (or earlier) to 2.0. So far there are none > that I know of though there is an expectation in HBASE-13373 that hfiles are > at least major version 2 and minor version 3. Lets list all such > expectations, etc., here. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-20516) Offheap read-path needs more detail
[ https://issues.apache.org/jira/browse/HBASE-20516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-20516: -- Fix Version/s: (was: 3.0.0) > Offheap read-path needs more detail > --- > > Key: HBASE-20516 > URL: https://issues.apache.org/jira/browse/HBASE-20516 > Project: HBase > Issue Type: Task > Components: Offheaping >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Attachments: 20516.initial.patch.txt, 20516.initial.patch.txt > > > Needs notes on what an operator should look for to see that all is on and > what to monitor in a running cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-20281) [DOC] upgrade section needs an explanation of changes to Bucket Cache
[
https://issues.apache.org/jira/browse/HBASE-20281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack updated HBASE-20281:
--
Fix Version/s: (was: 3.0.0)
> [DOC] upgrade section needs an explanation of changes to Bucket Cache
> -
>
> Key: HBASE-20281
> URL: https://issues.apache.org/jira/browse/HBASE-20281
> Project: HBase
> Issue Type: Bug
> Components: BucketCache, documentation
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Priority: Critical
>
> the first pass version of the upgrade docs has a brief note about the bucket
> cache changing:
> {quote}
> * hbase.bucketcache.combinedcache.enabled
> * hbase.bucketcache.ioengine no longer supports the 'heap' value.
> {quote}
> But the changes are substantial and warrant a section of their own under the
> "Changes of Note!" banner.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-20281) [DOC] upgrade section needs an explanation of changes to Bucket Cache
[
https://issues.apache.org/jira/browse/HBASE-20281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020742#comment-17020742
]
Michael Stack commented on HBASE-20281:
---
Unscheduled unaddressed subtask (made into an issue). If you get a chance
[~anoop.hbase] ? Thanks.
> [DOC] upgrade section needs an explanation of changes to Bucket Cache
> -
>
> Key: HBASE-20281
> URL: https://issues.apache.org/jira/browse/HBASE-20281
> Project: HBase
> Issue Type: Bug
> Components: BucketCache, documentation
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Priority: Critical
>
> the first pass version of the upgrade docs has a brief note about the bucket
> cache changing:
> {quote}
> * hbase.bucketcache.combinedcache.enabled
> * hbase.bucketcache.ioengine no longer supports the 'heap' value.
> {quote}
> But the changes are substantial and warrant a section of their own under the
> "Changes of Note!" banner.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-20281) [DOC] upgrade section needs an explanation of changes to Bucket Cache
[
https://issues.apache.org/jira/browse/HBASE-20281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack updated HBASE-20281:
--
Fix Version/s: (was: 2.3.0)
> [DOC] upgrade section needs an explanation of changes to Bucket Cache
> -
>
> Key: HBASE-20281
> URL: https://issues.apache.org/jira/browse/HBASE-20281
> Project: HBase
> Issue Type: Bug
> Components: BucketCache, documentation
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0
>
>
> the first pass version of the upgrade docs has a brief note about the bucket
> cache changing:
> {quote}
> * hbase.bucketcache.combinedcache.enabled
> * hbase.bucketcache.ioengine no longer supports the 'heap' value.
> {quote}
> But the changes are substantial and warrant a section of their own under the
> "Changes of Note!" banner.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-20281) [DOC] upgrade section needs an explanation of changes to Bucket Cache
[
https://issues.apache.org/jira/browse/HBASE-20281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack updated HBASE-20281:
--
Parent: (was: HBASE-13428)
Issue Type: Bug (was: Sub-task)
> [DOC] upgrade section needs an explanation of changes to Bucket Cache
> -
>
> Key: HBASE-20281
> URL: https://issues.apache.org/jira/browse/HBASE-20281
> Project: HBase
> Issue Type: Bug
> Components: BucketCache, documentation
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0, 2.3.0
>
>
> the first pass version of the upgrade docs has a brief note about the bucket
> cache changing:
> {quote}
> * hbase.bucketcache.combinedcache.enabled
> * hbase.bucketcache.ioengine no longer supports the 'heap' value.
> {quote}
> But the changes are substantial and warrant a section of their own under the
> "Changes of Note!" banner.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-20274) [DOC] additional metrics related changes for 2.0 upgrade section
[
https://issues.apache.org/jira/browse/HBASE-20274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020741#comment-17020741
]
Michael Stack commented on HBASE-20274:
---
Unscheduling a subtask that has gone told (made it into an issue).
> [DOC] additional metrics related changes for 2.0 upgrade section
>
>
> Key: HBASE-20274
> URL: https://issues.apache.org/jira/browse/HBASE-20274
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Priority: Major
>
> Feedback on HBASE-19158 from [~md...@cloudera.com]
> {quote}
> Metrics:
> HBASE-17957
> {quote}
> If folks find others before this gets done, feel free to drop here in
> comments.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-20274) [DOC] additional metrics related changes for 2.0 upgrade section
[
https://issues.apache.org/jira/browse/HBASE-20274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack updated HBASE-20274:
--
Parent: (was: HBASE-13428)
Issue Type: Bug (was: Sub-task)
> [DOC] additional metrics related changes for 2.0 upgrade section
>
>
> Key: HBASE-20274
> URL: https://issues.apache.org/jira/browse/HBASE-20274
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Priority: Major
>
> Feedback on HBASE-19158 from [~md...@cloudera.com]
> {quote}
> Metrics:
> HBASE-17957
> {quote}
> If folks find others before this gets done, feel free to drop here in
> comments.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-20277) [DOC] appendix with guidance on updating applications for HBase 2
[
https://issues.apache.org/jira/browse/HBASE-20277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020740#comment-17020740
]
Michael Stack commented on HBASE-20277:
---
Unscheduling an unaddressed sub-task made into an issue.
> [DOC] appendix with guidance on updating applications for HBase 2
> -
>
> Key: HBASE-20277
> URL: https://issues.apache.org/jira/browse/HBASE-20277
> Project: HBase
> Issue Type: Bug
> Components: Client, documentation
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Priority: Major
>
> Feedback from [~mdrob] on HBASE-19158:
> {quote}
> Classes/Methods moved:
> * HBASE-19425 (and subtasks) Put/Delete/Increment/Append
> * HBASE-19627 (and subtasks) Cell & Impl
> * HBASE-18978 (and subtasks) Async/Table
> * HBASE-18805 (and subtasks) Async/Admin
> * HBASE-14996 (and subtasks) Lots of misc changes
> * HBASE-13197 (and subtasks) Connection
> * HBASE-13844 KeyValue -> CellUtil
> * Maybe HBASE-19950 is worth calling out? Not really an upgrade issue though,
> but I know the old behavior of SCVF changed in a surprising way for the YARN
> ATS folks, when they were previously relying on what ended up being a bug.
> Could be mitigated by having a compatibility report, but would benefit from
> narrative messaging.
> Almost definitely need a blanket reminder that some deprecated APIs went away.
> {quote}
> from @busbey on HBASE-19158
> {quote}
> bq. * Classes/Methods moved:
> bq. * Almost definitely need a blanket reminder that some deprecated APIs
> went away.
> My intention was for both of these to be handled by this blurb:
> {code}
> +[[upgrade2.0.public.api]]
> +.Multiple breaking changes to source and binary compatibility for client API
> +The Java client API for HBase has a number of changes that break both source
> and binary compatibility for details see the Compatibility Check Report for
> the release you'll be upgrading to.
> +
> +
> +
> {code}
> I figure once we get an appendix that provides guidance on migrating an
> application from 1.x to 2.x we can also link to it.
> {quote}
> Get an appendix in place and then update the upgrade section as needed to
> draw attention to it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-20277) [DOC] appendix with guidance on updating applications for HBase 2
[
https://issues.apache.org/jira/browse/HBASE-20277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack updated HBASE-20277:
--
Parent: (was: HBASE-13428)
Issue Type: Bug (was: Sub-task)
> [DOC] appendix with guidance on updating applications for HBase 2
> -
>
> Key: HBASE-20277
> URL: https://issues.apache.org/jira/browse/HBASE-20277
> Project: HBase
> Issue Type: Bug
> Components: Client, documentation
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Priority: Major
>
> Feedback from [~mdrob] on HBASE-19158:
> {quote}
> Classes/Methods moved:
> * HBASE-19425 (and subtasks) Put/Delete/Increment/Append
> * HBASE-19627 (and subtasks) Cell & Impl
> * HBASE-18978 (and subtasks) Async/Table
> * HBASE-18805 (and subtasks) Async/Admin
> * HBASE-14996 (and subtasks) Lots of misc changes
> * HBASE-13197 (and subtasks) Connection
> * HBASE-13844 KeyValue -> CellUtil
> * Maybe HBASE-19950 is worth calling out? Not really an upgrade issue though,
> but I know the old behavior of SCVF changed in a surprising way for the YARN
> ATS folks, when they were previously relying on what ended up being a bug.
> Could be mitigated by having a compatibility report, but would benefit from
> narrative messaging.
> Almost definitely need a blanket reminder that some deprecated APIs went away.
> {quote}
> from @busbey on HBASE-19158
> {quote}
> bq. * Classes/Methods moved:
> bq. * Almost definitely need a blanket reminder that some deprecated APIs
> went away.
> My intention was for both of these to be handled by this blurb:
> {code}
> +[[upgrade2.0.public.api]]
> +.Multiple breaking changes to source and binary compatibility for client API
> +The Java client API for HBase has a number of changes that break both source
> and binary compatibility for details see the Compatibility Check Report for
> the release you'll be upgrading to.
> +
> +
> +
> {code}
> I figure once we get an appendix that provides guidance on migrating an
> application from 1.x to 2.x we can also link to it.
> {quote}
> Get an appendix in place and then update the upgrade section as needed to
> draw attention to it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-20279) [tooling] check for incompatible environment for HBase 2.0 upgrade
[
https://issues.apache.org/jira/browse/HBASE-20279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020739#comment-17020739
]
Michael Stack commented on HBASE-20279:
---
Moved unaddressed issue from subtask to standalone, unscheduled issue.
> [tooling] check for incompatible environment for HBase 2.0 upgrade
> --
>
> Key: HBASE-20279
> URL: https://issues.apache.org/jira/browse/HBASE-20279
> Project: HBase
> Issue Type: Bug
> Components: documentation, tooling
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Priority: Critical
>
> from [~stack] on HBASE-19158:
> {quote}
> bq. Server failure. HBase can still read HFiles written in the older version
> 2 format.
> Need a script that user can run pre-upgrade.
> {quote}
> Additional commentary from [~busbey]
> In addition to the config check stack mentions, such a tool could check
> * other problematic configs, like those for master-hosts-regions
> * inconsistent configs due to changed defaults (I think we should have some
> smell-tests around the change in default for hbase.security.authorization)
> * (optionally) check for unreadable WALs
> * (optionally) check for unreadable hfiles
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-20279) [tooling] check for incompatible environment for HBase 2.0 upgrade
[
https://issues.apache.org/jira/browse/HBASE-20279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack updated HBASE-20279:
--
Parent: (was: HBASE-13428)
Issue Type: Bug (was: Sub-task)
> [tooling] check for incompatible environment for HBase 2.0 upgrade
> --
>
> Key: HBASE-20279
> URL: https://issues.apache.org/jira/browse/HBASE-20279
> Project: HBase
> Issue Type: Bug
> Components: documentation, tooling
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Priority: Critical
>
> from [~stack] on HBASE-19158:
> {quote}
> bq. Server failure. HBase can still read HFiles written in the older version
> 2 format.
> Need a script that user can run pre-upgrade.
> {quote}
> Additional commentary from [~busbey]
> In addition to the config check stack mentions, such a tool could check
> * other problematic configs, like those for master-hosts-regions
> * inconsistent configs due to changed defaults (I think we should have some
> smell-tests around the change in default for hbase.security.authorization)
> * (optionally) check for unreadable WALs
> * (optionally) check for unreadable hfiles
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-20278) [DOC] include ref guide updates for HBase 2.0 HBCK
[
https://issues.apache.org/jira/browse/HBASE-20278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack resolved HBASE-20278.
---
Fix Version/s: 2.3.0
Assignee: Michael Stack (was: Umesh Agashe)
Resolution: Implemented
Resolving as implemented over in hbase-operator-tools in the hbck2 README:
https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2
> [DOC] include ref guide updates for HBase 2.0 HBCK
> --
>
> Key: HBASE-20278
> URL: https://issues.apache.org/jira/browse/HBASE-20278
> Project: HBase
> Issue Type: Sub-task
> Components: documentation, hbck
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Michael Stack
>Priority: Critical
> Fix For: 2.3.0
>
>
> Called out by [~stack] in HBASE-19158
> {quote}
> bq. HBCK tool from an earlier release against an HBase 2.0+ cluster will
> destructively alter said cluster in unrecoverable ways.
> Footnote or callout that says something like "Unfortunately we are unable to
> distinguish an HBCK client so cannot put in place guards against destructive
> HBCK changes." probably too much for this startup section on reread
> of what I've written here.
> bq. As of HBase 2.0, HBCK is a read-only tool that can report the status of
> some non-public system internals. You should not rely on the format nor
> content of these internals to remain consistent across HBase releases.
> Ugh. We need HBCK2 and a pointer here to it. Will do in a follow-on.
> {quote}
> then update the upgrade section to point to the docs
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-20516) Offheap read-path needs more detail
[ https://issues.apache.org/jira/browse/HBASE-20516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-20516: -- Fix Version/s: (was: 2.3.0) 3.0.0 Hadoop Flags: Reviewed Assignee: Michael Stack Resolution: Fixed Status: Resolved (was: Patch Available) Pushed what was here. Could do w/ more but this helps a bit at least. > Offheap read-path needs more detail > --- > > Key: HBASE-20516 > URL: https://issues.apache.org/jira/browse/HBASE-20516 > Project: HBase > Issue Type: Task > Components: Offheaping >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0 > > Attachments: 20516.initial.patch.txt, 20516.initial.patch.txt > > > Needs notes on what an operator should look for to see that all is on and > what to monitor in a running cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack merged pull request #1081: HBASE-20516 Offheap read-path needs more detail
saintstack merged pull request #1081: HBASE-20516 Offheap read-path needs more detail URL: https://github.com/apache/hbase/pull/1081 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1062: HBASE-23705 Add CellComparator to HFileContext
Apache-HBase commented on issue #1062: HBASE-23705 Add CellComparator to HFileContext URL: https://github.com/apache/hbase/pull/1062#issuecomment-576986412 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 17s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 20 new or modified test files. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 52s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 6m 52s | master passed | | +1 :green_heart: | compile | 2m 0s | master passed | | +1 :green_heart: | checkstyle | 2m 11s | master passed | | +1 :green_heart: | shadedjars | 5m 26s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 24s | master passed | | +0 :ok: | spotbugs | 5m 14s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 7m 6s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 44s | the patch passed | | +1 :green_heart: | compile | 1m 50s | the patch passed | | +1 :green_heart: | javac | 1m 50s | the patch passed | | +1 :green_heart: | checkstyle | 0m 24s | hbase-common: The patch generated 0 new + 17 unchanged - 32 fixed = 17 total (was 49) | | -1 :x: | checkstyle | 1m 12s | hbase-server: The patch generated 1 new + 54 unchanged - 156 fixed = 55 total (was 210) | | -1 :x: | checkstyle | 0m 18s | hbase-mapreduce: The patch generated 1 new + 0 unchanged - 18 fixed = 1 total (was 18) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 5m 0s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 17m 18s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 1m 12s | the patch passed | | +1 :green_heart: | findbugs | 6m 56s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 3m 10s | hbase-common in the patch passed. | | -1 :x: | unit | 33m 7s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 19m 20s | hbase-mapreduce in the patch passed. | | +1 :green_heart: | asflicense | 0m 54s | The patch does not generate ASF License warnings. | | | | 133m 27s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.regionserver.compactions.TestStripeCompactionPolicy | | | hadoop.hbase.regionserver.compactions.TestDateTieredCompactor | | | hadoop.hbase.regionserver.compactions.TestStripeCompactor | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1062/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1062 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux b5eec9873b25 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1062/out/precommit/personality/provided.sh | | git revision | master / ba3463d9de | | Default Java | 1.8.0_181 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1062/3/artifact/out/diff-checkstyle-hbase-server.txt | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1062/3/artifact/out/diff-checkstyle-hbase-mapreduce.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1062/3/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1062/3/testReport/ | | Max. process+thread count | 5347 (vs. ulimit of 1) | | modules | C: hbase-common hbase-server hbase-mapreduce U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1062/3/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
[GitHub] [hbase] virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log
virajjasani commented on a change in pull request #754: HBASE-22978 : Online
slow response log
URL: https://github.com/apache/hbase/pull/754#discussion_r369326567
##
File path: src/main/asciidoc/_chapters/ops_mgt.adoc
##
@@ -1914,6 +1914,85 @@ In the case that the call is not a client operation,
that detailed fingerprint i
This particular example, for example, would indicate that the likely cause of
slowness is simply a very large (on the order of 100MB) multiput, as we can
tell by the "vlen," or value length, fields of each put in the multiPut.
+ Get Slow Response Log from shell
+When an individual RPC exceeds a configurable time bound we log a complaint
+by way of the logging subsystem
+
+e.g.
+
+
+2019-10-02 10:10:22,195 WARN [,queue=15,port=60020] ipc.RpcServer -
(responseTooSlow):
+{"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
+"starttimems":1567203007549,
+"responsesize":6819737,
+"method":"Scan",
+"param":"region { type: REGION_NAME value:
\"t1,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
+"processingtimems":28646,
+"client":"10.253.196.215:41116",
+"queuetimems":22453,
+"class":"HRegionServer"}
+
+
+Unfortunately often the request parameters are truncated as per above Example.
+The truncation is unfortunate because it eliminates much of the utility of
+the warnings. For example, the region name, the start and end keys, and the
+filter hierarchy are all important clues for debugging performance problems
+caused by moderate to low selectivity queries or queries made at a high rate.
+
+HBASE-22978 introduces maintaining an in-memory ring buffer of requests that
were judged to
+be too slow in addition to the responseTooSlow logging. The in-memory
representation can be
+complete. There is some chance a high rate of requests will cause information
on other
+interesting requests to be overwritten before it can be read. This is an
acceptable trade off.
+
+shell commands to retrieve slowlog responses from RegionServers:
+
+
+Retrieve latest SlowLog Responses maintained by each or specific RegionServers.
+Specify 'all_rs' to include all RS otherwise array of 'server_name' for
specific
+RS. 'server_name' is the host, port plus startcode of a RegionServer.
+e.g.: host187.example.com,60020,1289493121758 (find servername in
+master ui or when you do detailed status in shell)
+
+Examples:
+
+ hbase> get_slowlog_responses 'all_rs'=> get
slowlog responses from all RS
+ hbase> get_slowlog_responses ['server_name1', 'server_name2']=> get
slowlog responses from server_name1,
+
server_name2
+ hbase> get_slowlog_responses 'all_rs', {'REGION_NAME' => 'hbase:meta,,1'}
+ => get
slowlog responses only related to meta
+ region
+ hbase> get_slowlog_responses 'all_rs', {'TABLE_NAME' => 't1'}=> get
slowlog responses only related to t1 table
+ hbase> get_slowlog_responses 'all_rs', {'CLIENT_IP' => '192.162.1.40:60225'}
+ => get
slowlog responses with given client
+ IP
address
+ hbase> get_slowlog_responses 'all_rs', {'REGION_NAME' => 'hbase:meta,,1',
'TABLE_NAME' => 't1'}
+ => get
slowlog responses with given region name
+ or table
name
+ hbase> get_slowlog_responses 'all_rs', {'USER' => 'user_name', 'CLIENT_IP'
=> '192.162.1.40:60225'}
+ => get
slowlog responses that match either
+ provided
client IP address or user name
+
+
+
+
+shell command to clear slowlog responses from RegionServer:
+
+
+Clears SlowLog Responses maintained by each or specific RegionServers.
+Specify array of 'server_name' for specific RS. 'server_name' is
+the host, port plus startcode of a RegionServer.
+e.g.: host187.example.com,60020,1289493121758 (find servername in
+master ui or when you do detailed status in shell)
+
+Examples:
+
+ hbase> clear_slowlog_responses => clears
slowlog responses from all RS
+ hbase> clear_slowlog_responses ['server_name1', 'server_name2']=> clears
slowlog responses from server_name1,
+
server_name2
+
+
+
+
Review comment:
Sure let me include `*` then
This is an automated message from
[GitHub] [hbase] virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log
virajjasani commented on a change in pull request #754: HBASE-22978 : Online
slow response log
URL: https://github.com/apache/hbase/pull/754#discussion_r369349567
##
File path: src/main/asciidoc/_chapters/ops_mgt.adoc
##
@@ -1914,6 +1914,85 @@ In the case that the call is not a client operation,
that detailed fingerprint i
This particular example, for example, would indicate that the likely cause of
slowness is simply a very large (on the order of 100MB) multiput, as we can
tell by the "vlen," or value length, fields of each put in the multiPut.
+ Get Slow Response Log from shell
+When an individual RPC exceeds a configurable time bound we log a complaint
+by way of the logging subsystem
+
+e.g.
+
+
+2019-10-02 10:10:22,195 WARN [,queue=15,port=60020] ipc.RpcServer -
(responseTooSlow):
+{"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
+"starttimems":1567203007549,
+"responsesize":6819737,
+"method":"Scan",
+"param":"region { type: REGION_NAME value:
\"t1,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
+"processingtimems":28646,
+"client":"10.253.196.215:41116",
+"queuetimems":22453,
+"class":"HRegionServer"}
+
+
+Unfortunately often the request parameters are truncated as per above Example.
+The truncation is unfortunate because it eliminates much of the utility of
+the warnings. For example, the region name, the start and end keys, and the
+filter hierarchy are all important clues for debugging performance problems
+caused by moderate to low selectivity queries or queries made at a high rate.
+
+HBASE-22978 introduces maintaining an in-memory ring buffer of requests that
were judged to
+be too slow in addition to the responseTooSlow logging. The in-memory
representation can be
+complete. There is some chance a high rate of requests will cause information
on other
+interesting requests to be overwritten before it can be read. This is an
acceptable trade off.
+
+shell commands to retrieve slowlog responses from RegionServers:
+
+
+Retrieve latest SlowLog Responses maintained by each or specific RegionServers.
+Specify 'all_rs' to include all RS otherwise array of 'server_name' for
specific
+RS. 'server_name' is the host, port plus startcode of a RegionServer.
+e.g.: host187.example.com,60020,1289493121758 (find servername in
+master ui or when you do detailed status in shell)
+
+Examples:
+
+ hbase> get_slowlog_responses 'all_rs'=> get
slowlog responses from all RS
+ hbase> get_slowlog_responses ['server_name1', 'server_name2']=> get
slowlog responses from server_name1,
+
server_name2
+ hbase> get_slowlog_responses 'all_rs', {'REGION_NAME' => 'hbase:meta,,1'}
+ => get
slowlog responses only related to meta
+ region
+ hbase> get_slowlog_responses 'all_rs', {'TABLE_NAME' => 't1'}=> get
slowlog responses only related to t1 table
+ hbase> get_slowlog_responses 'all_rs', {'CLIENT_IP' => '192.162.1.40:60225'}
+ => get
slowlog responses with given client
+ IP
address
+ hbase> get_slowlog_responses 'all_rs', {'REGION_NAME' => 'hbase:meta,,1',
'TABLE_NAME' => 't1'}
+ => get
slowlog responses with given region name
+ or table
name
+ hbase> get_slowlog_responses 'all_rs', {'USER' => 'user_name', 'CLIENT_IP'
=> '192.162.1.40:60225'}
+ => get
slowlog responses that match either
+ provided
client IP address or user name
+
+
+
+
+shell command to clear slowlog responses from RegionServer:
+
+
+Clears SlowLog Responses maintained by each or specific RegionServers.
+Specify array of 'server_name' for specific RS. 'server_name' is
+the host, port plus startcode of a RegionServer.
+e.g.: host187.example.com,60020,1289493121758 (find servername in
+master ui or when you do detailed status in shell)
+
+Examples:
+
+ hbase> clear_slowlog_responses => clears
slowlog responses from all RS
+ hbase> clear_slowlog_responses ['server_name1', 'server_name2']=> clears
slowlog responses from server_name1,
+
server_name2
+
+
+
+
Review comment:
Done
This is an automated message from the Apache Git Service.
[GitHub] [hbase] Apache-HBase commented on issue #936: HBASE-17115 Define UI admins via an ACL
Apache-HBase commented on issue #936: HBASE-17115 Define UI admins via an ACL URL: https://github.com/apache/hbase/pull/936#issuecomment-576981014 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 32s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 34s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 35s | master passed | | +1 :green_heart: | compile | 3m 16s | master passed | | +1 :green_heart: | checkstyle | 2m 35s | master passed | | +0 :ok: | refguide | 6m 39s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | shadedjars | 5m 9s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 4m 9s | master passed | | +0 :ok: | spotbugs | 4m 27s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 20m 55s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 26s | the patch passed | | +1 :green_heart: | compile | 3m 15s | the patch passed | | +1 :green_heart: | javac | 3m 15s | the patch passed | | -1 :x: | checkstyle | 2m 29s | root: The patch generated 1 new + 144 unchanged - 0 fixed = 145 total (was 144) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +0 :ok: | refguide | 6m 14s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | shadedjars | 5m 0s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 17m 3s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 3m 40s | the patch passed | | +1 :green_heart: | findbugs | 21m 25s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 232m 13s | root in the patch passed. | | +1 :green_heart: | asflicense | 1m 36s | The patch does not generate ASF License warnings. | | | | 356m 22s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-936/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/936 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile refguide | | uname | Linux 0cedeab57d46 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-936/out/precommit/personality/provided.sh | | git revision | master / 00e64d83e8 | | Default Java | 1.8.0_181 | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-936/2/artifact/out/branch-site/book.html | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-936/2/artifact/out/diff-checkstyle-root.txt | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-936/2/artifact/out/patch-site/book.html | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-936/2/testReport/ | | Max. process+thread count | 5263 (vs. ulimit of 1) | | modules | C: hbase-http hbase-server . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-936/2/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log
virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log URL: https://github.com/apache/hbase/pull/754#discussion_r369324644 ## File path: hbase-shell/src/main/ruby/shell/commands/get_slowlog_responses.rb ## @@ -0,0 +1,65 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with this +# work for additional information regarding copyright ownership. The ASF +# licenses this file to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +# Retrieve latest slowlog responses maintained in memory by RegionServers + +module Shell + module Commands +# Retrieve latest slowlog responses +class GetSlowlogResponses < Command + def help +<<-EOF +Retrieve latest SlowLog Responses maintained by each or specific RegionServers. +Specify 'all_rs' to include all RS otherwise array of 'server_name' for specific +RS. 'server_name' is the host, port plus startcode of a RegionServer. Review comment: I think we always pass server_name as `host:port:startcode` of a RegionServer right? That is what would work even here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log
virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log URL: https://github.com/apache/hbase/pull/754#discussion_r369342066 ## File path: hbase-shell/src/main/ruby/shell/commands/clear_slowlog_responses.rb ## @@ -0,0 +1,47 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with this +# work for additional information regarding copyright ownership. The ASF +# licenses this file to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +# Clear slowlog responses maintained in memory by RegionServers + +module Shell + module Commands +# Clear slowlog responses +class ClearSlowlogResponses < Command + def help +<<-EOF +Clears SlowLog Responses maintained by each or specific RegionServers. +Specify array of 'server_name' for specific RS. 'server_name' is +the host, port plus startcode of a RegionServer. +e.g.: host187.example.com,60020,1289493121758 (find servername in +master ui or when you do detailed status in shell) + +Examples: + + hbase> clear_slowlog_responses => clears slowlog responses from all RS + hbase> clear_slowlog_responses ['server_name1', 'server_name2']=> clears slowlog responses from server_name1, + server_name2 + + +EOF + end + + def command(server_names = nil) +admin.clear_slowlog_responses(server_names) + end +end + end +end Review comment: Oh but I think if we want to on/off dynamically, then we would need to involve ZNode just like any other `_switch` commands. Should we do that? Or may be do it as part of separate sub-task? What would be your preference? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (HBASE-22978) Online slow response log
[
https://issues.apache.org/jira/browse/HBASE-22978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020704#comment-17020704
]
Viraj Jasani edited comment on HBASE-22978 at 1/22/20 2:21 AM:
---
Thanks [~stack]
Sure let me try for 2.3.0 soon after we have enough reviews and confidence.
Will address comments.
Also, 2.3.0 backport might not take more time after trunk commit compared to
1.6.0 backport, will create backport subtask for 1.6.0.
Looking forward to all backports :)
was (Author: vjasani):
Thanks [~stack]
Sure let me try for 2.3.0 soon after we have enough reviews and confidence.
Will address comments.
Also, 2.3.0 backport might not take more time after trunk commit compared to
1.6.0 backport, will create backport subtask for 1.6.0.
> Online slow response log
>
>
> Key: HBASE-22978
> URL: https://issues.apache.org/jira/browse/HBASE-22978
> Project: HBase
> Issue Type: New Feature
> Components: Admin, Operability, regionserver, shell
>Affects Versions: 3.0.0, 2.3.0, 1.5.1
>Reporter: Andrew Kyle Purtell
>Assignee: Viraj Jasani
>Priority: Minor
> Fix For: 3.0.0, 2.3.0, 1.6.0
>
> Attachments: Screen Shot 2019-10-19 at 2.31.59 AM.png, Screen Shot
> 2019-10-19 at 2.32.54 AM.png, Screen Shot 2019-10-19 at 2.34.11 AM.png,
> Screen Shot 2019-10-19 at 2.36.14 AM.png
>
>
> Today when an individual RPC exceeds a configurable time bound we log a
> complaint by way of the logging subsystem. These log lines look like:
> {noformat}
> 2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
> (responseTooSlow):
> {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
> "starttimems":1567203007549,
> "responsesize":6819737,
> "method":"Scan",
> "param":"region { type: REGION_NAME value:
> \"tsdb,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
> "processingtimems":28646,
> "client":"10.253.196.215:41116",
> "queuetimems":22453,
> "class":"HRegionServer"}
> {noformat}
> Unfortunately we often truncate the request parameters, like in the above
> example. We do this because the human readable representation is verbose, the
> rate of too slow warnings may be high, and the combination of these things
> can overwhelm the log capture system. The truncation is unfortunate because
> it eliminates much of the utility of the warnings. For example, the region
> name, the start and end keys, and the filter hierarchy are all important
> clues for debugging performance problems caused by moderate to low
> selectivity queries or queries made at a high rate.
> We can maintain an in-memory ring buffer of requests that were judged to be
> too slow in addition to the responseTooSlow logging. The in-memory
> representation can be complete and compressed. A new admin API and shell
> command can provide access to the ring buffer for online performance
> debugging. A modest sizing of the ring buffer will prevent excessive memory
> utilization for a minor performance debugging feature by limiting the total
> number of retained records. There is some chance a high rate of requests will
> cause information on other interesting requests to be overwritten before it
> can be read. This is the nature of a ring buffer and an acceptable trade off.
> The write request types do not require us to retain all information submitted
> in the request. We don't need to retain all key-values in the mutation, which
> may be too large to comfortably retain. We only need a unique set of row
> keys, or even a min/max range, and total counts.
> The consumers of this information will be debugging tools. We can afford to
> apply fast compression to ring buffer entries (if codec support is
> available), something like snappy or zstandard, and decompress on the fly
> when servicing the retrieval API request. This will minimize the impact of
> retaining more information about slow requests than we do today.
> This proposal is for retention of request information only, the same
> information provided by responseTooSlow warnings. Total size of response
> serialization, possibly also total cell or row counts, should be sufficient
> to characterize the response.
> Optionally persist new entries added to the ring buffer into one or more
> files in HDFS in a write-behind manner. If the HDFS writer blocks or falls
> behind and we are unable to persist an entry before it is overwritten, that
> is fine. Response too slow logging is best effort. If we can detect this make
> a note of it in the log file. Provide a tool for parsing, dumping, filtering,
> and pretty printing the slow logs written to HDFS. The tool and the shell can
> share and reuse some utility
[jira] [Comment Edited] (HBASE-23279) Switch default block encoding to ROW_INDEX_V1
[
https://issues.apache.org/jira/browse/HBASE-23279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020718#comment-17020718
]
Viraj Jasani edited comment on HBASE-23279 at 1/22/20 2:11 AM:
---
It would be great to have HBASE-23705 landed to trunk and branch-2.
{quote}What is in the way of landing this issue other than HBASE-23705 ? You
still have size concerns? You want it in hbase-2.3.0?
{quote}
Yes apart from the above Jira, the only concern is with size.
[~larsh] [~anoop.hbase] Given that things look good with BucketCache block
counts and other calculations and also, the overall size increase is about 3%,
is it good to commit this?
Considering HBase handling block size increase carefully, may be it's good time
now to land this to trunk and 2.3.0. Sounds good?
was (Author: vjasani):
It would be great to have HBASE-23705 landed to trunk and branch-2.
{quote}What is in the way of landing this issue other than HBASE-23705 ? You
still have size concerns? You want it in hbase-2.3.0?
{quote}
Yes apart from the above Jira, the only concern is with size.
[~larsh] [~anoop.hbase] Given that things look good with BucketCache block
counts and other calculations and also, the overall size increase is about 3%,
is it good time to land this to trunk and 2.3.0?
Considering HBase handling block size increase carefully, may be it's good time
now to land this to trunk and 2.3.0. Sounds good?
> Switch default block encoding to ROW_INDEX_V1
> -
>
> Key: HBASE-23279
> URL: https://issues.apache.org/jira/browse/HBASE-23279
> Project: HBase
> Issue Type: Wish
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Lars Hofhansl
>Assignee: Viraj Jasani
>Priority: Minor
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HBASE-23279.master.000.patch,
> HBASE-23279.master.001.patch, HBASE-23279.master.002.patch,
> HBASE-23279.master.003.patch
>
>
> Currently we set both block encoding and compression to NONE.
> ROW_INDEX_V1 has many advantages and (almost) no disadvantages (the hfiles
> are slightly larger about 3% or so). I think that would a better default than
> NONE.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23279) Switch default block encoding to ROW_INDEX_V1
[
https://issues.apache.org/jira/browse/HBASE-23279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020718#comment-17020718
]
Viraj Jasani commented on HBASE-23279:
--
It would be great to have HBASE-23705 landed to trunk and branch-2.
{quote}What is in the way of landing this issue other than HBASE-23705 ? You
still have size concerns? You want it in hbase-2.3.0?
{quote}
Yes apart from the above Jira, the only concern is with size.
[~larsh] [~anoop.hbase] Given that things look good with BucketCache block
counts and other calculations and also, the overall size increase is about 3%,
is it good time to land this to trunk and 2.3.0?
Considering HBase handling block size increase carefully, may be it's good time
now to land this to trunk and 2.3.0. Sounds good?
> Switch default block encoding to ROW_INDEX_V1
> -
>
> Key: HBASE-23279
> URL: https://issues.apache.org/jira/browse/HBASE-23279
> Project: HBase
> Issue Type: Wish
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Lars Hofhansl
>Assignee: Viraj Jasani
>Priority: Minor
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HBASE-23279.master.000.patch,
> HBASE-23279.master.001.patch, HBASE-23279.master.002.patch,
> HBASE-23279.master.003.patch
>
>
> Currently we set both block encoding and compression to NONE.
> ROW_INDEX_V1 has many advantages and (almost) no disadvantages (the hfiles
> are slightly larger about 3% or so). I think that would a better default than
> NONE.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] joshelser commented on issue #1080: HBASE-23709 Unwrap the real user to properly dispatch proxy-user auth'n
joshelser commented on issue #1080: HBASE-23709 Unwrap the real user to properly dispatch proxy-user auth'n URL: https://github.com/apache/hbase/pull/1080#issuecomment-576974651 Pushing a second commit to try to get hbase-rest and hbase-thrift unit tests to run.. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase removed a comment on issue #754: HBASE-22978 : Online slow response log
Apache-HBase removed a comment on issue #754: HBASE-22978 : Online slow response log URL: https://github.com/apache/hbase/pull/754#issuecomment-558278036 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 1s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 35s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 14s | master passed | | +1 :green_heart: | compile | 2m 59s | master passed | | +1 :green_heart: | checkstyle | 2m 40s | master passed | | +0 :ok: | refguide | 5m 29s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | shadedjars | 4m 35s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 5m 6s | master passed | | +0 :ok: | spotbugs | 1m 20s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 23m 0s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 56s | the patch passed | | +1 :green_heart: | compile | 3m 2s | the patch passed | | +1 :green_heart: | cc | 3m 2s | the patch passed | | +1 :green_heart: | javac | 3m 2s | the patch passed | | +1 :green_heart: | checkstyle | 2m 44s | root: The patch generated 0 new + 450 unchanged - 1 fixed = 450 total (was 451) | | -1 :x: | rubocop | 0m 18s | The patch generated 19 new + 490 unchanged - 0 fixed = 509 total (was 490) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +0 :ok: | refguide | 5m 29s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | shadedjars | 4m 36s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 15m 44s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | hbaseprotoc | 8m 23s | the patch passed | | +1 :green_heart: | javadoc | 5m 8s | the patch passed | | +1 :green_heart: | findbugs | 24m 2s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 243m 7s | root in the patch passed. | | +1 :green_heart: | asflicense | 4m 49s | The patch does not generate ASF License warnings. | | | | 382m 7s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-754/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/754 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile refguide xml cc hbaseprotoc prototool rubocop | | uname | Linux c7335fdf852d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-754/out/precommit/personality/provided.sh | | git revision | master / dbbba7932c | | Default Java | 1.8.0_181 | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-754/1/artifact/out/branch-site/book.html | | rubocop | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-754/1/artifact/out/diff-patch-rubocop.txt | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-754/1/artifact/out/patch-site/book.html | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-754/1/testReport/ | | Max. process+thread count | 5492 (vs. ulimit of 1) | | modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-server hbase-thrift hbase-shell . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-754/1/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
[jira] [Commented] (HBASE-22978) Online slow response log
[
https://issues.apache.org/jira/browse/HBASE-22978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020704#comment-17020704
]
Viraj Jasani commented on HBASE-22978:
--
Thanks [~stack]
Sure let me try for 2.3.0 soon after we have enough reviews and confidence.
Will address comments.
Also, 2.3.0 backport might not take more time after trunk commit compared to
1.6.0 backport, will create backport subtask for 1.6.0.
> Online slow response log
>
>
> Key: HBASE-22978
> URL: https://issues.apache.org/jira/browse/HBASE-22978
> Project: HBase
> Issue Type: New Feature
> Components: Admin, Operability, regionserver, shell
>Affects Versions: 3.0.0, 2.3.0, 1.5.1
>Reporter: Andrew Kyle Purtell
>Assignee: Viraj Jasani
>Priority: Minor
> Fix For: 3.0.0, 2.3.0, 1.6.0
>
> Attachments: Screen Shot 2019-10-19 at 2.31.59 AM.png, Screen Shot
> 2019-10-19 at 2.32.54 AM.png, Screen Shot 2019-10-19 at 2.34.11 AM.png,
> Screen Shot 2019-10-19 at 2.36.14 AM.png
>
>
> Today when an individual RPC exceeds a configurable time bound we log a
> complaint by way of the logging subsystem. These log lines look like:
> {noformat}
> 2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
> (responseTooSlow):
> {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
> "starttimems":1567203007549,
> "responsesize":6819737,
> "method":"Scan",
> "param":"region { type: REGION_NAME value:
> \"tsdb,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
> "processingtimems":28646,
> "client":"10.253.196.215:41116",
> "queuetimems":22453,
> "class":"HRegionServer"}
> {noformat}
> Unfortunately we often truncate the request parameters, like in the above
> example. We do this because the human readable representation is verbose, the
> rate of too slow warnings may be high, and the combination of these things
> can overwhelm the log capture system. The truncation is unfortunate because
> it eliminates much of the utility of the warnings. For example, the region
> name, the start and end keys, and the filter hierarchy are all important
> clues for debugging performance problems caused by moderate to low
> selectivity queries or queries made at a high rate.
> We can maintain an in-memory ring buffer of requests that were judged to be
> too slow in addition to the responseTooSlow logging. The in-memory
> representation can be complete and compressed. A new admin API and shell
> command can provide access to the ring buffer for online performance
> debugging. A modest sizing of the ring buffer will prevent excessive memory
> utilization for a minor performance debugging feature by limiting the total
> number of retained records. There is some chance a high rate of requests will
> cause information on other interesting requests to be overwritten before it
> can be read. This is the nature of a ring buffer and an acceptable trade off.
> The write request types do not require us to retain all information submitted
> in the request. We don't need to retain all key-values in the mutation, which
> may be too large to comfortably retain. We only need a unique set of row
> keys, or even a min/max range, and total counts.
> The consumers of this information will be debugging tools. We can afford to
> apply fast compression to ring buffer entries (if codec support is
> available), something like snappy or zstandard, and decompress on the fly
> when servicing the retrieval API request. This will minimize the impact of
> retaining more information about slow requests than we do today.
> This proposal is for retention of request information only, the same
> information provided by responseTooSlow warnings. Total size of response
> serialization, possibly also total cell or row counts, should be sufficient
> to characterize the response.
> Optionally persist new entries added to the ring buffer into one or more
> files in HDFS in a write-behind manner. If the HDFS writer blocks or falls
> behind and we are unable to persist an entry before it is overwritten, that
> is fine. Response too slow logging is best effort. If we can detect this make
> a note of it in the log file. Provide a tool for parsing, dumping, filtering,
> and pretty printing the slow logs written to HDFS. The tool and the shell can
> share and reuse some utility classes and methods for accomplishing that.
> —
> New shell commands:
> {{get_slow_responses [ ... , ] [ , \{
> } ]}}
> Retrieve, decode, and pretty print the contents of the too slow response ring
> buffer maintained by the given list of servers; or all servers in the cluster
> if no list is provided. Optionally provide a map of parameters for filtering
>
[GitHub] [hbase] Apache-HBase commented on issue #1081: HBASE-20516 Offheap read-path needs more detail
Apache-HBase commented on issue #1081: HBASE-20516 Offheap read-path needs more detail URL: https://github.com/apache/hbase/pull/1081#issuecomment-576962283 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 1s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 7m 14s | master passed | | +0 :ok: | refguide | 8m 47s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 45s | the patch passed | | -1 :x: | whitespace | 0m 0s | The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +0 :ok: | refguide | 8m 33s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 21s | The patch does not generate ASF License warnings. | | | | 33m 40s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1081/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1081 | | Optional Tests | dupname asflicense refguide | | uname | Linux 77c2647e6197 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1081/out/precommit/personality/provided.sh | | git revision | master / ba3463d9de | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1081/1/artifact/out/branch-site/book.html | | whitespace | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1081/1/artifact/out/whitespace-eol.txt | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1081/1/artifact/out/patch-site/book.html | | Max. process+thread count | 70 (vs. ulimit of 1) | | modules | C: . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1081/1/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log
virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log URL: https://github.com/apache/hbase/pull/754#discussion_r369327238 ## File path: hbase-shell/src/main/ruby/shell/commands/clear_slowlog_responses.rb ## @@ -0,0 +1,47 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with this +# work for additional information regarding copyright ownership. The ASF +# licenses this file to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +# Clear slowlog responses maintained in memory by RegionServers + +module Shell + module Commands +# Clear slowlog responses +class ClearSlowlogResponses < Command + def help +<<-EOF +Clears SlowLog Responses maintained by each or specific RegionServers. +Specify array of 'server_name' for specific RS. 'server_name' is +the host, port plus startcode of a RegionServer. +e.g.: host187.example.com,60020,1289493121758 (find servername in +master ui or when you do detailed status in shell) + +Examples: + + hbase> clear_slowlog_responses => clears slowlog responses from all RS + hbase> clear_slowlog_responses ['server_name1', 'server_name2']=> clears slowlog responses from server_name1, + server_name2 + + +EOF + end + + def command(server_names = nil) +admin.clear_slowlog_responses(server_names) + end +end + end +end Review comment: Should not be an issue to provide on/off feature but should we really provide it? The reason being 255 entries might be significantly less to store in RAM anyways. But again, having this feature as on/off is also a good idea. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log
virajjasani commented on a change in pull request #754: HBASE-22978 : Online
slow response log
URL: https://github.com/apache/hbase/pull/754#discussion_r369326567
##
File path: src/main/asciidoc/_chapters/ops_mgt.adoc
##
@@ -1914,6 +1914,85 @@ In the case that the call is not a client operation,
that detailed fingerprint i
This particular example, for example, would indicate that the likely cause of
slowness is simply a very large (on the order of 100MB) multiput, as we can
tell by the "vlen," or value length, fields of each put in the multiPut.
+ Get Slow Response Log from shell
+When an individual RPC exceeds a configurable time bound we log a complaint
+by way of the logging subsystem
+
+e.g.
+
+
+2019-10-02 10:10:22,195 WARN [,queue=15,port=60020] ipc.RpcServer -
(responseTooSlow):
+{"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
+"starttimems":1567203007549,
+"responsesize":6819737,
+"method":"Scan",
+"param":"region { type: REGION_NAME value:
\"t1,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
+"processingtimems":28646,
+"client":"10.253.196.215:41116",
+"queuetimems":22453,
+"class":"HRegionServer"}
+
+
+Unfortunately often the request parameters are truncated as per above Example.
+The truncation is unfortunate because it eliminates much of the utility of
+the warnings. For example, the region name, the start and end keys, and the
+filter hierarchy are all important clues for debugging performance problems
+caused by moderate to low selectivity queries or queries made at a high rate.
+
+HBASE-22978 introduces maintaining an in-memory ring buffer of requests that
were judged to
+be too slow in addition to the responseTooSlow logging. The in-memory
representation can be
+complete. There is some chance a high rate of requests will cause information
on other
+interesting requests to be overwritten before it can be read. This is an
acceptable trade off.
+
+shell commands to retrieve slowlog responses from RegionServers:
+
+
+Retrieve latest SlowLog Responses maintained by each or specific RegionServers.
+Specify 'all_rs' to include all RS otherwise array of 'server_name' for
specific
+RS. 'server_name' is the host, port plus startcode of a RegionServer.
+e.g.: host187.example.com,60020,1289493121758 (find servername in
+master ui or when you do detailed status in shell)
+
+Examples:
+
+ hbase> get_slowlog_responses 'all_rs'=> get
slowlog responses from all RS
+ hbase> get_slowlog_responses ['server_name1', 'server_name2']=> get
slowlog responses from server_name1,
+
server_name2
+ hbase> get_slowlog_responses 'all_rs', {'REGION_NAME' => 'hbase:meta,,1'}
+ => get
slowlog responses only related to meta
+ region
+ hbase> get_slowlog_responses 'all_rs', {'TABLE_NAME' => 't1'}=> get
slowlog responses only related to t1 table
+ hbase> get_slowlog_responses 'all_rs', {'CLIENT_IP' => '192.162.1.40:60225'}
+ => get
slowlog responses with given client
+ IP
address
+ hbase> get_slowlog_responses 'all_rs', {'REGION_NAME' => 'hbase:meta,,1',
'TABLE_NAME' => 't1'}
+ => get
slowlog responses with given region name
+ or table
name
+ hbase> get_slowlog_responses 'all_rs', {'USER' => 'user_name', 'CLIENT_IP'
=> '192.162.1.40:60225'}
+ => get
slowlog responses that match either
+ provided
client IP address or user name
+
+
+
+
+shell command to clear slowlog responses from RegionServer:
+
+
+Clears SlowLog Responses maintained by each or specific RegionServers.
+Specify array of 'server_name' for specific RS. 'server_name' is
+the host, port plus startcode of a RegionServer.
+e.g.: host187.example.com,60020,1289493121758 (find servername in
+master ui or when you do detailed status in shell)
+
+Examples:
+
+ hbase> clear_slowlog_responses => clears
slowlog responses from all RS
+ hbase> clear_slowlog_responses ['server_name1', 'server_name2']=> clears
slowlog responses from server_name1,
+
server_name2
+
+
+
+
Review comment:
Sure let me include `*` then
This is an automated message from
[GitHub] [hbase] virajjasani commented on issue #754: HBASE-22978 : Online slow response log
virajjasani commented on issue #754: HBASE-22978 : Online slow response log URL: https://github.com/apache/hbase/pull/754#issuecomment-576961336 > Skimmed. > > One thought was that we'd want to run a buffer at the RS entrance too, not just at the point where we interact with HDFS. This patch would not preclude us doing such a thing? With the patch we would run at RS entrance only and for any slow running query it will go to this ring buffer running at RS. Just that this patch doesn't include storing slow logs in HDFS, which is also a good idea and probably up for a subtask given the priority is storing slow logs in ring buffer first. Storing them in HDFS would be just an option as part of another Jira(sub-task). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log
virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log URL: https://github.com/apache/hbase/pull/754#discussion_r369324644 ## File path: hbase-shell/src/main/ruby/shell/commands/get_slowlog_responses.rb ## @@ -0,0 +1,65 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with this +# work for additional information regarding copyright ownership. The ASF +# licenses this file to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +# Retrieve latest slowlog responses maintained in memory by RegionServers + +module Shell + module Commands +# Retrieve latest slowlog responses +class GetSlowlogResponses < Command + def help +<<-EOF +Retrieve latest SlowLog Responses maintained by each or specific RegionServers. +Specify 'all_rs' to include all RS otherwise array of 'server_name' for specific +RS. 'server_name' is the host, port plus startcode of a RegionServer. Review comment: I think we always pass server_name as `host:port:startcode` of a RegionServer right? That is what would work even here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1080: HBASE-23709 Unwrap the real user to properly dispatch proxy-user auth'n
Apache-HBase commented on issue #1080: HBASE-23709 Unwrap the real user to properly dispatch proxy-user auth'n URL: https://github.com/apache/hbase/pull/1080#issuecomment-576957873 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 2s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 51s | master passed | | +1 :green_heart: | compile | 0m 33s | master passed | | +1 :green_heart: | checkstyle | 0m 35s | master passed | | +1 :green_heart: | shadedjars | 5m 30s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 30s | master passed | | +0 :ok: | spotbugs | 1m 31s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 29s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 30s | the patch passed | | +1 :green_heart: | compile | 0m 28s | the patch passed | | +1 :green_heart: | javac | 0m 28s | the patch passed | | +1 :green_heart: | checkstyle | 0m 30s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 5m 36s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 21m 13s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 0m 25s | the patch passed | | +1 :green_heart: | findbugs | 1m 34s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 14s | hbase-client in the patch passed. | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 64m 26s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1080/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1080 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 95b8b8436864 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1080/out/precommit/personality/provided.sh | | git revision | master / ba3463d9de | | Default Java | 1.8.0_181 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1080/1/testReport/ | | Max. process+thread count | 290 (vs. ulimit of 1) | | modules | C: hbase-client U: hbase-client | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1080/1/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1062: HBASE-23705 Add CellComparator to HFileContext
Apache-HBase commented on issue #1062: HBASE-23705 Add CellComparator to HFileContext URL: https://github.com/apache/hbase/pull/1062#issuecomment-576956704 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 36s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 20 new or modified test files. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 44s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 8m 10s | master passed | | +1 :green_heart: | compile | 2m 24s | master passed | | +1 :green_heart: | checkstyle | 2m 17s | master passed | | +1 :green_heart: | shadedjars | 5m 9s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 15s | master passed | | +0 :ok: | spotbugs | 5m 4s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 6m 39s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 32s | the patch passed | | +1 :green_heart: | compile | 1m 46s | the patch passed | | +1 :green_heart: | javac | 1m 46s | the patch passed | | +1 :green_heart: | checkstyle | 0m 24s | hbase-common: The patch generated 0 new + 17 unchanged - 32 fixed = 17 total (was 49) | | -1 :x: | checkstyle | 1m 14s | hbase-server: The patch generated 1 new + 54 unchanged - 156 fixed = 55 total (was 210) | | -1 :x: | checkstyle | 0m 18s | hbase-mapreduce: The patch generated 1 new + 0 unchanged - 18 fixed = 1 total (was 18) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 5m 3s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 17m 29s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 1m 13s | the patch passed | | +1 :green_heart: | findbugs | 6m 55s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 3m 9s | hbase-common in the patch passed. | | -1 :x: | unit | 32m 25s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 18m 34s | hbase-mapreduce in the patch passed. | | +1 :green_heart: | asflicense | 1m 1s | The patch does not generate ASF License warnings. | | | | 132m 9s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.regionserver.compactions.TestStripeCompactionPolicy | | | hadoop.hbase.regionserver.compactions.TestDateTieredCompactor | | | hadoop.hbase.regionserver.compactions.TestStripeCompactor | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1062/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1062 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux e3c868edfbbf 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1062/out/precommit/personality/provided.sh | | git revision | master / 00e64d83e8 | | Default Java | 1.8.0_181 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1062/2/artifact/out/diff-checkstyle-hbase-server.txt | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1062/2/artifact/out/diff-checkstyle-hbase-mapreduce.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1062/2/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1062/2/testReport/ | | Max. process+thread count | 5380 (vs. ulimit of 1) | | modules | C: hbase-common hbase-server hbase-mapreduce U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1062/2/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
[jira] [Commented] (HBASE-20516) Offheap read-path needs more detail
[ https://issues.apache.org/jira/browse/HBASE-20516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020688#comment-17020688 ] Michael Stack commented on HBASE-20516: --- Put up the +1'd patch as a PR just to make sure it still good. > Offheap read-path needs more detail > --- > > Key: HBASE-20516 > URL: https://issues.apache.org/jira/browse/HBASE-20516 > Project: HBase > Issue Type: Task > Components: Offheaping >Reporter: Michael Stack >Priority: Major > Fix For: 2.3.0 > > Attachments: 20516.initial.patch.txt, 20516.initial.patch.txt > > > Needs notes on what an operator should look for to see that all is on and > what to monitor in a running cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack opened a new pull request #1081: HBASE-20516 Offheap read-path needs more detail
saintstack opened a new pull request #1081: HBASE-20516 Offheap read-path needs more detail URL: https://github.com/apache/hbase/pull/1081 Signed-off-by: Anoop Sam John This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-20553) Add dependency CVE checking to nightly tests
[
https://issues.apache.org/jira/browse/HBASE-20553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020678#comment-17020678
]
Michael Stack commented on HBASE-20553:
---
Unscheduling. No movement.
> Add dependency CVE checking to nightly tests
>
>
> Key: HBASE-20553
> URL: https://issues.apache.org/jira/browse/HBASE-20553
> Project: HBase
> Issue Type: Umbrella
> Components: dependencies
>Affects Versions: 3.0.0
>Reporter: Sean Busbey
>Assignee: Sakthi
>Priority: Major
>
> We should proactively work to flag dependencies with known CVEs so that we
> can then update them early in our development instead of near a release.
> YETUS-441 is working to add a plugin for this, we should grab a copy early to
> make sure it works for us.
> Rough outline:
> 1. [install yetus locally|http://yetus.apache.org/downloads/]
> 2. [install the dependency-check
> cli|https://www.owasp.org/index.php/OWASP_Dependency_Check] (homebrew
> instructions on right hand margin)
> 3. Get a local copy of the OWASP datafile ({{dependency-check --updateonly
> --data /some/local/path/to/dir}})
> 4. Run {{hbase_nightly_yetus.sh}} using matching environment variables from
> the “yetus general check” (currently [line #126 in our nightly
> Jenkinsfile|https://github.com/apache/hbase/blob/master/dev-support/Jenkinsfile#L126])
> 5. Grab the plugin definition and suppression file from from YETUS-441
> 6. put the plugin definition either in a directory of dev-support or into the
> hbase-personality.sh directly
> 7. Re-run {{hbase_nightly_yetus.sh}} to verify that the plugin results show
> up. (Probably this will involve adding new pointers for “where is the
> suppression file”, “where is the OWASP datafile” and pointing them somewhere
> locally.)
> Once all of that is in place we’ll get the changes needed into a branch that
> we can test out. Over in YETUS-441 I’ll need to add a jenkins job that’ll
> handle periodically updating a copy of the datafile for the OWASP dependency
> checker. Presuming I have that in place by the time we have a nightly branch
> to check this out, then we’ll also need to update our nightly Jenkinsfile to
> fetch the data file from that job.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-20553) Add dependency CVE checking to nightly tests
[
https://issues.apache.org/jira/browse/HBASE-20553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack updated HBASE-20553:
--
Fix Version/s: (was: 2.3.0)
(was: 3.0.0)
> Add dependency CVE checking to nightly tests
>
>
> Key: HBASE-20553
> URL: https://issues.apache.org/jira/browse/HBASE-20553
> Project: HBase
> Issue Type: Umbrella
> Components: dependencies
>Affects Versions: 3.0.0
>Reporter: Sean Busbey
>Assignee: Sakthi
>Priority: Major
>
> We should proactively work to flag dependencies with known CVEs so that we
> can then update them early in our development instead of near a release.
> YETUS-441 is working to add a plugin for this, we should grab a copy early to
> make sure it works for us.
> Rough outline:
> 1. [install yetus locally|http://yetus.apache.org/downloads/]
> 2. [install the dependency-check
> cli|https://www.owasp.org/index.php/OWASP_Dependency_Check] (homebrew
> instructions on right hand margin)
> 3. Get a local copy of the OWASP datafile ({{dependency-check --updateonly
> --data /some/local/path/to/dir}})
> 4. Run {{hbase_nightly_yetus.sh}} using matching environment variables from
> the “yetus general check” (currently [line #126 in our nightly
> Jenkinsfile|https://github.com/apache/hbase/blob/master/dev-support/Jenkinsfile#L126])
> 5. Grab the plugin definition and suppression file from from YETUS-441
> 6. put the plugin definition either in a directory of dev-support or into the
> hbase-personality.sh directly
> 7. Re-run {{hbase_nightly_yetus.sh}} to verify that the plugin results show
> up. (Probably this will involve adding new pointers for “where is the
> suppression file”, “where is the OWASP datafile” and pointing them somewhere
> locally.)
> Once all of that is in place we’ll get the changes needed into a branch that
> we can test out. Over in YETUS-441 I’ll need to add a jenkins job that’ll
> handle periodically updating a copy of the datafile for the OWASP dependency
> checker. Presuming I have that in place by the time we have a nightly branch
> to check this out, then we’ll also need to update our nightly Jenkinsfile to
> fetch the data file from that job.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-21454) Kill zk spew
[ https://issues.apache.org/jira/browse/HBASE-21454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-21454: -- Fix Version/s: (was: 2.3.0) (was: 3.0.0) > Kill zk spew > > > Key: HBASE-21454 > URL: https://issues.apache.org/jira/browse/HBASE-21454 > Project: HBase > Issue Type: Bug > Components: logging, Zookeeper >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Attachments: HBASE-21454.master.001.patch > > > Kill the zk spew. This is radical dropping startup listing of CLASSPATH and > all properties. Can dial back-in what we need after this patch goes in. > I get spew each time I run a little command in spark-shell. Annoying. Always > been annoying in all logs. > More might be needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22978) Online slow response log
[
https://issues.apache.org/jira/browse/HBASE-22978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020671#comment-17020671
]
Michael Stack commented on HBASE-22978:
---
This supposed to got into hbase-2.3.0? Left some comments on PR
> Online slow response log
>
>
> Key: HBASE-22978
> URL: https://issues.apache.org/jira/browse/HBASE-22978
> Project: HBase
> Issue Type: New Feature
> Components: Admin, Operability, regionserver, shell
>Affects Versions: 3.0.0, 2.3.0, 1.5.1
>Reporter: Andrew Kyle Purtell
>Assignee: Viraj Jasani
>Priority: Minor
> Fix For: 3.0.0, 2.3.0, 1.6.0
>
> Attachments: Screen Shot 2019-10-19 at 2.31.59 AM.png, Screen Shot
> 2019-10-19 at 2.32.54 AM.png, Screen Shot 2019-10-19 at 2.34.11 AM.png,
> Screen Shot 2019-10-19 at 2.36.14 AM.png
>
>
> Today when an individual RPC exceeds a configurable time bound we log a
> complaint by way of the logging subsystem. These log lines look like:
> {noformat}
> 2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
> (responseTooSlow):
> {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
> "starttimems":1567203007549,
> "responsesize":6819737,
> "method":"Scan",
> "param":"region { type: REGION_NAME value:
> \"tsdb,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
> "processingtimems":28646,
> "client":"10.253.196.215:41116",
> "queuetimems":22453,
> "class":"HRegionServer"}
> {noformat}
> Unfortunately we often truncate the request parameters, like in the above
> example. We do this because the human readable representation is verbose, the
> rate of too slow warnings may be high, and the combination of these things
> can overwhelm the log capture system. The truncation is unfortunate because
> it eliminates much of the utility of the warnings. For example, the region
> name, the start and end keys, and the filter hierarchy are all important
> clues for debugging performance problems caused by moderate to low
> selectivity queries or queries made at a high rate.
> We can maintain an in-memory ring buffer of requests that were judged to be
> too slow in addition to the responseTooSlow logging. The in-memory
> representation can be complete and compressed. A new admin API and shell
> command can provide access to the ring buffer for online performance
> debugging. A modest sizing of the ring buffer will prevent excessive memory
> utilization for a minor performance debugging feature by limiting the total
> number of retained records. There is some chance a high rate of requests will
> cause information on other interesting requests to be overwritten before it
> can be read. This is the nature of a ring buffer and an acceptable trade off.
> The write request types do not require us to retain all information submitted
> in the request. We don't need to retain all key-values in the mutation, which
> may be too large to comfortably retain. We only need a unique set of row
> keys, or even a min/max range, and total counts.
> The consumers of this information will be debugging tools. We can afford to
> apply fast compression to ring buffer entries (if codec support is
> available), something like snappy or zstandard, and decompress on the fly
> when servicing the retrieval API request. This will minimize the impact of
> retaining more information about slow requests than we do today.
> This proposal is for retention of request information only, the same
> information provided by responseTooSlow warnings. Total size of response
> serialization, possibly also total cell or row counts, should be sufficient
> to characterize the response.
> Optionally persist new entries added to the ring buffer into one or more
> files in HDFS in a write-behind manner. If the HDFS writer blocks or falls
> behind and we are unable to persist an entry before it is overwritten, that
> is fine. Response too slow logging is best effort. If we can detect this make
> a note of it in the log file. Provide a tool for parsing, dumping, filtering,
> and pretty printing the slow logs written to HDFS. The tool and the shell can
> share and reuse some utility classes and methods for accomplishing that.
> —
> New shell commands:
> {{get_slow_responses [ ... , ] [ , \{
> } ]}}
> Retrieve, decode, and pretty print the contents of the too slow response ring
> buffer maintained by the given list of servers; or all servers in the cluster
> if no list is provided. Optionally provide a map of parameters for filtering
> as additional argument. The TABLE filter, which expects a string containing a
> table name, will include only entries pertaining to that table. The REGION
> filter, which expects a
[GitHub] [hbase] saintstack commented on a change in pull request #754: HBASE-22978 : Online slow response log
saintstack commented on a change in pull request #754: HBASE-22978 : Online slow response log URL: https://github.com/apache/hbase/pull/754#discussion_r369308822 ## File path: hbase-shell/src/main/ruby/shell/commands/clear_slowlog_responses.rb ## @@ -0,0 +1,47 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with this +# work for additional information regarding copyright ownership. The ASF +# licenses this file to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +# Clear slowlog responses maintained in memory by RegionServers + +module Shell + module Commands +# Clear slowlog responses +class ClearSlowlogResponses < Command + def help +<<-EOF +Clears SlowLog Responses maintained by each or specific RegionServers. +Specify array of 'server_name' for specific RS. 'server_name' is +the host, port plus startcode of a RegionServer. +e.g.: host187.example.com,60020,1289493121758 (find servername in +master ui or when you do detailed status in shell) + +Examples: + + hbase> clear_slowlog_responses => clears slowlog responses from all RS + hbase> clear_slowlog_responses ['server_name1', 'server_name2']=> clears slowlog responses from server_name1, + server_name2 + + +EOF + end + + def command(server_names = nil) +admin.clear_slowlog_responses(server_names) + end +end + end +end Review comment: Would an on/off for recording be hard to do with off being default? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #754: HBASE-22978 : Online slow response log
saintstack commented on a change in pull request #754: HBASE-22978 : Online slow response log URL: https://github.com/apache/hbase/pull/754#discussion_r369308936 ## File path: hbase-shell/src/main/ruby/shell/commands/clear_slowlog_responses.rb ## @@ -0,0 +1,47 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with this +# work for additional information regarding copyright ownership. The ASF +# licenses this file to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +# Clear slowlog responses maintained in memory by RegionServers + +module Shell + module Commands +# Clear slowlog responses +class ClearSlowlogResponses < Command + def help +<<-EOF +Clears SlowLog Responses maintained by each or specific RegionServers. +Specify array of 'server_name' for specific RS. 'server_name' is Review comment: Ditto on server_name here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #754: HBASE-22978 : Online slow response log
saintstack commented on a change in pull request #754: HBASE-22978 : Online
slow response log
URL: https://github.com/apache/hbase/pull/754#discussion_r369308592
##
File path: hbase-shell/src/main/ruby/shell/commands/get_slowlog_responses.rb
##
@@ -0,0 +1,65 @@
+#
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with this
+# work for additional information regarding copyright ownership. The ASF
+# licenses this file to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+
+# Retrieve latest slowlog responses maintained in memory by RegionServers
+
+module Shell
+ module Commands
+# Retrieve latest slowlog responses
+class GetSlowlogResponses < Command
+ def help
+<<-EOF
+Retrieve latest SlowLog Responses maintained by each or specific RegionServers.
+Specify 'all_rs' to include all RS otherwise array of 'server_name' for
specific
+RS. 'server_name' is the host, port plus startcode of a RegionServer.
+e.g.: host187.example.com,60020,1289493121758 (find servername in
+master ui or when you do detailed status in shell)
+
+Provide optional filter parameters as Hash
+
+Examples:
+
+ hbase> get_slowlog_responses 'all_rs'=> get
slowlog responses from all RS
+ hbase> get_slowlog_responses ['server_name1', 'server_name2']=> get
slowlog responses from server_name1,
+
server_name2
+ hbase> get_slowlog_responses 'all_rs', {'REGION_NAME' => 'hbase:meta,,1'}
+ => get
slowlog responses only related to meta
+ region
+ hbase> get_slowlog_responses 'all_rs', {'TABLE_NAME' => 't1'}=> get
slowlog responses only related to t1 table
+ hbase> get_slowlog_responses 'all_rs', {'CLIENT_IP' => '192.162.1.40:60225'}
+ => get
slowlog responses with given client
+ IP
address
+ hbase> get_slowlog_responses 'all_rs', {'REGION_NAME' => 'hbase:meta,,1',
'TABLE_NAME' => 't1'}
+ => get
slowlog responses with given region name
+ or table
name
+ hbase> get_slowlog_responses 'all_rs', {'USER' => 'user_name', 'CLIENT_IP'
=> '192.162.1.40:60225'}
+ => get
slowlog responses that match either
+ provided
client IP address or user name
+
+EOF
+ end
+
+ def command(server_names, args = {})
+unless args.is_a? Hash
+ raise 'Filter parameters are not Hash'
+end
+
+admin.get_slowlog_responses(server_names, args)
+ end
+end
+ end
+end
Review comment:
Good
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #754: HBASE-22978 : Online slow response log
saintstack commented on a change in pull request #754: HBASE-22978 : Online
slow response log
URL: https://github.com/apache/hbase/pull/754#discussion_r369307786
##
File path: src/main/asciidoc/_chapters/ops_mgt.adoc
##
@@ -1914,6 +1914,85 @@ In the case that the call is not a client operation,
that detailed fingerprint i
This particular example, for example, would indicate that the likely cause of
slowness is simply a very large (on the order of 100MB) multiput, as we can
tell by the "vlen," or value length, fields of each put in the multiPut.
+ Get Slow Response Log from shell
+When an individual RPC exceeds a configurable time bound we log a complaint
+by way of the logging subsystem
+
+e.g.
+
+
+2019-10-02 10:10:22,195 WARN [,queue=15,port=60020] ipc.RpcServer -
(responseTooSlow):
+{"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
+"starttimems":1567203007549,
+"responsesize":6819737,
+"method":"Scan",
+"param":"region { type: REGION_NAME value:
\"t1,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
+"processingtimems":28646,
+"client":"10.253.196.215:41116",
+"queuetimems":22453,
+"class":"HRegionServer"}
+
+
+Unfortunately often the request parameters are truncated as per above Example.
+The truncation is unfortunate because it eliminates much of the utility of
+the warnings. For example, the region name, the start and end keys, and the
+filter hierarchy are all important clues for debugging performance problems
+caused by moderate to low selectivity queries or queries made at a high rate.
+
+HBASE-22978 introduces maintaining an in-memory ring buffer of requests that
were judged to
+be too slow in addition to the responseTooSlow logging. The in-memory
representation can be
+complete. There is some chance a high rate of requests will cause information
on other
+interesting requests to be overwritten before it can be read. This is an
acceptable trade off.
+
+shell commands to retrieve slowlog responses from RegionServers:
+
+
+Retrieve latest SlowLog Responses maintained by each or specific RegionServers.
+Specify 'all_rs' to include all RS otherwise array of 'server_name' for
specific
+RS. 'server_name' is the host, port plus startcode of a RegionServer.
+e.g.: host187.example.com,60020,1289493121758 (find servername in
+master ui or when you do detailed status in shell)
+
+Examples:
+
+ hbase> get_slowlog_responses 'all_rs'=> get
slowlog responses from all RS
+ hbase> get_slowlog_responses ['server_name1', 'server_name2']=> get
slowlog responses from server_name1,
+
server_name2
+ hbase> get_slowlog_responses 'all_rs', {'REGION_NAME' => 'hbase:meta,,1'}
+ => get
slowlog responses only related to meta
+ region
+ hbase> get_slowlog_responses 'all_rs', {'TABLE_NAME' => 't1'}=> get
slowlog responses only related to t1 table
+ hbase> get_slowlog_responses 'all_rs', {'CLIENT_IP' => '192.162.1.40:60225'}
+ => get
slowlog responses with given client
+ IP
address
+ hbase> get_slowlog_responses 'all_rs', {'REGION_NAME' => 'hbase:meta,,1',
'TABLE_NAME' => 't1'}
+ => get
slowlog responses with given region name
+ or table
name
+ hbase> get_slowlog_responses 'all_rs', {'USER' => 'user_name', 'CLIENT_IP'
=> '192.162.1.40:60225'}
+ => get
slowlog responses that match either
+ provided
client IP address or user name
+
+
+
+
+shell command to clear slowlog responses from RegionServer:
+
+
+Clears SlowLog Responses maintained by each or specific RegionServers.
+Specify array of 'server_name' for specific RS. 'server_name' is
+the host, port plus startcode of a RegionServer.
+e.g.: host187.example.com,60020,1289493121758 (find servername in
+master ui or when you do detailed status in shell)
+
+Examples:
+
+ hbase> clear_slowlog_responses => clears
slowlog responses from all RS
+ hbase> clear_slowlog_responses ['server_name1', 'server_name2']=> clears
slowlog responses from server_name1,
+
server_name2
+
+
+
+
Review comment:
Nice doc. Is 'all_rs' a special argument that means all? Why not just do '*'?
[GitHub] [hbase] saintstack commented on a change in pull request #754: HBASE-22978 : Online slow response log
saintstack commented on a change in pull request #754: HBASE-22978 : Online slow response log URL: https://github.com/apache/hbase/pull/754#discussion_r369309334 ## File path: hbase-shell/src/main/ruby/shell/commands/clear_slowlog_responses.rb ## @@ -0,0 +1,47 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with this +# work for additional information regarding copyright ownership. The ASF +# licenses this file to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +# Clear slowlog responses maintained in memory by RegionServers + +module Shell + module Commands +# Clear slowlog responses +class ClearSlowlogResponses < Command + def help +<<-EOF +Clears SlowLog Responses maintained by each or specific RegionServers. +Specify array of 'server_name' for specific RS. 'server_name' is Review comment: For example over in 'move', it talks of 'server name', not 'server_name'. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #754: HBASE-22978 : Online slow response log
saintstack commented on a change in pull request #754: HBASE-22978 : Online slow response log URL: https://github.com/apache/hbase/pull/754#discussion_r369308392 ## File path: hbase-shell/src/main/ruby/shell/commands/get_slowlog_responses.rb ## @@ -0,0 +1,65 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with this +# work for additional information regarding copyright ownership. The ASF +# licenses this file to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +# Retrieve latest slowlog responses maintained in memory by RegionServers + +module Shell + module Commands +# Retrieve latest slowlog responses +class GetSlowlogResponses < Command + def help +<<-EOF +Retrieve latest SlowLog Responses maintained by each or specific RegionServers. +Specify 'all_rs' to include all RS otherwise array of 'server_name' for specific +RS. 'server_name' is the host, port plus startcode of a RegionServer. Review comment: Yeah, use '*' instead of 'all_rs'? And for server_name, is this the ServerName ? See other commands for how they discuss passing ServerName because I don't think this aligns; e.g. no where else to we talk of 'server_name' IIRC This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
