[jira] [Commented] (HBASE-21186) Document hbase.regionserver.executor.openregion.threads in MTTR section

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16753937#comment-16753937
 ] 

Sahil Aggarwal commented on HBASE-21186:


>From my observation with ~ 2200 regions on 4 RS:
 * Master is able to call assign region rpc pretty quick hence didn't find any 
bottleneck at master side as RS queue up the RPC in the executor unbounded 
workQueue.
 * Region server was taking too much time in completing the assign region RPC 
as in parallel it will execute only 
hbase.regionserver.executor.openregion.threads number of RPCs at a time.
 * Increasing the threads probably only execute more assign region RPC 
parallely. 

 

Let me check on how Master is making assign region RPC and if depends on any 
other config too, then probably we can come up with the formula you are 
suggesting. 

 

 

> Document hbase.regionserver.executor.openregion.threads in MTTR section
> ---
>
> Key: HBASE-21186
> URL: https://issues.apache.org/jira/browse/HBASE-21186
> Project: HBase
>  Issue Type: Improvement
>  Components: documentation
>Reporter: Sahil Aggarwal
>Assignee: Sahil Aggarwal
>Priority: Minor
> Attachments: HBASE-21186.master.001.patch, 
> HBASE-21186.master.002.patch
>
>
> hbase.regionserver.executor.openregion.threads helps in improving MTTR by 
> increasing assign rpc processing rate at RS from HMaster but is not 
> documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20716) Unsafe access cleanup

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654544#comment-16654544
 ] 

Sahil Aggarwal commented on HBASE-20716:


[~stack] Good to go if you don't have any comments/suggestions. 

> Unsafe access cleanup
> -
>
> Key: HBASE-20716
> URL: https://issues.apache.org/jira/browse/HBASE-20716
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: Sahil Aggarwal
>Priority: Critical
>  Labels: beginner
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-20716.master.001.patch, 
> HBASE-20716.master.002.patch, HBASE-20716.master.003.patch, 
> HBASE-20716.master.004.patch, HBASE-20716.master.005.patch, 
> HBASE-20716.master.006.patch, HBASE-20716.master.007.patch, 
> HBASE-20716.master.008.patch, Screen Shot 2018-06-26 at 11.37.49 AM.png
>
>
> We have two means of getting at unsafe; UnsafeAccess and then internal to the 
> Bytes class. They are effectively doing the same thing. We should have one 
> avenue to Unsafe only.
> Many of our paths to Unsafe via UnsafeAccess traverse flags to check if 
> access is available, if it is aligned and the order in which words are 
> written on the machine. Each check costs -- especially if done millions of 
> times a second -- and on occasion adds bloat in hot code paths. The unsafe 
> access inside Bytes checks on startup what the machine is capable off and 
> then does a static assign of the appropriate class-to-use from there on out. 
> UnsafeAccess does not do this running the checks everytime. Would be good to 
> have the Bytes behavior pervasive.
> The benefit of one access to Unsafe only is plain. The benefits we gain 
> removing checks will be harder to measure though should be plain when you 
> disassemble a hot-path; in a (very) rare case, the saved byte codes could be 
> the difference between inlining or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20716) Unsafe access cleanup

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-20716:
---
Attachment: HBASE-20716.master.008.patch

> Unsafe access cleanup
> -
>
> Key: HBASE-20716
> URL: https://issues.apache.org/jira/browse/HBASE-20716
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: Sahil Aggarwal
>Priority: Critical
>  Labels: beginner
> Attachments: HBASE-20716.master.001.patch, 
> HBASE-20716.master.002.patch, HBASE-20716.master.003.patch, 
> HBASE-20716.master.004.patch, HBASE-20716.master.005.patch, 
> HBASE-20716.master.006.patch, HBASE-20716.master.007.patch, 
> HBASE-20716.master.008.patch, Screen Shot 2018-06-26 at 11.37.49 AM.png
>
>
> We have two means of getting at unsafe; UnsafeAccess and then internal to the 
> Bytes class. They are effectively doing the same thing. We should have one 
> avenue to Unsafe only.
> Many of our paths to Unsafe via UnsafeAccess traverse flags to check if 
> access is available, if it is aligned and the order in which words are 
> written on the machine. Each check costs -- especially if done millions of 
> times a second -- and on occasion adds bloat in hot code paths. The unsafe 
> access inside Bytes checks on startup what the machine is capable off and 
> then does a static assign of the appropriate class-to-use from there on out. 
> UnsafeAccess does not do this running the checks everytime. Would be good to 
> have the Bytes behavior pervasive.
> The benefit of one access to Unsafe only is plain. The benefits we gain 
> removing checks will be harder to measure though should be plain when you 
> disassemble a hot-path; in a (very) rare case, the saved byte codes could be 
> the difference between inlining or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20716) Unsafe access cleanup

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-20716:
---
Attachment: HBASE-20716.master.007.patch

> Unsafe access cleanup
> -
>
> Key: HBASE-20716
> URL: https://issues.apache.org/jira/browse/HBASE-20716
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: Sahil Aggarwal
>Priority: Critical
>  Labels: beginner
> Attachments: HBASE-20716.master.001.patch, 
> HBASE-20716.master.002.patch, HBASE-20716.master.003.patch, 
> HBASE-20716.master.004.patch, HBASE-20716.master.005.patch, 
> HBASE-20716.master.006.patch, HBASE-20716.master.007.patch, Screen Shot 
> 2018-06-26 at 11.37.49 AM.png
>
>
> We have two means of getting at unsafe; UnsafeAccess and then internal to the 
> Bytes class. They are effectively doing the same thing. We should have one 
> avenue to Unsafe only.
> Many of our paths to Unsafe via UnsafeAccess traverse flags to check if 
> access is available, if it is aligned and the order in which words are 
> written on the machine. Each check costs -- especially if done millions of 
> times a second -- and on occasion adds bloat in hot code paths. The unsafe 
> access inside Bytes checks on startup what the machine is capable off and 
> then does a static assign of the appropriate class-to-use from there on out. 
> UnsafeAccess does not do this running the checks everytime. Would be good to 
> have the Bytes behavior pervasive.
> The benefit of one access to Unsafe only is plain. The benefits we gain 
> removing checks will be harder to measure though should be plain when you 
> disassemble a hot-path; in a (very) rare case, the saved byte codes could be 
> the difference between inlining or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20716) Unsafe access cleanup

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-20716:
---
Attachment: HBASE-20716.master.006.patch

> Unsafe access cleanup
> -
>
> Key: HBASE-20716
> URL: https://issues.apache.org/jira/browse/HBASE-20716
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: Sahil Aggarwal
>Priority: Critical
>  Labels: beginner
> Attachments: HBASE-20716.master.001.patch, 
> HBASE-20716.master.002.patch, HBASE-20716.master.003.patch, 
> HBASE-20716.master.004.patch, HBASE-20716.master.005.patch, 
> HBASE-20716.master.006.patch, Screen Shot 2018-06-26 at 11.37.49 AM.png
>
>
> We have two means of getting at unsafe; UnsafeAccess and then internal to the 
> Bytes class. They are effectively doing the same thing. We should have one 
> avenue to Unsafe only.
> Many of our paths to Unsafe via UnsafeAccess traverse flags to check if 
> access is available, if it is aligned and the order in which words are 
> written on the machine. Each check costs -- especially if done millions of 
> times a second -- and on occasion adds bloat in hot code paths. The unsafe 
> access inside Bytes checks on startup what the machine is capable off and 
> then does a static assign of the appropriate class-to-use from there on out. 
> UnsafeAccess does not do this running the checks everytime. Would be good to 
> have the Bytes behavior pervasive.
> The benefit of one access to Unsafe only is plain. The benefits we gain 
> removing checks will be harder to measure though should be plain when you 
> disassemble a hot-path; in a (very) rare case, the saved byte codes could be 
> the difference between inlining or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21186) Document hbase.regionserver.executor.openregion.threads in MTTR section

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640591#comment-16640591
 ] 

Sahil Aggarwal commented on HBASE-21186:


Will do that Josh. On RS which was hosting around 800 regions, i had it set to 
200 to get the MTTR of 2-3mins. I think having large value of this executor 
pool should be harmless as on getting the work it will wakeup only 1 thread at 
a time and threads will be lying unused most of the time.

What would you recommend recommending users here?

> Document hbase.regionserver.executor.openregion.threads in MTTR section
> ---
>
> Key: HBASE-21186
> URL: https://issues.apache.org/jira/browse/HBASE-21186
> Project: HBase
>  Issue Type: Improvement
>  Components: documentation
>Reporter: Sahil Aggarwal
>Assignee: Sahil Aggarwal
>Priority: Minor
> Attachments: HBASE-21186.master.001.patch, 
> HBASE-21186.master.002.patch
>
>
> hbase.regionserver.executor.openregion.threads helps in improving MTTR by 
> increasing assign rpc processing rate at RS from HMaster but is not 
> documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20716) Unsafe access cleanup

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640583#comment-16640583
 ] 

Sahil Aggarwal commented on HBASE-20716:


Fixed the direct unsafe initialization and moved to FQCN way. 

-- What are you thinking? One class used by Bytes and BBUtil that does all 
unsafe? Or something else? It would be sweet if just one conduit to unsafe

   - I was thinking just not to expose UnsafeAccess.theUnsafe and let 
everything go through UnsafeAccess itself. For eg removing following such 
invocations:

   UnsafeAccess.theUnsafe.getLong(obj1, o1 + (long) i);

   UnsafeAccess.theUnsafe.getByte(obj1, o1 + i) & 0xFF);

 

 

> Unsafe access cleanup
> -
>
> Key: HBASE-20716
> URL: https://issues.apache.org/jira/browse/HBASE-20716
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: Sahil Aggarwal
>Priority: Critical
>  Labels: beginner
> Attachments: HBASE-20716.master.001.patch, 
> HBASE-20716.master.002.patch, HBASE-20716.master.003.patch, 
> HBASE-20716.master.004.patch, HBASE-20716.master.005.patch, Screen Shot 
> 2018-06-26 at 11.37.49 AM.png
>
>
> We have two means of getting at unsafe; UnsafeAccess and then internal to the 
> Bytes class. They are effectively doing the same thing. We should have one 
> avenue to Unsafe only.
> Many of our paths to Unsafe via UnsafeAccess traverse flags to check if 
> access is available, if it is aligned and the order in which words are 
> written on the machine. Each check costs -- especially if done millions of 
> times a second -- and on occasion adds bloat in hot code paths. The unsafe 
> access inside Bytes checks on startup what the machine is capable off and 
> then does a static assign of the appropriate class-to-use from there on out. 
> UnsafeAccess does not do this running the checks everytime. Would be good to 
> have the Bytes behavior pervasive.
> The benefit of one access to Unsafe only is plain. The benefits we gain 
> removing checks will be harder to measure though should be plain when you 
> disassemble a hot-path; in a (very) rare case, the saved byte codes could be 
> the difference between inlining or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20716) Unsafe access cleanup

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-20716:
---
Attachment: HBASE-20716.master.005.patch

> Unsafe access cleanup
> -
>
> Key: HBASE-20716
> URL: https://issues.apache.org/jira/browse/HBASE-20716
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: Sahil Aggarwal
>Priority: Critical
>  Labels: beginner
> Attachments: HBASE-20716.master.001.patch, 
> HBASE-20716.master.002.patch, HBASE-20716.master.003.patch, 
> HBASE-20716.master.004.patch, HBASE-20716.master.005.patch, Screen Shot 
> 2018-06-26 at 11.37.49 AM.png
>
>
> We have two means of getting at unsafe; UnsafeAccess and then internal to the 
> Bytes class. They are effectively doing the same thing. We should have one 
> avenue to Unsafe only.
> Many of our paths to Unsafe via UnsafeAccess traverse flags to check if 
> access is available, if it is aligned and the order in which words are 
> written on the machine. Each check costs -- especially if done millions of 
> times a second -- and on occasion adds bloat in hot code paths. The unsafe 
> access inside Bytes checks on startup what the machine is capable off and 
> then does a static assign of the appropriate class-to-use from there on out. 
> UnsafeAccess does not do this running the checks everytime. Would be good to 
> have the Bytes behavior pervasive.
> The benefit of one access to Unsafe only is plain. The benefits we gain 
> removing checks will be harder to measure though should be plain when you 
> disassemble a hot-path; in a (very) rare case, the saved byte codes could be 
> the difference between inlining or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20716) Unsafe access cleanup

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633258#comment-16633258
 ] 

Sahil Aggarwal commented on HBASE-20716:


If i understood it correctly, problem is direct static initialization of Unsafe 
object in Bytes.UnsafeConverter which can lead to class loading attempt for 
Unsafe class when not available, will change it to FQCN way.

Since in BBUtils we don't use Unsafe directly, are you suggesting removing any 
direct ref in Bytes class too?

> Unsafe access cleanup
> -
>
> Key: HBASE-20716
> URL: https://issues.apache.org/jira/browse/HBASE-20716
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: Sahil Aggarwal
>Priority: Critical
>  Labels: beginner
> Attachments: HBASE-20716.master.001.patch, 
> HBASE-20716.master.002.patch, HBASE-20716.master.003.patch, 
> HBASE-20716.master.004.patch, Screen Shot 2018-06-26 at 11.37.49 AM.png
>
>
> We have two means of getting at unsafe; UnsafeAccess and then internal to the 
> Bytes class. They are effectively doing the same thing. We should have one 
> avenue to Unsafe only.
> Many of our paths to Unsafe via UnsafeAccess traverse flags to check if 
> access is available, if it is aligned and the order in which words are 
> written on the machine. Each check costs -- especially if done millions of 
> times a second -- and on occasion adds bloat in hot code paths. The unsafe 
> access inside Bytes checks on startup what the machine is capable off and 
> then does a static assign of the appropriate class-to-use from there on out. 
> UnsafeAccess does not do this running the checks everytime. Would be good to 
> have the Bytes behavior pervasive.
> The benefit of one access to Unsafe only is plain. The benefits we gain 
> removing checks will be harder to measure though should be plain when you 
> disassemble a hot-path; in a (very) rare case, the saved byte codes could be 
> the difference between inlining or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21186) Document hbase.regionserver.executor.openregion.threads in MTTR section

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631331#comment-16631331
 ] 

Sahil Aggarwal commented on HBASE-21186:


Done.

> Document hbase.regionserver.executor.openregion.threads in MTTR section
> ---
>
> Key: HBASE-21186
> URL: https://issues.apache.org/jira/browse/HBASE-21186
> Project: HBase
>  Issue Type: Improvement
>  Components: documentation
>Reporter: Sahil Aggarwal
>Assignee: Sahil Aggarwal
>Priority: Minor
> Attachments: HBASE-21186.master.001.patch, 
> HBASE-21186.master.002.patch
>
>
> hbase.regionserver.executor.openregion.threads helps in improving MTTR by 
> increasing assign rpc processing rate at RS from HMaster but is not 
> documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-21186) Document hbase.regionserver.executor.openregion.threads in MTTR section

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-21186:
---
Attachment: HBASE-21186.master.002.patch

> Document hbase.regionserver.executor.openregion.threads in MTTR section
> ---
>
> Key: HBASE-21186
> URL: https://issues.apache.org/jira/browse/HBASE-21186
> Project: HBase
>  Issue Type: Improvement
>  Components: documentation
>Reporter: Sahil Aggarwal
>Assignee: Sahil Aggarwal
>Priority: Minor
> Attachments: HBASE-21186.master.001.patch, 
> HBASE-21186.master.002.patch
>
>
> hbase.regionserver.executor.openregion.threads helps in improving MTTR by 
> increasing assign rpc processing rate at RS from HMaster but is not 
> documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20716) Unsafe access cleanup

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631312#comment-16631312
 ] 

Sahil Aggarwal commented on HBASE-20716:


[~stack] Can you please have a look at the patch?

> Unsafe access cleanup
> -
>
> Key: HBASE-20716
> URL: https://issues.apache.org/jira/browse/HBASE-20716
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: Sahil Aggarwal
>Priority: Critical
>  Labels: beginner
> Attachments: HBASE-20716.master.001.patch, 
> HBASE-20716.master.002.patch, HBASE-20716.master.003.patch, 
> HBASE-20716.master.004.patch, Screen Shot 2018-06-26 at 11.37.49 AM.png
>
>
> We have two means of getting at unsafe; UnsafeAccess and then internal to the 
> Bytes class. They are effectively doing the same thing. We should have one 
> avenue to Unsafe only.
> Many of our paths to Unsafe via UnsafeAccess traverse flags to check if 
> access is available, if it is aligned and the order in which words are 
> written on the machine. Each check costs -- especially if done millions of 
> times a second -- and on occasion adds bloat in hot code paths. The unsafe 
> access inside Bytes checks on startup what the machine is capable off and 
> then does a static assign of the appropriate class-to-use from there on out. 
> UnsafeAccess does not do this running the checks everytime. Would be good to 
> have the Bytes behavior pervasive.
> The benefit of one access to Unsafe only is plain. The benefits we gain 
> removing checks will be harder to measure though should be plain when you 
> disassemble a hot-path; in a (very) rare case, the saved byte codes could be 
> the difference between inlining or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21186) Document hbase.regionserver.executor.openregion.threads in MTTR section

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611992#comment-16611992
 ] 

Sahil Aggarwal commented on HBASE-21186:


[~anoop.hbase] Can you please see if this makes sense ?

> Document hbase.regionserver.executor.openregion.threads in MTTR section
> ---
>
> Key: HBASE-21186
> URL: https://issues.apache.org/jira/browse/HBASE-21186
> Project: HBase
>  Issue Type: Improvement
>  Components: documentation
>Reporter: Sahil Aggarwal
>Assignee: Sahil Aggarwal
>Priority: Minor
> Attachments: HBASE-21186.master.001.patch
>
>
> hbase.regionserver.executor.openregion.threads helps in improving MTTR by 
> increasing assign rpc processing rate at RS from HMaster but is not 
> documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work started] (HBASE-21186) Document hbase.regionserver.executor.openregion.threads in MTTR section

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HBASE-21186 started by Sahil Aggarwal.
--
> Document hbase.regionserver.executor.openregion.threads in MTTR section
> ---
>
> Key: HBASE-21186
> URL: https://issues.apache.org/jira/browse/HBASE-21186
> Project: HBase
>  Issue Type: Improvement
>  Components: documentation
>Reporter: Sahil Aggarwal
>Assignee: Sahil Aggarwal
>Priority: Minor
> Attachments: HBASE-21186.master.001.patch
>
>
> hbase.regionserver.executor.openregion.threads helps in improving MTTR by 
> increasing assign rpc processing rate at RS from HMaster but is not 
> documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-21186) Document hbase.regionserver.executor.openregion.threads in MTTR section

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-21186:
---
Attachment: HBASE-21186.master.001.patch

> Document hbase.regionserver.executor.openregion.threads in MTTR section
> ---
>
> Key: HBASE-21186
> URL: https://issues.apache.org/jira/browse/HBASE-21186
> Project: HBase
>  Issue Type: Improvement
>  Components: documentation
>Reporter: Sahil Aggarwal
>Assignee: Sahil Aggarwal
>Priority: Minor
> Attachments: HBASE-21186.master.001.patch
>
>
> hbase.regionserver.executor.openregion.threads helps in improving MTTR by 
> increasing assign rpc processing rate at RS from HMaster but is not 
> documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-21186) Document hbase.regionserver.executor.openregion.threads in MTTR section

[Sahil Aggarwal (JIRA)]

Sahil Aggarwal created HBASE-21186:
--

 Summary: Document hbase.regionserver.executor.openregion.threads 
in MTTR section
 Key: HBASE-21186
 URL: https://issues.apache.org/jira/browse/HBASE-21186
 Project: HBase
  Issue Type: Improvement
  Components: documentation
Reporter: Sahil Aggarwal
Assignee: Sahil Aggarwal


hbase.regionserver.executor.openregion.threads helps in improving MTTR by 
increasing assign rpc processing rate at RS from HMaster but is not documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19316) Direct invocation Client-Server short-circuit without having to pass through the eye of a protobuf stub

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562396#comment-16562396
 ] 

Sahil Aggarwal commented on HBASE-19316:


[~stack] the description you added finally lead me to 
ShortCircuitingClusterConnection in ConnectionUtils. Now, what i understood is 
that it still uses protobuf service stubs for admin and client services and 
also returns the same for master. Here, probably we don't need to go through 
protobuf and can bypass it. 

If i got it right, any top of the head possible ways here?  As I see that it is 
tied to the ClusterConnection interface which has methods declared to return 
such stubs. Connection seem to be independent of such stubs and can be easily 
decoupled i guess.

 

Thanks!

 

> Direct invocation Client-Server short-circuit without having to pass through 
> the eye of a protobuf stub
> ---
>
> Key: HBASE-19316
> URL: https://issues.apache.org/jira/browse/HBASE-19316
> Project: HBase
>  Issue Type: Improvement
>  Components: rpc
>Reporter: stack
>Priority: Major
>  Labels: beginner
>
> In hbase, on server-side, we have a short-circuit facility that bypasses RPC 
> by directly hooking the client and server protobuf Services up to each other.
> Passing through the Protobuf Service stub requires that the invocation be 
> cast as protobufs -- the invocation itself and all params are converted to 
> protobuf to pass through the eye of the protobuf Service stub. Can we do 
> better and make direct invocations w/o having to do the protobuf 
> marshalling/unmarshalling? (Can we do it in a way that is not brittle in need 
> of careful repair whenever a change is made?). It would make for some nice 
> savings.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20731) Incorrect folders in documentation

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561745#comment-16561745
 ] 

Sahil Aggarwal commented on HBASE-20731:


Thanks [~psomogyi] for quick review and push.

> Incorrect folders in documentation
> --
>
> Key: HBASE-20731
> URL: https://issues.apache.org/jira/browse/HBASE-20731
> Project: HBase
>  Issue Type: Bug
>  Components: documentation
>Reporter: Peter Somogyi
>Assignee: Sahil Aggarwal
>Priority: Minor
>  Labels: beginner
> Fix For: 3.0.0
>
> Attachments: HBASE-20731.master.001.patch
>
>
> Unexpected Filesystem Growth chapter in Reference Guide mentions .snapshots 
> and .archive directories. Both of these were changed long ago.
>  /hbase/.snapshots -> /hbase/.hbase-snapshot
>  /hbase/.archive -> /hbase/archive
> [https://hbase.apache.org/book.html#_unexpected_filesystem_growth]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20731) Incorrect folders in documentation

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-20731:
---
Attachment: HBASE-20731.master.001.patch

> Incorrect folders in documentation
> --
>
> Key: HBASE-20731
> URL: https://issues.apache.org/jira/browse/HBASE-20731
> Project: HBase
>  Issue Type: Bug
>  Components: documentation
>Reporter: Peter Somogyi
>Priority: Minor
>  Labels: beginner
> Attachments: HBASE-20731.master.001.patch
>
>
> Unexpected Filesystem Growth chapter in Reference Guide mentions .snapshots 
> and .archive directories. Both of these were changed long ago.
>  /hbase/.snapshots -> /hbase/.hbase-snapshot
>  /hbase/.archive -> /hbase/archive
> [https://hbase.apache.org/book.html#_unexpected_filesystem_growth]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20753) reference guide should direct security related issues to priv...@hbase.apache.org

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-20753:
---
Attachment: HBASE-20753.master.001.patch

> reference guide should direct security related issues to 
> priv...@hbase.apache.org
> -
>
> Key: HBASE-20753
> URL: https://issues.apache.org/jira/browse/HBASE-20753
> Project: HBase
>  Issue Type: Bug
>  Components: documentation, security
>Reporter: Sean Busbey
>Priority: Critical
>  Labels: beginner
> Attachments: HBASE-20753.master.001.patch
>
>
> the reference guide currently directs folks to send security issues to 
> priv...@apache.org:
> {quote}
> To protect existing HBase installations from new vulnerabilities, please do 
> not use JIRA to report security-related bugs. Instead, send your report to 
> the mailing list priv...@apache.org, which allows anyone to send messages, 
> but restricts who can read them. Someone on that list will contact you to 
> follow up on your report.
> {quote}
> This address does not exist. It should tell folks to send the email to 
> priv...@hbase.apache.org.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20766) Verify Replication Tool Has Typo "remove cluster"

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-20766:
---
Attachment: HBASE-20766.master.001.patch

> Verify Replication Tool Has Typo "remove cluster"
> -
>
> Key: HBASE-20766
> URL: https://issues.apache.org/jira/browse/HBASE-20766
> Project: HBase
>  Issue Type: Bug
>Reporter: Clay B.
>Priority: Trivial
>  Labels: beginner
> Attachments: HBASE-20766.master.001.patch
>
>
> The verify replication tool has a trivial typo "remove cluster" instead of 
> "remote cluster": 
> https://github.com/apache/hbase/blob/a6eeb26cc0b4d0af3fff50b5b931b6847df1f9d2/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/replication/VerifyReplication.java#L355



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20851) Change rubocop config for max line length of 100

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561009#comment-16561009
 ] 

Sahil Aggarwal commented on HBASE-20851:


Running rubocop currently shows many offenses and even after adding above 
configuration there multiple line length exceeding offenses. Scope of this task 
is just to add the config or even fix the violations?

> Change rubocop config for max line length of 100
> 
>
> Key: HBASE-20851
> URL: https://issues.apache.org/jira/browse/HBASE-20851
> Project: HBase
>  Issue Type: Bug
>  Components: community, shell
>Affects Versions: 2.0.1
>Reporter: Umesh Agashe
>Priority: Minor
>  Labels: beginner, beginners
>
> Existing ruby and Java code uses max line length of 100 characters. Change 
> rubocop config with:
> {code:java}
> Metrics/LineLength:
>   Max: 100
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20716) Unsafe access cleanup

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560802#comment-16560802
 ] 

Sahil Aggarwal commented on HBASE-20716:


[~anoop.hbase] Changed the BBUtils too. 

> Unsafe access cleanup
> -
>
> Key: HBASE-20716
> URL: https://issues.apache.org/jira/browse/HBASE-20716
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: Sahil Aggarwal
>Priority: Critical
>  Labels: beginner
> Attachments: HBASE-20716.master.001.patch, 
> HBASE-20716.master.002.patch, HBASE-20716.master.003.patch, 
> HBASE-20716.master.004.patch, Screen Shot 2018-06-26 at 11.37.49 AM.png
>
>
> We have two means of getting at unsafe; UnsafeAccess and then internal to the 
> Bytes class. They are effectively doing the same thing. We should have one 
> avenue to Unsafe only.
> Many of our paths to Unsafe via UnsafeAccess traverse flags to check if 
> access is available, if it is aligned and the order in which words are 
> written on the machine. Each check costs -- especially if done millions of 
> times a second -- and on occasion adds bloat in hot code paths. The unsafe 
> access inside Bytes checks on startup what the machine is capable off and 
> then does a static assign of the appropriate class-to-use from there on out. 
> UnsafeAccess does not do this running the checks everytime. Would be good to 
> have the Bytes behavior pervasive.
> The benefit of one access to Unsafe only is plain. The benefits we gain 
> removing checks will be harder to measure though should be plain when you 
> disassemble a hot-path; in a (very) rare case, the saved byte codes could be 
> the difference between inlining or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20716) Unsafe access cleanup

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-20716:
---
Attachment: HBASE-20716.master.004.patch

> Unsafe access cleanup
> -
>
> Key: HBASE-20716
> URL: https://issues.apache.org/jira/browse/HBASE-20716
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: Sahil Aggarwal
>Priority: Critical
>  Labels: beginner
> Attachments: HBASE-20716.master.001.patch, 
> HBASE-20716.master.002.patch, HBASE-20716.master.003.patch, 
> HBASE-20716.master.004.patch, Screen Shot 2018-06-26 at 11.37.49 AM.png
>
>
> We have two means of getting at unsafe; UnsafeAccess and then internal to the 
> Bytes class. They are effectively doing the same thing. We should have one 
> avenue to Unsafe only.
> Many of our paths to Unsafe via UnsafeAccess traverse flags to check if 
> access is available, if it is aligned and the order in which words are 
> written on the machine. Each check costs -- especially if done millions of 
> times a second -- and on occasion adds bloat in hot code paths. The unsafe 
> access inside Bytes checks on startup what the machine is capable off and 
> then does a static assign of the appropriate class-to-use from there on out. 
> UnsafeAccess does not do this running the checks everytime. Would be good to 
> have the Bytes behavior pervasive.
> The benefit of one access to Unsafe only is plain. The benefits we gain 
> removing checks will be harder to measure though should be plain when you 
> disassemble a hot-path; in a (very) rare case, the saved byte codes could be 
> the difference between inlining or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20716) Unsafe access cleanup

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555407#comment-16555407
 ] 

Sahil Aggarwal commented on HBASE-20716:


I intend to finish it all in this tasks itself. To me ByteBufferUtils seems to 
be only other class where we do this check and dispatch. Will changing the 
ByteBufferUtils too be all for this task?

> Unsafe access cleanup
> -
>
> Key: HBASE-20716
> URL: https://issues.apache.org/jira/browse/HBASE-20716
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: Sahil Aggarwal
>Priority: Critical
>  Labels: beginner
> Attachments: HBASE-20716.master.001.patch, 
> HBASE-20716.master.002.patch, HBASE-20716.master.003.patch, Screen Shot 
> 2018-06-26 at 11.37.49 AM.png
>
>
> We have two means of getting at unsafe; UnsafeAccess and then internal to the 
> Bytes class. They are effectively doing the same thing. We should have one 
> avenue to Unsafe only.
> Many of our paths to Unsafe via UnsafeAccess traverse flags to check if 
> access is available, if it is aligned and the order in which words are 
> written on the machine. Each check costs -- especially if done millions of 
> times a second -- and on occasion adds bloat in hot code paths. The unsafe 
> access inside Bytes checks on startup what the machine is capable off and 
> then does a static assign of the appropriate class-to-use from there on out. 
> UnsafeAccess does not do this running the checks everytime. Would be good to 
> have the Bytes behavior pervasive.
> The benefit of one access to Unsafe only is plain. The benefits we gain 
> removing checks will be harder to measure though should be plain when you 
> disassemble a hot-path; in a (very) rare case, the saved byte codes could be 
> the difference between inlining or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20716) Unsafe access cleanup

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16528539#comment-16528539
 ] 

Sahil Aggarwal commented on HBASE-20716:


Thanks [~stack] [~anoop.hbase] for all the prompt reviews. Have attached the 
patch which includes conversion of other types too.

> Unsafe access cleanup
> -
>
> Key: HBASE-20716
> URL: https://issues.apache.org/jira/browse/HBASE-20716
> Project: HBase
>  Issue Type: Improvement
>  Components: Performance
>Reporter: stack
>Assignee: Sahil Aggarwal
>Priority: Critical
>  Labels: beginner
> Attachments: HBASE-20716.master.001.patch, 
> HBASE-20716.master.002.patch, HBASE-20716.master.003.patch, Screen Shot 
> 2018-06-26 at 11.37.49 AM.png
>
>
> We have two means of getting at unsafe; UnsafeAccess and then internal to the 
> Bytes class. They are effectively doing the same thing. We should have one 
> avenue to Unsafe only.
> Many of our paths to Unsafe via UnsafeAccess traverse flags to check if 
> access is available, if it is aligned and the order in which words are 
> written on the machine. Each check costs -- especially if done millions of 
> times a second -- and on occasion adds bloat in hot code paths. The unsafe 
> access inside Bytes checks on startup what the machine is capable off and 
> then does a static assign of the appropriate class-to-use from there on out. 
> UnsafeAccess does not do this running the checks everytime. Would be good to 
> have the Bytes behavior pervasive.
> The benefit of one access to Unsafe only is plain. The benefits we gain 
> removing checks will be harder to measure though should be plain when you 
> disassemble a hot-path; in a (very) rare case, the saved byte codes could be 
> the difference between inlining or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20716) Unsafe access cleanup

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-20716:
---
Attachment: HBASE-20716.master.003.patch

> Unsafe access cleanup
> -
>
> Key: HBASE-20716
> URL: https://issues.apache.org/jira/browse/HBASE-20716
> Project: HBase
>  Issue Type: Improvement
>  Components: Performance
>Reporter: stack
>Assignee: Sahil Aggarwal
>Priority: Critical
>  Labels: beginner
> Attachments: HBASE-20716.master.001.patch, 
> HBASE-20716.master.002.patch, HBASE-20716.master.003.patch, Screen Shot 
> 2018-06-26 at 11.37.49 AM.png
>
>
> We have two means of getting at unsafe; UnsafeAccess and then internal to the 
> Bytes class. They are effectively doing the same thing. We should have one 
> avenue to Unsafe only.
> Many of our paths to Unsafe via UnsafeAccess traverse flags to check if 
> access is available, if it is aligned and the order in which words are 
> written on the machine. Each check costs -- especially if done millions of 
> times a second -- and on occasion adds bloat in hot code paths. The unsafe 
> access inside Bytes checks on startup what the machine is capable off and 
> then does a static assign of the appropriate class-to-use from there on out. 
> UnsafeAccess does not do this running the checks everytime. Would be good to 
> have the Bytes behavior pervasive.
> The benefit of one access to Unsafe only is plain. The benefits we gain 
> removing checks will be harder to measure though should be plain when you 
> disassemble a hot-path; in a (very) rare case, the saved byte codes could be 
> the difference between inlining or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20716) Unsafe access cleanup

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-20716:
---
Attachment: HBASE-20716.master.002.patch

> Unsafe access cleanup
> -
>
> Key: HBASE-20716
> URL: https://issues.apache.org/jira/browse/HBASE-20716
> Project: HBase
>  Issue Type: Improvement
>  Components: Performance
>Reporter: stack
>Assignee: Sahil Aggarwal
>Priority: Critical
>  Labels: beginner
> Attachments: HBASE-20716.master.001.patch, 
> HBASE-20716.master.002.patch, Screen Shot 2018-06-26 at 11.37.49 AM.png
>
>
> We have two means of getting at unsafe; UnsafeAccess and then internal to the 
> Bytes class. They are effectively doing the same thing. We should have one 
> avenue to Unsafe only.
> Many of our paths to Unsafe via UnsafeAccess traverse flags to check if 
> access is available, if it is aligned and the order in which words are 
> written on the machine. Each check costs -- especially if done millions of 
> times a second -- and on occasion adds bloat in hot code paths. The unsafe 
> access inside Bytes checks on startup what the machine is capable off and 
> then does a static assign of the appropriate class-to-use from there on out. 
> UnsafeAccess does not do this running the checks everytime. Would be good to 
> have the Bytes behavior pervasive.
> The benefit of one access to Unsafe only is plain. The benefits we gain 
> removing checks will be harder to measure though should be plain when you 
> disassemble a hot-path; in a (very) rare case, the saved byte codes could be 
> the difference between inlining or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20716) Unsafe access cleanup

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527637#comment-16527637
 ] 

Sahil Aggarwal commented on HBASE-20716:


UnsafeAccess#LITTLE_ENDIAN and Bytes#UNSAFE_UNALIGNED are already final in 
master. Initially I had JMH'd the 2 versions: one without any check(direct 
dispatch) and another with check on final boolean which showed same performance 
just to see if removing the check and doing the direct call gives any perf 
benefit. But the both the booleans were already final.

 

HbaseBytes.testToLongCheckAndDispatch is exactly what we are doing currently.

> Unsafe access cleanup
> -
>
> Key: HBASE-20716
> URL: https://issues.apache.org/jira/browse/HBASE-20716
> Project: HBase
>  Issue Type: Improvement
>  Components: Performance
>Reporter: stack
>Assignee: Sahil Aggarwal
>Priority: Critical
>  Labels: beginner
> Attachments: HBASE-20716.master.001.patch, Screen Shot 2018-06-26 at 
> 11.37.49 AM.png
>
>
> We have two means of getting at unsafe; UnsafeAccess and then internal to the 
> Bytes class. They are effectively doing the same thing. We should have one 
> avenue to Unsafe only.
> Many of our paths to Unsafe via UnsafeAccess traverse flags to check if 
> access is available, if it is aligned and the order in which words are 
> written on the machine. Each check costs -- especially if done millions of 
> times a second -- and on occasion adds bloat in hot code paths. The unsafe 
> access inside Bytes checks on startup what the machine is capable off and 
> then does a static assign of the appropriate class-to-use from there on out. 
> UnsafeAccess does not do this running the checks everytime. Would be good to 
> have the Bytes behavior pervasive.
> The benefit of one access to Unsafe only is plain. The benefits we gain 
> removing checks will be harder to measure though should be plain when you 
> disassemble a hot-path; in a (very) rare case, the saved byte codes could be 
> the difference between inlining or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20716) Unsafe access cleanup

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527215#comment-16527215
 ] 

Sahil Aggarwal commented on HBASE-20716:


Benchmark                                                       Mode      Cnt   
 Score  Error  Units

HbaseBytes.testToLongCheckAndDispatch     thrpt        6       352925235.054 ± 
7423950.697  ops/s

HbaseBytes.testToLongStaticInvoke                thrpt        6       
353652817.747 ± 6536569.256  ops/s

 

org.apache.hadoop.hbase.util.Bytes::toLongCheckAndDispatch (73 bytes)    - 
current impl

org.apache.hadoop.hbase.util.Bytes::toLong (33 bytes)   - impl using 
BEST_CONVERTER

Reduced the size of method by 40 bytes.

> Unsafe access cleanup
> -
>
> Key: HBASE-20716
> URL: https://issues.apache.org/jira/browse/HBASE-20716
> Project: HBase
>  Issue Type: Improvement
>  Components: Performance
>Reporter: stack
>Assignee: Sahil Aggarwal
>Priority: Critical
>  Labels: beginner
> Attachments: HBASE-20716.master.001.patch, Screen Shot 2018-06-26 at 
> 11.37.49 AM.png
>
>
> We have two means of getting at unsafe; UnsafeAccess and then internal to the 
> Bytes class. They are effectively doing the same thing. We should have one 
> avenue to Unsafe only.
> Many of our paths to Unsafe via UnsafeAccess traverse flags to check if 
> access is available, if it is aligned and the order in which words are 
> written on the machine. Each check costs -- especially if done millions of 
> times a second -- and on occasion adds bloat in hot code paths. The unsafe 
> access inside Bytes checks on startup what the machine is capable off and 
> then does a static assign of the appropriate class-to-use from there on out. 
> UnsafeAccess does not do this running the checks everytime. Would be good to 
> have the Bytes behavior pervasive.
> The benefit of one access to Unsafe only is plain. The benefits we gain 
> removing checks will be harder to measure though should be plain when you 
> disassemble a hot-path; in a (very) rare case, the saved byte codes could be 
> the difference between inlining or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20716) Unsafe access cleanup

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526462#comment-16526462
 ] 

Sahil Aggarwal commented on HBASE-20716:


Added patch for just long conversion. Can someone verify if I am heading in 
right direction? If yes, will do the JMH on it to see the actual gains or any 
additional cost.

> Unsafe access cleanup
> -
>
> Key: HBASE-20716
> URL: https://issues.apache.org/jira/browse/HBASE-20716
> Project: HBase
>  Issue Type: Improvement
>  Components: Performance
>Reporter: stack
>Assignee: Sahil Aggarwal
>Priority: Critical
>  Labels: beginner
> Attachments: HBASE-20716.master.001.patch, Screen Shot 2018-06-26 at 
> 11.37.49 AM.png
>
>
> We have two means of getting at unsafe; UnsafeAccess and then internal to the 
> Bytes class. They are effectively doing the same thing. We should have one 
> avenue to Unsafe only.
> Many of our paths to Unsafe via UnsafeAccess traverse flags to check if 
> access is available, if it is aligned and the order in which words are 
> written on the machine. Each check costs -- especially if done millions of 
> times a second -- and on occasion adds bloat in hot code paths. The unsafe 
> access inside Bytes checks on startup what the machine is capable off and 
> then does a static assign of the appropriate class-to-use from there on out. 
> UnsafeAccess does not do this running the checks everytime. Would be good to 
> have the Bytes behavior pervasive.
> The benefit of one access to Unsafe only is plain. The benefits we gain 
> removing checks will be harder to measure though should be plain when you 
> disassemble a hot-path; in a (very) rare case, the saved byte codes could be 
> the difference between inlining or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20716) Unsafe access cleanup

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-20716:
---
Attachment: HBASE-20716.master.001.patch

> Unsafe access cleanup
> -
>
> Key: HBASE-20716
> URL: https://issues.apache.org/jira/browse/HBASE-20716
> Project: HBase
>  Issue Type: Improvement
>  Components: Performance
>Reporter: stack
>Assignee: Sahil Aggarwal
>Priority: Critical
>  Labels: beginner
> Attachments: HBASE-20716.master.001.patch, Screen Shot 2018-06-26 at 
> 11.37.49 AM.png
>
>
> We have two means of getting at unsafe; UnsafeAccess and then internal to the 
> Bytes class. They are effectively doing the same thing. We should have one 
> avenue to Unsafe only.
> Many of our paths to Unsafe via UnsafeAccess traverse flags to check if 
> access is available, if it is aligned and the order in which words are 
> written on the machine. Each check costs -- especially if done millions of 
> times a second -- and on occasion adds bloat in hot code paths. The unsafe 
> access inside Bytes checks on startup what the machine is capable off and 
> then does a static assign of the appropriate class-to-use from there on out. 
> UnsafeAccess does not do this running the checks everytime. Would be good to 
> have the Bytes behavior pervasive.
> The benefit of one access to Unsafe only is plain. The benefits we gain 
> removing checks will be harder to measure though should be plain when you 
> disassemble a hot-path; in a (very) rare case, the saved byte codes could be 
> the difference between inlining or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HBASE-20716) Unsafe access cleanup

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal reassigned HBASE-20716:
--

Assignee: Sahil Aggarwal

> Unsafe access cleanup
> -
>
> Key: HBASE-20716
> URL: https://issues.apache.org/jira/browse/HBASE-20716
> Project: HBase
>  Issue Type: Improvement
>  Components: Performance
>Reporter: stack
>Assignee: Sahil Aggarwal
>Priority: Critical
>  Labels: beginner
> Attachments: Screen Shot 2018-06-26 at 11.37.49 AM.png
>
>
> We have two means of getting at unsafe; UnsafeAccess and then internal to the 
> Bytes class. They are effectively doing the same thing. We should have one 
> avenue to Unsafe only.
> Many of our paths to Unsafe via UnsafeAccess traverse flags to check if 
> access is available, if it is aligned and the order in which words are 
> written on the machine. Each check costs -- especially if done millions of 
> times a second -- and on occasion adds bloat in hot code paths. The unsafe 
> access inside Bytes checks on startup what the machine is capable off and 
> then does a static assign of the appropriate class-to-use from there on out. 
> UnsafeAccess does not do this running the checks everytime. Would be good to 
> have the Bytes behavior pervasive.
> The benefit of one access to Unsafe only is plain. The benefits we gain 
> removing checks will be harder to measure though should be plain when you 
> disassemble a hot-path; in a (very) rare case, the saved byte codes could be 
> the difference between inlining or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19164) Avoid UUID.randomUUID in tests

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525216#comment-16525216
 ] 

Sahil Aggarwal commented on HBASE-19164:


Thanks a lot [~mdrob] for prompt reviews. 

> Avoid UUID.randomUUID in tests
> --
>
> Key: HBASE-19164
> URL: https://issues.apache.org/jira/browse/HBASE-19164
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>Priority: Major
>  Labels: beginner
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-19164.master.001.patch, 
> HBASE-19164.master.002.patch, HBASE-19164.master.003.patch, 
> HBASE-19164.master.004.patch, HBASE-19164.master.005.patch, 
> HBASE-19164.master.006.patch, HBASE-19164.master.007.patch
>
>
> We have a lot of places in our test code where we use {{UUID.randomUUID}} to 
> generate table names or paths for uniqueness. Unfortunately, this uses up a 
> good chunk of system entropy, since Sun chose that random UUID's should use 
> the NativePRNGBlocking implementation.
> We don't need to block on entropy for random bits to pick a random table name 
> in a test, so we can use something that doesn't strain the system too much - 
> secure random can be a source of problems on some VM or containers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19164) Avoid UUID.randomUUID in tests

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-19164:
---
Attachment: HBASE-19164.master.007.patch

> Avoid UUID.randomUUID in tests
> --
>
> Key: HBASE-19164
> URL: https://issues.apache.org/jira/browse/HBASE-19164
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>Priority: Major
>  Labels: beginner
> Attachments: HBASE-19164.master.001.patch, 
> HBASE-19164.master.002.patch, HBASE-19164.master.003.patch, 
> HBASE-19164.master.004.patch, HBASE-19164.master.005.patch, 
> HBASE-19164.master.006.patch, HBASE-19164.master.007.patch
>
>
> We have a lot of places in our test code where we use {{UUID.randomUUID}} to 
> generate table names or paths for uniqueness. Unfortunately, this uses up a 
> good chunk of system entropy, since Sun chose that random UUID's should use 
> the NativePRNGBlocking implementation.
> We don't need to block on entropy for random bits to pick a random table name 
> in a test, so we can use something that doesn't strain the system too much - 
> secure random can be a source of problems on some VM or containers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20716) Unsafe access cleanup

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16524707#comment-16524707
 ] 

Sahil Aggarwal commented on HBASE-20716:


Shall I take it up ?

Will abstract out:

putInt, getInt, putShort, getShort, putLong, getLong into Abstract class 
BytesCoverter have 2 impls safe and unsafe. I see that we do these conversions 
in Bytes and ByteBufferUtils.

Also, in ByteBufferUtils we still use check and compare mechanism whereas in 
Bytes we have BEST_COMPARER which is statically initialized at runtime, shall 
we also change it too or it falls under different ticket?

 

> Unsafe access cleanup
> -
>
> Key: HBASE-20716
> URL: https://issues.apache.org/jira/browse/HBASE-20716
> Project: HBase
>  Issue Type: Improvement
>  Components: Performance
>Reporter: stack
>Priority: Critical
>  Labels: beginner
> Attachments: Screen Shot 2018-06-26 at 11.37.49 AM.png
>
>
> We have two means of getting at unsafe; UnsafeAccess and then internal to the 
> Bytes class. They are effectively doing the same thing. We should have one 
> avenue to Unsafe only.
> Many of our paths to Unsafe via UnsafeAccess traverse flags to check if 
> access is available, if it is aligned and the order in which words are 
> written on the machine. Each check costs -- especially if done millions of 
> times a second -- and on occasion adds bloat in hot code paths. The unsafe 
> access inside Bytes checks on startup what the machine is capable off and 
> then does a static assign of the appropriate class-to-use from there on out. 
> UnsafeAccess does not do this running the checks everytime. Would be good to 
> have the Bytes behavior pervasive.
> The benefit of one access to Unsafe only is plain. The benefits we gain 
> removing checks will be harder to measure though should be plain when you 
> disassemble a hot-path; in a (very) rare case, the saved byte codes could be 
> the difference between inlining or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20716) Unsafe access cleanup

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16524578#comment-16524578
 ] 

Sahil Aggarwal commented on HBASE-20716:


Here's small benchmark for Bytes.toShort

@Benchmark
public void testGetShortCheckAndDispatch() {
 if (UNSAFE_UNALIGNED) {  
     UnsafeAccess.toShort(bytes, 0);
  } else {
    getShort(bytes, 0);
  }
}

@Benchmark
public void testGetShortDispatch() {
   UnsafeAccess.toShort(bytes, 0);
}

private short getShort(byte[] bytes, int offset) {
   short n = 0;
   n = (short) ((n ^ bytes[offset]) & 0xFF);
   n = (short) (n << 8);
   n = (short) ((n ^ bytes[offset+1]) & 0xFF);
   return n;
}

Benchmark                                                                       
        Mode    Cnt     Score                     Error        Units
UnsafeAccessBenchmark.testGetShortCheckAndDispatch    thrpt       4       
2195811074.302                    ops/s
UnsafeAccessBenchmark.testGetShortDispatch                     thrpt      4     
  2196669275.206                    ops/s

 

Was curious on affect on inlining too, disabling the inlining on 
UnsafeAccess.toShort:

Benchmark                                                                       
        Mode    Cnt     Score                     Error        Units
UnsafeAccessBenchmark.testGetShortCheckAndDispatch    thrpt       2       
474847927.534                    ops/s

UnsafeAccessBenchmark.testGetShortDispatch                     thrpt      2     
  486521295.364                    ops/s

 

 

~ 78% reduction on disabling inlining.

 

And, size difference b/w these methods:

org.sample.UnsafeAccessBenchmark::testGetShortCheckAndDispatch (27 bytes)   
force inline by CompileOracle

org.sample.UnsafeAccessBenchmark::testGetShortDispatch (9 bytes)   force inline 
by CompileOracle

 

 

> Unsafe access cleanup
> -
>
> Key: HBASE-20716
> URL: https://issues.apache.org/jira/browse/HBASE-20716
> Project: HBase
>  Issue Type: Improvement
>  Components: Performance
>Reporter: stack
>Priority: Critical
>  Labels: beginner
> Attachments: Screen Shot 2018-06-26 at 11.37.49 AM.png
>
>
> We have two means of getting at unsafe; UnsafeAccess and then internal to the 
> Bytes class. They are effectively doing the same thing. We should have one 
> avenue to Unsafe only.
> Many of our paths to Unsafe via UnsafeAccess traverse flags to check if 
> access is available, if it is aligned and the order in which words are 
> written on the machine. Each check costs -- especially if done millions of 
> times a second -- and on occasion adds bloat in hot code paths. The unsafe 
> access inside Bytes checks on startup what the machine is capable off and 
> then does a static assign of the appropriate class-to-use from there on out. 
> UnsafeAccess does not do this running the checks everytime. Would be good to 
> have the Bytes behavior pervasive.
> The benefit of one access to Unsafe only is plain. The benefits we gain 
> removing checks will be harder to measure though should be plain when you 
> disassemble a hot-path; in a (very) rare case, the saved byte codes could be 
> the difference between inlining or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19164) Avoid UUID.randomUUID in tests

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-19164:
---
Attachment: HBASE-19164.master.006.patch

> Avoid UUID.randomUUID in tests
> --
>
> Key: HBASE-19164
> URL: https://issues.apache.org/jira/browse/HBASE-19164
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>Priority: Major
>  Labels: beginner
> Attachments: HBASE-19164.master.001.patch, 
> HBASE-19164.master.002.patch, HBASE-19164.master.003.patch, 
> HBASE-19164.master.004.patch, HBASE-19164.master.005.patch, 
> HBASE-19164.master.006.patch
>
>
> We have a lot of places in our test code where we use {{UUID.randomUUID}} to 
> generate table names or paths for uniqueness. Unfortunately, this uses up a 
> good chunk of system entropy, since Sun chose that random UUID's should use 
> the NativePRNGBlocking implementation.
> We don't need to block on entropy for random bits to pick a random table name 
> in a test, so we can use something that doesn't strain the system too much - 
> secure random can be a source of problems on some VM or containers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20716) Unsafe access cleanup

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523681#comment-16523681
 ] 

Sahil Aggarwal commented on HBASE-20716:


[~stack] 

What I got from description and some code reading is that we want to remove 
these booleans:

UNSAFE_UNALIGNED

LITTLE_ENDIAN

probably because these checks happen in hot code path and can be avoided by 
checking the same at startup and assigned the relevant handler. Tried running 
jmh on 2 versions: one with flag and another without flag, after making it 
final there seem to be almost no difference. 

 

+non-final flag LITTLE_ENDIAN:+

Result "org.sample.UnsafeAccessBenchmark.testUnsafeWithFlag": 2178237030.784 
ops/s

Result "org.sample.UnsafeAccessBenchmark.testUnsafeWithoutFlag": 3255613805.960 
ops/s

 

+final flag LITTLE_ENDIAN:+

Result "org.sample.UnsafeAccessBenchmark.testUnsafeWithFlag": 3243770223.012 
ops/s

Result "org.sample.UnsafeAccessBenchmark.testUnsafeWithoutFlag": 3256975533.152 
ops/s

 

 

> Unsafe access cleanup
> -
>
> Key: HBASE-20716
> URL: https://issues.apache.org/jira/browse/HBASE-20716
> Project: HBase
>  Issue Type: Improvement
>  Components: Performance
>Reporter: stack
>Priority: Critical
>  Labels: beginner
>
> We have two means of getting at unsafe; UnsafeAccess and then internal to the 
> Bytes class. They are effectively doing the same thing. We should have one 
> avenue to Unsafe only.
> Many of our paths to Unsafe via UnsafeAccess traverse flags to check if 
> access is available, if it is aligned and the order in which words are 
> written on the machine. Each check costs -- especially if done millions of 
> times a second -- and on occasion adds bloat in hot code paths. The unsafe 
> access inside Bytes checks on startup what the machine is capable off and 
> then does a static assign of the appropriate class-to-use from there on out. 
> UnsafeAccess does not do this running the checks everytime. Would be good to 
> have the Bytes behavior pervasive.
> The benefit of one access to Unsafe only is plain. The benefits we gain 
> removing checks will be harder to measure though should be plain when you 
> disassemble a hot-path; in a (very) rare case, the saved byte codes could be 
> the difference between inlining or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19164) Avoid UUID.randomUUID in tests

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-19164:
---
Attachment: HBASE-19164.master.005.patch

> Avoid UUID.randomUUID in tests
> --
>
> Key: HBASE-19164
> URL: https://issues.apache.org/jira/browse/HBASE-19164
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>Priority: Major
>  Labels: beginner
> Attachments: HBASE-19164.master.001.patch, 
> HBASE-19164.master.002.patch, HBASE-19164.master.003.patch, 
> HBASE-19164.master.004.patch, HBASE-19164.master.005.patch
>
>
> We have a lot of places in our test code where we use {{UUID.randomUUID}} to 
> generate table names or paths for uniqueness. Unfortunately, this uses up a 
> good chunk of system entropy, since Sun chose that random UUID's should use 
> the NativePRNGBlocking implementation.
> We don't need to block on entropy for random bits to pick a random table name 
> in a test, so we can use something that doesn't strain the system too much - 
> secure random can be a source of problems on some VM or containers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19164) Avoid UUID.randomUUID in tests

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523153#comment-16523153
 ] 

Sahil Aggarwal commented on HBASE-19164:


Rebased the patch.

> Avoid UUID.randomUUID in tests
> --
>
> Key: HBASE-19164
> URL: https://issues.apache.org/jira/browse/HBASE-19164
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>Priority: Major
>  Labels: beginner
> Attachments: HBASE-19164.master.001.patch, 
> HBASE-19164.master.002.patch, HBASE-19164.master.003.patch, 
> HBASE-19164.master.004.patch
>
>
> We have a lot of places in our test code where we use {{UUID.randomUUID}} to 
> generate table names or paths for uniqueness. Unfortunately, this uses up a 
> good chunk of system entropy, since Sun chose that random UUID's should use 
> the NativePRNGBlocking implementation.
> We don't need to block on entropy for random bits to pick a random table name 
> in a test, so we can use something that doesn't strain the system too much - 
> secure random can be a source of problems on some VM or containers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19164) Avoid UUID.randomUUID in tests

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-19164:
---
Attachment: HBASE-19164.master.004.patch

> Avoid UUID.randomUUID in tests
> --
>
> Key: HBASE-19164
> URL: https://issues.apache.org/jira/browse/HBASE-19164
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>Priority: Major
>  Labels: beginner
> Attachments: HBASE-19164.master.001.patch, 
> HBASE-19164.master.002.patch, HBASE-19164.master.003.patch, 
> HBASE-19164.master.004.patch
>
>
> We have a lot of places in our test code where we use {{UUID.randomUUID}} to 
> generate table names or paths for uniqueness. Unfortunately, this uses up a 
> good chunk of system entropy, since Sun chose that random UUID's should use 
> the NativePRNGBlocking implementation.
> We don't need to block on entropy for random bits to pick a random table name 
> in a test, so we can use something that doesn't strain the system too much - 
> secure random can be a source of problems on some VM or containers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20326) Doc building hbase with native libraries

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504770#comment-16504770
 ] 

Sahil Aggarwal commented on HBASE-20326:


Shall we add this under "Building against various hadoop versions." in refguide?

> Doc building hbase with native libraries
> 
>
> Key: HBASE-20326
> URL: https://issues.apache.org/jira/browse/HBASE-20326
> Project: HBase
>  Issue Type: Bug
>  Components: documentation
>Reporter: stack
>Priority: Major
>  Labels: beginner
>
> Summarize in refguide the discussion up on 'building hbase with native 
> libraries' on dev list.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-18612) Update comparators to be more declarative

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504568#comment-16504568
 ] 

Sahil Aggarwal commented on HBASE-18612:


[~mdrob] On cursory look I see that at some places we use comparators where we 
compare multiple things, do we intend to convert them to different comparators 
and chain them instead of having one fat comparator? For eg:

 

In TableDescriptor.getComparator, we compare tableName, then we compare 
columnFamilies. 

in RegionInfo.COMPARATOR, we compare TableName, startkey, endkey, regionId, 
replicaId. 

 

Do we intend to chain each compare in all the fat compare methods?

> Update comparators to be more declarative
> -
>
> Key: HBASE-18612
> URL: https://issues.apache.org/jira/browse/HBASE-18612
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mike Drob
>Priority: Major
>  Labels: beginner, java8
> Attachments: HBASE-18612-WIP.patch
>
>
> We can write less code if we use factory methods present on Comparator to 
> build the chains. Also has the advantage of being easier to update in the 
> future and hopefully easier for incoming folks to understand.
> See also: 
> https://praveer09.github.io/technology/2016/06/21/writing-comparators-the-java8-way/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19453) Use maven filtering instead of ant replace task for project version in hbase-defaults.xml

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-19453:
---
Attachment: HBASE-19453.master.003.patch

> Use maven filtering instead of ant replace task for project version in 
> hbase-defaults.xml
> -
>
> Key: HBASE-19453
> URL: https://issues.apache.org/jira/browse/HBASE-19453
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>Priority: Minor
>  Labels: beginner
> Attachments: HBASE-19453.master.001.patch, 
> HBASE-19453.master.002.patch, HBASE-19453.master.003.patch
>
>
> In {{hbase-common.xml}} we use the antrun plugin to filter our 
> {{hbase-defaults.xml}} file to include the current version. This would be 
> easier using the maven resources plugin to do filtering, and gets better IDE 
> integration.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19453) Use maven filtering instead of ant replace task for project version in hbase-defaults.xml

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503672#comment-16503672
 ] 

Sahil Aggarwal commented on HBASE-19453:


Yes, you are right. Removed it.

> Use maven filtering instead of ant replace task for project version in 
> hbase-defaults.xml
> -
>
> Key: HBASE-19453
> URL: https://issues.apache.org/jira/browse/HBASE-19453
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>Priority: Minor
>  Labels: beginner
> Attachments: HBASE-19453.master.001.patch, 
> HBASE-19453.master.002.patch, HBASE-19453.master.003.patch
>
>
> In {{hbase-common.xml}} we use the antrun plugin to filter our 
> {{hbase-defaults.xml}} file to include the current version. This would be 
> easier using the maven resources plugin to do filtering, and gets better IDE 
> integration.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19453) Use maven filtering instead of ant replace task for project version in hbase-defaults.xml

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503300#comment-16503300
 ] 

Sahil Aggarwal commented on HBASE-19453:


[~mdrob] Thanks for the quick review. Yes, as you pointed most of the stuff was 
default. Have made the changes and verified. 

> Use maven filtering instead of ant replace task for project version in 
> hbase-defaults.xml
> -
>
> Key: HBASE-19453
> URL: https://issues.apache.org/jira/browse/HBASE-19453
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>Priority: Minor
>  Labels: beginner
> Attachments: HBASE-19453.master.001.patch, 
> HBASE-19453.master.002.patch
>
>
> In {{hbase-common.xml}} we use the antrun plugin to filter our 
> {{hbase-defaults.xml}} file to include the current version. This would be 
> easier using the maven resources plugin to do filtering, and gets better IDE 
> integration.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19453) Use maven filtering instead of ant replace task for project version in hbase-defaults.xml

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-19453:
---
Attachment: HBASE-19453.master.002.patch

> Use maven filtering instead of ant replace task for project version in 
> hbase-defaults.xml
> -
>
> Key: HBASE-19453
> URL: https://issues.apache.org/jira/browse/HBASE-19453
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>Priority: Minor
>  Labels: beginner
> Attachments: HBASE-19453.master.001.patch, 
> HBASE-19453.master.002.patch
>
>
> In {{hbase-common.xml}} we use the antrun plugin to filter our 
> {{hbase-defaults.xml}} file to include the current version. This would be 
> easier using the maven resources plugin to do filtering, and gets better IDE 
> integration.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HBASE-19453) Use maven filtering instead of ant replace task for project version in hbase-defaults.xml

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal reassigned HBASE-19453:
--

Assignee: Sahil Aggarwal

> Use maven filtering instead of ant replace task for project version in 
> hbase-defaults.xml
> -
>
> Key: HBASE-19453
> URL: https://issues.apache.org/jira/browse/HBASE-19453
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>Priority: Minor
>  Labels: beginner
> Attachments: HBASE-19453.master.001.patch
>
>
> In {{hbase-common.xml}} we use the antrun plugin to filter our 
> {{hbase-defaults.xml}} file to include the current version. This would be 
> easier using the maven resources plugin to do filtering, and gets better IDE 
> integration.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19453) Use maven filtering instead of ant replace task for project version in hbase-defaults.xml

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-19453:
---
Attachment: HBASE-19453.master.001.patch

> Use maven filtering instead of ant replace task for project version in 
> hbase-defaults.xml
> -
>
> Key: HBASE-19453
> URL: https://issues.apache.org/jira/browse/HBASE-19453
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Reporter: Mike Drob
>Priority: Minor
>  Labels: beginner
> Attachments: HBASE-19453.master.001.patch
>
>
> In {{hbase-common.xml}} we use the antrun plugin to filter our 
> {{hbase-defaults.xml}} file to include the current version. This would be 
> easier using the maven resources plugin to do filtering, and gets better IDE 
> integration.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20474) Show non-RPC tasks on master/regionserver Web UI by default

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454578#comment-16454578
 ] 

Sahil Aggarwal commented on HBASE-20474:


[~andrewcheng] Tested it, works fine.

> Show non-RPC tasks on master/regionserver Web UI  by default
> 
>
> Key: HBASE-20474
> URL: https://issues.apache.org/jira/browse/HBASE-20474
> Project: HBase
>  Issue Type: Improvement
>  Components: UI
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20474.master.001.patch
>
>
> Now, when opening master or regionserver pages, all tasks will be displayed 
> on the page, however, but in most cases we will pay more attention to non-RPC 
> tasks.
> In addition, if all tasks are displayed by default, a large number of pages 
> will be occupied.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20474) Show non-RPC tasks on master/regionserver Web UI by default

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-20474:
---
Attachment: (was: HBASE-20474.master.002.patch)

> Show non-RPC tasks on master/regionserver Web UI  by default
> 
>
> Key: HBASE-20474
> URL: https://issues.apache.org/jira/browse/HBASE-20474
> Project: HBase
>  Issue Type: Improvement
>  Components: UI
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20474.master.001.patch
>
>
> Now, when opening master or regionserver pages, all tasks will be displayed 
> on the page, however, but in most cases we will pay more attention to non-RPC 
> tasks.
> In addition, if all tasks are displayed by default, a large number of pages 
> will be occupied.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20474) Show non-RPC tasks on master/regionserver Web UI by default

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-20474:
---
Attachment: HBASE-20474.master.002.patch

> Show non-RPC tasks on master/regionserver Web UI  by default
> 
>
> Key: HBASE-20474
> URL: https://issues.apache.org/jira/browse/HBASE-20474
> Project: HBase
>  Issue Type: Improvement
>  Components: UI
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20474.master.001.patch, 
> HBASE-20474.master.002.patch
>
>
> Now, when opening master or regionserver pages, all tasks will be displayed 
> on the page, however, but in most cases we will pay more attention to non-RPC 
> tasks.
> In addition, if all tasks are displayed by default, a large number of pages 
> will be occupied.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20351) Shell dumps netty properties on startup

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426489#comment-16426489
 ] 

Sahil Aggarwal commented on HBASE-20351:


Can you please help me reproduce. I tried on branch-2

```

Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
Version 2.1.0-SNAPSHOT, r039bc73571d2cc89378749573dfeec74c247b0b9, Thu Apr 5 
10:33:31 IST 2018
Took 0.0030 seconds 
hbase(main):001:0> 
hbase(main):002:0* list
TABLE 
0 row(s)
Took 0.3629 seconds 
hbase(main):003:0>

```

couldn't see such logs, even tried turning everything to DEBUG mode in log 
config.

> Shell dumps netty properties on startup
> ---
>
> Key: HBASE-20351
> URL: https://issues.apache.org/jira/browse/HBASE-20351
> Project: HBase
>  Issue Type: Bug
>  Components: pain-in-the-ass, shell
>Reporter: stack
>Priority: Major
> Fix For: 2.0.0
>
>
> {code}
> stack@ve0524:~$ ./hbase/bin/hbase --config conf_hbase shell
> 2018-04-04 19:58:02,187 DEBUG [main] logging.InternalLoggerFactory: Using 
> SLF4J as the default logging framework
> 2018-04-04 19:58:02,191 DEBUG [main] util.ResourceLeakDetector: 
> -Dorg.apache.hbase.thirdparty.io.netty.leakDetection.level: simple
> 2018-04-04 19:58:02,192 DEBUG [main] util.ResourceLeakDetector: 
> -Dorg.apache.hbase.thirdparty.io.netty.leakDetection.targetRecords: 4
> 2018-04-04 19:58:02,214 DEBUG [main] internal.PlatformDependent0: 
> -Dio.netty.noUnsafe: false
> 2018-04-04 19:58:02,215 DEBUG [main] internal.PlatformDependent0: Java 
> version: 8
> 2018-04-04 19:58:02,216 DEBUG [main] internal.PlatformDependent0: 
> sun.misc.Unsafe.theUnsafe: available
> 2018-04-04 19:58:02,216 DEBUG [main] internal.PlatformDependent0: 
> sun.misc.Unsafe.copyMemory: available
> 2018-04-04 19:58:02,217 DEBUG [main] internal.PlatformDependent0: 
> java.nio.Buffer.address: available
> 2018-04-04 19:58:02,217 DEBUG [main] internal.PlatformDependent0: direct 
> buffer constructor: available
> 2018-04-04 19:58:02,218 DEBUG [main] internal.PlatformDependent0: 
> java.nio.Bits.unaligned: available, true
> 2018-04-04 19:58:02,218 DEBUG [main] internal.PlatformDependent0: 
> jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable prior 
> to Java9
> 2018-04-04 19:58:02,218 DEBUG [main] internal.PlatformDependent0: 
> java.nio.DirectByteBuffer.(long, int): available
> 2018-04-04 19:58:02,218 DEBUG [main] internal.PlatformDependent: 
> sun.misc.Unsafe: available
> 2018-04-04 19:58:02,218 DEBUG [main] internal.PlatformDependent: 
> -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
> 2018-04-04 19:58:02,218 DEBUG [main] internal.PlatformDependent: 
> -Dio.netty.bitMode: 64 (sun.arch.data.model)
> 2018-04-04 19:58:02,219 DEBUG [main] internal.PlatformDependent: 
> -Dio.netty.noPreferDirect: false
> 2018-04-04 19:58:02,219 DEBUG [main] internal.PlatformDependent: 
> -Dio.netty.maxDirectMemory: 1073741824 bytes
> 2018-04-04 19:58:02,219 DEBUG [main] internal.PlatformDependent: 
> -Dio.netty.uninitializedArrayAllocationThreshold: -1
> 2018-04-04 19:58:02,220 DEBUG [main] internal.CleanerJava6: 
> java.nio.ByteBuffer.cleaner(): available
> 2018-04-04 19:58:02,220 DEBUG [main] util.ResourceLeakDetectorFactory: Loaded 
> default ResourceLeakDetector: 
> org.apache.hbase.thirdparty.io.netty.util.ResourceLeakDetector@7dbae40
> 2018-04-04 19:58:02,229 DEBUG [main] internal.PlatformDependent: 
> org.jctools-core.MpscChunkedArrayQueue: available
> 2018-04-04 19:58:02,260 DEBUG [main] channel.MultithreadEventLoopGroup: 
> -Dio.netty.eventLoopThreads: 96
> 2018-04-04 19:58:02,282 DEBUG [main] nio.NioEventLoop: 
> -Dio.netty.noKeySetOptimization: false
> 2018-04-04 19:58:02,282 DEBUG [main] nio.NioEventLoop: 
> -Dio.netty.selectorAutoRebuildThreshold: 512
> HBase Shell
> Use "help" to get list of supported commands.
> Use "exit" to quit this interactive shell.
> Version 2.0.0, r0db342d312784a6663b406fdb0f7b3b3c1fa928d, Mon Apr  2 22:54:56 
> PDT 2018
> Took 0.0028 seconds
> hbase(main):001:0>
> {code}
> Does it each time I run a command
> {code}
> hbase(main):001:0> describe 'ycsb'
> 2018-04-04 19:59:00,084 DEBUG [main] buffer.AbstractByteBuf: 
> -Dorg.apache.hbase.thirdparty.io.netty.buffer.bytebuf.checkAccessible: true
> 2018-04-04 19:59:00,084 DEBUG [main] util.ResourceLeakDetectorFactory: Loaded 
> default ResourceLeakDetector: 
> org.apache.hbase.thirdparty.io.netty.util.ResourceLeakDetector@66ab924
> 2018-04-04 19:59:00,121 DEBUG [main] channel.DefaultChannelId: 
> -Dio.netty.processId: 697 (auto-detected)
> 2018-04-04 19:59:00,123 DEBUG [main] util.NetUtil: 
> -Djava.net.preferIPv4Stack: true
> 2018-04-04 19:59:00,123 DEBUG [main] util.NetUtil: 
> -Djava.net.preferIPv6Addresses: false
> 2018-04-04 19:59:00,124 DEBUG [main] util.NetUtil: Loopback interface: lo 
> (lo, 127.0.0.1)
> 

[jira] [Commented] (HBASE-17104) Improve cryptic error message "Memstore size is" on region close

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16371898#comment-16371898
 ] 

Sahil Aggarwal commented on HBASE-17104:


Sorry for the delay here.

Earlier we were just logging it as ERROR and continuing further where we are 
executing post-close hooks for coprocessor and closing the metricregion, I 
thought that should still continue. Thoughts?

> Improve cryptic error message "Memstore size is" on region close
> 
>
> Key: HBASE-17104
> URL: https://issues.apache.org/jira/browse/HBASE-17104
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Reporter: Matteo Bertozzi
>Assignee: Sahil Aggarwal
>Priority: Trivial
>  Labels: beginner, noob
> Fix For: 2.0.0
>
> Attachments: HBASE-17104.master.001 (1) (1).patch, 
> HBASE-17104.master.001 (1).patch, HBASE-17104.master.001.patch
>
>
> while grepping my RS log for ERROR I found a cryptic
> {noformat}
> ERROR [RS_CLOSE_REGION-u1604vm:35021-1] regionserver.HRegion(1601): Memstore 
> size is 33744
> {noformat}
> from the code looks like we seems to want to notify the user about the fact 
> that on close the rs was not able to flush and there were things in the RS. 
> https://github.com/apache/hbase/blob/c3685760f004450667920144f926383eb307de53/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L1601
> {code}
> if (!canFlush) {
>   this.decrMemstoreSize(new MemstoreSize(memstoreDataSize.get(), 
> getMemstoreHeapOverhead()));
> } else if (memstoreDataSize.get() != 0) {
>   LOG.error("Memstore size is " + memstoreDataSize.get());
> }
> {code}
> this should probably not even be an error but a warn or even info, unless we 
> have puts that specifically asked to not be written to the wal,  otherwise 
> the data in the memstore should be safe in the wals. 
> In any case it will be nice to have a message describing what is going on and 
> why we are notifying about the memstore size.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19075) Task tabs on master UI cause page scroll

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-19075:
---
Attachment: HBASE-19075.master.002.patch

> Task tabs on master UI cause page scroll
> 
>
> Key: HBASE-19075
> URL: https://issues.apache.org/jira/browse/HBASE-19075
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>Priority: Major
>  Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-19075.master.001.patch, 
> HBASE-19075.master.002.patch
>
>
> On the master info page, the clicking the tabs under Tasks causes the page to 
> scroll back to the top of the page.
> {noformat}
> Tasks
> Show All Monitored Tasks Show non-RPC Tasks Show All RPC Handler Tasks Show 
> Active RPC Calls Show Client Operations View as JSON
> {noformat}
> ^^ Any of those
> The other tab-like links on the page keep the scroll in the same location.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19075) Task tabs on master UI cause page scroll

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16371959#comment-16371959
 ] 

Sahil Aggarwal commented on HBASE-19075:


Fixed it. Moved "view as Json" from nav bar to each respective tab-pane.

> Task tabs on master UI cause page scroll
> 
>
> Key: HBASE-19075
> URL: https://issues.apache.org/jira/browse/HBASE-19075
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>Priority: Major
>  Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-19075.master.001.patch, 
> HBASE-19075.master.002.patch
>
>
> On the master info page, the clicking the tabs under Tasks causes the page to 
> scroll back to the top of the page.
> {noformat}
> Tasks
> Show All Monitored Tasks Show non-RPC Tasks Show All RPC Handler Tasks Show 
> Active RPC Calls Show Client Operations View as JSON
> {noformat}
> ^^ Any of those
> The other tab-like links on the page keep the scroll in the same location.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19164) Avoid UUID.randomUUID in tests

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16248817#comment-16248817
 ] 

Sahil Aggarwal commented on HBASE-19164:


These tests are passing on my local but timing out on build machine. How do i 
debug it further?

> Avoid UUID.randomUUID in tests
> --
>
> Key: HBASE-19164
> URL: https://issues.apache.org/jira/browse/HBASE-19164
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>  Labels: beginner
> Attachments: HBASE-19164.master.001.patch, 
> HBASE-19164.master.002.patch, HBASE-19164.master.003.patch
>
>
> We have a lot of places in our test code where we use {{UUID.randomUUID}} to 
> generate table names or paths for uniqueness. Unfortunately, this uses up a 
> good chunk of system entropy, since Sun chose that random UUID's should use 
> the NativePRNGBlocking implementation.
> We don't need to block on entropy for random bits to pick a random table name 
> in a test, so we can use something that doesn't strain the system too much - 
> secure random can be a source of problems on some VM or containers.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-19164) Avoid UUID.randomUUID in tests

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-19164:
---
Status: Open  (was: Patch Available)

> Avoid UUID.randomUUID in tests
> --
>
> Key: HBASE-19164
> URL: https://issues.apache.org/jira/browse/HBASE-19164
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>  Labels: beginner
> Attachments: HBASE-19164.master.001.patch, 
> HBASE-19164.master.002.patch, HBASE-19164.master.003.patch
>
>
> We have a lot of places in our test code where we use {{UUID.randomUUID}} to 
> generate table names or paths for uniqueness. Unfortunately, this uses up a 
> good chunk of system entropy, since Sun chose that random UUID's should use 
> the NativePRNGBlocking implementation.
> We don't need to block on entropy for random bits to pick a random table name 
> in a test, so we can use something that doesn't strain the system too much - 
> secure random can be a source of problems on some VM or containers.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-19164) Avoid UUID.randomUUID in tests

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-19164:
---
Status: Patch Available  (was: Open)

> Avoid UUID.randomUUID in tests
> --
>
> Key: HBASE-19164
> URL: https://issues.apache.org/jira/browse/HBASE-19164
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>  Labels: beginner
> Attachments: HBASE-19164.master.001.patch, 
> HBASE-19164.master.002.patch, HBASE-19164.master.003.patch
>
>
> We have a lot of places in our test code where we use {{UUID.randomUUID}} to 
> generate table names or paths for uniqueness. Unfortunately, this uses up a 
> good chunk of system entropy, since Sun chose that random UUID's should use 
> the NativePRNGBlocking implementation.
> We don't need to block on entropy for random bits to pick a random table name 
> in a test, so we can use something that doesn't strain the system too much - 
> secure random can be a source of problems on some VM or containers.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-19164) Avoid UUID.randomUUID in tests

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-19164:
---
Attachment: HBASE-19164.master.003.patch

> Avoid UUID.randomUUID in tests
> --
>
> Key: HBASE-19164
> URL: https://issues.apache.org/jira/browse/HBASE-19164
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>  Labels: beginner
> Attachments: HBASE-19164.master.001.patch, 
> HBASE-19164.master.002.patch, HBASE-19164.master.003.patch
>
>
> We have a lot of places in our test code where we use {{UUID.randomUUID}} to 
> generate table names or paths for uniqueness. Unfortunately, this uses up a 
> good chunk of system entropy, since Sun chose that random UUID's should use 
> the NativePRNGBlocking implementation.
> We don't need to block on entropy for random bits to pick a random table name 
> in a test, so we can use something that doesn't strain the system too much - 
> secure random can be a source of problems on some VM or containers.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-19164) Avoid UUID.randomUUID in tests

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-19164:
---
Attachment: HBASE-19164.master.002.patch

> Avoid UUID.randomUUID in tests
> --
>
> Key: HBASE-19164
> URL: https://issues.apache.org/jira/browse/HBASE-19164
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>  Labels: beginner
> Attachments: HBASE-19164.master.001.patch, 
> HBASE-19164.master.002.patch
>
>
> We have a lot of places in our test code where we use {{UUID.randomUUID}} to 
> generate table names or paths for uniqueness. Unfortunately, this uses up a 
> good chunk of system entropy, since Sun chose that random UUID's should use 
> the NativePRNGBlocking implementation.
> We don't need to block on entropy for random bits to pick a random table name 
> in a test, so we can use something that doesn't strain the system too much - 
> secure random can be a source of problems on some VM or containers.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-19164) Avoid UUID.randomUUID in tests

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-19164:
---
Status: Patch Available  (was: Open)

> Avoid UUID.randomUUID in tests
> --
>
> Key: HBASE-19164
> URL: https://issues.apache.org/jira/browse/HBASE-19164
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>  Labels: beginner
> Attachments: HBASE-19164.master.001.patch, 
> HBASE-19164.master.002.patch
>
>
> We have a lot of places in our test code where we use {{UUID.randomUUID}} to 
> generate table names or paths for uniqueness. Unfortunately, this uses up a 
> good chunk of system entropy, since Sun chose that random UUID's should use 
> the NativePRNGBlocking implementation.
> We don't need to block on entropy for random bits to pick a random table name 
> in a test, so we can use something that doesn't strain the system too much - 
> secure random can be a source of problems on some VM or containers.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-19164) Avoid UUID.randomUUID in tests

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246986#comment-16246986
 ] 

Sahil Aggarwal commented on HBASE-19164:


Attached the patch. 

Have added helper method in HbaseCommonTestingUtility which was being used by 
most of the tests exception being RandomStringGeneratorImpl.

 Shall i do "Submit Patch" to trigger the QA?

> Avoid UUID.randomUUID in tests
> --
>
> Key: HBASE-19164
> URL: https://issues.apache.org/jira/browse/HBASE-19164
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>  Labels: beginner
> Attachments: HBASE-19164.master.001.patch
>
>
> We have a lot of places in our test code where we use {{UUID.randomUUID}} to 
> generate table names or paths for uniqueness. Unfortunately, this uses up a 
> good chunk of system entropy, since Sun chose that random UUID's should use 
> the NativePRNGBlocking implementation.
> We don't need to block on entropy for random bits to pick a random table name 
> in a test, so we can use something that doesn't strain the system too much - 
> secure random can be a source of problems on some VM or containers.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-19164) Avoid UUID.randomUUID in tests

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-19164:
---
Attachment: HBASE-19164.master.001.patch

> Avoid UUID.randomUUID in tests
> --
>
> Key: HBASE-19164
> URL: https://issues.apache.org/jira/browse/HBASE-19164
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>  Labels: beginner
> Attachments: HBASE-19164.master.001.patch
>
>
> We have a lot of places in our test code where we use {{UUID.randomUUID}} to 
> generate table names or paths for uniqueness. Unfortunately, this uses up a 
> good chunk of system entropy, since Sun chose that random UUID's should use 
> the NativePRNGBlocking implementation.
> We don't need to block on entropy for random bits to pick a random table name 
> in a test, so we can use something that doesn't strain the system too much - 
> secure random can be a source of problems on some VM or containers.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HBASE-19164) Avoid UUID.randomUUID in tests

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal reassigned HBASE-19164:
--

Assignee: Sahil Aggarwal

> Avoid UUID.randomUUID in tests
> --
>
> Key: HBASE-19164
> URL: https://issues.apache.org/jira/browse/HBASE-19164
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>  Labels: beginner
>
> We have a lot of places in our test code where we use {{UUID.randomUUID}} to 
> generate table names or paths for uniqueness. Unfortunately, this uses up a 
> good chunk of system entropy, since Sun chose that random UUID's should use 
> the NativePRNGBlocking implementation.
> We don't need to block on entropy for random bits to pick a random table name 
> in a test, so we can use something that doesn't strain the system too much - 
> secure random can be a source of problems on some VM or containers.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (HBASE-19164) Avoid UUID.randomUUID in tests

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241483#comment-16241483
 ] 

Sahil Aggarwal edited comment on HBASE-19164 at 11/7/17 4:53 AM:
-

I see multiple tests using 
{code:java}
UUID.randomUUID()
{code}
, do we just need to replace it with 
{code:java}
new UUID(ThreadLocalRandom.current().nextLong(), 
ThreadLocalRandom.current().nextLong()
{code}
)


was (Author: awked06):
I see multiple tests using ```UUID.randomUUID()```, do we just need to replace 
it with ```new UUID(ThreadLocalRandom.current().nextLong(), 
ThreadLocalRandom.current().nextLong())```

> Avoid UUID.randomUUID in tests
> --
>
> Key: HBASE-19164
> URL: https://issues.apache.org/jira/browse/HBASE-19164
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Reporter: Mike Drob
>  Labels: beginner
>
> We have a lot of places in our test code where we use {{UUID.randomUUID}} to 
> generate table names or paths for uniqueness. Unfortunately, this uses up a 
> good chunk of system entropy, since Sun chose that random UUID's should use 
> the NativePRNGBlocking implementation.
> We don't need to block on entropy for random bits to pick a random table name 
> in a test, so we can use something that doesn't strain the system too much - 
> secure random can be a source of problems on some VM or containers.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-19164) Avoid UUID.randomUUID in tests

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241483#comment-16241483
 ] 

Sahil Aggarwal commented on HBASE-19164:


I see multiple tests using ```UUID.randomUUID()```, do we just need to replace 
it with ```new UUID(ThreadLocalRandom.current().nextLong(), 
ThreadLocalRandom.current().nextLong())```

> Avoid UUID.randomUUID in tests
> --
>
> Key: HBASE-19164
> URL: https://issues.apache.org/jira/browse/HBASE-19164
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Reporter: Mike Drob
>  Labels: beginner
>
> We have a lot of places in our test code where we use {{UUID.randomUUID}} to 
> generate table names or paths for uniqueness. Unfortunately, this uses up a 
> good chunk of system entropy, since Sun chose that random UUID's should use 
> the NativePRNGBlocking implementation.
> We don't need to block on entropy for random bits to pick a random table name 
> in a test, so we can use something that doesn't strain the system too much - 
> secure random can be a source of problems on some VM or containers.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18722) Replace copied guava code w/ pointer to the (shaded) lib

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241444#comment-16241444
 ] 

Sahil Aggarwal commented on HBASE-18722:


[~azafir] You working on it or shall i take it up?

> Replace copied guava code w/ pointer to the (shaded) lib
> 
>
> Key: HBASE-18722
> URL: https://issues.apache.org/jira/browse/HBASE-18722
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Priority: Minor
>  Labels: beginner
>
> I found a few bits of code copied from guava src:
> {code}
> ./src/main/java/org/apache/hadoop/hbase/io/LimitInputStream.java: * Copied 
> from guava source code v15 (LimitedInputStream)
> ./src/main/java/org/apache/hadoop/hbase/io/LimitInputStream.java: * allows to 
> be compatible with guava 11 to 15+.
> ./src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java: * 
> from guava v21
> ./src/main/java/org/apache/hadoop/hbase/util/Bytes.java: * 
> comparator from guava v21
> {code}
> We can instead point back at guava implementations now we use shaded guava 
> internally. No point having it in our code base when latest guava is on 
> latest CLASSPATH (This wasn't always the case; this is why the code was 
> copied into hbase).
> Also, there are other copies other than the above.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-19075) Task tabs on master UI cause page scroll

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241435#comment-16241435
 ] 

Sahil Aggarwal commented on HBASE-19075:


I had to do "Submit Patch" to move it to "Path Available", earlier i submitted 
patch by just attaching the file.

> Task tabs on master UI cause page scroll
> 
>
> Key: HBASE-19075
> URL: https://issues.apache.org/jira/browse/HBASE-19075
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>  Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-19075.master.001.patch
>
>
> On the master info page, the clicking the tabs under Tasks causes the page to 
> scroll back to the top of the page.
> {noformat}
> Tasks
> Show All Monitored Tasks Show non-RPC Tasks Show All RPC Handler Tasks Show 
> Active RPC Calls Show Client Operations View as JSON
> {noformat}
> ^^ Any of those
> The other tab-like links on the page keep the scroll in the same location.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-19075) Task tabs on master UI cause page scroll

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-19075:
---
Status: Patch Available  (was: Open)

> Task tabs on master UI cause page scroll
> 
>
> Key: HBASE-19075
> URL: https://issues.apache.org/jira/browse/HBASE-19075
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>  Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-19075.master.001.patch
>
>
> On the master info page, the clicking the tabs under Tasks causes the page to 
> scroll back to the top of the page.
> {noformat}
> Tasks
> Show All Monitored Tasks Show non-RPC Tasks Show All RPC Handler Tasks Show 
> Active RPC Calls Show Client Operations View as JSON
> {noformat}
> ^^ Any of those
> The other tab-like links on the page keep the scroll in the same location.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Work stopped] (HBASE-19075) Task tabs on master UI cause page scroll

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HBASE-19075 stopped by Sahil Aggarwal.
--
> Task tabs on master UI cause page scroll
> 
>
> Key: HBASE-19075
> URL: https://issues.apache.org/jira/browse/HBASE-19075
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>  Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-19075.master.001.patch
>
>
> On the master info page, the clicking the tabs under Tasks causes the page to 
> scroll back to the top of the page.
> {noformat}
> Tasks
> Show All Monitored Tasks Show non-RPC Tasks Show All RPC Handler Tasks Show 
> Active RPC Calls Show Client Operations View as JSON
> {noformat}
> ^^ Any of those
> The other tab-like links on the page keep the scroll in the same location.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-19075) Task tabs on master UI cause page scroll

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16239970#comment-16239970
 ] 

Sahil Aggarwal commented on HBASE-19075:


[~mdrob] Can you please review the patch whenever you can :)

> Task tabs on master UI cause page scroll
> 
>
> Key: HBASE-19075
> URL: https://issues.apache.org/jira/browse/HBASE-19075
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>  Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-19075.master.001.patch
>
>
> On the master info page, the clicking the tabs under Tasks causes the page to 
> scroll back to the top of the page.
> {noformat}
> Tasks
> Show All Monitored Tasks Show non-RPC Tasks Show All RPC Handler Tasks Show 
> Active RPC Calls Show Client Operations View as JSON
> {noformat}
> ^^ Any of those
> The other tab-like links on the page keep the scroll in the same location.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Work started] (HBASE-19075) Task tabs on master UI cause page scroll

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HBASE-19075 started by Sahil Aggarwal.
--
> Task tabs on master UI cause page scroll
> 
>
> Key: HBASE-19075
> URL: https://issues.apache.org/jira/browse/HBASE-19075
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>  Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-19075.master.001.patch
>
>
> On the master info page, the clicking the tabs under Tasks causes the page to 
> scroll back to the top of the page.
> {noformat}
> Tasks
> Show All Monitored Tasks Show non-RPC Tasks Show All RPC Handler Tasks Show 
> Active RPC Calls Show Client Operations View as JSON
> {noformat}
> ^^ Any of those
> The other tab-like links on the page keep the scroll in the same location.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-19075) Task tabs on master UI cause page scroll

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-19075:
---
Attachment: HBASE-19075.master.001.patch

> Task tabs on master UI cause page scroll
> 
>
> Key: HBASE-19075
> URL: https://issues.apache.org/jira/browse/HBASE-19075
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>Priority: Major
>  Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-19075.master.001.patch
>
>
> On the master info page, the clicking the tabs under Tasks causes the page to 
> scroll back to the top of the page.
> {noformat}
> Tasks
> Show All Monitored Tasks Show non-RPC Tasks Show All RPC Handler Tasks Show 
> Active RPC Calls Show Client Operations View as JSON
> {noformat}
> ^^ Any of those
> The other tab-like links on the page keep the scroll in the same location.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-19075) Task tabs on master UI cause page scroll

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238925#comment-16238925
 ] 

Sahil Aggarwal commented on HBASE-19075:


I am getting *Response: 403 Forbidden* while submitting patch. I tried: 

python dev-support/submit-patch.py -jid HBASE-19075 -srb 

> Task tabs on master UI cause page scroll
> 
>
> Key: HBASE-19075
> URL: https://issues.apache.org/jira/browse/HBASE-19075
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>Priority: Major
>  Labels: beginner
> Fix For: 2.0.0
>
>
> On the master info page, the clicking the tabs under Tasks causes the page to 
> scroll back to the top of the page.
> {noformat}
> Tasks
> Show All Monitored Tasks Show non-RPC Tasks Show All RPC Handler Tasks Show 
> Active RPC Calls Show Client Operations View as JSON
> {noformat}
> ^^ Any of those
> The other tab-like links on the page keep the scroll in the same location.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HBASE-19075) Task tabs on master UI cause page scroll

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal reassigned HBASE-19075:
--

Assignee: Sahil Aggarwal

> Task tabs on master UI cause page scroll
> 
>
> Key: HBASE-19075
> URL: https://issues.apache.org/jira/browse/HBASE-19075
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>Priority: Major
>  Labels: beginner
> Fix For: 2.0.0
>
>
> On the master info page, the clicking the tabs under Tasks causes the page to 
> scroll back to the top of the page.
> {noformat}
> Tasks
> Show All Monitored Tasks Show non-RPC Tasks Show All RPC Handler Tasks Show 
> Active RPC Calls Show Client Operations View as JSON
> {noformat}
> ^^ Any of those
> The other tab-like links on the page keep the scroll in the same location.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-19075) Task tabs on master UI cause page scroll

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235204#comment-16235204
 ] 

Sahil Aggarwal commented on HBASE-19075:


Hi Mike,

Other tables uses href="#div" which just changes the div on tab click but this 
table changes the query param and rely on page refresh to show the requested 
tab. Shall we change its impl to work in similar way as others or we just add 
the scipt to scroll to the selected anchor?

> Task tabs on master UI cause page scroll
> 
>
> Key: HBASE-19075
> URL: https://issues.apache.org/jira/browse/HBASE-19075
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Reporter: Mike Drob
>Priority: Major
>  Labels: beginner
> Fix For: 2.0.0
>
>
> On the master info page, the clicking the tabs under Tasks causes the page to 
> scroll back to the top of the page.
> {noformat}
> Tasks
> Show All Monitored Tasks Show non-RPC Tasks Show All RPC Handler Tasks Show 
> Active RPC Calls Show Client Operations View as JSON
> {noformat}
> ^^ Any of those
> The other tab-like links on the page keep the scroll in the same location.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HBASE-18439) Subclasses of o.a.h.h.chaos.actions.Action all use the same logger

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal reassigned HBASE-18439:
--

Assignee: Jui-Yu Hsieh  (was: Sahil Aggarwal)

> Subclasses of o.a.h.h.chaos.actions.Action all use the same logger
> --
>
> Key: HBASE-18439
> URL: https://issues.apache.org/jira/browse/HBASE-18439
> Project: HBase
>  Issue Type: Bug
>  Components: integration tests
>Reporter: Mike Drob
>Assignee: Jui-Yu Hsieh
>Priority: Minor
>  Labels: beginner
>
> A bunch of the actions all use the same logger inherited from the super 
> class. We should have them declare distinct loggers, either each one in class 
> or perhaps we can do something dynamically like 
> {{LogFactory.getLogger(MethodHandles.lookup().lookupClass()}} and drop the 
> static modifier on the log field.
> I'm not sure that exact incantation would actually work, but the 
> MethodHandles approach in general is how logger resolution happens in Solr 
> and it actually works out pretty well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18439) Subclasses of o.a.h.h.chaos.actions.Action all use the same logger

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115731#comment-16115731
 ] 

Sahil Aggarwal commented on HBASE-18439:


[~johnfly2003] Sure, i haven't started on it yet. I can't find you to assign it 
to you, probably because you are not added to contributers.

> Subclasses of o.a.h.h.chaos.actions.Action all use the same logger
> --
>
> Key: HBASE-18439
> URL: https://issues.apache.org/jira/browse/HBASE-18439
> Project: HBase
>  Issue Type: Bug
>  Components: integration tests
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>Priority: Minor
>  Labels: beginner
>
> A bunch of the actions all use the same logger inherited from the super 
> class. We should have them declare distinct loggers, either each one in class 
> or perhaps we can do something dynamically like 
> {{LogFactory.getLogger(MethodHandles.lookup().lookupClass()}} and drop the 
> static modifier on the log field.
> I'm not sure that exact incantation would actually work, but the 
> MethodHandles approach in general is how logger resolution happens in Solr 
> and it actually works out pretty well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HBASE-18439) Subclasses of o.a.h.h.chaos.actions.Action all use the same logger

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal reassigned HBASE-18439:
--

Assignee: Sahil Aggarwal

> Subclasses of o.a.h.h.chaos.actions.Action all use the same logger
> --
>
> Key: HBASE-18439
> URL: https://issues.apache.org/jira/browse/HBASE-18439
> Project: HBase
>  Issue Type: Bug
>  Components: integration tests
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>Priority: Minor
>  Labels: beginner
>
> A bunch of the actions all use the same logger inherited from the super 
> class. We should have them declare distinct loggers, either each one in class 
> or perhaps we can do something dynamically like 
> {{LogFactory.getLogger(MethodHandles.lookup().lookupClass()}} and drop the 
> static modifier on the log field.
> I'm not sure that exact incantation would actually work, but the 
> MethodHandles approach in general is how logger resolution happens in Solr 
> and it actually works out pretty well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HBASE-18439) Subclasses of o.a.h.h.chaos.actions.Action all use the same logger

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal reassigned HBASE-18439:
--

Assignee: (was: Sahil Aggarwal)

> Subclasses of o.a.h.h.chaos.actions.Action all use the same logger
> --
>
> Key: HBASE-18439
> URL: https://issues.apache.org/jira/browse/HBASE-18439
> Project: HBase
>  Issue Type: Bug
>  Components: integration tests
>Reporter: Mike Drob
>Priority: Minor
>  Labels: beginner
>
> A bunch of the actions all use the same logger inherited from the super 
> class. We should have them declare distinct loggers, either each one in class 
> or perhaps we can do something dynamically like 
> {{LogFactory.getLogger(MethodHandles.lookup().lookupClass()}} and drop the 
> static modifier on the log field.
> I'm not sure that exact incantation would actually work, but the 
> MethodHandles approach in general is how logger resolution happens in Solr 
> and it actually works out pretty well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HBASE-18211) Encryption of exisiting data in Stripe Compaction

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal reassigned HBASE-18211:
--

Assignee: Sahil Aggarwal

> Encryption of exisiting data in Stripe Compaction
> -
>
> Key: HBASE-18211
> URL: https://issues.apache.org/jira/browse/HBASE-18211
> Project: HBase
>  Issue Type: Bug
>  Components: Compaction, encryption
>Reporter: Karthick
>Assignee: Sahil Aggarwal
>Priority: Critical
>  Labels: compaction, encryption
>
> We have a table which has time series data with Stripe Compaction enabled. 
> After encryption has been enabled for this table the newer entries are 
> encrypted and inserted. However to encrypt the existing data in the table, a 
> major compaction has to run. Since, stripe compaction doesn't allow a major 
> compaction to run, we are unable to encrypt the previous data. 
> see this 
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/StripeCompactionPolicy.java



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18211) Encryption of exisiting data in Stripe Compaction

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112689#comment-16112689
 ] 

Sahil Aggarwal commented on HBASE-18211:


Would like to give it a shot. Assigning to myself.

> Encryption of exisiting data in Stripe Compaction
> -
>
> Key: HBASE-18211
> URL: https://issues.apache.org/jira/browse/HBASE-18211
> Project: HBase
>  Issue Type: Bug
>  Components: Compaction, encryption
>Reporter: Karthick
>Priority: Critical
>  Labels: compaction, encryption
>
> We have a table which has time series data with Stripe Compaction enabled. 
> After encryption has been enabled for this table the newer entries are 
> encrypted and inserted. However to encrypt the existing data in the table, a 
> major compaction has to run. Since, stripe compaction doesn't allow a major 
> compaction to run, we are unable to encrypt the previous data. 
> see this 
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/StripeCompactionPolicy.java



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (HBASE-17104) Improve cryptic error message "Memstore size is" on region close

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110233#comment-16110233
 ] 

Sahil Aggarwal edited comment on HBASE-17104 at 8/2/17 3:48 AM:


[~tedyu] How do i request for review?


was (Author: awked06):
How do i request for review?

> Improve cryptic error message "Memstore size is" on region close
> 
>
> Key: HBASE-17104
> URL: https://issues.apache.org/jira/browse/HBASE-17104
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Reporter: Matteo Bertozzi
>Assignee: Sahil Aggarwal
>Priority: Trivial
>  Labels: beginner, noob
> Fix For: 2.0.0
>
> Attachments: HBASE-17104.master.001 (1).patch, 
> HBASE-17104.master.001.patch
>
>
> while grepping my RS log for ERROR I found a cryptic
> {noformat}
> ERROR [RS_CLOSE_REGION-u1604vm:35021-1] regionserver.HRegion(1601): Memstore 
> size is 33744
> {noformat}
> from the code looks like we seems to want to notify the user about the fact 
> that on close the rs was not able to flush and there were things in the RS. 
> https://github.com/apache/hbase/blob/c3685760f004450667920144f926383eb307de53/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L1601
> {code}
> if (!canFlush) {
>   this.decrMemstoreSize(new MemstoreSize(memstoreDataSize.get(), 
> getMemstoreHeapOverhead()));
> } else if (memstoreDataSize.get() != 0) {
>   LOG.error("Memstore size is " + memstoreDataSize.get());
> }
> {code}
> this should probably not even be an error but a warn or even info, unless we 
> have puts that specifically asked to not be written to the wal,  otherwise 
> the data in the memstore should be safe in the wals. 
> In any case it will be nice to have a message describing what is going on and 
> why we are notifying about the memstore size.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-17104) Improve cryptic error message "Memstore size is" on region close

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110233#comment-16110233
 ] 

Sahil Aggarwal commented on HBASE-17104:


How do i request for review?

> Improve cryptic error message "Memstore size is" on region close
> 
>
> Key: HBASE-17104
> URL: https://issues.apache.org/jira/browse/HBASE-17104
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Reporter: Matteo Bertozzi
>Assignee: Sahil Aggarwal
>Priority: Trivial
>  Labels: beginner, noob
> Fix For: 2.0.0
>
> Attachments: HBASE-17104.master.001 (1).patch, 
> HBASE-17104.master.001.patch
>
>
> while grepping my RS log for ERROR I found a cryptic
> {noformat}
> ERROR [RS_CLOSE_REGION-u1604vm:35021-1] regionserver.HRegion(1601): Memstore 
> size is 33744
> {noformat}
> from the code looks like we seems to want to notify the user about the fact 
> that on close the rs was not able to flush and there were things in the RS. 
> https://github.com/apache/hbase/blob/c3685760f004450667920144f926383eb307de53/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L1601
> {code}
> if (!canFlush) {
>   this.decrMemstoreSize(new MemstoreSize(memstoreDataSize.get(), 
> getMemstoreHeapOverhead()));
> } else if (memstoreDataSize.get() != 0) {
>   LOG.error("Memstore size is " + memstoreDataSize.get());
> }
> {code}
> this should probably not even be an error but a warn or even info, unless we 
> have puts that specifically asked to not be written to the wal,  otherwise 
> the data in the memstore should be safe in the wals. 
> In any case it will be nice to have a message describing what is going on and 
> why we are notifying about the memstore size.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Work started] (HBASE-17104) Improve cryptic error message "Memstore size is" on region close

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HBASE-17104 started by Sahil Aggarwal.
--
> Improve cryptic error message "Memstore size is" on region close
> 
>
> Key: HBASE-17104
> URL: https://issues.apache.org/jira/browse/HBASE-17104
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Reporter: Matteo Bertozzi
>Assignee: Sahil Aggarwal
>Priority: Trivial
>  Labels: beginner, noob
> Fix For: 2.0.0
>
> Attachments: HBASE-17104.master.001 (1).patch, 
> HBASE-17104.master.001.patch
>
>
> while grepping my RS log for ERROR I found a cryptic
> {noformat}
> ERROR [RS_CLOSE_REGION-u1604vm:35021-1] regionserver.HRegion(1601): Memstore 
> size is 33744
> {noformat}
> from the code looks like we seems to want to notify the user about the fact 
> that on close the rs was not able to flush and there were things in the RS. 
> https://github.com/apache/hbase/blob/c3685760f004450667920144f926383eb307de53/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L1601
> {code}
> if (!canFlush) {
>   this.decrMemstoreSize(new MemstoreSize(memstoreDataSize.get(), 
> getMemstoreHeapOverhead()));
> } else if (memstoreDataSize.get() != 0) {
>   LOG.error("Memstore size is " + memstoreDataSize.get());
> }
> {code}
> this should probably not even be an error but a warn or even info, unless we 
> have puts that specifically asked to not be written to the wal,  otherwise 
> the data in the memstore should be safe in the wals. 
> In any case it will be nice to have a message describing what is going on and 
> why we are notifying about the memstore size.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Work stopped] (HBASE-17104) Improve cryptic error message "Memstore size is" on region close

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HBASE-17104 stopped by Sahil Aggarwal.
--
> Improve cryptic error message "Memstore size is" on region close
> 
>
> Key: HBASE-17104
> URL: https://issues.apache.org/jira/browse/HBASE-17104
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Reporter: Matteo Bertozzi
>Assignee: Sahil Aggarwal
>Priority: Trivial
>  Labels: beginner, noob
> Fix For: 2.0.0
>
> Attachments: HBASE-17104.master.001 (1).patch, 
> HBASE-17104.master.001.patch
>
>
> while grepping my RS log for ERROR I found a cryptic
> {noformat}
> ERROR [RS_CLOSE_REGION-u1604vm:35021-1] regionserver.HRegion(1601): Memstore 
> size is 33744
> {noformat}
> from the code looks like we seems to want to notify the user about the fact 
> that on close the rs was not able to flush and there were things in the RS. 
> https://github.com/apache/hbase/blob/c3685760f004450667920144f926383eb307de53/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L1601
> {code}
> if (!canFlush) {
>   this.decrMemstoreSize(new MemstoreSize(memstoreDataSize.get(), 
> getMemstoreHeapOverhead()));
> } else if (memstoreDataSize.get() != 0) {
>   LOG.error("Memstore size is " + memstoreDataSize.get());
> }
> {code}
> this should probably not even be an error but a warn or even info, unless we 
> have puts that specifically asked to not be written to the wal,  otherwise 
> the data in the memstore should be safe in the wals. 
> In any case it will be nice to have a message describing what is going on and 
> why we are notifying about the memstore size.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Work started] (HBASE-17104) Improve cryptic error message "Memstore size is" on region close

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HBASE-17104 started by Sahil Aggarwal.
--
> Improve cryptic error message "Memstore size is" on region close
> 
>
> Key: HBASE-17104
> URL: https://issues.apache.org/jira/browse/HBASE-17104
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Reporter: Matteo Bertozzi
>Assignee: Sahil Aggarwal
>Priority: Trivial
>  Labels: beginner, noob
> Fix For: 2.0.0
>
> Attachments: HBASE-17104.master.001 (1).patch, 
> HBASE-17104.master.001.patch
>
>
> while grepping my RS log for ERROR I found a cryptic
> {noformat}
> ERROR [RS_CLOSE_REGION-u1604vm:35021-1] regionserver.HRegion(1601): Memstore 
> size is 33744
> {noformat}
> from the code looks like we seems to want to notify the user about the fact 
> that on close the rs was not able to flush and there were things in the RS. 
> https://github.com/apache/hbase/blob/c3685760f004450667920144f926383eb307de53/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L1601
> {code}
> if (!canFlush) {
>   this.decrMemstoreSize(new MemstoreSize(memstoreDataSize.get(), 
> getMemstoreHeapOverhead()));
> } else if (memstoreDataSize.get() != 0) {
>   LOG.error("Memstore size is " + memstoreDataSize.get());
> }
> {code}
> this should probably not even be an error but a warn or even info, unless we 
> have puts that specifically asked to not be written to the wal,  otherwise 
> the data in the memstore should be safe in the wals. 
> In any case it will be nice to have a message describing what is going on and 
> why we are notifying about the memstore size.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HBASE-18439) Subclasses of o.a.h.h.chaos.actions.Action all use the same logger

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal reassigned HBASE-18439:
--

Assignee: Sahil Aggarwal

> Subclasses of o.a.h.h.chaos.actions.Action all use the same logger
> --
>
> Key: HBASE-18439
> URL: https://issues.apache.org/jira/browse/HBASE-18439
> Project: HBase
>  Issue Type: Bug
>  Components: integration tests
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>Priority: Minor
>  Labels: beginner
>
> A bunch of the actions all use the same logger inherited from the super 
> class. We should have them declare distinct loggers, either each one in class 
> or perhaps we can do something dynamically like 
> {{LogFactory.getLogger(MethodHandles.lookup().lookupClass()}} and drop the 
> static modifier on the log field.
> I'm not sure that exact incantation would actually work, but the 
> MethodHandles approach in general is how logger resolution happens in Solr 
> and it actually works out pretty well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18142) Deletion of a cell deletes the previous versions too

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110214#comment-16110214
 ] 

Sahil Aggarwal commented on HBASE-18142:


[~chia7712] Thanks for clarification. We probably then need to decouple 
_createdelete_internal for deleteall and delete, something like 
_createdelete_internal and _createdeleteall_internal or just pass flag in 
_createdelete_internal to call d.addColumns/d.addColumn.

> Deletion of a cell deletes the previous versions too
> 
>
> Key: HBASE-18142
> URL: https://issues.apache.org/jira/browse/HBASE-18142
> Project: HBase
>  Issue Type: Bug
>  Components: API
>Affects Versions: 3.0.0
>Reporter: Karthick
>Assignee: ChunHao
>  Labels: beginner
> Fix For: 3.0.0
>
> Attachments: HBASE-18142.master.v0.patch
>
>
> When I tried to delete a cell using it's timestamp in the Hbase Shell, the 
> previous versions of the same cell also got deleted. But when I tried the 
> same using the Java API, then the previous versions are not deleted and I can 
> retrive the previous values.
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java
> see this file to fix the issue. This method (public Delete addColumns(final 
> byte [] family, final byte [] qualifier, final long timestamp)) only deletes 
> the current version of the cell. The previous versions are not deleted.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18142) Deletion of a cell deletes the previous versions too

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104838#comment-16104838
 ] 

Sahil Aggarwal commented on HBASE-18142:


Noob quest:

On looking at  _deleteall_internal in table.rb, if row name is a Hash then we 
call _deleterows_internal but here we don't honor the timestamp provided in row 
hash and use latest timestamp, could this be the reason we are deleting all the 
versions of all the cells in that row?

> Deletion of a cell deletes the previous versions too
> 
>
> Key: HBASE-18142
> URL: https://issues.apache.org/jira/browse/HBASE-18142
> Project: HBase
>  Issue Type: Bug
>  Components: API
>Reporter: Karthick
>  Labels: beginner
>
> When I tried to delete a cell using it's timestamp in the Hbase Shell, the 
> previous versions of the same cell also got deleted. But when I tried the 
> same using the Java API, then the previous versions are not deleted and I can 
> retrive the previous values.
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java
> see this file to fix the issue. This method (public Delete addColumns(final 
> byte [] family, final byte [] qualifier, final long timestamp)) only deletes 
> the current version of the cell. The previous versions are not deleted.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-4214) Per-region request counters should be clearer about scope

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15844004#comment-15844004
 ] 

Sahil Aggarwal commented on HBASE-4214:
---

Will changing the heading from 'Read Request Count' -> 'Total Read requests' 
and 'Write RequestCount' -> 'Total Write Requests' help here? 

Moreover, to get Read/Write count per second can't we use 
MetricsRegionServerWrapperImpl to use 
Map/Map  and calculate it the 
way we are calculating 'Requests per second' for a region server ? 

> Per-region request counters should be clearer about scope
> -
>
> Key: HBASE-4214
> URL: https://issues.apache.org/jira/browse/HBASE-4214
> Project: HBase
>  Issue Type: Bug
>  Components: metrics, regionserver
>Affects Versions: 2.0.0
>Reporter: Todd Lipcon
>Priority: Trivial
>  Labels: beginner
>
> In testing trunk, I noticed that per-region request counters shown on 
> table.jsp are lifetime-scoped, rather than per-second or some other time 
> range. However, I'm pretty sure they reset when the region is moved. So, it's 
> hard to use them to judge relative hotness of regions from the web UI without 
> hooking it up to something lik OpenTSDB



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12408) Clarify ref guide findbugs section's use of "Apache licensed version of annotations"

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-12408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-12408:
---
Attachment: HBASE-12408.master.001.patch

> Clarify ref guide findbugs section's use of "Apache licensed version of 
> annotations"
> 
>
> Key: HBASE-12408
> URL: https://issues.apache.org/jira/browse/HBASE-12408
> Project: HBase
>  Issue Type: Improvement
>  Components: documentation
>Reporter: Sean Busbey
>Priority: Trivial
>  Labels: beginner
> Attachments: HBASE-12408.master.001.patch
>
>
> The ref guide has a [findbugs 
> section|http://hbase.apache.org/book.html#common.patch.feedback.findbugs] 
> that ends with this line
> {quote}
> It is important to use the Apache-licensed version of the annotations.
> {quote}
> without any additional context, it's hard to figure out what we're talking 
> about or why. The site for Findbugs itself only ever talks about its own 
> license, which is LGPL.
> I think what we're getting at is that we use the fully qualified 
> SupressWarnings instead of the newer SuppressFBWarnings because we rely on 
> the [cleanroom ASL reimplementation of findbugs 
> annotations|https://github.com/stephenc/findbugs-annotations].
> We should clarify that and provide some links.
> * [findbugs project page|http://findbugs.sourceforge.net/]
> * [cleanroom project page|https://github.com/stephenc/findbugs-annotations]
> * [javadocs|http://stephenc.github.io/findbugs-annotations/apidocs/]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16390) Fix documentation around setAutoFlush

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-16390:
---
Attachment: HBASE-16390.master.001.patch

> Fix documentation around setAutoFlush
> -
>
> Key: HBASE-16390
> URL: https://issues.apache.org/jira/browse/HBASE-16390
> Project: HBase
>  Issue Type: Bug
>  Components: documentation
>Reporter: stack
>Assignee: Sahil Aggarwal
>Priority: Minor
>  Labels: beginner
> Attachments: HBASE-16390.master.001.patch
>
>
> Our documentation is a little confused around setAutoFlush. Talks of Table 
> but setAutoFlush is not in the Table interface. It was on HTable but was 
> deprecated and since removed. Clean up the doc:
> {code}
> 100.4. HBase Client: AutoFlush
> When performing a lot of Puts, make sure that setAutoFlush is set to false
> on your Table
> 
> instance.
> Otherwise, the Puts will be sent one at a time to the RegionServer. Puts
> added via table.add(Put) and table.add(  Put) wind up in the same
> write buffer. If autoFlush = false, these messages are not sent until the
> write-buffer is filled. To explicitly flush the messages, call flushCommits.
> Calling close on the Table instance will invoke flushCommits
> {code}
> Spotted by Jeff Shmain.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16390) Fix documentation around setAutoFlush

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833291#comment-15833291
 ] 

Sahil Aggarwal commented on HBASE-16390:


We still have setAutoFlush in HTableInterface but HTableInterface is deprecated 
and not being used anywhere. Should i clean up HTableInterface too or separate 
ticket should be created for it ? 

> Fix documentation around setAutoFlush
> -
>
> Key: HBASE-16390
> URL: https://issues.apache.org/jira/browse/HBASE-16390
> Project: HBase
>  Issue Type: Bug
>  Components: documentation
>Reporter: stack
>Priority: Minor
>  Labels: beginner
>
> Our documentation is a little confused around setAutoFlush. Talks of Table 
> but setAutoFlush is not in the Table interface. It was on HTable but was 
> deprecated and since removed. Clean up the doc:
> {code}
> 100.4. HBase Client: AutoFlush
> When performing a lot of Puts, make sure that setAutoFlush is set to false
> on your Table
> 
> instance.
> Otherwise, the Puts will be sent one at a time to the RegionServer. Puts
> added via table.add(Put) and table.add(  Put) wind up in the same
> write buffer. If autoFlush = false, these messages are not sent until the
> write-buffer is filled. To explicitly flush the messages, call flushCommits.
> Calling close on the Table instance will invoke flushCommits
> {code}
> Spotted by Jeff Shmain.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17104) Improve cryptic error message "Memstore size is" on region close

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833288#comment-15833288
 ] 

Sahil Aggarwal commented on HBASE-17104:


Thanks [~yuzhih...@gmail.com] and [~stack]

> Improve cryptic error message "Memstore size is" on region close
> 
>
> Key: HBASE-17104
> URL: https://issues.apache.org/jira/browse/HBASE-17104
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Reporter: Matteo Bertozzi
>Assignee: Sahil Aggarwal
>Priority: Trivial
>  Labels: beginner, noob
> Fix For: 2.0.0
>
> Attachments: HBASE-17104.master.001.patch
>
>
> while grepping my RS log for ERROR I found a cryptic
> {noformat}
> ERROR [RS_CLOSE_REGION-u1604vm:35021-1] regionserver.HRegion(1601): Memstore 
> size is 33744
> {noformat}
> from the code looks like we seems to want to notify the user about the fact 
> that on close the rs was not able to flush and there were things in the RS. 
> https://github.com/apache/hbase/blob/c3685760f004450667920144f926383eb307de53/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L1601
> {code}
> if (!canFlush) {
>   this.decrMemstoreSize(new MemstoreSize(memstoreDataSize.get(), 
> getMemstoreHeapOverhead()));
> } else if (memstoreDataSize.get() != 0) {
>   LOG.error("Memstore size is " + memstoreDataSize.get());
> }
> {code}
> this should probably not even be an error but a warn or even info, unless we 
> have puts that specifically asked to not be written to the wal,  otherwise 
> the data in the memstore should be safe in the wals. 
> In any case it will be nice to have a message describing what is going on and 
> why we are notifying about the memstore size.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17104) Improve cryptic error message "Memstore size is" on region close

[Sahil Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Aggarwal updated HBASE-17104:
---
Attachment: HBASE-17104.master.001.patch

> Improve cryptic error message "Memstore size is" on region close
> 
>
> Key: HBASE-17104
> URL: https://issues.apache.org/jira/browse/HBASE-17104
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Reporter: Matteo Bertozzi
>Assignee: Sahil Aggarwal
>Priority: Trivial
>  Labels: beginner, noob
> Fix For: 2.0.0
>
> Attachments: HBASE-17104.master.001.patch
>
>
> while grepping my RS log for ERROR I found a cryptic
> {noformat}
> ERROR [RS_CLOSE_REGION-u1604vm:35021-1] regionserver.HRegion(1601): Memstore 
> size is 33744
> {noformat}
> from the code looks like we seems to want to notify the user about the fact 
> that on close the rs was not able to flush and there were things in the RS. 
> https://github.com/apache/hbase/blob/c3685760f004450667920144f926383eb307de53/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L1601
> {code}
> if (!canFlush) {
>   this.decrMemstoreSize(new MemstoreSize(memstoreDataSize.get(), 
> getMemstoreHeapOverhead()));
> } else if (memstoreDataSize.get() != 0) {
>   LOG.error("Memstore size is " + memstoreDataSize.get());
> }
> {code}
> this should probably not even be an error but a warn or even info, unless we 
> have puts that specifically asked to not be written to the wal,  otherwise 
> the data in the memstore should be safe in the wals. 
> In any case it will be nice to have a message describing what is going on and 
> why we are notifying about the memstore size.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17104) Improve cryptic error message "Memstore size is" on region close

[Sahil Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833050#comment-15833050
 ] 

Sahil Aggarwal commented on HBASE-17104:


We end up here only when abort=true and canFlush=true(region is writable) in 
which case we don't flush memstore. If abort=false then we should have 
definitely flushed memstore by now and size would be 0 if canFlush=true.

I have created patch for it but not able to submit using 
dev-support/submit-patch.py script. Getting error: 

CRITICAL:submit-patch: Oops, something went wrong when uploading patch to jira.
Response: 403 Forbidden

> Improve cryptic error message "Memstore size is" on region close
> 
>
> Key: HBASE-17104
> URL: https://issues.apache.org/jira/browse/HBASE-17104
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Reporter: Matteo Bertozzi
>Priority: Trivial
>  Labels: beginner, noob
> Fix For: 2.0.0
>
>
> while grepping my RS log for ERROR I found a cryptic
> {noformat}
> ERROR [RS_CLOSE_REGION-u1604vm:35021-1] regionserver.HRegion(1601): Memstore 
> size is 33744
> {noformat}
> from the code looks like we seems to want to notify the user about the fact 
> that on close the rs was not able to flush and there were things in the RS. 
> https://github.com/apache/hbase/blob/c3685760f004450667920144f926383eb307de53/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L1601
> {code}
> if (!canFlush) {
>   this.decrMemstoreSize(new MemstoreSize(memstoreDataSize.get(), 
> getMemstoreHeapOverhead()));
> } else if (memstoreDataSize.get() != 0) {
>   LOG.error("Memstore size is " + memstoreDataSize.get());
> }
> {code}
> this should probably not even be an error but a warn or even info, unless we 
> have puts that specifically asked to not be written to the wal,  otherwise 
> the data in the memstore should be safe in the wals. 
> In any case it will be nice to have a message describing what is going on and 
> why we are notifying about the memstore size.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)