[jira] [Commented] (HBASE-14643) Avoid Splits from once again opening a closed reader for fetching the first and last key

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964702#comment-14964702
 ] 

Hadoop QA commented on HBASE-14643:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12767531/HBASE-14643_v3.patch
  against master branch at commit c1f0442045f44fcbb3935f9244794929a5d0caea.
  ATTACHMENT ID: 12767531

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 4 new 
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 
2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:red}-1 checkstyle{color}.  The applied patch generated 
1748 checkstyle errors (more than the master's current 1747 errors).

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16108//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16108//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16108//artifact/patchprocess/checkstyle-aggregate.html

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16108//console

This message is automatically generated.

> Avoid Splits from once again opening a closed reader for fetching the first 
> and last key
> 
>
> Key: HBASE-14643
> URL: https://issues.apache.org/jira/browse/HBASE-14643
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: Heng Chen
> Attachments: HBASE-14643.patch, HBASE-14643_v1.patch, 
> HBASE-14643_v2.patch, HBASE-14643_v3.patch, HBASE-14643_v4.patch
>
>
> Currently split flow is such that we close the parent region and all its 
> store file readers are also closed.  After that inorder to split the 
> reference files we need the first and last keys for which once again open the 
> readers on those store files.  This could be costlier operation considering 
> the fact that it has to contact the HDFS for this close and open operation. 
> This JIRA is to see if we can improve this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14636) Clear HFileScannerImpl#prevBlocks in between Compaction flow

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964705#comment-14964705
 ] 

Hadoop QA commented on HBASE-14636:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12767532/HBASE-14636_V2.patch
  against master branch at commit c1f0442045f44fcbb3935f9244794929a5d0caea.
  ATTACHMENT ID: 12767532

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 
2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16109//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16109//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16109//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16109//console

This message is automatically generated.

> Clear HFileScannerImpl#prevBlocks in between Compaction flow
> 
>
> Key: HBASE-14636
> URL: https://issues.apache.org/jira/browse/HBASE-14636
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver, Scanners
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-14636.patch, HBASE-14636.patch, 
> HBASE-14636_V2.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14643) Avoid Splits from once again opening a closed reader for fetching the first and last key

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964708#comment-14964708
 ] 

Anoop Sam John commented on HBASE-14643:


+1
nit:
public Comparator getCellComparator() {
Method name can be getComparator(). 

Which all branches we need this?  At least to branch-1 and branch-1.2 also IMO.

> Avoid Splits from once again opening a closed reader for fetching the first 
> and last key
> 
>
> Key: HBASE-14643
> URL: https://issues.apache.org/jira/browse/HBASE-14643
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: Heng Chen
> Attachments: HBASE-14643.patch, HBASE-14643_v1.patch, 
> HBASE-14643_v2.patch, HBASE-14643_v3.patch, HBASE-14643_v4.patch
>
>
> Currently split flow is such that we close the parent region and all its 
> store file readers are also closed.  After that inorder to split the 
> reference files we need the first and last keys for which once again open the 
> readers on those store files.  This could be costlier operation considering 
> the fact that it has to contact the HDFS for this close and open operation. 
> This JIRA is to see if we can improve this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14603) Disable timestamping of generated Javadocs so they are not all modified by each build

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964730#comment-14964730
 ] 

Hadoop QA commented on HBASE-14603:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12767533/HBASE-14603-v6.patch
  against master branch at commit c1f0442045f44fcbb3935f9244794929a5d0caea.
  ATTACHMENT ID: 12767533

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+0 tests included{color}.  The patch appears to be a 
documentation patch that doesn't require tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 
2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 lineLengths{color}.  The patch introduces the following lines 
longer than 100:
+
org.apache.hadoop.hbase.backup*:org.apache.hadoop.hbase.catalog:org.apache.hadoop.hbase.client.coprocessor:org.apache.hadoop.hbase.client.metrics:org.apache.hadoop.hbase.codec*:org.apache.hadoop.hbase.constraint:org.apache.hadoop.hbase.coprocessor.*:org.apache.hadoop.hbase.executor:org.apache.hadoop.hbase.fs:*.generated.*:org.apache.hadoop.hbase.io.hfile.*:org.apache.hadoop.hbase.mapreduce.hadoopbackport:org.apache.hadoop.hbase.mapreduce.replication:org.apache.hadoop.hbase.master.*:org.apache.hadoop.hbase.metrics*:org.apache.hadoop.hbase.migration:org.apache.hadoop.hbase.monitoring:org.apache.hadoop.hbase.p*:org.apache.hadoop.hbase.regionserver.compactions:org.apache.hadoop.hbase.regionserver.handler:org.apache.hadoop.hbase.regionserver.snapshot:org.apache.hadoop.hbase.replication.*:org.apache.hadoop.hbase.rest.filter:org.apache.hadoop.hbase.rest.model:org.apache.hadoop.hbase.rest.p*:org.apache.hadoop.hbase.security.*:org.apache.hadoop.hbase.thrift*:org.apache.hadoop.hbase.tmpl.*:org.apache.hadoop.hbase.tool:org.apache.hadoop.hbase.trace:org.apache.hadoop.hbase.util.byterange*:org.apache.hadoop.hbase.util.test:org.apache.hadoop.hbase.util.vint:org.apache.hadoop.hbase.zookeeper.lock:org.apache.hadoop.metrics2*
+
org.apache.hbase:hbase-annotations
+
+

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16110//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16110//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16110//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16110//console

This message is automatically generated.

> Disable timestamping of generated Javadocs so they are not all modified by 
> each build
> -
>
> Key: HBASE-14603
> URL: https://issues.apache.org/jira/browse/HBASE-14603
> Project: HBase
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 2.0.0
>Reporter: Misty Stanley-Jones
>Assignee: Misty Stanley-Jones
> Fix For: 2.0.0
>
> Attachments: HBASE-14603-v2.patch, HBASE-14603-v3.patch, 
> HBASE-14603-v4.patch, HBASE-14603-v5.patch, HBASE-14603-v6.patch, 
> HBASE-14603.patch
>
>
> If we disable the timestamps (which are in HTML comments), then every Javadoc 
> file won't show up as different every time we rebuild the site.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14636) Clear HFileScannerImpl#prevBlocks in between Compaction flow

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-14636:
---
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Pushed to master. Thanks for reviews Stack and Ram..

Thanks a lot Stack for taking time to test it again and confirm back that the 
OOME no longer exists.

> Clear HFileScannerImpl#prevBlocks in between Compaction flow
> 
>
> Key: HBASE-14636
> URL: https://issues.apache.org/jira/browse/HBASE-14636
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver, Scanners
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-14636.patch, HBASE-14636.patch, 
> HBASE-14636_V2.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964824#comment-14964824
 ] 

Ashish Singhi commented on HBASE-13153:
---

During a offline discussion with Anoop on this, we found that when the source 
hfiles are in a different FS and if the hfile requires a split then 
LoadIncrementalHFiles will open a remote reader to source hfile, scan the file 
and append the data to each of the file split. 
Since we anyway copy the hfiles to the local FS if the source hfiles are in 
remote FS later, so we thought we can optimize this by copying the hfiles to a 
temp directory in local FS if source hfiles are in a different FS first and 
then do a local read and write.

This is related to LoadIncrementalHFiles, when ever the source hfiles are in a 
different FS so I will handle this as part of another jira which will be 
subtask of this.
So in this jira there will be no change in the patch or doc related to this.

Any further review comments on the patch will be really appreciated.
Thanks Ted, Ram, Anoop and Matteo for the reviews till now.

> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, 
> HBASE-13153-v11.patch, HBASE-13153-v2.patch, HBASE-13153-v3.patch, 
> HBASE-13153-v4.patch, HBASE-13153-v5.patch, HBASE-13153-v6.patch, 
> HBASE-13153-v7.patch, HBASE-13153-v8.patch, HBASE-13153-v9.patch, 
> HBASE-13153.patch, HBase Bulk Load Replication-v1-1.pdf, HBase Bulk Load 
> Replication-v2.pdf, HBase Bulk Load Replication.pdf
>
>
> Currently we plan to use HBase Replication feature to deal with disaster 
> tolerance scenario.But we encounter an issue that we will use bulkload very 
> frequently,because bulkload bypass write path, and will not generate WAL, so 
> the data will not be replicated to backup cluster. It's inappropriate to 
> bukload twice both on active cluster and backup cluster. So i advise do some 
> modification to bulkload feature to enable bukload to both active cluster and 
> backup cluster



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14222) Improve DrainBarrier

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964828#comment-14964828
 ] 

Hadoop QA commented on HBASE-14222:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12767534/HBASE-14222-V2.patch
  against master branch at commit c1f0442045f44fcbb3935f9244794929a5d0caea.
  ATTACHMENT ID: 12767534

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 3 new 
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 
2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
 

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16111//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16111//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16111//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16111//console

This message is automatically generated.

> Improve DrainBarrier
> 
>
> Key: HBASE-14222
> URL: https://issues.apache.org/jira/browse/HBASE-14222
> Project: HBase
>  Issue Type: Bug
>  Components: util
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: HBASE-14222-V2.patch, HBASE-14222-V2.patch, 
> HBASE-14222.patch
>
>
> 1. {{DrainBarrier.stopAndDrainOps}} may wait forever if 
> {{DrainBarrier.endOp}} changes its state and calls {{Object.notifyAll}} just 
> before {{stopAndDrainOps}} enters the synchronized block to call 
> {{Object.wait}}. Moreover, {{Object.wait}} may wake up false-positively, and 
> {{stopAndDrainOps}} may break the block before outstanding operations are 
> complete.
> 2. Some tests for {{DrainBarrier}} catch and ignore {{AssertionError}} 
> explicitly thrown JUnit's {{fail}} method.
> The implementation of {{DrainBarrier}} is a little complex, and I'll fix and 
> refactor it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14331) a single callQueue related improvements

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964791#comment-14964791
 ] 

Hadoop QA commented on HBASE-14331:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12767545/HBASE-14331-V6.patch
  against master branch at commit c1f0442045f44fcbb3935f9244794929a5d0caea.
  ATTACHMENT ID: 12767545

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 6 new 
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 
2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:red}-1 checkstyle{color}.  The applied patch generated 
1751 checkstyle errors (more than the master's current 1747 errors).

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
 

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16113//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16113//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16113//artifact/patchprocess/checkstyle-aggregate.html

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16113//console

This message is automatically generated.

> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
>  Issue Type: Improvement
>  Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf, 
> BlockingQueuesPerformanceTestApp-output.txt, 
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java, 
> HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331-V4.patch, 
> HBASE-14331-V5.patch, HBASE-14331-V6.patch, HBASE-14331.patch, 
> HBASE-14331.patch, SemaphoreBasedBlockingQueue.java, 
> SemaphoreBasedLinkedBlockingQueue.java, 
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and 
> the {{put}} method, but not between takers, and not between putters. These 
> methods are implemented to take locks at the almost beginning of their logic. 
> HBASE-11355 introduces multiple call-queues to reduce such possible 
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using 
> multiple queues, since {{BlockingQueue}} blocks threads it is required to 
> prepare enough threads for each queue. It is possible that there is a queue 
> starving for threads while there is another queue where threads are idle. 
> Even if you can tune parameters to avoid such situations, the tuning is not 
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14643) Avoid Splits from once again opening a closed reader for fetching the first and last key

[Heng Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Heng Chen updated HBASE-14643:
--
Attachment: HBASE-14643_v4.patch

> Avoid Splits from once again opening a closed reader for fetching the first 
> and last key
> 
>
> Key: HBASE-14643
> URL: https://issues.apache.org/jira/browse/HBASE-14643
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: Heng Chen
> Attachments: HBASE-14643.patch, HBASE-14643_v1.patch, 
> HBASE-14643_v2.patch, HBASE-14643_v3.patch, HBASE-14643_v4.patch
>
>
> Currently split flow is such that we close the parent region and all its 
> store file readers are also closed.  After that inorder to split the 
> reference files we need the first and last keys for which once again open the 
> readers on those store files.  This could be costlier operation considering 
> the fact that it has to contact the HDFS for this close and open operation. 
> This JIRA is to see if we can improve this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14652) Improve / update publish-website script in dev-support

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964612#comment-14964612
 ] 

stack commented on HBASE-14652:
---

Looks great. It works?  Best thing is check it in so we can try it and find any 
issues... +1

> Improve / update publish-website script in dev-support
> --
>
> Key: HBASE-14652
> URL: https://issues.apache.org/jira/browse/HBASE-14652
> Project: HBase
>  Issue Type: Task
>  Components: scripts
>Affects Versions: 2.0.0
>Reporter: Misty Stanley-Jones
>Assignee: Misty Stanley-Jones
> Fix For: 2.0.0
>
> Attachments: HBASE-14652.patch
>
>
> Since I wrote the script some parts of the build process have changed. Need 
> to update it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14603) Disable timestamping of generated Javadocs so they are not all modified by each build

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964629#comment-14964629
 ] 

Hadoop QA commented on HBASE-14603:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12767525/HBASE-14603-v4.patch
  against master branch at commit c1f0442045f44fcbb3935f9244794929a5d0caea.
  ATTACHMENT ID: 12767525

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+0 tests included{color}.  The patch appears to be a 
documentation patch that doesn't require tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 
2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 lineLengths{color}.  The patch introduces the following lines 
longer than 100:
+
org.apache.hadoop.hbase.backup*:org.apache.hadoop.hbase.catalog:org.apache.hadoop.hbase.client.coprocessor:org.apache.hadoop.hbase.client.metrics:org.apache.hadoop.hbase.codec*:org.apache.hadoop.hbase.constraint:org.apache.hadoop.hbase.coprocessor.*:org.apache.hadoop.hbase.executor:org.apache.hadoop.hbase.fs:*.generated.*:org.apache.hadoop.hbase.io.hfile.*:org.apache.hadoop.hbase.mapreduce.hadoopbackport:org.apache.hadoop.hbase.mapreduce.replication:org.apache.hadoop.hbase.master.*:org.apache.hadoop.hbase.metrics*:org.apache.hadoop.hbase.migration:org.apache.hadoop.hbase.monitoring:org.apache.hadoop.hbase.p*:org.apache.hadoop.hbase.regionserver.compactions:org.apache.hadoop.hbase.regionserver.handler:org.apache.hadoop.hbase.regionserver.snapshot:org.apache.hadoop.hbase.replication.*:org.apache.hadoop.hbase.rest.filter:org.apache.hadoop.hbase.rest.model:org.apache.hadoop.hbase.rest.p*:org.apache.hadoop.hbase.security.*:org.apache.hadoop.hbase.thrift*:org.apache.hadoop.hbase.tmpl.*:org.apache.hadoop.hbase.tool:org.apache.hadoop.hbase.trace:org.apache.hadoop.hbase.util.byterange*:org.apache.hadoop.hbase.util.test:org.apache.hadoop.hbase.util.vint:org.apache.hadoop.hbase.zookeeper.lock:org.apache.hadoop.metrics2*
+
org.apache.hbase:hbase-annotations

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16106//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16106//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16106//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16106//console

This message is automatically generated.

> Disable timestamping of generated Javadocs so they are not all modified by 
> each build
> -
>
> Key: HBASE-14603
> URL: https://issues.apache.org/jira/browse/HBASE-14603
> Project: HBase
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 2.0.0
>Reporter: Misty Stanley-Jones
>Assignee: Misty Stanley-Jones
> Fix For: 2.0.0
>
> Attachments: HBASE-14603-v2.patch, HBASE-14603-v3.patch, 
> HBASE-14603-v4.patch, HBASE-14603-v5.patch, HBASE-14603-v6.patch, 
> HBASE-14603.patch
>
>
> If we disable the timestamps (which are in HTML comments), then every Javadoc 
> file won't show up as different every time we rebuild the site.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14643) Avoid Splits from once again opening a closed reader for fetching the first and last key

[Heng Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Heng Chen updated HBASE-14643:
--
Attachment: (was: HBASE-14643_v4.patch)

> Avoid Splits from once again opening a closed reader for fetching the first 
> and last key
> 
>
> Key: HBASE-14643
> URL: https://issues.apache.org/jira/browse/HBASE-14643
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: Heng Chen
> Attachments: HBASE-14643.patch, HBASE-14643_v1.patch, 
> HBASE-14643_v2.patch, HBASE-14643_v3.patch
>
>
> Currently split flow is such that we close the parent region and all its 
> store file readers are also closed.  After that inorder to split the 
> reference files we need the first and last keys for which once again open the 
> readers on those store files.  This could be costlier operation considering 
> the fact that it has to contact the HDFS for this close and open operation. 
> This JIRA is to see if we can improve this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14643) Avoid Splits from once again opening a closed reader for fetching the first and last key

[Heng Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Heng Chen updated HBASE-14643:
--
Attachment: HBASE-14643_v4.patch

Thanks [~anoop.hbase] and [~ram_krish].

I update patch as your suggestion.

> Avoid Splits from once again opening a closed reader for fetching the first 
> and last key
> 
>
> Key: HBASE-14643
> URL: https://issues.apache.org/jira/browse/HBASE-14643
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: Heng Chen
> Attachments: HBASE-14643.patch, HBASE-14643_v1.patch, 
> HBASE-14643_v2.patch, HBASE-14643_v3.patch, HBASE-14643_v4.patch
>
>
> Currently split flow is such that we close the parent region and all its 
> store file readers are also closed.  After that inorder to split the 
> reference files we need the first and last keys for which once again open the 
> readers on those store files.  This could be costlier operation considering 
> the fact that it has to contact the HDFS for this close and open operation. 
> This JIRA is to see if we can improve this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14643) Avoid Splits from once again opening a closed reader for fetching the first and last key

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964635#comment-14964635
 ] 

ramkrishna.s.vasudevan commented on HBASE-14643:


+1 on patch. Splits can be some what more faster now I think after this goes 
in. 

> Avoid Splits from once again opening a closed reader for fetching the first 
> and last key
> 
>
> Key: HBASE-14643
> URL: https://issues.apache.org/jira/browse/HBASE-14643
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: Heng Chen
> Attachments: HBASE-14643.patch, HBASE-14643_v1.patch, 
> HBASE-14643_v2.patch, HBASE-14643_v3.patch, HBASE-14643_v4.patch
>
>
> Currently split flow is such that we close the parent region and all its 
> store file readers are also closed.  After that inorder to split the 
> reference files we need the first and last keys for which once again open the 
> readers on those store files.  This could be costlier operation considering 
> the fact that it has to contact the HDFS for this close and open operation. 
> This JIRA is to see if we can improve this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-14604) Improve MoveCostFunction in StochasticLoadBalancer

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HBASE-14604.

Resolution: Fixed

Thanks for the patch, Guanghao

> Improve MoveCostFunction in StochasticLoadBalancer
> --
>
> Key: HBASE-14604
> URL: https://issues.apache.org/jira/browse/HBASE-14604
> Project: HBase
>  Issue Type: Bug
>  Components: Balancer
>Affects Versions: 0.98.15
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0, 1.3.0, 0.98.16
>
> Attachments: HBASE-14604-0.98.patch, HBASE-14604-0.98_v1.patch, 
> HBASE-14604-trunk.patch, HBASE-14604-trunk_v1.patch, HBASE-14604_98.diff, 
> HBASE-14604_98_with_ut.diff
>
>
> The code in MoveCoseFunction:
> {code}
> return scale(0, cluster.numRegions + META_MOVE_COST_MULT, moveCost);
> {code}
> It uses cluster.numRegions + META_MOVE_COST_MULT as the max value when scale 
> moveCost to [0,1]. But this should use maxMoves as the max value when cluster 
> have a lot of regions.
> Assume a cluster have 1 regions, maxMoves is 2500, it only scale moveCost 
> to [0, 0.25].
> Improve moveCost by use Math.min(cluster.numRegions, maxMoves) as the max 
> cost, so it can scale moveCost to [0,1].
> {code}
> return scale(0, Math.min(cluster.numRegions, maxMoves) + META_MOVE_COST_MULT, 
> moveCost);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14633) Try fluid width UI

[Elliott Clark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-14633:
--
Attachment: Screen Shot 2015-10-20 at 2.40.23 AM.png
Screen Shot 2015-10-20 at 2.40.34 AM.png

Screenshots of current state. Notice how everything has more room on a larger 
screen. It feels more utilitarian, but it's got more room for info.

> Try fluid width UI
> --
>
> Key: HBASE-14633
> URL: https://issues.apache.org/jira/browse/HBASE-14633
> Project: HBase
>  Issue Type: Bug
>  Components: UI
>Affects Versions: 2.0.0, 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-14633-v1.patch, HBASE-14633-v2.patch, 
> HBASE-14633.patch, Screen Shot 2015-10-20 at 2.40.23 AM.png, Screen Shot 
> 2015-10-20 at 2.40.34 AM.png
>
>
> Our UI is often too long. Lets give it more room if available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14654) Reenable TestMultiParallel#testActiveThreadsCount

[Heng Chen (JIRA)]

Heng Chen created HBASE-14654:
-

 Summary: Reenable TestMultiParallel#testActiveThreadsCount
 Key: HBASE-14654
 URL: https://issues.apache.org/jira/browse/HBASE-14654
 Project: HBase
  Issue Type: Bug
Reporter: Heng Chen


It was disabled in HBASE-14642,  this issue should reenable it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14654) Reenable TestMultiParallel#testActiveThreadsCount

[Heng Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Heng Chen updated HBASE-14654:
--
Attachment: HBASE-14654.patch

> Reenable TestMultiParallel#testActiveThreadsCount
> -
>
> Key: HBASE-14654
> URL: https://issues.apache.org/jira/browse/HBASE-14654
> Project: HBase
>  Issue Type: Bug
>Reporter: Heng Chen
> Attachments: HBASE-14654.patch
>
>
> It was disabled in HBASE-14642,  this issue should reenable it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14604) Improve MoveCostFunction in StochasticLoadBalancer

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14604:
---
 Hadoop Flags: Reviewed
Fix Version/s: 0.98.16
   1.3.0
   2.0.0

> Improve MoveCostFunction in StochasticLoadBalancer
> --
>
> Key: HBASE-14604
> URL: https://issues.apache.org/jira/browse/HBASE-14604
> Project: HBase
>  Issue Type: Bug
>  Components: Balancer
>Affects Versions: 0.98.15
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0, 1.3.0, 0.98.16
>
> Attachments: HBASE-14604-0.98.patch, HBASE-14604-0.98_v1.patch, 
> HBASE-14604-trunk.patch, HBASE-14604-trunk_v1.patch, HBASE-14604_98.diff, 
> HBASE-14604_98_with_ut.diff
>
>
> The code in MoveCoseFunction:
> {code}
> return scale(0, cluster.numRegions + META_MOVE_COST_MULT, moveCost);
> {code}
> It uses cluster.numRegions + META_MOVE_COST_MULT as the max value when scale 
> moveCost to [0,1]. But this should use maxMoves as the max value when cluster 
> have a lot of regions.
> Assume a cluster have 1 regions, maxMoves is 2500, it only scale moveCost 
> to [0, 0.25].
> Improve moveCost by use Math.min(cluster.numRegions, maxMoves) as the max 
> cost, so it can scale moveCost to [0,1].
> {code}
> return scale(0, Math.min(cluster.numRegions, maxMoves) + META_MOVE_COST_MULT, 
> moveCost);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14643) Avoid Splits from once again opening a closed reader for fetching the first and last key

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964872#comment-14964872
 ] 

Hadoop QA commented on HBASE-14643:
---

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12767549/HBASE-14643_v4.patch
  against master branch at commit c1f0442045f44fcbb3935f9244794929a5d0caea.
  ATTACHMENT ID: 12767549

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 4 new 
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 
2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16114//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16114//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16114//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16114//console

This message is automatically generated.

> Avoid Splits from once again opening a closed reader for fetching the first 
> and last key
> 
>
> Key: HBASE-14643
> URL: https://issues.apache.org/jira/browse/HBASE-14643
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: Heng Chen
> Attachments: HBASE-14643.patch, HBASE-14643_v1.patch, 
> HBASE-14643_v2.patch, HBASE-14643_v3.patch, HBASE-14643_v4.patch, 
> HBASE-14643_v5.patch
>
>
> Currently split flow is such that we close the parent region and all its 
> store file readers are also closed.  After that inorder to split the 
> reference files we need the first and last keys for which once again open the 
> readers on those store files.  This could be costlier operation considering 
> the fact that it has to contact the HDFS for this close and open operation. 
> This JIRA is to see if we can improve this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14633) Try fluid width UI

[Elliott Clark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-14633:
--
Attachment: HBASE-14633-v3.patch

Patch with updated css tweaks.

* Less space at the top.
* Make the hbase logo better positioned.
* margin on the left to keep things from feeling too crowded.

> Try fluid width UI
> --
>
> Key: HBASE-14633
> URL: https://issues.apache.org/jira/browse/HBASE-14633
> Project: HBase
>  Issue Type: Bug
>  Components: UI
>Affects Versions: 2.0.0, 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-14633-v1.patch, HBASE-14633-v2.patch, 
> HBASE-14633-v3.patch, HBASE-14633.patch, Screen Shot 2015-10-20 at 2.40.23 
> AM.png, Screen Shot 2015-10-20 at 2.40.34 AM.png, Screen Shot 2015-10-20 at 
> 2.44.11 AM.png
>
>
> Our UI is often too long. Lets give it more room if available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14633) Try fluid width UI

[Elliott Clark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-14633:
--
Attachment: Screen Shot 2015-10-20 at 2.44.11 AM.png

Safari

> Try fluid width UI
> --
>
> Key: HBASE-14633
> URL: https://issues.apache.org/jira/browse/HBASE-14633
> Project: HBase
>  Issue Type: Bug
>  Components: UI
>Affects Versions: 2.0.0, 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-14633-v1.patch, HBASE-14633-v2.patch, 
> HBASE-14633-v3.patch, HBASE-14633.patch, Screen Shot 2015-10-20 at 2.40.23 
> AM.png, Screen Shot 2015-10-20 at 2.40.34 AM.png, Screen Shot 2015-10-20 at 
> 2.44.11 AM.png
>
>
> Our UI is often too long. Lets give it more room if available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14633) Try fluid width UI

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964908#comment-14964908
 ] 

Hadoop QA commented on HBASE-14633:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12767575/HBASE-14633-v3.patch
  against master branch at commit 60465964039acd05f43f268cdb4f909a150a0f41.
  ATTACHMENT ID: 12767575

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16116//console

This message is automatically generated.

> Try fluid width UI
> --
>
> Key: HBASE-14633
> URL: https://issues.apache.org/jira/browse/HBASE-14633
> Project: HBase
>  Issue Type: Bug
>  Components: UI
>Affects Versions: 2.0.0, 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-14633-v1.patch, HBASE-14633-v2.patch, 
> HBASE-14633-v3.patch, HBASE-14633.patch, Screen Shot 2015-10-20 at 2.40.23 
> AM.png, Screen Shot 2015-10-20 at 2.40.34 AM.png, Screen Shot 2015-10-20 at 
> 2.44.11 AM.png
>
>
> Our UI is often too long. Lets give it more room if available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14366) NPE in case visibility expression is not present in labels table during importtsv run

[Bhupendra Kumar Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964954#comment-14964954
 ] 

Bhupendra Kumar Jain commented on HBASE-14366:
--

Test case failures are not related to this patch.

> NPE in case visibility expression is not present in labels table during 
> importtsv run
> -
>
> Key: HBASE-14366
> URL: https://issues.apache.org/jira/browse/HBASE-14366
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
> Attachments: 0001-HBASE-14366.patch, 0001-HBASE-14366_1.patch, 
> HBASE-14366-0.98.patch, HBASE-14366-branch-1.1.patch, 
> HBASE-14366-branch-1.1_v1.patch, HBASE-14366-branch-1.2.patch, 
> HBASE-14366-branch-1.2_v1.patch, HBASE-14366-branch-1.patch, 
> HBASE-14366_2(1).patch, HBASE-14366_2.patch
>
>
> Below exception is shown in logs if visibility expression is not present in 
> labels table during importtsv run. Appropriate exception / message should be 
> logged for the user to take further action.
> {code}
> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver$1.getLabelOrdinal(DefaultVisibilityExpressionResolver.java:127)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.getLabelOrdinals(VisibilityUtils.java:358)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.createVisibilityExpTags(VisibilityUtils.java:323)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.createVisibilityExpTags(DefaultVisibilityExpressionResolver.java:137)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.populatePut(TsvImporterMapper.java:205)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:165)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14366) NPE in case visibility expression is not present in labels table during importtsv run

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-14366:
---
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 0.98.16
   1.1.3
   1.0.3
   1.3.0
   1.2.0
   2.0.0
   Status: Resolved  (was: Patch Available)

Pushed to 0.98+ branches.  Thanks for the patch Bhupendra.

> NPE in case visibility expression is not present in labels table during 
> importtsv run
> -
>
> Key: HBASE-14366
> URL: https://issues.apache.org/jira/browse/HBASE-14366
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: 0001-HBASE-14366.patch, 0001-HBASE-14366_1.patch, 
> HBASE-14366-0.98.patch, HBASE-14366-branch-1.1.patch, 
> HBASE-14366-branch-1.1_v1.patch, HBASE-14366-branch-1.2.patch, 
> HBASE-14366-branch-1.2_v1.patch, HBASE-14366-branch-1.patch, 
> HBASE-14366_2(1).patch, HBASE-14366_2.patch
>
>
> Below exception is shown in logs if visibility expression is not present in 
> labels table during importtsv run. Appropriate exception / message should be 
> logged for the user to take further action.
> {code}
> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver$1.getLabelOrdinal(DefaultVisibilityExpressionResolver.java:127)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.getLabelOrdinals(VisibilityUtils.java:358)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.createVisibilityExpTags(VisibilityUtils.java:323)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.createVisibilityExpTags(DefaultVisibilityExpressionResolver.java:137)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.populatePut(TsvImporterMapper.java:205)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:165)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14642) Disable flakey TestMultiParallel#testActiveThreadsCount

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964891#comment-14964891
 ] 

Hudson commented on HBASE-14642:


SUCCESS: Integrated in HBase-TRUNK #6929 (See 
[https://builds.apache.org/job/HBase-TRUNK/6929/])
HBASE-14642 Update website-publishing script (mstanleyjones: rev 
51693b9cdeb1671ddaf8aa3eeb6eca2a57761bd6)
* dev-support/publish_hbase_website.sh


> Disable flakey TestMultiParallel#testActiveThreadsCount
> ---
>
> Key: HBASE-14642
> URL: https://issues.apache.org/jira/browse/HBASE-14642
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
> Attachments: 14642.txt
>
>
> Failed twice in a row on 1.2 build... Disabling for now Unless someone 
> wants to dig in and fix it that is...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14654) Reenable TestMultiParallel#testActiveThreadsCount

[Heng Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964911#comment-14964911
 ] 

Heng Chen commented on HBASE-14654:
---

I think the reason is that.
We submit batch request using ThreadPool, the relates code is in 
{{AsyncProcess.sendMultiAction}},  like below:
{code}
// run all the runnables
for (Runnable runnable : runnables) {
  if ((--actionsRemaining == 0) && reuseThread) {
runnable.run();
  } else {
try {
  pool.submit(runnable);  //Notice 
} catch (Throwable t) {
  .
  }
}
{code}

And this thread pool was constructed in {{HTable}} 
{code}
 ThreadPoolExecutor pool = new ThreadPoolExecutor(1, maxThreads, keepAliveTime, 
TimeUnit.SECONDS,
new SynchronousQueue(), 
Threads.newDaemonThreadFactory("htable"));
{code}

We notice that the first param (corePoolSize) is 1.  As ThreadPoolExecutor 
comments said.
{code}
 * When a new task is submitted in method {@link #execute(Runnable)},
 * and fewer than corePoolSize threads are running, a new thread is
 * created to handle the request, even if other worker threads are
 * idle.  If there are more than corePoolSize but less than
 * maximumPoolSize threads running, a new thread will be created only
 * if the queue is full. 
{code}

As current logic, corePoolSize less than maxPoolSize,  So it will be possible 
to reuse idle thread in ThreadPool.

So we can increase the corePoolSize to make sure every new request will be 
handled by a new thread.




> Reenable TestMultiParallel#testActiveThreadsCount
> -
>
> Key: HBASE-14654
> URL: https://issues.apache.org/jira/browse/HBASE-14654
> Project: HBase
>  Issue Type: Bug
>Reporter: Heng Chen
>
> It was disabled in HBASE-14642,  this issue should reenable it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14425) In Secure Zookeeper cluster superuser will not have sufficient permission if multiple values are configured in "hbase.superuser"

[Pankaj Kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pankaj Kumar updated HBASE-14425:
-
Attachment: HBASE-14425-V2.patch

Thanks [~enis] for reviewing the patch and pointing out the integration test 
changes. 
Sorry for late reply, have attached the V2 patch.

> In Secure Zookeeper cluster superuser will not have sufficient permission if 
> multiple values are configured in "hbase.superuser"
> 
>
> Key: HBASE-14425
> URL: https://issues.apache.org/jira/browse/HBASE-14425
> Project: HBase
>  Issue Type: Bug
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 2.0.0
>
> Attachments: HBASE-14425-V2.patch, HBASE-14425.patch
>
>
> During master intialization we are setting ACLs for the znodes.
> In ZKUtil.createACL(ZooKeeperWatcher zkw, String node, boolean 
> isSecureZooKeeper),
> {code}
>   String superUser = zkw.getConfiguration().get("hbase.superuser");
>   ArrayList acls = new ArrayList();
>   // add permission to hbase supper user
>   if (superUser != null) {
> acls.add(new ACL(Perms.ALL, new Id("auth", superUser)));
>   }
> {code}
> Here we are directly setting "hbase.superuser" value to Znode which will 
> cause an issue when multiple values are configured. In "hbase.superuser" 
> multiple superusers and supergroups can be configured separated by comma. We 
> need to iterate them and set ACL.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14651) Default minimum compaction size is too high

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964953#comment-14964953
 ] 

Anoop Sam John commented on HBASE-14651:


But the cluster configs to be adjusted so as to avoid the premature memstore 
flushes no?  So the flushed files will have sizes of this memstore flush size.  
If we reduce the def size of min compaction to be much smaller than this, will 
that be good.. at least we want the immediate flushed files to be compacted..  
And if user dont want that, he can always reduce the config value.  Do we 
really need to reduce this default value?

> Default minimum compaction size is too high
> ---
>
> Key: HBASE-14651
> URL: https://issues.apache.org/jira/browse/HBASE-14651
> Project: HBase
>  Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-14651-v1.patch
>
>
> *hbase.hstore.compaction.min.size* defines minimum selection size which is 
> always eligible for minor compaction (no compaction ratio check is performed 
> on such file selections). Default size is equals to memstore flush size 
> (128MB).  First of all, even this value is too high for some (many) 
> deployments, especially for write intensive, because of  a small sizes of a 
> memstore flushes, and if user increases memstore flush size (they usually set 
> it to at least 256MB), they have no idea how will it impact the overall 
> compaction process efficiency. With 256MB of minimum size to compact, 
> compactor most of the time skips necessary file ratio checks and this will 
> result in increased read/write IO during compactions, because of the 
> unbalanced selections where relatively large files can be mixed with a newly 
> created small store files. I think we should set this default minimum  to 
> 64MB and not to link it to memstore flush size at all. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14648) Reenable TestWALProcedureStoreOnHDFS#testWalRollOnLowReplication

[Heng Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Heng Chen updated HBASE-14648:
--
Status: Patch Available  (was: Open)

> Reenable TestWALProcedureStoreOnHDFS#testWalRollOnLowReplication
> 
>
> Key: HBASE-14648
> URL: https://issues.apache.org/jira/browse/HBASE-14648
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-14648.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14654) Reenable TestMultiParallel#testActiveThreadsCount

[Heng Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Heng Chen updated HBASE-14654:
--
Status: Patch Available  (was: Open)

> Reenable TestMultiParallel#testActiveThreadsCount
> -
>
> Key: HBASE-14654
> URL: https://issues.apache.org/jira/browse/HBASE-14654
> Project: HBase
>  Issue Type: Bug
>Reporter: Heng Chen
> Attachments: HBASE-14654.patch
>
>
> It was disabled in HBASE-14642,  this issue should reenable it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14604) Improve MoveCostFunction in StochasticLoadBalancer

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965140#comment-14965140
 ] 

Hudson commented on HBASE-14604:


FAILURE: Integrated in HBase-0.98 #1162 (See 
[https://builds.apache.org/job/HBase-0.98/1162/])
HBASE-14604 Improve MoveCostFunction in StochasticLoadBalancer (Guanghao 
(tedyu: rev 76a14b9ba682c76a81baa06168239a2de6d2c7de)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java


> Improve MoveCostFunction in StochasticLoadBalancer
> --
>
> Key: HBASE-14604
> URL: https://issues.apache.org/jira/browse/HBASE-14604
> Project: HBase
>  Issue Type: Bug
>  Components: Balancer
>Affects Versions: 0.98.15
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0, 1.3.0, 0.98.16
>
> Attachments: HBASE-14604-0.98.patch, HBASE-14604-0.98_v1.patch, 
> HBASE-14604-trunk.patch, HBASE-14604-trunk_v1.patch, HBASE-14604_98.diff, 
> HBASE-14604_98_with_ut.diff
>
>
> The code in MoveCoseFunction:
> {code}
> return scale(0, cluster.numRegions + META_MOVE_COST_MULT, moveCost);
> {code}
> It uses cluster.numRegions + META_MOVE_COST_MULT as the max value when scale 
> moveCost to [0,1]. But this should use maxMoves as the max value when cluster 
> have a lot of regions.
> Assume a cluster have 1 regions, maxMoves is 2500, it only scale moveCost 
> to [0, 0.25].
> Improve moveCost by use Math.min(cluster.numRegions, maxMoves) as the max 
> cost, so it can scale moveCost to [0,1].
> {code}
> return scale(0, Math.min(cluster.numRegions, maxMoves) + META_MOVE_COST_MULT, 
> moveCost);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14636) Clear HFileScannerImpl#prevBlocks in between Compaction flow

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965030#comment-14965030
 ] 

Hudson commented on HBASE-14636:


FAILURE: Integrated in HBase-TRUNK #6930 (See 
[https://builds.apache.org/job/HBase-TRUNK/6930/])
HBASE-14636 Clear HFileScannerImpl#prevBlocks in between Compaction 
(anoopsamjohn: rev c9523a569d45e9edc2c2d7b8d4d9cbf05f46a100)
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java


> Clear HFileScannerImpl#prevBlocks in between Compaction flow
> 
>
> Key: HBASE-14636
> URL: https://issues.apache.org/jira/browse/HBASE-14636
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver, Scanners
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-14636.patch, HBASE-14636.patch, 
> HBASE-14636_V2.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14604) Improve MoveCostFunction in StochasticLoadBalancer

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965029#comment-14965029
 ] 

Hudson commented on HBASE-14604:


FAILURE: Integrated in HBase-TRUNK #6930 (See 
[https://builds.apache.org/job/HBase-TRUNK/6930/])
HBASE-14604 Improve MoveCostFunction in StochasticLoadBalancer (Guanghao 
(tedyu: rev 60465964039acd05f43f268cdb4f909a150a0f41)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java


> Improve MoveCostFunction in StochasticLoadBalancer
> --
>
> Key: HBASE-14604
> URL: https://issues.apache.org/jira/browse/HBASE-14604
> Project: HBase
>  Issue Type: Bug
>  Components: Balancer
>Affects Versions: 0.98.15
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0, 1.3.0, 0.98.16
>
> Attachments: HBASE-14604-0.98.patch, HBASE-14604-0.98_v1.patch, 
> HBASE-14604-trunk.patch, HBASE-14604-trunk_v1.patch, HBASE-14604_98.diff, 
> HBASE-14604_98_with_ut.diff
>
>
> The code in MoveCoseFunction:
> {code}
> return scale(0, cluster.numRegions + META_MOVE_COST_MULT, moveCost);
> {code}
> It uses cluster.numRegions + META_MOVE_COST_MULT as the max value when scale 
> moveCost to [0,1]. But this should use maxMoves as the max value when cluster 
> have a lot of regions.
> Assume a cluster have 1 regions, maxMoves is 2500, it only scale moveCost 
> to [0, 0.25].
> Improve moveCost by use Math.min(cluster.numRegions, maxMoves) as the max 
> cost, so it can scale moveCost to [0,1].
> {code}
> return scale(0, Math.min(cluster.numRegions, maxMoves) + META_MOVE_COST_MULT, 
> moveCost);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14643) Avoid Splits from once again opening a closed reader for fetching the first and last key

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965003#comment-14965003
 ] 

Hadoop QA commented on HBASE-14643:
---

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12767554/HBASE-14643_v5.patch
  against master branch at commit c9523a569d45e9edc2c2d7b8d4d9cbf05f46a100.
  ATTACHMENT ID: 12767554

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 4 new 
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 
2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16115//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16115//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16115//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16115//console

This message is automatically generated.

> Avoid Splits from once again opening a closed reader for fetching the first 
> and last key
> 
>
> Key: HBASE-14643
> URL: https://issues.apache.org/jira/browse/HBASE-14643
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: Heng Chen
> Attachments: HBASE-14643.patch, HBASE-14643_v1.patch, 
> HBASE-14643_v2.patch, HBASE-14643_v3.patch, HBASE-14643_v4.patch, 
> HBASE-14643_v5.patch
>
>
> Currently split flow is such that we close the parent region and all its 
> store file readers are also closed.  After that inorder to split the 
> reference files we need the first and last keys for which once again open the 
> readers on those store files.  This could be costlier operation considering 
> the fact that it has to contact the HDFS for this close and open operation. 
> This JIRA is to see if we can improve this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14604) Improve MoveCostFunction in StochasticLoadBalancer

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965058#comment-14965058
 ] 

Hudson commented on HBASE-14604:


SUCCESS: Integrated in HBase-1.3 #282 (See 
[https://builds.apache.org/job/HBase-1.3/282/])
HBASE-14604 Improve MoveCostFunction in StochasticLoadBalancer (Guanghao 
(tedyu: rev b076d7dbd193bd83fee3ab54edf4570505192ddd)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java


> Improve MoveCostFunction in StochasticLoadBalancer
> --
>
> Key: HBASE-14604
> URL: https://issues.apache.org/jira/browse/HBASE-14604
> Project: HBase
>  Issue Type: Bug
>  Components: Balancer
>Affects Versions: 0.98.15
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0, 1.3.0, 0.98.16
>
> Attachments: HBASE-14604-0.98.patch, HBASE-14604-0.98_v1.patch, 
> HBASE-14604-trunk.patch, HBASE-14604-trunk_v1.patch, HBASE-14604_98.diff, 
> HBASE-14604_98_with_ut.diff
>
>
> The code in MoveCoseFunction:
> {code}
> return scale(0, cluster.numRegions + META_MOVE_COST_MULT, moveCost);
> {code}
> It uses cluster.numRegions + META_MOVE_COST_MULT as the max value when scale 
> moveCost to [0,1]. But this should use maxMoves as the max value when cluster 
> have a lot of regions.
> Assume a cluster have 1 regions, maxMoves is 2500, it only scale moveCost 
> to [0, 0.25].
> Improve moveCost by use Math.min(cluster.numRegions, maxMoves) as the max 
> cost, so it can scale moveCost to [0,1].
> {code}
> return scale(0, Math.min(cluster.numRegions, maxMoves) + META_MOVE_COST_MULT, 
> moveCost);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (HBASE-14652) Improve / update publish-website script in dev-support

[Misty Stanley-Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HBASE-14652 started by Misty Stanley-Jones.
---
> Improve / update publish-website script in dev-support
> --
>
> Key: HBASE-14652
> URL: https://issues.apache.org/jira/browse/HBASE-14652
> Project: HBase
>  Issue Type: Task
>  Components: scripts
>Affects Versions: 2.0.0
>Reporter: Misty Stanley-Jones
>Assignee: Misty Stanley-Jones
> Fix For: 2.0.0
>
> Attachments: HBASE-14652.patch
>
>
> Since I wrote the script some parts of the build process have changed. Need 
> to update it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14603) Disable timestamping of generated Javadocs so they are not all modified by each build

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964713#comment-14964713
 ] 

Hadoop QA commented on HBASE-14603:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12767533/HBASE-14603-v6.patch
  against master branch at commit c1f0442045f44fcbb3935f9244794929a5d0caea.
  ATTACHMENT ID: 12767533

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+0 tests included{color}.  The patch appears to be a 
documentation patch that doesn't require tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 
2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 lineLengths{color}.  The patch introduces the following lines 
longer than 100:
+
org.apache.hadoop.hbase.backup*:org.apache.hadoop.hbase.catalog:org.apache.hadoop.hbase.client.coprocessor:org.apache.hadoop.hbase.client.metrics:org.apache.hadoop.hbase.codec*:org.apache.hadoop.hbase.constraint:org.apache.hadoop.hbase.coprocessor.*:org.apache.hadoop.hbase.executor:org.apache.hadoop.hbase.fs:*.generated.*:org.apache.hadoop.hbase.io.hfile.*:org.apache.hadoop.hbase.mapreduce.hadoopbackport:org.apache.hadoop.hbase.mapreduce.replication:org.apache.hadoop.hbase.master.*:org.apache.hadoop.hbase.metrics*:org.apache.hadoop.hbase.migration:org.apache.hadoop.hbase.monitoring:org.apache.hadoop.hbase.p*:org.apache.hadoop.hbase.regionserver.compactions:org.apache.hadoop.hbase.regionserver.handler:org.apache.hadoop.hbase.regionserver.snapshot:org.apache.hadoop.hbase.replication.*:org.apache.hadoop.hbase.rest.filter:org.apache.hadoop.hbase.rest.model:org.apache.hadoop.hbase.rest.p*:org.apache.hadoop.hbase.security.*:org.apache.hadoop.hbase.thrift*:org.apache.hadoop.hbase.tmpl.*:org.apache.hadoop.hbase.tool:org.apache.hadoop.hbase.trace:org.apache.hadoop.hbase.util.byterange*:org.apache.hadoop.hbase.util.test:org.apache.hadoop.hbase.util.vint:org.apache.hadoop.hbase.zookeeper.lock:org.apache.hadoop.metrics2*
+
org.apache.hbase:hbase-annotations
+
+

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16107//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16107//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16107//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16107//console

This message is automatically generated.

> Disable timestamping of generated Javadocs so they are not all modified by 
> each build
> -
>
> Key: HBASE-14603
> URL: https://issues.apache.org/jira/browse/HBASE-14603
> Project: HBase
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 2.0.0
>Reporter: Misty Stanley-Jones
>Assignee: Misty Stanley-Jones
> Fix For: 2.0.0
>
> Attachments: HBASE-14603-v2.patch, HBASE-14603-v3.patch, 
> HBASE-14603-v4.patch, HBASE-14603-v5.patch, HBASE-14603-v6.patch, 
> HBASE-14603.patch
>
>
> If we disable the timestamps (which are in HTML comments), then every Javadoc 
> file won't show up as different every time we rebuild the site.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14643) Avoid Splits from once again opening a closed reader for fetching the first and last key

[Heng Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Heng Chen updated HBASE-14643:
--
Attachment: HBASE-14643_v5.patch

{quote}
public Comparator getCellComparator() {
Method name can be getComparator().
{quote}

Update patch as suggestion.

> Avoid Splits from once again opening a closed reader for fetching the first 
> and last key
> 
>
> Key: HBASE-14643
> URL: https://issues.apache.org/jira/browse/HBASE-14643
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: Heng Chen
> Attachments: HBASE-14643.patch, HBASE-14643_v1.patch, 
> HBASE-14643_v2.patch, HBASE-14643_v3.patch, HBASE-14643_v4.patch, 
> HBASE-14643_v5.patch
>
>
> Currently split flow is such that we close the parent region and all its 
> store file readers are also closed.  After that inorder to split the 
> reference files we need the first and last keys for which once again open the 
> readers on those store files.  This could be costlier operation considering 
> the fact that it has to contact the HDFS for this close and open operation. 
> This JIRA is to see if we can improve this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14652) Improve / update publish-website script in dev-support

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964786#comment-14964786
 ] 

Hadoop QA commented on HBASE-14652:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12767546/HBASE-14652.patch
  against master branch at commit c1f0442045f44fcbb3935f9244794929a5d0caea.
  ATTACHMENT ID: 12767546

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+0 tests included{color}.  The patch appears to be a 
documentation, build,
or dev-support patch that doesn't require tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 
2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 lineLengths{color}.  The patch introduces the following lines 
longer than 100:
+find . -type d  ! -path '*0.94*' ! -path '*apidocs*' ! -path '*xref*' ! 
-path '*book*' ! -path '*svn*' | \
+find . -type f  ! -path '*0.94*' ! -path '*apidocs*' ! -path '*xref*' ! -path 
'*book*' ! -path '*svn*' | \
+echo "Stale files or directories exist, but not taking care of stale 
directories and files in auto mode." |tee -a /tmp/out.txt

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
 

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16112//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16112//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16112//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16112//console

This message is automatically generated.

> Improve / update publish-website script in dev-support
> --
>
> Key: HBASE-14652
> URL: https://issues.apache.org/jira/browse/HBASE-14652
> Project: HBase
>  Issue Type: Task
>  Components: scripts
>Affects Versions: 2.0.0
>Reporter: Misty Stanley-Jones
>Assignee: Misty Stanley-Jones
> Fix For: 2.0.0
>
> Attachments: HBASE-14652.patch
>
>
> Since I wrote the script some parts of the build process have changed. Need 
> to update it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14652) Improve / update publish-website script in dev-support

[Misty Stanley-Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Misty Stanley-Jones updated HBASE-14652:

Hadoop Flags: Reviewed

Committed to master. Leaving open for a bit more review.

> Improve / update publish-website script in dev-support
> --
>
> Key: HBASE-14652
> URL: https://issues.apache.org/jira/browse/HBASE-14652
> Project: HBase
>  Issue Type: Task
>  Components: scripts
>Affects Versions: 2.0.0
>Reporter: Misty Stanley-Jones
>Assignee: Misty Stanley-Jones
> Fix For: 2.0.0
>
> Attachments: HBASE-14652.patch
>
>
> Since I wrote the script some parts of the build process have changed. Need 
> to update it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14652) Improve / update publish-website script in dev-support

[Misty Stanley-Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Misty Stanley-Jones updated HBASE-14652:

Status: Open  (was: Patch Available)

> Improve / update publish-website script in dev-support
> --
>
> Key: HBASE-14652
> URL: https://issues.apache.org/jira/browse/HBASE-14652
> Project: HBase
>  Issue Type: Task
>  Components: scripts
>Affects Versions: 2.0.0
>Reporter: Misty Stanley-Jones
>Assignee: Misty Stanley-Jones
> Fix For: 2.0.0
>
> Attachments: HBASE-14652.patch
>
>
> Since I wrote the script some parts of the build process have changed. Need 
> to update it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14648) Reenable TestWALProcedureStoreOnHDFS#testWalRollOnLowReplication

[Heng Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964676#comment-14964676
 ] 

Heng Chen commented on HBASE-14648:
---

I have a question about this testcase.

The testcase is to do test about Roll under low replication.  But why we set 
dfs.namenode.replication.min to be 3.

{{dfs.namenode.replication.min}} means when we write to hdfs if replica under 
this value, write will be failed. 

And our dn number is 3 too.  So when we restart, if the dn come back slowly, 
all insert action will be failed. 

But what we want to test is low replication( < 3) . It means under this 
situation, {{dfs.namenode.replication.min}} will never be satisfied. 

Do you think we should set {{dfs.namenode.replication.min}} to be 1? 


> Reenable TestWALProcedureStoreOnHDFS#testWalRollOnLowReplication
> 
>
> Key: HBASE-14648
> URL: https://issues.apache.org/jira/browse/HBASE-14648
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964695#comment-14964695
 ] 

ramkrishna.s.vasudevan commented on HBASE-13082:


bq.This status is in reader? It suits better in StoreFile no?
The reason for doing this (I just now checked the code once again) is because 
when we create a scanner on the Storefile we do it on the reader associated 
with the Storefile and not on the store file.  So hence do determine whether 
this store file can be used in the scanner or not the state and ref count if it 
is in the reader it will be easy to use that info. Already info like isBulkLoad 
and setSeqId everything is happening on the reader now and not on the 
StoreFile. So may be if we can introduce a getStoreFileScanner at the Storefile 
level rather than the Reader as how it is currently is, then we can make this 
change of adding the ref count and the state to the StoreFile. 

> Coarsen StoreScanner locks to RegionScanner
> ---
>
> Key: HBASE-13082
> URL: https://issues.apache.org/jira/browse/HBASE-13082
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: ramkrishna.s.vasudevan
> Attachments: 13082-test.txt, 13082-v2.txt, 13082-v3.txt, 
> 13082-v4.txt, 13082.txt, 13082.txt, HBASE-13082.pdf, HBASE-13082_1_WIP.patch, 
> HBASE-13082_2_WIP.patch, HBASE-13082_3.patch, HBASE-13082_4.patch, gc.png, 
> gc.png, gc.png, hits.png, next.png, next.png
>
>
> Continuing where HBASE-10015 left of.
> We can avoid locking (and memory fencing) inside StoreScanner by deferring to 
> the lock already held by the RegionScanner.
> In tests this shows quite a scan improvement and reduced CPU (the fences make 
> the cores wait for memory fetches).
> There are some drawbacks too:
> * All calls to RegionScanner need to be remain synchronized
> * Implementors of coprocessors need to be diligent in following the locking 
> contract. For example Phoenix does not lock RegionScanner.nextRaw() and 
> required in the documentation (not picking on Phoenix, this one is my fault 
> as I told them it's OK)
> * possible starving of flushes and compaction with heavy read load. 
> RegionScanner operations would keep getting the locks and the 
> flushes/compactions would not be able finalize the set of files.
> I'll have a patch soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14648) Reenable TestWALProcedureStoreOnHDFS#testWalRollOnLowReplication

[Heng Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Heng Chen updated HBASE-14648:
--
Attachment: HBASE-14648.patch

Set {{dfs.namenode.replication.min}} to be 1.

Inorder to do it, i move the whole MiniDFSCluster into each function. 

I have a feeling this time we can finish this flaky testcase. :)

> Reenable TestWALProcedureStoreOnHDFS#testWalRollOnLowReplication
> 
>
> Key: HBASE-14648
> URL: https://issues.apache.org/jira/browse/HBASE-14648
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-14648.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14366) NPE in case visibility expression is not present in labels table during importtsv run

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965159#comment-14965159
 ] 

Hudson commented on HBASE-14366:


SUCCESS: Integrated in HBase-1.2 #277 (See 
[https://builds.apache.org/job/HBase-1.2/277/])
HBASE-14366 NPE in case visibility expression is not present in labels 
(anoopsamjohn: rev 258cde6a917e1399fe9770bdfba645a0c1470e66)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTSVWithVisibilityLabels.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityConstants.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TsvImporterMapper.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityLabelsCache.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TextSortReducer.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/DefaultVisibilityExpressionResolver.java


> NPE in case visibility expression is not present in labels table during 
> importtsv run
> -
>
> Key: HBASE-14366
> URL: https://issues.apache.org/jira/browse/HBASE-14366
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: 0001-HBASE-14366.patch, 0001-HBASE-14366_1.patch, 
> HBASE-14366-0.98.patch, HBASE-14366-branch-1.1.patch, 
> HBASE-14366-branch-1.1_v1.patch, HBASE-14366-branch-1.2.patch, 
> HBASE-14366-branch-1.2_v1.patch, HBASE-14366-branch-1.patch, 
> HBASE-14366_2(1).patch, HBASE-14366_2.patch
>
>
> Below exception is shown in logs if visibility expression is not present in 
> labels table during importtsv run. Appropriate exception / message should be 
> logged for the user to take further action.
> {code}
> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver$1.getLabelOrdinal(DefaultVisibilityExpressionResolver.java:127)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.getLabelOrdinals(VisibilityUtils.java:358)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.createVisibilityExpTags(VisibilityUtils.java:323)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.createVisibilityExpTags(DefaultVisibilityExpressionResolver.java:137)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.populatePut(TsvImporterMapper.java:205)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:165)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14634) Disable flakey TestSnapshotCloneIndependence.testOnlineSnapshotDeleteIndependent

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965585#comment-14965585
 ] 

stack commented on HBASE-14634:
---

testOnlineSnapshotAppendIndependent is what I disabled, not the item in the 
topic. Let me fix since it continues to fail...

> Disable flakey 
> TestSnapshotCloneIndependence.testOnlineSnapshotDeleteIndependent
> 
>
> Key: HBASE-14634
> URL: https://issues.apache.org/jira/browse/HBASE-14634
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
>
> See HBASE-14585 for some history on this test. It just failed on patch build 
> twice in a row: 
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-TRUNK/6921/consoleText
>  and 
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-TRUNK/6922/consoleText
> Disabling for now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14633) Try fluid width UI

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965294#comment-14965294
 ] 

Sean Busbey commented on HBASE-14633:
-

looks close enough to be fine on 1.2.

would really like to see a test-patch run where applying works.

from scanning the patch:

* bootstrap upgrade requires updating the copyright year(s) in our 
LICENSE/NOTICE files
* bootstrap upgrade changed the license, will need suitable updates to our 
LICENSE / NOTICE files.
* jquery upgrade requires updating the copyright year(s) in our LICENSE/NOTICE 
files

> Try fluid width UI
> --
>
> Key: HBASE-14633
> URL: https://issues.apache.org/jira/browse/HBASE-14633
> Project: HBase
>  Issue Type: Bug
>  Components: UI
>Affects Versions: 2.0.0, 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-14633-v1.patch, HBASE-14633-v2.patch, 
> HBASE-14633-v3.patch, HBASE-14633.patch, Screen Shot 2015-10-20 at 2.40.23 
> AM.png, Screen Shot 2015-10-20 at 2.40.34 AM.png, Screen Shot 2015-10-20 at 
> 2.44.11 AM.png
>
>
> Our UI is often too long. Lets give it more room if available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14654) Reenable TestMultiParallel#testActiveThreadsCount

[Heng Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965310#comment-14965310
 ] 

Heng Chen commented on HBASE-14654:
---

findHangingTests.py shows 
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat timeout.

> Reenable TestMultiParallel#testActiveThreadsCount
> -
>
> Key: HBASE-14654
> URL: https://issues.apache.org/jira/browse/HBASE-14654
> Project: HBase
>  Issue Type: Bug
>Reporter: Heng Chen
> Attachments: HBASE-14654.patch, HBASE-14654.patch
>
>
> It was disabled in HBASE-14642,  this issue should reenable it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14366) NPE in case visibility expression is not present in labels table during importtsv run

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965434#comment-14965434
 ] 

Hudson commented on HBASE-14366:


FAILURE: Integrated in HBase-0.98 #1163 (See 
[https://builds.apache.org/job/HBase-0.98/1163/])
HBASE-14366 NPE in case visibility expression is not present in labels 
(anoopsamjohn: rev 834035bac118d45dd93ce6ae96372a0fe0f83a2e)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TsvImporterMapper.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityConstants.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/DefaultVisibilityExpressionResolver.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityLabelsCache.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TextSortReducer.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTSVWithVisibilityLabels.java


> NPE in case visibility expression is not present in labels table during 
> importtsv run
> -
>
> Key: HBASE-14366
> URL: https://issues.apache.org/jira/browse/HBASE-14366
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: 0001-HBASE-14366.patch, 0001-HBASE-14366_1.patch, 
> HBASE-14366-0.98.patch, HBASE-14366-branch-1.1.patch, 
> HBASE-14366-branch-1.1_v1.patch, HBASE-14366-branch-1.2.patch, 
> HBASE-14366-branch-1.2_v1.patch, HBASE-14366-branch-1.patch, 
> HBASE-14366_2(1).patch, HBASE-14366_2.patch
>
>
> Below exception is shown in logs if visibility expression is not present in 
> labels table during importtsv run. Appropriate exception / message should be 
> logged for the user to take further action.
> {code}
> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver$1.getLabelOrdinal(DefaultVisibilityExpressionResolver.java:127)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.getLabelOrdinals(VisibilityUtils.java:358)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.createVisibilityExpTags(VisibilityUtils.java:323)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.createVisibilityExpTags(DefaultVisibilityExpressionResolver.java:137)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.populatePut(TsvImporterMapper.java:205)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:165)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14654) Reenable TestMultiParallel#testActiveThreadsCount

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965575#comment-14965575
 ] 

Hadoop QA commented on HBASE-14654:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12767609/HBASE-14654.patch
  against master branch at commit 60465964039acd05f43f268cdb4f909a150a0f41.
  ATTACHMENT ID: 12767609

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 4 new 
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 
2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:red}-1 checkstyle{color}.  The applied patch generated 
1748 checkstyle errors (more than the master's current 1747 errors).

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16121//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16121//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16121//artifact/patchprocess/checkstyle-aggregate.html

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16121//console

This message is automatically generated.

> Reenable TestMultiParallel#testActiveThreadsCount
> -
>
> Key: HBASE-14654
> URL: https://issues.apache.org/jira/browse/HBASE-14654
> Project: HBase
>  Issue Type: Bug
>Reporter: Heng Chen
> Attachments: HBASE-14654.patch, HBASE-14654.patch
>
>
> It was disabled in HBASE-14642,  this issue should reenable it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14463) Severe performance downgrade when parallel reading a single key from BucketCache

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965431#comment-14965431
 ] 

Anoop Sam John commented on HBASE-14463:


I am doing a cluster testing with this patch

> Severe performance downgrade when parallel reading a single key from 
> BucketCache
> 
>
> Key: HBASE-14463
> URL: https://issues.apache.org/jira/browse/HBASE-14463
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.14, 1.1.2
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.16
>
> Attachments: GC_with_WeakObjectPool.png, HBASE-14463.patch, 
> HBASE-14463_v11.patch, HBASE-14463_v12.patch, HBASE-14463_v2.patch, 
> HBASE-14463_v3.patch, HBASE-14463_v4.patch, HBASE-14463_v5.patch, 
> TestBucketCache-new_with_IdLock.png, 
> TestBucketCache-new_with_IdReadWriteLock.png, 
> TestBucketCache_with_IdLock-latest.png, TestBucketCache_with_IdLock.png, 
> TestBucketCache_with_IdReadWriteLock-latest.png, 
> TestBucketCache_with_IdReadWriteLock-resolveLockLeak.png, 
> TestBucketCache_with_IdReadWriteLock.png
>
>
> We store feature data of online items in HBase, do machine learning on these 
> features, and supply the outputs to our online search engine. In such 
> scenario we will launch hundreds of yarn workers and each worker will read 
> all features of one item(i.e. single rowkey in HBase), so there'll be heavy 
> parallel reading on a single rowkey.
> We were using LruCache but start to try BucketCache recently to resolve gc 
> issue, and just as titled we have observed severe performance downgrade. 
> After some analytics we found the root cause is the lock in 
> BucketCache#getBlock, as shown below
> {code}
>   try {
> lockEntry = offsetLock.getLockEntry(bucketEntry.offset());
> // ...
> if (bucketEntry.equals(backingMap.get(key))) {
>   // ...
>   int len = bucketEntry.getLength();
>   Cacheable cachedBlock = ioEngine.read(bucketEntry.offset(), len,
>   bucketEntry.deserializerReference(this.deserialiserMap));
> {code}
> Since ioEnging.read involves array copy, it's much more time-costed than the 
> operation in LruCache. And since we're using synchronized in 
> IdLock#getLockEntry, parallel read dropping on the same bucket would be 
> executed in serial, which causes a really bad performance.
> To resolve the problem, we propose to use ReentranceReadWriteLock in 
> BucketCache, and introduce a new class called IdReadWriteLock to implement it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14420) Zombie Stomping Session

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965502#comment-14965502
 ] 

Hadoop QA commented on HBASE-14420:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12767603/none_fix.txt
  against master branch at commit 60465964039acd05f43f268cdb4f909a150a0f41.
  ATTACHMENT ID: 12767603

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+0 tests included{color}.  The patch appears to be a 
documentation, build,
or dev-support patch that doesn't require tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 
2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
 

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16120//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16120//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16120//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16120//console

This message is automatically generated.

> Zombie Stomping Session
> ---
>
> Key: HBASE-14420
> URL: https://issues.apache.org/jira/browse/HBASE-14420
> Project: HBase
>  Issue Type: Umbrella
>  Components: test
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: hangers.txt, none_fix (1).txt, none_fix.txt, 
> none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, 
> none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, 
> none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, 
> none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, 
> none_fix.txt, none_fix.txt, none_fix.txt
>
>
> Patch build are now failing most of the time because we are dropping zombies. 
> I confirm we are doing this on non-apache build boxes too.
> Left-over zombies consume resources on build boxes (OOME cannot create native 
> threads). Having to do multiple test runs in the hope that we can get a 
> non-zombie-making build or making (arbitrary) rulings that the zombies are 
> 'not related' is a productivity sink. And so on...
> This is an umbrella issue for a zombie stomping session that started earlier 
> this week. Will hang sub-issues of this one. Am running builds back-to-back 
> on little cluster to turn out the monsters.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14633) Try fluid width UI

[Elliott Clark (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965558#comment-14965558
 ] 

Elliott Clark commented on HBASE-14633:
---

bq.would really like to see a test-patch run where applying works.
For that I think that we would need to change the test patch stuff to try git 
am rather than just patch -p0/1

Let me try and get the patch without upgrading things and we can do that in a 
different jira.

> Try fluid width UI
> --
>
> Key: HBASE-14633
> URL: https://issues.apache.org/jira/browse/HBASE-14633
> Project: HBase
>  Issue Type: Bug
>  Components: UI
>Affects Versions: 2.0.0, 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-14633-v1.patch, HBASE-14633-v2.patch, 
> HBASE-14633-v3.patch, HBASE-14633.patch, Screen Shot 2015-10-20 at 2.40.23 
> AM.png, Screen Shot 2015-10-20 at 2.40.34 AM.png, Screen Shot 2015-10-20 at 
> 2.44.11 AM.png
>
>
> Our UI is often too long. Lets give it more room if available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14633) Try fluid width UI

[Elliott Clark (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965592#comment-14965592
 ] 

Elliott Clark commented on HBASE-14633:
---

Yeah works on jdk8 that's what's running in the screenshots. However if we move 
the upgrading stuff to a different jira I bet I can get a clean pass.

> Try fluid width UI
> --
>
> Key: HBASE-14633
> URL: https://issues.apache.org/jira/browse/HBASE-14633
> Project: HBase
>  Issue Type: Bug
>  Components: UI
>Affects Versions: 2.0.0, 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-14633-v1.patch, HBASE-14633-v2.patch, 
> HBASE-14633-v3.patch, HBASE-14633.patch, Screen Shot 2015-10-20 at 2.40.23 
> AM.png, Screen Shot 2015-10-20 at 2.40.34 AM.png, Screen Shot 2015-10-20 at 
> 2.44.11 AM.png
>
>
> Our UI is often too long. Lets give it more room if available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14634) Disable flakey TestSnapshotCloneIndependence.testOnlineSnapshotDeleteIndependent

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-14634:
--
Attachment: 14634.addendum.txt

Addendum pushed to branch-1.2+

> Disable flakey 
> TestSnapshotCloneIndependence.testOnlineSnapshotDeleteIndependent
> 
>
> Key: HBASE-14634
> URL: https://issues.apache.org/jira/browse/HBASE-14634
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14634.addendum.txt
>
>
> See HBASE-14585 for some history on this test. It just failed on patch build 
> twice in a row: 
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-TRUNK/6921/consoleText
>  and 
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-TRUNK/6922/consoleText
> Disabling for now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14633) Try fluid width UI

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965581#comment-14965581
 ] 

Sean Busbey commented on HBASE-14633:
-

bq. For that I think that we would need to change the test patch stuff to try 
git am rather than just patch -p0/1

So I need to get a Yetus release. :) let's not block on that then. Just someone 
tell me that {{mvn -DskipTests verify}} works on jdk8.

> Try fluid width UI
> --
>
> Key: HBASE-14633
> URL: https://issues.apache.org/jira/browse/HBASE-14633
> Project: HBase
>  Issue Type: Bug
>  Components: UI
>Affects Versions: 2.0.0, 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-14633-v1.patch, HBASE-14633-v2.patch, 
> HBASE-14633-v3.patch, HBASE-14633.patch, Screen Shot 2015-10-20 at 2.40.23 
> AM.png, Screen Shot 2015-10-20 at 2.40.34 AM.png, Screen Shot 2015-10-20 at 
> 2.44.11 AM.png
>
>
> Our UI is often too long. Lets give it more room if available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14633) Try fluid width UI

[Elliott Clark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-14633:
--
Attachment: HBASE-14633-v5.patch

margin is a little different now.

> Try fluid width UI
> --
>
> Key: HBASE-14633
> URL: https://issues.apache.org/jira/browse/HBASE-14633
> Project: HBase
>  Issue Type: Bug
>  Components: UI
>Affects Versions: 2.0.0, 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-14633-v1.patch, HBASE-14633-v2.patch, 
> HBASE-14633-v3.patch, HBASE-14633-v4.patch, HBASE-14633-v5.patch, 
> HBASE-14633.patch, Screen Shot 2015-10-20 at 2.40.23 AM.png, Screen Shot 
> 2015-10-20 at 2.40.34 AM.png, Screen Shot 2015-10-20 at 2.44.11 AM.png
>
>
> Our UI is often too long. Lets give it more room if available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14355) Scan different TimeRange for each column family

[churro morales (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965508#comment-14965508
 ] 

churro morales commented on HBASE-14355:


We have to add to the API to have this feature.  Is there anyone else that 
wants to chime in whether or not this is acceptable?

> Scan different TimeRange for each column family
> ---
>
> Key: HBASE-14355
> URL: https://issues.apache.org/jira/browse/HBASE-14355
> Project: HBase
>  Issue Type: New Feature
>  Components: Client, regionserver, Scanners
>Reporter: Dave Latham
>Assignee: churro morales
> Fix For: 2.0.0, 1.3.0, 0.98.16
>
> Attachments: HBASE-14355-v1.patch, HBASE-14355.patch
>
>
> At present the Scan API supports only table level time range. We have 
> specific use cases that will benefit from per column family time range. (See 
> background discussion at 
> https://mail-archives.apache.org/mod_mbox/hbase-user/201508.mbox/%3ccaa4mzom00ef5eoxstk0hetxeby8mqss61gbvgttgpaspmhq...@mail.gmail.com%3E)
> There are a couple of choices that would be good to validate.  First - how to 
> update the Scan API to support family and table level updates.  One proposal 
> would be to add Scan.setTimeRange(byte family, long minTime, long maxTime), 
> then store it in a Map.  When executing the scan, if a 
> family has a specified TimeRange, then use it, otherwise fall back to using 
> the table level TimeRange.  Clients using the new API against old region 
> servers would not get the families correctly filterd.  Old clients sending 
> scans to new region servers would work correctly.
> The other question is how to get StoreFileScanner.shouldUseScanner to match 
> up the proper family and time range.  It has the Scan available but doesn't 
> currently have available which family it is a part of.  One option would be 
> to try to pass down the column family in each constructor path.  Another 
> would be to instead alter shouldUseScanner to pass down the specific 
> TimeRange to use (similar to how it currently passes down the columns to use 
> which also appears to be a workaround for not having the family available). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14633) Try fluid width UI

[Elliott Clark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-14633:
--
Attachment: HBASE-14633-v4.patch

Just the css/layout changes without the upgrade of bootstrap/jquery

> Try fluid width UI
> --
>
> Key: HBASE-14633
> URL: https://issues.apache.org/jira/browse/HBASE-14633
> Project: HBase
>  Issue Type: Bug
>  Components: UI
>Affects Versions: 2.0.0, 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-14633-v1.patch, HBASE-14633-v2.patch, 
> HBASE-14633-v3.patch, HBASE-14633-v4.patch, HBASE-14633.patch, Screen Shot 
> 2015-10-20 at 2.40.23 AM.png, Screen Shot 2015-10-20 at 2.40.34 AM.png, 
> Screen Shot 2015-10-20 at 2.44.11 AM.png
>
>
> Our UI is often too long. Lets give it more room if available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14633) Try fluid width UI

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965269#comment-14965269
 ] 

stack commented on HBASE-14633:
---

+1 Pretty. Try it.  Put it on master and branch-1?  [~busbey] new UI layout for 
1.2?

> Try fluid width UI
> --
>
> Key: HBASE-14633
> URL: https://issues.apache.org/jira/browse/HBASE-14633
> Project: HBase
>  Issue Type: Bug
>  Components: UI
>Affects Versions: 2.0.0, 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-14633-v1.patch, HBASE-14633-v2.patch, 
> HBASE-14633-v3.patch, HBASE-14633.patch, Screen Shot 2015-10-20 at 2.40.23 
> AM.png, Screen Shot 2015-10-20 at 2.40.34 AM.png, Screen Shot 2015-10-20 at 
> 2.44.11 AM.png
>
>
> Our UI is often too long. Lets give it more room if available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14643) Avoid Splits from once again opening a closed reader for fetching the first and last key

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965339#comment-14965339
 ] 

ramkrishna.s.vasudevan commented on HBASE-14643:


Will commit this to master, branch-1 and branch-2. Any objections?  Will commit 
this tomorrow. 

> Avoid Splits from once again opening a closed reader for fetching the first 
> and last key
> 
>
> Key: HBASE-14643
> URL: https://issues.apache.org/jira/browse/HBASE-14643
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: Heng Chen
> Attachments: HBASE-14643.patch, HBASE-14643_v1.patch, 
> HBASE-14643_v2.patch, HBASE-14643_v3.patch, HBASE-14643_v4.patch, 
> HBASE-14643_v5.patch
>
>
> Currently split flow is such that we close the parent region and all its 
> store file readers are also closed.  After that inorder to split the 
> reference files we need the first and last keys for which once again open the 
> readers on those store files.  This could be costlier operation considering 
> the fact that it has to contact the HDFS for this close and open operation. 
> This JIRA is to see if we can improve this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14628) [0.98] Save object creation for scanning with block encodings

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965338#comment-14965338
 ] 

Andrew Purtell commented on HBASE-14628:


bq. The DataBlockEncoder interface is marked private, so I should be able to 
simply remove the method from the DataBlockEncoder.EncodedSeeker interface.

+1, that's why we have the annotations :-) 

> [0.98] Save object creation for scanning with block encodings
> -
>
> Key: HBASE-14628
> URL: https://issues.apache.org/jira/browse/HBASE-14628
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.98.16
>
> Attachments: 14628-0.98-v2.txt, 14628-0.98.txt
>
>
> I noticed that (at least in 0.98 - master is entirely different) we create 
> ByteBuffer just to create a byte[], which is then used to create a KeyValue.
> We can save the creation of the ByteBuffer and hence save allocating an extra 
> object for each KV we find by creating the byte[] directly.
> In a Phoenix count\(*) query that saved from 10% of runtime.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14648) Reenable TestWALProcedureStoreOnHDFS#testWalRollOnLowReplication

[Heng Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965326#comment-14965326
 ] 

Heng Chen commented on HBASE-14648:
---

In HBASE-14634,  we should ignore testOnlineSnapshotDeleteIndependent,  but it 
seems testOfflineSnapshotDeleteIndependent was ignored  Is it right? 
[~stack]

> Reenable TestWALProcedureStoreOnHDFS#testWalRollOnLowReplication
> 
>
> Key: HBASE-14648
> URL: https://issues.apache.org/jira/browse/HBASE-14648
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-14648.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14655) Narrow the scope of doAs() calls to region observer notifications for compaction

[Ted Yu (JIRA)]

Ted Yu created HBASE-14655:
--

 Summary: Narrow the scope of doAs() calls to region observer 
notifications for compaction
 Key: HBASE-14655
 URL: https://issues.apache.org/jira/browse/HBASE-14655
 Project: HBase
  Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
 Attachments: 14655-v1.txt

As what has been done in HBASE-14631 and HBASE-14605, the scope of calling 
doAs() for compaction related region observer notifications should be narrowed.
User object is passed from CompactSplitThread down to the methods where region 
observer notifications are made.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14654) Reenable TestMultiParallel#testActiveThreadsCount

[Heng Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965353#comment-14965353
 ] 

Heng Chen commented on HBASE-14654:
---

I notice that when this QA running,  there is other QA running simultaneously 
on same machine H0.

And the two QA all show up that TestHFileOutputFormat was hanging. 

I doubt If there are some common resources the two QA conflicts when running  
TestHFileOutputFormat simultaneously. I will dig it tomorrow.

> Reenable TestMultiParallel#testActiveThreadsCount
> -
>
> Key: HBASE-14654
> URL: https://issues.apache.org/jira/browse/HBASE-14654
> Project: HBase
>  Issue Type: Bug
>Reporter: Heng Chen
> Attachments: HBASE-14654.patch, HBASE-14654.patch
>
>
> It was disabled in HBASE-14642,  this issue should reenable it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14604) Improve MoveCostFunction in StochasticLoadBalancer

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965250#comment-14965250
 ] 

Hudson commented on HBASE-14604:


FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1115 (See 
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1115/])
HBASE-14604 Improve MoveCostFunction in StochasticLoadBalancer (Guanghao 
(tedyu: rev 76a14b9ba682c76a81baa06168239a2de6d2c7de)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java


> Improve MoveCostFunction in StochasticLoadBalancer
> --
>
> Key: HBASE-14604
> URL: https://issues.apache.org/jira/browse/HBASE-14604
> Project: HBase
>  Issue Type: Bug
>  Components: Balancer
>Affects Versions: 0.98.15
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0, 1.3.0, 0.98.16
>
> Attachments: HBASE-14604-0.98.patch, HBASE-14604-0.98_v1.patch, 
> HBASE-14604-trunk.patch, HBASE-14604-trunk_v1.patch, HBASE-14604_98.diff, 
> HBASE-14604_98_with_ut.diff
>
>
> The code in MoveCoseFunction:
> {code}
> return scale(0, cluster.numRegions + META_MOVE_COST_MULT, moveCost);
> {code}
> It uses cluster.numRegions + META_MOVE_COST_MULT as the max value when scale 
> moveCost to [0,1]. But this should use maxMoves as the max value when cluster 
> have a lot of regions.
> Assume a cluster have 1 regions, maxMoves is 2500, it only scale moveCost 
> to [0, 0.25].
> Improve moveCost by use Math.min(cluster.numRegions, maxMoves) as the max 
> cost, so it can scale moveCost to [0,1].
> {code}
> return scale(0, Math.min(cluster.numRegions, maxMoves) + META_MOVE_COST_MULT, 
> moveCost);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14654) Reenable TestMultiParallel#testActiveThreadsCount

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965273#comment-14965273
 ] 

stack commented on HBASE-14654:
---

Says "[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.18.1:test 
(secondPartTestsExecution) on project hbase-server: There was a timeout or 
other error in the fork -> [Help 1]"

> Reenable TestMultiParallel#testActiveThreadsCount
> -
>
> Key: HBASE-14654
> URL: https://issues.apache.org/jira/browse/HBASE-14654
> Project: HBase
>  Issue Type: Bug
>Reporter: Heng Chen
> Attachments: HBASE-14654.patch, HBASE-14654.patch
>
>
> It was disabled in HBASE-14642,  this issue should reenable it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14654) Reenable TestMultiParallel#testActiveThreadsCount

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-14654:
--
Attachment: HBASE-14654.patch

> Reenable TestMultiParallel#testActiveThreadsCount
> -
>
> Key: HBASE-14654
> URL: https://issues.apache.org/jira/browse/HBASE-14654
> Project: HBase
>  Issue Type: Bug
>Reporter: Heng Chen
> Attachments: HBASE-14654.patch, HBASE-14654.patch
>
>
> It was disabled in HBASE-14642,  this issue should reenable it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14655) Narrow the scope of doAs() calls to region observer notifications for compaction

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14655:
---
Status: Patch Available  (was: Open)

> Narrow the scope of doAs() calls to region observer notifications for 
> compaction
> 
>
> Key: HBASE-14655
> URL: https://issues.apache.org/jira/browse/HBASE-14655
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 14655-v1.txt
>
>
> As what has been done in HBASE-14631 and HBASE-14605, the scope of calling 
> doAs() for compaction related region observer notifications should be 
> narrowed.
> User object is passed from CompactSplitThread down to the methods where 
> region observer notifications are made.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14643) Avoid Splits from once again opening a closed reader for fetching the first and last key

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965347#comment-14965347
 ] 

Anoop Sam John commented on HBASE-14643:


u mean branch-1 and branch-1.2?

> Avoid Splits from once again opening a closed reader for fetching the first 
> and last key
> 
>
> Key: HBASE-14643
> URL: https://issues.apache.org/jira/browse/HBASE-14643
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: Heng Chen
> Attachments: HBASE-14643.patch, HBASE-14643_v1.patch, 
> HBASE-14643_v2.patch, HBASE-14643_v3.patch, HBASE-14643_v4.patch, 
> HBASE-14643_v5.patch
>
>
> Currently split flow is such that we close the parent region and all its 
> store file readers are also closed.  After that inorder to split the 
> reference files we need the first and last keys for which once again open the 
> readers on those store files.  This could be costlier operation considering 
> the fact that it has to contact the HDFS for this close and open operation. 
> This JIRA is to see if we can improve this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14420) Zombie Stomping Session

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-14420:
--
Attachment: none_fix.txt

> Zombie Stomping Session
> ---
>
> Key: HBASE-14420
> URL: https://issues.apache.org/jira/browse/HBASE-14420
> Project: HBase
>  Issue Type: Umbrella
>  Components: test
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: hangers.txt, none_fix (1).txt, none_fix.txt, 
> none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, 
> none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, 
> none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, 
> none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, 
> none_fix.txt, none_fix.txt, none_fix.txt
>
>
> Patch build are now failing most of the time because we are dropping zombies. 
> I confirm we are doing this on non-apache build boxes too.
> Left-over zombies consume resources on build boxes (OOME cannot create native 
> threads). Having to do multiple test runs in the hope that we can get a 
> non-zombie-making build or making (arbitrary) rulings that the zombies are 
> 'not related' is a productivity sink. And so on...
> This is an umbrella issue for a zombie stomping session that started earlier 
> this week. Will hang sub-issues of this one. Am running builds back-to-back 
> on little cluster to turn out the monsters.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14648) Reenable TestWALProcedureStoreOnHDFS#testWalRollOnLowReplication

[Heng Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965317#comment-14965317
 ] 

Heng Chen commented on HBASE-14648:
---

Printing hanging tests
Hanging test : org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
Printing Failing tests
Failing test : org.apache.hadoop.hbase.client.TestSnapshotCloneIndependence

> Reenable TestWALProcedureStoreOnHDFS#testWalRollOnLowReplication
> 
>
> Key: HBASE-14648
> URL: https://issues.apache.org/jira/browse/HBASE-14648
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-14648.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14655) Narrow the scope of doAs() calls to region observer notifications for compaction

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14655:
---
Attachment: 14655-v1.txt

> Narrow the scope of doAs() calls to region observer notifications for 
> compaction
> 
>
> Key: HBASE-14655
> URL: https://issues.apache.org/jira/browse/HBASE-14655
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 14655-v1.txt
>
>
> As what has been done in HBASE-14631 and HBASE-14605, the scope of calling 
> doAs() for compaction related region observer notifications should be 
> narrowed.
> User object is passed from CompactSplitThread down to the methods where 
> region observer notifications are made.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14643) Avoid Splits from once again opening a closed reader for fetching the first and last key

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-14643:
---
Fix Version/s: 1.3.0
   1.2.0
   2.0.0

> Avoid Splits from once again opening a closed reader for fetching the first 
> and last key
> 
>
> Key: HBASE-14643
> URL: https://issues.apache.org/jira/browse/HBASE-14643
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: Heng Chen
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14643.patch, HBASE-14643_v1.patch, 
> HBASE-14643_v2.patch, HBASE-14643_v3.patch, HBASE-14643_v4.patch, 
> HBASE-14643_v5.patch
>
>
> Currently split flow is such that we close the parent region and all its 
> store file readers are also closed.  After that inorder to split the 
> reference files we need the first and last keys for which once again open the 
> readers on those store files.  This could be costlier operation considering 
> the fact that it has to contact the HDFS for this close and open operation. 
> This JIRA is to see if we can improve this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14654) Reenable TestMultiParallel#testActiveThreadsCount

[Heng Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965357#comment-14965357
 ] 

Heng Chen commented on HBASE-14654:
---

the two QA information:

https://builds.apache.org/job/PreCommit-HBASE-Build/16118/console
https://builds.apache.org/job/PreCommit-HBASE-Build/16117/console


> Reenable TestMultiParallel#testActiveThreadsCount
> -
>
> Key: HBASE-14654
> URL: https://issues.apache.org/jira/browse/HBASE-14654
> Project: HBase
>  Issue Type: Bug
>Reporter: Heng Chen
> Attachments: HBASE-14654.patch, HBASE-14654.patch
>
>
> It was disabled in HBASE-14642,  this issue should reenable it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-14656) Move TestAssignmentManager from medium to large category

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-14656.
---
   Resolution: Fixed
 Assignee: stack
Fix Version/s: 1.3.0
   1.2.0
   2.0.0

Pushed this to branch-1 and branch-1.2. Test is not in master.

> Move TestAssignmentManager from medium to large category
> 
>
> Key: HBASE-14656
> URL: https://issues.apache.org/jira/browse/HBASE-14656
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14656.patch
>
>
> On internal rig, this test timedout. Its medium category and then I go this:
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal org.apache.maven.plugins:maven-surefire-plugin:2.18.1:test 
> (secondPartTestsExecution) on project hbase-server: There are test failures.
> Hopefully there is correlation. Let me move this test to large from medium so 
> it runs in its own jvm... and we see it timeout without breaking next stage 
> run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14463) Severe performance downgrade when parallel reading a single key from BucketCache

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965679#comment-14965679
 ] 

Anoop Sam John commented on HBASE-14463:


On a single node cluster with 100 GB data.  The whole data is loaded in to off 
heap bucket cache.
Doing multi get test with performance evaluation tool. Having 25 client threads.
The patch slows down the operation a bit.  5% or so.
The read pattern may be such that different threads requesting for same block 
is rare.   Still there should not be a slow down.  

> Severe performance downgrade when parallel reading a single key from 
> BucketCache
> 
>
> Key: HBASE-14463
> URL: https://issues.apache.org/jira/browse/HBASE-14463
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.14, 1.1.2
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.16
>
> Attachments: GC_with_WeakObjectPool.png, HBASE-14463.patch, 
> HBASE-14463_v11.patch, HBASE-14463_v12.patch, HBASE-14463_v2.patch, 
> HBASE-14463_v3.patch, HBASE-14463_v4.patch, HBASE-14463_v5.patch, 
> TestBucketCache-new_with_IdLock.png, 
> TestBucketCache-new_with_IdReadWriteLock.png, 
> TestBucketCache_with_IdLock-latest.png, TestBucketCache_with_IdLock.png, 
> TestBucketCache_with_IdReadWriteLock-latest.png, 
> TestBucketCache_with_IdReadWriteLock-resolveLockLeak.png, 
> TestBucketCache_with_IdReadWriteLock.png
>
>
> We store feature data of online items in HBase, do machine learning on these 
> features, and supply the outputs to our online search engine. In such 
> scenario we will launch hundreds of yarn workers and each worker will read 
> all features of one item(i.e. single rowkey in HBase), so there'll be heavy 
> parallel reading on a single rowkey.
> We were using LruCache but start to try BucketCache recently to resolve gc 
> issue, and just as titled we have observed severe performance downgrade. 
> After some analytics we found the root cause is the lock in 
> BucketCache#getBlock, as shown below
> {code}
>   try {
> lockEntry = offsetLock.getLockEntry(bucketEntry.offset());
> // ...
> if (bucketEntry.equals(backingMap.get(key))) {
>   // ...
>   int len = bucketEntry.getLength();
>   Cacheable cachedBlock = ioEngine.read(bucketEntry.offset(), len,
>   bucketEntry.deserializerReference(this.deserialiserMap));
> {code}
> Since ioEnging.read involves array copy, it's much more time-costed than the 
> operation in LruCache. And since we're using synchronized in 
> IdLock#getLockEntry, parallel read dropping on the same bucket would be 
> executed in serial, which causes a really bad performance.
> To resolve the problem, we propose to use ReentranceReadWriteLock in 
> BucketCache, and introduce a new class called IdReadWriteLock to implement it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14628) [0.98] Save object creation for scanning with block encodings

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965745#comment-14965745
 ] 

Lars Hofhansl commented on HBASE-14628:
---

And agreed on removing getValueShallowCopy. Return a ByteBuffer from any method 
leaks internal implementation that we should avoid (even in in internal 
interface).

> [0.98] Save object creation for scanning with block encodings
> -
>
> Key: HBASE-14628
> URL: https://issues.apache.org/jira/browse/HBASE-14628
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.98.16
>
> Attachments: 14628-0.98-v2.txt, 14628-0.98.txt
>
>
> I noticed that (at least in 0.98 - master is entirely different) we create 
> ByteBuffer just to create a byte[], which is then used to create a KeyValue.
> We can save the creation of the ByteBuffer and hence save allocating an extra 
> object for each KV we find by creating the byte[] directly.
> In a Phoenix count\(*) query that saved from 10% of runtime.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14628) [0.98] Save object creation for scanning with block encodings

[Lars Hofhansl (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-14628:
--
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Committed to 0.98. Thanks for looking.

> [0.98] Save object creation for scanning with block encodings
> -
>
> Key: HBASE-14628
> URL: https://issues.apache.org/jira/browse/HBASE-14628
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.98.16
>
> Attachments: 14628-0.98-v2.txt, 14628-0.98.txt
>
>
> I noticed that (at least in 0.98 - master is entirely different) we create 
> ByteBuffer just to create a byte[], which is then used to create a KeyValue.
> We can save the creation of the ByteBuffer and hence save allocating an extra 
> object for each KV we find by creating the byte[] directly.
> In a Phoenix count\(*) query that saved from 10% of runtime.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14283) Reverse scan doesn’t work with HFile inline index/bloom blocks

[Ben Lau (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965794#comment-14965794
 ] 

Ben Lau commented on HBASE-14283:
-

[~andrew.purt...@gmail.com] Is the above explanation agreeable?

> Reverse scan doesn’t work with HFile inline index/bloom blocks
> --
>
> Key: HBASE-14283
> URL: https://issues.apache.org/jira/browse/HBASE-14283
> Project: HBase
>  Issue Type: Bug
>Reporter: Ben Lau
>Assignee: Ben Lau
> Attachments: HBASE-14283-0.98.patch, HBASE-14283-branch-1.0.patch, 
> HBASE-14283-branch-1.1.patch, HBASE-14283-branch-1.2.patch, 
> HBASE-14283-branch-1.patch, HBASE-14283-master.patch, 
> HBASE-14283-reupload-master.patch, HBASE-14283-v2.patch, HBASE-14283.patch, 
> hfile-seek-before.patch
>
>
> Reverse scans do not work if an HFile contains inline bloom blocks or leaf 
> level index blocks.  The reason is because the seekBefore() call calculates 
> the previous data block’s size by assuming data blocks are contiguous which 
> is not the case in HFile V2 and beyond.
> Attached is a first cut patch (targeting 
> bcef28eefaf192b0ad48c8011f98b8e944340da5 on trunk) which includes:
> (1) a unit test which exposes the bug and demonstrates failures for both 
> inline bloom blocks and inline index blocks
> (2) a proposed fix for inline index blocks that does not require a new HFile 
> version change, but is only performant for 1 and 2-level indexes and not 3+.  
> 3+ requires an HFile format update for optimal performance.
> This patch does not fix the bloom filter blocks bug.  But the fix should be 
> similar to the case of inline index blocks.  The reason I haven’t made the 
> change yet is I want to confirm that you guys would be fine with me revising 
> the HFile.Reader interface.
> Specifically, these 2 functions (getGeneralBloomFilterMetadata and 
> getDeleteBloomFilterMetadata) need to return the BloomFilter.  Right now the 
> HFileReader class doesn’t have a reference to the bloom filters (and hence 
> their indices) and only constructs the IO streams and hence has no way to 
> know where the bloom blocks are in the HFile.  It seems that the HFile.Reader 
> bloom method comments state that they “know nothing about how that metadata 
> is structured” but I do not know if that is a requirement of the abstraction 
> (why?) or just an incidental current property. 
> We would like to do 3 things with community approval:
> (1) Update the HFile.Reader interface and implementation to contain and 
> return BloomFilters directly rather than unstructured IO streams
> (2) Merge the fixes for index blocks and bloom blocks into open source
> (3) Create a new Jira ticket for open source HBase to add a ‘prevBlockSize’ 
> field in the block header in the next HFile version, so that seekBefore() 
> calls can not only be correct but performant in all cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14656) Move TestAssignmentManager from medium to large category

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965818#comment-14965818
 ] 

Hudson commented on HBASE-14656:


SUCCESS: Integrated in HBase-1.3-IT #254 (See 
[https://builds.apache.org/job/HBase-1.3-IT/254/])
HBASE-14656 Move TestAssignmentManager from medium to large category (stack: 
rev ec1f8df523d2d329375b943f2b22d0c49830c59f)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java


> Move TestAssignmentManager from medium to large category
> 
>
> Key: HBASE-14656
> URL: https://issues.apache.org/jira/browse/HBASE-14656
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14656.patch
>
>
> On internal rig, this test timedout. Its medium category and then I go this:
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal org.apache.maven.plugins:maven-surefire-plugin:2.18.1:test 
> (secondPartTestsExecution) on project hbase-server: There are test failures.
> Hopefully there is correlation. Let me move this test to large from medium so 
> it runs in its own jvm... and we see it timeout without breaking next stage 
> run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14634) Disable flakey TestSnapshotCloneIndependence.testOnlineSnapshotDeleteIndependent

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965819#comment-14965819
 ] 

Hudson commented on HBASE-14634:


SUCCESS: Integrated in HBase-1.3-IT #254 (See 
[https://builds.apache.org/job/HBase-1.3-IT/254/])
HBASE-14634 Disable flakey (stack: rev 4950db5110fbd5e343f14f97f7ee933ffd2d4353)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotCloneIndependence.java


> Disable flakey 
> TestSnapshotCloneIndependence.testOnlineSnapshotDeleteIndependent
> 
>
> Key: HBASE-14634
> URL: https://issues.apache.org/jira/browse/HBASE-14634
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14634.addendum.txt
>
>
> See HBASE-14585 for some history on this test. It just failed on patch build 
> twice in a row: 
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-TRUNK/6921/consoleText
>  and 
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-TRUNK/6922/consoleText
> Disabling for now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14655) Narrow the scope of doAs() calls to region observer notifications for compaction

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14655:
---
Attachment: 14655-v2.txt

Patch v2 fixes TestCompactionWithCoprocessor

> Narrow the scope of doAs() calls to region observer notifications for 
> compaction
> 
>
> Key: HBASE-14655
> URL: https://issues.apache.org/jira/browse/HBASE-14655
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 14655-v1.txt, 14655-v2.txt
>
>
> As what has been done in HBASE-14631 and HBASE-14605, the scope of calling 
> doAs() for compaction related region observer notifications should be 
> narrowed.
> User object is passed from CompactSplitThread down to the methods where 
> region observer notifications are made.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14634) Disable flakey TestSnapshotCloneIndependence.testOnlineSnapshotDeleteIndependent

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965758#comment-14965758
 ] 

Hudson commented on HBASE-14634:


SUCCESS: Integrated in HBase-1.2-IT #228 (See 
[https://builds.apache.org/job/HBase-1.2-IT/228/])
HBASE-14634 Disable flakey (stack: rev 2689f2a97e581524163a5eda5242cdb544b17908)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotCloneIndependence.java


> Disable flakey 
> TestSnapshotCloneIndependence.testOnlineSnapshotDeleteIndependent
> 
>
> Key: HBASE-14634
> URL: https://issues.apache.org/jira/browse/HBASE-14634
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14634.addendum.txt
>
>
> See HBASE-14585 for some history on this test. It just failed on patch build 
> twice in a row: 
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-TRUNK/6921/consoleText
>  and 
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-TRUNK/6922/consoleText
> Disabling for now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14634) Disable flakey TestSnapshotCloneIndependence.testOnlineSnapshotDeleteIndependent

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965837#comment-14965837
 ] 

Hudson commented on HBASE-14634:


SUCCESS: Integrated in HBase-1.3 #284 (See 
[https://builds.apache.org/job/HBase-1.3/284/])
HBASE-14634 Disable flakey (stack: rev 4950db5110fbd5e343f14f97f7ee933ffd2d4353)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotCloneIndependence.java


> Disable flakey 
> TestSnapshotCloneIndependence.testOnlineSnapshotDeleteIndependent
> 
>
> Key: HBASE-14634
> URL: https://issues.apache.org/jira/browse/HBASE-14634
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14634.addendum.txt
>
>
> See HBASE-14585 for some history on this test. It just failed on patch build 
> twice in a row: 
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-TRUNK/6921/consoleText
>  and 
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-TRUNK/6922/consoleText
> Disabling for now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14463) Severe performance downgrade when parallel reading a single key from BucketCache

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965840#comment-14965840
 ] 

Lars Hofhansl commented on HBASE-14463:
---

Bit late to the party. We do have the IdLock in order to only lock the block(s) 
in question, and not take a "global" lock in a sense. That probably causes the 
5% degradation. I'd assume that'd be worse if we only hit random blocks _and_ 
we have to load the blocks.

On first blush this does not strike as the right solution.

In HFileReaderXXX we do double-checked locking in order to avoid taking the 
lock completely when the block is already cached. Can we do something this that 
here?

> Severe performance downgrade when parallel reading a single key from 
> BucketCache
> 
>
> Key: HBASE-14463
> URL: https://issues.apache.org/jira/browse/HBASE-14463
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.14, 1.1.2
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.16
>
> Attachments: GC_with_WeakObjectPool.png, HBASE-14463.patch, 
> HBASE-14463_v11.patch, HBASE-14463_v12.patch, HBASE-14463_v2.patch, 
> HBASE-14463_v3.patch, HBASE-14463_v4.patch, HBASE-14463_v5.patch, 
> TestBucketCache-new_with_IdLock.png, 
> TestBucketCache-new_with_IdReadWriteLock.png, 
> TestBucketCache_with_IdLock-latest.png, TestBucketCache_with_IdLock.png, 
> TestBucketCache_with_IdReadWriteLock-latest.png, 
> TestBucketCache_with_IdReadWriteLock-resolveLockLeak.png, 
> TestBucketCache_with_IdReadWriteLock.png
>
>
> We store feature data of online items in HBase, do machine learning on these 
> features, and supply the outputs to our online search engine. In such 
> scenario we will launch hundreds of yarn workers and each worker will read 
> all features of one item(i.e. single rowkey in HBase), so there'll be heavy 
> parallel reading on a single rowkey.
> We were using LruCache but start to try BucketCache recently to resolve gc 
> issue, and just as titled we have observed severe performance downgrade. 
> After some analytics we found the root cause is the lock in 
> BucketCache#getBlock, as shown below
> {code}
>   try {
> lockEntry = offsetLock.getLockEntry(bucketEntry.offset());
> // ...
> if (bucketEntry.equals(backingMap.get(key))) {
>   // ...
>   int len = bucketEntry.getLength();
>   Cacheable cachedBlock = ioEngine.read(bucketEntry.offset(), len,
>   bucketEntry.deserializerReference(this.deserialiserMap));
> {code}
> Since ioEnging.read involves array copy, it's much more time-costed than the 
> operation in LruCache. And since we're using synchronized in 
> IdLock#getLockEntry, parallel read dropping on the same bucket would be 
> executed in serial, which causes a really bad performance.
> To resolve the problem, we propose to use ReentranceReadWriteLock in 
> BucketCache, and introduce a new class called IdReadWriteLock to implement it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14366) NPE in case visibility expression is not present in labels table during importtsv run

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965693#comment-14965693
 ] 

Hudson commented on HBASE-14366:


FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1116 (See 
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1116/])
HBASE-14366 NPE in case visibility expression is not present in labels 
(anoopsamjohn: rev 834035bac118d45dd93ce6ae96372a0fe0f83a2e)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTSVWithVisibilityLabels.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityLabelsCache.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TsvImporterMapper.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/DefaultVisibilityExpressionResolver.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TextSortReducer.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityConstants.java


> NPE in case visibility expression is not present in labels table during 
> importtsv run
> -
>
> Key: HBASE-14366
> URL: https://issues.apache.org/jira/browse/HBASE-14366
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: 0001-HBASE-14366.patch, 0001-HBASE-14366_1.patch, 
> HBASE-14366-0.98.patch, HBASE-14366-branch-1.1.patch, 
> HBASE-14366-branch-1.1_v1.patch, HBASE-14366-branch-1.2.patch, 
> HBASE-14366-branch-1.2_v1.patch, HBASE-14366-branch-1.patch, 
> HBASE-14366_2(1).patch, HBASE-14366_2.patch
>
>
> Below exception is shown in logs if visibility expression is not present in 
> labels table during importtsv run. Appropriate exception / message should be 
> logged for the user to take further action.
> {code}
> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver$1.getLabelOrdinal(DefaultVisibilityExpressionResolver.java:127)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.getLabelOrdinals(VisibilityUtils.java:358)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.createVisibilityExpTags(VisibilityUtils.java:323)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.createVisibilityExpTags(DefaultVisibilityExpressionResolver.java:137)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.populatePut(TsvImporterMapper.java:205)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:165)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14628) [0.98] Save object creation for scanning with block encodings

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965720#comment-14965720
 ] 

Lars Hofhansl commented on HBASE-14628:
---

Filed sub task (HBASE-14657) to remove the API from EncodedSeeker in 1.0+

> [0.98] Save object creation for scanning with block encodings
> -
>
> Key: HBASE-14628
> URL: https://issues.apache.org/jira/browse/HBASE-14628
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.98.16
>
> Attachments: 14628-0.98-v2.txt, 14628-0.98.txt
>
>
> I noticed that (at least in 0.98 - master is entirely different) we create 
> ByteBuffer just to create a byte[], which is then used to create a KeyValue.
> We can save the creation of the ByteBuffer and hence save allocating an extra 
> object for each KV we find by creating the byte[] directly.
> In a Phoenix count\(*) query that saved from 10% of runtime.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14657) Remove unneeded API from EncodedSeeker

[Lars Hofhansl (JIRA)]

Lars Hofhansl created HBASE-14657:
-

 Summary: Remove unneeded API from EncodedSeeker
 Key: HBASE-14657
 URL: https://issues.apache.org/jira/browse/HBASE-14657
 Project: HBase
  Issue Type: Sub-task
Reporter: Lars Hofhansl
 Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3


See parent. We do not need getKeyValueBuffer. It's only used for tests, and 
parent patch fixes all tests to use getKeyValue instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14656) Move TestAssignmentManager from medium to large category

[stack (JIRA)]

stack created HBASE-14656:
-

 Summary: Move TestAssignmentManager from medium to large category
 Key: HBASE-14656
 URL: https://issues.apache.org/jira/browse/HBASE-14656
 Project: HBase
  Issue Type: Sub-task
Reporter: stack


On internal rig, this test timedout. Its medium category and then I go this:

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.18.1:test 
(secondPartTestsExecution) on project hbase-server: There are test failures.


Hopefully there is correlation. Let me move this test to large from medium so 
it runs in its own jvm... and we see it timeout without breaking next stage run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14656) Move TestAssignmentManager from medium to large category

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-14656:
--
Attachment: 14656.patch

> Move TestAssignmentManager from medium to large category
> 
>
> Key: HBASE-14656
> URL: https://issues.apache.org/jira/browse/HBASE-14656
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
> Attachments: 14656.patch
>
>
> On internal rig, this test timedout. Its medium category and then I go this:
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal org.apache.maven.plugins:maven-surefire-plugin:2.18.1:test 
> (secondPartTestsExecution) on project hbase-server: There are test failures.
> Hopefully there is correlation. Let me move this test to large from medium so 
> it runs in its own jvm... and we see it timeout without breaking next stage 
> run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14656) Move TestAssignmentManager from medium to large category

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965757#comment-14965757
 ] 

Hudson commented on HBASE-14656:


SUCCESS: Integrated in HBase-1.2-IT #228 (See 
[https://builds.apache.org/job/HBase-1.2-IT/228/])
HBASE-14656 Move TestAssignmentManager from medium to large category (stack: 
rev 0090b718c38f3261d10d0c1e08daeee105385436)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java


> Move TestAssignmentManager from medium to large category
> 
>
> Key: HBASE-14656
> URL: https://issues.apache.org/jira/browse/HBASE-14656
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14656.patch
>
>
> On internal rig, this test timedout. Its medium category and then I go this:
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal org.apache.maven.plugins:maven-surefire-plugin:2.18.1:test 
> (secondPartTestsExecution) on project hbase-server: There are test failures.
> Hopefully there is correlation. Let me move this test to large from medium so 
> it runs in its own jvm... and we see it timeout without breaking next stage 
> run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14387) Compaction improvements: Maximum off-peak compaction size

[Vladimir Rodionov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov updated HBASE-14387:
--
Status: Patch Available  (was: Open)

> Compaction improvements: Maximum off-peak compaction size
> -
>
> Key: HBASE-14387
> URL: https://issues.apache.org/jira/browse/HBASE-14387
> Project: HBase
>  Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-14387-v1.patch, HBASE-14387-v2.patch
>
>
> Make max compaction size for peak and off peak separate config options.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14387) Compaction improvements: Maximum off-peak compaction size

[Vladimir Rodionov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov updated HBASE-14387:
--
Attachment: HBASE-14387-v2.patch

> Compaction improvements: Maximum off-peak compaction size
> -
>
> Key: HBASE-14387
> URL: https://issues.apache.org/jira/browse/HBASE-14387
> Project: HBase
>  Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-14387-v1.patch, HBASE-14387-v2.patch
>
>
> Make max compaction size for peak and off peak separate config options.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14463) Severe performance downgrade when parallel reading a single key from BucketCache

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965843#comment-14965843
 ] 

Lars Hofhansl commented on HBASE-14463:
---

Never mind. Looked at the patch again - should have taken a closer look before 
I sent the previous. Looks good.

> Severe performance downgrade when parallel reading a single key from 
> BucketCache
> 
>
> Key: HBASE-14463
> URL: https://issues.apache.org/jira/browse/HBASE-14463
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.14, 1.1.2
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.16
>
> Attachments: GC_with_WeakObjectPool.png, HBASE-14463.patch, 
> HBASE-14463_v11.patch, HBASE-14463_v12.patch, HBASE-14463_v2.patch, 
> HBASE-14463_v3.patch, HBASE-14463_v4.patch, HBASE-14463_v5.patch, 
> TestBucketCache-new_with_IdLock.png, 
> TestBucketCache-new_with_IdReadWriteLock.png, 
> TestBucketCache_with_IdLock-latest.png, TestBucketCache_with_IdLock.png, 
> TestBucketCache_with_IdReadWriteLock-latest.png, 
> TestBucketCache_with_IdReadWriteLock-resolveLockLeak.png, 
> TestBucketCache_with_IdReadWriteLock.png
>
>
> We store feature data of online items in HBase, do machine learning on these 
> features, and supply the outputs to our online search engine. In such 
> scenario we will launch hundreds of yarn workers and each worker will read 
> all features of one item(i.e. single rowkey in HBase), so there'll be heavy 
> parallel reading on a single rowkey.
> We were using LruCache but start to try BucketCache recently to resolve gc 
> issue, and just as titled we have observed severe performance downgrade. 
> After some analytics we found the root cause is the lock in 
> BucketCache#getBlock, as shown below
> {code}
>   try {
> lockEntry = offsetLock.getLockEntry(bucketEntry.offset());
> // ...
> if (bucketEntry.equals(backingMap.get(key))) {
>   // ...
>   int len = bucketEntry.getLength();
>   Cacheable cachedBlock = ioEngine.read(bucketEntry.offset(), len,
>   bucketEntry.deserializerReference(this.deserialiserMap));
> {code}
> Since ioEnging.read involves array copy, it's much more time-costed than the 
> operation in LruCache. And since we're using synchronized in 
> IdLock#getLockEntry, parallel read dropping on the same bucket would be 
> executed in serial, which causes a really bad performance.
> To resolve the problem, we propose to use ReentranceReadWriteLock in 
> BucketCache, and introduce a new class called IdReadWriteLock to implement it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14633) Try fluid width UI

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965875#comment-14965875
 ] 

Hadoop QA commented on HBASE-14633:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12767651/HBASE-14633-v5.patch
  against master branch at commit a532ed73d808f543909c5563999405c82fa9b1d5.
  ATTACHMENT ID: 12767651

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 
2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
 

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16124//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16124//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16124//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16124//console

This message is automatically generated.

> Try fluid width UI
> --
>
> Key: HBASE-14633
> URL: https://issues.apache.org/jira/browse/HBASE-14633
> Project: HBase
>  Issue Type: Bug
>  Components: UI
>Affects Versions: 2.0.0, 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-14633-v1.patch, HBASE-14633-v2.patch, 
> HBASE-14633-v3.patch, HBASE-14633-v4.patch, HBASE-14633-v5.patch, 
> HBASE-14633.patch, Screen Shot 2015-10-20 at 2.40.23 AM.png, Screen Shot 
> 2015-10-20 at 2.40.34 AM.png, Screen Shot 2015-10-20 at 2.44.11 AM.png
>
>
> Our UI is often too long. Lets give it more room if available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14628) [0.98] Save object creation for scanning with block encodings

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965890#comment-14965890
 ] 

Lars Hofhansl commented on HBASE-14628:
---

[~giacomotaylor], FYI. Shaves another 5-10% of many Phoenix queries that 
traverse a lot of KVs.

> [0.98] Save object creation for scanning with block encodings
> -
>
> Key: HBASE-14628
> URL: https://issues.apache.org/jira/browse/HBASE-14628
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.98.16
>
> Attachments: 14628-0.98-v2.txt, 14628-0.98.txt
>
>
> I noticed that (at least in 0.98 - master is entirely different) we create 
> ByteBuffer just to create a byte[], which is then used to create a KeyValue.
> We can save the creation of the ByteBuffer and hence save allocating an extra 
> object for each KV we find by creating the byte[] directly.
> In a Phoenix count\(*) query that saved from 10% of runtime.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14634) Disable flakey TestSnapshotCloneIndependence.testOnlineSnapshotDeleteIndependent

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965860#comment-14965860
 ] 

Hudson commented on HBASE-14634:


FAILURE: Integrated in HBase-1.2 #279 (See 
[https://builds.apache.org/job/HBase-1.2/279/])
HBASE-14634 Disable flakey (stack: rev 2689f2a97e581524163a5eda5242cdb544b17908)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotCloneIndependence.java


> Disable flakey 
> TestSnapshotCloneIndependence.testOnlineSnapshotDeleteIndependent
> 
>
> Key: HBASE-14634
> URL: https://issues.apache.org/jira/browse/HBASE-14634
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14634.addendum.txt
>
>
> See HBASE-14585 for some history on this test. It just failed on patch build 
> twice in a row: 
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-TRUNK/6921/consoleText
>  and 
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-TRUNK/6922/consoleText
> Disabling for now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14656) Move TestAssignmentManager from medium to large category

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965859#comment-14965859
 ] 

Hudson commented on HBASE-14656:


FAILURE: Integrated in HBase-1.2 #279 (See 
[https://builds.apache.org/job/HBase-1.2/279/])
HBASE-14656 Move TestAssignmentManager from medium to large category (stack: 
rev 0090b718c38f3261d10d0c1e08daeee105385436)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java


> Move TestAssignmentManager from medium to large category
> 
>
> Key: HBASE-14656
> URL: https://issues.apache.org/jira/browse/HBASE-14656
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14656.patch
>
>
> On internal rig, this test timedout. Its medium category and then I go this:
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal org.apache.maven.plugins:maven-surefire-plugin:2.18.1:test 
> (secondPartTestsExecution) on project hbase-server: There are test failures.
> Hopefully there is correlation. Let me move this test to large from medium so 
> it runs in its own jvm... and we see it timeout without breaking next stage 
> run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)