date:20161113

[jira] [Commented] (HBASE-17074) PreCommit job always fails because of OOM

2016-11-13 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-17074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663079#comment-15663079
 ] 

Hadoop QA commented on HBASE-17074:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 45s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 59s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 41s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 41s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
29m 11s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 1s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 88m 8s {color} 
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
19s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 136m 51s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.client.TestSplitOrMergeStatus 
|
|   | org.apache.hadoop.hbase.client.TestAdmin1 |
|   | org.apache.hadoop.hbase.client.TestHCM |
|   | org.apache.hadoop.hbase.client.TestSnapshotFromClientWithRegionReplicas |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12838712/HBASE-17074.patch |
| JIRA Issue | HBASE-17074 |
| Optional Tests |  asflicense  javac  javadoc  unit  xml  compile  |
| uname | Linux dc7d35637f56 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 9250bf8 |
| Default Java | 1.8.0_101 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4453/artifact/patchprocess/patch-unit-root.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-HBASE-Build/4453/artifact/patchprocess/patch-unit-root.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4453/testReport/ |
| modules | C: . U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4453/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> PreCommit job always fails because of OOM
> -
>
> Key: HBASE-17074
> URL: https://issues.apache.org/jira/browse/HBASE-17074
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Attachments: HBASE-17074.patch
>
>
>

[jira] [Commented] (HBASE-15786) Create DBB backed MSLAB pool

2016-11-13 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663026#comment-15663026
 ] 

Anoop Sam John commented on HBASE-15786:


bq.MemorySizeUtil needs class comment explaining it. Don't you have a nice 
write up on how this thing operators that you could copy/paste in here?
That is flush related decision points which is handled by another patch..  Will 
make sure we have comments copied from the google doc there.


> Create DBB backed MSLAB pool
> 
>
> Key: HBASE-15786
> URL: https://issues.apache.org/jira/browse/HBASE-15786
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Reporter: ramkrishna.s.vasudevan
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15786.patch
>
>
> We can make use of MSLAB pool for this off heap memstore. 
> Right now one can specify the global memstore size (heap size) as a % of max 
> memory using a config. We will add another config with which one can specify 
> the global off heap memstore size. This will be exact size not as %. When off 
> heap memstore in use, we will give this entire area for the MSLAB pool and 
> that will create off heap chunks. So when cells are added to memstore, the 
> cell data gets copied into the off heap MSLAB chunk spaces. Note that when 
> the pool size is not really enough and we need additional chunk creation, we 
> wont use off heap area for that.  We dony want to create so many on demand 
> DBBs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15786) Create DBB backed MSLAB pool

2016-11-13 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663024#comment-15663024
 ] 

Anoop Sam John commented on HBASE-15786:


bq.Sometimes we are saying ByteBuffered as in ByteBufferedCell and now we are 
starting to say ByteBuffer instead as in ByteBufferWriter or ByteBufferUtil
Ya we call ByteBufferedCell and there some other places in Codec (Which will 
get changed by HBASE-15788 patch any way).. So we can change ByteBufferedCell 
to be ByteBufferCell?

> Create DBB backed MSLAB pool
> 
>
> Key: HBASE-15786
> URL: https://issues.apache.org/jira/browse/HBASE-15786
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Reporter: ramkrishna.s.vasudevan
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15786.patch
>
>
> We can make use of MSLAB pool for this off heap memstore. 
> Right now one can specify the global memstore size (heap size) as a % of max 
> memory using a config. We will add another config with which one can specify 
> the global off heap memstore size. This will be exact size not as %. When off 
> heap memstore in use, we will give this entire area for the MSLAB pool and 
> that will create off heap chunks. So when cells are added to memstore, the 
> cell data gets copied into the off heap MSLAB chunk spaces. Note that when 
> the pool size is not really enough and we need additional chunk creation, we 
> wont use off heap area for that.  We dony want to create so many on demand 
> DBBs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17087) Enable Aliasing for CodedInputStream created by ByteInputByteString#newCodedInput

2016-11-13 Thread Anoop Sam John (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-17087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-17087:
---
Status: Patch Available  (was: Open)

> Enable Aliasing for CodedInputStream created by 
> ByteInputByteString#newCodedInput
> -
>
> Key: HBASE-17087
> URL: https://issues.apache.org/jira/browse/HBASE-17087
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-17087.patch, HBASE-17087.patch
>
>
> Missed setting this while doing HBASE-15789.  We make CIS with 
> 'bufferIsImmutable' as true but we should do enableAliasing also to avoid 
> copy while building PB objects from this new CIS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync

2016-11-13 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663000#comment-15663000
 ] 

Hadoop QA commented on HBASE-17085:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
0s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
27m 7s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 58s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
14s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 117m 0s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | 
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDefaultVisLabelService
 |
|   | 
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithCustomVisLabService
 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:7bda515 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12838709/HBASE-17085-v1.patch |
| JIRA Issue | HBASE-17085 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 5e271956dbf9 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 9250bf8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4452/artifact/patchprocess/patch-unit-hbase-server.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-HBASE-Build/4452/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4452/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4452/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



>

[jira] [Updated] (HBASE-17087) Enable Aliasing for CodedInputStream created by ByteInputByteString#newCodedInput

2016-11-13 Thread Anoop Sam John (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-17087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-17087:
---
Attachment: HBASE-17087.patch

Simple patch. [~saint@gmail.com] This is the way we need PB patches to be 
right?

> Enable Aliasing for CodedInputStream created by 
> ByteInputByteString#newCodedInput
> -
>
> Key: HBASE-17087
> URL: https://issues.apache.org/jira/browse/HBASE-17087
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-17087.patch, HBASE-17087.patch
>
>
> Missed setting this while doing HBASE-15789.  We make CIS with 
> 'bufferIsImmutable' as true but we should do enableAliasing also to avoid 
> copy while building PB objects from this new CIS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17087) Enable Aliasing for CodedInputStream created by ByteInputByteString#newCodedInput

2016-11-13 Thread Anoop Sam John (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-17087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-17087:
---
Attachment: HBASE-17087.patch

> Enable Aliasing for CodedInputStream created by 
> ByteInputByteString#newCodedInput
> -
>
> Key: HBASE-17087
> URL: https://issues.apache.org/jira/browse/HBASE-17087
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-17087.patch
>
>
> Missed setting this while doing HBASE-15789.  We make CIS with 
> 'bufferIsImmutable' as true but we should do enableAliasing also to avoid 
> copy while building PB objects from this new CIS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17087) Enable Aliasing for CodedInputStream created by ByteInputByteString#newCodedInput

2016-11-13 Thread Anoop Sam John (JIRA)

Anoop Sam John created HBASE-17087:
--

 Summary: Enable Aliasing for CodedInputStream created by 
ByteInputByteString#newCodedInput
 Key: HBASE-17087
 URL: https://issues.apache.org/jira/browse/HBASE-17087
 Project: HBase
  Issue Type: Sub-task
Affects Versions: 2.0.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
 Fix For: 2.0.0


Missed setting this while doing HBASE-15789.  We make CIS with 
'bufferIsImmutable' as true but we should do enableAliasing also to avoid copy 
while building PB objects from this new CIS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17074) PreCommit job always fails because of OOM

2016-11-13 Thread Duo Zhang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-17074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang updated HBASE-17074:
--
Assignee: Duo Zhang
  Status: Patch Available  (was: Open)

> PreCommit job always fails because of OOM
> -
>
> Key: HBASE-17074
> URL: https://issues.apache.org/jira/browse/HBASE-17074
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Attachments: HBASE-17074.patch
>
>
> https://builds.apache.org/job/PreCommit-HBASE-Build/4434/artifact/patchprocess/patch-unit-hbase-server.txt
> {noformat}
> Exception in thread "Thread-2369" java.lang.OutOfMemoryError: Java heap space
>   at java.util.Arrays.copyOf(Arrays.java:3332)
>   at 
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
>   at 
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:596)
>   at java.lang.StringBuffer.append(StringBuffer.java:367)
>   at java.io.BufferedReader.readLine(BufferedReader.java:370)
>   at java.io.BufferedReader.readLine(BufferedReader.java:389)
>   at 
> org.apache.maven.surefire.shade.org.apache.maven.shared.utils.cli.StreamPumper.run(StreamPumper.java:66)
> Exception in thread "Thread-2357" java.lang.OutOfMemoryError: Java heap space
> Exception in thread "Thread-2365" java.lang.OutOfMemoryError: Java heap space
> Running org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd
> Running org.apache.hadoop.hbase.filter.TestFilterListOrOperatorWithBlkCnt
> Exception in thread "Thread-2383" java.lang.OutOfMemoryError: Java heap space
> Exception in thread "Thread-2397" java.lang.OutOfMemoryError: Java heap space
> Exception in thread "Thread-2401" java.lang.OutOfMemoryError: Java heap space
> Running org.apache.hadoop.hbase.TestHBaseTestingUtility
> Exception in thread "Thread-2407" java.lang.OutOfMemoryError: Java heap space
> Exception in thread "Thread-2411" java.lang.OutOfMemoryError: Java heap space
> Exception in thread "Thread-2413" java.lang.OutOfMemoryError: Java heap space
> {noformat}
> The OOM happens in the surefire plugin when reading the stdout or stderr of 
> the running test...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17074) PreCommit job always fails because of OOM

2016-11-13 Thread Duo Zhang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-17074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang updated HBASE-17074:
--
Attachment: HBASE-17074.patch

I‘ve changed the heap size of precommit job from 3100m to 6100m(The heap size 
of our HBase-trunk job is 6100m).

Let's see if it helps.

> PreCommit job always fails because of OOM
> -
>
> Key: HBASE-17074
> URL: https://issues.apache.org/jira/browse/HBASE-17074
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Reporter: Duo Zhang
>Priority: Critical
> Attachments: HBASE-17074.patch
>
>
> https://builds.apache.org/job/PreCommit-HBASE-Build/4434/artifact/patchprocess/patch-unit-hbase-server.txt
> {noformat}
> Exception in thread "Thread-2369" java.lang.OutOfMemoryError: Java heap space
>   at java.util.Arrays.copyOf(Arrays.java:3332)
>   at 
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
>   at 
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:596)
>   at java.lang.StringBuffer.append(StringBuffer.java:367)
>   at java.io.BufferedReader.readLine(BufferedReader.java:370)
>   at java.io.BufferedReader.readLine(BufferedReader.java:389)
>   at 
> org.apache.maven.surefire.shade.org.apache.maven.shared.utils.cli.StreamPumper.run(StreamPumper.java:66)
> Exception in thread "Thread-2357" java.lang.OutOfMemoryError: Java heap space
> Exception in thread "Thread-2365" java.lang.OutOfMemoryError: Java heap space
> Running org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd
> Running org.apache.hadoop.hbase.filter.TestFilterListOrOperatorWithBlkCnt
> Exception in thread "Thread-2383" java.lang.OutOfMemoryError: Java heap space
> Exception in thread "Thread-2397" java.lang.OutOfMemoryError: Java heap space
> Exception in thread "Thread-2401" java.lang.OutOfMemoryError: Java heap space
> Running org.apache.hadoop.hbase.TestHBaseTestingUtility
> Exception in thread "Thread-2407" java.lang.OutOfMemoryError: Java heap space
> Exception in thread "Thread-2411" java.lang.OutOfMemoryError: Java heap space
> Exception in thread "Thread-2413" java.lang.OutOfMemoryError: Java heap space
> {noformat}
> The OOM happens in the surefire plugin when reading the stdout or stderr of 
> the running test...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync

2016-11-13 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662819#comment-15662819
 ] 

ramkrishna.s.vasudevan commented on HBASE-17085:


Am just seeing your -v1 patch.
{code}
 if (!syncFutures.isEmpty()
457 && syncFutures.last().getTxid() > 
highestProcessedAppendTxidAtLastSync) {
{code}
This seems to be your fix. I can try your patch in my cluster. 
{code}
And there is no duplicated record. It is just a pipeline. 
{code}
bq.we will always issue an AsyncDFSOutput.sync after an append even if there is 
no new sync request.
True. So you mean that since append has happend only that new data will be 
again called for sync. And because of that the number of syncs may be more and 
here you plan to aggregate more. 

> AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
> 
>
> Key: HBASE-17085
> URL: https://issues.apache.org/jira/browse/HBASE-17085
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17085-v1.patch, HBASE-17085.patch
>
>
> The problem is in appendAndSync method, we will issue an  AsyncDFSOutput.sync 
> if syncFutures is not empty. The SyncFutures in syncFutures can only be 
> removed after an AsyncDFSOutput.sync comes back, so before the 
> AsyncDFSOutput.sync actually returns, we will always issue an  
> AsyncDFSOutput.sync after an append even if there is no new sync request.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12341) Overhaul logging; log4j2, machine-readable, etc.

2016-11-13 Thread Nick Dimiduk (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-12341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662821#comment-15662821
 ] 

Nick Dimiduk commented on HBASE-12341:
--

Has there been a design discussion on this topic? Anyone mention moving to a 
log API (ie, slf4j) and leaving the backend up to runtime decision?

> Overhaul logging; log4j2, machine-readable, etc.
> 
>
> Key: HBASE-12341
> URL: https://issues.apache.org/jira/browse/HBASE-12341
> Project: HBase
>  Issue Type: Umbrella
>Reporter: stack
>Priority: Critical
> Fix For: 2.0.0
>
>
> This is a general umbrella issue for 2.x logging improvements. Hang related 
> work off this one.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17086) Cell#getTagsLength() returns a int, but in KeyValue, tags length is a short

2016-11-13 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-17086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662817#comment-15662817
 ] 

Anoop Sam John commented on HBASE-17086:


{code}
/**
   * @return the total length of the tags in the Cell.
   * @deprecated use {@link #getTagsLengthUnsigned()} which can handle tags 
length upto 65535.
   */
  @Deprecated
  @InterfaceStability.Unstable
  short getTagsLength();
{code}
The above is from 0.98.  We first added tags length as short.  But 0.98 only 
deprecated it and went with int.  This is because, even if we store it in 2 
bytes, we make use of the fact that the length can not be -ve. So make use of 
the sign bit also and can go double the max short value.  So in order to return 
that, the type has to be bigger than short and that is int.  Hope am clarifying 
it for u.


> Cell#getTagsLength() returns a int, but in KeyValue, tags length is a short
> ---
>
> Key: HBASE-17086
> URL: https://issues.apache.org/jira/browse/HBASE-17086
> Project: HBase
>  Issue Type: Bug
>  Components:  Interface
>Affects Versions: 2.0.0
>Reporter: Xiang Li
>
> In the Cell interface, getTagsLength() returns a int
> But in the KeyValue implementation, tags length is a short. Also in 
> ExtendedCell, when explaining the KeyValue format, tags length is stated to 
> be 2 bytes
> Any plan to update Cell interface to make getTagsLength() returns a short ?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync

2016-11-13 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662813#comment-15662813
 ] 

Duo Zhang commented on HBASE-17085:
---

I used to tried limiting the concurrent sync requests but it did not help. I 
think we could try again after this patch is applied.

And there is no duplicated record. It is just a pipeline. In FSHLog we also use 
5 threads to do syncing(That's why I limited the number of concurrent sync 
requests to 5)

If no concern, let's commit the patch here first? Thanks.

> AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
> 
>
> Key: HBASE-17085
> URL: https://issues.apache.org/jira/browse/HBASE-17085
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17085-v1.patch, HBASE-17085.patch
>
>
> The problem is in appendAndSync method, we will issue an  AsyncDFSOutput.sync 
> if syncFutures is not empty. The SyncFutures in syncFutures can only be 
> removed after an AsyncDFSOutput.sync comes back, so before the 
> AsyncDFSOutput.sync actually returns, we will always issue an  
> AsyncDFSOutput.sync after an append even if there is no new sync request.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync

2016-11-13 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662804#comment-15662804
 ] 

ramkrishna.s.vasudevan commented on HBASE-17085:


Can we try limiting the sync that happens? I know you tried with some sync 
count of 5. So even that did not help. 
And do think we had duplicate records created as we were syncing even if the 
current sync did not get completed.

> AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
> 
>
> Key: HBASE-17085
> URL: https://issues.apache.org/jira/browse/HBASE-17085
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17085-v1.patch, HBASE-17085.patch
>
>
> The problem is in appendAndSync method, we will issue an  AsyncDFSOutput.sync 
> if syncFutures is not empty. The SyncFutures in syncFutures can only be 
> removed after an AsyncDFSOutput.sync comes back, so before the 
> AsyncDFSOutput.sync actually returns, we will always issue an  
> AsyncDFSOutput.sync after an append even if there is no new sync request.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17049) Find out why AsyncFSWAL issues much more syncs than FSHLog

2016-11-13 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662787#comment-15662787
 ] 

Duo Zhang commented on HBASE-17049:
---

With HBASE-17085 we can perform a little better in the PE.

{noformat}
./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred 
--presplit=50 --size=50 --columns=50 --valueSize=200 --writeToWAL=true 
--bloomFilter=NONE randomWrite 50

FSHLog: Min: 494940ms   Max: 527185ms   Avg: 520070ms
AsyncFSWAL-old: Min: 899553ms   Max: 937701ms   Avg: 931112ms
AsyncFSWAL-new:Min: 745193ms   Max: 770322ms   Avg: 764724ms
{noformat}

Still slower than FSHLog. Need to find out more.

Thanks.

> Find out why AsyncFSWAL issues much more syncs than FSHLog
> --
>
> Key: HBASE-17049
> URL: https://issues.apache.org/jira/browse/HBASE-17049
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
> Fix For: 2.0.0
>
>
> https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17086) Cell#getTagsLength() returns a int, but in KeyValue, tags length is a short

2016-11-13 Thread Xiang Li (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-17086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiang Li updated HBASE-17086:
-
Summary: Cell#getTagsLength() returns a int, but in KeyValue, tags length 
is a short  (was: Cell#getTagsLength() returns a int but in KeyValue, tags 
length is a short)

> Cell#getTagsLength() returns a int, but in KeyValue, tags length is a short
> ---
>
> Key: HBASE-17086
> URL: https://issues.apache.org/jira/browse/HBASE-17086
> Project: HBase
>  Issue Type: Bug
>  Components:  Interface
>Affects Versions: 2.0.0
>Reporter: Xiang Li
>
> In the Cell interface, getTagsLength() returns a int
> But in the KeyValue implementation, tags length is a short. Also in 
> ExtendedCell, when explaining the KeyValue format, tags length is stated to 
> be 2 bytes
> Any plan to update Cell interface to make getTagsLength() returns a short ?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17086) Cell#getTagsLength() returns a int but in KeyValue, tags length is a short

2016-11-13 Thread Xiang Li (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-17086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662782#comment-15662782
 ] 

Xiang Li commented on HBASE-17086:
--

[~anoop.hbase], you are the expert here. Any comments? Too much changes to make 
if we would like to update the Cell interface?

> Cell#getTagsLength() returns a int but in KeyValue, tags length is a short
> --
>
> Key: HBASE-17086
> URL: https://issues.apache.org/jira/browse/HBASE-17086
> Project: HBase
>  Issue Type: Bug
>  Components:  Interface
>Affects Versions: 2.0.0
>Reporter: Xiang Li
>
> In the Cell interface, getTagsLength() returns a int
> But in the KeyValue implementation, tags length is a short. Also in 
> ExtendedCell, when explaining the KeyValue format, tags length is stated to 
> be 2 bytes
> Any plan to update Cell interface to make getTagsLength() returns a short ?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync

2016-11-13 Thread Duo Zhang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang updated HBASE-17085:
--
Attachment: HBASE-17085-v1.patch

> AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
> 
>
> Key: HBASE-17085
> URL: https://issues.apache.org/jira/browse/HBASE-17085
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17085-v1.patch, HBASE-17085.patch
>
>
> The problem is in appendAndSync method, we will issue an  AsyncDFSOutput.sync 
> if syncFutures is not empty. The SyncFutures in syncFutures can only be 
> removed after an AsyncDFSOutput.sync comes back, so before the 
> AsyncDFSOutput.sync actually returns, we will always issue an  
> AsyncDFSOutput.sync after an append even if there is no new sync request.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17086) Cell#getTagsLength() returns a int but in KeyValue, tags length is a short

2016-11-13 Thread Xiang Li (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-17086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiang Li updated HBASE-17086:
-
Description: 
In the Cell interface, getTagsLength() returns a int
But in the KeyValue implementation, tags length is a short. Also in 
ExtendedCell, when explaining the KeyValue format, tags length is stated to be 
2 bytes

Any plan to update Cell interface to make getTagsLength() returns a short ?

> Cell#getTagsLength() returns a int but in KeyValue, tags length is a short
> --
>
> Key: HBASE-17086
> URL: https://issues.apache.org/jira/browse/HBASE-17086
> Project: HBase
>  Issue Type: Bug
>  Components:  Interface
>Affects Versions: 2.0.0
>Reporter: Xiang Li
>
> In the Cell interface, getTagsLength() returns a int
> But in the KeyValue implementation, tags length is a short. Also in 
> ExtendedCell, when explaining the KeyValue format, tags length is stated to 
> be 2 bytes
> Any plan to update Cell interface to make getTagsLength() returns a short ?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17086) Cell#getTagsLength() returns a int but in KeyValue, tags length is a short

2016-11-13 Thread Xiang Li (JIRA)

Xiang Li created HBASE-17086:


 Summary: Cell#getTagsLength() returns a int but in KeyValue, tags 
length is a short
 Key: HBASE-17086
 URL: https://issues.apache.org/jira/browse/HBASE-17086
 Project: HBase
  Issue Type: Bug
  Components:  Interface
Affects Versions: 2.0.0
Reporter: Xiang Li






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17081) Flush the entire CompactingMemStore content to disk

2016-11-13 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-17081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662734#comment-15662734
 ] 

ramkrishna.s.vasudevan commented on HBASE-17081:


So [~anastas] - can I take this up. 

> Flush the entire CompactingMemStore content to disk
> ---
>
> Key: HBASE-17081
> URL: https://issues.apache.org/jira/browse/HBASE-17081
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Anastasia Braginsky
>
> Part of CompactingMemStore's memory is held by an active segment, and another 
> part is divided between immutable segments in the compacting pipeline. Upon 
> flush-to-disk request we want to flush all of it to disk, in contrast to 
> flushing only tail of the compacting pipeline.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16417) In-Memory MemStore Policy for Flattening and Compactions

2016-11-13 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-16417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662732#comment-15662732
 ] 

ramkrishna.s.vasudevan commented on HBASE-16417:


Is this JIRA due to this
bq.This bug is about to be fixed in a new patch Anastasia is working on.

> In-Memory MemStore Policy for Flattening and Compactions
> 
>
> Key: HBASE-16417
> URL: https://issues.apache.org/jira/browse/HBASE-16417
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Eshcar Hillel
> Fix For: 2.0.0
>
> Attachments: HBASE-16417-benchmarkresults-20161101.pdf, 
> HBASE-16417-benchmarkresults-20161110.pdf
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16417) In-Memory MemStore Policy for Flattening and Compactions

2016-11-13 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-16417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662731#comment-15662731
 ] 

ramkrishna.s.vasudevan commented on HBASE-16417:


ACtually we had a discussion around this. But as Anoop said we cannot go with 
first memstore and then HFiles as per the reasons stated above. 
For a given row - if there are 100 qualifiers (single CF), you could add them 
in 100 different puts one by one. 
So in this case when we say get that row - we are not sure if by this time if 
any flush had happened moving some of the qualifiers to the file. 
The InternalScan is not exposed but if you want a behaviour where you are sure 
that you do frequent updates and fetch only the recent one (the catch is that 
if the recent is not in memstore - you won't get any result) then we may have 
to expose readOnlyMemstore type of API in scan. But am not sure how beneficial 
it i will be where there are more chances for missing results.

> In-Memory MemStore Policy for Flattening and Compactions
> 
>
> Key: HBASE-16417
> URL: https://issues.apache.org/jira/browse/HBASE-16417
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Eshcar Hillel
> Fix For: 2.0.0
>
> Attachments: HBASE-16417-benchmarkresults-20161101.pdf, 
> HBASE-16417-benchmarkresults-20161110.pdf
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15199) Move jruby jar so only on hbase-shell module classpath; currently globally available

2016-11-13 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662698#comment-15662698
 ] 

Hadoop QA commented on HBASE-15199:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 2s 
{color} | {color:blue} The patch file was not named according to hbase's naming 
conventions. Please see 
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for 
instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
1s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 46s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 56s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 46s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 46s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 3s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
27m 13s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 59s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 49s 
{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 3s {color} 
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
27s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 150m 59s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestMetaWithReplicas |
|   | org.apache.hadoop.hbase.master.TestTableLockManager |
|   | org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor |
|   | org.apache.hadoop.hbase.client.TestHCM |
|   | org.apache.hadoop.hbase.client.TestTableSnapshotScanner |
|   | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:7bda515 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12838695/15199.txt |
| JIRA Issue | HBASE-15199 |
| Optional Tests |  asflicense  javac  javadoc  unit  xml  compile  |
| uname | Linux 744f378a9983 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 9250bf8 |
| Default Java | 1.8.0_111 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4451/artifact/patchprocess/patch-unit-root.txt
 |
|

[jira] [Commented] (HBASE-14255) Simplify Cell creation post 1.0

2016-11-13 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662626#comment-15662626
 ] 

Anoop Sam John commented on HBASE-14255:


In CellUtil, which is public, we have diff versions of createCell APIs.
bq.public static Cell createCell(final byte [] row, final byte [] family, final 
byte [] qualifier) 
Here we don't have value to be passed.  We should be having an overloaded API 
which takes value byte[] also..  Seems that is what Lars looking for.
Should be a simple API. I can give a quick patch.  Or u r working on it in some 
other way [~saint@gmail.com]?


> Simplify Cell creation post 1.0
> ---
>
> Key: HBASE-14255
> URL: https://issues.apache.org/jira/browse/HBASE-14255
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Lars George
>Priority: Critical
> Fix For: 2.0.0
>
>
> After the switch to the new Cell based client API, and making KeyValue 
> private (but especially as soon as DBB backed Cells land) it is rather 
> difficult to create a {{Cell}} instance. I am using this now:
> {code}
>  @Override
>   public void postGetOp(ObserverContext e,
> Get get, List results) throws IOException {
> Put put = new Put(get.getRow());
> put.addColumn(get.getRow(), FIXED_COLUMN, Bytes.toBytes(counter.get()));
> CellScanner scanner = put.cellScanner();
> scanner.advance();
> Cell cell = scanner.current();
> LOG.debug("Adding fake cell: " + cell);
> results.add(cell);
>   }
> {code}
> That is, I have to create a {{Put}} instance to add a cell and then retrieve 
> its instance. The {{KeyValue}} methods are private now and should not be 
> used. Create a CellBuilder helper?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16417) In-Memory MemStore Policy for Flattening and Compactions

2016-11-13 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-16417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662618#comment-15662618
 ] 

Anoop Sam John commented on HBASE-16417:


It is not just TS being applied by client issue.  (Yes that is also one and 
when we are sure that all TS applied by server only and it is strictly 
increasing some other optimizations also possible.  There is some jira around 
that. Forgot id. But LarsH raised that)
The main thing is that when we do a row get, how u will know that u have done 
with all the columns in that row? HBase being a CF oriented NoSQL, we don't 
know the columns within a CF and it can differ from row to row.  But when we 
know the column qualifiers always and specify in Get, and we look for only one 
version and we are sure abt the TS increasing nature, ya the optimization is 
possible. The ColumnTracker always track and allow only the given columns in 
Get are being selected.  And the other 2 also stands, then we can do this 
optimization I believe..  I did not do any code read or deep analysis..I 
get ur point also.. In usecase like u described, it is some thing to really 
think abt
{code}
/**
 * Special scanner, currently used for increment operations to
 * allow additional server-side arguments for Scan operations.
 * 
 * Rather than adding new options/parameters to the public Scan API, this new
 * class has been created.
 * 
 * Supports adding an option to only read from the MemStore with
 * {@link #checkOnlyMemStore()} or to only read from StoreFiles with
 * {@link #checkOnlyStoreFiles()}.
 */
@InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.COPROC)
public class InternalScan extends Scan {
{code}
This is not public client side exposed.  Only can be used within CPs.  JFYI

> In-Memory MemStore Policy for Flattening and Compactions
> 
>
> Key: HBASE-16417
> URL: https://issues.apache.org/jira/browse/HBASE-16417
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Eshcar Hillel
> Fix For: 2.0.0
>
> Attachments: HBASE-16417-benchmarkresults-20161101.pdf, 
> HBASE-16417-benchmarkresults-20161110.pdf
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14255) Simplify Cell creation post 1.0

2016-11-13 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-14255:
--
Fix Version/s: 2.0.0

> Simplify Cell creation post 1.0
> ---
>
> Key: HBASE-14255
> URL: https://issues.apache.org/jira/browse/HBASE-14255
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Lars George
>Priority: Critical
> Fix For: 2.0.0
>
>
> After the switch to the new Cell based client API, and making KeyValue 
> private (but especially as soon as DBB backed Cells land) it is rather 
> difficult to create a {{Cell}} instance. I am using this now:
> {code}
>  @Override
>   public void postGetOp(ObserverContext e,
> Get get, List results) throws IOException {
> Put put = new Put(get.getRow());
> put.addColumn(get.getRow(), FIXED_COLUMN, Bytes.toBytes(counter.get()));
> CellScanner scanner = put.cellScanner();
> scanner.advance();
> Cell cell = scanner.current();
> LOG.debug("Adding fake cell: " + cell);
> results.add(cell);
>   }
> {code}
> That is, I have to create a {{Put}} instance to add a cell and then retrieve 
> its instance. The {{KeyValue}} methods are private now and should not be 
> used. Create a CellBuilder helper?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14282) Remove metrics2

2016-11-13 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662490#comment-15662490
 ] 

stack commented on HBASE-14282:
---

Good idea. Critical actually but no progress so removing from 2.0.0.

> Remove metrics2
> ---
>
> Key: HBASE-14282
> URL: https://issues.apache.org/jira/browse/HBASE-14282
> Project: HBase
>  Issue Type: Improvement
>  Components: metrics
>Reporter: Elliott Clark
>Priority: Critical
> Attachments: HBASE-14282.patch
>
>
> Metrics2 has a whole bunch of race conditions and weird edges because of all 
> of the caching that metrics2 does.
> Additionally tying something so integral to hadoop who has lots of versions 
> we support has been a maintenance nightmare. It would allow us to completely 
> get rid of the compat modules.
> Rip it out.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14282) Remove metrics2

2016-11-13 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-14282:
--
Fix Version/s: (was: 2.0.0)

> Remove metrics2
> ---
>
> Key: HBASE-14282
> URL: https://issues.apache.org/jira/browse/HBASE-14282
> Project: HBase
>  Issue Type: Improvement
>  Components: metrics
>Reporter: Elliott Clark
>Priority: Critical
> Attachments: HBASE-14282.patch
>
>
> Metrics2 has a whole bunch of race conditions and weird edges because of all 
> of the caching that metrics2 does.
> Additionally tying something so integral to hadoop who has lots of versions 
> we support has been a maintenance nightmare. It would allow us to completely 
> get rid of the compat modules.
> Rip it out.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17075) Old regionStates are not clearing for truncated tables

2016-11-13 Thread Allan Yang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-17075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662487#comment-15662487
 ] 

Allan Yang commented on HBASE-17075:


this issues should have been fixed in HBASE-16649

> Old regionStates are not clearing for truncated tables 
> ---
>
> Key: HBASE-17075
> URL: https://issues.apache.org/jira/browse/HBASE-17075
> Project: HBase
>  Issue Type: Bug
>  Components: Region Assignment
>Affects Versions: 1.1.8
>Reporter: Y. SREENIVASULU REDDY
>Priority: Minor
> Fix For: 1.1.8
>
> Attachments: HBASE-17075-branch-1.1.patch, HBASE-17075.001.patch
>
>
> For truncated tables, not clearing the region states from master in-memory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15199) Move jruby jar so only on hbase-shell module classpath; currently globally available

2016-11-13 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662480#comment-15662480
 ] 

stack commented on HBASE-15199:
---

Do it for 2.0.0 only.

> Move jruby jar so only on hbase-shell module classpath; currently globally 
> available
> 
>
> Key: HBASE-15199
> URL: https://issues.apache.org/jira/browse/HBASE-15199
> Project: HBase
>  Issue Type: Task
>  Components: dependencies, jruby, shell
>Reporter: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 15199.txt
>
>
> A suggestion that came up out of internal issue (filed by Mr Jan Van Besien) 
> was to move the scope of the jruby include down so it is only a dependency 
> for the hbase-shell. jruby jar brings in a bunch of dependencies (joda time 
> for example) which can clash with the includes of others. Our Sean suggests 
> that could be good to shut down exploit possibilities if jruby was not 
> globally available. Only downside I can think is that it may no longer be 
> available to our bin/*rb scripts if we move the jar but perhaps these can be 
> changed so they can find the ruby jar in new location.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15199) Move jruby jar so only on hbase-shell module classpath; currently globally available

2016-11-13 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15199:
--
Fix Version/s: (was: 1.4.0)

> Move jruby jar so only on hbase-shell module classpath; currently globally 
> available
> 
>
> Key: HBASE-15199
> URL: https://issues.apache.org/jira/browse/HBASE-15199
> Project: HBase
>  Issue Type: Task
>  Components: dependencies, jruby, shell
>Reporter: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 15199.txt
>
>
> A suggestion that came up out of internal issue (filed by Mr Jan Van Besien) 
> was to move the scope of the jruby include down so it is only a dependency 
> for the hbase-shell. jruby jar brings in a bunch of dependencies (joda time 
> for example) which can clash with the includes of others. Our Sean suggests 
> that could be good to shut down exploit possibilities if jruby was not 
> globally available. Only downside I can think is that it may no longer be 
> available to our bin/*rb scripts if we move the jar but perhaps these can be 
> changed so they can find the ruby jar in new location.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15199) Move jruby jar so only on hbase-shell module classpath; currently globally available

2016-11-13 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15199:
--
Status: Patch Available  (was: Open)

> Move jruby jar so only on hbase-shell module classpath; currently globally 
> available
> 
>
> Key: HBASE-15199
> URL: https://issues.apache.org/jira/browse/HBASE-15199
> Project: HBase
>  Issue Type: Task
>  Components: dependencies, jruby, shell
>Reporter: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 15199.txt
>
>
> A suggestion that came up out of internal issue (filed by Mr Jan Van Besien) 
> was to move the scope of the jruby include down so it is only a dependency 
> for the hbase-shell. jruby jar brings in a bunch of dependencies (joda time 
> for example) which can clash with the includes of others. Our Sean suggests 
> that could be good to shut down exploit possibilities if jruby was not 
> globally available. Only downside I can think is that it may no longer be 
> available to our bin/*rb scripts if we move the jar but perhaps these can be 
> changed so they can find the ruby jar in new location.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15199) Move jruby jar so only on hbase-shell module classpath; currently globally available

2016-11-13 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15199:
--
Attachment: 15199.txt

Small change.

> Move jruby jar so only on hbase-shell module classpath; currently globally 
> available
> 
>
> Key: HBASE-15199
> URL: https://issues.apache.org/jira/browse/HBASE-15199
> Project: HBase
>  Issue Type: Task
>  Components: dependencies, jruby, shell
>Reporter: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 15199.txt
>
>
> A suggestion that came up out of internal issue (filed by Mr Jan Van Besien) 
> was to move the scope of the jruby include down so it is only a dependency 
> for the hbase-shell. jruby jar brings in a bunch of dependencies (joda time 
> for example) which can clash with the includes of others. Our Sean suggests 
> that could be good to shut down exploit possibilities if jruby was not 
> globally available. Only downside I can think is that it may no longer be 
> available to our bin/*rb scripts if we move the jar but perhaps these can be 
> changed so they can find the ruby jar in new location.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12907) Test patch should look for undeclared dependencies

2016-11-13 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-12907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-12907:
--
Fix Version/s: (was: 2.0.0)

> Test patch should look for undeclared dependencies
> --
>
> Key: HBASE-12907
> URL: https://issues.apache.org/jira/browse/HBASE-12907
> Project: HBase
>  Issue Type: Improvement
>  Components: build
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>
> Given the breadth of used undeclared dependencies in HBASE-12898, we should 
> expand the QA patch tester to look for them as they show up.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12907) Test patch should look for undeclared dependencies

2016-11-13 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-12907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662473#comment-15662473
 ] 

stack commented on HBASE-12907:
---

Removing specific version on a tooling improvement.

> Test patch should look for undeclared dependencies
> --
>
> Key: HBASE-12907
> URL: https://issues.apache.org/jira/browse/HBASE-12907
> Project: HBase
>  Issue Type: Improvement
>  Components: build
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>
> Given the breadth of used undeclared dependencies in HBASE-12898, we should 
> expand the QA patch tester to look for them as they show up.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14583) Enabled client-side metrics by default

2016-11-13 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-14583:
--
Priority: Critical  (was: Minor)

> Enabled client-side metrics by default
> --
>
> Key: HBASE-14583
> URL: https://issues.apache.org/jira/browse/HBASE-14583
> Project: HBase
>  Issue Type: Task
>  Components: Client, Operability, Performance
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Critical
> Fix For: 2.0.0, 1.3.1
>
> Attachments: 14583.00.branch-1.patch, 14583.00.patch
>
>
> Enabling this feature by default for master and branch-1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-16332) update LMAX disruptor to latest

2016-11-13 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-16332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-16332.
---
Resolution: Implemented

This commit moved us to 3.3.6 as a side-effect:

commit 3b629d632ae660b618422b3e2f67533a6fdc7106
Author: zhangduo 
Date:   Wed Nov 9 09:24:12 2016 +0800

HBASE-17021 Use RingBuffer to reduce the contention in AsyncFSWAL

Resolving as 'implemented'

> update LMAX disruptor to latest
> ---
>
> Key: HBASE-16332
> URL: https://issues.apache.org/jira/browse/HBASE-16332
> Project: HBase
>  Issue Type: Task
>  Components: dependencies
>Reporter: Sean Busbey
>Priority: Minor
> Fix For: 2.0.0
>
>
> our version of the LMAX disruptor library is ~2 years old. update to 3.3.5+
> https://github.com/LMAX-Exchange/disruptor/releases



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-2484) refactor package names from o.a.h.h to o.a.hbase

2016-11-13 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-2484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-2484:
-
Fix Version/s: (was: 2.0.0)
   3.0.0

> refactor package names from o.a.h.h to o.a.hbase
> 
>
> Key: HBASE-2484
> URL: https://issues.apache.org/jira/browse/HBASE-2484
> Project: HBase
>  Issue Type: Task
>Reporter: Karthik K
> Fix For: 3.0.0
>
>
> After becoming a TLP, it makes sense to refactor to o.a.hbase instead of 
> o.a.h.hbase as it exists now. 
> There is a consensus among the team, but concerns about the migration effects 
> on the end-user remain.  placeholder ticket for the refactoring + opinions on 
> the migration cost of the API. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12341) Overhaul logging; log4j2, machine-readable, etc.

2016-11-13 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-12341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-12341:
--
Priority: Critical  (was: Major)

> Overhaul logging; log4j2, machine-readable, etc.
> 
>
> Key: HBASE-12341
> URL: https://issues.apache.org/jira/browse/HBASE-12341
> Project: HBase
>  Issue Type: Umbrella
>Reporter: stack
>Priority: Critical
> Fix For: 2.0.0
>
>
> This is a general umbrella issue for 2.x logging improvements. Hang related 
> work off this one.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17084) Delete HMerge (dead code)

2016-11-13 Thread Appy (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-17084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662353#comment-15662353
 ] 

Appy commented on HBASE-17084:
--

That's probably 
[Merge|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/util/Merge.java]
 since it extends Tool.
It seems weird to me too, and am not confident of this cleanup. But then, we 
might wonder the same thing again a year later.

> Delete HMerge (dead code)
> -
>
> Key: HBASE-17084
> URL: https://issues.apache.org/jira/browse/HBASE-17084
> Project: HBase
>  Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
>Priority: Trivial
> Attachments: HBASE-17084.master.001.patch
>
>
> HMerge isn't used anywhere. Can we delete it?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17084) Delete HMerge (dead code)

2016-11-13 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-17084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662316#comment-15662316
 ] 

stack commented on HBASE-17084:
---

Is this not how we do merges? Or we have another tool to do that?
Thanks for trying to do a bit of cleanup [~appy]


> Delete HMerge (dead code)
> -
>
> Key: HBASE-17084
> URL: https://issues.apache.org/jira/browse/HBASE-17084
> Project: HBase
>  Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
>Priority: Trivial
> Attachments: HBASE-17084.master.001.patch
>
>
> HMerge isn't used anywhere. Can we delete it?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17046) Add 1.1 doc to hbase.apache.org

2016-11-13 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-17046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662305#comment-15662305
 ] 

stack commented on HBASE-17046:
---

You could update it but could also just leave it as is. Its 1.1.7 so unlikely 
to change. Thats how we treat the 0.94. It didn't get updated with each new 
release. Thanks [~ndimiduk]

> Add 1.1 doc to hbase.apache.org
> ---
>
> Key: HBASE-17046
> URL: https://issues.apache.org/jira/browse/HBASE-17046
> Project: HBase
>  Issue Type: Task
>  Components: site
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
>
> Add link to 1.1 doc to the site.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16417) In-Memory MemStore Policy for Flattening and Compactions

2016-11-13 Thread Eshcar Hillel (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-16417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662097#comment-15662097
 ] 

Eshcar Hillel commented on HBASE-16417:
---

bq. then also even if we read all qualifiers from memstore itself, how we know 
there are no other versions of this in other HFiles.

I think I understand. So it is not enough just to check after the first 
(memstores scan) round that the result is not empty -- we need to go on and 
check that we retrieved data for all qualifiers in the get query, and if not do 
the second round which seeks the other Hfile. any problem with this?

bq. U can see there is an InternalScan extension for Scan where one can say use 
memstore only or not.

I am not familiar with this internal scanner. Where/when is it being used? 

bq. So this can be done with certain hard limitations. Only version should be 
present and/or ts on puts are always increasing.

Yes, if the application manipulates ts so that it is not always increasing you 
have a point. Is there a way to know this for sure? I don't think so.

Bottom line, [~anoop.hbase] you raise some valid concerns. So it might be that 
we cannot apply this optimization for all cases, however I am confident that 
99.99% of the application can benefit from such optimization, it is highly 
unreasonable not to apply it just to allow corner cases like manipulating 
timestamps.
Could we have the application "announce" that it is manipulating ts so then we 
avoid this optimization but apply it in all other common cases (?)

> In-Memory MemStore Policy for Flattening and Compactions
> 
>
> Key: HBASE-16417
> URL: https://issues.apache.org/jira/browse/HBASE-16417
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Eshcar Hillel
> Fix For: 2.0.0
>
> Attachments: HBASE-16417-benchmarkresults-20161101.pdf, 
> HBASE-16417-benchmarkresults-20161110.pdf
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16417) In-Memory MemStore Policy for Flattening and Compactions

2016-11-13 Thread Edward Bortnikov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-16417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15661998#comment-15661998
 ] 

Edward Bortnikov commented on HBASE-16417:
--

[~anoop.hbase], we certainly appreciate the input, feel free to fire the first 
thoughts going fwd (smile). Yes, we thought about the multi-cf case. We are 
speaking of single-row get only. The idea was trying to fetch from the set of 
the memstore scanners first. If the data can be retrieved, no need to go look 
in HFiles - isn't it? Am I missing something here? 

> In-Memory MemStore Policy for Flattening and Compactions
> 
>
> Key: HBASE-16417
> URL: https://issues.apache.org/jira/browse/HBASE-16417
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Eshcar Hillel
> Fix For: 2.0.0
>
> Attachments: HBASE-16417-benchmarkresults-20161101.pdf, 
> HBASE-16417-benchmarkresults-20161110.pdf
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16417) In-Memory MemStore Policy for Flattening and Compactions

2016-11-13 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-16417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15661784#comment-15661784
 ] 

Anoop Sam John commented on HBASE-16417:


bq.4. Change read (get) implementation to first seek for the key in memstore(s) 
only, and only if no matching entry is found seek in all memstore segments and 
all relevant store files. This could be a subject of another Jira. We believe 
this would be beneficial also with no compaction, and even more when 
index-/data-compaction is employed. Any thought on this direction
 
There are few issues for this. When we do a get for a row, memstore+HFiles read 
happens per cf wise. But how we can know all possible qualifiers in that. My Q 
is when we able to find an entry for the rk in memstore, how we can be sure 
that there are no other entries for this rk in HFiles? So suppose there are 
qualifiers also mentioned in the Get, (Get#addColumn(byte [] family, byte [] 
qualifier)) then also even if we read all qualifiers from memstore itself, how 
we know there are no other versions of this in other HFiles. U can see there is 
an InternalScan extension for Scan where one can say use memstore only or not.  
But am getting what u r saying is bit diff.   So this can be done with certain 
hard limitations. Only version should be present and/or ts on puts are always 
increasing. Gets are issues with columns and/or while doing writes all columns 
of a CF are put together.  Just noting down whatever comes at first thought.

> In-Memory MemStore Policy for Flattening and Compactions
> 
>
> Key: HBASE-16417
> URL: https://issues.apache.org/jira/browse/HBASE-16417
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Eshcar Hillel
> Fix For: 2.0.0
>
> Attachments: HBASE-16417-benchmarkresults-20161101.pdf, 
> HBASE-16417-benchmarkresults-20161110.pdf
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17081) Flush the entire CompactingMemStore content to disk

2016-11-13 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-17081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15661765#comment-15661765
 ] 

Anoop Sam John commented on HBASE-17081:


+1 for the task

> Flush the entire CompactingMemStore content to disk
> ---
>
> Key: HBASE-17081
> URL: https://issues.apache.org/jira/browse/HBASE-17081
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Anastasia Braginsky
>
> Part of CompactingMemStore's memory is held by an active segment, and another 
> part is divided between immutable segments in the compacting pipeline. Upon 
> flush-to-disk request we want to flush all of it to disk, in contrast to 
> flushing only tail of the compacting pipeline.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync

2016-11-13 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15661767#comment-15661767
 ] 

ramkrishna.s.vasudevan commented on HBASE-17085:


Thanks for the patch. I don't have system with me. I have seen this issue and 
was working on a fix. But first tried to do some workaround which did not pay 
off . My other doubt was if this happens then we will have duplicate entries or 
is it just empty packets. I was verifying that and as it was week end could not 
do that

> AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
> 
>
> Key: HBASE-17085
> URL: https://issues.apache.org/jira/browse/HBASE-17085
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17085.patch
>
>
> The problem is in appendAndSync method, we will issue an  AsyncDFSOutput.sync 
> if syncFutures is not empty. The SyncFutures in syncFutures can only be 
> removed after an AsyncDFSOutput.sync comes back, so before the 
> AsyncDFSOutput.sync actually returns, we will always issue an  
> AsyncDFSOutput.sync after an append even if there is no new sync request.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync

2016-11-13 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15661756#comment-15661756
 ] 

Hadoop QA commented on HBASE-17085:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
1s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
28m 43s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 80m 34s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
13s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 121m 47s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.mapred.TestTableMapReduceUtil 
|
|   | org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor |
|   | org.apache.hadoop.hbase.master.TestWarmupRegion |
|   | org.apache.hadoop.hbase.mapred.TestMultiTableSnapshotInputFormat |
|   | org.apache.hadoop.hbase.TestPartialResultsFromClientSide |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:7bda515 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12838688/HBASE-17085.patch |
| JIRA Issue | HBASE-17085 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux a6138a0d1f67 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 3f919dd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4450/artifact/patchprocess/patch-unit-hbase-server.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-HBASE-Build/4450/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4450/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |

[jira] [Updated] (HBASE-17077) Don't copy the replication queue belonging to the peer which has been deleted

2016-11-13 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-17077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-17077:
---
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 1.4.0
   2.0.0
   Status: Resolved  (was: Patch Available)

Thanks for the patch, Guanghao

> Don't copy the replication queue belonging to the peer which has been deleted
> -
>
> Key: HBASE-17077
> URL: https://issues.apache.org/jira/browse/HBASE-17077
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17077-v1.patch, HBASE-17077.patch
>
>
> When a region server is dead, then other live region servers will transfer 
> the dead rs's replication queue to their own queue. Now the live rs first 
> copy the wals queue to its own znode, then create a new replication source to 
> replicate the wals. But if the queue belong to a peer has been deleted, it 
> copy the queue, too. The current steps is:
> 1. copy the queue to its own znode
> 2. found the queue belong to a peer has been deleted
> 3. remove the queue and don't create a new replication source for it
> There is a small improvement. The live region server doesn't need to copy the 
> queue to its own znode. The new steps is:
> 1. found the queue belong to a peer has been deleted
> 2. remove the queue directly instead of copy it



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17077) Don't copy the replication queue belonging to the peer which has been deleted

2016-11-13 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-17077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-17077:
---
Summary: Don't copy the replication queue belonging to the peer which has 
been deleted  (was: Don't copy the replication queue which belong to the peer 
has been deleted)

> Don't copy the replication queue belonging to the peer which has been deleted
> -
>
> Key: HBASE-17077
> URL: https://issues.apache.org/jira/browse/HBASE-17077
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-17077-v1.patch, HBASE-17077.patch
>
>
> When a region server is dead, then other live region servers will transfer 
> the dead rs's replication queue to their own queue. Now the live rs first 
> copy the wals queue to its own znode, then create a new replication source to 
> replicate the wals. But if the queue belong to a peer has been deleted, it 
> copy the queue, too. The current steps is:
> 1. copy the queue to its own znode
> 2. found the queue belong to a peer has been deleted
> 3. remove the queue and don't create a new replication source for it
> There is a small improvement. The live region server doesn't need to copy the 
> queue to its own znode. The new steps is:
> 1. found the queue belong to a peer has been deleted
> 2. remove the queue directly instead of copy it



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17049) Find out why AsyncFSWAL issues much more syncs than FSHLog

2016-11-13 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15661591#comment-15661591
 ] 

Duo Zhang commented on HBASE-17049:
---

HBASE-17085 fix a problem which may cause unnecessary DFSOutput.sync. Will test 
the performance with the patch in place.

> Find out why AsyncFSWAL issues much more syncs than FSHLog
> --
>
> Key: HBASE-17049
> URL: https://issues.apache.org/jira/browse/HBASE-17049
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
> Fix For: 2.0.0
>
>
> https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync

2016-11-13 Thread Duo Zhang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang updated HBASE-17085:
--
Status: Patch Available  (was: Open)

> AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
> 
>
> Key: HBASE-17085
> URL: https://issues.apache.org/jira/browse/HBASE-17085
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17085.patch
>
>
> The problem is in appendAndSync method, we will issue an  AsyncDFSOutput.sync 
> if syncFutures is not empty. The SyncFutures in syncFutures can only be 
> removed after an AsyncDFSOutput.sync comes back, so before the 
> AsyncDFSOutput.sync actually returns, we will always issue an  
> AsyncDFSOutput.sync after an append even if there is no new sync request.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync

2016-11-13 Thread Duo Zhang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang updated HBASE-17085:
--
Attachment: HBASE-17085.patch

> AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
> 
>
> Key: HBASE-17085
> URL: https://issues.apache.org/jira/browse/HBASE-17085
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17085.patch
>
>
> The problem is in appendAndSync method, we will issue an  AsyncDFSOutput.sync 
> if syncFutures is not empty. The SyncFutures in syncFutures can only be 
> removed after an AsyncDFSOutput.sync comes back, so before the 
> AsyncDFSOutput.sync actually returns, we will always issue an  
> AsyncDFSOutput.sync after an append even if there is no new sync request.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync

2016-11-13 Thread Duo Zhang (JIRA)

Duo Zhang created HBASE-17085:
-

 Summary: AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
 Key: HBASE-17085
 URL: https://issues.apache.org/jira/browse/HBASE-17085
 Project: HBase
  Issue Type: Sub-task
  Components: wal
Affects Versions: 2.0.0
Reporter: Duo Zhang
Assignee: Duo Zhang
 Fix For: 2.0.0


The problem is in appendAndSync method, we will issue an  AsyncDFSOutput.sync 
if syncFutures is not empty. The SyncFutures in syncFutures can only be removed 
after an AsyncDFSOutput.sync comes back, so before the AsyncDFSOutput.sync 
actually returns, we will always issue an  AsyncDFSOutput.sync after an append 
even if there is no new sync request.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17077) Don't copy the replication queue which belong to the peer has been deleted

2016-11-13 Thread Guanghao Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-17077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15661409#comment-15661409
 ] 

Guanghao Zhang commented on HBASE-17077:


The timeout uts passed locally.

> Don't copy the replication queue which belong to the peer has been deleted
> --
>
> Key: HBASE-17077
> URL: https://issues.apache.org/jira/browse/HBASE-17077
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-17077-v1.patch, HBASE-17077.patch
>
>
> When a region server is dead, then other live region servers will transfer 
> the dead rs's replication queue to their own queue. Now the live rs first 
> copy the wals queue to its own znode, then create a new replication source to 
> replicate the wals. But if the queue belong to a peer has been deleted, it 
> copy the queue, too. The current steps is:
> 1. copy the queue to its own znode
> 2. found the queue belong to a peer has been deleted
> 3. remove the queue and don't create a new replication source for it
> There is a small improvement. The live region server doesn't need to copy the 
> queue to its own znode. The new steps is:
> 1. found the queue belong to a peer has been deleted
> 2. remove the queue directly instead of copy it



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

57 matches

Mail list logo