[jira] [Updated] (HBASE-16414) Improve performance for RPC encryption with Apache Common Crypto
[ https://issues.apache.org/jira/browse/HBASE-16414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Ma updated HBASE-16414: - Status: Patch Available (was: Open) > Improve performance for RPC encryption with Apache Common Crypto > > > Key: HBASE-16414 > URL: https://issues.apache.org/jira/browse/HBASE-16414 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 2.0.0 >Reporter: Colin Ma >Assignee: Colin Ma > Attachments: HBASE-16414.001.patch, HbaseRpcEncryptionWithCrypoto.docx > > > Hbase RPC encryption is enabled by setting “hbase.rpc.protection” to > "privacy". With the token authentication, it utilized DIGEST-MD5 mechanisms > for secure authentication and data protection. For DIGEST-MD5, it uses DES, > 3DES or RC4 to do encryption and it is very slow, especially for Scan. This > will become the bottleneck of the RPC throughput. > Apache Commons Crypto is a cryptographic library optimized with AES-NI. It > provides Java API for both cipher level and Java stream level. Developers can > use it to implement high performance AES encryption/decryption with the > minimum code and effort. Compare with the current implementation of > org.apache.hadoop.hbase.io.crypto.aes.AES, Crypto supports both JCE Cipher > and OpenSSL Cipher which is better performance than JCE Cipher. User can > configure the cipher type and the default is JCE Cipher. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16414) Improve performance for RPC encryption with Apache Common Crypto
[ https://issues.apache.org/jira/browse/HBASE-16414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Ma updated HBASE-16414: - Attachment: HBASE-16414.001.patch Upload the initial patch and add the link for review board. > Improve performance for RPC encryption with Apache Common Crypto > > > Key: HBASE-16414 > URL: https://issues.apache.org/jira/browse/HBASE-16414 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 2.0.0 >Reporter: Colin Ma >Assignee: Colin Ma > Attachments: HBASE-16414.001.patch, HbaseRpcEncryptionWithCrypoto.docx > > > Hbase RPC encryption is enabled by setting “hbase.rpc.protection” to > "privacy". With the token authentication, it utilized DIGEST-MD5 mechanisms > for secure authentication and data protection. For DIGEST-MD5, it uses DES, > 3DES or RC4 to do encryption and it is very slow, especially for Scan. This > will become the bottleneck of the RPC throughput. > Apache Commons Crypto is a cryptographic library optimized with AES-NI. It > provides Java API for both cipher level and Java stream level. Developers can > use it to implement high performance AES encryption/decryption with the > minimum code and effort. Compare with the current implementation of > org.apache.hadoop.hbase.io.crypto.aes.AES, Crypto supports both JCE Cipher > and OpenSSL Cipher which is better performance than JCE Cipher. User can > configure the cipher type and the default is JCE Cipher. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16414) Improve performance for RPC encryption with Apache Common Crypto
[ https://issues.apache.org/jira/browse/HBASE-16414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Ma updated HBASE-16414: - Attachment: HbaseRpcEncryptionWithCrypoto.docx > Improve performance for RPC encryption with Apache Common Crypto > > > Key: HBASE-16414 > URL: https://issues.apache.org/jira/browse/HBASE-16414 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 2.0.0 >Reporter: Colin Ma >Assignee: Colin Ma > Attachments: HbaseRpcEncryptionWithCrypoto.docx > > > Hbase RPC encryption is enabled by setting “hbase.rpc.protection” to > "privacy". With the token authentication, it utilized DIGEST-MD5 mechanisms > for secure authentication and data protection. For DIGEST-MD5, it uses DES, > 3DES or RC4 to do encryption and it is very slow, especially for Scan. This > will become the bottleneck of the RPC throughput. > Apache Commons Crypto is a cryptographic library optimized with AES-NI. It > provides Java API for both cipher level and Java stream level. Developers can > use it to implement high performance AES encryption/decryption with the > minimum code and effort. Compare with the current implementation of > org.apache.hadoop.hbase.io.crypto.aes.AES, Crypto supports both JCE Cipher > and OpenSSL Cipher which is better performance than JCE Cipher. User can > configure the cipher type and the default is JCE Cipher. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16414) Improve performance for RPC encryption with Apache Common Crypto
Colin Ma created HBASE-16414: Summary: Improve performance for RPC encryption with Apache Common Crypto Key: HBASE-16414 URL: https://issues.apache.org/jira/browse/HBASE-16414 Project: HBase Issue Type: Improvement Components: IPC/RPC Affects Versions: 2.0.0 Reporter: Colin Ma Assignee: Colin Ma Hbase RPC encryption is enabled by setting “hbase.rpc.protection” to "privacy". With the token authentication, it utilized DIGEST-MD5 mechanisms for secure authentication and data protection. For DIGEST-MD5, it uses DES, 3DES or RC4 to do encryption and it is very slow, especially for Scan. This will become the bottleneck of the RPC throughput. Apache Commons Crypto is a cryptographic library optimized with AES-NI. It provides Java API for both cipher level and Java stream level. Developers can use it to implement high performance AES encryption/decryption with the minimum code and effort. Compare with the current implementation of org.apache.hadoop.hbase.io.crypto.aes.AES, Crypto supports both JCE Cipher and OpenSSL Cipher which is better performance than JCE Cipher. User can configure the cipher type and the default is JCE Cipher. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-15982) Interface ReplicationEndpoint extends Guava's Service
[ https://issues.apache.org/jira/browse/HBASE-15982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15420656#comment-15420656 ] Andrew Purtell edited comment on HBASE-15982 at 8/15/16 6:32 AM: - Here's a new wrinkle. Now that we've banned jsr305, some versions of Guava cannot be used because they cause a compilation failure at ReplicationEndpoint: {noformat} [ERROR] .m2/repository/com/google/guava/guava/14.0.1/guava-14.0.1.jar (com/google/common/util/concurrent/AbstractService.class): warning: Cannot find annotation method 'value()' in type 'GuardedBy': class file for javax.annotation.concurrent.GuardedBy not found {noformat} was (Author: apurtell): Here's a new wrinkle. Now that we've banned jsr305-findbugs, some versions of Guava cannot be used because they cause a compilation failure at ReplicationEndpoint: {noformat} [ERROR] .m2/repository/com/google/guava/guava/14.0.1/guava-14.0.1.jar (com/google/common/util/concurrent/AbstractService.class): warning: Cannot find annotation method 'value()' in type 'GuardedBy': class file for javax.annotation.concurrent.GuardedBy not found {noformat} > Interface ReplicationEndpoint extends Guava's Service > - > > Key: HBASE-15982 > URL: https://issues.apache.org/jira/browse/HBASE-15982 > Project: HBase > Issue Type: Bug >Reporter: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 0.98.22 > > > We have Guava's Service leaking into the LimitedPrivate interface > ReplicationEndpoint: > {code} > public interface ReplicationEndpoint extends Service, > ReplicationPeerConfigListener > {code} > This required a private patch when I updated Guava for our internal > deployments. This is going to be a problem for us for long term maintenance > and implenters of pluggable replication endpoints. LP is only less than > public by a degree. We shouldn't leak types from third part code into either > Public or LP APIs in my opinion. Let's fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15982) Interface ReplicationEndpoint extends Guava's Service
[ https://issues.apache.org/jira/browse/HBASE-15982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15420658#comment-15420658 ] Andrew Purtell commented on HBASE-15982: bq. I still want to use Guava as our long term Service interface, because the interface it pretty well defined. However, maybe we can wrap it so that RE is not exposed to it directly. It might be difficult indeed to use Guava as the basis for our Service interface if it embeds a requirement for a Category X dependency. We harmonize on Guava 14 for internal builds after finding something that works across HDFS, HBase, Hive, Spark, and others. It's easy to revert the jsr305-findbugs ban on an internal build but that won't be a workable solution upstream. > Interface ReplicationEndpoint extends Guava's Service > - > > Key: HBASE-15982 > URL: https://issues.apache.org/jira/browse/HBASE-15982 > Project: HBase > Issue Type: Bug >Reporter: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 0.98.22 > > > We have Guava's Service leaking into the LimitedPrivate interface > ReplicationEndpoint: > {code} > public interface ReplicationEndpoint extends Service, > ReplicationPeerConfigListener > {code} > This required a private patch when I updated Guava for our internal > deployments. This is going to be a problem for us for long term maintenance > and implenters of pluggable replication endpoints. LP is only less than > public by a degree. We shouldn't leak types from third part code into either > Public or LP APIs in my opinion. Let's fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15982) Interface ReplicationEndpoint extends Guava's Service
[ https://issues.apache.org/jira/browse/HBASE-15982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15420656#comment-15420656 ] Andrew Purtell commented on HBASE-15982: Here's a new wrinkle. Now that we've banned jsr305-findbugs, some versions of Guava cannot be used because they cause a compilation failure at ReplicationEndpoint: {noformat} [ERROR] .m2/repository/com/google/guava/guava/14.0.1/guava-14.0.1.jar (com/google/common/util/concurrent/AbstractService.class): warning: Cannot find annotation method 'value()' in type 'GuardedBy': class file for javax.annotation.concurrent.GuardedBy not found {noformat} > Interface ReplicationEndpoint extends Guava's Service > - > > Key: HBASE-15982 > URL: https://issues.apache.org/jira/browse/HBASE-15982 > Project: HBase > Issue Type: Bug >Reporter: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 0.98.22 > > > We have Guava's Service leaking into the LimitedPrivate interface > ReplicationEndpoint: > {code} > public interface ReplicationEndpoint extends Service, > ReplicationPeerConfigListener > {code} > This required a private patch when I updated Guava for our internal > deployments. This is going to be a problem for us for long term maintenance > and implenters of pluggable replication endpoints. LP is only less than > public by a degree. We shouldn't leak types from third part code into either > Public or LP APIs in my opinion. Let's fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16148) Hybrid Logical Clocks(placeholder for running tests)
[ https://issues.apache.org/jira/browse/HBASE-16148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15420622#comment-15420622 ] stack commented on HBASE-16148: --- Hey [~saitejar] Suggest you do up a nice commit message when you post a patch detailing whats in it. No harm in using the nice @appy tool ./dev-support/submit-patch.py It'll post for you to review board too and do the patch with format-patch so your nice writeup gets included. Just a suggestion boss. > Hybrid Logical Clocks(placeholder for running tests) > > > Key: HBASE-16148 > URL: https://issues.apache.org/jira/browse/HBASE-16148 > Project: HBase > Issue Type: Sub-task > Components: API >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Minor > Labels: test-patch > Attachments: HBASE-16148.master.001.patch, > HBASE-16148.master.6.patch, HBASE-16148.master.test.1.patch, > HBASE-16148.master.test.2.patch, HBASE-16148.master.test.3.patch, > HBASE-16148.master.test.4.patch, HBASE-16148.master.test.5.patch, > HLC.1.patch, HLC.2.patch, HLC.3.patch, HLC.4.patch, HLC.5.patch, HLC.6.patch, > HLC.patch > > > This JIRA is just a placeholder to test Hybrid Logical Clocks code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16413) Add apache-hbase.slack.com #users channel to ref guide
Dima Spivak created HBASE-16413: --- Summary: Add apache-hbase.slack.com #users channel to ref guide Key: HBASE-16413 URL: https://issues.apache.org/jira/browse/HBASE-16413 Project: HBase Issue Type: Task Components: documentation Reporter: Dima Spivak Priority: Minor We have a user channel on Slack at [http://apache-hbase.slack.com], but no reference to it in the [Getting Involved section|https://hbase.apache.org/book.html#getting.involved] of our ref guide. We should fix that and probably note that new users who want access and don't have an {{apache.org}} email address should request an invite by emailing [dev@|mailto:d...@hbase.apache.org]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16148) Hybrid Logical Clocks(placeholder for running tests)
[ https://issues.apache.org/jira/browse/HBASE-16148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15420555#comment-15420555 ] Hadoop QA commented on HBASE-16148: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 14 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 9s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s {color} | {color:green} master passed with JDK v1.8.0_101 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 10s {color} | {color:green} master passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 6s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 39s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 37s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s {color} | {color:green} master passed with JDK v1.8.0_101 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s {color} | {color:green} master passed with JDK v1.7.0_101 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 23s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s {color} | {color:green} the patch passed with JDK v1.8.0_101 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 59s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 10s {color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 36s {color} | {color:red} hbase-server-jdk1.7.0_101 with JDK v1.7.0_101 generated 2 new + 4 unchanged - 2 fixed = 6 total (was 6) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 7s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 40s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 27m 40s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} | | {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 42s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 17s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s {color} | {color:green} the patch passed with JDK v1.8.0_101 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s {color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 43s {color} | {color:green} hbase-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 59s {color} | {color:green} hbase-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 92m 22s {color} | {color:green} hbase-server in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 49s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 150m 21s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.11.2 Server=
[jira] [Updated] (HBASE-16411) BackupDistCp#execute() should throw exception if the distcp job fails
[ https://issues.apache.org/jira/browse/HBASE-16411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16411: --- Description: In some case, IncrementalTableBackupProcedure#incrementalCopy() wouldn't detect the failed job, leading to potential data loss. BackupDistCp#execute() should throw exception if the wrapped distcp job fails. This would allow IncrementalTableBackupProcedure#incrementalCopy() to properly respond to job failure. was: User may run hadoop release which doesn't have fix for HADOOP-13489. In that case, IncrementalTableBackupProcedure#incrementalCopy() wouldn't detect the failed job, leading to potential data loss. BackupDistCp#execute() should throw exception if the wrapped distcp job fails. This would allow IncrementalTableBackupProcedure#incrementalCopy() to properly respond to job failure. > BackupDistCp#execute() should throw exception if the distcp job fails > - > > Key: HBASE-16411 > URL: https://issues.apache.org/jira/browse/HBASE-16411 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Labels: backup > Attachments: 16411.v1.txt > > > In some case, IncrementalTableBackupProcedure#incrementalCopy() wouldn't > detect the failed job, leading to potential data loss. > BackupDistCp#execute() should throw exception if the wrapped distcp job > fails. This would allow IncrementalTableBackupProcedure#incrementalCopy() to > properly respond to job failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16148) Hybrid Logical Clocks(placeholder for running tests)
[ https://issues.apache.org/jira/browse/HBASE-16148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Teja Ranuva updated HBASE-16148: Status: Open (was: Patch Available) > Hybrid Logical Clocks(placeholder for running tests) > > > Key: HBASE-16148 > URL: https://issues.apache.org/jira/browse/HBASE-16148 > Project: HBase > Issue Type: Sub-task > Components: API >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Minor > Labels: test-patch > Attachments: HBASE-16148.master.001.patch, > HBASE-16148.master.6.patch, HBASE-16148.master.test.1.patch, > HBASE-16148.master.test.2.patch, HBASE-16148.master.test.3.patch, > HBASE-16148.master.test.4.patch, HBASE-16148.master.test.5.patch, > HLC.1.patch, HLC.2.patch, HLC.3.patch, HLC.4.patch, HLC.5.patch, HLC.6.patch, > HLC.patch > > > This JIRA is just a placeholder to test Hybrid Logical Clocks code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16148) Hybrid Logical Clocks(placeholder for running tests)
[ https://issues.apache.org/jira/browse/HBASE-16148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Teja Ranuva updated HBASE-16148: Attachment: HLC.6.patch > Hybrid Logical Clocks(placeholder for running tests) > > > Key: HBASE-16148 > URL: https://issues.apache.org/jira/browse/HBASE-16148 > Project: HBase > Issue Type: Sub-task > Components: API >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Minor > Labels: test-patch > Attachments: HBASE-16148.master.001.patch, > HBASE-16148.master.6.patch, HBASE-16148.master.test.1.patch, > HBASE-16148.master.test.2.patch, HBASE-16148.master.test.3.patch, > HBASE-16148.master.test.4.patch, HBASE-16148.master.test.5.patch, > HLC.1.patch, HLC.2.patch, HLC.3.patch, HLC.4.patch, HLC.5.patch, HLC.6.patch, > HLC.patch > > > This JIRA is just a placeholder to test Hybrid Logical Clocks code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16148) Hybrid Logical Clocks(placeholder for running tests)
[ https://issues.apache.org/jira/browse/HBASE-16148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Teja Ranuva updated HBASE-16148: Status: Patch Available (was: Open) > Hybrid Logical Clocks(placeholder for running tests) > > > Key: HBASE-16148 > URL: https://issues.apache.org/jira/browse/HBASE-16148 > Project: HBase > Issue Type: Sub-task > Components: API >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Minor > Labels: test-patch > Attachments: HBASE-16148.master.001.patch, > HBASE-16148.master.6.patch, HBASE-16148.master.test.1.patch, > HBASE-16148.master.test.2.patch, HBASE-16148.master.test.3.patch, > HBASE-16148.master.test.4.patch, HBASE-16148.master.test.5.patch, > HLC.1.patch, HLC.2.patch, HLC.3.patch, HLC.4.patch, HLC.5.patch, HLC.6.patch, > HLC.patch > > > This JIRA is just a placeholder to test Hybrid Logical Clocks code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14921) Memory optimizations
[ https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15420309#comment-15420309 ] Anastasia Braginsky commented on HBASE-14921: - bq. Makes sense. So were you able to observe the problems that we highlighted in the previous discussions - We got two problems one is the invalid negative size for memstore and the 'Too many open files'? Were you able to get this patch running with the PE tool and the configs that we had updated in the earlier discussions? We are yet unable to use the PE tool. Even on DefaultMemStore it doesn't work properly. We are currently running with YCSB, stressing the writes. We didn't get the exceptions that you mentioned, neither "negative size" nor "too many open files". However, we were able to see the problem of "too many open files" without getting the exception. From here, the quick fix of limiting the number of segments in the pipeline. This to be managed by a proper policy later. Later today I am going to publish the patch where we have instrumentation measuring the time it takes to do the speculative scan. Will you be able to give it a run on your PE tool? You can by the way take a look if you are still seeing the "negative size" exception. This one (if still exists) can later be investigated while dealing with the sizes issues. > Memory optimizations > > > Key: HBASE-14921 > URL: https://issues.apache.org/jira/browse/HBASE-14921 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Eshcar Hillel >Assignee: Anastasia Braginsky > Attachments: CellBlocksSegmentInMemStore.pdf, > CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, > HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, > HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, > HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, > HBASE-14921-V09-CAO.patch, InitialCellArrayMapEvaluation.pdf, > IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, > MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx > > > Memory optimizations including compressed format representation and offheap > allocations -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16304) regionserver should shutdown but it is blocked
[ https://issues.apache.org/jira/browse/HBASE-16304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15420308#comment-15420308 ] Hadoop QA commented on HBASE-16304: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s {color} | {color:blue} Docker mode activated. {color} | | {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 2s {color} | {color:blue} The patch file was not named according to hbase's naming conventions. Please see https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for instructions. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s {color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 16s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s {color} | {color:green} master passed with JDK v1.8.0_101 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s {color} | {color:green} master passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 51s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 17s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 56s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s {color} | {color:green} master passed with JDK v1.8.0_101 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s {color} | {color:green} master passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 46s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s {color} | {color:green} the patch passed with JDK v1.8.0_101 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s {color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 49s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 17s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s {color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 27m 12s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} | | {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 8s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s {color} | {color:green} the patch passed with JDK v1.8.0_101 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s {color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 91m 57s {color} | {color:green} hbase-server in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m 6s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 135m 40s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-08-14 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12823619/16304.v5.txt | | JIRA Issue | HBASE-16304 | | Optional Tests | asflicense javac javadoc unit findb
[jira] [Updated] (HBASE-16304) regionserver should shutdown but it is blocked
[ https://issues.apache.org/jira/browse/HBASE-16304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16304: --- Attachment: 16304.v5.txt > regionserver should shutdown but it is blocked > -- > > Key: HBASE-16304 > URL: https://issues.apache.org/jira/browse/HBASE-16304 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.2 >Reporter: mingmin xu >Assignee: Ted Yu >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: 16304.branch-1.2.v4.txt, 16304.branch-1.2.v5.txt, > 16304.branch-1.v1.txt, 16304.v1.txt, 16304.v3.txt, 16304.v4.txt, > 16304.v4.txt, 16304.v5.txt > > > here is my jvm stack: > {code} > 2016-07-29 16:36:56 > Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.72-b04 mixed mode): > "Timer for 'HBase' metrics system" daemon prio=10 tid=0x7f205cf38000 > nid=0xafa5 in Object.wait() [0x7f203b353000] >java.lang.Thread.State: TIMED_WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.util.TimerThread.mainLoop(Timer.java:552) > - locked <0x00063503c790> (a java.util.TaskQueue) > at java.util.TimerThread.run(Timer.java:505) > "Attach Listener" daemon prio=10 tid=0x7f205d017800 nid=0x1300 waiting on > condition [0x] >java.lang.Thread.State: RUNNABLE > "IPC Parameter Sending Thread #2" daemon prio=10 tid=0x7f205c7c4000 > nid=0x4f1a waiting on condition [0x7f20362e1000] >java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00066f996718> (a > java.util.concurrent.SynchronousQueue$TransferStack) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at > java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) > at > java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "RS_LOG_REPLAY_OPS-hadoop-datanode-0042:16020-1" prio=10 > tid=0x7f2054ec8000 nid=0x832d waiting on condition [0x7f2039a18000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00066ffb5950> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "RS_LOG_REPLAY_OPS-hadoop-datanode-0042:16020-0" prio=10 > tid=0x7f20542ca800 nid=0x5a5d waiting on condition [0x7f2033bba000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00066ffb5950> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "hadoop-datanode-0042.corp.cootek.com,16020,1469690065288_ChoreService_2" > daemon prio=10 tid=0x7f205d0d4000 nid=0x72af waiting on condition > [0x7f203b151000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00066fd70dd8> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.jav
[jira] [Updated] (HBASE-16304) regionserver should shutdown but it is blocked
[ https://issues.apache.org/jira/browse/HBASE-16304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16304: --- Attachment: 16304.branch-1.2.v5.txt > regionserver should shutdown but it is blocked > -- > > Key: HBASE-16304 > URL: https://issues.apache.org/jira/browse/HBASE-16304 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.2 >Reporter: mingmin xu >Assignee: Ted Yu >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: 16304.branch-1.2.v4.txt, 16304.branch-1.2.v5.txt, > 16304.branch-1.v1.txt, 16304.v1.txt, 16304.v3.txt, 16304.v4.txt, 16304.v4.txt > > > here is my jvm stack: > {code} > 2016-07-29 16:36:56 > Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.72-b04 mixed mode): > "Timer for 'HBase' metrics system" daemon prio=10 tid=0x7f205cf38000 > nid=0xafa5 in Object.wait() [0x7f203b353000] >java.lang.Thread.State: TIMED_WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.util.TimerThread.mainLoop(Timer.java:552) > - locked <0x00063503c790> (a java.util.TaskQueue) > at java.util.TimerThread.run(Timer.java:505) > "Attach Listener" daemon prio=10 tid=0x7f205d017800 nid=0x1300 waiting on > condition [0x] >java.lang.Thread.State: RUNNABLE > "IPC Parameter Sending Thread #2" daemon prio=10 tid=0x7f205c7c4000 > nid=0x4f1a waiting on condition [0x7f20362e1000] >java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00066f996718> (a > java.util.concurrent.SynchronousQueue$TransferStack) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at > java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) > at > java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "RS_LOG_REPLAY_OPS-hadoop-datanode-0042:16020-1" prio=10 > tid=0x7f2054ec8000 nid=0x832d waiting on condition [0x7f2039a18000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00066ffb5950> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "RS_LOG_REPLAY_OPS-hadoop-datanode-0042:16020-0" prio=10 > tid=0x7f20542ca800 nid=0x5a5d waiting on condition [0x7f2033bba000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00066ffb5950> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "hadoop-datanode-0042.corp.cootek.com,16020,1469690065288_ChoreService_2" > daemon prio=10 tid=0x7f205d0d4000 nid=0x72af waiting on condition > [0x7f203b151000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00066fd70dd8> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
[jira] [Commented] (HBASE-16341) Missing bit on "Regression: Random Read/WorkloadC slower in 1.x than 0.98"
[ https://issues.apache.org/jira/browse/HBASE-16341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15420260#comment-15420260 ] Hadoop QA commented on HBASE-16341: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m 13s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s {color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m 52s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s {color} | {color:green} master passed with JDK v1.8.0_101 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s {color} | {color:green} master passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 26s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 37s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 40s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s {color} | {color:green} master passed with JDK v1.8.0_101 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s {color} | {color:green} master passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 0s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s {color} | {color:green} the patch passed with JDK v1.8.0_101 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 48s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s {color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 48s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 0s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 23s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 38m 30s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} | | {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 22s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 54s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s {color} | {color:green} the patch passed with JDK v1.8.0_101 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s {color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 135m 2s {color} | {color:green} hbase-server in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 44s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 222m 55s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-08-14 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12823606/HBASE-16341.master.001.patch | | JIRA Issue | HBASE-16341 | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux a2cf8a683083 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/compo