[jira] [Commented] (HBASE-8625) Dependency version upgrade

[Nicolas Liochon (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13669041#comment-13669041
 ] 

Nicolas Liochon commented on HBASE-8625:


I can do that. I also plan to propose an upgrade to hdfs.


> Dependency version upgrade
> --
>
> Key: HBASE-8625
> URL: https://issues.apache.org/jira/browse/HBASE-8625
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.98.0, 0.95.0
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
>Priority: Minor
> Fix For: 0.98.0
>
> Attachments: 8625.v1.patch, 8625.v2.patch
>
>
> junit dependency should be scoped "test"
> We should use a newer versions of jaxb-api. One of our 3rd party would prefer 
> a newer one: 
>  javax.xml.bind:jaxb-api:jar:2.1:compile (version managed from 2.2.2)
>  Last is 2.2.4.
>  
> Not mandatory, but should be done:
>  guava 14.0.1
>  netty 3.6.6.Final
>  commons-codec.version 1.8
>  jackson.version 1.9.3

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8630) Share Socket Connections for different HConnectionImplementations

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13669036#comment-13669036
 ] 

Hadoop QA commented on HBASE-8630:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12585159/8630-trunk-v2.txt
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 
1.0 profile.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   
org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor
  org.apache.hadoop.hbase.master.TestTableLockManager

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5862//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5862//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5862//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5862//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5862//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5862//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5862//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5862//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5862//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5862//console

This message is automatically generated.

> Share Socket Connections for different HConnectionImplementations
> -
>
> Key: HBASE-8630
> URL: https://issues.apache.org/jira/browse/HBASE-8630
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Affects Versions: 0.94.3
>Reporter: cuijianwei
> Attachments: 8630-trunk-v1.txt, 8630-trunk-v2.txt
>
>
> In org.apache.hadoop.hbase.ipc.HBaseClient.java, socket connections are 
> pooled by map as:
> {code} protected final PoolMap connections; {code}
> The hashCode of ConnectionId is defined as:
> {code} public int hashCode() {
>   return (address.hashCode() + PRIME * (
>   PRIME * System.identityHashCode(protocol) ^
>  (ticket == null ? 0 : ticket.hashCode()) )) ^ rpcTimeout;
> } {code}
> As we can see, ticket.hashCode() will contribute to hashCode of ConnectionId. 
> For hbase without authentication, the ticket should be a HadoopUser; while 
> for hbase with authentication, the ticket should be a SecureHadoopUser. 
> Neither HadoopUser nor SecureHadoopUser override hashCode() method, 
> therefore, two tickets have the same hashCode only when they refer to the 
> same object.
>   On the other hand, when we use HTable to access hbase, firstly, we will 
> invoke HBaseRPC.waitForProxy(...) to create a proxy for region server as 
> follows:
> {code}  server = (HRegionInterface) HBaseRPC.waitForProxy(
>   serverInterfaceClass, HRegionInterface.VERSION,
>   address, this.conf,
>   this.maxRPCAttempts, this.rpcTimeout, this.rpcTimeout); 

[jira] [Updated] (HBASE-8639) very poor performance of htable.getscanner in multithread env.

[Raymond Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raymond Liu updated HBASE-8639:
---

Description: 
Hi, I am running a app on top of phoenix which will fork say around 100+ thread 
to call htable.getscanner(scan) to do parallel scan ( say each scan is actually 
targeting one Region), And each scan will only match a few result and return 
thus will be very fast.

under this case, I found that the htable.getscanner(scan) op itself runs pretty 
slow. by profiling with jvisualvm. I found 90% of app time is cost on 
org.apache.hadoop.net.DNS.getDefaultHost. Which been invoked by 
scannnercallable.checkIfRegionServerIsRemote. 

The root cause is that DNS.getDefaultHost involves synchronized methods in 
java.net.Inet4AddressImpl which have the 100+ thread to lock and wait upon each 
other. each call to DNS.getDefaultHost cost around 30ms, while in another case, 
I run single thread to call 100K times DNS.getDefaultHost , each cost leas than 
0.06ms.

By hacking the code and remove the call to checkIfRegionServerIsRemote, my app 
runs 5 times faster, say, 50K op in my app cost 200+ seconds instead of 1000+ 
seconds.

by check the code further, I found this checkIfRegionServerIsRemote seems just 
for use of metrics collection. ( or maybe retry logic?) I am wondering that 
could this been removed or switch to some other implementation? so that cases 
like mine which run large number of small scan with multi threads could 
performance way better?


  was:
Hi, I am running a app on top of phoenix which will fork say around 100+ thread 
to call htable.getscanner(scan) to do parallel scan ( say each scan is actually 
targeting one Region), And each scan will only match a few result and return 
thus will be very fast.

under this case, I found that the htable.getscanner(scan) op itself runs pretty 
slow. by profiling with jvisualvm. I found 90% of app time is cost on 
org.apache.hadoop.net.DNS.getDefaultHost. Which been invoked by 
scannnercallable.checkIfRegionServerIsRemote. 

The root cause is that DNS.getDefaultHost involves synchronized methods in 
java.net.Inet4AddressImpl which have the 100+ thread to lock and wait upon each 
other. each call to DNS.getDefaultHost cost around 30ms, while in another case, 
I run single thread to call 10K times DNS.getDefaultHost , each cost leas than 
0.06ms.

By hacking the code and remove the call to checkIfRegionServerIsRemote, my app 
runs 5 times faster, say, 50K op in my app cost 200+ seconds instead of 1000+ 
seconds.

by check the code further, I found this checkIfRegionServerIsRemote seems just 
for use of metrics collection. ( or maybe retry logic?) I am wondering that 
could this been removed or switch to some other implementation? so that cases 
like mine which run large number of small scan with multi threads could 
performance way better?



> very poor performance of htable.getscanner in multithread env.
> --
>
> Key: HBASE-8639
> URL: https://issues.apache.org/jira/browse/HBASE-8639
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.94.7
>Reporter: Raymond Liu
>
> Hi, I am running a app on top of phoenix which will fork say around 100+ 
> thread to call htable.getscanner(scan) to do parallel scan ( say each scan is 
> actually targeting one Region), And each scan will only match a few result 
> and return thus will be very fast.
> under this case, I found that the htable.getscanner(scan) op itself runs 
> pretty slow. by profiling with jvisualvm. I found 90% of app time is cost on 
> org.apache.hadoop.net.DNS.getDefaultHost. Which been invoked by 
> scannnercallable.checkIfRegionServerIsRemote. 
> The root cause is that DNS.getDefaultHost involves synchronized methods in 
> java.net.Inet4AddressImpl which have the 100+ thread to lock and wait upon 
> each other. each call to DNS.getDefaultHost cost around 30ms, while in 
> another case, I run single thread to call 100K times DNS.getDefaultHost , 
> each cost leas than 0.06ms.
> By hacking the code and remove the call to checkIfRegionServerIsRemote, my 
> app runs 5 times faster, say, 50K op in my app cost 200+ seconds instead of 
> 1000+ seconds.
> by check the code further, I found this checkIfRegionServerIsRemote seems 
> just for use of metrics collection. ( or maybe retry logic?) I am wondering 
> that could this been removed or switch to some other implementation? so that 
> cases like mine which run large number of small scan with multi threads could 
> performance way better?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8439) [replication] Remove ReplicationZookeeper class

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-8439:
--

Component/s: Replication

> [replication] Remove ReplicationZookeeper class
> ---
>
> Key: HBASE-8439
> URL: https://issues.apache.org/jira/browse/HBASE-8439
> Project: HBase
>  Issue Type: Improvement
>  Components: Replication
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Blocker
>
> Once all of the logic in ReplicationZookeeper has been refactored into three 
> interfaces (for status, queues, and peers), there is almost no logic in 
> ReplicationZookeeper. It can now be removed and classes that call it should 
> be refactored to call the state interfaces directly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8440) [replication] Rename ReplicationStateInterface and ReplicationStateImpl

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-8440:
--

Component/s: Replication

> [replication] Rename ReplicationStateInterface and ReplicationStateImpl
> ---
>
> Key: HBASE-8440
> URL: https://issues.apache.org/jira/browse/HBASE-8440
> Project: HBase
>  Issue Type: Improvement
>  Components: Replication
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Trivial
>
> As per Stack and J-D's comments:
> ReplicationStateInterface should drop the interface part and 
> ReplicationStateImpl should indicate that it is the Zookeeper implementation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-8639) very poor performance of htable.getscanner in multithread env.

[Raymond Liu (JIRA)]

Raymond Liu created HBASE-8639:
--

 Summary: very poor performance of htable.getscanner in multithread 
env.
 Key: HBASE-8639
 URL: https://issues.apache.org/jira/browse/HBASE-8639
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.7
Reporter: Raymond Liu


Hi, I am running a app on top of phoenix which will fork say around 100+ thread 
to call htable.getscanner(scan) to do parallel scan ( say each scan is actually 
targeting one Region), And each scan will only match a few result and return 
thus will be very fast.

under this case, I found that the htable.getscanner(scan) op itself runs pretty 
slow. by profiling with jvisualvm. I found 90% of app time is cost on 
org.apache.hadoop.net.DNS.getDefaultHost. Which been invoked by 
scannnercallable.checkIfRegionServerIsRemote. 

The root cause is that DNS.getDefaultHost involves synchronized methods in 
java.net.Inet4AddressImpl which have the 100+ thread to lock and wait upon each 
other. each call to DNS.getDefaultHost cost around 30ms, while in another case, 
I run single thread to call 10K times DNS.getDefaultHost , each cost leas than 
0.06ms.

By hacking the code and remove the call to checkIfRegionServerIsRemote, my app 
runs 5 times faster, say, 50K op in my app cost 200+ seconds instead of 1000+ 
seconds.

by check the code further, I found this checkIfRegionServerIsRemote seems just 
for use of metrics collection. ( or maybe retry logic?) I am wondering that 
could this been removed or switch to some other implementation? so that cases 
like mine which run large number of small scan with multi threads could 
performance way better?


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8609) Make the CopyTable support startRow, stopRow options

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13669011#comment-13669011
 ] 

Hudson commented on HBASE-8609:
---

Integrated in HBase-0.94-security #154 (See 
[https://builds.apache.org/job/HBase-0.94-security/154/])
HBASE-8609 Make the CopyTable support startRow, stopRow options (Shaohui) 
(Revision 1487228)

 Result = SUCCESS
tedyu : 
Files : 
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/CopyTable.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestCopyTable.java


> Make the CopyTable support startRow, stopRow options
> 
>
> Key: HBASE-8609
> URL: https://issues.apache.org/jira/browse/HBASE-8609
> Project: HBase
>  Issue Type: Improvement
>  Components: mapreduce
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: 0.98.0, 0.95.2, 0.94.9
>
> Attachments: 8609-trunk-v4.patch, HBASE-8609-0.94-v0.patch, 
> HBASE-8609-0.94-v1.patch, HBASE-8609-0.94-v2.patch, 
> HBASE-8609-trunk-v0.patch, HBASE-8609-trunk-v1.patch, 
> HBASE-8609-trunk-v2.patch, HBASE-8609-trunk-v3.patch
>
>
> Make the CopyTable support startRow, stopRow options.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8630) Share Socket Connections for different HConnectionImplementations

[cuijianwei (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

cuijianwei updated HBASE-8630:
--

Attachment: 8630-trunk-v2.txt

remove long line

> Share Socket Connections for different HConnectionImplementations
> -
>
> Key: HBASE-8630
> URL: https://issues.apache.org/jira/browse/HBASE-8630
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Affects Versions: 0.94.3
>Reporter: cuijianwei
> Attachments: 8630-trunk-v1.txt, 8630-trunk-v2.txt
>
>
> In org.apache.hadoop.hbase.ipc.HBaseClient.java, socket connections are 
> pooled by map as:
> {code} protected final PoolMap connections; {code}
> The hashCode of ConnectionId is defined as:
> {code} public int hashCode() {
>   return (address.hashCode() + PRIME * (
>   PRIME * System.identityHashCode(protocol) ^
>  (ticket == null ? 0 : ticket.hashCode()) )) ^ rpcTimeout;
> } {code}
> As we can see, ticket.hashCode() will contribute to hashCode of ConnectionId. 
> For hbase without authentication, the ticket should be a HadoopUser; while 
> for hbase with authentication, the ticket should be a SecureHadoopUser. 
> Neither HadoopUser nor SecureHadoopUser override hashCode() method, 
> therefore, two tickets have the same hashCode only when they refer to the 
> same object.
>   On the other hand, when we use HTable to access hbase, firstly, we will 
> invoke HBaseRPC.waitForProxy(...) to create a proxy for region server as 
> follows:
> {code}  server = (HRegionInterface) HBaseRPC.waitForProxy(
>   serverInterfaceClass, HRegionInterface.VERSION,
>   address, this.conf,
>   this.maxRPCAttempts, this.rpcTimeout, this.rpcTimeout); 
> {code}
> Then HBaseRpc.getProxy(...) will be called as follows:
> {code}public static VersionedProtocol getProxy(Class VersionedProtocol> protocol,
>   long clientVersion, InetSocketAddress addr, Configuration conf,
>   SocketFactory factory, int rpcTimeout) throws IOException {
> return getProxy(protocol, clientVersion, addr,
> User.getCurrent(), conf, factory, rpcTimeout);
>   } {code}
> We can see, User.getCurrent() will be invoked to generate the ticket to build 
> socket connection. User.getCurrent() is defined as:
> {code}
> public static User getCurrent() throws IOException {
> User user;
> if (IS_SECURE_HADOOP) {
>   user = new SecureHadoopUser();
> } else {
>   user = new HadoopUser();
> }
> if (user.getUGI() == null) {
>   return null;
> }
> return user;
>   }
> {code}
> Therefore, we will get different tickets when we create different proxies for 
> the same region server, so that these proxies can't share the created socket 
> connections and will create new socket connections even if they have the same 
> HBaseConfiguration.
> We can use the following case to validate the description above:
> {code}
> public static void main(String args[]) throws Exception {
> Configuration conf = HBaseConfiguration.create();
> for (int i = 0;; ++i) {
>   HTable table = new HTable(conf, TestTable.testTableName);
>   table.close();
> }
> }
> {code}
> Each time we close the HTable, the created region server proxies will be 
> closed as the underlying HConnectionImplementation will be closed. However, 
> the created socket connections won't be closed and wait to be shared in 
> future. Then, when we create HTable in the next turn, we will create server 
> proxy again, get a new ticket and consequently create new socket connections. 
> The created socket connections last turn can not be used any more. As the 
> loop goes on, thousands of socket will be created to connect region servers 
> until we get an exception to show no more sockets could be created.
> To fix the problem, maybe, we can use ticket.getName().hashCode() instead of 
> ticket.hashCode()?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8630) Share Socket Connections for different HConnectionImplementations

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13669005#comment-13669005
 ] 

Hadoop QA commented on HBASE-8630:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12585151/8630-trunk-v1.txt
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 
1.0 profile.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 lineLengths{color}.  The patch introduces lines longer than 
100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5861//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5861//console

This message is automatically generated.

> Share Socket Connections for different HConnectionImplementations
> -
>
> Key: HBASE-8630
> URL: https://issues.apache.org/jira/browse/HBASE-8630
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Affects Versions: 0.94.3
>Reporter: cuijianwei
> Attachments: 8630-trunk-v1.txt
>
>
> In org.apache.hadoop.hbase.ipc.HBaseClient.java, socket connections are 
> pooled by map as:
> {code} protected final PoolMap connections; {code}
> The hashCode of ConnectionId is defined as:
> {code} public int hashCode() {
>   return (address.hashCode() + PRIME * (
>   PRIME * System.identityHashCode(protocol) ^
>  (ticket == null ? 0 : ticket.hashCode()) )) ^ rpcTimeout;
> } {code}
> As we can see, ticket.hashCode() will contribute to hashCode of ConnectionId. 
> For hbase without authentication, the ticket should be a HadoopUser; while 
> for hbase with authentication, the ticket should be a SecureHadoopUser. 
> Neither HadoopUser nor SecureHadoopUser override hashCode() method, 
> therefore, two tickets have the same hashCode only when they refer to the 
> same object.
>   On the other hand, when we use HTable to access hbase, firstly, we will 
> invoke HBaseRPC.waitForProxy(...) to create a proxy for region server as 
> follows:
> {code}  server = (HRegionInterface) HBaseRPC.waitForProxy(
>   serverInterfaceClass, HRegionInterface.VERSION,
>   address, this.conf,
>   this.maxRPCAttempts, this.rpcTimeout, this.rpcTimeout); 
> {code}
> Then HBaseRpc.getProxy(...) will be called as follows:
> {code}public static VersionedProtocol getProxy(Class VersionedProtocol> protocol,
>   long clientVersion, InetSocketA

[jira] [Commented] (HBASE-8634) Fix potential null pointer dereference in HRegionServer and TableLockChecker

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668997#comment-13668997
 ] 

Hadoop QA commented on HBASE-8634:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12585152/8634-v2.txt
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 
1.0 profile.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5860//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5860//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5860//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5860//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5860//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5860//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5860//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5860//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5860//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5860//console

This message is automatically generated.

> Fix potential null pointer dereference in HRegionServer and TableLockChecker
> 
>
> Key: HBASE-8634
> URL: https://issues.apache.org/jira/browse/HBASE-8634
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: 8634-v1.txt, 8634-v2.txt
>
>
> From 
> https://builds.apache.org/job/PreCommit-HBASE-Build/5852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
>  :
> {code}
> Possible null pointer dereference of rpcc in 
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(RpcController, 
> ClientProtos$MultiRequest)
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.regionserver.HRegionServerIn 
> method 
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(RpcController, 
> ClientProtos$MultiRequest)Value loaded from rpccDereferenced at 
> HRegionServer.java:[line 3176]Known null at HRegionServer.java:[line 
> 3174]
> ...
> Possible null pointer dereference of rpcc in 
> org.apache.hadoop.hbase.regionserver.HRegionServer.mutate(RpcController, 
> ClientProtos$MutateRequest)
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.regionserver.HRegionServerIn 
> method 
> org.apache.hadoop.hbase.regionserver.HRegionServer.mutate(RpcController, 
> ClientProtos$MutateRequest)Value loaded from rpccDereferenced at 
> HRegionServer.java:[line 2832]Known null at HRegionServer.java:[line 
> 2830]
> ...
> Possible null pointer dereference of rpcc in 
> org.apache.hadoop.hbase.regionserver.HRegionServer.replay(RpcController, 
> ClientProtos$MultiRequest)
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 

[jira] [Commented] (HBASE-8626) RowMutations fail when Delete and Put on same columnFamily/column/row

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668996#comment-13668996
 ] 

Lars Hofhansl commented on HBASE-8626:
--

-1 on changing this. This is working as designed.

It does not make any logical sense to do an atomic Put and Delete to the *same* 
column. What does that even mean? You want both the Put and the Delete, but not 
really the Put, because the Delete should win.

Will close as "Won't fix" unless I hear objections.

> RowMutations fail when Delete and Put on same columnFamily/column/row
> -
>
> Key: HBASE-8626
> URL: https://issues.apache.org/jira/browse/HBASE-8626
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.94.7, 0.95.0
> Environment: Ubuntu 12.04, HBase 0.94.7
>Reporter: Vinod
>Assignee: Ted Yu
> Fix For: 0.94.7, 0.95.1
>
> Attachments: 8626-v1.txt, TestRowMutations.java, 
> tests_for_row_mutations1.patch
>
>
> When RowMutations have a Delete followed by Put to same column family or 
> columns or rows, only the Delete is happening while the Put is ignored so 
> atomicity of RowMutations is broken for such cases.
> Attached is a unit test where the following tests are failing:
> - testDeleteCFThenPutInSameCF: Delete a column family and then Put to same 
> column family.
> - testDeleteColumnThenPutSameColumn: Delete a column and then Put to same 
> column.
> - testDeleteRowThenPutSameRow: Delete a row and then Put to same row

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8609) Make the CopyTable support startRow, stopRow options

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668987#comment-13668987
 ] 

Hudson commented on HBASE-8609:
---

Integrated in HBase-0.94 #999 (See 
[https://builds.apache.org/job/HBase-0.94/999/])
HBASE-8609 Make the CopyTable support startRow, stopRow options (Shaohui) 
(Revision 1487228)

 Result = FAILURE
tedyu : 
Files : 
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/CopyTable.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestCopyTable.java


> Make the CopyTable support startRow, stopRow options
> 
>
> Key: HBASE-8609
> URL: https://issues.apache.org/jira/browse/HBASE-8609
> Project: HBase
>  Issue Type: Improvement
>  Components: mapreduce
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: 0.98.0, 0.95.2, 0.94.9
>
> Attachments: 8609-trunk-v4.patch, HBASE-8609-0.94-v0.patch, 
> HBASE-8609-0.94-v1.patch, HBASE-8609-0.94-v2.patch, 
> HBASE-8609-trunk-v0.patch, HBASE-8609-trunk-v1.patch, 
> HBASE-8609-trunk-v2.patch, HBASE-8609-trunk-v3.patch
>
>
> Make the CopyTable support startRow, stopRow options.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5083) Backport from trunk: Backup HMaster should have http infoport open with link to the active master

[Lars Hofhansl (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5083:
-

Fix Version/s: (was: 0.94.8)
   0.94.9

> Backport from trunk: Backup HMaster should have http infoport open with link 
> to the active master
> -
>
> Key: HBASE-5083
> URL: https://issues.apache.org/jira/browse/HBASE-5083
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Affects Versions: 0.92.0
>Reporter: Jonathan Hsieh
>Assignee: Cody Marcel
> Fix For: 0.94.9
>
> Attachments: backup_master.png, HBASE-5083.patch, HBASE-5083.patch, 
> master.png
>
>
> Without ssh'ing and jps/ps'ing, it is difficult to see if a backup hmaster is 
> up.  It seems like it would be good for a backup hmaster to have a basic web 
> page up on the info port so that users could see that it is up.  Also it 
> should probably either provide a link to the active master or automatically 
> forward to the active master.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8638) add logging to compaction policy

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668984#comment-13668984
 ] 

Ted Yu commented on HBASE-8638:
---

{code}
+  LOG.info("Compaction failed at original callstack: " + 
formatStackTrace(ex));
{code}
Should the above log be at DEBUG level ?

> add logging to compaction policy
> 
>
> Key: HBASE-8638
> URL: https://issues.apache.org/jira/browse/HBASE-8638
> Project: HBase
>  Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Trivial
> Attachments: HBASE-8638-v0.patch
>
>
> We are seeing some strange patterns with current compaction policy in some 
> contexts (with normal writes, no bulk load). It seems like some logging is 
> needed to understand what is going on, similar to old default policy

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-8609) Make the CopyTable support startRow, stopRow options

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John resolved HBASE-8609.
---

  Resolution: Fixed
Hadoop Flags: Reviewed

> Make the CopyTable support startRow, stopRow options
> 
>
> Key: HBASE-8609
> URL: https://issues.apache.org/jira/browse/HBASE-8609
> Project: HBase
>  Issue Type: Improvement
>  Components: mapreduce
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: 0.98.0, 0.95.2, 0.94.9
>
> Attachments: 8609-trunk-v4.patch, HBASE-8609-0.94-v0.patch, 
> HBASE-8609-0.94-v1.patch, HBASE-8609-0.94-v2.patch, 
> HBASE-8609-trunk-v0.patch, HBASE-8609-trunk-v1.patch, 
> HBASE-8609-trunk-v2.patch, HBASE-8609-trunk-v3.patch
>
>
> Make the CopyTable support startRow, stopRow options.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8492) Introduce master failover into a new AccessController unit test

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-8492:
--

Component/s: security

> Introduce master failover into a new AccessController unit test
> ---
>
> Key: HBASE-8492
> URL: https://issues.apache.org/jira/browse/HBASE-8492
> Project: HBase
>  Issue Type: Test
>  Components: security
>Reporter: Ted Yu
> Attachments: 8492-0.94-v1.txt
>
>
> Currently TestAccessController starts a cluster with one master.
> This JIRA introduces two masters in the cluster where the active master is 
> stopped. Expectation is that standby master should take over and perform 
> permission verification.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8615) TestReplicationQueueFailoverCompressed#queueFailover fails on hadoop 2.0 due to IndexOutOfBoundsException

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-8615:
--

Component/s: Replication

> TestReplicationQueueFailoverCompressed#queueFailover fails on hadoop 2.0 due 
> to IndexOutOfBoundsException
> -
>
> Key: HBASE-8615
> URL: https://issues.apache.org/jira/browse/HBASE-8615
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Ted Yu
> Attachments: 
> org.apache.hadoop.hbase.replication.TestReplicationQueueFailoverCompressed-output.txt
>
>
> In a recent test run, I noticed the following in test output:
> {code}
> 2013-05-24 22:01:02,424 DEBUG 
> [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2]
>  fs.HFileSystem$ReorderWALBlocks(327): 
> /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428
>  is an HLog file, so reordering blocks, last hostname will 
> be:kiyo.gq1.ygridcore.net
> 2013-05-24 22:01:02,429 DEBUG 
> [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2]
>  wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 
> 132235, fileLength: 132243, trailerPresent: true
> 2013-05-24 22:01:02,438 ERROR 
> [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2]
>  wal.ProtobufLogReader(236): Error  while reading 691 WAL KVs; started 
> reading at 53272 and read up to 65538
> 2013-05-24 22:01:02,438 WARN  
> [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2]
>  regionserver.ReplicationSource(324): 2 Got:
> java.io.IOException: Error  while reading 691 WAL KVs; started reading at 
> 53272 and read up to 65538
> at 
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:237)
> at 
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:96)
> at 
> org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.readNextAndSetPosition(ReplicationHLogReaderManager.java:89)
> at 
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:404)
> at 
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:320)
> Caused by: java.lang.IndexOutOfBoundsException: index (30062) must be less 
> than size (1)
> at 
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
> at 
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
> at 
> org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.get(LRUDictionary.java:124)
> at 
> org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.access$000(LRUDictionary.java:71)
> at 
> org.apache.hadoop.hbase.regionserver.wal.LRUDictionary.getEntry(LRUDictionary.java:42)
> at 
> org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.readIntoArray(WALCellCodec.java:210)
> at 
> org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.parseCell(WALCellCodec.java:184)
> at 
> org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:46)
> at 
> org.apache.hadoop.hbase.regionserver.wal.WALEdit.readFromCells(WALEdit.java:213)
> at 
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:217)
> ... 4 more
> 2013-05-24 22:01:02,439 DEBUG 
> [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2]
>  regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100 
> times 10
> {code}
> Will attach test output.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8609) Make the CopyTable support startRow, stopRow options

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668979#comment-13668979
 ] 

Ted Yu commented on HBASE-8609:
---

0.94 security test suite result was good.

Integrated to 0.94 as well.

> Make the CopyTable support startRow, stopRow options
> 
>
> Key: HBASE-8609
> URL: https://issues.apache.org/jira/browse/HBASE-8609
> Project: HBase
>  Issue Type: Improvement
>  Components: mapreduce
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: 0.98.0, 0.95.2, 0.94.9
>
> Attachments: 8609-trunk-v4.patch, HBASE-8609-0.94-v0.patch, 
> HBASE-8609-0.94-v1.patch, HBASE-8609-0.94-v2.patch, 
> HBASE-8609-trunk-v0.patch, HBASE-8609-trunk-v1.patch, 
> HBASE-8609-trunk-v2.patch, HBASE-8609-trunk-v3.patch
>
>
> Make the CopyTable support startRow, stopRow options.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8630) Share Socket Connections for different HConnectionImplementations

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668975#comment-13668975
 ] 

Ted Yu commented on HBASE-8630:
---

[~ghelmling]:
What do you think ?

> Share Socket Connections for different HConnectionImplementations
> -
>
> Key: HBASE-8630
> URL: https://issues.apache.org/jira/browse/HBASE-8630
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Affects Versions: 0.94.3
>Reporter: cuijianwei
> Attachments: 8630-trunk-v1.txt
>
>
> In org.apache.hadoop.hbase.ipc.HBaseClient.java, socket connections are 
> pooled by map as:
> {code} protected final PoolMap connections; {code}
> The hashCode of ConnectionId is defined as:
> {code} public int hashCode() {
>   return (address.hashCode() + PRIME * (
>   PRIME * System.identityHashCode(protocol) ^
>  (ticket == null ? 0 : ticket.hashCode()) )) ^ rpcTimeout;
> } {code}
> As we can see, ticket.hashCode() will contribute to hashCode of ConnectionId. 
> For hbase without authentication, the ticket should be a HadoopUser; while 
> for hbase with authentication, the ticket should be a SecureHadoopUser. 
> Neither HadoopUser nor SecureHadoopUser override hashCode() method, 
> therefore, two tickets have the same hashCode only when they refer to the 
> same object.
>   On the other hand, when we use HTable to access hbase, firstly, we will 
> invoke HBaseRPC.waitForProxy(...) to create a proxy for region server as 
> follows:
> {code}  server = (HRegionInterface) HBaseRPC.waitForProxy(
>   serverInterfaceClass, HRegionInterface.VERSION,
>   address, this.conf,
>   this.maxRPCAttempts, this.rpcTimeout, this.rpcTimeout); 
> {code}
> Then HBaseRpc.getProxy(...) will be called as follows:
> {code}public static VersionedProtocol getProxy(Class VersionedProtocol> protocol,
>   long clientVersion, InetSocketAddress addr, Configuration conf,
>   SocketFactory factory, int rpcTimeout) throws IOException {
> return getProxy(protocol, clientVersion, addr,
> User.getCurrent(), conf, factory, rpcTimeout);
>   } {code}
> We can see, User.getCurrent() will be invoked to generate the ticket to build 
> socket connection. User.getCurrent() is defined as:
> {code}
> public static User getCurrent() throws IOException {
> User user;
> if (IS_SECURE_HADOOP) {
>   user = new SecureHadoopUser();
> } else {
>   user = new HadoopUser();
> }
> if (user.getUGI() == null) {
>   return null;
> }
> return user;
>   }
> {code}
> Therefore, we will get different tickets when we create different proxies for 
> the same region server, so that these proxies can't share the created socket 
> connections and will create new socket connections even if they have the same 
> HBaseConfiguration.
> We can use the following case to validate the description above:
> {code}
> public static void main(String args[]) throws Exception {
> Configuration conf = HBaseConfiguration.create();
> for (int i = 0;; ++i) {
>   HTable table = new HTable(conf, TestTable.testTableName);
>   table.close();
> }
> }
> {code}
> Each time we close the HTable, the created region server proxies will be 
> closed as the underlying HConnectionImplementation will be closed. However, 
> the created socket connections won't be closed and wait to be shared in 
> future. Then, when we create HTable in the next turn, we will create server 
> proxy again, get a new ticket and consequently create new socket connections. 
> The created socket connections last turn can not be used any more. As the 
> loop goes on, thousands of socket will be created to connect region servers 
> until we get an exception to show no more sockets could be created.
> To fix the problem, maybe, we can use ticket.getName().hashCode() instead of 
> ticket.hashCode()?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8630) Share Socket Connections for different HConnectionImplementations

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-8630:
--

Status: Patch Available  (was: Open)

> Share Socket Connections for different HConnectionImplementations
> -
>
> Key: HBASE-8630
> URL: https://issues.apache.org/jira/browse/HBASE-8630
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Affects Versions: 0.94.3
>Reporter: cuijianwei
> Attachments: 8630-trunk-v1.txt
>
>
> In org.apache.hadoop.hbase.ipc.HBaseClient.java, socket connections are 
> pooled by map as:
> {code} protected final PoolMap connections; {code}
> The hashCode of ConnectionId is defined as:
> {code} public int hashCode() {
>   return (address.hashCode() + PRIME * (
>   PRIME * System.identityHashCode(protocol) ^
>  (ticket == null ? 0 : ticket.hashCode()) )) ^ rpcTimeout;
> } {code}
> As we can see, ticket.hashCode() will contribute to hashCode of ConnectionId. 
> For hbase without authentication, the ticket should be a HadoopUser; while 
> for hbase with authentication, the ticket should be a SecureHadoopUser. 
> Neither HadoopUser nor SecureHadoopUser override hashCode() method, 
> therefore, two tickets have the same hashCode only when they refer to the 
> same object.
>   On the other hand, when we use HTable to access hbase, firstly, we will 
> invoke HBaseRPC.waitForProxy(...) to create a proxy for region server as 
> follows:
> {code}  server = (HRegionInterface) HBaseRPC.waitForProxy(
>   serverInterfaceClass, HRegionInterface.VERSION,
>   address, this.conf,
>   this.maxRPCAttempts, this.rpcTimeout, this.rpcTimeout); 
> {code}
> Then HBaseRpc.getProxy(...) will be called as follows:
> {code}public static VersionedProtocol getProxy(Class VersionedProtocol> protocol,
>   long clientVersion, InetSocketAddress addr, Configuration conf,
>   SocketFactory factory, int rpcTimeout) throws IOException {
> return getProxy(protocol, clientVersion, addr,
> User.getCurrent(), conf, factory, rpcTimeout);
>   } {code}
> We can see, User.getCurrent() will be invoked to generate the ticket to build 
> socket connection. User.getCurrent() is defined as:
> {code}
> public static User getCurrent() throws IOException {
> User user;
> if (IS_SECURE_HADOOP) {
>   user = new SecureHadoopUser();
> } else {
>   user = new HadoopUser();
> }
> if (user.getUGI() == null) {
>   return null;
> }
> return user;
>   }
> {code}
> Therefore, we will get different tickets when we create different proxies for 
> the same region server, so that these proxies can't share the created socket 
> connections and will create new socket connections even if they have the same 
> HBaseConfiguration.
> We can use the following case to validate the description above:
> {code}
> public static void main(String args[]) throws Exception {
> Configuration conf = HBaseConfiguration.create();
> for (int i = 0;; ++i) {
>   HTable table = new HTable(conf, TestTable.testTableName);
>   table.close();
> }
> }
> {code}
> Each time we close the HTable, the created region server proxies will be 
> closed as the underlying HConnectionImplementation will be closed. However, 
> the created socket connections won't be closed and wait to be shared in 
> future. Then, when we create HTable in the next turn, we will create server 
> proxy again, get a new ticket and consequently create new socket connections. 
> The created socket connections last turn can not be used any more. As the 
> loop goes on, thousands of socket will be created to connect region servers 
> until we get an exception to show no more sockets could be created.
> To fix the problem, maybe, we can use ticket.getName().hashCode() instead of 
> ticket.hashCode()?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8634) Fix potential null pointer dereference in HRegionServer and TableLockChecker

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-8634:
--

Attachment: 8634-v2.txt

> Fix potential null pointer dereference in HRegionServer and TableLockChecker
> 
>
> Key: HBASE-8634
> URL: https://issues.apache.org/jira/browse/HBASE-8634
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: 8634-v1.txt, 8634-v2.txt
>
>
> From 
> https://builds.apache.org/job/PreCommit-HBASE-Build/5852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
>  :
> {code}
> Possible null pointer dereference of rpcc in 
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(RpcController, 
> ClientProtos$MultiRequest)
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.regionserver.HRegionServerIn 
> method 
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(RpcController, 
> ClientProtos$MultiRequest)Value loaded from rpccDereferenced at 
> HRegionServer.java:[line 3176]Known null at HRegionServer.java:[line 
> 3174]
> ...
> Possible null pointer dereference of rpcc in 
> org.apache.hadoop.hbase.regionserver.HRegionServer.mutate(RpcController, 
> ClientProtos$MutateRequest)
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.regionserver.HRegionServerIn 
> method 
> org.apache.hadoop.hbase.regionserver.HRegionServer.mutate(RpcController, 
> ClientProtos$MutateRequest)Value loaded from rpccDereferenced at 
> HRegionServer.java:[line 2832]Known null at HRegionServer.java:[line 
> 2830]
> ...
> Possible null pointer dereference of rpcc in 
> org.apache.hadoop.hbase.regionserver.HRegionServer.replay(RpcController, 
> ClientProtos$MultiRequest)
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.regionserver.HRegionServerIn 
> method 
> org.apache.hadoop.hbase.regionserver.HRegionServer.replay(RpcController, 
> ClientProtos$MultiRequest)Value loaded from rpccDereferenced at 
> HRegionServer.java:[line 3742]Known null at HRegionServer.java:[line 
> 3740]
> ...
> Possible null pointer dereference of data in 
> org.apache.hadoop.hbase.util.hbck.TableLockChecker$1.handleMetadata(byte[])
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.util.hbck.TableLockChecker$1In 
> method 
> org.apache.hadoop.hbase.util.hbck.TableLockChecker$1.handleMetadata(byte[])Value
>  loaded from dataDereferenced at TableLockChecker.java:[line 
> 68]Known null at TableLockChecker.java:[line 60]
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8630) Share Socket Connections for different HConnectionImplementations

[cuijianwei (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

cuijianwei updated HBASE-8630:
--

Attachment: 8630-trunk-v1.txt

using ticket.getUGI().hashCode()

> Share Socket Connections for different HConnectionImplementations
> -
>
> Key: HBASE-8630
> URL: https://issues.apache.org/jira/browse/HBASE-8630
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Affects Versions: 0.94.3
>Reporter: cuijianwei
> Attachments: 8630-trunk-v1.txt
>
>
> In org.apache.hadoop.hbase.ipc.HBaseClient.java, socket connections are 
> pooled by map as:
> {code} protected final PoolMap connections; {code}
> The hashCode of ConnectionId is defined as:
> {code} public int hashCode() {
>   return (address.hashCode() + PRIME * (
>   PRIME * System.identityHashCode(protocol) ^
>  (ticket == null ? 0 : ticket.hashCode()) )) ^ rpcTimeout;
> } {code}
> As we can see, ticket.hashCode() will contribute to hashCode of ConnectionId. 
> For hbase without authentication, the ticket should be a HadoopUser; while 
> for hbase with authentication, the ticket should be a SecureHadoopUser. 
> Neither HadoopUser nor SecureHadoopUser override hashCode() method, 
> therefore, two tickets have the same hashCode only when they refer to the 
> same object.
>   On the other hand, when we use HTable to access hbase, firstly, we will 
> invoke HBaseRPC.waitForProxy(...) to create a proxy for region server as 
> follows:
> {code}  server = (HRegionInterface) HBaseRPC.waitForProxy(
>   serverInterfaceClass, HRegionInterface.VERSION,
>   address, this.conf,
>   this.maxRPCAttempts, this.rpcTimeout, this.rpcTimeout); 
> {code}
> Then HBaseRpc.getProxy(...) will be called as follows:
> {code}public static VersionedProtocol getProxy(Class VersionedProtocol> protocol,
>   long clientVersion, InetSocketAddress addr, Configuration conf,
>   SocketFactory factory, int rpcTimeout) throws IOException {
> return getProxy(protocol, clientVersion, addr,
> User.getCurrent(), conf, factory, rpcTimeout);
>   } {code}
> We can see, User.getCurrent() will be invoked to generate the ticket to build 
> socket connection. User.getCurrent() is defined as:
> {code}
> public static User getCurrent() throws IOException {
> User user;
> if (IS_SECURE_HADOOP) {
>   user = new SecureHadoopUser();
> } else {
>   user = new HadoopUser();
> }
> if (user.getUGI() == null) {
>   return null;
> }
> return user;
>   }
> {code}
> Therefore, we will get different tickets when we create different proxies for 
> the same region server, so that these proxies can't share the created socket 
> connections and will create new socket connections even if they have the same 
> HBaseConfiguration.
> We can use the following case to validate the description above:
> {code}
> public static void main(String args[]) throws Exception {
> Configuration conf = HBaseConfiguration.create();
> for (int i = 0;; ++i) {
>   HTable table = new HTable(conf, TestTable.testTableName);
>   table.close();
> }
> }
> {code}
> Each time we close the HTable, the created region server proxies will be 
> closed as the underlying HConnectionImplementation will be closed. However, 
> the created socket connections won't be closed and wait to be shared in 
> future. Then, when we create HTable in the next turn, we will create server 
> proxy again, get a new ticket and consequently create new socket connections. 
> The created socket connections last turn can not be used any more. As the 
> loop goes on, thousands of socket will be created to connect region servers 
> until we get an exception to show no more sockets could be created.
> To fix the problem, maybe, we can use ticket.getName().hashCode() instead of 
> ticket.hashCode()?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8630) Share Socket Connections for different HConnectionImplementations

[cuijianwei (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668970#comment-13668970
 ] 

cuijianwei commented on HBASE-8630:
---

I search hbase project, and find that loginUserFromKeytab() won't be called. If 
users call loginUserFromKeytab() themselves, they should response for the 
results. In SecureClient, reloginFromKeytab() or reloginFromTicketCache() could 
be called, these two functions won't change the current user. Therefore, 
ticket.getUGI().hashCode() seems good for this problem. I provide a trunk patch 
8630-trunk-v1.txt using ticket.getUGI().hashCode().

> Share Socket Connections for different HConnectionImplementations
> -
>
> Key: HBASE-8630
> URL: https://issues.apache.org/jira/browse/HBASE-8630
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Affects Versions: 0.94.3
>Reporter: cuijianwei
> Attachments: 8630-trunk-v1.txt
>
>
> In org.apache.hadoop.hbase.ipc.HBaseClient.java, socket connections are 
> pooled by map as:
> {code} protected final PoolMap connections; {code}
> The hashCode of ConnectionId is defined as:
> {code} public int hashCode() {
>   return (address.hashCode() + PRIME * (
>   PRIME * System.identityHashCode(protocol) ^
>  (ticket == null ? 0 : ticket.hashCode()) )) ^ rpcTimeout;
> } {code}
> As we can see, ticket.hashCode() will contribute to hashCode of ConnectionId. 
> For hbase without authentication, the ticket should be a HadoopUser; while 
> for hbase with authentication, the ticket should be a SecureHadoopUser. 
> Neither HadoopUser nor SecureHadoopUser override hashCode() method, 
> therefore, two tickets have the same hashCode only when they refer to the 
> same object.
>   On the other hand, when we use HTable to access hbase, firstly, we will 
> invoke HBaseRPC.waitForProxy(...) to create a proxy for region server as 
> follows:
> {code}  server = (HRegionInterface) HBaseRPC.waitForProxy(
>   serverInterfaceClass, HRegionInterface.VERSION,
>   address, this.conf,
>   this.maxRPCAttempts, this.rpcTimeout, this.rpcTimeout); 
> {code}
> Then HBaseRpc.getProxy(...) will be called as follows:
> {code}public static VersionedProtocol getProxy(Class VersionedProtocol> protocol,
>   long clientVersion, InetSocketAddress addr, Configuration conf,
>   SocketFactory factory, int rpcTimeout) throws IOException {
> return getProxy(protocol, clientVersion, addr,
> User.getCurrent(), conf, factory, rpcTimeout);
>   } {code}
> We can see, User.getCurrent() will be invoked to generate the ticket to build 
> socket connection. User.getCurrent() is defined as:
> {code}
> public static User getCurrent() throws IOException {
> User user;
> if (IS_SECURE_HADOOP) {
>   user = new SecureHadoopUser();
> } else {
>   user = new HadoopUser();
> }
> if (user.getUGI() == null) {
>   return null;
> }
> return user;
>   }
> {code}
> Therefore, we will get different tickets when we create different proxies for 
> the same region server, so that these proxies can't share the created socket 
> connections and will create new socket connections even if they have the same 
> HBaseConfiguration.
> We can use the following case to validate the description above:
> {code}
> public static void main(String args[]) throws Exception {
> Configuration conf = HBaseConfiguration.create();
> for (int i = 0;; ++i) {
>   HTable table = new HTable(conf, TestTable.testTableName);
>   table.close();
> }
> }
> {code}
> Each time we close the HTable, the created region server proxies will be 
> closed as the underlying HConnectionImplementation will be closed. However, 
> the created socket connections won't be closed and wait to be shared in 
> future. Then, when we create HTable in the next turn, we will create server 
> proxy again, get a new ticket and consequently create new socket connections. 
> The created socket connections last turn can not be used any more. As the 
> loop goes on, thousands of socket will be created to connect region servers 
> until we get an exception to show no more sockets could be created.
> To fix the problem, maybe, we can use ticket.getName().hashCode() instead of 
> ticket.hashCode()?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8634) Fix potential null pointer dereference in HRegionServer and TableLockChecker

[chunhui shen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668962#comment-13668962
 ] 

chunhui shen commented on HBASE-8634:
-

line 2918 in HRegionServer:
{code}
  if (cellsToReturn != null) {
controller.setCellScanner(cellsToReturn.cellScanner());
  }
{code}
I think NULL check should also be done here.

patch is ok, +1 

> Fix potential null pointer dereference in HRegionServer and TableLockChecker
> 
>
> Key: HBASE-8634
> URL: https://issues.apache.org/jira/browse/HBASE-8634
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: 8634-v1.txt
>
>
> From 
> https://builds.apache.org/job/PreCommit-HBASE-Build/5852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
>  :
> {code}
> Possible null pointer dereference of rpcc in 
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(RpcController, 
> ClientProtos$MultiRequest)
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.regionserver.HRegionServerIn 
> method 
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(RpcController, 
> ClientProtos$MultiRequest)Value loaded from rpccDereferenced at 
> HRegionServer.java:[line 3176]Known null at HRegionServer.java:[line 
> 3174]
> ...
> Possible null pointer dereference of rpcc in 
> org.apache.hadoop.hbase.regionserver.HRegionServer.mutate(RpcController, 
> ClientProtos$MutateRequest)
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.regionserver.HRegionServerIn 
> method 
> org.apache.hadoop.hbase.regionserver.HRegionServer.mutate(RpcController, 
> ClientProtos$MutateRequest)Value loaded from rpccDereferenced at 
> HRegionServer.java:[line 2832]Known null at HRegionServer.java:[line 
> 2830]
> ...
> Possible null pointer dereference of rpcc in 
> org.apache.hadoop.hbase.regionserver.HRegionServer.replay(RpcController, 
> ClientProtos$MultiRequest)
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.regionserver.HRegionServerIn 
> method 
> org.apache.hadoop.hbase.regionserver.HRegionServer.replay(RpcController, 
> ClientProtos$MultiRequest)Value loaded from rpccDereferenced at 
> HRegionServer.java:[line 3742]Known null at HRegionServer.java:[line 
> 3740]
> ...
> Possible null pointer dereference of data in 
> org.apache.hadoop.hbase.util.hbck.TableLockChecker$1.handleMetadata(byte[])
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.util.hbck.TableLockChecker$1In 
> method 
> org.apache.hadoop.hbase.util.hbck.TableLockChecker$1.handleMetadata(byte[])Value
>  loaded from dataDereferenced at TableLockChecker.java:[line 
> 68]Known null at TableLockChecker.java:[line 60]
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8609) Make the CopyTable support startRow, stopRow options

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668948#comment-13668948
 ] 

Hudson commented on HBASE-8609:
---

Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #546 (See 
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/546/])
HBASE-8609 Make the CopyTable support startRow, stopRow options (Liu 
Shaohui) (Revision 1487032)

 Result = FAILURE
tedyu : 
Files : 
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/CopyTable.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestCopyTable.java


> Make the CopyTable support startRow, stopRow options
> 
>
> Key: HBASE-8609
> URL: https://issues.apache.org/jira/browse/HBASE-8609
> Project: HBase
>  Issue Type: Improvement
>  Components: mapreduce
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: 0.98.0, 0.95.2, 0.94.9
>
> Attachments: 8609-trunk-v4.patch, HBASE-8609-0.94-v0.patch, 
> HBASE-8609-0.94-v1.patch, HBASE-8609-0.94-v2.patch, 
> HBASE-8609-trunk-v0.patch, HBASE-8609-trunk-v1.patch, 
> HBASE-8609-trunk-v2.patch, HBASE-8609-trunk-v3.patch
>
>
> Make the CopyTable support startRow, stopRow options.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8609) Make the CopyTable support startRow, stopRow options

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-8609:
--

Fix Version/s: 0.94.9

> Make the CopyTable support startRow, stopRow options
> 
>
> Key: HBASE-8609
> URL: https://issues.apache.org/jira/browse/HBASE-8609
> Project: HBase
>  Issue Type: Improvement
>  Components: mapreduce
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: 0.98.0, 0.95.2, 0.94.9
>
> Attachments: 8609-trunk-v4.patch, HBASE-8609-0.94-v0.patch, 
> HBASE-8609-0.94-v1.patch, HBASE-8609-0.94-v2.patch, 
> HBASE-8609-trunk-v0.patch, HBASE-8609-trunk-v1.patch, 
> HBASE-8609-trunk-v2.patch, HBASE-8609-trunk-v3.patch
>
>
> Make the CopyTable support startRow, stopRow options.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8630) Share Socket Connections for different HConnectionImplementations

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668946#comment-13668946
 ] 

Ted Yu commented on HBASE-8630:
---

Looking at User class in HBase, it calls the following method of 
UserGroupInformation:
{code}
  static UserGroupInformation getLoginUser() throws IOException {
{code}
from:
{code}
ugi = (UserGroupInformation) callStatic("getCurrentUser");
{code}
Can you tell me where loginUserFromKeytab() would be called ?

> Share Socket Connections for different HConnectionImplementations
> -
>
> Key: HBASE-8630
> URL: https://issues.apache.org/jira/browse/HBASE-8630
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Affects Versions: 0.94.3
>Reporter: cuijianwei
>
> In org.apache.hadoop.hbase.ipc.HBaseClient.java, socket connections are 
> pooled by map as:
> {code} protected final PoolMap connections; {code}
> The hashCode of ConnectionId is defined as:
> {code} public int hashCode() {
>   return (address.hashCode() + PRIME * (
>   PRIME * System.identityHashCode(protocol) ^
>  (ticket == null ? 0 : ticket.hashCode()) )) ^ rpcTimeout;
> } {code}
> As we can see, ticket.hashCode() will contribute to hashCode of ConnectionId. 
> For hbase without authentication, the ticket should be a HadoopUser; while 
> for hbase with authentication, the ticket should be a SecureHadoopUser. 
> Neither HadoopUser nor SecureHadoopUser override hashCode() method, 
> therefore, two tickets have the same hashCode only when they refer to the 
> same object.
>   On the other hand, when we use HTable to access hbase, firstly, we will 
> invoke HBaseRPC.waitForProxy(...) to create a proxy for region server as 
> follows:
> {code}  server = (HRegionInterface) HBaseRPC.waitForProxy(
>   serverInterfaceClass, HRegionInterface.VERSION,
>   address, this.conf,
>   this.maxRPCAttempts, this.rpcTimeout, this.rpcTimeout); 
> {code}
> Then HBaseRpc.getProxy(...) will be called as follows:
> {code}public static VersionedProtocol getProxy(Class VersionedProtocol> protocol,
>   long clientVersion, InetSocketAddress addr, Configuration conf,
>   SocketFactory factory, int rpcTimeout) throws IOException {
> return getProxy(protocol, clientVersion, addr,
> User.getCurrent(), conf, factory, rpcTimeout);
>   } {code}
> We can see, User.getCurrent() will be invoked to generate the ticket to build 
> socket connection. User.getCurrent() is defined as:
> {code}
> public static User getCurrent() throws IOException {
> User user;
> if (IS_SECURE_HADOOP) {
>   user = new SecureHadoopUser();
> } else {
>   user = new HadoopUser();
> }
> if (user.getUGI() == null) {
>   return null;
> }
> return user;
>   }
> {code}
> Therefore, we will get different tickets when we create different proxies for 
> the same region server, so that these proxies can't share the created socket 
> connections and will create new socket connections even if they have the same 
> HBaseConfiguration.
> We can use the following case to validate the description above:
> {code}
> public static void main(String args[]) throws Exception {
> Configuration conf = HBaseConfiguration.create();
> for (int i = 0;; ++i) {
>   HTable table = new HTable(conf, TestTable.testTableName);
>   table.close();
> }
> }
> {code}
> Each time we close the HTable, the created region server proxies will be 
> closed as the underlying HConnectionImplementation will be closed. However, 
> the created socket connections won't be closed and wait to be shared in 
> future. Then, when we create HTable in the next turn, we will create server 
> proxy again, get a new ticket and consequently create new socket connections. 
> The created socket connections last turn can not be used any more. As the 
> loop goes on, thousands of socket will be created to connect region servers 
> until we get an exception to show no more sockets could be created.
> To fix the problem, maybe, we can use ticket.getName().hashCode() instead of 
> ticket.hashCode()?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8630) Share Socket Connections for different HConnectionImplementations

[cuijianwei (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668939#comment-13668939
 ] 

cuijianwei commented on HBASE-8630:
---

Thanks for your comment. I think ticket.getUGI().hashCode() seems OK. If we 
login many times in a process, such as calling 
{code}
  public synchronized
  static void loginUserFromKeytab(String user,
  String path
  ) throws IOException
{code}
We will get different ugis, consequently different ticket.getUGI().hashCode(), 
does this meet our expectation?

> Share Socket Connections for different HConnectionImplementations
> -
>
> Key: HBASE-8630
> URL: https://issues.apache.org/jira/browse/HBASE-8630
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Affects Versions: 0.94.3
>Reporter: cuijianwei
>
> In org.apache.hadoop.hbase.ipc.HBaseClient.java, socket connections are 
> pooled by map as:
> {code} protected final PoolMap connections; {code}
> The hashCode of ConnectionId is defined as:
> {code} public int hashCode() {
>   return (address.hashCode() + PRIME * (
>   PRIME * System.identityHashCode(protocol) ^
>  (ticket == null ? 0 : ticket.hashCode()) )) ^ rpcTimeout;
> } {code}
> As we can see, ticket.hashCode() will contribute to hashCode of ConnectionId. 
> For hbase without authentication, the ticket should be a HadoopUser; while 
> for hbase with authentication, the ticket should be a SecureHadoopUser. 
> Neither HadoopUser nor SecureHadoopUser override hashCode() method, 
> therefore, two tickets have the same hashCode only when they refer to the 
> same object.
>   On the other hand, when we use HTable to access hbase, firstly, we will 
> invoke HBaseRPC.waitForProxy(...) to create a proxy for region server as 
> follows:
> {code}  server = (HRegionInterface) HBaseRPC.waitForProxy(
>   serverInterfaceClass, HRegionInterface.VERSION,
>   address, this.conf,
>   this.maxRPCAttempts, this.rpcTimeout, this.rpcTimeout); 
> {code}
> Then HBaseRpc.getProxy(...) will be called as follows:
> {code}public static VersionedProtocol getProxy(Class VersionedProtocol> protocol,
>   long clientVersion, InetSocketAddress addr, Configuration conf,
>   SocketFactory factory, int rpcTimeout) throws IOException {
> return getProxy(protocol, clientVersion, addr,
> User.getCurrent(), conf, factory, rpcTimeout);
>   } {code}
> We can see, User.getCurrent() will be invoked to generate the ticket to build 
> socket connection. User.getCurrent() is defined as:
> {code}
> public static User getCurrent() throws IOException {
> User user;
> if (IS_SECURE_HADOOP) {
>   user = new SecureHadoopUser();
> } else {
>   user = new HadoopUser();
> }
> if (user.getUGI() == null) {
>   return null;
> }
> return user;
>   }
> {code}
> Therefore, we will get different tickets when we create different proxies for 
> the same region server, so that these proxies can't share the created socket 
> connections and will create new socket connections even if they have the same 
> HBaseConfiguration.
> We can use the following case to validate the description above:
> {code}
> public static void main(String args[]) throws Exception {
> Configuration conf = HBaseConfiguration.create();
> for (int i = 0;; ++i) {
>   HTable table = new HTable(conf, TestTable.testTableName);
>   table.close();
> }
> }
> {code}
> Each time we close the HTable, the created region server proxies will be 
> closed as the underlying HConnectionImplementation will be closed. However, 
> the created socket connections won't be closed and wait to be shared in 
> future. Then, when we create HTable in the next turn, we will create server 
> proxy again, get a new ticket and consequently create new socket connections. 
> The created socket connections last turn can not be used any more. As the 
> loop goes on, thousands of socket will be created to connect region servers 
> until we get an exception to show no more sockets could be created.
> To fix the problem, maybe, we can use ticket.getName().hashCode() instead of 
> ticket.hashCode()?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8634) Fix potential null pointer dereference in HRegionServer and TableLockChecker

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668934#comment-13668934
 ] 

Hadoop QA commented on HBASE-8634:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12585137/8634-v1.txt
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 
1.0 profile.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5858//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5858//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5858//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5858//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5858//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5858//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5858//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5858//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5858//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5858//console

This message is automatically generated.

> Fix potential null pointer dereference in HRegionServer and TableLockChecker
> 
>
> Key: HBASE-8634
> URL: https://issues.apache.org/jira/browse/HBASE-8634
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: 8634-v1.txt
>
>
> From 
> https://builds.apache.org/job/PreCommit-HBASE-Build/5852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
>  :
> {code}
> Possible null pointer dereference of rpcc in 
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(RpcController, 
> ClientProtos$MultiRequest)
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.regionserver.HRegionServerIn 
> method 
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(RpcController, 
> ClientProtos$MultiRequest)Value loaded from rpccDereferenced at 
> HRegionServer.java:[line 3176]Known null at HRegionServer.java:[line 
> 3174]
> ...
> Possible null pointer dereference of rpcc in 
> org.apache.hadoop.hbase.regionserver.HRegionServer.mutate(RpcController, 
> ClientProtos$MutateRequest)
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.regionserver.HRegionServerIn 
> method 
> org.apache.hadoop.hbase.regionserver.HRegionServer.mutate(RpcController, 
> ClientProtos$MutateRequest)Value loaded from rpccDereferenced at 
> HRegionServer.java:[line 2832]Known null at HRegionServer.java:[line 
> 2830]
> ...
> Possible null pointer dereference of rpcc in 
> org.apache.hadoop.hbase.regionserver.HRegionServer.replay(RpcController, 
> ClientProtos$MultiRequest)
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> 

[jira] [Commented] (HBASE-8344) Improve the assignment when node failures happen to choose the secondary RS as the new primary RS

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668932#comment-13668932
 ] 

Hadoop QA commented on HBASE-8344:
--

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12585136/hbase-8344-2.2.txt
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 6 new 
or modified tests.

{color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 
1.0 profile.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5859//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5859//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5859//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5859//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5859//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5859//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5859//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5859//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5859//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5859//console

This message is automatically generated.

> Improve the assignment when node failures happen to choose the secondary RS 
> as the new primary RS
> -
>
> Key: HBASE-8344
> URL: https://issues.apache.org/jira/browse/HBASE-8344
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Devaraj Das
>Assignee: Devaraj Das
>Priority: Critical
> Fix For: 0.95.2
>
> Attachments: hbase-8344-1.txt, hbase-8344-2.1.txt, hbase-8344-2.2.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8609) Make the CopyTable support startRow, stopRow options

[Liu Shaohui (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liu Shaohui updated HBASE-8609:
---

Attachment: HBASE-8609-0.94-v2.patch

Backport the change in the trunk review to 0.94 patch

> Make the CopyTable support startRow, stopRow options
> 
>
> Key: HBASE-8609
> URL: https://issues.apache.org/jira/browse/HBASE-8609
> Project: HBase
>  Issue Type: Improvement
>  Components: mapreduce
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: 0.98.0, 0.95.2
>
> Attachments: 8609-trunk-v4.patch, HBASE-8609-0.94-v0.patch, 
> HBASE-8609-0.94-v1.patch, HBASE-8609-0.94-v2.patch, 
> HBASE-8609-trunk-v0.patch, HBASE-8609-trunk-v1.patch, 
> HBASE-8609-trunk-v2.patch, HBASE-8609-trunk-v3.patch
>
>
> Make the CopyTable support startRow, stopRow options.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8638) add logging to compaction policy

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668915#comment-13668915
 ] 

Hadoop QA commented on HBASE-8638:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12585130/HBASE-8638-v0.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 
1.0 profile.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5857//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5857//console

This message is automatically generated.

> add logging to compaction policy
> 
>
> Key: HBASE-8638
> URL: https://issues.apache.org/jira/browse/HBASE-8638
> Project: HBase
>  Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Trivial
> Attachments: HBASE-8638-v0.patch
>
>
> We are seeing some strange patterns with current compaction policy in some 
> contexts (with normal writes, no bulk load). It seems like some logging is 
> needed to understand what is going on, similar to old default policy

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8634) Fix potential null pointer dereference in HRegionServer and TableLockChecker

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-8634:
--

Attachment: (was: 8634-v1.txt)

> Fix potential null pointer dereference in HRegionServer and TableLockChecker
> 
>
> Key: HBASE-8634
> URL: https://issues.apache.org/jira/browse/HBASE-8634
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: 8634-v1.txt
>
>
> From 
> https://builds.apache.org/job/PreCommit-HBASE-Build/5852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
>  :
> {code}
> Possible null pointer dereference of rpcc in 
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(RpcController, 
> ClientProtos$MultiRequest)
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.regionserver.HRegionServerIn 
> method 
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(RpcController, 
> ClientProtos$MultiRequest)Value loaded from rpccDereferenced at 
> HRegionServer.java:[line 3176]Known null at HRegionServer.java:[line 
> 3174]
> ...
> Possible null pointer dereference of rpcc in 
> org.apache.hadoop.hbase.regionserver.HRegionServer.mutate(RpcController, 
> ClientProtos$MutateRequest)
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.regionserver.HRegionServerIn 
> method 
> org.apache.hadoop.hbase.regionserver.HRegionServer.mutate(RpcController, 
> ClientProtos$MutateRequest)Value loaded from rpccDereferenced at 
> HRegionServer.java:[line 2832]Known null at HRegionServer.java:[line 
> 2830]
> ...
> Possible null pointer dereference of rpcc in 
> org.apache.hadoop.hbase.regionserver.HRegionServer.replay(RpcController, 
> ClientProtos$MultiRequest)
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.regionserver.HRegionServerIn 
> method 
> org.apache.hadoop.hbase.regionserver.HRegionServer.replay(RpcController, 
> ClientProtos$MultiRequest)Value loaded from rpccDereferenced at 
> HRegionServer.java:[line 3742]Known null at HRegionServer.java:[line 
> 3740]
> ...
> Possible null pointer dereference of data in 
> org.apache.hadoop.hbase.util.hbck.TableLockChecker$1.handleMetadata(byte[])
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.util.hbck.TableLockChecker$1In 
> method 
> org.apache.hadoop.hbase.util.hbck.TableLockChecker$1.handleMetadata(byte[])Value
>  loaded from dataDereferenced at TableLockChecker.java:[line 
> 68]Known null at TableLockChecker.java:[line 60]
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8609) Make the CopyTable support startRow, stopRow options

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668890#comment-13668890
 ] 

Hudson commented on HBASE-8609:
---

Integrated in hbase-0.95-on-hadoop2 #115 (See 
[https://builds.apache.org/job/hbase-0.95-on-hadoop2/115/])
HBASE-8609 Make the CopyTable support startRow, stopRow options (Liu 
Shaohui) (Revision 1487033)

 Result = FAILURE
tedyu : 
Files : 
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/CopyTable.java
* 
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestCopyTable.java


> Make the CopyTable support startRow, stopRow options
> 
>
> Key: HBASE-8609
> URL: https://issues.apache.org/jira/browse/HBASE-8609
> Project: HBase
>  Issue Type: Improvement
>  Components: mapreduce
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: 0.98.0, 0.95.2
>
> Attachments: 8609-trunk-v4.patch, HBASE-8609-0.94-v0.patch, 
> HBASE-8609-0.94-v1.patch, HBASE-8609-trunk-v0.patch, 
> HBASE-8609-trunk-v1.patch, HBASE-8609-trunk-v2.patch, 
> HBASE-8609-trunk-v3.patch
>
>
> Make the CopyTable support startRow, stopRow options.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8634) Fix potential null pointer dereference in HRegionServer and TableLockChecker

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-8634:
--

Attachment: 8634-v1.txt

> Fix potential null pointer dereference in HRegionServer and TableLockChecker
> 
>
> Key: HBASE-8634
> URL: https://issues.apache.org/jira/browse/HBASE-8634
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: 8634-v1.txt, 8634-v1.txt
>
>
> From 
> https://builds.apache.org/job/PreCommit-HBASE-Build/5852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
>  :
> {code}
> Possible null pointer dereference of rpcc in 
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(RpcController, 
> ClientProtos$MultiRequest)
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.regionserver.HRegionServerIn 
> method 
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(RpcController, 
> ClientProtos$MultiRequest)Value loaded from rpccDereferenced at 
> HRegionServer.java:[line 3176]Known null at HRegionServer.java:[line 
> 3174]
> ...
> Possible null pointer dereference of rpcc in 
> org.apache.hadoop.hbase.regionserver.HRegionServer.mutate(RpcController, 
> ClientProtos$MutateRequest)
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.regionserver.HRegionServerIn 
> method 
> org.apache.hadoop.hbase.regionserver.HRegionServer.mutate(RpcController, 
> ClientProtos$MutateRequest)Value loaded from rpccDereferenced at 
> HRegionServer.java:[line 2832]Known null at HRegionServer.java:[line 
> 2830]
> ...
> Possible null pointer dereference of rpcc in 
> org.apache.hadoop.hbase.regionserver.HRegionServer.replay(RpcController, 
> ClientProtos$MultiRequest)
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.regionserver.HRegionServerIn 
> method 
> org.apache.hadoop.hbase.regionserver.HRegionServer.replay(RpcController, 
> ClientProtos$MultiRequest)Value loaded from rpccDereferenced at 
> HRegionServer.java:[line 3742]Known null at HRegionServer.java:[line 
> 3740]
> ...
> Possible null pointer dereference of data in 
> org.apache.hadoop.hbase.util.hbck.TableLockChecker$1.handleMetadata(byte[])
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.util.hbck.TableLockChecker$1In 
> method 
> org.apache.hadoop.hbase.util.hbck.TableLockChecker$1.handleMetadata(byte[])Value
>  loaded from dataDereferenced at TableLockChecker.java:[line 
> 68]Known null at TableLockChecker.java:[line 60]
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8631) Meta Region First Recovery

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1366#comment-1366
 ] 

Ted Yu commented on HBASE-8631:
---

+1

> Meta Region First Recovery
> --
>
> Key: HBASE-8631
> URL: https://issues.apache.org/jira/browse/HBASE-8631
> Project: HBase
>  Issue Type: Bug
>  Components: MTTR
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
> Attachments: hbase-8631.patch, hbase-8631-v2.patch
>
>
> We have a separate wal for meta region. While log splitting logic haven't 
> taken the advantage of this and splitlogworker still picks a wal file 
> randomly. Imaging if we have multiple region servers including meta RS fails 
> about the same time while meta wal is recovered last, all failed regions have 
> to wait meta recovered and then can be online again. 
> The open JIRA is to let splitlogworker to pick a meta wal file firstly and 
> then others.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8344) Improve the assignment when node failures happen to choose the secondary RS as the new primary RS

[Devaraj Das (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj Das updated HBASE-8344:
---

Attachment: hbase-8344-2.2.txt

Fixes a bug to do with comparing of ServerName objects - makes it so that the 
startcode is ignored in the FavoredNodeLoadBalancer's assignment calls.

> Improve the assignment when node failures happen to choose the secondary RS 
> as the new primary RS
> -
>
> Key: HBASE-8344
> URL: https://issues.apache.org/jira/browse/HBASE-8344
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Devaraj Das
>Assignee: Devaraj Das
>Priority: Critical
> Fix For: 0.95.2
>
> Attachments: hbase-8344-1.txt, hbase-8344-2.1.txt, hbase-8344-2.2.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7963) HBase VerifyReplication not working when security enabled

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-7963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668880#comment-13668880
 ] 

Enis Soztutar commented on HBASE-7963:
--

Could you please explain a bit more why this is needed:
{code}
-ZKUtil.createWithParents(this.zookeeper, this.peersZNode);
+ZKUtil.createWithParentsIfNotExists(this.zookeeper, this.peersZNode);
{code}
Is this because of (1) above? 


> HBase VerifyReplication not working when security enabled
> -
>
> Key: HBASE-7963
> URL: https://issues.apache.org/jira/browse/HBASE-7963
> Project: HBase
>  Issue Type: Bug
>  Components: Replication, security
>Affects Versions: 0.94.5
> Environment: Security is enabled on both clusters and all principals 
> are in the same realm.
>Reporter: Water Chow
> Attachments: HBASE-7963-0.94.patch, HBASE-7963-0.94-v1.patch
>
>
> When security is enabled, HBase VerifyReplication fails for two reasons:
> 1.MapReduce do not have the auth to read the replication paths 
> "/hbase/replication/*" on ZK;
> 2.VerifyReplication does not get the token for slave cluster, it's different 
> from HBASE-7442, this VerifyReplication does not have the output.
> {noformat}
> WARN [main] 
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
> RemoteException connecting to RS
> javax.security.sasl.SaslException: DIGEST-MD5: digest response format 
> violation. Mismatched response.
>at 
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.readStatus(HBaseSaslRpcClient.java:112)
>at 
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:174)
>at 
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConnection(SecureClient.java:177)
>at 
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.access$500(SecureClient.java:85)
>at 
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:284)
>at 
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:281)
>at java.security.AccessController.doPrivileged(Native Method)
>at javax.security.auth.Subject.doAs(Subject.java:396)
>at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>at java.lang.reflect.Method.invoke(Method.java:597)
>at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
>at org.apache.hadoop.hbase.security.User.call(User.java:586)
>at 
> org.apache.hadoop.hbase.security.User.access$700(User.java:50)
>at 
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:440)
>at 
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(SecureClient.java:280)
>at 
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1150)
>at 
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1000)
>at 
> org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(SecureRpcEngine.java:164)
>at $Proxy13.getProtocolVersion(Unknown Source)
>at 
> org.apache.hadoop.hbase.ipc.SecureRpcEngine.getProxy(SecureRpcEngine.java:208)
>at 
> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:335)
>at 
> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:312)
>at 
> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:364)
>at 
> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:236)
>at 
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1313)
>at 
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1269)
>at 
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1256)
>at 
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:965)
>at 
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:860)
>at 
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(

[jira] [Commented] (HBASE-8636) Backport KeyValue Codec to 0.94 (HBASE-7413)

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668878#comment-13668878
 ] 

Sergey Shelukhin commented on HBASE-8636:
-

Actually, I wonder about the relations of compressionContext and codec.
Codec is set on objects like WALEdit, and when setCompressionContext is called 
on WALEdit, compression context is set on the codec. That is kind of confusing, 
both because it's already set as far as I can tell, and because it is set for 
everyone potentially sharing this codec. I wonder if there should just be the 
ability to set one thing?

> Backport KeyValue Codec to 0.94 (HBASE-7413)
> 
>
> Key: HBASE-8636
> URL: https://issues.apache.org/jira/browse/HBASE-8636
> Project: HBase
>  Issue Type: Improvement
>  Components: wal
>Affects Versions: 0.94.9
>Reporter: Jesse Yates
>Assignee: Jesse Yates
>Priority: Minor
> Fix For: 0.94.9
>
> Attachments: HBASE-8636-0.94.patch
>
>
> Backport the KeyValueCodec/WALEditCodec stuff from trunk to 0.94 to support a 
> more pluggable KeyValue serialization. This doesn't include all the Cell code 
> added to trunk, but rather just providing a small layer of indirection for 
> the serialization of a KeyValue in the WAL.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8637) IntegrationTestBigLinkedListWithChaosMonkey uses the wrong table name

[Jeffrey Zhong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668877#comment-13668877
 ] 

Jeffrey Zhong commented on HBASE-8637:
--

I've run the Integration test using the following command several times 
successfully.
{code}
mvn failsafe:integration-test 
-Dit.test=IntegrationTestBigLinkedListWithChaosMonkey
{code}


> IntegrationTestBigLinkedListWithChaosMonkey uses the wrong table name
> -
>
> Key: HBASE-8637
> URL: https://issues.apache.org/jira/browse/HBASE-8637
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Reporter: Elliott Clark
>
> IntegrationTestBigLinkedListWithChaosMonkey creates a table named 
> IntegrationTestBigLinkedListWithChaosMonkey but when inserting data it 
> doesn't insert any data into it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8636) Backport KeyValue Codec to 0.94 (HBASE-7413)

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668875#comment-13668875
 ] 

Sergey Shelukhin commented on HBASE-8636:
-

looks like a straightforward backport. Does it need new tests?

> Backport KeyValue Codec to 0.94 (HBASE-7413)
> 
>
> Key: HBASE-8636
> URL: https://issues.apache.org/jira/browse/HBASE-8636
> Project: HBase
>  Issue Type: Improvement
>  Components: wal
>Affects Versions: 0.94.9
>Reporter: Jesse Yates
>Assignee: Jesse Yates
>Priority: Minor
> Fix For: 0.94.9
>
> Attachments: HBASE-8636-0.94.patch
>
>
> Backport the KeyValueCodec/WALEditCodec stuff from trunk to 0.94 to support a 
> more pluggable KeyValue serialization. This doesn't include all the Cell code 
> added to trunk, but rather just providing a small layer of indirection for 
> the serialization of a KeyValue in the WAL.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8625) Dependency version upgrade

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668874#comment-13668874
 ] 

Enis Soztutar commented on HBASE-8625:
--

If I remember correctly, there were issues with version mismatch between 
Hadoop's guava ours. Hadoop 2 seems to be using guava 11, should we at least 
run a couple of test with hadoop2? 

> Dependency version upgrade
> --
>
> Key: HBASE-8625
> URL: https://issues.apache.org/jira/browse/HBASE-8625
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.98.0, 0.95.0
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
>Priority: Minor
> Fix For: 0.98.0
>
> Attachments: 8625.v1.patch, 8625.v2.patch
>
>
> junit dependency should be scoped "test"
> We should use a newer versions of jaxb-api. One of our 3rd party would prefer 
> a newer one: 
>  javax.xml.bind:jaxb-api:jar:2.1:compile (version managed from 2.2.2)
>  Last is 2.2.4.
>  
> Not mandatory, but should be done:
>  guava 14.0.1
>  netty 3.6.6.Final
>  commons-codec.version 1.8
>  jackson.version 1.9.3

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8637) IntegrationTestBigLinkedListWithChaosMonkey uses the wrong table name

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668871#comment-13668871
 ] 

Enis Soztutar commented on HBASE-8637:
--

Elliot, how did you test this? This is on distributed setup, right? Do you have 
steps to repro? 

> IntegrationTestBigLinkedListWithChaosMonkey uses the wrong table name
> -
>
> Key: HBASE-8637
> URL: https://issues.apache.org/jira/browse/HBASE-8637
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Reporter: Elliott Clark
>
> IntegrationTestBigLinkedListWithChaosMonkey creates a table named 
> IntegrationTestBigLinkedListWithChaosMonkey but when inserting data it 
> doesn't insert any data into it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8638) add logging to compaction policy

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HBASE-8638:


Status: Patch Available  (was: Open)

> add logging to compaction policy
> 
>
> Key: HBASE-8638
> URL: https://issues.apache.org/jira/browse/HBASE-8638
> Project: HBase
>  Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Trivial
> Attachments: HBASE-8638-v0.patch
>
>
> We are seeing some strange patterns with current compaction policy in some 
> contexts (with normal writes, no bulk load). It seems like some logging is 
> needed to understand what is going on, similar to old default policy

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8638) add logging to compaction policy

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HBASE-8638:


Attachment: HBASE-8638-v0.patch

> add logging to compaction policy
> 
>
> Key: HBASE-8638
> URL: https://issues.apache.org/jira/browse/HBASE-8638
> Project: HBase
>  Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Trivial
> Attachments: HBASE-8638-v0.patch
>
>
> We are seeing some strange patterns with current compaction policy in some 
> contexts (with normal writes, no bulk load). It seems like some logging is 
> needed to understand what is going on, similar to old default policy

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8638) add logging to compaction policy

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HBASE-8638:


Issue Type: Improvement  (was: Bug)

> add logging to compaction policy
> 
>
> Key: HBASE-8638
> URL: https://issues.apache.org/jira/browse/HBASE-8638
> Project: HBase
>  Issue Type: Improvement
>Reporter: Sergey Shelukhin
>
> We are seeing some strange patterns with current compaction policy in some 
> contexts (with normal writes, no bulk load). It seems like some logging is 
> needed to understand what is going on, similar to old default policy

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-8638) add logging to compaction policy

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HBASE-8638:
---

Assignee: Sergey Shelukhin

> add logging to compaction policy
> 
>
> Key: HBASE-8638
> URL: https://issues.apache.org/jira/browse/HBASE-8638
> Project: HBase
>  Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Trivial
>
> We are seeing some strange patterns with current compaction policy in some 
> contexts (with normal writes, no bulk load). It seems like some logging is 
> needed to understand what is going on, similar to old default policy

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8638) add logging to compaction policy

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HBASE-8638:


Priority: Trivial  (was: Major)

> add logging to compaction policy
> 
>
> Key: HBASE-8638
> URL: https://issues.apache.org/jira/browse/HBASE-8638
> Project: HBase
>  Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Priority: Trivial
>
> We are seeing some strange patterns with current compaction policy in some 
> contexts (with normal writes, no bulk load). It seems like some logging is 
> needed to understand what is going on, similar to old default policy

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8631) Meta Region First Recovery

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668861#comment-13668861
 ] 

Hadoop QA commented on HBASE-8631:
--

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12585116/hbase-8631-v2.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 6 new 
or modified tests.

{color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 
1.0 profile.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5856//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5856//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5856//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5856//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5856//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5856//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5856//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5856//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5856//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5856//console

This message is automatically generated.

> Meta Region First Recovery
> --
>
> Key: HBASE-8631
> URL: https://issues.apache.org/jira/browse/HBASE-8631
> Project: HBase
>  Issue Type: Bug
>  Components: MTTR
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
> Attachments: hbase-8631.patch, hbase-8631-v2.patch
>
>
> We have a separate wal for meta region. While log splitting logic haven't 
> taken the advantage of this and splitlogworker still picks a wal file 
> randomly. Imaging if we have multiple region servers including meta RS fails 
> about the same time while meta wal is recovered last, all failed regions have 
> to wait meta recovered and then can be online again. 
> The open JIRA is to let splitlogworker to pick a meta wal file firstly and 
> then others.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8609) Make the CopyTable support startRow, stopRow options

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668860#comment-13668860
 ] 

Hudson commented on HBASE-8609:
---

Integrated in HBase-TRUNK #4146 (See 
[https://builds.apache.org/job/HBase-TRUNK/4146/])
HBASE-8609 Make the CopyTable support startRow, stopRow options (Liu 
Shaohui) (Revision 1487032)

 Result = SUCCESS
tedyu : 
Files : 
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/CopyTable.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestCopyTable.java


> Make the CopyTable support startRow, stopRow options
> 
>
> Key: HBASE-8609
> URL: https://issues.apache.org/jira/browse/HBASE-8609
> Project: HBase
>  Issue Type: Improvement
>  Components: mapreduce
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: 0.98.0, 0.95.2
>
> Attachments: 8609-trunk-v4.patch, HBASE-8609-0.94-v0.patch, 
> HBASE-8609-0.94-v1.patch, HBASE-8609-trunk-v0.patch, 
> HBASE-8609-trunk-v1.patch, HBASE-8609-trunk-v2.patch, 
> HBASE-8609-trunk-v3.patch
>
>
> Make the CopyTable support startRow, stopRow options.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-8638) add logging to compaction policy

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HBASE-8638:
---

 Summary: add logging to compaction policy
 Key: HBASE-8638
 URL: https://issues.apache.org/jira/browse/HBASE-8638
 Project: HBase
  Issue Type: Bug
Reporter: Sergey Shelukhin


We are seeing some strange patterns with current compaction policy in some 
contexts (with normal writes, no bulk load). It seems like some logging is 
needed to understand what is going on, similar to old default policy

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8322) Re-enable hbase checksums by default

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668853#comment-13668853
 ] 

Enis Soztutar commented on HBASE-8322:
--

This issue refers to the fix done at HBASE-6868, but reverts the default value 
change done there. We can improve on the warn message saying that your 
checksums won't be verified, and you might get corrupt data if that setting is 
enabled. Would that work? 

> Re-enable hbase checksums by default
> 
>
> Key: HBASE-8322
> URL: https://issues.apache.org/jira/browse/HBASE-8322
> Project: HBase
>  Issue Type: Improvement
>  Components: Filesystem Integration
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 0.98.0, 0.95.1
>
> Attachments: hbase-8322_v1.patch
>
>
> Double checksumming issues in HBase level checksums(HBASE-5074) has been 
> fixed in HBASE-6868. However, that patch also disables hbase checksums by 
> default. I think we should re-enable it by default, and document the 
> interaction with shortcircuit reads. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-8637) IntegrationTestBigLinkedListWithChaosMonkey uses the wrong table name

[Elliott Clark (JIRA)]

Elliott Clark created HBASE-8637:


 Summary: IntegrationTestBigLinkedListWithChaosMonkey uses the 
wrong table name
 Key: HBASE-8637
 URL: https://issues.apache.org/jira/browse/HBASE-8637
 Project: HBase
  Issue Type: Bug
  Components: test
Reporter: Elliott Clark


IntegrationTestBigLinkedListWithChaosMonkey creates a table named 
IntegrationTestBigLinkedListWithChaosMonkey but when inserting data it doesn't 
insert any data into it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8534) fix coverage org.apache.hadoop.hbase.mapreduce

[Nick Dimiduk (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668844#comment-13668844
 ] 

Nick Dimiduk commented on HBASE-8534:
-

It looks like HBASE-8609 went in already, you'll need to update TestCopyTable 
as appropriate.

> fix coverage org.apache.hadoop.hbase.mapreduce
> --
>
> Key: HBASE-8534
> URL: https://issues.apache.org/jira/browse/HBASE-8534
> Project: HBase
>  Issue Type: Test
>Affects Versions: 0.94.8, 0.95.2
>Reporter: Aleksey Gorshkov
> Attachments: HBASE-8534-0.94-d.patch, HBASE-8534-0.94.patch, 
> HBASE-8534-trunk-a.patch, HBASE-8534-trunk-b.patch, HBASE-8534-trunk-c.patch, 
> HBASE-8534-trunk-d.patch, HBASE-8534-trunk.patch
>
>
> fix coverage org.apache.hadoop.hbase.mapreduce
> patch HBASE-8534-0.94.patch for branch-0.94
> patch HBASE-8534-trunk.patch for branch-0.95 and trunk

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8634) Fix potential null pointer dereference in HRegionServer and TableLockChecker

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668839#comment-13668839
 ] 

Hadoop QA commented on HBASE-8634:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12585107/8634-v1.txt
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 
1.0 profile.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   org.apache.hadoop.hbase.master.TestTableLockManager

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5854//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5854//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5854//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5854//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5854//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5854//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5854//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5854//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5854//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5854//console

This message is automatically generated.

> Fix potential null pointer dereference in HRegionServer and TableLockChecker
> 
>
> Key: HBASE-8634
> URL: https://issues.apache.org/jira/browse/HBASE-8634
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: 8634-v1.txt
>
>
> From 
> https://builds.apache.org/job/PreCommit-HBASE-Build/5852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
>  :
> {code}
> Possible null pointer dereference of rpcc in 
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(RpcController, 
> ClientProtos$MultiRequest)
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.regionserver.HRegionServerIn 
> method 
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(RpcController, 
> ClientProtos$MultiRequest)Value loaded from rpccDereferenced at 
> HRegionServer.java:[line 3176]Known null at HRegionServer.java:[line 
> 3174]
> ...
> Possible null pointer dereference of rpcc in 
> org.apache.hadoop.hbase.regionserver.HRegionServer.mutate(RpcController, 
> ClientProtos$MutateRequest)
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.regionserver.HRegionServerIn 
> method 
> org.apache.hadoop.hbase.regionserver.HRegionServer.mutate(RpcController, 
> ClientProtos$MutateRequest)Value loaded from rpccDereferenced at 
> HRegionServer.java:[line 2832]Known null at HRegionServer.java:[line 
> 2830]
> ...
> Possible null pointer dereference of rpcc in 
> org.apache.hadoop.hbase.regionserver.HRegionServer.replay(RpcController, 
> ClientProtos$MultiReque

[jira] [Commented] (HBASE-8636) Backport KeyValue Codec to 0.94 (HBASE-7413)

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668824#comment-13668824
 ] 

Ted Yu commented on HBASE-8636:
---

Codec.java and Decoder.java miss license.

> Backport KeyValue Codec to 0.94 (HBASE-7413)
> 
>
> Key: HBASE-8636
> URL: https://issues.apache.org/jira/browse/HBASE-8636
> Project: HBase
>  Issue Type: Improvement
>  Components: wal
>Affects Versions: 0.94.9
>Reporter: Jesse Yates
>Assignee: Jesse Yates
>Priority: Minor
> Fix For: 0.94.9
>
> Attachments: HBASE-8636-0.94.patch
>
>
> Backport the KeyValueCodec/WALEditCodec stuff from trunk to 0.94 to support a 
> more pluggable KeyValue serialization. This doesn't include all the Cell code 
> added to trunk, but rather just providing a small layer of indirection for 
> the serialization of a KeyValue in the WAL.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8609) Make the CopyTable support startRow, stopRow options

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668817#comment-13668817
 ] 

Hudson commented on HBASE-8609:
---

Integrated in hbase-0.95 #219 (See 
[https://builds.apache.org/job/hbase-0.95/219/])
HBASE-8609 Make the CopyTable support startRow, stopRow options (Liu 
Shaohui) (Revision 1487033)

 Result = FAILURE
tedyu : 
Files : 
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/CopyTable.java
* 
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestCopyTable.java


> Make the CopyTable support startRow, stopRow options
> 
>
> Key: HBASE-8609
> URL: https://issues.apache.org/jira/browse/HBASE-8609
> Project: HBase
>  Issue Type: Improvement
>  Components: mapreduce
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: 0.98.0, 0.95.2
>
> Attachments: 8609-trunk-v4.patch, HBASE-8609-0.94-v0.patch, 
> HBASE-8609-0.94-v1.patch, HBASE-8609-trunk-v0.patch, 
> HBASE-8609-trunk-v1.patch, HBASE-8609-trunk-v2.patch, 
> HBASE-8609-trunk-v3.patch
>
>
> Make the CopyTable support startRow, stopRow options.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8636) Backport KeyValue Codec to 0.94 (HBASE-7413)

[Jesse Yates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesse Yates updated HBASE-8636:
---

Attachment: HBASE-8636-0.94.patch

Attaching backport patch. A majority of the changes are just copying the Codec 
stuff back - everything else is really just a couple lines in the 
SequenceFileLogReader/Writer and the WALEdit.

> Backport KeyValue Codec to 0.94 (HBASE-7413)
> 
>
> Key: HBASE-8636
> URL: https://issues.apache.org/jira/browse/HBASE-8636
> Project: HBase
>  Issue Type: Improvement
>  Components: wal
>Affects Versions: 0.94.9
>Reporter: Jesse Yates
>Assignee: Jesse Yates
>Priority: Minor
> Fix For: 0.94.9
>
> Attachments: HBASE-8636-0.94.patch
>
>
> Backport the KeyValueCodec/WALEditCodec stuff from trunk to 0.94 to support a 
> more pluggable KeyValue serialization. This doesn't include all the Cell code 
> added to trunk, but rather just providing a small layer of indirection for 
> the serialization of a KeyValue in the WAL.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3787) Increment is non-idempotent but client retries RPC

[Nick Dimiduk (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-3787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668808#comment-13668808
 ] 

Nick Dimiduk commented on HBASE-3787:
-

[~sershe] I like the idea of giving users the option to sacrifice throughput in 
exchange for reliability.

> Increment is non-idempotent but client retries RPC
> --
>
> Key: HBASE-3787
> URL: https://issues.apache.org/jira/browse/HBASE-3787
> Project: HBase
>  Issue Type: Bug
>  Components: Client
>Affects Versions: 0.94.4, 0.95.2
>Reporter: dhruba borthakur
>Assignee: Sergey Shelukhin
>Priority: Critical
> Fix For: 0.95.1
>
> Attachments: HBASE-3787-partial.patch, HBASE-3787-v0.patch, 
> HBASE-3787-v1.patch, HBASE-3787-v2.patch, HBASE-3787-v3.patch, 
> HBASE-3787-v4.patch, HBASE-3787-v5.patch, HBASE-3787-v5.patch
>
>
> The HTable.increment() operation is non-idempotent. The client retries the 
> increment RPC a few times (as specified by configuration) before throwing an 
> error to the application. This makes it possible that the same increment call 
> be applied twice at the server.
> For increment operations, is it better to use 
> HConnectionManager.getRegionServerWithoutRetries()? Another  option would be 
> to enhance the IPC module to make the RPC server correctly identify if the 
> RPC is a retry attempt and handle accordingly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8631) Meta Region First Recovery

[Jeffrey Zhong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeffrey Zhong updated HBASE-8631:
-

Attachment: hbase-8631-v2.patch

v2 patch which includes Ted's review comments. Thanks [~te...@apache.org] for 
the review!

> Meta Region First Recovery
> --
>
> Key: HBASE-8631
> URL: https://issues.apache.org/jira/browse/HBASE-8631
> Project: HBase
>  Issue Type: Bug
>  Components: MTTR
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
> Attachments: hbase-8631.patch, hbase-8631-v2.patch
>
>
> We have a separate wal for meta region. While log splitting logic haven't 
> taken the advantage of this and splitlogworker still picks a wal file 
> randomly. Imaging if we have multiple region servers including meta RS fails 
> about the same time while meta wal is recovered last, all failed regions have 
> to wait meta recovered and then can be online again. 
> The open JIRA is to let splitlogworker to pick a meta wal file firstly and 
> then others.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-8636) Backport KeyValue Codec to 0.94 (HBASE-7413)

[Jesse Yates (JIRA)]

Jesse Yates created HBASE-8636:
--

 Summary: Backport KeyValue Codec to 0.94 (HBASE-7413)
 Key: HBASE-8636
 URL: https://issues.apache.org/jira/browse/HBASE-8636
 Project: HBase
  Issue Type: Improvement
  Components: wal
Affects Versions: 0.94.9
Reporter: Jesse Yates
Assignee: Jesse Yates
Priority: Minor
 Fix For: 0.94.9


Backport the KeyValueCodec/WALEditCodec stuff from trunk to 0.94 to support a 
more pluggable KeyValue serialization. This doesn't include all the Cell code 
added to trunk, but rather just providing a small layer of indirection for the 
serialization of a KeyValue in the WAL.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5083) Backport from trunk: Backup HMaster should have http infoport open with link to the active master

[Cody Marcel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cody Marcel updated HBASE-5083:
---

Summary: Backport from trunk: Backup HMaster should have http infoport open 
with link to the active master  (was: Backup HMaster should have http infoport 
open with link to the active master)

> Backport from trunk: Backup HMaster should have http infoport open with link 
> to the active master
> -
>
> Key: HBASE-5083
> URL: https://issues.apache.org/jira/browse/HBASE-5083
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Affects Versions: 0.92.0
>Reporter: Jonathan Hsieh
>Assignee: Cody Marcel
> Fix For: 0.94.8
>
> Attachments: backup_master.png, HBASE-5083.patch, HBASE-5083.patch, 
> master.png
>
>
> Without ssh'ing and jps/ps'ing, it is difficult to see if a backup hmaster is 
> up.  It seems like it would be good for a backup hmaster to have a basic web 
> page up on the info port so that users could see that it is up.  Also it 
> should probably either provide a link to the active master or automatically 
> forward to the active master.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5083) Backup HMaster should have http infoport open with link to the active master

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668799#comment-13668799
 ] 

Hadoop QA commented on HBASE-5083:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12585108/HBASE-5083.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5855//console

This message is automatically generated.

> Backup HMaster should have http infoport open with link to the active master
> 
>
> Key: HBASE-5083
> URL: https://issues.apache.org/jira/browse/HBASE-5083
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Affects Versions: 0.92.0
>Reporter: Jonathan Hsieh
>Assignee: Cody Marcel
> Fix For: 0.94.8
>
> Attachments: backup_master.png, HBASE-5083.patch, HBASE-5083.patch, 
> master.png
>
>
> Without ssh'ing and jps/ps'ing, it is difficult to see if a backup hmaster is 
> up.  It seems like it would be good for a backup hmaster to have a basic web 
> page up on the info port so that users could see that it is up.  Also it 
> should probably either provide a link to the active master or automatically 
> forward to the active master.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7055) port HBASE-6371 tier-based compaction from 0.89-fb to trunk (with changes)

[Nick Dimiduk (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668797#comment-13668797
 ] 

Nick Dimiduk commented on HBASE-7055:
-

bq. My sense is no one will use is though just because it so complicated 
requiring so much operator interjection and study.

[~stack] have you developed cold feet about inclusion of this policy on trunk?

[~sershe] FYI, v7.patch no longer applies cleanly on trunk.

> port HBASE-6371 tier-based compaction from 0.89-fb to trunk (with changes)
> --
>
> Key: HBASE-7055
> URL: https://issues.apache.org/jira/browse/HBASE-7055
> Project: HBase
>  Issue Type: Task
>  Components: Compaction
>Affects Versions: 0.95.2
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.95.1
>
> Attachments: HBASE-6371-squashed.patch, HBASE-6371-v2-squashed.patch, 
> HBASE-6371-v3-refactor-only-squashed.patch, 
> HBASE-6371-v4-refactor-only-squashed.patch, 
> HBASE-6371-v5-refactor-only-squashed.patch, HBASE-7055-v0.patch, 
> HBASE-7055-v1.patch, HBASE-7055-v2.patch, HBASE-7055-v3.patch, 
> HBASE-7055-v4.patch, HBASE-7055-v5.patch, HBASE-7055-v6.patch, 
> HBASE-7055-v7.patch, HBASE-7055-v7.patch
>
>
> See HBASE-6371 for details.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5083) Backup HMaster should have http infoport open with link to the active master

[Cody Marcel (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668796#comment-13668796
 ] 

Cody Marcel commented on HBASE-5083:


Review board is up now with the fixed diff.

> Backup HMaster should have http infoport open with link to the active master
> 
>
> Key: HBASE-5083
> URL: https://issues.apache.org/jira/browse/HBASE-5083
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Affects Versions: 0.92.0
>Reporter: Jonathan Hsieh
>Assignee: Cody Marcel
> Fix For: 0.94.8
>
> Attachments: backup_master.png, HBASE-5083.patch, HBASE-5083.patch, 
> master.png
>
>
> Without ssh'ing and jps/ps'ing, it is difficult to see if a backup hmaster is 
> up.  It seems like it would be good for a backup hmaster to have a basic web 
> page up on the info port so that users could see that it is up.  Also it 
> should probably either provide a link to the active master or automatically 
> forward to the active master.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8322) Re-enable hbase checksums by default

[Jean-Daniel Cryans (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668795#comment-13668795
 ] 

Jean-Daniel Cryans commented on HBASE-8322:
---

bq. My understanding is that the best practice is to not run with checksums 
off, hence the warning

If that's indeed what they would be doing, how would it relate to HBASE-6868 
then? It seems to me that a more general message like "kiss your data goodbye" 
would be more appropriate ;)

> Re-enable hbase checksums by default
> 
>
> Key: HBASE-8322
> URL: https://issues.apache.org/jira/browse/HBASE-8322
> Project: HBase
>  Issue Type: Improvement
>  Components: Filesystem Integration
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 0.98.0, 0.95.1
>
> Attachments: hbase-8322_v1.patch
>
>
> Double checksumming issues in HBase level checksums(HBASE-5074) has been 
> fixed in HBASE-6868. However, that patch also disables hbase checksums by 
> default. I think we should re-enable it by default, and document the 
> interaction with shortcircuit reads. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5083) Backup HMaster should have http infoport open with link to the active master

[Jonathan Hsieh (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668794#comment-13668794
 ] 

Jonathan Hsieh commented on HBASE-5083:
---

I didn't realize that (I've been testing against at 0.95.0 and it doesn't seem 
to be there).  Can you find the jira, add a link to it and also rename this to 
be a backport?  If we port to 0.94 we should have it in 0.95 as well!



> Backup HMaster should have http infoport open with link to the active master
> 
>
> Key: HBASE-5083
> URL: https://issues.apache.org/jira/browse/HBASE-5083
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Affects Versions: 0.92.0
>Reporter: Jonathan Hsieh
>Assignee: Cody Marcel
> Fix For: 0.94.8
>
> Attachments: backup_master.png, HBASE-5083.patch, HBASE-5083.patch, 
> master.png
>
>
> Without ssh'ing and jps/ps'ing, it is difficult to see if a backup hmaster is 
> up.  It seems like it would be good for a backup hmaster to have a basic web 
> page up on the info port so that users could see that it is up.  Also it 
> should probably either provide a link to the active master or automatically 
> forward to the active master.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5083) Backup HMaster should have http infoport open with link to the active master

[Cody Marcel (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668791#comment-13668791
 ] 

Cody Marcel commented on HBASE-5083:


[~jmhsieh] THis already exists in trunk, mostly. I am backporting to 0.94.

> Backup HMaster should have http infoport open with link to the active master
> 
>
> Key: HBASE-5083
> URL: https://issues.apache.org/jira/browse/HBASE-5083
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Affects Versions: 0.92.0
>Reporter: Jonathan Hsieh
>Assignee: Cody Marcel
> Fix For: 0.94.8
>
> Attachments: backup_master.png, HBASE-5083.patch, HBASE-5083.patch, 
> master.png
>
>
> Without ssh'ing and jps/ps'ing, it is difficult to see if a backup hmaster is 
> up.  It seems like it would be good for a backup hmaster to have a basic web 
> page up on the info port so that users could see that it is up.  Also it 
> should probably either provide a link to the active master or automatically 
> forward to the active master.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5083) Backup HMaster should have http infoport open with link to the active master

[Cody Marcel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cody Marcel updated HBASE-5083:
---

Attachment: HBASE-5083.patch

Merged local commits to correct the diff.

> Backup HMaster should have http infoport open with link to the active master
> 
>
> Key: HBASE-5083
> URL: https://issues.apache.org/jira/browse/HBASE-5083
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Affects Versions: 0.92.0
>Reporter: Jonathan Hsieh
>Assignee: Cody Marcel
> Fix For: 0.94.8
>
> Attachments: backup_master.png, HBASE-5083.patch, HBASE-5083.patch, 
> master.png
>
>
> Without ssh'ing and jps/ps'ing, it is difficult to see if a backup hmaster is 
> up.  It seems like it would be good for a backup hmaster to have a basic web 
> page up on the info port so that users could see that it is up.  Also it 
> should probably either provide a link to the active master or automatically 
> forward to the active master.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-8634) Fix potential null pointer dereference in HRegionServer and TableLockChecker

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu reassigned HBASE-8634:
-

Assignee: Ted Yu

> Fix potential null pointer dereference in HRegionServer and TableLockChecker
> 
>
> Key: HBASE-8634
> URL: https://issues.apache.org/jira/browse/HBASE-8634
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: 8634-v1.txt
>
>
> From 
> https://builds.apache.org/job/PreCommit-HBASE-Build/5852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
>  :
> {code}
> Possible null pointer dereference of rpcc in 
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(RpcController, 
> ClientProtos$MultiRequest)
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.regionserver.HRegionServerIn 
> method 
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(RpcController, 
> ClientProtos$MultiRequest)Value loaded from rpccDereferenced at 
> HRegionServer.java:[line 3176]Known null at HRegionServer.java:[line 
> 3174]
> ...
> Possible null pointer dereference of rpcc in 
> org.apache.hadoop.hbase.regionserver.HRegionServer.mutate(RpcController, 
> ClientProtos$MutateRequest)
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.regionserver.HRegionServerIn 
> method 
> org.apache.hadoop.hbase.regionserver.HRegionServer.mutate(RpcController, 
> ClientProtos$MutateRequest)Value loaded from rpccDereferenced at 
> HRegionServer.java:[line 2832]Known null at HRegionServer.java:[line 
> 2830]
> ...
> Possible null pointer dereference of rpcc in 
> org.apache.hadoop.hbase.regionserver.HRegionServer.replay(RpcController, 
> ClientProtos$MultiRequest)
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.regionserver.HRegionServerIn 
> method 
> org.apache.hadoop.hbase.regionserver.HRegionServer.replay(RpcController, 
> ClientProtos$MultiRequest)Value loaded from rpccDereferenced at 
> HRegionServer.java:[line 3742]Known null at HRegionServer.java:[line 
> 3740]
> ...
> Possible null pointer dereference of data in 
> org.apache.hadoop.hbase.util.hbck.TableLockChecker$1.handleMetadata(byte[])
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.util.hbck.TableLockChecker$1In 
> method 
> org.apache.hadoop.hbase.util.hbck.TableLockChecker$1.handleMetadata(byte[])Value
>  loaded from dataDereferenced at TableLockChecker.java:[line 
> 68]Known null at TableLockChecker.java:[line 60]
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8634) Fix potential null pointer dereference in HRegionServer and TableLockChecker

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-8634:
--

Attachment: 8634-v1.txt

> Fix potential null pointer dereference in HRegionServer and TableLockChecker
> 
>
> Key: HBASE-8634
> URL: https://issues.apache.org/jira/browse/HBASE-8634
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
> Attachments: 8634-v1.txt
>
>
> From 
> https://builds.apache.org/job/PreCommit-HBASE-Build/5852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
>  :
> {code}
> Possible null pointer dereference of rpcc in 
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(RpcController, 
> ClientProtos$MultiRequest)
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.regionserver.HRegionServerIn 
> method 
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(RpcController, 
> ClientProtos$MultiRequest)Value loaded from rpccDereferenced at 
> HRegionServer.java:[line 3176]Known null at HRegionServer.java:[line 
> 3174]
> ...
> Possible null pointer dereference of rpcc in 
> org.apache.hadoop.hbase.regionserver.HRegionServer.mutate(RpcController, 
> ClientProtos$MutateRequest)
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.regionserver.HRegionServerIn 
> method 
> org.apache.hadoop.hbase.regionserver.HRegionServer.mutate(RpcController, 
> ClientProtos$MutateRequest)Value loaded from rpccDereferenced at 
> HRegionServer.java:[line 2832]Known null at HRegionServer.java:[line 
> 2830]
> ...
> Possible null pointer dereference of rpcc in 
> org.apache.hadoop.hbase.regionserver.HRegionServer.replay(RpcController, 
> ClientProtos$MultiRequest)
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.regionserver.HRegionServerIn 
> method 
> org.apache.hadoop.hbase.regionserver.HRegionServer.replay(RpcController, 
> ClientProtos$MultiRequest)Value loaded from rpccDereferenced at 
> HRegionServer.java:[line 3742]Known null at HRegionServer.java:[line 
> 3740]
> ...
> Possible null pointer dereference of data in 
> org.apache.hadoop.hbase.util.hbck.TableLockChecker$1.handleMetadata(byte[])
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.util.hbck.TableLockChecker$1In 
> method 
> org.apache.hadoop.hbase.util.hbck.TableLockChecker$1.handleMetadata(byte[])Value
>  loaded from dataDereferenced at TableLockChecker.java:[line 
> 68]Known null at TableLockChecker.java:[line 60]
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8634) Fix potential null pointer dereference in HRegionServer and TableLockChecker

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-8634:
--

Status: Patch Available  (was: Open)

> Fix potential null pointer dereference in HRegionServer and TableLockChecker
> 
>
> Key: HBASE-8634
> URL: https://issues.apache.org/jira/browse/HBASE-8634
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: 8634-v1.txt
>
>
> From 
> https://builds.apache.org/job/PreCommit-HBASE-Build/5852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
>  :
> {code}
> Possible null pointer dereference of rpcc in 
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(RpcController, 
> ClientProtos$MultiRequest)
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.regionserver.HRegionServerIn 
> method 
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(RpcController, 
> ClientProtos$MultiRequest)Value loaded from rpccDereferenced at 
> HRegionServer.java:[line 3176]Known null at HRegionServer.java:[line 
> 3174]
> ...
> Possible null pointer dereference of rpcc in 
> org.apache.hadoop.hbase.regionserver.HRegionServer.mutate(RpcController, 
> ClientProtos$MutateRequest)
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.regionserver.HRegionServerIn 
> method 
> org.apache.hadoop.hbase.regionserver.HRegionServer.mutate(RpcController, 
> ClientProtos$MutateRequest)Value loaded from rpccDereferenced at 
> HRegionServer.java:[line 2832]Known null at HRegionServer.java:[line 
> 2830]
> ...
> Possible null pointer dereference of rpcc in 
> org.apache.hadoop.hbase.regionserver.HRegionServer.replay(RpcController, 
> ClientProtos$MultiRequest)
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.regionserver.HRegionServerIn 
> method 
> org.apache.hadoop.hbase.regionserver.HRegionServer.replay(RpcController, 
> ClientProtos$MultiRequest)Value loaded from rpccDereferenced at 
> HRegionServer.java:[line 3742]Known null at HRegionServer.java:[line 
> 3740]
> ...
> Possible null pointer dereference of data in 
> org.apache.hadoop.hbase.util.hbck.TableLockChecker$1.handleMetadata(byte[])
> 
> 
> 
> 
> 
> Bug type NP_NULL_ON_SOME_PATH (click for 
> details)
> In class org.apache.hadoop.hbase.util.hbck.TableLockChecker$1In 
> method 
> org.apache.hadoop.hbase.util.hbck.TableLockChecker$1.handleMetadata(byte[])Value
>  loaded from dataDereferenced at TableLockChecker.java:[line 
> 68]Known null at TableLockChecker.java:[line 60]
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8322) Re-enable hbase checksums by default

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668783#comment-13668783
 ] 

Enis Soztutar commented on HBASE-8322:
--

bq. But what about if the user really wants to set it to true? Live with the 
WARN message?
My understanding is that the best practice is to not run with checksums off, 
hence the warning. They should suffer the warning, if they should chose to 
ignore the wisdom :) 

> Re-enable hbase checksums by default
> 
>
> Key: HBASE-8322
> URL: https://issues.apache.org/jira/browse/HBASE-8322
> Project: HBase
>  Issue Type: Improvement
>  Components: Filesystem Integration
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 0.98.0, 0.95.1
>
> Attachments: hbase-8322_v1.patch
>
>
> Double checksumming issues in HBase level checksums(HBASE-5074) has been 
> fixed in HBASE-6868. However, that patch also disables hbase checksums by 
> default. I think we should re-enable it by default, and document the 
> interaction with shortcircuit reads. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5083) Backup HMaster should have http infoport open with link to the active master

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668777#comment-13668777
 ] 

Hadoop QA commented on HBASE-5083:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12585102/HBASE-5083.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5853//console

This message is automatically generated.

> Backup HMaster should have http infoport open with link to the active master
> 
>
> Key: HBASE-5083
> URL: https://issues.apache.org/jira/browse/HBASE-5083
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Affects Versions: 0.92.0
>Reporter: Jonathan Hsieh
>Assignee: Cody Marcel
> Fix For: 0.94.8
>
> Attachments: backup_master.png, HBASE-5083.patch, master.png
>
>
> Without ssh'ing and jps/ps'ing, it is difficult to see if a backup hmaster is 
> up.  It seems like it would be good for a backup hmaster to have a basic web 
> page up on the info port so that users could see that it is up.  Also it 
> should probably either provide a link to the active master or automatically 
> forward to the active master.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5083) Backup HMaster should have http infoport open with link to the active master

[Jonathan Hsieh (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668779#comment-13668779
 ] 

Jonathan Hsieh commented on HBASE-5083:
---

looks very 0.94. there will be a version for trunk as well?

> Backup HMaster should have http infoport open with link to the active master
> 
>
> Key: HBASE-5083
> URL: https://issues.apache.org/jira/browse/HBASE-5083
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Affects Versions: 0.92.0
>Reporter: Jonathan Hsieh
>Assignee: Cody Marcel
> Fix For: 0.94.8
>
> Attachments: backup_master.png, HBASE-5083.patch, master.png
>
>
> Without ssh'ing and jps/ps'ing, it is difficult to see if a backup hmaster is 
> up.  It seems like it would be good for a backup hmaster to have a basic web 
> page up on the info port so that users could see that it is up.  Also it 
> should probably either provide a link to the active master or automatically 
> forward to the active master.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5083) Backup HMaster should have http infoport open with link to the active master

[Jesse Yates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668776#comment-13668776
 ] 

Jesse Yates commented on HBASE-5083:


hmmm, something is wonky with this patch... Looks like its diffing itself. 
Wanna try again?

> Backup HMaster should have http infoport open with link to the active master
> 
>
> Key: HBASE-5083
> URL: https://issues.apache.org/jira/browse/HBASE-5083
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Affects Versions: 0.92.0
>Reporter: Jonathan Hsieh
>Assignee: Cody Marcel
> Fix For: 0.94.8
>
> Attachments: backup_master.png, HBASE-5083.patch, master.png
>
>
> Without ssh'ing and jps/ps'ing, it is difficult to see if a backup hmaster is 
> up.  It seems like it would be good for a backup hmaster to have a basic web 
> page up on the info port so that users could see that it is up.  Also it 
> should probably either provide a link to the active master or automatically 
> forward to the active master.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-8635) Define prefetcher.resultsize.max as percentage

[Ted Yu (JIRA)]

Ted Yu created HBASE-8635:
-

 Summary: Define prefetcher.resultsize.max as percentage
 Key: HBASE-8635
 URL: https://issues.apache.org/jira/browse/HBASE-8635
 Project: HBase
  Issue Type: Improvement
Reporter: Ted Yu
Priority: Minor


Currently "hbase.hregionserver.prefetcher.resultsize.max" defines global limit 
for prefetching.
The default value is 256MB.

It would be more flexible to define this measure as a percentage of the heap.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5083) Backup HMaster should have http infoport open with link to the active master

[Cody Marcel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cody Marcel updated HBASE-5083:
---

Attachment: HBASE-5083.patch

> Backup HMaster should have http infoport open with link to the active master
> 
>
> Key: HBASE-5083
> URL: https://issues.apache.org/jira/browse/HBASE-5083
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Affects Versions: 0.92.0
>Reporter: Jonathan Hsieh
>Assignee: Cody Marcel
> Fix For: 0.94.8
>
> Attachments: backup_master.png, HBASE-5083.patch, master.png
>
>
> Without ssh'ing and jps/ps'ing, it is difficult to see if a backup hmaster is 
> up.  It seems like it would be good for a backup hmaster to have a basic web 
> page up on the info port so that users could see that it is up.  Also it 
> should probably either provide a link to the active master or automatically 
> forward to the active master.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-8634) Fix potential null pointer dereference in HRegionServer and TableLockChecker

[Ted Yu (JIRA)]

Ted Yu created HBASE-8634:
-

 Summary: Fix potential null pointer dereference in HRegionServer 
and TableLockChecker
 Key: HBASE-8634
 URL: https://issues.apache.org/jira/browse/HBASE-8634
 Project: HBase
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


>From 
>https://builds.apache.org/job/PreCommit-HBASE-Build/5852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
> :
{code}
Possible null pointer dereference of rpcc in 
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(RpcController, 
ClientProtos$MultiRequest)





Bug type NP_NULL_ON_SOME_PATH (click for 
details)
In class org.apache.hadoop.hbase.regionserver.HRegionServerIn method 
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(RpcController, 
ClientProtos$MultiRequest)Value loaded from rpccDereferenced at 
HRegionServer.java:[line 3176]Known null at HRegionServer.java:[line 
3174]
...
Possible null pointer dereference of rpcc in 
org.apache.hadoop.hbase.regionserver.HRegionServer.mutate(RpcController, 
ClientProtos$MutateRequest)





Bug type NP_NULL_ON_SOME_PATH (click for 
details)
In class org.apache.hadoop.hbase.regionserver.HRegionServerIn method 
org.apache.hadoop.hbase.regionserver.HRegionServer.mutate(RpcController, 
ClientProtos$MutateRequest)Value loaded from rpccDereferenced at 
HRegionServer.java:[line 2832]Known null at HRegionServer.java:[line 
2830]
...
Possible null pointer dereference of rpcc in 
org.apache.hadoop.hbase.regionserver.HRegionServer.replay(RpcController, 
ClientProtos$MultiRequest)





Bug type NP_NULL_ON_SOME_PATH (click for 
details)
In class org.apache.hadoop.hbase.regionserver.HRegionServerIn method 
org.apache.hadoop.hbase.regionserver.HRegionServer.replay(RpcController, 
ClientProtos$MultiRequest)Value loaded from rpccDereferenced at 
HRegionServer.java:[line 3742]Known null at HRegionServer.java:[line 
3740]
...
Possible null pointer dereference of data in 
org.apache.hadoop.hbase.util.hbck.TableLockChecker$1.handleMetadata(byte[])





Bug type NP_NULL_ON_SOME_PATH (click for 
details)
In class org.apache.hadoop.hbase.util.hbck.TableLockChecker$1In 
method 
org.apache.hadoop.hbase.util.hbck.TableLockChecker$1.handleMetadata(byte[])Value
 loaded from dataDereferenced at TableLockChecker.java:[line 68]Known 
null at TableLockChecker.java:[line 60]
{code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5083) Backup HMaster should have http infoport open with link to the active master

[Cody Marcel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cody Marcel updated HBASE-5083:
---

Fix Version/s: 0.94.8
 Release Note: This patch includes adding a table for the backup masters to 
the UI of the master server. It also has a link from the backup master server's 
UI to point to the primary.
   Status: Patch Available  (was: Open)

I tried setting up a review for this, but it failed: The file 
'src/main/jamon/org/apache/hadoop/hbase/tmpl/master/BackupMasterStatusTmpl.jamon'
 (r238d2ad) could not be found in the repository

> Backup HMaster should have http infoport open with link to the active master
> 
>
> Key: HBASE-5083
> URL: https://issues.apache.org/jira/browse/HBASE-5083
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Affects Versions: 0.92.0
>Reporter: Jonathan Hsieh
>Assignee: Cody Marcel
> Fix For: 0.94.8
>
> Attachments: backup_master.png, master.png
>
>
> Without ssh'ing and jps/ps'ing, it is difficult to see if a backup hmaster is 
> up.  It seems like it would be good for a backup hmaster to have a basic web 
> page up on the info port so that users could see that it is up.  Also it 
> should probably either provide a link to the active master or automatically 
> forward to the active master.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5083) Backup HMaster should have http infoport open with link to the active master

[Cody Marcel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cody Marcel updated HBASE-5083:
---

Attachment: master.png
backup_master.png

Screen shots for the UI

> Backup HMaster should have http infoport open with link to the active master
> 
>
> Key: HBASE-5083
> URL: https://issues.apache.org/jira/browse/HBASE-5083
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Affects Versions: 0.92.0
>Reporter: Jonathan Hsieh
>Assignee: Cody Marcel
> Fix For: 0.94.8
>
> Attachments: backup_master.png, master.png
>
>
> Without ssh'ing and jps/ps'ing, it is difficult to see if a backup hmaster is 
> up.  It seems like it would be good for a backup hmaster to have a basic web 
> page up on the info port so that users could see that it is up.  Also it 
> should probably either provide a link to the active master or automatically 
> forward to the active master.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8322) Re-enable hbase checksums by default

[Jean-Daniel Cryans (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668764#comment-13668764
 ] 

Jean-Daniel Cryans commented on HBASE-8322:
---

bq. The thing is that we want checksums for everything other than reading hfile 
blocks.

Ah ok, that makes sense. But what about if the user really wants to set it to 
true? Live with the WARN message?

bq. Was there performance comparison between HBase checksum for HFiles vs. 
hadoop checksum through native CRC ?

http://apache-hbase.679495.n3.nabble.com/Unscientific-comparison-of-fully-cached-zipfian-reading-td4042911.html

4th VS 7th column.

> Re-enable hbase checksums by default
> 
>
> Key: HBASE-8322
> URL: https://issues.apache.org/jira/browse/HBASE-8322
> Project: HBase
>  Issue Type: Improvement
>  Components: Filesystem Integration
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 0.98.0, 0.95.1
>
> Attachments: hbase-8322_v1.patch
>
>
> Double checksumming issues in HBase level checksums(HBASE-5074) has been 
> fixed in HBASE-6868. However, that patch also disables hbase checksums by 
> default. I think we should re-enable it by default, and document the 
> interaction with shortcircuit reads. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7839) Add a script to simulate a dead machine during the integration tests

[Nick Dimiduk (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-7839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668740#comment-13668740
 ] 

Nick Dimiduk commented on HBASE-7839:
-

Why is this in C instead of something like Ruby or Python? Python os package 
contains all these APIs, as far as I can tell.

> Add a script to simulate a dead machine during the integration tests
> 
>
> Key: HBASE-7839
> URL: https://issues.apache.org/jira/browse/HBASE-7839
> Project: HBase
>  Issue Type: New Feature
>  Components: test
>Affects Versions: 0.95.2
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
>Priority: Minor
> Fix For: 0.95.1
>
> Attachments: 7839.v1.patch
>
>
> A kill -9 cannot be used to simulate a dead box. With a kill -9:
> 1) the sockets are closed by the OS
> 2) the next calls will get immediately a connection refused.
> Physically unplugging the cable does work.
> To automate this, the best solution seems to be using a firewall config.
> It's a little bit complicated however, because we need to refuse the incoming 
> packets. If we block the out connection, it's detected by the OS and there is 
> no wait on the socket.
> Moreover you need to be root, or use a program with a sticky bit to launch 
> the script that changes the firewall config.
> This will be usable from the integration tests.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-8633) Document namespaces in HBase book

[Enis Soztutar (JIRA)]

Enis Soztutar created HBASE-8633:


 Summary: Document namespaces in HBase book
 Key: HBASE-8633
 URL: https://issues.apache.org/jira/browse/HBASE-8633
 Project: HBase
  Issue Type: Sub-task
Reporter: Enis Soztutar


We need to add documentation about the namespaces feature. It should go into 
the HBase book. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6295) Possible performance improvement in client batch operations: presplit and send in background

[Nicolas Liochon (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668712#comment-13668712
 ] 

Nicolas Liochon commented on HBASE-6295:


bq. can you please file a jira for Process code de-duping/removal
I've removed the Process class already :-)
I need to look at these test errors. I haven't got them locally.

> Possible performance improvement in client batch operations: presplit and 
> send in background
> 
>
> Key: HBASE-6295
> URL: https://issues.apache.org/jira/browse/HBASE-6295
> Project: HBase
>  Issue Type: Improvement
>  Components: Client, Performance
>Affects Versions: 0.95.2
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
>  Labels: noob
> Fix For: 0.98.0
>
> Attachments: 6295.v1.patch, 6295.v2.patch, 6295.v3.patch, 
> 6295.v4.patch, 6295.v5.patch, 6295.v6.patch, 6295.v8.patch, 6295.v9.patch
>
>
> today batch algo is:
> {noformat}
> for Operation o: List{
>   add o to todolist
>   if todolist > maxsize or o last in list
> split todolist per location
> send split lists to region servers
> clear todolist
> wait
> }
> {noformat}
> We could:
> - create immediately the final object instead of an intermediate array
> - split per location immediately
> - instead of sending when the list as a whole is full, send it when there is 
> enough data for a single location
> It would be:
> {noformat}
> for Operation o: List{
>   get location
>   add o to todo location.todolist
>   if (location.todolist > maxLocationSize)
> send location.todolist to region server 
> clear location.todolist
> // don't wait, continue the loop
> }
> send remaining
> wait
> {noformat}
> It's not trivial to write if you add error management: retried list must be 
> shared with the operations added in the todolist. But it's doable.
> It's interesting mainly for 'big' writes

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8632) TestAccessController#testGlobalAuthorizationForNewRegisteredRS fails intermittently due to prolonged region movement

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-8632:
--

Summary: TestAccessController#testGlobalAuthorizationForNewRegisteredRS 
fails intermittently due to prolonged region movement  (was: 
TestAccessController#testGlobalAuthorizationForNewRegisteredRS fails 
intermittently due to )

> TestAccessController#testGlobalAuthorizationForNewRegisteredRS fails 
> intermittently due to prolonged region movement
> 
>
> Key: HBASE-8632
> URL: https://issues.apache.org/jira/browse/HBASE-8632
> Project: HBase
>  Issue Type: Test
>Reporter: Ted Yu
>
> Here is test output from 
> https://builds.apache.org/job/PreCommit-HBASE-Build/5852/testReport/org.apache.hadoop.hbase.security.access/TestAccessController/testGlobalAuthorizationForNewRegisteredRS/:
> {code}
> 2013-05-28 19:54:07,360 INFO  [PRI IPC Server handler 0 on 58791] 
> regionserver.HRegionServer(3535): Received close region: 
> 4700bdd89a351b506da910dd6a268d44Transitioning in ZK: yes. Version of ZK 
> closing node:0. Destination 
> server:asf001.sp2.ygridcore.net,36189,1369770847288
> 2013-05-28 19:54:07,360 DEBUG 
> [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,58791,136977068-1] 
> handler.CloseRegionHandler(125): Processing close of 
> testtable2,,1369770846203.4700bdd89a351b506da910dd6a268d44.
> 2013-05-28 19:54:07,360 DEBUG [IPC Server handler 2 on 55436] 
> master.AssignmentManager(1672): Sent CLOSE to 
> asf001.sp2.ygridcore.net,58791,136977068 for region 
> testtable2,,1369770846203.4700bdd89a351b506da910dd6a268d44.
> 2013-05-28 19:54:07,360 DEBUG [pool-1-thread-1] 
> access.TestAccessController(1955): Waiting for region to be opened. Already 
> retried 0 times.
> 2013-05-28 19:54:07,361 DEBUG 
> [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,58791,136977068-1] 
> regionserver.HRegion(928): Closing 
> testtable2,,1369770846203.4700bdd89a351b506da910dd6a268d44.: disabling 
> compactions & flushes
> 2013-05-28 19:54:07,361 DEBUG 
> [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,58791,136977068-1] 
> regionserver.HRegion(950): Updates disabled for region 
> testtable2,,1369770846203.4700bdd89a351b506da910dd6a268d44.
> 2013-05-28 19:54:07,362 INFO  
> [StoreCloserThread-testtable2,,1369770846203.4700bdd89a351b506da910dd6a268d44.-1]
>  regionserver.HStore(663): Closed f1
> 2013-05-28 19:54:07,362 INFO  
> [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,58791,136977068-1] 
> regionserver.HRegion(1007): Closed 
> testtable2,,1369770846203.4700bdd89a351b506da910dd6a268d44.
> 2013-05-28 19:54:07,362 INFO  
> [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,58791,136977068-1] 
> regionserver.HRegionServer(4060): Adding moved region record: 
> 4700bdd89a351b506da910dd6a268d44 to 
> asf001.sp2.ygridcore.net,36189,1369770847288:36189 as of 1
> 2013-05-28 19:54:07,363 DEBUG 
> [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,58791,136977068-1] 
> zookeeper.ZKAssign(786): regionserver:58791-0x13eecb1db960001 Attempting to 
> transition node 4700bdd89a351b506da910dd6a268d44 from M_ZK_REGION_CLOSING to 
> RS_ZK_REGION_CLOSED
> 2013-05-28 19:54:07,561 DEBUG [pool-1-thread-1] 
> access.TestAccessController(1955): Waiting for region to be opened. Already 
> retried 1 times.
> 2013-05-28 19:54:07,761 DEBUG [pool-1-thread-1] 
> access.TestAccessController(1955): Waiting for region to be opened. Already 
> retried 2 times.
> 2013-05-28 19:54:07,961 DEBUG [pool-1-thread-1] 
> access.TestAccessController(1955): Waiting for region to be opened. Already 
> retried 3 times.
> 2013-05-28 19:54:08,162 DEBUG [pool-1-thread-1] 
> access.TestAccessController(1955): Waiting for region to be opened. Already 
> retried 4 times.
> 2013-05-28 19:54:08,362 DEBUG [pool-1-thread-1] 
> access.TestAccessController(1955): Waiting for region to be opened. Already 
> retried 5 times.
> 2013-05-28 19:54:08,562 DEBUG [pool-1-thread-1] 
> access.TestAccessController(1955): Waiting for region to be opened. Already 
> retried 6 times.
> 2013-05-28 19:54:08,763 DEBUG [pool-1-thread-1] 
> access.TestAccessController(1955): Waiting for region to be opened. Already 
> retried 7 times.
> 2013-05-28 19:54:08,963 DEBUG [pool-1-thread-1] 
> access.TestAccessController(1955): Waiting for region to be opened. Already 
> retried 8 times.
> 2013-05-28 19:54:09,164 INFO  [pool-1-thread-1] client.HBaseAdmin$5(834): 
> Started disable of testtable
> 2013-05-28 19:54:09,320 DEBUG [pool-1-thread-1-EventThread] 
> zookeeper.ZooKeeperWatcher(307): master:55436-0x13eecb1db96 Received 
> ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, 
> path=/hbase/region-in-transition/4700bdd89a351b506da910dd6a268d44
> 2013-05-28 19:54:09,320 DEBUG 
> [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,58791,136977068-1] 
> 

[jira] [Created] (HBASE-8632) TestAccessController#testGlobalAuthorizationForNewRegisteredRS fails intermittently due to

[Ted Yu (JIRA)]

Ted Yu created HBASE-8632:
-

 Summary: 
TestAccessController#testGlobalAuthorizationForNewRegisteredRS fails 
intermittently due to 
 Key: HBASE-8632
 URL: https://issues.apache.org/jira/browse/HBASE-8632
 Project: HBase
  Issue Type: Test
Reporter: Ted Yu


Here is test output from 
https://builds.apache.org/job/PreCommit-HBASE-Build/5852/testReport/org.apache.hadoop.hbase.security.access/TestAccessController/testGlobalAuthorizationForNewRegisteredRS/:
{code}
2013-05-28 19:54:07,360 INFO  [PRI IPC Server handler 0 on 58791] 
regionserver.HRegionServer(3535): Received close region: 
4700bdd89a351b506da910dd6a268d44Transitioning in ZK: yes. Version of ZK closing 
node:0. Destination server:asf001.sp2.ygridcore.net,36189,1369770847288
2013-05-28 19:54:07,360 DEBUG 
[RS_CLOSE_REGION-asf001.sp2.ygridcore.net,58791,136977068-1] 
handler.CloseRegionHandler(125): Processing close of 
testtable2,,1369770846203.4700bdd89a351b506da910dd6a268d44.
2013-05-28 19:54:07,360 DEBUG [IPC Server handler 2 on 55436] 
master.AssignmentManager(1672): Sent CLOSE to 
asf001.sp2.ygridcore.net,58791,136977068 for region 
testtable2,,1369770846203.4700bdd89a351b506da910dd6a268d44.
2013-05-28 19:54:07,360 DEBUG [pool-1-thread-1] 
access.TestAccessController(1955): Waiting for region to be opened. Already 
retried 0 times.
2013-05-28 19:54:07,361 DEBUG 
[RS_CLOSE_REGION-asf001.sp2.ygridcore.net,58791,136977068-1] 
regionserver.HRegion(928): Closing 
testtable2,,1369770846203.4700bdd89a351b506da910dd6a268d44.: disabling 
compactions & flushes
2013-05-28 19:54:07,361 DEBUG 
[RS_CLOSE_REGION-asf001.sp2.ygridcore.net,58791,136977068-1] 
regionserver.HRegion(950): Updates disabled for region 
testtable2,,1369770846203.4700bdd89a351b506da910dd6a268d44.
2013-05-28 19:54:07,362 INFO  
[StoreCloserThread-testtable2,,1369770846203.4700bdd89a351b506da910dd6a268d44.-1]
 regionserver.HStore(663): Closed f1
2013-05-28 19:54:07,362 INFO  
[RS_CLOSE_REGION-asf001.sp2.ygridcore.net,58791,136977068-1] 
regionserver.HRegion(1007): Closed 
testtable2,,1369770846203.4700bdd89a351b506da910dd6a268d44.
2013-05-28 19:54:07,362 INFO  
[RS_CLOSE_REGION-asf001.sp2.ygridcore.net,58791,136977068-1] 
regionserver.HRegionServer(4060): Adding moved region record: 
4700bdd89a351b506da910dd6a268d44 to 
asf001.sp2.ygridcore.net,36189,1369770847288:36189 as of 1
2013-05-28 19:54:07,363 DEBUG 
[RS_CLOSE_REGION-asf001.sp2.ygridcore.net,58791,136977068-1] 
zookeeper.ZKAssign(786): regionserver:58791-0x13eecb1db960001 Attempting to 
transition node 4700bdd89a351b506da910dd6a268d44 from M_ZK_REGION_CLOSING to 
RS_ZK_REGION_CLOSED
2013-05-28 19:54:07,561 DEBUG [pool-1-thread-1] 
access.TestAccessController(1955): Waiting for region to be opened. Already 
retried 1 times.
2013-05-28 19:54:07,761 DEBUG [pool-1-thread-1] 
access.TestAccessController(1955): Waiting for region to be opened. Already 
retried 2 times.
2013-05-28 19:54:07,961 DEBUG [pool-1-thread-1] 
access.TestAccessController(1955): Waiting for region to be opened. Already 
retried 3 times.
2013-05-28 19:54:08,162 DEBUG [pool-1-thread-1] 
access.TestAccessController(1955): Waiting for region to be opened. Already 
retried 4 times.
2013-05-28 19:54:08,362 DEBUG [pool-1-thread-1] 
access.TestAccessController(1955): Waiting for region to be opened. Already 
retried 5 times.
2013-05-28 19:54:08,562 DEBUG [pool-1-thread-1] 
access.TestAccessController(1955): Waiting for region to be opened. Already 
retried 6 times.
2013-05-28 19:54:08,763 DEBUG [pool-1-thread-1] 
access.TestAccessController(1955): Waiting for region to be opened. Already 
retried 7 times.
2013-05-28 19:54:08,963 DEBUG [pool-1-thread-1] 
access.TestAccessController(1955): Waiting for region to be opened. Already 
retried 8 times.
2013-05-28 19:54:09,164 INFO  [pool-1-thread-1] client.HBaseAdmin$5(834): 
Started disable of testtable
2013-05-28 19:54:09,320 DEBUG [pool-1-thread-1-EventThread] 
zookeeper.ZooKeeperWatcher(307): master:55436-0x13eecb1db96 Received 
ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, 
path=/hbase/region-in-transition/4700bdd89a351b506da910dd6a268d44
2013-05-28 19:54:09,320 DEBUG 
[RS_CLOSE_REGION-asf001.sp2.ygridcore.net,58791,136977068-1] 
zookeeper.ZKAssign(862): regionserver:58791-0x13eecb1db960001 Successfully 
transitioned node 4700bdd89a351b506da910dd6a268d44 from M_ZK_REGION_CLOSING to 
RS_ZK_REGION_CLOSED
2013-05-28 19:54:09,321 DEBUG 
[RS_CLOSE_REGION-asf001.sp2.ygridcore.net,58791,136977068-1] 
handler.CloseRegionHandler(168): set region closed state in zk successfully for 
region testtable2,,1369770846203.4700bdd89a351b506da910dd6a268d44. sn name: 
asf001.sp2.ygridcore.net,58791,136977068
2013-05-28 19:54:09,321 DEBUG 
[RS_CLOSE_REGION-asf001.sp2.ygridcore.net,58791,136977068-1] 
handler.CloseRegionHandler(177): Closed region 
testtable2,,1369770846203.4700b

[jira] [Commented] (HBASE-6295) Possible performance improvement in client batch operations: presplit and send in background

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668693#comment-13668693
 ] 

Sergey Shelukhin commented on HBASE-6295:
-

patch looks reasonable so far... can you please file a jira for Process code 
de-duping/removal

> Possible performance improvement in client batch operations: presplit and 
> send in background
> 
>
> Key: HBASE-6295
> URL: https://issues.apache.org/jira/browse/HBASE-6295
> Project: HBase
>  Issue Type: Improvement
>  Components: Client, Performance
>Affects Versions: 0.95.2
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
>  Labels: noob
> Fix For: 0.98.0
>
> Attachments: 6295.v1.patch, 6295.v2.patch, 6295.v3.patch, 
> 6295.v4.patch, 6295.v5.patch, 6295.v6.patch, 6295.v8.patch, 6295.v9.patch
>
>
> today batch algo is:
> {noformat}
> for Operation o: List{
>   add o to todolist
>   if todolist > maxsize or o last in list
> split todolist per location
> send split lists to region servers
> clear todolist
> wait
> }
> {noformat}
> We could:
> - create immediately the final object instead of an intermediate array
> - split per location immediately
> - instead of sending when the list as a whole is full, send it when there is 
> enough data for a single location
> It would be:
> {noformat}
> for Operation o: List{
>   get location
>   add o to todo location.todolist
>   if (location.todolist > maxLocationSize)
> send location.todolist to region server 
> clear location.todolist
> // don't wait, continue the loop
> }
> send remaining
> wait
> {noformat}
> It's not trivial to write if you add error management: retried list must be 
> shared with the operations added in the todolist. But it's doable.
> It's interesting mainly for 'big' writes

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8611) Improve test coverage in pkg org.apache.hadoop.hbase.mapred

[Nick Dimiduk (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668690#comment-13668690
 ] 

Nick Dimiduk commented on HBASE-8611:
-

Read through trunk-b.patch, looks good. Ran these tests on trunk, here's the 
output:

{noformat}
---
 T E S T S
---
Running org.apache.hadoop.hbase.mapred.TestDriver
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.584 sec
Running org.apache.hadoop.hbase.mapred.TestGroupingTableMap
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.542 sec
Running org.apache.hadoop.hbase.mapred.TestIdentityTableMap
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.893 sec
Running org.apache.hadoop.hbase.mapred.TestRowCounter
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.023 sec
Running org.apache.hadoop.hbase.mapred.TestSplitTable
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.623 sec
Running org.apache.hadoop.hbase.mapred.TestTableInputFormat
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.652 sec
Running org.apache.hadoop.hbase.mapred.TestTableMapReduce
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 189.506 sec
Running org.apache.hadoop.hbase.mapred.TestTableMapReduceUtil
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.731 sec

Results :

Tests run: 28, Failures: 0, Errors: 0, Skipped: 0
{noformat}

My only request is for more comments. Tests names are nice, but test comments 
are helpful for test maintainers and super helpful for new users looking to 
learn APIs.

+1 from me otherwise.

> Improve test coverage in pkg org.apache.hadoop.hbase.mapred
> ---
>
> Key: HBASE-8611
> URL: https://issues.apache.org/jira/browse/HBASE-8611
> Project: HBase
>  Issue Type: Test
>Reporter: Vadim Bondarev
>Assignee: Vadim Bondarev
> Attachments: HBASE-8611-0.94-a.patch, HBASE-8611-trunk-a.patch, 
> HBASE-8611-trunk-b.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8629) More log edits: we log too much

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668683#comment-13668683
 ] 

Sergey Shelukhin commented on HBASE-8629:
-

interesting point about info logging, it's really impossible to figure out 
anything without debug logging at the moment. Should info be aimed at 
debugging, or at admins (I know, I know it's named info, but we are in 0.x 
version... :)). In the latter case we might want to actually prune it.

Thread names (or tids) should be kept imho.

Wrt long winded entity logging like regions, perhaps log it fully first time 
(under reasonable definition of first time, such as when region is 
opened/split?), then log short only, so the long description can be grepped 
from the logs if necessary?



> More log edits: we log too much
> ---
>
> Key: HBASE-8629
> URL: https://issues.apache.org/jira/browse/HBASE-8629
> Project: HBase
>  Issue Type: Umbrella
>Reporter: stack
>
> + Our thread names are nice and descriptive and are useful particularly when 
> running standlone mode but it gets silly printing out full thread names when 
> distributed on each log (we could turn off printing thread name but can be 
> helpful).
> + Do we have to print the fully qualified path for all files every time?  
> Lines get really long and hard to read.  Ditto for region names.
> + Can we print out just the class name rather than full package qualified 
> class name.
> For example:
> {code}
> 2013-05-25 12:21:01,912 DEBUG 
> [RS_OPEN_REGION-sss-2.ent.cloudera.com,60020,1369507494038-2] 
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: region 
> transitioned to opened in zookeeper: {NAME => 
> 'IntegrationTestDataIngestWithChaosMonkey,c28f5c19,1369509660096.4e91d414f14a75cf367609ce9c4377c7.',
>  STARTKEY => 'c28f5c19', ENDKEY => 'ccbc', ENCODED => 
> 4e91d414f14a75cf367609ce9c4377c7,}, server: 
> sss-2.ent.cloudera.com,60020,1369507494038
> {code}
> Should the above just be:
> {code}
> 2013-05-25 12:21:01,912 DEBUG 
> [RS_OPEN_REGION-sss-2.ent.cloudera.com,60020,1369507494038-2] 
> OpenRegionHandler: region 4e91d414f14a75cf367609ce9c4377c7 transitioned to 
> opened
> {code}
> + Some logging is bound to freak operators.  We print out the full stack 
> trace when we are logging failed assignment because of connection refused.
> + Should  make sure we tell a decent story when INFO level only enabled.  At 
> moment it is scattershot.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8365) Duplicated ZK notifications cause Master abort (or other unknown issues)

[Jeffrey Zhong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668679#comment-13668679
 ] 

Jeffrey Zhong commented on HBASE-8365:
--

[~ram_krish] Actually it's very unlikely to cause any issue in trunk(Zookeeper 
may still generate dup notifications). Because trunk Assignment Manager code 
handle one region at one time and only re-register watcher right before AM 
handling a notification. Let's say in trunk we firstly get a "failed_open" ZK 
notification, we only re-register a watcher right before the handling starts. 
In other words, the time window between we received a notification and we 
process the notification, we won't receive any ZK notification. 

In 0.94, because of the async(executorService.submit) nature, we immediately 
reregister watcher on a region which triggered the notification while actual 
handling may happen later at some time. In addition, there is no sync control, 
multiple sources could touch same region concurrently. It creates the 
possibility that two handlers to handle same notification in 0.94.

> Duplicated ZK notifications cause Master abort (or other unknown issues)
> 
>
> Key: HBASE-8365
> URL: https://issues.apache.org/jira/browse/HBASE-8365
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.94.6
>Reporter: Jeffrey Zhong
>Assignee: ramkrishna.s.vasudevan
> Attachments: TestResult.txt, TestZookeeper.txt
>
>
> The duplicated ZK notifications should happen in trunk as well. Since the way 
> we handle ZK notifications is different in trunk, we don't see the issue 
> there. I'll explain later.
> The issue is causing TestMetaReaderEditor.testRetrying flaky with error 
> message {code}reader: count=2, t=null{code} A related link is at 
> https://builds.apache.org/job/HBase-0.94/941/testReport/junit/org.apache.hadoop.hbase.catalog/TestMetaReaderEditor/testRetrying/
> The test case failure is due to an IllegalStateException and master is 
> aborted so the rest test cases also failed after testRetrying.
> Below are steps why the issue is happening(region 
> fa0e7a5590feb69bd065fbc99c228b36 is in interests):
> 1) Got first notification event RS_ZK_REGION_FAILED_OPEN at 2013-04-04 
> 17:39:01,197
> {code} DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager(744): 
> Handling transition=RS_ZK_REGION_FAILED_OPEN, 
> server=janus.apache.org,42093,1365097126155, 
> region=fa0e7a5590feb69bd065fbc99c228b36{code}
> In the step, AM tries to open the region on another RS in a separate thread
> 2) Got second notification event RS_ZK_REGION_FAILED_OPEN at 2013-04-04 
> 17:39:01,200 
> {code}DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager(744): 
> Handling transition=RS_ZK_REGION_FAILED_OPEN, 
> server=janus.apache.org,42093,1365097126155, 
> region=fa0e7a5590feb69bd065fbc99c228b36{code}
> 3) Later got opening notification event result from the step 1 at 2013-04-04 
> 17:39:01,288 
> {code} DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager(744): 
> Handling transition=RS_ZK_REGION_OPENING, 
> server=janus.apache.org,54833,1365097126175, 
> region=fa0e7a5590feb69bd065fbc99c228b36{code}
> Step 2 ClosedRegionHandler throws IllegalStateException because "Cannot 
> transit it to OFFLINE"(state is in opening from notification 3) and abort 
> Master. This could happen in 0.94 because we handle notifications using 
> executorService which opens the door to handle events out of order through 
> receive them in order of updates.
> I've confirmed that we don't have duplicated AM listeners and both events 
> triggered by same ZK data of exact same version. The issue can be reproduced 
> once by running testRetrying test case 20 times in a loop.
> There are several issues for the failure:
> 1) duplicated ZK notifications. Since ZK watcher is one time trigger, the 
> duplicated notification should not happen from the same data of the same 
> version in the first place
> 2) ZooKeeper watcher handling is wrong in both 0.94 and trunk as following:
> a) 0.94 handle notifications in async way which may lead to handle 
> notifications out of order of the events happened
> b) In trunk, we handle ZK notifications synchronously which slows down other 
> components such as SSH, LogSplitting etc. because we have a single 
> notification queue
> c) In trunk & 0.94, we could use stale event data because we have a long 
> listener list. ZK node state could have changed at the time when handling the 
> event. If a listener needs to act upon latest state, it should re-fetch the 
> data to verify if the data triggered the handler hasn't changed.
> Suggestions:
> For 0.94, we can bandit the CloseRegionHandler to pass in the expected ZK 
> data version to skip event handling on stale data with min i

[jira] [Commented] (HBASE-8631) Meta Region First Recovery

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668649#comment-13668649
 ] 

Ted Yu commented on HBASE-8631:
---

{code}
+  // we split meta regions and user regions separately therefore logfiles 
are either all for
{code}
'meta regions' -> 'meta region'
{code}
+if (carryingMeta && 
!region.equalsIgnoreCase(metaEncodeRegionName)) {
+  // skip non-meta regions because we recover meta regions and 
user regions in
+  // two separate calls.
+  continue;
+} else if (!carryingMeta && 
region.equalsIgnoreCase(metaEncodeRegionName)) {
{code}
The conditions for above two checks can be unified, right ?
{code}
carryingMeta != region.equalsIgnoreCase(metaEncodeRegionName)
{code}
{code}
-Set previouslyFailedDeletoins = null;
+List, Boolean>> previouslyFailedDeletoins = null;
{code}
typo: Deletoin
{code}
+// set a smaller retries to fast fail otherwise splitlogworker 
could be blocked for
+// quite a while inside HConnection layer. The worker won't 
available for other
+// tasks even after current task is preempted after a split task 
times out.
{code}
Should the new conf be created inside LogReplayOutputSink ctor ?

> Meta Region First Recovery
> --
>
> Key: HBASE-8631
> URL: https://issues.apache.org/jira/browse/HBASE-8631
> Project: HBase
>  Issue Type: Bug
>  Components: MTTR
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
> Attachments: hbase-8631.patch
>
>
> We have a separate wal for meta region. While log splitting logic haven't 
> taken the advantage of this and splitlogworker still picks a wal file 
> randomly. Imaging if we have multiple region servers including meta RS fails 
> about the same time while meta wal is recovered last, all failed regions have 
> to wait meta recovered and then can be online again. 
> The open JIRA is to let splitlogworker to pick a meta wal file firstly and 
> then others.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8631) Meta Region First Recovery

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668636#comment-13668636
 ] 

Hadoop QA commented on HBASE-8631:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12585062/hbase-8631.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 6 new 
or modified tests.

{color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 
1.0 profile.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   
org.apache.hadoop.hbase.security.access.TestAccessController

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5852//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5852//console

This message is automatically generated.

> Meta Region First Recovery
> --
>
> Key: HBASE-8631
> URL: https://issues.apache.org/jira/browse/HBASE-8631
> Project: HBase
>  Issue Type: Bug
>  Components: MTTR
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
> Attachments: hbase-8631.patch
>
>
> We have a separate wal for meta region. While log splitting logic haven't 
> taken the advantage of this and splitlogworker still picks a wal file 
> randomly. Imaging if we have multiple region servers including meta RS fails 
> about the same time while meta wal is recovered last, all failed regions have 
> to wait meta recovered and then can be online again. 
> The open JIRA is to let splitlogworker to pick a meta wal file firstly and 
> then others.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8488) HBase transitive dependencies not being pulled in when building apps like Flume which depend on HBase

[Roshan Naik (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668586#comment-13668586
 ] 

Roshan Naik commented on HBASE-8488:


mvn -X output


[WARNING] The POM for 
org.apache.hbase:hbase-common:jar:tests:0.95.0-hadoop2-20130524.222913-2 is 
invalid, transitive dependencies (if any) will not be available: 2 problems 
were encountered while building the effective model for 
org.apache.hbase:hbase-common:0.95.0-hadoop2-SNAPSHOT
[ERROR] 'dependencyManagement.dependencies.dependency.artifactId' for 
org.apache.hbase:${compat.module}:jar with value '${compat.module}' does not 
match a valid id pattern. @ 
[ERROR] 'dependencyManagement.dependencies.dependency.artifactId' for 
org.apache.hbase:${compat.module}:test-jar with value '${compat.module}' does 
not match a valid id pattern. @ 



> HBase transitive dependencies not being pulled in when building apps like 
> Flume which depend on HBase
> -
>
> Key: HBASE-8488
> URL: https://issues.apache.org/jira/browse/HBASE-8488
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.95.0
>Reporter: Roshan Naik
>
> Here is a snippet of the errors seen when building against Hbase
> {code}
> [WARNING] Invalid POM for org.apache.hbase:hbase-common:jar:0.97.0-SNAPSHOT, 
> transitive dependencies (if any) will not be available, enable debug logging 
> for more details: Some problems were encountered while processing the POMs:
> [ERROR] 'dependencyManagement.dependencies.dependency.artifactId' for 
> org.apache.hbase:${compat.module}:jar with value '${compat.module}' does not 
> match a valid id pattern. @ org.apache.hbase:hbase:0.97.0-SNAPSHOT, 
> /Users/rnaik/.m2/repository/org/apache/hbase/hbase/0.97.0-SNAPSHOT/hbase-0.97.0-SNAPSHOT.pom,
>  line 982, column 21
> [ERROR] 'dependencyManagement.dependencies.dependency.artifactId' for 
> org.apache.hbase:${compat.module}:test-jar with value '${compat.module}' does 
> not match a valid id pattern. @ org.apache.hbase:hbase:0.97.0-SNAPSHOT, 
> /Users/rnaik/.m2/repository/org/apache/hbase/hbase/0.97.0-SNAPSHOT/hbase-0.97.0-SNAPSHOT.pom,
>  line 987, column 21
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8488) HBase transitive dependencies not being pulled in when building apps like Flume which depend on HBase

[Roshan Naik (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668582#comment-13668582
 ] 

Roshan Naik commented on HBASE-8488:


The issue with transitive dependencies not getting pulled remains..


[WARNING] The POM for 
org.apache.hbase:hbase-common:jar:tests:0.95.0-hadoop2-20130524.222913-2 is 
invalid, transitive dependencies (if any) will not be available, enable debug 
logging for more details
[WARNING] The POM for 
org.apache.hbase:hbase-client:jar:0.95.0-hadoop2-20130524.222944-2 is invalid, 
transitive dependencies (if any) will not be available, enable debug logging 
for more details
[WARNING] The POM for 
org.apache.hbase:hbase-server:jar:0.95.0-hadoop2-20130524.223047-2 is invalid, 
transitive dependencies (if any) will not be available, enable debug logging 
for more details
[WARNING] The POM for 
org.apache.hbase:hbase-server:jar:tests:0.95.0-hadoop2-20130524.223047-2 is 
invalid, transitive dependencies (if any) will not be available, enable debug 
logging for more details

> HBase transitive dependencies not being pulled in when building apps like 
> Flume which depend on HBase
> -
>
> Key: HBASE-8488
> URL: https://issues.apache.org/jira/browse/HBASE-8488
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.95.0
>Reporter: Roshan Naik
>
> Here is a snippet of the errors seen when building against Hbase
> {code}
> [WARNING] Invalid POM for org.apache.hbase:hbase-common:jar:0.97.0-SNAPSHOT, 
> transitive dependencies (if any) will not be available, enable debug logging 
> for more details: Some problems were encountered while processing the POMs:
> [ERROR] 'dependencyManagement.dependencies.dependency.artifactId' for 
> org.apache.hbase:${compat.module}:jar with value '${compat.module}' does not 
> match a valid id pattern. @ org.apache.hbase:hbase:0.97.0-SNAPSHOT, 
> /Users/rnaik/.m2/repository/org/apache/hbase/hbase/0.97.0-SNAPSHOT/hbase-0.97.0-SNAPSHOT.pom,
>  line 982, column 21
> [ERROR] 'dependencyManagement.dependencies.dependency.artifactId' for 
> org.apache.hbase:${compat.module}:test-jar with value '${compat.module}' does 
> not match a valid id pattern. @ org.apache.hbase:hbase:0.97.0-SNAPSHOT, 
> /Users/rnaik/.m2/repository/org/apache/hbase/hbase/0.97.0-SNAPSHOT/hbase-0.97.0-SNAPSHOT.pom,
>  line 987, column 21
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6295) Possible performance improvement in client batch operations: presplit and send in background

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668571#comment-13668571
 ] 

Hadoop QA commented on HBASE-6295:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12585051/6295.v9.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 5 new 
or modified tests.

{color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 
1.0 profile.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 lineLengths{color}.  The patch introduces lines longer than 
100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   
org.apache.hadoop.hbase.replication.TestReplicationQueueFailoverCompressed
  
org.apache.hadoop.hbase.replication.TestReplicationQueueFailover
  org.apache.hadoop.hbase.client.TestHCM

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5851//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5851//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5851//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5851//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5851//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5851//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5851//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5851//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5851//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5851//console

This message is automatically generated.

> Possible performance improvement in client batch operations: presplit and 
> send in background
> 
>
> Key: HBASE-6295
> URL: https://issues.apache.org/jira/browse/HBASE-6295
> Project: HBase
>  Issue Type: Improvement
>  Components: Client, Performance
>Affects Versions: 0.95.2
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
>  Labels: noob
> Fix For: 0.98.0
>
> Attachments: 6295.v1.patch, 6295.v2.patch, 6295.v3.patch, 
> 6295.v4.patch, 6295.v5.patch, 6295.v6.patch, 6295.v8.patch, 6295.v9.patch
>
>
> today batch algo is:
> {noformat}
> for Operation o: List{
>   add o to todolist
>   if todolist > maxsize or o last in list
> split todolist per location
> send split lists to region servers
> clear todolist
> wait
> }
> {noformat}
> We could:
> - create immediately the final object instead of an intermediate array
> - split per location immediately
> - instead of sending when the list as a whole is full, send it when there is 
> enough data for a single location
> It would be:
> {noformat}
> for Operation o: List{
>   get location
>   add o to todo location.todolist
>   if (location.todolist > maxLocationSize)
> send location.todolist to region server 
> clear location.todolist
> // don't wait, continue the loop
> }
> send remaining
> wait
> {noformat}
> It's not trivial to write if you add error management: retried list must be 
> shared with the operations added in the todolist. But it's doable.
> It's interesting mainly for 'big' writes

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administr

[jira] [Updated] (HBASE-8631) Meta Region First Recovery

[Jeffrey Zhong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeffrey Zhong updated HBASE-8631:
-

Status: Patch Available  (was: Open)

> Meta Region First Recovery
> --
>
> Key: HBASE-8631
> URL: https://issues.apache.org/jira/browse/HBASE-8631
> Project: HBase
>  Issue Type: Bug
>  Components: MTTR
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
> Attachments: hbase-8631.patch
>
>
> We have a separate wal for meta region. While log splitting logic haven't 
> taken the advantage of this and splitlogworker still picks a wal file 
> randomly. Imaging if we have multiple region servers including meta RS fails 
> about the same time while meta wal is recovered last, all failed regions have 
> to wait meta recovered and then can be online again. 
> The open JIRA is to let splitlogworker to pick a meta wal file firstly and 
> then others.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8488) HBase transitive dependencies not being pulled in when building apps like Flume which depend on HBase

[Roshan Naik (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668565#comment-13668565
 ] 

Roshan Naik commented on HBASE-8488:


taking a look. will revert back soon.

> HBase transitive dependencies not being pulled in when building apps like 
> Flume which depend on HBase
> -
>
> Key: HBASE-8488
> URL: https://issues.apache.org/jira/browse/HBASE-8488
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.95.0
>Reporter: Roshan Naik
>
> Here is a snippet of the errors seen when building against Hbase
> {code}
> [WARNING] Invalid POM for org.apache.hbase:hbase-common:jar:0.97.0-SNAPSHOT, 
> transitive dependencies (if any) will not be available, enable debug logging 
> for more details: Some problems were encountered while processing the POMs:
> [ERROR] 'dependencyManagement.dependencies.dependency.artifactId' for 
> org.apache.hbase:${compat.module}:jar with value '${compat.module}' does not 
> match a valid id pattern. @ org.apache.hbase:hbase:0.97.0-SNAPSHOT, 
> /Users/rnaik/.m2/repository/org/apache/hbase/hbase/0.97.0-SNAPSHOT/hbase-0.97.0-SNAPSHOT.pom,
>  line 982, column 21
> [ERROR] 'dependencyManagement.dependencies.dependency.artifactId' for 
> org.apache.hbase:${compat.module}:test-jar with value '${compat.module}' does 
> not match a valid id pattern. @ org.apache.hbase:hbase:0.97.0-SNAPSHOT, 
> /Users/rnaik/.m2/repository/org/apache/hbase/hbase/0.97.0-SNAPSHOT/hbase-0.97.0-SNAPSHOT.pom,
>  line 987, column 21
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8631) Meta Region First Recovery

[Jeffrey Zhong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeffrey Zhong updated HBASE-8631:
-

Attachment: hbase-8631.patch

This patch includes several changes from my integration testing.

1) pick wal firstly inside splitlogworker
2) decrease retries(fast fail) in distributedLogReplay mode so a splitlogworker 
won't be blocked(currently server to server retries are set to 100 by default. 
It means 50 mins wait)
3) Add support of removing recovering region when a RS has both meta and user 
region situation



> Meta Region First Recovery
> --
>
> Key: HBASE-8631
> URL: https://issues.apache.org/jira/browse/HBASE-8631
> Project: HBase
>  Issue Type: Bug
>  Components: MTTR
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
> Attachments: hbase-8631.patch
>
>
> We have a separate wal for meta region. While log splitting logic haven't 
> taken the advantage of this and splitlogworker still picks a wal file 
> randomly. Imaging if we have multiple region servers including meta RS fails 
> about the same time while meta wal is recovered last, all failed regions have 
> to wait meta recovered and then can be online again. 
> The open JIRA is to let splitlogworker to pick a meta wal file firstly and 
> then others.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8631) Meta Region First Recovery

[Jeffrey Zhong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668558#comment-13668558
 ] 

Jeffrey Zhong commented on HBASE-8631:
--

{quote}
I think it is handled in the MetaServerShutdownHandler's process()?
{quote}
MetaSSH can deal with one RS failure which put meta wal firstly. While even in 
that case, if we already have bunch of wals files in log splitting process, it 
still has issue.

> Meta Region First Recovery
> --
>
> Key: HBASE-8631
> URL: https://issues.apache.org/jira/browse/HBASE-8631
> Project: HBase
>  Issue Type: Bug
>  Components: MTTR
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
>
> We have a separate wal for meta region. While log splitting logic haven't 
> taken the advantage of this and splitlogworker still picks a wal file 
> randomly. Imaging if we have multiple region servers including meta RS fails 
> about the same time while meta wal is recovered last, all failed regions have 
> to wait meta recovered and then can be online again. 
> The open JIRA is to let splitlogworker to pick a meta wal file firstly and 
> then others.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira