date:20151103


[ 
https://issues.apache.org/jira/browse/HBASE-14589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14986919#comment-14986919
 ] 

Hudson commented on HBASE-14589:


SUCCESS: Integrated in HBase-TRUNK #6989 (See 
[https://builds.apache.org/job/HBase-TRUNK/6989/])
HBASE-14589 Looking for the surefire-killer; builds being killed...; (stack: 
rev a82d55e3cfcc71d6b40ce6baa825d6e5c88b1bed)
* dev-support/test-patch.sh
HBASE-14589 Looking for the surefire-killer; builds being killed...; (stack: 
rev 870c74e270dc456f6328cf11ec5a3678e1db73e3)
* dev-support/test-patch.sh


> Looking for the surefire-killer; builds being killed...
> ---
>
> Key: HBASE-14589
> URL: https://issues.apache.org/jira/browse/HBASE-14589
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Attachments: 14589.mx.patch, 14589.purge.zombie.killer.patch, 
> 14589.timeout.txt, 14589.txt, 14598.addendum.sufire.timeout.patch, 
> bad_fix.patch
>
>
> I see this in a build that started at two hours ago... about 6:45... its 
> build 15941 on ubuntu-6
> {code}
> WARNING: 2 rogue build processes detected, terminating.
> /bin/kill -9 18640 
> /bin/kill -9 22625 
> {code}
> If I back up to build 15939, started about 3 1/2 hours ago, say, 5:15  I 
> see:
> Running org.apache.hadoop.hbase.client.TestShell
> Killed
> ... but it was running on ubuntu-1 so it doesn't look like we are killing 
> ourselves...  when we do this in test-patch.sh
>   ### Kill any rogue build processes from the last attempt
>   $PS auxwww | $GREP ${PROJECT_NAME}PatchProcess | $AWK '{print $2}' | 
> /usr/bin/xargs -t -I {} /bin/kill -9 {} > /dev/null
> The above code runs in a few places... in test-patch.sh.
> Let me try and add some more info around what is being killed... 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14749) Make changes to region_mover.rb to use RegionMover Java tool

Abhishek Singh Chouhan created HBASE-14749:
--

 Summary: Make changes to region_mover.rb to use RegionMover Java 
tool
 Key: HBASE-14749
 URL: https://issues.apache.org/jira/browse/HBASE-14749
 Project: HBase
  Issue Type: Improvement
Reporter: Abhishek Singh Chouhan
Assignee: Abhishek Singh Chouhan


With HBASE-13014 in, we can now replace the ruby script such that it invokes 
the Java Tool. Also expose timeout and no-ack mode which were added.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14605) Split fails due to 'No valid credentials' error when SecureBulkLoadEndpoint#start tries to access hdfs


[ 
https://issues.apache.org/jira/browse/HBASE-14605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14986887#comment-14986887
 ] 

Hudson commented on HBASE-14605:


FAILURE: Integrated in HBase-1.1-JDK7 #1582 (See 
[https://builds.apache.org/job/HBase-1.1-JDK7/1582/])
HBASE-14605 Addendum restores two methods in SplitTransactionImpl (apurtell: 
rev 79b6b91cc5a2a1ad87a713e55964ba1f31380e6b)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransactionImpl.java


> Split fails due to 'No valid credentials' error when 
> SecureBulkLoadEndpoint#start tries to access hdfs
> --
>
> Key: HBASE-14605
> URL: https://issues.apache.org/jira/browse/HBASE-14605
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: 144605-branch-1-v3.txt, 14605-0.98-v5.txt, 
> 14605-branch-1-addendum.txt, 14605-branch-1-v4.txt, 14605-branch-1-v5.txt, 
> 14605-branch-1.0-v5.txt, 14605-v1.txt, 14605-v2.txt, 14605-v3.txt, 
> 14605-v3.txt, 14605-v3.txt, 14605-v4.txt, 14605-v5.txt, 14605.alt
>
>
> During recent testing in secure cluster (with HBASE-14475), we found the 
> following when user X (non-super user) split a table with region replica:
> {code}
> 2015-10-12 10:58:18,955 ERROR [FifoRpcScheduler.handler1-thread-9] 
> master.HMaster: Region server hbase-4-4.novalocal,60020,1444645588137 
> reported a fatal error:
> ABORTING region server hbase-4-4.novalocal,60020,1444645588137: The 
> coprocessor org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint 
> threw an unexpected   exception
> Cause:
> java.lang.IllegalStateException: Failed to get FileSystem instance
>   at 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.start(SecureBulkLoadEndpoint.java:148)
>   at 
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:415)
>   at 
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:257)
>   at 
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadSystemCoprocessors(CoprocessorHost.java:160)
>   at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.(RegionCoprocessorHost.java:192)
>   at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:701)
>   at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:608)
> ...
> Caused by: java.io.IOException: Failed on local exception: 
> java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed 
> [Caused by GSSException: No valid  credentials provided (Mechanism 
> level: Failed to find any Kerberos tgt)]; Host Details : local host is: 
> "hbase-4-4/172.22.66.186"; destination host is: "os-r6-  
> okarus-hbase-4-2.novalocal":8020;
>   at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1473)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1400)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
>   at com.sun.proxy.$Proxy18.mkdirs(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:555)
>   at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
>   at com.sun.proxy.$Proxy19.mkdirs(Unknown Source)
>   at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2775)
>   at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2746)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:967)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:963)
>   at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> {code}
> The cause was that SecureBulkLoadEndpoint#start tried to create staging dir 
> in hdfs as user X but didn't pass authentication.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14631) Region merge request should be audited with request user through proper scope of doAs() calls to region observer notifications


[ 
https://issues.apache.org/jira/browse/HBASE-14631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14986886#comment-14986886
 ] 

Hudson commented on HBASE-14631:


FAILURE: Integrated in HBase-1.1-JDK7 #1582 (See 
[https://builds.apache.org/job/HBase-1.1-JDK7/1582/])
HBASE-14631 Addendum restores a method in RegionMergeTransactionImpl (apurtell: 
rev 074d512fa67f7a00d2fd3771d56c383d9c7da299)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionMergeTransactionImpl.java


> Region merge request should be audited with request user through proper scope 
> of doAs() calls to region observer notifications
> --
>
> Key: HBASE-14631
> URL: https://issues.apache.org/jira/browse/HBASE-14631
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: 14631-branch-0.98.txt, 14631-branch-1.0.txt, 
> 14631-branch-1.txt, 14631-v1.txt, HBASE-14631-addendum.patch
>
>
> HBASE-14475 and HBASE-14605 narrowed the scope of doAs() calls to region 
> observer notifications for region splitting.
> During review of HBASE-14605, Andrew brought up the case for region merge.
> This JIRA is to implement similar scope narrowing technique for region 
> merging.
> The majority of the change would be in RegionMergeTransactionImpl class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-12822) Option for Unloading regions through region_mover.rb without Acknowledging


 [ 
https://issues.apache.org/jira/browse/HBASE-12822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Singh Chouhan resolved HBASE-12822.

   Resolution: Fixed
Fix Version/s: 2.0.0
 Release Note: Incorporated in HBASE-13014.

> Option for Unloading regions through region_mover.rb without Acknowledging
> --
>
> Key: HBASE-12822
> URL: https://issues.apache.org/jira/browse/HBASE-12822
> Project: HBase
>  Issue Type: Improvement
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
>Priority: Minor
> Fix For: 2.0.0
>
>
> Currently we acknowledge if region is up in Target RS after being moved.We 
> can improve performance of Graceful Stop/Rolling restarts if we introduce a 
> flag based mode which does not Acknowledge if regions are successfully up on 
> the Target region server.This would work fine as Regions will be moved once 
> Region Server shuts down anyways.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-12822) Option for Unloading regions through region_mover.rb without Acknowledging


 [ 
https://issues.apache.org/jira/browse/HBASE-12822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Singh Chouhan reassigned HBASE-12822:
--

Assignee: Abhishek Singh Chouhan

> Option for Unloading regions through region_mover.rb without Acknowledging
> --
>
> Key: HBASE-12822
> URL: https://issues.apache.org/jira/browse/HBASE-12822
> Project: HBase
>  Issue Type: Improvement
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
>Priority: Minor
> Fix For: 2.0.0
>
>
> Currently we acknowledge if region is up in Target RS after being moved.We 
> can improve performance of Graceful Stop/Rolling restarts if we introduce a 
> flag based mode which does not Acknowledge if regions are successfully up on 
> the Target region server.This would work fine as Regions will be moved once 
> Region Server shuts down anyways.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14589) Looking for the surefire-killer; builds being killed...


[ 
https://issues.apache.org/jira/browse/HBASE-14589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14986871#comment-14986871
 ] 

Hudson commented on HBASE-14589:


FAILURE: Integrated in HBase-Trunk_matrix #426 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/426/])
HBASE-14589 Looking for the surefire-killer; builds being killed...; (stack: 
rev a82d55e3cfcc71d6b40ce6baa825d6e5c88b1bed)
* dev-support/test-patch.sh
HBASE-14589 Looking for the surefire-killer; builds being killed...; (stack: 
rev 870c74e270dc456f6328cf11ec5a3678e1db73e3)
* dev-support/test-patch.sh


> Looking for the surefire-killer; builds being killed...
> ---
>
> Key: HBASE-14589
> URL: https://issues.apache.org/jira/browse/HBASE-14589
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Attachments: 14589.mx.patch, 14589.purge.zombie.killer.patch, 
> 14589.timeout.txt, 14589.txt, 14598.addendum.sufire.timeout.patch, 
> bad_fix.patch
>
>
> I see this in a build that started at two hours ago... about 6:45... its 
> build 15941 on ubuntu-6
> {code}
> WARNING: 2 rogue build processes detected, terminating.
> /bin/kill -9 18640 
> /bin/kill -9 22625 
> {code}
> If I back up to build 15939, started about 3 1/2 hours ago, say, 5:15  I 
> see:
> Running org.apache.hadoop.hbase.client.TestShell
> Killed
> ... but it was running on ubuntu-1 so it doesn't look like we are killing 
> ourselves...  when we do this in test-patch.sh
>   ### Kill any rogue build processes from the last attempt
>   $PS auxwww | $GREP ${PROJECT_NAME}PatchProcess | $AWK '{print $2}' | 
> /usr/bin/xargs -t -I {} /bin/kill -9 {} > /dev/null
> The above code runs in a few places... in test-patch.sh.
> Let me try and add some more info around what is being killed... 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12822) Option for Unloading regions through region_mover.rb without Acknowledging


[ 
https://issues.apache.org/jira/browse/HBASE-12822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14986987#comment-14986987
 ] 

Abhishek Singh Chouhan commented on HBASE-12822:


Marking this fixed as HBASE-13014 is now in.

> Option for Unloading regions through region_mover.rb without Acknowledging
> --
>
> Key: HBASE-12822
> URL: https://issues.apache.org/jira/browse/HBASE-12822
> Project: HBase
>  Issue Type: Improvement
>Reporter: Abhishek Singh Chouhan
>Priority: Minor
>
> Currently we acknowledge if region is up in Target RS after being moved.We 
> can improve performance of Graceful Stop/Rolling restarts if we introduce a 
> flag based mode which does not Acknowledge if regions are successfully up on 
> the Target region server.This would work fine as Regions will be moved once 
> Region Server shuts down anyways.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14420) Zombie Stomping Session


[ 
https://issues.apache.org/jira/browse/HBASE-14420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987414#comment-14987414
 ] 

stack commented on HBASE-14420:
---

Latest is that we were still killing legit tests as zombies. See HBASE-14589. 
Surefire "ExecutionException: java.lang.RuntimeException..." does not seem to 
be about over-consumption of resources, at least in some instances.

> Zombie Stomping Session
> ---
>
> Key: HBASE-14420
> URL: https://issues.apache.org/jira/browse/HBASE-14420
> Project: HBase
>  Issue Type: Umbrella
>  Components: test
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: hangers.txt, none_fix (1).txt, none_fix.txt, 
> none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, 
> none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, 
> none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, 
> none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, 
> none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, 
> none_fix.txt, none_fix.txt
>
>
> Patch build are now failing most of the time because we are dropping zombies. 
> I confirm we are doing this on non-apache build boxes too.
> Left-over zombies consume resources on build boxes (OOME cannot create native 
> threads). Having to do multiple test runs in the hope that we can get a 
> non-zombie-making build or making (arbitrary) rulings that the zombies are 
> 'not related' is a productivity sink. And so on...
> This is an umbrella issue for a zombie stomping session that started earlier 
> this week. Will hang sub-issues of this one. Am running builds back-to-back 
> on little cluster to turn out the monsters.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14700) Support a "permissive" mode for secure clusters to allow "simple" auth clients


[ 
https://issues.apache.org/jira/browse/HBASE-14700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987179#comment-14987179
 ] 

Hudson commented on HBASE-14700:


FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1127 (See 
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1127/])
HBASE-14700 Support a permissive mode for secure clusters to allow (garyh: rev 
e66eec33d7ce59285aa84692f50a10c6b7b4f09c)
* hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServer.java
* hbase-common/src/main/resources/hbase-default.xml
* 
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSource.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestSecureRPC.java
* 
hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSourceImpl.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java


> Support a "permissive" mode for secure clusters to allow "simple" auth clients
> --
>
> Key: HBASE-14700
> URL: https://issues.apache.org/jira/browse/HBASE-14700
> Project: HBase
>  Issue Type: Improvement
>  Components: security
>Reporter: Gary Helmling
>Assignee: Gary Helmling
> Fix For: 2.0.0, 1.2.0, 0.98.16
>
> Attachments: HBASE-14700-v2.patch, HBASE-14700-v3.patch, 
> HBASE-14700.patch, HBASE-14700_0.98-v1.patch
>
>
> When implementing HBase security for an existing cluster, it can be useful to 
> support mixed secure and insecure clients while all client configurations are 
> migrated over to secure authentication.  
> We currently have an option to allow secure clients to fallback to simple 
> auth against insecure clusters.  By providing an analogous setting for 
> servers, we would allow a phased rollout of security:
> # First, security can be enabled on the cluster servers, with the 
> "permissive" mode enabled
> # Clients can be converting to using secure authentication incrementally
> # The server audit logs allow identification of clients still using simple 
> auth to connect
> # Finally, when sufficient clients have been converted to secure operation, 
> the server-side "permissive" mode can be removed, allowing completely secure 
> operation.
> Obviously with this enabled, there is no effective access control, but this 
> would still be a useful tool to enable a smooth operational rollout of 
> security.  Permissive mode would of course be disabled by default.  Enabling 
> it should provide a big scary warning in the logs on startup, and possibly be 
> flagged on relevant UIs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-8927) Use nano time instead of mili time everywhere

2015-11-03 Thread Jeroen van Bemmel (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987237#comment-14987237
 ] 

Jeroen van Bemmel commented on HBASE-8927:
--

A shift in system time ( e.g. ntpdate -u or date -s on Linux ) can cause a 
Zookeeper server to close all client connections due to session timeout; it 
becomes unavailable for several minutes until it recovers.
Using System.nanoTime() (in the right places) can potentially make the system 
immune to such local time shifts, on platforms that support the monotonically 
increasing clock

> Use nano time instead of mili time everywhere
> -
>
> Key: HBASE-8927
> URL: https://issues.apache.org/jira/browse/HBASE-8927
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>  Labels: phoenix
> Attachments: 8927-WIP-TEST.txt, 8927.txt
>
>
> Less collisions and we are paying the price of a long anyways so might as 
> well fill it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14631) Region merge request should be audited with request user through proper scope of doAs() calls to region observer notifications


[ 
https://issues.apache.org/jira/browse/HBASE-14631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987178#comment-14987178
 ] 

Hudson commented on HBASE-14631:


FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1127 (See 
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1127/])
HBASE-14631 Addendum restores a method in RegionMergeTransaction (apurtell: rev 
34abb947b03aa8107e17a1ccc33630168114c79d)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionMergeTransaction.java


> Region merge request should be audited with request user through proper scope 
> of doAs() calls to region observer notifications
> --
>
> Key: HBASE-14631
> URL: https://issues.apache.org/jira/browse/HBASE-14631
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: 14631-branch-0.98.txt, 14631-branch-1.0.txt, 
> 14631-branch-1.txt, 14631-v1.txt, HBASE-14631-addendum.patch
>
>
> HBASE-14475 and HBASE-14605 narrowed the scope of doAs() calls to region 
> observer notifications for region splitting.
> During review of HBASE-14605, Andrew brought up the case for region merge.
> This JIRA is to implement similar scope narrowing technique for region 
> merging.
> The majority of the change would be in RegionMergeTransactionImpl class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14631) Region merge request should be audited with request user through proper scope of doAs() calls to region observer notifications


[ 
https://issues.apache.org/jira/browse/HBASE-14631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987257#comment-14987257
 ] 

Hudson commented on HBASE-14631:


FAILURE: Integrated in HBase-0.98-matrix #255 (See 
[https://builds.apache.org/job/HBase-0.98-matrix/255/])
HBASE-14631 Addendum restores a method in RegionMergeTransaction (apurtell: rev 
34abb947b03aa8107e17a1ccc33630168114c79d)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionMergeTransaction.java


> Region merge request should be audited with request user through proper scope 
> of doAs() calls to region observer notifications
> --
>
> Key: HBASE-14631
> URL: https://issues.apache.org/jira/browse/HBASE-14631
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: 14631-branch-0.98.txt, 14631-branch-1.0.txt, 
> 14631-branch-1.txt, 14631-v1.txt, HBASE-14631-addendum.patch
>
>
> HBASE-14475 and HBASE-14605 narrowed the scope of doAs() calls to region 
> observer notifications for region splitting.
> During review of HBASE-14605, Andrew brought up the case for region merge.
> This JIRA is to implement similar scope narrowing technique for region 
> merging.
> The majority of the change would be in RegionMergeTransactionImpl class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14700) Support a "permissive" mode for secure clusters to allow "simple" auth clients


[ 
https://issues.apache.org/jira/browse/HBASE-14700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987258#comment-14987258
 ] 

Hudson commented on HBASE-14700:


FAILURE: Integrated in HBase-0.98-matrix #255 (See 
[https://builds.apache.org/job/HBase-0.98-matrix/255/])
HBASE-14700 Support a permissive mode for secure clusters to allow (garyh: rev 
e66eec33d7ce59285aa84692f50a10c6b7b4f09c)
* hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
* 
hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSourceImpl.java
* 
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSource.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServer.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestSecureRPC.java
* hbase-common/src/main/resources/hbase-default.xml


> Support a "permissive" mode for secure clusters to allow "simple" auth clients
> --
>
> Key: HBASE-14700
> URL: https://issues.apache.org/jira/browse/HBASE-14700
> Project: HBase
>  Issue Type: Improvement
>  Components: security
>Reporter: Gary Helmling
>Assignee: Gary Helmling
> Fix For: 2.0.0, 1.2.0, 0.98.16
>
> Attachments: HBASE-14700-v2.patch, HBASE-14700-v3.patch, 
> HBASE-14700.patch, HBASE-14700_0.98-v1.patch
>
>
> When implementing HBase security for an existing cluster, it can be useful to 
> support mixed secure and insecure clients while all client configurations are 
> migrated over to secure authentication.  
> We currently have an option to allow secure clients to fallback to simple 
> auth against insecure clusters.  By providing an analogous setting for 
> servers, we would allow a phased rollout of security:
> # First, security can be enabled on the cluster servers, with the 
> "permissive" mode enabled
> # Clients can be converting to using secure authentication incrementally
> # The server audit logs allow identification of clients still using simple 
> auth to connect
> # Finally, when sufficient clients have been converted to secure operation, 
> the server-side "permissive" mode can be removed, allowing completely secure 
> operation.
> Obviously with this enabled, there is no effective access control, but this 
> would still be a useful tool to enable a smooth operational rollout of 
> security.  Permissive mode would of course be disabled by default.  Enabling 
> it should provide a big scary warning in the logs on startup, and possibly be 
> flagged on relevant UIs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11393) Replication TableCfs should be a PB object rather than a string


[ 
https://issues.apache.org/jira/browse/HBASE-11393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987083#comment-14987083
 ] 

Hadoop QA commented on HBASE-11393:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12770273/HBASE-11393_v6.patch
  against master branch at commit 870c74e270dc456f6328cf11ec5a3678e1db73e3.
  ATTACHMENT ID: 12770273

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 15 new 
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 
2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:red}-1 checkstyle{color}.  The applied patch generated 
1731 checkstyle errors (more than the master's current 1726 errors).

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16360//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16360//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16360//artifact/patchprocess/checkstyle-aggregate.html

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16360//console

This message is automatically generated.

> Replication TableCfs should be a PB object rather than a string
> ---
>
> Key: HBASE-11393
> URL: https://issues.apache.org/jira/browse/HBASE-11393
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Enis Soztutar
> Fix For: 2.0.0
>
> Attachments: HBASE-11393.patch, HBASE-11393_v1.patch, 
> HBASE-11393_v2.patch, HBASE-11393_v3.patch, HBASE-11393_v4.patch, 
> HBASE-11393_v5.patch, HBASE-11393_v6.patch, HBASE-11393_v7.patch
>
>
> We concatenate the list of tables and column families in format  
> "table1:cf1,cf2;table2:cfA,cfB" in zookeeper for table-cf to replication peer 
> mapping. 
> This results in ugly parsing code. We should do this a PB object. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-11393) Replication TableCfs should be a PB object rather than a string

2015-11-03 Thread Heng Chen (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-11393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Heng Chen updated HBASE-11393:
--
Attachment: HBASE-11393_v7.patch

Update patch as [~enis] suggestions.

> Replication TableCfs should be a PB object rather than a string
> ---
>
> Key: HBASE-11393
> URL: https://issues.apache.org/jira/browse/HBASE-11393
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Enis Soztutar
> Fix For: 2.0.0
>
> Attachments: HBASE-11393.patch, HBASE-11393_v1.patch, 
> HBASE-11393_v2.patch, HBASE-11393_v3.patch, HBASE-11393_v4.patch, 
> HBASE-11393_v5.patch, HBASE-11393_v6.patch, HBASE-11393_v7.patch
>
>
> We concatenate the list of tables and column families in format  
> "table1:cf1,cf2;table2:cfA,cfB" in zookeeper for table-cf to replication peer 
> mapping. 
> This results in ugly parsing code. We should do this a PB object. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13408) HBase In-Memory Memstore Compaction

2015-11-03 Thread Edward Bortnikov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987470#comment-14987470
 ] 

Edward Bortnikov commented on HBASE-13408:
--

Part of the next patch will be configurable in-memory-compaction per-store 
property, unrelated to the in-memory property. 

General request - please speak up about any additional major issues now. We'd 
prefer this re-design round be the final before landing the feature. 

Thanks!

> HBase In-Memory Memstore Compaction
> ---
>
> Key: HBASE-13408
> URL: https://issues.apache.org/jira/browse/HBASE-13408
> Project: HBase
>  Issue Type: New Feature
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
> Fix For: 2.0.0
>
> Attachments: HBASE-13408-trunk-v01.patch, 
> HBASE-13408-trunk-v02.patch, HBASE-13408-trunk-v03.patch, 
> HBASE-13408-trunk-v04.patch, HBASE-13408-trunk-v05.patch, 
> HBASE-13408-trunk-v06.patch, HBASE-13408-trunk-v07.patch, 
> HBASE-13408-trunk-v08.patch, 
> HBaseIn-MemoryMemstoreCompactionDesignDocument-ver02.pdf, 
> HBaseIn-MemoryMemstoreCompactionDesignDocument-ver03.pdf, 
> HBaseIn-MemoryMemstoreCompactionDesignDocument.pdf, 
> InMemoryMemstoreCompactionEvaluationResults.pdf, 
> InMemoryMemstoreCompactionMasterEvaluationResults.pdf, 
> InMemoryMemstoreCompactionScansEvaluationResults.pdf, 
> StoreSegmentandStoreSegmentScannerClassHierarchies.pdf
>
>
> A store unit holds a column family in a region, where the memstore is its 
> in-memory component. The memstore absorbs all updates to the store; from time 
> to time these updates are flushed to a file on disk, where they are 
> compacted. Unlike disk components, the memstore is not compacted until it is 
> written to the filesystem and optionally to block-cache. This may result in 
> underutilization of the memory due to duplicate entries per row, for example, 
> when hot data is continuously updated. 
> Generally, the faster the data is accumulated in memory, more flushes are 
> triggered, the data sinks to disk more frequently, slowing down retrieval of 
> data, even if very recent.
> In high-churn workloads, compacting the memstore can help maintain the data 
> in memory, and thereby speed up data retrieval. 
> We suggest a new compacted memstore with the following principles:
> 1.The data is kept in memory for as long as possible
> 2.Memstore data is either compacted or in process of being compacted 
> 3.Allow a panic mode, which may interrupt an in-progress compaction and 
> force a flush of part of the memstore.
> We suggest applying this optimization only to in-memory column families.
> A design document is attached.
> This feature was previously discussed in HBASE-5311.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14750) HBaseConfiguration.create() doesn't load properties

2015-11-03 Thread Niels Basjes (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niels Basjes updated HBASE-14750:
-
Description: 
While writing an application that uses HBase I ran into the problem that 
although I have setup the hbase-site.xml in the {{HBASE_CONF_DIR}}; the 
settings in this file are not picked up in my application.

In several places I find instructions to include things like the 
{{hbase.zookeeper.quorum}} in a properties file and the explicitly set these 
value from within the application (I have actually done this in the past quite 
a few times).

Although this works I expect the system to automatically pickup the settings I 
have installed already which are picked up by tools like the hbase shell and 
pig.

So I ended up writing this helper method:
{code:java}
public static Configuration createConfiguration() {
String hbaseConfDir = System.getenv("HBASE_CONF_DIR");
if (hbaseConfDir == null) {
hbaseConfDir = "/etc/hbase/conf";
}

Configuration conf = HBaseConfiguration.create();
conf.addResource(new Path(hbaseConfDir + "/hbase-site.xml"));
return conf;
}
{code}

I expect HBaseConfiguration.create() to give me a working config in an 
environment where everything for all the other HBase clients has already been 
setup correctly.

My proposal is to change the HBaseConfiguration.create() to effectively include 
what my helper method does.


  was:
While writing an application that uses HBase I ran into the problem that 
although I have setup the hbase-site.xml in the {{HBASE_CONF_DIR}}; the 
settings in this file are not picked up in my application.

In several places I find instructions to include things like the 
{{hbase.zookeeper.quorum}} in a properties file and the explicitly set these 
value from within the application (I have actually done this in the past quite 
a few times).

Although this works I expect the system to automatically pickup the settings I 
have installed already which are picked up by tools like the hbase shell and 
pig.

So I ended up writing this helper method:
{code:java}
public static Configuration createConfiguration() {
String hbaseConfDir = System.getenv("HBASE_CONF_DIR");
if (hbaseConfDir == null) {
hbaseConfDir = "/etc/hbase";
}

Configuration conf = HBaseConfiguration.create();
conf.addResource(new Path(hbaseConfDir + "/hbase-site.xml"));
return conf;
}
{code}

I expect HBaseConfiguration.create() to give me a working config in an 
environment where everything for all the other HBase clients has already been 
setup correctly.

My proposal is to change the HBaseConfiguration.create() to effectively include 
what my helper method does.



> HBaseConfiguration.create() doesn't load properties
> ---
>
> Key: HBASE-14750
> URL: https://issues.apache.org/jira/browse/HBASE-14750
> Project: HBase
>  Issue Type: Bug
>Reporter: Niels Basjes
>
> While writing an application that uses HBase I ran into the problem that 
> although I have setup the hbase-site.xml in the {{HBASE_CONF_DIR}}; the 
> settings in this file are not picked up in my application.
> In several places I find instructions to include things like the 
> {{hbase.zookeeper.quorum}} in a properties file and the explicitly set these 
> value from within the application (I have actually done this in the past 
> quite a few times).
> Although this works I expect the system to automatically pickup the settings 
> I have installed already which are picked up by tools like the hbase shell 
> and pig.
> So I ended up writing this helper method:
> {code:java}
> public static Configuration createConfiguration() {
> String hbaseConfDir = System.getenv("HBASE_CONF_DIR");
> if (hbaseConfDir == null) {
> hbaseConfDir = "/etc/hbase/conf";
> }
> Configuration conf = HBaseConfiguration.create();
> conf.addResource(new Path(hbaseConfDir + "/hbase-site.xml"));
> return conf;
> }
> {code}
> I expect HBaseConfiguration.create() to give me a working config in an 
> environment where everything for all the other HBase clients has already been 
> setup correctly.
> My proposal is to change the HBaseConfiguration.create() to effectively 
> include what my helper method does.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14575) Reduce scope of compactions holding region lock

2015-11-03 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14575:
---
Attachment: 14575-v5.patch

Patch v5 addresses what Jerry and Ram pointed out:
When creating StoreFileScanner's, region read lock is not needed.
This is because CompactionRequest contains StoreFile's to be compacted which 
wouldn't change.

> Reduce scope of compactions holding region lock
> ---
>
> Key: HBASE-14575
> URL: https://issues.apache.org/jira/browse/HBASE-14575
> Project: HBase
>  Issue Type: Sub-task
>  Components: Compaction, regionserver
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Attachments: 14575-v1.patch, 14575-v2.patch, 14575-v3.patch, 
> 14575-v4.patch, 14575-v5.patch, 14575.v00.patch
>
>
> Per [~devaraj]'s idea on parent issue, let's see if we can reduce the scope 
> of critical section under which compactions hold the region read lock.
> Here is summary from parent issue:
> Another idea is we can reduce the scope of when the read lock is held during 
> compaction. In theory the compactor only needs a region read lock while 
> deciding what files to compact and at the time of committing the compaction. 
> We're protected from the case of region close events because compactions are 
> checking (every 10k bytes written) if the store has been closed in order to 
> abort in such a case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14481) Decommission HBase wiki


[ 
https://issues.apache.org/jira/browse/HBASE-14481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987634#comment-14987634
 ] 

stack commented on HBASE-14481:
---

+1

When you close this issue, can you paste the description as the release note. 
Someone is going to ask what is up w/ the wiki! And can paste the release note 
at them.

Excellent. Glad this project is done.

> Decommission HBase wiki
> ---
>
> Key: HBASE-14481
> URL: https://issues.apache.org/jira/browse/HBASE-14481
> Project: HBase
>  Issue Type: Task
>  Components: documentation
>Reporter: Nick Dimiduk
>Assignee: Misty Stanley-Jones
>Priority: Blocker
>  Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-14481.patch
>
>
> We have an awesome community resource in our [online 
> book|http://hbase.apache.org/book.html]. It's maintained and looked after 
> with diligence. We also have an HBase section on the [hadoop 
> wiki|http://wiki.apache.org/hadoop/Hbase] that hasn't been updated since 
> 2012. Let's sift through the pages of the wiki, bring over any content that's 
> still relevant and not already present in the book, and kill the wiki.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14744) BulkDelete in Hbase Table

2015-11-03 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987735#comment-14987735
 ] 

Anoop Sam John commented on HBASE-14744:


Can you pls do imports in your test class and check. Seems java compile issues..

> BulkDelete in Hbase Table
> -
>
> Key: HBASE-14744
> URL: https://issues.apache.org/jira/browse/HBASE-14744
> Project: HBase
>  Issue Type: Bug
>  Components: Deletes
>Affects Versions: 0.98.7
> Environment: OS : Unix
> Java Version : java version "1.8.0_25"
> Hbase Version: Version 0.98.7.12.1509250618_h2
>Reporter: Marimuthu
>
> Hi Anoop/Ted,
> I need to delete bulk records in Hbase table based on certain criteria.
> I copied one piece of code from similar thread and it throws "can not find  
> symbol" error for most of the code.
> Can you please provide me right code which would be used to delete bulk 
> records
> Here are the errors
> DeleteRowsTest.java:75: error: cannot find symbol
>HTable tableName = new HTable(conf, "salestoolsdata:account");
>  ^
>   symbol:   variable conf
>   location: class DeleteRowsTest
> DeleteRowsTest.java:79: error: no suitable constructor found for 
> SingleColumnValueFilter(char,String,CompareOp,byte[])
> SingleColumnValueFilter scvf = new 
> SingleColumnValueFilter('d',"ISDELETED",CompareOp.EQUAL, 
> Bytes.toBytes("true"));
>^
> constructor 
> SingleColumnValueFilter.SingleColumnValueFilter(byte[],byte[],CompareOp,byte[])
>  is not applicable
>   (argument mismatch; char cannot be converted to byte[])
> constructor 
> SingleColumnValueFilter.SingleColumnValueFilter(byte[],byte[],CompareOp,ByteArrayComparable)
>  is not applicable
>   (argument mismatch; char cannot be converted to byte[])
> DeleteRowsTest.java:85: error: cannot find symbol
> long noOfRowsDeleted = invokeBulkDeleteProtocol(tableName, scan, 500, 
> DeleteType.ROW, null);
>   ^
>   symbol:   variable DeleteType
>   location: class DeleteRowsTest
> DeleteRowsTest.java:89: error: cannot find symbol
> for (Result result : ht.getScanner(new Scan())) {
>  ^
>   symbol:   variable ht
>   location: class DeleteRowsTest
> DeleteRowsTest.java:98: error: cannot find symbol
> HTable ht = new HTable(conf, tableName);
>^
>   symbol:   variable conf
>   location: class DeleteRowsTest
> DeleteRowsTest.java:100: error: cannot find symbol
> Batch.Call callable = 
>^
>   symbol:   class BulkDeleteProtocol
>   location: class DeleteRowsTest
> DeleteRowsTest.java:100: error: cannot find symbol
> Batch.Call callable = 
>^
>   symbol:   class BulkDeleteResponse
>   location: class DeleteRowsTest
> DeleteRowsTest.java:101: error: cannot find symbol
> new Batch.Call() {
>^
>   symbol:   class BulkDeleteProtocol
>   location: class DeleteRowsTest
> DeleteRowsTest.java:101: error: cannot find symbol
> new Batch.Call() {
>^
>   symbol:   class BulkDeleteResponse
>   location: class DeleteRowsTest
> DeleteRowsTest.java:102: error: cannot find symbol
>   public BulkDeleteResponse call(BulkDeleteProtocol instance) throws 
> IOException {
>  ^
>   symbol: class BulkDeleteProtocol
> DeleteRowsTest.java:102: error: cannot find symbol
>   public BulkDeleteResponse call(BulkDeleteProtocol instance) throws 
> IOException {
>  ^
>   symbol: class BulkDeleteResponse
> DeleteRowsTest.java:106: error: cannot find symbol
> Map result = 
> ht.coprocessorExec(BulkDeleteProtocol.class,
> ^
>   symbol:   class BulkDeleteResponse
>   location: class DeleteRowsTest
> DeleteRowsTest.java:106: error: cannot find symbol
> Map result = 
> ht.coprocessorExec(BulkDeleteProtocol.class,
> ^
>   symbol:   class BulkDeleteProtocol
>   location: class DeleteRowsTest
> DeleteRowsTest.java:108: error: cannot find symbol
> for (BulkDeleteResponse response : result.values()) {
>  ^
>   symbol:   class BulkDeleteResponse
>   location: class DeleteRowsTest
> Note: Some messages have been simplified; recompile with -Xdiags:verbose to 
> get full output
> 18 errors
> Thanks
> Marimuthu



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14744) BulkDelete in Hbase Table

2015-11-03 Thread Marimuthu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987830#comment-14987830
 ] 

Marimuthu commented on HBASE-14744:
---

Thank you Anoop.
Can you please send me the updated link from where I can get the script and
compile

Thanks
Marimuthu

On Tue, Nov 3, 2015 at 10:04 AM, Anoop Sam John (JIRA) 



> BulkDelete in Hbase Table
> -
>
> Key: HBASE-14744
> URL: https://issues.apache.org/jira/browse/HBASE-14744
> Project: HBase
>  Issue Type: Bug
>  Components: Deletes
>Affects Versions: 0.98.7
> Environment: OS : Unix
> Java Version : java version "1.8.0_25"
> Hbase Version: Version 0.98.7.12.1509250618_h2
>Reporter: Marimuthu
>
> Hi Anoop/Ted,
> I need to delete bulk records in Hbase table based on certain criteria.
> I copied one piece of code from similar thread and it throws "can not find  
> symbol" error for most of the code.
> Can you please provide me right code which would be used to delete bulk 
> records
> Here are the errors
> DeleteRowsTest.java:75: error: cannot find symbol
>HTable tableName = new HTable(conf, "salestoolsdata:account");
>  ^
>   symbol:   variable conf
>   location: class DeleteRowsTest
> DeleteRowsTest.java:79: error: no suitable constructor found for 
> SingleColumnValueFilter(char,String,CompareOp,byte[])
> SingleColumnValueFilter scvf = new 
> SingleColumnValueFilter('d',"ISDELETED",CompareOp.EQUAL, 
> Bytes.toBytes("true"));
>^
> constructor 
> SingleColumnValueFilter.SingleColumnValueFilter(byte[],byte[],CompareOp,byte[])
>  is not applicable
>   (argument mismatch; char cannot be converted to byte[])
> constructor 
> SingleColumnValueFilter.SingleColumnValueFilter(byte[],byte[],CompareOp,ByteArrayComparable)
>  is not applicable
>   (argument mismatch; char cannot be converted to byte[])
> DeleteRowsTest.java:85: error: cannot find symbol
> long noOfRowsDeleted = invokeBulkDeleteProtocol(tableName, scan, 500, 
> DeleteType.ROW, null);
>   ^
>   symbol:   variable DeleteType
>   location: class DeleteRowsTest
> DeleteRowsTest.java:89: error: cannot find symbol
> for (Result result : ht.getScanner(new Scan())) {
>  ^
>   symbol:   variable ht
>   location: class DeleteRowsTest
> DeleteRowsTest.java:98: error: cannot find symbol
> HTable ht = new HTable(conf, tableName);
>^
>   symbol:   variable conf
>   location: class DeleteRowsTest
> DeleteRowsTest.java:100: error: cannot find symbol
> Batch.Call callable = 
>^
>   symbol:   class BulkDeleteProtocol
>   location: class DeleteRowsTest
> DeleteRowsTest.java:100: error: cannot find symbol
> Batch.Call callable = 
>^
>   symbol:   class BulkDeleteResponse
>   location: class DeleteRowsTest
> DeleteRowsTest.java:101: error: cannot find symbol
> new Batch.Call() {
>^
>   symbol:   class BulkDeleteProtocol
>   location: class DeleteRowsTest
> DeleteRowsTest.java:101: error: cannot find symbol
> new Batch.Call() {
>^
>   symbol:   class BulkDeleteResponse
>   location: class DeleteRowsTest
> DeleteRowsTest.java:102: error: cannot find symbol
>   public BulkDeleteResponse call(BulkDeleteProtocol instance) throws 
> IOException {
>  ^
>   symbol: class BulkDeleteProtocol
> DeleteRowsTest.java:102: error: cannot find symbol
>   public BulkDeleteResponse call(BulkDeleteProtocol instance) throws 
> IOException {
>  ^
>   symbol: class BulkDeleteResponse
> DeleteRowsTest.java:106: error: cannot find symbol
> Map result = 
> ht.coprocessorExec(BulkDeleteProtocol.class,
> ^
>   symbol:   class BulkDeleteResponse
>   location: class DeleteRowsTest
> DeleteRowsTest.java:106: error: cannot find symbol
> Map result = 
> ht.coprocessorExec(BulkDeleteProtocol.class,
> ^
>   symbol:   class BulkDeleteProtocol
>   location: class DeleteRowsTest
> DeleteRowsTest.java:108: error: cannot find symbol
> for (BulkDeleteResponse response : result.values()) {
>  ^
>   symbol:   class BulkDeleteResponse
>   location: class DeleteRowsTest
> Note: Some messages have been simplified; recompile with -Xdiags:verbose to 
> get full output
> 18 errors
> Thanks
> Marimuthu



--
This message was sent

[jira] [Commented] (HBASE-14490) [RpcServer] reuse request read buffer


[ 
https://issues.apache.org/jira/browse/HBASE-14490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987648#comment-14987648
 ] 

Hadoop QA commented on HBASE-14490:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12770346/HBASE-14490-v12.patch
  against master branch at commit 0eae729ffa95828462737c54c7204b34bb73182a.
  ATTACHMENT ID: 12770346

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 3 new 
or modified tests.

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16365//console

This message is automatically generated.

> [RpcServer] reuse request read buffer
> -
>
> Key: HBASE-14490
> URL: https://issues.apache.org/jira/browse/HBASE-14490
> Project: HBase
>  Issue Type: Improvement
>  Components: IPC/RPC
>Affects Versions: 2.0.0, 1.0.2
>Reporter: Zephyr Guo
>Assignee: Zephyr Guo
>  Labels: performance
> Fix For: 2.0.0, 1.0.2
>
> Attachments: ByteBufferPool.java, HBASE-14490-v1.patch, 
> HBASE-14490-v10.patch, HBASE-14490-v11.patch, HBASE-14490-v12.patch, 
> HBASE-14490-v2.patch, HBASE-14490-v3.patch, HBASE-14490-v4.patch, 
> HBASE-14490-v5.patch, HBASE-14490-v6.patch, HBASE-14490-v7.patch, 
> HBASE-14490-v8.patch, HBASE-14490-v9.patch
>
>
> Reuse buffer to read request.It's not necessary to every request free 
> buffer.The idea of optimization is to reduce the times that allocate 
> ByteBuffer.
> *Modification*
> 1. {{saslReadAndProcess}} ,{{processOneRpc}}, {{processUnwrappedData}}, 
> {{processConnectionHeader}} accept a ByteBuffer instead of byte[].They can 
> move {{ByteBuffer.position}} correctly when we have read the data.
> 2. {{processUnwrappedData}} no longer use any extra memory.
> 3. Maintaining a reused ByteBuffer in each {{Connection}}.
> a) Size is dynamic to fit specific client.
> b) If size deviate far from average size, we replace a new one.
> c) Using a SoftReference to reference the buffer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14575) Reduce scope of compactions holding region lock


[ 
https://issues.apache.org/jira/browse/HBASE-14575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987797#comment-14987797
 ] 

Hadoop QA commented on HBASE-14575:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12770335/14575-v5.patch
  against master branch at commit 0eae729ffa95828462737c54c7204b34bb73182a.
  ATTACHMENT ID: 12770335

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 18 new 
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 
2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   
org.apache.hadoop.hbase.security.token.TestGenerateDelegationToken

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16364//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16364//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16364//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16364//console

This message is automatically generated.

> Reduce scope of compactions holding region lock
> ---
>
> Key: HBASE-14575
> URL: https://issues.apache.org/jira/browse/HBASE-14575
> Project: HBase
>  Issue Type: Sub-task
>  Components: Compaction, regionserver
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Attachments: 14575-v1.patch, 14575-v2.patch, 14575-v3.patch, 
> 14575-v4.patch, 14575-v5.patch, 14575.v00.patch
>
>
> Per [~devaraj]'s idea on parent issue, let's see if we can reduce the scope 
> of critical section under which compactions hold the region read lock.
> Here is summary from parent issue:
> Another idea is we can reduce the scope of when the read lock is held during 
> compaction. In theory the compactor only needs a region read lock while 
> deciding what files to compact and at the time of committing the compaction. 
> We're protected from the case of region close events because compactions are 
> checking (every 10k bytes written) if the store has been closed in order to 
> abort in such a case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14589) Looking for the surefire-killer; builds being killed...ExecutionException: java.lang.RuntimeException: The forked VM terminated without properly saying goodbye. VM cra


[ 
https://issues.apache.org/jira/browse/HBASE-14589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987912#comment-14987912
 ] 

Hudson commented on HBASE-14589:


FAILURE: Integrated in HBase-Trunk_matrix #427 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/427/])
HBASE-14589 Looking for the surefire-killer; builds being killed...; (stack: 
rev 0eae729ffa95828462737c54c7204b34bb73182a)
* dev-support/test-patch.sh


> Looking for the surefire-killer; builds being killed...ExecutionException: 
> java.lang.RuntimeException: The forked VM terminated without properly saying 
> goodbye. VM crash or System.exit called?
> 
>
> Key: HBASE-14589
> URL: https://issues.apache.org/jira/browse/HBASE-14589
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Attachments: 14589.mx.patch, 14589.purge.zombie.killer.patch, 
> 14589.timeout.txt, 14589.txt, 14598.addendum.sufire.timeout.patch, 
> bad_fix.patch
>
>
> I see this in a build that started at two hours ago... about 6:45... its 
> build 15941 on ubuntu-6
> {code}
> WARNING: 2 rogue build processes detected, terminating.
> /bin/kill -9 18640 
> /bin/kill -9 22625 
> {code}
> If I back up to build 15939, started about 3 1/2 hours ago, say, 5:15  I 
> see:
> Running org.apache.hadoop.hbase.client.TestShell
> Killed
> ... but it was running on ubuntu-1 so it doesn't look like we are killing 
> ourselves...  when we do this in test-patch.sh
>   ### Kill any rogue build processes from the last attempt
>   $PS auxwww | $GREP ${PROJECT_NAME}PatchProcess | $AWK '{print $2}' | 
> /usr/bin/xargs -t -I {} /bin/kill -9 {} > /dev/null
> The above code runs in a few places... in test-patch.sh.
> Let me try and add some more info around what is being killed... 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14723) Fix IT tests split too many times


 [ 
https://issues.apache.org/jira/browse/HBASE-14723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-14723:
--
Attachment: HBASE-14723.patch

Retry. Hopefully the above failure...

ExecutionException Error occurred in starting fork, check output in log -> 
[Help 1]

... is less frequent now.

> Fix IT tests split too many times
> -
>
> Key: HBASE-14723
> URL: https://issues.apache.org/jira/browse/HBASE-14723
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14723.patch, HBASE-14723.patch
>
>
> Splitting the whole table is happening too often. Lets make this happen less 
> frequently as there are more regions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14631) Region merge request should be audited with request user through proper scope of doAs() calls to region observer notifications


[ 
https://issues.apache.org/jira/browse/HBASE-14631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987599#comment-14987599
 ] 

Hudson commented on HBASE-14631:


FAILURE: Integrated in HBase-1.0 #1106 (See 
[https://builds.apache.org/job/HBase-1.0/1106/])
HBASE-14631 Addendum restores a method in RegionMergeTransaction (apurtell: rev 
5cbf5d6a6ba38f44ad9b9ad029d483f8c6bd6feb)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionMergeTransaction.java


> Region merge request should be audited with request user through proper scope 
> of doAs() calls to region observer notifications
> --
>
> Key: HBASE-14631
> URL: https://issues.apache.org/jira/browse/HBASE-14631
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: 14631-branch-0.98.txt, 14631-branch-1.0.txt, 
> 14631-branch-1.txt, 14631-v1.txt, HBASE-14631-addendum.patch
>
>
> HBASE-14475 and HBASE-14605 narrowed the scope of doAs() calls to region 
> observer notifications for region splitting.
> During review of HBASE-14605, Andrew brought up the case for region merge.
> This JIRA is to implement similar scope narrowing technique for region 
> merging.
> The majority of the change would be in RegionMergeTransactionImpl class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12790) Support fairness across parallelized scans


[ 
https://issues.apache.org/jira/browse/HBASE-12790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987863#comment-14987863
 ] 

Andrew Purtell commented on HBASE-12790:


bq.  So if there is no 1-1 mapping from Mutation/Operation to RPC things get 
ambiguity.

Yes, and we need to document this, but if the proposal is to only support group 
ID for reads and not writes, then I'm not in favor of this change. Clients send 
mixed workloads to the server. Only supporting reads means only half the work 
is done and the feature would be only half useful.

> Support fairness across parallelized scans
> --
>
> Key: HBASE-12790
> URL: https://issues.apache.org/jira/browse/HBASE-12790
> Project: HBase
>  Issue Type: New Feature
>Reporter: James Taylor
>Assignee: ramkrishna.s.vasudevan
>  Labels: Phoenix
> Attachments: AbstractRoundRobinQueue.java, HBASE-12790.patch, 
> HBASE-12790_1.patch, HBASE-12790_5.patch, HBASE-12790_callwrapper.patch, 
> HBASE-12790_trunk_1.patch, PHOENIX_4.5.3-HBase-0.98-2317-SNAPSHOT.zip
>
>
> Some HBase clients parallelize the execution of a scan to reduce latency in 
> getting back results. This can lead to starvation with a loaded cluster and 
> interleaved scans, since the RPC queue will be ordered and processed on a 
> FIFO basis. For example, if there are two clients, A & B that submit largish 
> scans at the same time. Say each scan is broken down into 100 scans by the 
> client (broken down into equal depth chunks along the row key), and the 100 
> scans of client A are queued first, followed immediately by the 100 scans of 
> client B. In this case, client B will be starved out of getting any results 
> back until the scans for client A complete.
> One solution to this is to use the attached AbstractRoundRobinQueue instead 
> of the standard FIFO queue. The queue to be used could be (maybe it already 
> is) configurable based on a new config parameter. Using this queue would 
> require the client to have the same identifier for all of the 100 parallel 
> scans that represent a single logical scan from the clients point of view. 
> With this information, the round robin queue would pick off a task from the 
> queue in a round robin fashion (instead of a strictly FIFO manner) to prevent 
> starvation over interleaved parallelized scans.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14746) Run less client threads in tests


 [ 
https://issues.apache.org/jira/browse/HBASE-14746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-14746:
--
Attachment: lessthreads.txt

Thanks for review [~jmhsieh]

Failure could be related.

Would like to strip more threads if possible... will keep looking here.

> Run less client threads in tests
> 
>
> Key: HBASE-14746
> URL: https://issues.apache.org/jira/browse/HBASE-14746
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Attachments: lessthreads.txt, lessthreads.txt
>
>
> Looking at thread counts reported by the nice 'after:' log message, I see the 
> likes of TestReplicationKillSlaveRS and all of its derivatives reporting 
> hundreds of threads (shows as over 1000 in one run here).
> Running the test standalone, I see the above run up the threads to 700 
> (only!). Need to figure why the discrepancy.
> The attached patch makes some difference -- cutting down the client thread 
> count -- but not enough. Let me look more.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14589) Looking for the surefire-killer; builds being killed...ExecutionException: java.lang.RuntimeException: The forked VM terminated without properly saying goodbye. VM cra


[ 
https://issues.apache.org/jira/browse/HBASE-14589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987697#comment-14987697
 ] 

Hudson commented on HBASE-14589:


SUCCESS: Integrated in HBase-TRUNK #6990 (See 
[https://builds.apache.org/job/HBase-TRUNK/6990/])
HBASE-14589 Looking for the surefire-killer; builds being killed...; (stack: 
rev 0eae729ffa95828462737c54c7204b34bb73182a)
* dev-support/test-patch.sh


> Looking for the surefire-killer; builds being killed...ExecutionException: 
> java.lang.RuntimeException: The forked VM terminated without properly saying 
> goodbye. VM crash or System.exit called?
> 
>
> Key: HBASE-14589
> URL: https://issues.apache.org/jira/browse/HBASE-14589
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Attachments: 14589.mx.patch, 14589.purge.zombie.killer.patch, 
> 14589.timeout.txt, 14589.txt, 14598.addendum.sufire.timeout.patch, 
> bad_fix.patch
>
>
> I see this in a build that started at two hours ago... about 6:45... its 
> build 15941 on ubuntu-6
> {code}
> WARNING: 2 rogue build processes detected, terminating.
> /bin/kill -9 18640 
> /bin/kill -9 22625 
> {code}
> If I back up to build 15939, started about 3 1/2 hours ago, say, 5:15  I 
> see:
> Running org.apache.hadoop.hbase.client.TestShell
> Killed
> ... but it was running on ubuntu-1 so it doesn't look like we are killing 
> ourselves...  when we do this in test-patch.sh
>   ### Kill any rogue build processes from the last attempt
>   $PS auxwww | $GREP ${PROJECT_NAME}PatchProcess | $AWK '{print $2}' | 
> /usr/bin/xargs -t -I {} /bin/kill -9 {} > /dev/null
> The above code runs in a few places... in test-patch.sh.
> Let me try and add some more info around what is being killed... 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12790) Support fairness across parallelized scans

2015-11-03 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-12790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987853#comment-14987853
 ] 

Anoop Sam John commented on HBASE-12790:


Basically 'groupId' make sense for RPC requests but that is not really exposed 
to user.  So if there is no 1-1 mapping from Mutation/Operation to RPC things 
get ambiguity.

> Support fairness across parallelized scans
> --
>
> Key: HBASE-12790
> URL: https://issues.apache.org/jira/browse/HBASE-12790
> Project: HBase
>  Issue Type: New Feature
>Reporter: James Taylor
>Assignee: ramkrishna.s.vasudevan
>  Labels: Phoenix
> Attachments: AbstractRoundRobinQueue.java, HBASE-12790.patch, 
> HBASE-12790_1.patch, HBASE-12790_5.patch, HBASE-12790_callwrapper.patch, 
> HBASE-12790_trunk_1.patch, PHOENIX_4.5.3-HBase-0.98-2317-SNAPSHOT.zip
>
>
> Some HBase clients parallelize the execution of a scan to reduce latency in 
> getting back results. This can lead to starvation with a loaded cluster and 
> interleaved scans, since the RPC queue will be ordered and processed on a 
> FIFO basis. For example, if there are two clients, A & B that submit largish 
> scans at the same time. Say each scan is broken down into 100 scans by the 
> client (broken down into equal depth chunks along the row key), and the 100 
> scans of client A are queued first, followed immediately by the 100 scans of 
> client B. In this case, client B will be starved out of getting any results 
> back until the scans for client A complete.
> One solution to this is to use the attached AbstractRoundRobinQueue instead 
> of the standard FIFO queue. The queue to be used could be (maybe it already 
> is) configurable based on a new config parameter. Using this queue would 
> require the client to have the same identifier for all of the 100 parallel 
> scans that represent a single logical scan from the clients point of view. 
> With this information, the round robin queue would pick off a task from the 
> queue in a round robin fashion (instead of a strictly FIFO manner) to prevent 
> starvation over interleaved parallelized scans.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14575) Reduce scope of compactions holding region lock


[ 
https://issues.apache.org/jira/browse/HBASE-14575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987609#comment-14987609
 ] 

stack commented on HBASE-14575:
---

You saw my comments [~ted_yu]  I repeat main thrust below:

bq. The original author says of the approach "... I'm not sure if it's 
correct..." I'd be interested in a paragraph on why the current author thinks 
this patch is (correct). Also, what testing has been done on this approach? 
(Testing to verify the approach is what the original fellows working on this 
issue were most concerned about).

> Reduce scope of compactions holding region lock
> ---
>
> Key: HBASE-14575
> URL: https://issues.apache.org/jira/browse/HBASE-14575
> Project: HBase
>  Issue Type: Sub-task
>  Components: Compaction, regionserver
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Attachments: 14575-v1.patch, 14575-v2.patch, 14575-v3.patch, 
> 14575-v4.patch, 14575-v5.patch, 14575.v00.patch
>
>
> Per [~devaraj]'s idea on parent issue, let's see if we can reduce the scope 
> of critical section under which compactions hold the region read lock.
> Here is summary from parent issue:
> Another idea is we can reduce the scope of when the read lock is held during 
> compaction. In theory the compactor only needs a region read lock while 
> deciding what files to compact and at the time of committing the compaction. 
> We're protected from the case of region close events because compactions are 
> checking (every 10k bytes written) if the store has been closed in order to 
> abort in such a case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14723) Fix IT tests split too many times


[ 
https://issues.apache.org/jira/browse/HBASE-14723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987610#comment-14987610
 ] 

Andrew Purtell commented on HBASE-14723:


+1

> Fix IT tests split too many times
> -
>
> Key: HBASE-14723
> URL: https://issues.apache.org/jira/browse/HBASE-14723
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14723.patch
>
>
> Splitting the whole table is happening too often. Lets make this happen less 
> frequently as there are more regions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14575) Reduce scope of compactions holding region lock

2015-11-03 Thread Ted Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987615#comment-14987615
 ] 

Ted Yu commented on HBASE-14575:


Yes, I did see the above comment.

Preparation of such paragraph is under way.

> Reduce scope of compactions holding region lock
> ---
>
> Key: HBASE-14575
> URL: https://issues.apache.org/jira/browse/HBASE-14575
> Project: HBase
>  Issue Type: Sub-task
>  Components: Compaction, regionserver
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Attachments: 14575-v1.patch, 14575-v2.patch, 14575-v3.patch, 
> 14575-v4.patch, 14575-v5.patch, 14575.v00.patch
>
>
> Per [~devaraj]'s idea on parent issue, let's see if we can reduce the scope 
> of critical section under which compactions hold the region read lock.
> Here is summary from parent issue:
> Another idea is we can reduce the scope of when the read lock is held during 
> compaction. In theory the compactor only needs a region read lock while 
> deciding what files to compact and at the time of committing the compaction. 
> We're protected from the case of region close events because compactions are 
> checking (every 10k bytes written) if the store has been closed in order to 
> abort in such a case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14223) Meta WALs are not cleared if meta region was closed and RS aborts


[ 
https://issues.apache.org/jira/browse/HBASE-14223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987872#comment-14987872
 ] 

Enis Soztutar commented on HBASE-14223:
---

Thanks for following up. 
bq. So i have removed filter parameter from SplitLogManager#getFileList()
The filter is needed since we are doing WAL splitting for meta and the regular 
WALs differently. Removing the filter has the effect of splitting (and hance 
recovering) the meta WALs that were not needed to be recovered. This may end up 
bringing back old data that is already deleted from meta, so it is not 
something safe. 

bq.  I'm not sure why file name was deformed (i suspect on PathFilter filter). 
The znodes have the hdfs path encoded in them. The split task is coordinated 
via znodes, so that file name to split is turned into a znode path. 

> Meta WALs are not cleared if meta region was closed and RS aborts
> -
>
> Key: HBASE-14223
> URL: https://issues.apache.org/jira/browse/HBASE-14223
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.4
>
> Attachments: HBASE-14223logs, hbase-14223_v0.patch
>
>
> When an RS opens meta, and later closes it, the WAL(FSHlog) is not closed. 
> The last WAL file just sits there in the RS WAL directory. If RS stops 
> gracefully, the WAL file for meta is deleted. Otherwise if RS aborts, WAL for 
> meta is not cleaned. It is also not split (which is correct) since master 
> determines that the RS no longer hosts meta at the time of RS abort. 
> From a cluster after running ITBLL with CM, I see a lot of {{-splitting}} 
> directories left uncleaned: 
> {code}
> [root@os-enis-dal-test-jun-4-7 cluster-os]# sudo -u hdfs hadoop fs -ls 
> /apps/hbase/data/WALs
> Found 31 items
> drwxr-xr-x   - hbase hadoop  0 2015-06-05 01:14 
> /apps/hbase/data/WALs/hregion-58203265
> drwxr-xr-x   - hbase hadoop  0 2015-06-05 07:54 
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433489308745-splitting
> drwxr-xr-x   - hbase hadoop  0 2015-06-05 09:28 
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433494382959-splitting
> drwxr-xr-x   - hbase hadoop  0 2015-06-05 10:01 
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433498252205-splitting
> ...
> {code}
> The directories contain WALs from meta: 
> {code}
> [root@os-enis-dal-test-jun-4-7 cluster-os]# sudo -u hdfs hadoop fs -ls 
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting
> Found 2 items
> -rw-r--r--   3 hbase hadoop 201608 2015-06-05 03:15 
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433470511501.meta
> -rw-r--r--   3 hbase hadoop  44420 2015-06-05 04:36 
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433474111645.meta
> {code}
> The RS hosted the meta region for some time: 
> {code}
> 2015-06-05 03:14:28,692 INFO  [PostOpenDeployTasks:1588230740] 
> zookeeper.MetaTableLocator: Setting hbase:meta region location in ZooKeeper 
> as os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285
> ...
> 2015-06-05 03:15:17,302 INFO  
> [RS_CLOSE_META-os-enis-dal-test-jun-4-5:16020-0] regionserver.HRegion: Closed 
> hbase:meta,,1.1588230740
> {code}
> In between, a WAL is created: 
> {code}
> 2015-06-05 03:15:11,707 INFO  
> [RS_OPEN_META-os-enis-dal-test-jun-4-5:16020-0-MetaLogRoller] wal.FSHLog: 
> Rolled WAL 
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433470511501.meta
>  with entries=385, filesize=196.88 KB; new WAL 
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433474111645.meta
> {code}
> When CM killed the region server later master did not see these WAL files: 
> {code}
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:46,075 
> INFO  [MASTER_SERVER_OPERATIONS-os-enis-dal-test-jun-4-3:16000-0] 
> master.SplitLogManager: started splitting 2 logs in 
> [hdfs://os-enis-dal-test-jun-4-1.openstacklocal:8020/apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting]
>  for [os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285]
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:47,300 
> INFO  [main-EventThread] wal.WALSplitter: Archived processed log 
>

[jira] [Commented] (HBASE-12790) Support fairness across parallelized scans

2015-11-03 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-12790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987759#comment-14987759
 ] 

ramkrishna.s.vasudevan commented on HBASE-12790:


Thanks for all the comments on the RB. Had an offline discussion with Andy, 
James and Anoop.  I would like to update the discussion here.
We will extend the groupid concept to all the client requests. That includes 
scan, gets, MutateRequest, MultiRequest, Bulkloadrequest etc.
In order to do this we expose the groupId API at the Operation level. This will 
allow every Put, Delete, Increment, Append, Get and Scan to have a grouping id. 
Now at the Rpc layer the scan and gets have one to one mapping with the scan 
requests. So the groupid set on the individual scan/gets can be used to do the 
round robin.
But for MultiRequest there could be 'n' number of actions like Puts, deletes, 
gets etc. And every thing will be mapped to one multiRequest. Since we expose 
groupId at the Operation level it will mean that different actions can have 
different groupids set but at the Rpc layer we take the first groupId as the id 
for the entire multiRequest. I had a concern with this part because users will 
be allowed to set different groupIds but internally we will be using only one 
of them and this point gets hidden from the user totally. May be it could 
confuse the user is what I thought. Overall this groupingId concept is not a 
direct parameter that affects the users result whereas it is more on how the 
server is going to handle the request. 
I can update the patch based on the above feedbacks/discussions. Any more 
queries and feedback are welcome!!


> Support fairness across parallelized scans
> --
>
> Key: HBASE-12790
> URL: https://issues.apache.org/jira/browse/HBASE-12790
> Project: HBase
>  Issue Type: New Feature
>Reporter: James Taylor
>Assignee: ramkrishna.s.vasudevan
>  Labels: Phoenix
> Attachments: AbstractRoundRobinQueue.java, HBASE-12790.patch, 
> HBASE-12790_1.patch, HBASE-12790_5.patch, HBASE-12790_callwrapper.patch, 
> HBASE-12790_trunk_1.patch, PHOENIX_4.5.3-HBase-0.98-2317-SNAPSHOT.zip
>
>
> Some HBase clients parallelize the execution of a scan to reduce latency in 
> getting back results. This can lead to starvation with a loaded cluster and 
> interleaved scans, since the RPC queue will be ordered and processed on a 
> FIFO basis. For example, if there are two clients, A & B that submit largish 
> scans at the same time. Say each scan is broken down into 100 scans by the 
> client (broken down into equal depth chunks along the row key), and the 100 
> scans of client A are queued first, followed immediately by the 100 scans of 
> client B. In this case, client B will be starved out of getting any results 
> back until the scans for client A complete.
> One solution to this is to use the attached AbstractRoundRobinQueue instead 
> of the standard FIFO queue. The queue to be used could be (maybe it already 
> is) configurable based on a new config parameter. Using this queue would 
> require the client to have the same identifier for all of the 100 parallel 
> scans that represent a single logical scan from the clients point of view. 
> With this information, the round robin queue would pick off a task from the 
> queue in a round robin fashion (instead of a strictly FIFO manner) to prevent 
> starvation over interleaved parallelized scans.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-14744) BulkDelete in Hbase Table


 [ 
https://issues.apache.org/jira/browse/HBASE-14744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-14744.
---
Resolution: Invalid

> BulkDelete in Hbase Table
> -
>
> Key: HBASE-14744
> URL: https://issues.apache.org/jira/browse/HBASE-14744
> Project: HBase
>  Issue Type: Bug
>  Components: Deletes
>Affects Versions: 0.98.7
> Environment: OS : Unix
> Java Version : java version "1.8.0_25"
> Hbase Version: Version 0.98.7.12.1509250618_h2
>Reporter: Marimuthu
>
> Hi Anoop/Ted,
> I need to delete bulk records in Hbase table based on certain criteria.
> I copied one piece of code from similar thread and it throws "can not find  
> symbol" error for most of the code.
> Can you please provide me right code which would be used to delete bulk 
> records
> Here are the errors
> DeleteRowsTest.java:75: error: cannot find symbol
>HTable tableName = new HTable(conf, "salestoolsdata:account");
>  ^
>   symbol:   variable conf
>   location: class DeleteRowsTest
> DeleteRowsTest.java:79: error: no suitable constructor found for 
> SingleColumnValueFilter(char,String,CompareOp,byte[])
> SingleColumnValueFilter scvf = new 
> SingleColumnValueFilter('d',"ISDELETED",CompareOp.EQUAL, 
> Bytes.toBytes("true"));
>^
> constructor 
> SingleColumnValueFilter.SingleColumnValueFilter(byte[],byte[],CompareOp,byte[])
>  is not applicable
>   (argument mismatch; char cannot be converted to byte[])
> constructor 
> SingleColumnValueFilter.SingleColumnValueFilter(byte[],byte[],CompareOp,ByteArrayComparable)
>  is not applicable
>   (argument mismatch; char cannot be converted to byte[])
> DeleteRowsTest.java:85: error: cannot find symbol
> long noOfRowsDeleted = invokeBulkDeleteProtocol(tableName, scan, 500, 
> DeleteType.ROW, null);
>   ^
>   symbol:   variable DeleteType
>   location: class DeleteRowsTest
> DeleteRowsTest.java:89: error: cannot find symbol
> for (Result result : ht.getScanner(new Scan())) {
>  ^
>   symbol:   variable ht
>   location: class DeleteRowsTest
> DeleteRowsTest.java:98: error: cannot find symbol
> HTable ht = new HTable(conf, tableName);
>^
>   symbol:   variable conf
>   location: class DeleteRowsTest
> DeleteRowsTest.java:100: error: cannot find symbol
> Batch.Call callable = 
>^
>   symbol:   class BulkDeleteProtocol
>   location: class DeleteRowsTest
> DeleteRowsTest.java:100: error: cannot find symbol
> Batch.Call callable = 
>^
>   symbol:   class BulkDeleteResponse
>   location: class DeleteRowsTest
> DeleteRowsTest.java:101: error: cannot find symbol
> new Batch.Call() {
>^
>   symbol:   class BulkDeleteProtocol
>   location: class DeleteRowsTest
> DeleteRowsTest.java:101: error: cannot find symbol
> new Batch.Call() {
>^
>   symbol:   class BulkDeleteResponse
>   location: class DeleteRowsTest
> DeleteRowsTest.java:102: error: cannot find symbol
>   public BulkDeleteResponse call(BulkDeleteProtocol instance) throws 
> IOException {
>  ^
>   symbol: class BulkDeleteProtocol
> DeleteRowsTest.java:102: error: cannot find symbol
>   public BulkDeleteResponse call(BulkDeleteProtocol instance) throws 
> IOException {
>  ^
>   symbol: class BulkDeleteResponse
> DeleteRowsTest.java:106: error: cannot find symbol
> Map result = 
> ht.coprocessorExec(BulkDeleteProtocol.class,
> ^
>   symbol:   class BulkDeleteResponse
>   location: class DeleteRowsTest
> DeleteRowsTest.java:106: error: cannot find symbol
> Map result = 
> ht.coprocessorExec(BulkDeleteProtocol.class,
> ^
>   symbol:   class BulkDeleteProtocol
>   location: class DeleteRowsTest
> DeleteRowsTest.java:108: error: cannot find symbol
> for (BulkDeleteResponse response : result.values()) {
>  ^
>   symbol:   class BulkDeleteResponse
>   location: class DeleteRowsTest
> Note: Some messages have been simplified; recompile with -Xdiags:verbose to 
> get full output
> 18 errors
> Thanks
> Marimuthu



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14744) BulkDelete in Hbase Table


[ 
https://issues.apache.org/jira/browse/HBASE-14744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987793#comment-14987793
 ] 

stack commented on HBASE-14744:
---

This is a question for the dev list. Not appropriate to a JIRA.

> BulkDelete in Hbase Table
> -
>
> Key: HBASE-14744
> URL: https://issues.apache.org/jira/browse/HBASE-14744
> Project: HBase
>  Issue Type: Bug
>  Components: Deletes
>Affects Versions: 0.98.7
> Environment: OS : Unix
> Java Version : java version "1.8.0_25"
> Hbase Version: Version 0.98.7.12.1509250618_h2
>Reporter: Marimuthu
>
> Hi Anoop/Ted,
> I need to delete bulk records in Hbase table based on certain criteria.
> I copied one piece of code from similar thread and it throws "can not find  
> symbol" error for most of the code.
> Can you please provide me right code which would be used to delete bulk 
> records
> Here are the errors
> DeleteRowsTest.java:75: error: cannot find symbol
>HTable tableName = new HTable(conf, "salestoolsdata:account");
>  ^
>   symbol:   variable conf
>   location: class DeleteRowsTest
> DeleteRowsTest.java:79: error: no suitable constructor found for 
> SingleColumnValueFilter(char,String,CompareOp,byte[])
> SingleColumnValueFilter scvf = new 
> SingleColumnValueFilter('d',"ISDELETED",CompareOp.EQUAL, 
> Bytes.toBytes("true"));
>^
> constructor 
> SingleColumnValueFilter.SingleColumnValueFilter(byte[],byte[],CompareOp,byte[])
>  is not applicable
>   (argument mismatch; char cannot be converted to byte[])
> constructor 
> SingleColumnValueFilter.SingleColumnValueFilter(byte[],byte[],CompareOp,ByteArrayComparable)
>  is not applicable
>   (argument mismatch; char cannot be converted to byte[])
> DeleteRowsTest.java:85: error: cannot find symbol
> long noOfRowsDeleted = invokeBulkDeleteProtocol(tableName, scan, 500, 
> DeleteType.ROW, null);
>   ^
>   symbol:   variable DeleteType
>   location: class DeleteRowsTest
> DeleteRowsTest.java:89: error: cannot find symbol
> for (Result result : ht.getScanner(new Scan())) {
>  ^
>   symbol:   variable ht
>   location: class DeleteRowsTest
> DeleteRowsTest.java:98: error: cannot find symbol
> HTable ht = new HTable(conf, tableName);
>^
>   symbol:   variable conf
>   location: class DeleteRowsTest
> DeleteRowsTest.java:100: error: cannot find symbol
> Batch.Call callable = 
>^
>   symbol:   class BulkDeleteProtocol
>   location: class DeleteRowsTest
> DeleteRowsTest.java:100: error: cannot find symbol
> Batch.Call callable = 
>^
>   symbol:   class BulkDeleteResponse
>   location: class DeleteRowsTest
> DeleteRowsTest.java:101: error: cannot find symbol
> new Batch.Call() {
>^
>   symbol:   class BulkDeleteProtocol
>   location: class DeleteRowsTest
> DeleteRowsTest.java:101: error: cannot find symbol
> new Batch.Call() {
>^
>   symbol:   class BulkDeleteResponse
>   location: class DeleteRowsTest
> DeleteRowsTest.java:102: error: cannot find symbol
>   public BulkDeleteResponse call(BulkDeleteProtocol instance) throws 
> IOException {
>  ^
>   symbol: class BulkDeleteProtocol
> DeleteRowsTest.java:102: error: cannot find symbol
>   public BulkDeleteResponse call(BulkDeleteProtocol instance) throws 
> IOException {
>  ^
>   symbol: class BulkDeleteResponse
> DeleteRowsTest.java:106: error: cannot find symbol
> Map result = 
> ht.coprocessorExec(BulkDeleteProtocol.class,
> ^
>   symbol:   class BulkDeleteResponse
>   location: class DeleteRowsTest
> DeleteRowsTest.java:106: error: cannot find symbol
> Map result = 
> ht.coprocessorExec(BulkDeleteProtocol.class,
> ^
>   symbol:   class BulkDeleteProtocol
>   location: class DeleteRowsTest
> DeleteRowsTest.java:108: error: cannot find symbol
> for (BulkDeleteResponse response : result.values()) {
>  ^
>   symbol:   class BulkDeleteResponse
>   location: class DeleteRowsTest
> Note: Some messages have been simplified; recompile with -Xdiags:verbose to 
> get full output
> 18 errors
> Thanks
> Marimuthu



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14030) HBase Backup/Restore Phase 1

2015-11-03 Thread Vladimir Rodionov (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov updated HBASE-14030:
--
Status: Patch Available  (was: Open)

> HBase Backup/Restore Phase 1
> 
>
> Key: HBASE-14030
> URL: https://issues.apache.org/jira/browse/HBASE-14030
> Project: HBase
>  Issue Type: Umbrella
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-14030-v0.patch, HBASE-14030-v1.patch, 
> HBASE-14030-v10.patch, HBASE-14030-v11.patch, HBASE-14030-v12.patch, 
> HBASE-14030-v13.patch, HBASE-14030-v14.patch, HBASE-14030-v15.patch, 
> HBASE-14030-v2.patch, HBASE-14030-v3.patch, HBASE-14030-v4.patch, 
> HBASE-14030-v5.patch, HBASE-14030-v6.patch, HBASE-14030-v7.patch, 
> HBASE-14030-v8.patch
>
>
> This is the umbrella ticket for Backup/Restore Phase 1. See HBASE-7912 design 
> doc for the phase description.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-9049) Generalize ServerCallable creation to support custom callables


 [ 
https://issues.apache.org/jira/browse/HBASE-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-9049:
--
Fix Version/s: 0.98.0
   0.95.2
   0.94.11

> Generalize ServerCallable creation to support custom callables
> --
>
> Key: HBASE-9049
> URL: https://issues.apache.org/jira/browse/HBASE-9049
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.0, 0.95.2, 0.94.11
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Fix For: 0.98.0, 0.95.2, 0.94.11, 2.0.0
>
> Attachments: hbase-9049-trunk-v0.patch, hbase-9049-trunk-v1.patch, 
> hbase-9049-trunk-v2.patch, hbase-9049-trunk-v3.patch, 
> hbase-9049-trunk-v4.patch
>
>
> Currently, sever callables are instantiated via direct calls. Instead, we can 
> use a single factory and that allows more specialized callable 
> implementations, for instance, using a circuit-breaker pattern (or the 
> Hystrix implementation!) to minimize attempts to contact the server.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14030) HBase Backup/Restore Phase 1

2015-11-03 Thread Vladimir Rodionov (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov updated HBASE-14030:
--
Attachment: HBASE-14030-v15.patch

v15. rebased to current master

> HBase Backup/Restore Phase 1
> 
>
> Key: HBASE-14030
> URL: https://issues.apache.org/jira/browse/HBASE-14030
> Project: HBase
>  Issue Type: Umbrella
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-14030-v0.patch, HBASE-14030-v1.patch, 
> HBASE-14030-v10.patch, HBASE-14030-v11.patch, HBASE-14030-v12.patch, 
> HBASE-14030-v13.patch, HBASE-14030-v14.patch, HBASE-14030-v15.patch, 
> HBASE-14030-v2.patch, HBASE-14030-v3.patch, HBASE-14030-v4.patch, 
> HBASE-14030-v5.patch, HBASE-14030-v6.patch, HBASE-14030-v7.patch, 
> HBASE-14030-v8.patch
>
>
> This is the umbrella ticket for Backup/Restore Phase 1. See HBASE-7912 design 
> doc for the phase description.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14736) ITBLL debugging search tool OOMEs on big dataset


[ 
https://issues.apache.org/jira/browse/HBASE-14736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988294#comment-14988294
 ] 

stack commented on HBASE-14736:
---

I tried just getting first 1k in each partition. Was then having interesting 
issue where WALs were moving out from under me while the task ran... they were 
no longer found.  I got the first-1k job to pass but it found no keys Need 
to dig in more. Seems like when the scale is large, these tools as they are 
written no longer work (I lost use of cluster so could pursue no further for 
the time being).

> ITBLL debugging search tool OOMEs on big dataset
> 
>
> Key: HBASE-14736
> URL: https://issues.apache.org/jira/browse/HBASE-14736
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>
> I ran an ITBLL on an 80 node cluster sized to do 100B items. The job failed 
> with 300M undefined items (branch-1). I tried to run the search tool 
> debugging the loss -- see 
> https://docs.google.com/document/d/14Tvu5yWYNBDFkh8xCqLkU9tlyNWhJv3GjDGOkqZU1eE/edit#
>  -- but it OOME'd:
> {code}
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at 
> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:834)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:697)
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
> at 
> org.apache.hadoop.io.SequenceFile$Reader.nextRawKey(SequenceFile.java:2452)
> at 
> org.apache.hadoop.mapreduce.lib.input.SequenceFileAsBinaryInputFormat$SequenceFileAsBinaryRecordReader.nextKeyValue(SequenceFileAsBinaryInputFormat.java:119)
> at 
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Search.readFileToSearch(IntegrationTestBigLinkedList.java:775)
> at 
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Search.readKeysToSearch(IntegrationTestBigLinkedList.java:757)
> at 
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Search.run(IntegrationTestBigLinkedList.java:726)
> at 
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Search.run(IntegrationTestBigLinkedList.java:657)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at 
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList.runTestFromCommandLine(IntegrationTestBigLinkedList.java:1646)
> at 
> org.apache.hadoop.hbase.IntegrationTestBase.doWork(IntegrationTestBase.java:123)
> at 
> org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:112)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at 
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList.main(IntegrationTestBigLinkedList.java:1686)
> {code}
> Its trying to build a sorted set out of the 300M items Dang.
> The 10B test passed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14738) Backport HBASE-11927 (Use Native Hadoop Library for HFile checksum) to 0.98


[ 
https://issues.apache.org/jira/browse/HBASE-14738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988273#comment-14988273
 ] 

Andrew Purtell commented on HBASE-14738:


What do you think [~lhofhansl], [~stack]. Looking to get this into 0.98.16

> Backport HBASE-11927 (Use Native Hadoop Library for HFile checksum) to 0.98
> ---
>
> Key: HBASE-14738
> URL: https://issues.apache.org/jira/browse/HBASE-14738
> Project: HBase
>  Issue Type: Task
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
> Fix For: 0.98.16
>
> Attachments: HBASE-14738-0.98.patch
>
>
> Profiling 0.98.15 I see 20-30% of CPU time spent in Hadoop's PureJavaCrc32. 
> Not surprising given previous results described on HBASE-11927. Backport.
> There are two issues with the backport:
> # The patch on 11927 changes the default CRC type from CRC32 to CRC32C. 
> Although the changes are backwards compatible -files with either CRC type 
> will be handled correctly in a transparent manner - we should probably leave 
> the default alone in 0.98 and advise users on a site configuration change to 
> use CRC32C if desired, for potential hardware acceleration.
> # Need a shim for differences between Hadoop's DataChecksum type.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14664) Master failover issue: Backup master is unable to start if active master is killed and started in short time interval


 [ 
https://issues.apache.org/jira/browse/HBASE-14664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-14664:
--
Attachment: HBASE-14664.patch

TestRegionMover was flakey for a while. Retry.

> Master failover issue: Backup master is unable to start if active master is 
> killed and started in short time interval
> -
>
> Key: HBASE-14664
> URL: https://issues.apache.org/jira/browse/HBASE-14664
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 2.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-14664.patch, HBASE-14664.patch
>
>
> I notice this issue while running IntegrationTestDDLMasterFailover, it can be 
> simply reproduced by executing this on active master (tested on two masters + 
> 3rs cluster setup)
> {code}
> $ kill -9 master_pid; hbase-daemon.sh  start master
> {code} 
> Logs show that new active master is trying to locate hbase:meta table on 
> restarted active master
> {code}
> 2015-10-21 19:28:20,804 INFO  [hnode2:16000.activeMasterManager] 
> zookeeper.MetaTableLocator: Failed verification of hbase:meta,,1 at 
> address=hnode1,16000,1445447051681, 
> exception=org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is 
> not running yet
> at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1092)
> at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionInfo(RSRpcServices.java:1330)
> at 
> org.apache.hadoop.hbase.master.MasterRpcServices.getRegionInfo(MasterRpcServices.java:1525)
> at 
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22233)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2136)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106)
> at 
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> 2015-10-21 19:28:20,805 INFO  [hnode2:16000.activeMasterManager] 
> master.HMaster: Meta was in transition on hnode1,16000,1445447051681
> 2015-10-21 19:28:20,805 INFO  [hnode2:16000.activeMasterManager] 
> master.AssignmentManager: Processing {1588230740 state=OPEN, 
> ts=1445448500598, server=hnode1,16000,1445447051681
> {code}
>  and because of above master is unable to read hbase:meta table:
> {code}
> 2015-10-21 19:28:49,429 INFO  [hconnection-0x6e9cebcc-shared--pool6-t1] 
> client.AsyncProcess: #2, table=hbase:meta, attempt=10/351 failed=1ops, last 
> exception: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: 
> org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not 
> running yet
> at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1092)
> at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2083)
> at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32462)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2136)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106)
> at 
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> which cause master is unable to complete start. 
> I have also notices that in this case value of /hbase/meta-region-server 
> znode is always pointing on restarted active master (hnode1 in my cluster ).
> I was able to workaround this issue by repeating same scenario with following:
> {code}
> $ kill -9 master_pid; hbase zkcli rmr /hbase/meta-region-server; 
> hbase-daemon.sh start master
> {code}
> So issue is probably caused by staled value in /hbase/meta-region-server 
> znode. I will try to create patch based on above.   
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14753) TestShell is not invoked anymore

Enis Soztutar created HBASE-14753:
-

 Summary: TestShell is not invoked anymore
 Key: HBASE-14753
 URL: https://issues.apache.org/jira/browse/HBASE-14753
 Project: HBase
  Issue Type: Bug
Reporter: Enis Soztutar
 Fix For: 2.0.0, 1.2.0, 1.3.0


Not sure whether it is due to HBASE-14679 or not. 

{code}
mvn clean  test -Dtest=TestShell
{code}
does not run any test at all. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11544) [Ergonomics] hbase.client.scanner.caching is dogged and will try to return batch even if it means OOME


[ 
https://issues.apache.org/jira/browse/HBASE-11544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988360#comment-14988360
 ] 

stack commented on HBASE-11544:
---

I think the Scan setting needs to be propagated regardless.  Config won't work 
for those who want the partials on a per-scan basis. Thanks.

> [Ergonomics] hbase.client.scanner.caching is dogged and will try to return 
> batch even if it means OOME
> --
>
> Key: HBASE-11544
> URL: https://issues.apache.org/jira/browse/HBASE-11544
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: Jonathan Lawlor
>Priority: Critical
> Fix For: 2.0.0, 1.1.0
>
> Attachments: Allocation_Hot_Spots.html, 
> HBASE-11544-addendum-v1.patch, HBASE-11544-addendum-v2.patch, 
> HBASE-11544-branch_1_0-v1.patch, HBASE-11544-branch_1_0-v2.patch, 
> HBASE-11544-v1.patch, HBASE-11544-v2.patch, HBASE-11544-v3.patch, 
> HBASE-11544-v4.patch, HBASE-11544-v5.patch, HBASE-11544-v6.patch, 
> HBASE-11544-v6.patch, HBASE-11544-v6.patch, HBASE-11544-v7.patch, 
> HBASE-11544-v8-branch-1.patch, HBASE-11544-v8.patch, gc.j.png, h.png, 
> hits.j.png, m.png, mean.png, net.j.png, q (2).png
>
>
> Running some tests, I set hbase.client.scanner.caching=1000.  Dataset has 
> large cells.  I kept OOME'ing.
> Serverside, we should measure how much we've accumulated and return to the 
> client whatever we've gathered once we pass out a certain size threshold 
> rather than keep accumulating till we OOME.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11544) [Ergonomics] hbase.client.scanner.caching is dogged and will try to return batch even if it means OOME


[ 
https://issues.apache.org/jira/browse/HBASE-11544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988358#comment-14988358
 ] 

stack commented on HBASE-11544:
---

This seems like a bug. File an issue please [~mindaugas.genu...@nomagic.com]

> [Ergonomics] hbase.client.scanner.caching is dogged and will try to return 
> batch even if it means OOME
> --
>
> Key: HBASE-11544
> URL: https://issues.apache.org/jira/browse/HBASE-11544
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: Jonathan Lawlor
>Priority: Critical
> Fix For: 2.0.0, 1.1.0
>
> Attachments: Allocation_Hot_Spots.html, 
> HBASE-11544-addendum-v1.patch, HBASE-11544-addendum-v2.patch, 
> HBASE-11544-branch_1_0-v1.patch, HBASE-11544-branch_1_0-v2.patch, 
> HBASE-11544-v1.patch, HBASE-11544-v2.patch, HBASE-11544-v3.patch, 
> HBASE-11544-v4.patch, HBASE-11544-v5.patch, HBASE-11544-v6.patch, 
> HBASE-11544-v6.patch, HBASE-11544-v6.patch, HBASE-11544-v7.patch, 
> HBASE-11544-v8-branch-1.patch, HBASE-11544-v8.patch, gc.j.png, h.png, 
> hits.j.png, m.png, mean.png, net.j.png, q (2).png
>
>
> Running some tests, I set hbase.client.scanner.caching=1000.  Dataset has 
> large cells.  I kept OOME'ing.
> Serverside, we should measure how much we've accumulated and return to the 
> client whatever we've gathered once we pass out a certain size threshold 
> rather than keep accumulating till we OOME.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex


[ 
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988401#comment-14988401
 ] 

stack commented on HBASE-14279:
---

Can we get the LockedStripedBag into a state where it could be tried in place 
of ConcurrentIndex?

> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
>  Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch, 
> LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible 
> to remove a non-empty set, and to add a value to a removed set. Also 
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes 
> trace the current state and sometimes doesn't.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14481) Decommission HBase wiki


 [ 
https://issues.apache.org/jira/browse/HBASE-14481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Misty Stanley-Jones updated HBASE-14481:

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Changed the commit message as per request, and pushed to master.

> Decommission HBase wiki
> ---
>
> Key: HBASE-14481
> URL: https://issues.apache.org/jira/browse/HBASE-14481
> Project: HBase
>  Issue Type: Task
>  Components: documentation
>Reporter: Nick Dimiduk
>Assignee: Misty Stanley-Jones
>Priority: Blocker
>  Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-14481.patch
>
>
> We have an awesome community resource in our [online 
> book|http://hbase.apache.org/book.html]. It's maintained and looked after 
> with diligence. We also have an HBase section on the [hadoop 
> wiki|http://wiki.apache.org/hadoop/Hbase] that hasn't been updated since 
> 2012. Let's sift through the pages of the wiki, bring over any content that's 
> still relevant and not already present in the book, and kill the wiki.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14490) [RpcServer] reuse request read buffer


[ 
https://issues.apache.org/jira/browse/HBASE-14490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988163#comment-14988163
 ] 

stack commented on HBASE-14490:
---

Would you like me to try out the patch [~gzh1992n]? Isn't the 
BoundedByteBufferPool offheap by default now? Was it in your test?  If so, 
especially if the throughput was close, it would be an added advantage going 
the BBBP route.

> [RpcServer] reuse request read buffer
> -
>
> Key: HBASE-14490
> URL: https://issues.apache.org/jira/browse/HBASE-14490
> Project: HBase
>  Issue Type: Improvement
>  Components: IPC/RPC
>Affects Versions: 2.0.0, 1.0.2
>Reporter: Zephyr Guo
>Assignee: Zephyr Guo
>  Labels: performance
> Fix For: 2.0.0, 1.0.2
>
> Attachments: ByteBufferPool.java, HBASE-14490-v1.patch, 
> HBASE-14490-v10.patch, HBASE-14490-v11.patch, HBASE-14490-v12.patch, 
> HBASE-14490-v2.patch, HBASE-14490-v3.patch, HBASE-14490-v4.patch, 
> HBASE-14490-v5.patch, HBASE-14490-v6.patch, HBASE-14490-v7.patch, 
> HBASE-14490-v8.patch, HBASE-14490-v9.patch
>
>
> Reuse buffer to read request.It's not necessary to every request free 
> buffer.The idea of optimization is to reduce the times that allocate 
> ByteBuffer.
> *Modification*
> 1. {{saslReadAndProcess}} ,{{processOneRpc}}, {{processUnwrappedData}}, 
> {{processConnectionHeader}} accept a ByteBuffer instead of byte[].They can 
> move {{ByteBuffer.position}} correctly when we have read the data.
> 2. {{processUnwrappedData}} no longer use any extra memory.
> 3. Maintaining a reused ByteBuffer in each {{Connection}}.
> a) Size is dynamic to fit specific client.
> b) If size deviate far from average size, we replace a new one.
> c) Using a SoftReference to reference the buffer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14715) Add javadocs to DelegatingRetryingCallable


 [ 
https://issues.apache.org/jira/browse/HBASE-14715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-14715:
---
Fix Version/s: 0.98.16
   1.1.3
   1.0.3
   1.3.0
   1.2.0

> Add javadocs to DelegatingRetryingCallable
> --
>
> Key: HBASE-14715
> URL: https://issues.apache.org/jira/browse/HBASE-14715
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Reporter: Jesse Yates
>Assignee: Jesse Yates
>Priority: Trivial
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: hbase-14715-trunk-v0.patch
>
>
> Follow up cleanup from HBASE-9049



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14715) Add javadocs to DelegatingRetryingCallable


[ 
https://issues.apache.org/jira/browse/HBASE-14715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988198#comment-14988198
 ] 

Andrew Purtell commented on HBASE-14715:


This changes something that went in into ~0.95, picked back to all relevant 
branches. 

> Add javadocs to DelegatingRetryingCallable
> --
>
> Key: HBASE-14715
> URL: https://issues.apache.org/jira/browse/HBASE-14715
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Reporter: Jesse Yates
>Assignee: Jesse Yates
>Priority: Trivial
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: hbase-14715-trunk-v0.patch
>
>
> Follow up cleanup from HBASE-9049



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14747) Make it possible to build Javadoc and xref reports for 0.94 again


[ 
https://issues.apache.org/jira/browse/HBASE-14747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988267#comment-14988267
 ] 

stack commented on HBASE-14747:
---

I'd say just push this Misty. I'll try it out up on hbase.org.

> Make it possible to build Javadoc and xref reports for 0.94 again
> -
>
> Key: HBASE-14747
> URL: https://issues.apache.org/jira/browse/HBASE-14747
> Project: HBase
>  Issue Type: Task
>  Components: build
>Affects Versions: 0.94.27
>Reporter: Misty Stanley-Jones
>Assignee: Misty Stanley-Jones
> Fix For: 0.94.28
>
> Attachments: HBASE-14747-0.94.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14523) rolling-restart.sh --graceful will start regionserver process on master node


 [ 
https://issues.apache.org/jira/browse/HBASE-14523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-14523:
--
Attachment: HBASE-14523v2.patch

Retry

Sorry [~asamir] about the killed test run. I think these have been addressed 
now (HBASE-14589). The last run failed fork a test which likely lack of 
resources on test machine... TBD.

> rolling-restart.sh --graceful will start regionserver process on master node
> 
>
> Key: HBASE-14523
> URL: https://issues.apache.org/jira/browse/HBASE-14523
> Project: HBase
>  Issue Type: Bug
>  Components: scripts
>Affects Versions: 2.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
> Fix For: 2.0.0
>
> Attachments: HBASE-14523.patch, HBASE-14523v2.patch, 
> HBASE-14523v2.patch, HBASE-14523v2.patch
>
>
> In master branch master acts also as regionserver hosting 'hbase:meta' table 
> and it has ephemeral znode created in '/hbase/rs'. Because of this 
> rolling-restart.sh --graceful will pick up master server from zk at this line:
> {code}
> online_regionservers=`$bin/hbase zkcli ls $zkrs 2>&1 | tail -1 | sed "s/\[//" 
> | sed "s/\]//"`
> {code}
> and will restart it a long with rest of regionservers. 
> I'm planing to add some code to rolling-restart.sh script to filter master 
> server from above  list. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-14495) TestHRegion#testFlushCacheWhileScanning goes zombie


[ 
https://issues.apache.org/jira/browse/HBASE-14495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988308#comment-14988308
 ] 

stack edited comment on HBASE-14495 at 11/3/15 10:34 PM:
-

Yeah. Thats STUCK. I don't have windows box to see diff but something to do w/ 
coarser timer? Can you repro on linux? Maybe there is a means of getting a 
windows timer on a linux box? If we had environmentedge everywhere, I could 
fake it


was (Author: stack):
Yeah. Thats STUCK. I don't have windows box to see diff but something to do w/ 
coarser timer?

> TestHRegion#testFlushCacheWhileScanning goes zombie
> ---
>
> Key: HBASE-14495
> URL: https://issues.apache.org/jira/browse/HBASE-14495
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
> Attachments: 14495.txt, 14495.txt, 14495v3.txt, 14495v6.txt, 
> 14495v7.txt, 14495v9.txt
>
>
> This test goes zombie on us, most recently, here: 
> https://builds.apache.org/job/PreCommit-HBASE-Build/15744//console
> It does not fail on my internal rig runs nor locally on laptop when run in a 
> loop.
> Its hung up in close of the region:
> {code}
> "main" prio=10 tid=0x7fc49800a800 nid=0x6053 in Object.wait() 
> [0x7fc4a02c9000]
>java.lang.Thread.State: WAITING (on object monitor)
>   at java.lang.Object.wait(Native Method)
>   - waiting on <0x0007d07c3478> (a java.lang.Object)
>   at 
> org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.waitForRead(MultiVersionConcurrencyControl.java:207)
>   - locked <0x0007d07c3478> (a java.lang.Object)
>   at 
> org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.completeAndWait(MultiVersionConcurrencyControl.java:143)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2257)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2061)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2026)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2016)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1423)
>   at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1344)
>   - locked <0x0007d07c34a8> (a java.lang.Object)
>   at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1295)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.closeRegionAndWAL(HBaseTestingUtility.java:352)
>   at 
> org.apache.hadoop.hbase.regionserver.TestHRegion.testFlushCacheWhileScanning(TestHRegion.java:3756)
> {code}
> It is waiting on mvcc to catch up.
> There is this comment at the point where we are hung:
> // TODO: Lets see if we hang here, if there is a scenario where 
> an outstanding reader
> // with a read point is in advance of this write point.
> mvcc.completeAndWait(writeEntry);
> The above came in with HBASE-12751. The comment was added at v29:
> https://issues.apache.org/jira/secure/attachment/12754775/12751.rebased.v29.txt
> Looks like I added it so must have had predilection that this might be 
> dodgy... Let me take a look... 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14664) Master failover issue: Backup master is unable to start if active master is killed and started in short time interval


 [ 
https://issues.apache.org/jira/browse/HBASE-14664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-14664:
--
Priority: Critical  (was: Major)

Marking critical. This looks bad w/ a simple fix. I think it needs a test 
though since it a pretty radical fix. deleting the cached hbase;meta. If 
too hard to write a test [~asamir] -- its not that straight-forward -- say so 
and I can help out.

> Master failover issue: Backup master is unable to start if active master is 
> killed and started in short time interval
> -
>
> Key: HBASE-14664
> URL: https://issues.apache.org/jira/browse/HBASE-14664
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 2.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-14664.patch
>
>
> I notice this issue while running IntegrationTestDDLMasterFailover, it can be 
> simply reproduced by executing this on active master (tested on two masters + 
> 3rs cluster setup)
> {code}
> $ kill -9 master_pid; hbase-daemon.sh  start master
> {code} 
> Logs show that new active master is trying to locate hbase:meta table on 
> restarted active master
> {code}
> 2015-10-21 19:28:20,804 INFO  [hnode2:16000.activeMasterManager] 
> zookeeper.MetaTableLocator: Failed verification of hbase:meta,,1 at 
> address=hnode1,16000,1445447051681, 
> exception=org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is 
> not running yet
> at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1092)
> at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionInfo(RSRpcServices.java:1330)
> at 
> org.apache.hadoop.hbase.master.MasterRpcServices.getRegionInfo(MasterRpcServices.java:1525)
> at 
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22233)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2136)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106)
> at 
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> 2015-10-21 19:28:20,805 INFO  [hnode2:16000.activeMasterManager] 
> master.HMaster: Meta was in transition on hnode1,16000,1445447051681
> 2015-10-21 19:28:20,805 INFO  [hnode2:16000.activeMasterManager] 
> master.AssignmentManager: Processing {1588230740 state=OPEN, 
> ts=1445448500598, server=hnode1,16000,1445447051681
> {code}
>  and because of above master is unable to read hbase:meta table:
> {code}
> 2015-10-21 19:28:49,429 INFO  [hconnection-0x6e9cebcc-shared--pool6-t1] 
> client.AsyncProcess: #2, table=hbase:meta, attempt=10/351 failed=1ops, last 
> exception: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: 
> org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not 
> running yet
> at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1092)
> at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2083)
> at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32462)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2136)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106)
> at 
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> which cause master is unable to complete start. 
> I have also notices that in this case value of /hbase/meta-region-server 
> znode is always pointing on restarted active master (hnode1 in my cluster ).
> I was able to workaround this issue by repeating same scenario with following:
> {code}
> $ kill -9 master_pid; hbase zkcli rmr /hbase/meta-region-server; 
> hbase-daemon.sh start master
> {code}
> So issue is probably caused by staled value in /hbase/meta-region-server 
> znode. I will try to create patch based on above.   
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14751) Book seems to be broken


[ 
https://issues.apache.org/jira/browse/HBASE-14751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988373#comment-14988373
 ] 

Hudson commented on HBASE-14751:


FAILURE: Integrated in HBase-Trunk_matrix #428 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/428/])
HBASE-14751 Fix Asciidoc in external API chapter (mstanleyjones: rev 
090fbd3ec862f8c85aa511172dd8e591b3b79332)
* src/main/asciidoc/_chapters/external_apis.adoc


> Book seems to be broken
> ---
>
> Key: HBASE-14751
> URL: https://issues.apache.org/jira/browse/HBASE-14751
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Misty Stanley-Jones
> Fix For: 2.0.0
>
> Attachments: HBASE-14751.patch
>
>
> Seems that the content after: 
> https://hbase.apache.org/book.html#jython seems to be broken. No more titles 
> and links. 
> [~misty] FYI. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14481) Decommission HBase wiki


[ 
https://issues.apache.org/jira/browse/HBASE-14481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988422#comment-14988422
 ] 

Misty Stanley-Jones commented on HBASE-14481:
-

Changed the front page of the Wiki: https://wiki.apache.org/hadoop/Hbase

> Decommission HBase wiki
> ---
>
> Key: HBASE-14481
> URL: https://issues.apache.org/jira/browse/HBASE-14481
> Project: HBase
>  Issue Type: Task
>  Components: documentation
>Reporter: Nick Dimiduk
>Assignee: Misty Stanley-Jones
>Priority: Blocker
>  Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-14481.patch
>
>
> We have an awesome community resource in our [online 
> book|http://hbase.apache.org/book.html]. It's maintained and looked after 
> with diligence. We also have an HBase section on the [hadoop 
> wiki|http://wiki.apache.org/hadoop/Hbase] that hasn't been updated since 
> 2012. Let's sift through the pages of the wiki, bring over any content that's 
> still relevant and not already present in the book, and kill the wiki.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14752) Add example of using the HBase client in a multi-threaded environment


 [ 
https://issues.apache.org/jira/browse/HBASE-14752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-14752:
--
Affects Version/s: 2.0.0
   Status: Patch Available  (was: Open)

> Add example of using the HBase client in a multi-threaded environment
> -
>
> Key: HBASE-14752
> URL: https://issues.apache.org/jira/browse/HBASE-14752
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Affects Versions: 2.0.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Minor
> Attachments: HBASE-14752-v1.patch, HBASE-14752-v2.patch, 
> HBASE-14752-v3.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-14707) NPE spew getting metrics via jmx


 [ 
https://issues.apache.org/jira/browse/HBASE-14707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-14707.
---
Resolution: Invalid

It is not spewing anymore. Will open a new issue when it happens again with a 
fix. This one keep showing up...

> NPE spew getting metrics via jmx
> 
>
> Key: HBASE-14707
> URL: https://issues.apache.org/jira/browse/HBASE-14707
> Project: HBase
>  Issue Type: Bug
>  Components: metrics
>Reporter: stack
>
> See this in branch-1 tip:
> {code}
> 2015-10-27 08:01:08,954 INFO  [main-EventThread] 
> replication.ReplicationTrackerZKImpl: 
> /hbase/rs/e1101.halxg.cloudera.com,16020,1445958006576 znode expired, 
> triggering replicatorRemoved event
> 2015-10-27 08:01:20,645 ERROR [685943200@qtp-893835279-134] util.JSONBean: 
> getting attribute Value of 
> "org.apache.hadoop.hbase.client":type="MetricsConnection",scope="hconnection-0x33abd9d3",name="executorPoolActiveThreads"
>  threw an exception
> javax.management.RuntimeMBeanException: java.lang.NullPointerException
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651)
> at 
> com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
> at 
> org.apache.hadoop.hbase.util.JSONBean.writeAttribute(JSONBean.java:235)
> at org.apache.hadoop.hbase.util.JSONBean.write(JSONBean.java:209)
> at org.apache.hadoop.hbase.util.JSONBean.access$000(JSONBean.java:53)
> at org.apache.hadoop.hbase.util.JSONBean$1.write(JSONBean.java:96)
> at 
> org.apache.hadoop.hbase.http.jmx.JMXJsonServlet.doGet(JMXJsonServlet.java:202)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at 
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> at 
> org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:113)
> at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at 
> org.apache.hadoop.hbase.http.ClickjackingPreventionFilter.doFilter(ClickjackingPreventionFilter.java:48)
> at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at 
> org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1354)
> at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at 
> org.apache.hadoop.hbase.http.NoCacheFilter.doFilter(NoCacheFilter.java:49)
> at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at 
> org.apache.hadoop.hbase.http.NoCacheFilter.doFilter(NoCacheFilter.java:49)
> at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at 
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at 
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at 
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at 
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at 
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at 
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at 
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at 
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at 
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at 
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> at 
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Caused by: java.lang.NullPointerException
> {code}
> TSDB is trying to get metrics on a period



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11544) [Ergonomics] hbase.client.scanner.caching is dogged and will try to return batch even if it means OOME


[ 
https://issues.apache.org/jira/browse/HBASE-11544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988359#comment-14988359
 ] 

stack commented on HBASE-11544:
---

I think the Scan setting needs to be propagated regardless.  Config won't work 
for those who want the partials on a per-scan basis. Thanks.

> [Ergonomics] hbase.client.scanner.caching is dogged and will try to return 
> batch even if it means OOME
> --
>
> Key: HBASE-11544
> URL: https://issues.apache.org/jira/browse/HBASE-11544
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: Jonathan Lawlor
>Priority: Critical
> Fix For: 2.0.0, 1.1.0
>
> Attachments: Allocation_Hot_Spots.html, 
> HBASE-11544-addendum-v1.patch, HBASE-11544-addendum-v2.patch, 
> HBASE-11544-branch_1_0-v1.patch, HBASE-11544-branch_1_0-v2.patch, 
> HBASE-11544-v1.patch, HBASE-11544-v2.patch, HBASE-11544-v3.patch, 
> HBASE-11544-v4.patch, HBASE-11544-v5.patch, HBASE-11544-v6.patch, 
> HBASE-11544-v6.patch, HBASE-11544-v6.patch, HBASE-11544-v7.patch, 
> HBASE-11544-v8-branch-1.patch, HBASE-11544-v8.patch, gc.j.png, h.png, 
> hits.j.png, m.png, mean.png, net.j.png, q (2).png
>
>
> Running some tests, I set hbase.client.scanner.caching=1000.  Dataset has 
> large cells.  I kept OOME'ing.
> Serverside, we should measure how much we've accumulated and return to the 
> client whatever we've gathered once we pass out a certain size threshold 
> rather than keep accumulating till we OOME.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14753) TestShell is not invoked anymore


[ 
https://issues.apache.org/jira/browse/HBASE-14753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988379#comment-14988379
 ] 

stack commented on HBASE-14753:
---

It was disabled over in HBASE-14678 along with TestReplicationShell as part of 
the effort at stabilizing builds.

Builds seem to be settling. Will start reenabling the stuff removed over in 
HBASE-14678 after we get a good run of blue builds.

> TestShell is not invoked anymore
> 
>
> Key: HBASE-14753
> URL: https://issues.apache.org/jira/browse/HBASE-14753
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
>
> Not sure whether it is due to HBASE-14679 or not. 
> {code}
> mvn clean  test -Dtest=TestShell
> {code}
> does not run any test at all. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14753) TestShell is not invoked anymore


[ 
https://issues.apache.org/jira/browse/HBASE-14753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988430#comment-14988430
 ] 

Enis Soztutar commented on HBASE-14753:
---

Ok, sorry for the spam then. I was trying to write a shell test, and was 
confused. Do we need to keep this open for tracking re-enabling or HBASE-14678 
covers that? 

> TestShell is not invoked anymore
> 
>
> Key: HBASE-14753
> URL: https://issues.apache.org/jira/browse/HBASE-14753
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
>
> Not sure whether it is due to HBASE-14679 or not. 
> {code}
> mvn clean  test -Dtest=TestShell
> {code}
> does not run any test at all. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14030) HBase Backup/Restore Phase 1

2015-11-03 Thread Vladimir Rodionov (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov updated HBASE-14030:
--
Status: Open  (was: Patch Available)

> HBase Backup/Restore Phase 1
> 
>
> Key: HBASE-14030
> URL: https://issues.apache.org/jira/browse/HBASE-14030
> Project: HBase
>  Issue Type: Umbrella
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-14030-v0.patch, HBASE-14030-v1.patch, 
> HBASE-14030-v10.patch, HBASE-14030-v11.patch, HBASE-14030-v12.patch, 
> HBASE-14030-v13.patch, HBASE-14030-v14.patch, HBASE-14030-v2.patch, 
> HBASE-14030-v3.patch, HBASE-14030-v4.patch, HBASE-14030-v5.patch, 
> HBASE-14030-v6.patch, HBASE-14030-v7.patch, HBASE-14030-v8.patch
>
>
> This is the umbrella ticket for Backup/Restore Phase 1. See HBASE-7912 design 
> doc for the phase description.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-9049) Generalize ServerCallable creation to support custom callables


 [ 
https://issues.apache.org/jira/browse/HBASE-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-9049:
--
Fix Version/s: 2.0.0

> Generalize ServerCallable creation to support custom callables
> --
>
> Key: HBASE-9049
> URL: https://issues.apache.org/jira/browse/HBASE-9049
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.0, 0.95.2, 0.94.11
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Fix For: 2.0.0
>
> Attachments: hbase-9049-trunk-v0.patch, hbase-9049-trunk-v1.patch, 
> hbase-9049-trunk-v2.patch, hbase-9049-trunk-v3.patch, 
> hbase-9049-trunk-v4.patch
>
>
> Currently, sever callables are instantiated via direct calls. Instead, we can 
> use a single factory and that allows more specialized callable 
> implementations, for instance, using a circuit-breaker pattern (or the 
> Hystrix implementation!) to minimize attempts to contact the server.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14715) Add javadocs to DelegatingRetryingCallable

2015-11-03 Thread Jesse Yates (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988200#comment-14988200
 ] 

Jesse Yates commented on HBASE-14715:
-

Thanks andrew!

> Add javadocs to DelegatingRetryingCallable
> --
>
> Key: HBASE-14715
> URL: https://issues.apache.org/jira/browse/HBASE-14715
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Reporter: Jesse Yates
>Assignee: Jesse Yates
>Priority: Trivial
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: hbase-14715-trunk-v0.patch
>
>
> Follow up cleanup from HBASE-9049



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14747) Make it possible to build Javadoc and xref reports for 0.94 again


 [ 
https://issues.apache.org/jira/browse/HBASE-14747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Misty Stanley-Jones updated HBASE-14747:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Pushed to 0.94 branch. Thanks [~stack]

> Make it possible to build Javadoc and xref reports for 0.94 again
> -
>
> Key: HBASE-14747
> URL: https://issues.apache.org/jira/browse/HBASE-14747
> Project: HBase
>  Issue Type: Task
>  Components: build
>Affects Versions: 0.94.27
>Reporter: Misty Stanley-Jones
>Assignee: Misty Stanley-Jones
> Fix For: 0.94.28
>
> Attachments: HBASE-14747-0.94.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-14748) Update 0.94 apidocs and xref on website


 [ 
https://issues.apache.org/jira/browse/HBASE-14748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Misty Stanley-Jones resolved HBASE-14748.
-
Resolution: Fixed

Pushed to SVN yesterday.

> Update 0.94 apidocs and xref on website
> ---
>
> Key: HBASE-14748
> URL: https://issues.apache.org/jira/browse/HBASE-14748
> Project: HBase
>  Issue Type: Sub-task
>  Components: build
>Reporter: Misty Stanley-Jones
>Assignee: Misty Stanley-Jones
> Fix For: 0.94.28
>
>
> When HBASE-14747 has been merged, update the subdirectories of 0.94/ in SVN. 
> There are lots of broken links there.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14704) Block cache instances should be mapped to each region servers in standalone mode or HBaseTestingUtility


[ 
https://issues.apache.org/jira/browse/HBASE-14704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988333#comment-14988333
 ] 

stack commented on HBASE-14704:
---

Could we do without the static instance completely?

> Block cache instances should be mapped to each region servers in standalone 
> mode or HBaseTestingUtility
> ---
>
> Key: HBASE-14704
> URL: https://issues.apache.org/jira/browse/HBASE-14704
> Project: HBase
>  Issue Type: Bug
>Reporter: Eungsop Yoo
>Priority: Minor
> Attachments: HBASE-14704.patch
>
>
> CacheConfig has a single and static block cache instance. When HBase is 
> running in standalone mode or HBaseTestingUtility, a single instance of block 
> cache causes incorrect cache stats. So I suggest to change a single instance 
> of block cache to a map of region server and block cache.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14700) Support a "permissive" mode for secure clusters to allow "simple" auth clients

2015-11-03 Thread Gary Helmling (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Helmling updated HBASE-14700:
--
Attachment: HBASE-14700_0.98-addendum.patch

Attaching the addendum committed to 0.98, fixing MetricsHBaseServerSourceImpl 
in hadoop1-compat module.

> Support a "permissive" mode for secure clusters to allow "simple" auth clients
> --
>
> Key: HBASE-14700
> URL: https://issues.apache.org/jira/browse/HBASE-14700
> Project: HBase
>  Issue Type: Improvement
>  Components: security
>Reporter: Gary Helmling
>Assignee: Gary Helmling
> Fix For: 2.0.0, 1.2.0, 0.98.16
>
> Attachments: HBASE-14700-v2.patch, HBASE-14700-v3.patch, 
> HBASE-14700.patch, HBASE-14700_0.98-addendum.patch, HBASE-14700_0.98-v1.patch
>
>
> When implementing HBase security for an existing cluster, it can be useful to 
> support mixed secure and insecure clients while all client configurations are 
> migrated over to secure authentication.  
> We currently have an option to allow secure clients to fallback to simple 
> auth against insecure clusters.  By providing an analogous setting for 
> servers, we would allow a phased rollout of security:
> # First, security can be enabled on the cluster servers, with the 
> "permissive" mode enabled
> # Clients can be converting to using secure authentication incrementally
> # The server audit logs allow identification of clients still using simple 
> auth to connect
> # Finally, when sufficient clients have been converted to secure operation, 
> the server-side "permissive" mode can be removed, allowing completely secure 
> operation.
> Obviously with this enabled, there is no effective access control, but this 
> would still be a useful tool to enable a smooth operational rollout of 
> security.  Permissive mode would of course be disabled by default.  Enabling 
> it should provide a big scary warning in the logs on startup, and possibly be 
> flagged on relevant UIs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11393) Replication TableCfs should be a PB object rather than a string


[ 
https://issues.apache.org/jira/browse/HBASE-11393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988011#comment-14988011
 ] 

Enis Soztutar commented on HBASE-11393:
---

Now, thinking more about it, I think we should do a couple of more changes to 
the client-visible API. First, the String based configuration for table cfs 
should completely be deprecated. Second, I think we should unify the 
ReplicationPeerConfiguration and table-cf configuration so that the table CF's 
to filter goes inside the ReplicationPeerConfiguration. Wdyt? 

These are the functions to be deprecated at least:  
{code}
getPeerTableCFs(String id)
appendPeerTableCFs(String id, String tableCfs)
removePeerTableCFs(String id, String tableCf)
{code}



> Replication TableCfs should be a PB object rather than a string
> ---
>
> Key: HBASE-11393
> URL: https://issues.apache.org/jira/browse/HBASE-11393
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Enis Soztutar
> Fix For: 2.0.0
>
> Attachments: HBASE-11393.patch, HBASE-11393_v1.patch, 
> HBASE-11393_v2.patch, HBASE-11393_v3.patch, HBASE-11393_v4.patch, 
> HBASE-11393_v5.patch, HBASE-11393_v6.patch, HBASE-11393_v7.patch
>
>
> We concatenate the list of tables and column families in format  
> "table1:cf1,cf2;table2:cfA,cfB" in zookeeper for table-cf to replication peer 
> mapping. 
> This results in ugly parsing code. We should do this a PB object. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14751) Book seems to be broken


 [ 
https://issues.apache.org/jira/browse/HBASE-14751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Misty Stanley-Jones updated HBASE-14751:

   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.0.0
   Status: Resolved  (was: Patch Available)

> Book seems to be broken
> ---
>
> Key: HBASE-14751
> URL: https://issues.apache.org/jira/browse/HBASE-14751
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Misty Stanley-Jones
> Fix For: 2.0.0
>
> Attachments: HBASE-14751.patch
>
>
> Seems that the content after: 
> https://hbase.apache.org/book.html#jython seems to be broken. No more titles 
> and links. 
> [~misty] FYI. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14752) Add example of using the HBase client in a multi-threaded environment


 [ 
https://issues.apache.org/jira/browse/HBASE-14752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-14752:
--
Attachment: HBASE-14752-v1.patch

Very rough example. It needs a lot more clean up and comments but it's a 
starting place.

It shows how to have different connections for different purposes. How to 
create and throw away tables.

> Add example of using the HBase client in a multi-threaded environment
> -
>
> Key: HBASE-14752
> URL: https://issues.apache.org/jira/browse/HBASE-14752
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Minor
> Attachments: HBASE-14752-v1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14751) Book seems to be broken


[ 
https://issues.apache.org/jira/browse/HBASE-14751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988134#comment-14988134
 ] 

stack commented on HBASE-14751:
---

+

> Book seems to be broken
> ---
>
> Key: HBASE-14751
> URL: https://issues.apache.org/jira/browse/HBASE-14751
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Misty Stanley-Jones
> Attachments: HBASE-14751.patch
>
>
> Seems that the content after: 
> https://hbase.apache.org/book.html#jython seems to be broken. No more titles 
> and links. 
> [~misty] FYI. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14752) Add example of using the HBase client in a multi-threaded environment


 [ 
https://issues.apache.org/jira/browse/HBASE-14752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-14752:
--
Attachment: HBASE-14752-v2.patch

Need to wait for all the futures.

> Add example of using the HBase client in a multi-threaded environment
> -
>
> Key: HBASE-14752
> URL: https://issues.apache.org/jira/browse/HBASE-14752
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Minor
> Attachments: HBASE-14752-v1.patch, HBASE-14752-v2.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14751) Book seems to be broken


 [ 
https://issues.apache.org/jira/browse/HBASE-14751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Misty Stanley-Jones updated HBASE-14751:

Status: Patch Available  (was: Open)

> Book seems to be broken
> ---
>
> Key: HBASE-14751
> URL: https://issues.apache.org/jira/browse/HBASE-14751
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Misty Stanley-Jones
> Attachments: HBASE-14751.patch
>
>
> Seems that the content after: 
> https://hbase.apache.org/book.html#jython seems to be broken. No more titles 
> and links. 
> [~misty] FYI. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14752) Add example of using the HBase client in a multi-threaded environment

Elliott Clark created HBASE-14752:
-

 Summary: Add example of using the HBase client in a multi-threaded 
environment
 Key: HBASE-14752
 URL: https://issues.apache.org/jira/browse/HBASE-14752
 Project: HBase
  Issue Type: Improvement
  Components: Client
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14751) Book seems to be broken


 [ 
https://issues.apache.org/jira/browse/HBASE-14751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Misty Stanley-Jones updated HBASE-14751:

Attachment: HBASE-14751.patch

Fixed leftovers from a bad conflict resolution.

> Book seems to be broken
> ---
>
> Key: HBASE-14751
> URL: https://issues.apache.org/jira/browse/HBASE-14751
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Misty Stanley-Jones
> Attachments: HBASE-14751.patch
>
>
> Seems that the content after: 
> https://hbase.apache.org/book.html#jython seems to be broken. No more titles 
> and links. 
> [~misty] FYI. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-14751) Book seems to be broken


[ 
https://issues.apache.org/jira/browse/HBASE-14751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988134#comment-14988134
 ] 

stack edited comment on HBASE-14751 at 11/3/15 9:13 PM:


+1


was (Author: stack):
+

> Book seems to be broken
> ---
>
> Key: HBASE-14751
> URL: https://issues.apache.org/jira/browse/HBASE-14751
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Misty Stanley-Jones
> Attachments: HBASE-14751.patch
>
>
> Seems that the content after: 
> https://hbase.apache.org/book.html#jython seems to be broken. No more titles 
> and links. 
> [~misty] FYI. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13408) HBase In-Memory Memstore Compaction


[ 
https://issues.apache.org/jira/browse/HBASE-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988156#comment-14988156
 ] 

stack commented on HBASE-13408:
---

I took a look at a bit of review board. I left some comments there. Seems like 
the whole notion of snapshot should not be exposed to the client. It is an 
implementation detail of the original memstore, the defaultmemstore, something 
that we should try not expose.

> HBase In-Memory Memstore Compaction
> ---
>
> Key: HBASE-13408
> URL: https://issues.apache.org/jira/browse/HBASE-13408
> Project: HBase
>  Issue Type: New Feature
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
> Fix For: 2.0.0
>
> Attachments: HBASE-13408-trunk-v01.patch, 
> HBASE-13408-trunk-v02.patch, HBASE-13408-trunk-v03.patch, 
> HBASE-13408-trunk-v04.patch, HBASE-13408-trunk-v05.patch, 
> HBASE-13408-trunk-v06.patch, HBASE-13408-trunk-v07.patch, 
> HBASE-13408-trunk-v08.patch, 
> HBaseIn-MemoryMemstoreCompactionDesignDocument-ver02.pdf, 
> HBaseIn-MemoryMemstoreCompactionDesignDocument-ver03.pdf, 
> HBaseIn-MemoryMemstoreCompactionDesignDocument.pdf, 
> InMemoryMemstoreCompactionEvaluationResults.pdf, 
> InMemoryMemstoreCompactionMasterEvaluationResults.pdf, 
> InMemoryMemstoreCompactionScansEvaluationResults.pdf, 
> StoreSegmentandStoreSegmentScannerClassHierarchies.pdf
>
>
> A store unit holds a column family in a region, where the memstore is its 
> in-memory component. The memstore absorbs all updates to the store; from time 
> to time these updates are flushed to a file on disk, where they are 
> compacted. Unlike disk components, the memstore is not compacted until it is 
> written to the filesystem and optionally to block-cache. This may result in 
> underutilization of the memory due to duplicate entries per row, for example, 
> when hot data is continuously updated. 
> Generally, the faster the data is accumulated in memory, more flushes are 
> triggered, the data sinks to disk more frequently, slowing down retrieval of 
> data, even if very recent.
> In high-churn workloads, compacting the memstore can help maintain the data 
> in memory, and thereby speed up data retrieval. 
> We suggest a new compacted memstore with the following principles:
> 1.The data is kept in memory for as long as possible
> 2.Memstore data is either compacted or in process of being compacted 
> 3.Allow a panic mode, which may interrupt an in-progress compaction and 
> force a flush of part of the memstore.
> We suggest applying this optimization only to in-memory column families.
> A design document is attached.
> This feature was previously discussed in HBASE-5311.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14751) Book seems to be broken

Enis Soztutar created HBASE-14751:
-

 Summary: Book seems to be broken
 Key: HBASE-14751
 URL: https://issues.apache.org/jira/browse/HBASE-14751
 Project: HBase
  Issue Type: Bug
Reporter: Enis Soztutar


Seems that the content after: 
https://hbase.apache.org/book.html#jython seems to be broken. No more titles 
and links. 
[~misty] FYI. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-14751) Book seems to be broken


 [ 
https://issues.apache.org/jira/browse/HBASE-14751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Misty Stanley-Jones reassigned HBASE-14751:
---

Assignee: Misty Stanley-Jones

> Book seems to be broken
> ---
>
> Key: HBASE-14751
> URL: https://issues.apache.org/jira/browse/HBASE-14751
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Misty Stanley-Jones
>
> Seems that the content after: 
> https://hbase.apache.org/book.html#jython seems to be broken. No more titles 
> and links. 
> [~misty] FYI. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14751) Book seems to be broken


[ 
https://issues.apache.org/jira/browse/HBASE-14751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988149#comment-14988149
 ] 

Enis Soztutar commented on HBASE-14751:
---

Go for it! 

> Book seems to be broken
> ---
>
> Key: HBASE-14751
> URL: https://issues.apache.org/jira/browse/HBASE-14751
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Misty Stanley-Jones
> Fix For: 2.0.0
>
> Attachments: HBASE-14751.patch
>
>
> Seems that the content after: 
> https://hbase.apache.org/book.html#jython seems to be broken. No more titles 
> and links. 
> [~misty] FYI. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13408) HBase In-Memory Memstore Compaction

[
https://issues.apache.org/jira/browse/HBASE-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988099#comment-14988099
]

stack commented on HBASE-13408:
---

Thanks for the update to the design doc. It is great.

"This may result in underutilization of the memory due to duplicate entries per
row, for example, when hot data is continuously updated."

Other reasons for compactions in memory would be to move data from an
expensive, fat if-convenient data structure -- the skip list -- to a data
structure that is more compact in memory and more efficient to access (can
exploit the fact that it is read-only, can trigger the in-memory flush when
memstore hits some 'small' size -- should improve perf). The latter is
especially pertinent in the case where a flush is encoded and/or compressed
such that the in-memory representation may be, say 128MB, but then the hfile on
disk is much smaller than this. IIRC we have seen cases where it takes longer
to pull an entry from a large skip list than it does from cache. We could also
remove 'duplicates' by coalescing increments, applying deletes to deleted data
if present in memory.

Downsides to in-memory compaction is that we carry a larger backlog of WALs
making MTTR take longer.

bq. Data is kept in memory for as long as possible, by periodically compacting
the memstore content.

I think the principal should be more that you will afford yourself the luxury
of being able to make smarter decisions on when to flush (Keeping data in
memory as long as possible could actual be detrimental over the long run).

bq. A flush call may interrupt an inprogress compaction and flush to the disk
part of the data.

The part that will be flushed is the 'compacted' part? Just trying to
understand better.

On name of the config., I think it should be IN_MEMORY_COMPACTION rather than
COMPACTED. We already have a compaction process. Add the IN_MEMORY prefix to
show where it applies. Also, this feature should be on by default. You might
want to say that in the doc as being your intent. Lets prove it makes life
better so we can indeed enable it by default.

Can the in-memory flush use same code as the flush-to-disk flush? Ditto on
compaction?

bq. . An additional flushtotalsize is defined as the average of flushsize
and blockingflushsize.

Pardon me, what is the above for?

bq. In case of FlushLargeStoresPolicy, the decision is based on the size of
the active segment.

I'm not clear on when a flush to disk will happen. I get you need headroom for
in-memory flush and compaction to run but can you be more clear on where the
threshold for flush to disk is? Thanks.

What is a snapshot in this scheme? It is just the last segment in the pipeline?
Each pipeline segment because an hfile? Or we have to do a merge sort on flush
to make the hfile?

{code}
Upon inmemory flush, we take advantage of the existing blocking mechanism
the active
segment is pushed to the pipeline while holding the region updatesLockin
exclusive mode.
Then, a compaction is applied at the background to all pipeline segments
resulting in a single
immutable segment. This procedure is executed at the scope of the store level,
and specifically
in the context of the compacted memstore.
{code}

Do we hold the region lock while we compact the in-memory segments on a column
family? Every time a compaction runs, it compacts all segments in the pipeline?

I'm not sure I follow the approximation of oldest sequence id. Why does it have
to to be an approximation? Can't we keep a running oldest sequenceid seen?
(Removing it too if compactions remove the cell that is oldest -- I suppose
this could complicate acccounting... how you find the next oldest). So, you
have a map of timestamp to oldest sequenceid in the segment? And when a segment
is flushed to disk, the next oldest becomes oldest edit in memory?

Do you have a rig where you can try out your implementation apart from running
it inside a regionserver?

Should be CompactingMemStore rather than CompactedMemStore.

The class diagram looks great.

So, on design, we talking about adding one more thread -- a compacting thread
-- per Store? Do we have to do this? HBase has too many threads already.
Minimally, these threads can be run by an executor so they don't live for ever
while the process is up? It could be trigger by an in-memory flush.

On compactionpipeline, add rather than push-in and remove rather than pull-out?

On MemstoreScanner, we are keeping the fact that the implementation is crossing
Segments an internal implementation detail? +1 on making the MemStoreScanner
first-class detached from DefaultMemStore.

MemStoreCompactor should not be first class, right? It should be internal to
the CompactingMemStore implementation?

Yeah, its fine if internally the kv sets go into a StoreSegment...but clients
don't have to know this, right?

How will you implement

[jira] [Commented] (HBASE-14753) TestShell is not invoked anymore


[ 
https://issues.apache.org/jira/browse/HBASE-14753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988475#comment-14988475
 ] 

stack commented on HBASE-14753:
---

I was thinking HBASE-14678 would do. I'd file issues for stuff I don't want to 
bring back in because flakey still...  tests on shell are pretty important.

> TestShell is not invoked anymore
> 
>
> Key: HBASE-14753
> URL: https://issues.apache.org/jira/browse/HBASE-14753
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
>
> Not sure whether it is due to HBASE-14679 or not. 
> {code}
> mvn clean  test -Dtest=TestShell
> {code}
> does not run any test at all. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14030) HBase Backup/Restore Phase 1


[ 
https://issues.apache.org/jira/browse/HBASE-14030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988645#comment-14988645
 ] 

Hadoop QA commented on HBASE-14030:
---

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12770410/HBASE-14030-v15.patch
  against master branch at commit 090fbd3ec862f8c85aa511172dd8e591b3b79332.
  ATTACHMENT ID: 12770410

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 23 new 
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 
2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16368//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16368//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16368//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16368//console

This message is automatically generated.

> HBase Backup/Restore Phase 1
> 
>
> Key: HBASE-14030
> URL: https://issues.apache.org/jira/browse/HBASE-14030
> Project: HBase
>  Issue Type: Umbrella
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-14030-v0.patch, HBASE-14030-v1.patch, 
> HBASE-14030-v10.patch, HBASE-14030-v11.patch, HBASE-14030-v12.patch, 
> HBASE-14030-v13.patch, HBASE-14030-v14.patch, HBASE-14030-v15.patch, 
> HBASE-14030-v2.patch, HBASE-14030-v3.patch, HBASE-14030-v4.patch, 
> HBASE-14030-v5.patch, HBASE-14030-v6.patch, HBASE-14030-v7.patch, 
> HBASE-14030-v8.patch
>
>
> This is the umbrella ticket for Backup/Restore Phase 1. See HBASE-7912 design 
> doc for the phase description.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14734) TestGenerateDelegationToken fails with BindAddress clash


[ 
https://issues.apache.org/jira/browse/HBASE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988681#comment-14988681
 ] 

Andrew Purtell commented on HBASE-14734:


In any event it's out of our hands I think. Either ApacheDS or Hadoop 
(MiniKDC). 

> TestGenerateDelegationToken fails with BindAddress clash
> 
>
> Key: HBASE-14734
> URL: https://issues.apache.org/jira/browse/HBASE-14734
> Project: HBase
>  Issue Type: Bug
>  Components: flakey, test
>Reporter: stack
>
> From 
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/330/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.security.token/TestGenerateDelegationToken/org_apache_hadoop_hbase_security_token_TestGenerateDelegationToken/
> Error Message
> Address already in use
> Stacktrace
> java.net.BindException: Address already in use
>   at sun.nio.ch.Net.bind0(Native Method)
>   at sun.nio.ch.Net.bind(Net.java:444)
>   at sun.nio.ch.Net.bind(Net.java:436)
>   at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
>   at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
>   at 
> org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198)
>   at 
> org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51)
>   at 
> org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547)
>   at 
> org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68)
>   at 
> org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422)
>   at 
> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Can this utility be made to not fail if address taken? Try another?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-7105) RS throws NPE on forcing compaction from HBase shell on a single bulk imported file.


 [ 
https://issues.apache.org/jira/browse/HBASE-7105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-7105:
--
 Assignee: (was: Cosmin Lehene)
Fix Version/s: (was: 1.1.4)
   (was: 1.0.3)
   (was: 1.2.1)

No movement. Unscheduling except from master and branch-1. 

> RS throws NPE on forcing compaction from HBase shell on a single bulk 
> imported file.
> 
>
> Key: HBASE-7105
> URL: https://issues.apache.org/jira/browse/HBASE-7105
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Reporter: Karthik Ranganathan
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 
> 0001-HBASE-7105-RS-throws-NPE-on-forcing-compaction-from-.patch, 
> 0001-HBASE-7105-RS-throws-NPE-on-forcing-compaction-from-.patch
>
>
> In StoreFile, we have:
> private AtomicBoolean majorCompaction = null;
> In StoreFile.open(), we do:
> b = metadataMap.get(MAJOR_COMPACTION_KEY);
> if (b != null) {
>   // init majorCompaction variable
> }
> Because the file was bulk imported, this is never initialized. Any subsequent 
> call to isMajorCompaction() NPE's.
> Fix is to initialize it to false.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-11850) RpcClient can get stuck when closing


 [ 
https://issues.apache.org/jira/browse/HBASE-11850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-11850:
---
 Assignee: (was: Nicolas Liochon)
Fix Version/s: (was: 1.1.4)
   (was: 1.0.3)
   (was: 1.2.1)

No movement. Unscheduling except from master and branch-1. 

> RpcClient can get stuck when closing
> 
>
> Key: HBASE-11850
> URL: https://issues.apache.org/jira/browse/HBASE-11850
> Project: HBase
>  Issue Type: Bug
>  Components: Client
>Affects Versions: 0.99.0, 2.0.0
>Reporter: Nicolas Liochon
> Fix For: 2.0.0, 1.3.0
>
>
> we don't stop until the map in 'connections' is empty.
> the new connection is put in this map at creation, but if this connection is 
> not used it will never be removed.
> No totally sure of the fix yet. Probability is low (but not zero. I saw it 
> happening).
> The code is different in 0.98. It's 0.99+ issue only.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-12273) Generate .tabledesc file during upgrading if missing


 [ 
https://issues.apache.org/jira/browse/HBASE-12273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell resolved HBASE-12273.

   Resolution: Incomplete
Fix Version/s: (was: 0.98.17)
   (was: 1.0.3)

This looks like contributor missing in action. Resolving as Incomplete. Reopen 
when we can make progress.

> Generate .tabledesc file during upgrading if missing
> 
>
> Key: HBASE-12273
> URL: https://issues.apache.org/jira/browse/HBASE-12273
> Project: HBase
>  Issue Type: Sub-task
>  Components: Admin
>Affects Versions: 1.0.0, 0.98.7
>Reporter: Yi Deng
>  Labels: upgrade
> Attachments: 
> 1.0-0001-HBASE-12273-Add-a-tool-for-fixing-missing-TableDescr.patch, 
> 1.0-0001-INTERNAL-Add-a-tool-for-fixing-missing-TableDescript.patch
>
>
> Generate .tabledesc file during upgrading if missing



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13452) HRegion warning about memstore size miscalculation is not actionable


 [ 
https://issues.apache.org/jira/browse/HBASE-13452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-13452:
---
 Assignee: (was: Mikhail Antonov)
Fix Version/s: (was: 1.1.4)
   (was: 1.0.3)
   (was: 1.2.1)
   1.3.0

No movement. Unscheduled except from master and branch-1.

> HRegion warning about memstore size miscalculation is not actionable
> 
>
> Key: HBASE-13452
> URL: https://issues.apache.org/jira/browse/HBASE-13452
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Dev Lakhani
>Priority: Critical
> Fix For: 2.0.0, 1.3.0
>
>
> During normal operation the HRegion class reports a message related to 
> memstore flushing in HRegion.class :
>   if (!canFlush) {
> addAndGetGlobalMemstoreSize(-memstoreSize.get());
>   } else if (memstoreSize.get() != 0) {
> LOG.error("Memstore size is " + memstoreSize.get());
>   }
> The log file is filled with lots of 
> Memstore size is 558744
> Memstore size is 4390632
> Memstore size is 558744 
> ...
> These message are uninformative, clog up the logs and offers no root cause 
> nor solution. Maybe the message needs to be more informative, changed to WARN 
> or some further information provided.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13667) Backport HBASE-12975 to 1.0 and 0.98 without changing coprocessors hooks


 [ 
https://issues.apache.org/jira/browse/HBASE-13667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-13667:
---
Fix Version/s: (was: 1.0.3)
   1.0.4

> Backport HBASE-12975 to 1.0 and 0.98 without changing coprocessors hooks
> 
>
> Key: HBASE-13667
> URL: https://issues.apache.org/jira/browse/HBASE-13667
> Project: HBase
>  Issue Type: Bug
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 0.98.17, 1.0.4
>
>
> We can backport Split transaction, region merge transaction interfaces to 
> branch 1.0 and 0.98 without changing coprocessor hooks. Then it should be 
> compatible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13667) Backport HBASE-12975 to 1.0 and 0.98 without changing coprocessors hooks


[ 
https://issues.apache.org/jira/browse/HBASE-13667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988750#comment-14988750
 ] 

Andrew Purtell commented on HBASE-13667:


Any plans to do this [~rajeshbabu] ? Or close this as Wont Fix? 

> Backport HBASE-12975 to 1.0 and 0.98 without changing coprocessors hooks
> 
>
> Key: HBASE-13667
> URL: https://issues.apache.org/jira/browse/HBASE-13667
> Project: HBase
>  Issue Type: Bug
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 0.98.17, 1.0.4
>
>
> We can backport Split transaction, region merge transaction interfaces to 
> branch 1.0 and 0.98 without changing coprocessor hooks. Then it should be 
> compatible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-14754) TestFastFailWithoutTestUtil failing on trunk now in #testPreemptiveFastFailException50Times


 [ 
https://issues.apache.org/jira/browse/HBASE-14754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-14754.
---
   Resolution: Fixed
 Assignee: stack
Fix Version/s: 1.3.0
   1.2.0
   2.0.0

I just git rm'd this test from branch-1.2+ Go to HBASE-14422 if interested in 
restoring/fixing.

> TestFastFailWithoutTestUtil failing on trunk now in 
> #testPreemptiveFastFailException50Times
> ---
>
> Key: HBASE-14754
> URL: https://issues.apache.org/jira/browse/HBASE-14754
> Project: HBase
>  Issue Type: Bug
>  Components: flakey, test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
>
> I'm just going to remove this suite. Its caused loads of pain over last few 
> months. See HBASE-14484 and HBASE-14421 with all their amendments.
> HBASE-14422 was filed a while ago to fix this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14715) Add javadocs to DelegatingRetryingCallable


[ 
https://issues.apache.org/jira/browse/HBASE-14715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988559#comment-14988559
 ] 

Hudson commented on HBASE-14715:


SUCCESS: Integrated in HBase-1.3-IT #290 (See 
[https://builds.apache.org/job/HBase-1.3-IT/290/])
HBASE-14715 Add javadocs to DelegatingRetryingCallable (apurtell: rev 
ed3b6bda4b8b6423e47c5dd6202d067e2af96bd5)
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RetryingCallable.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/DelegatingRetryingCallable.java


> Add javadocs to DelegatingRetryingCallable
> --
>
> Key: HBASE-14715
> URL: https://issues.apache.org/jira/browse/HBASE-14715
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Reporter: Jesse Yates
>Assignee: Jesse Yates
>Priority: Trivial
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: hbase-14715-trunk-v0.patch
>
>
> Follow up cleanup from HBASE-9049



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14715) Add javadocs to DelegatingRetryingCallable


[ 
https://issues.apache.org/jira/browse/HBASE-14715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988647#comment-14988647
 ] 

Hudson commented on HBASE-14715:


SUCCESS: Integrated in HBase-1.2-IT #261 (See 
[https://builds.apache.org/job/HBase-1.2-IT/261/])
HBASE-14715 Add javadocs to DelegatingRetryingCallable (apurtell: rev 
f3bd085d2ae53db898cfc4f62d9b502a2f51a154)
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RetryingCallable.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/DelegatingRetryingCallable.java


> Add javadocs to DelegatingRetryingCallable
> --
>
> Key: HBASE-14715
> URL: https://issues.apache.org/jira/browse/HBASE-14715
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Reporter: Jesse Yates
>Assignee: Jesse Yates
>Priority: Trivial
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: hbase-14715-trunk-v0.patch
>
>
> Follow up cleanup from HBASE-9049



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14696) Support setting allowPartialResults in mapreduce Mappers


[ 
https://issues.apache.org/jira/browse/HBASE-14696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988690#comment-14988690
 ] 

Andrew Purtell commented on HBASE-14696:


Why wasn't this committed to any releaseable branch? Doesn't do much good where 
it is right now. Picking it back. 

> Support setting allowPartialResults in mapreduce Mappers
> 
>
> Key: HBASE-14696
> URL: https://issues.apache.org/jira/browse/HBASE-14696
> Project: HBase
>  Issue Type: Improvement
>  Components: mapreduce
>Affects Versions: 2.0.0, 1.1.0
>Reporter: Mindaugas Kairys
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 14696-branch-1-v1.txt, 14696-branch-1-v2.txt, 
> 14696-branch-1-v2.txt, 14696-v1.txt, 14696-v2.txt
>
>
> It is currently impossible to get partial results in mapreduce mapper jobs.
> When setting setAllowPartialResults(true) for scan jobs, they still fail with 
> OOME on large rows.
> The reason is that Scan field allowPartialResults is lost during job creation:
>   1. User creates a Job and sets a scan object via 
> TableMapReduceUtil.initTableMapperJob(table_name, scanObj,...) -> which puts 
> a result of TableMapReduceUtil.convertScanToString(scanObj) to the job config.
>   2. When the job starts - method TableInputFormat.setConfig retrieves a scan 
> string from config and converts it to Scan object by calling 
> TableMapReduceUtil.convertStringToScan - which results in a Scan object with 
> a field allowPartialResults always set to false.
> I have tried to experiment and modify a TableInputFormat method setConfig() 
> by forcing all scans to allow partial results and after this all jobs 
> succeeded with no more OOME and I also noticed that mappers began to get 
> partial results (Result.isPartial()).
> My use case is very simple - I just have large rows and expect a mapper to 
> get them partially - to get same rowid several times with different key/value 
> records.
> This would allow me not to worry about implementing my own result 
> partitioning solution, which i would encounter in case the big amount of 
> result key values could be transparently returned for a single large row.
> And from the other side - if a Scan object can return several records for the 
> same rowid (partial results), perhaps the mapper should do the same.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14708) Use copy on write TreeMap for region location cache


 [ 
https://issues.apache.org/jira/browse/HBASE-14708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-14708:
--
Attachment: HBASE-14708-v9.patch

> Use copy on write TreeMap for region location cache
> ---
>
> Key: HBASE-14708
> URL: https://issues.apache.org/jira/browse/HBASE-14708
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Affects Versions: 1.1.2
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14708-v2.patch, HBASE-14708-v3.patch, 
> HBASE-14708-v4.patch, HBASE-14708-v5.patch, HBASE-14708-v6.patch, 
> HBASE-14708-v7.patch, HBASE-14708-v8.patch, HBASE-14708-v9.patch, 
> HBASE-14708.patch, anotherbench.zip, location_cache_times.pdf, result.csv
>
>
> Internally a co-worker profiled their application that was talking to HBase. 
> > 60% of the time was spent in locating a region. This was while the cluster 
> was stable and no regions were moving.
> To figure out if there was a faster way to cache region location I wrote up a 
> benchmark here: https://github.com/elliottneilclark/benchmark-hbase-cache
> This tries to simulate a heavy load on the location cache. 
> * 24 different threads.
> * 2 Deleting location data
> * 2 Adding location data
> * Using floor to get the result.
> To repeat my work just run ./run.sh and it should produce a result.csv
> Results:
> ConcurrentSkiplistMap is a good middle ground. It's got equal speed for 
> reading and writing.
> However most operations will not need to remove or add a region location. 
> There will be potentially several orders of magnitude more reads for cached 
> locations than there will be on clearing the cache.
> So I propose a copy on write tree map.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11544) [Ergonomics] hbase.client.scanner.caching is dogged and will try to return batch even if it means OOME