[jira] [Commented] (HBASE-14203) remove duplicate code getTableDescriptor in HTable

2015-08-12 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14693580#comment-14693580
 ] 

Hadoop QA commented on HBASE-14203:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12750059/HBASE-14203_v4.patch
  against master branch at commit aa3538f80278f5c0ba1bc8ca903066fa02ac79ec.
  ATTACHMENT ID: 12750059

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:red}-1 checkstyle{color}.  The applied patch generated 
1861 checkstyle errors (more than the master's current 1857 errors).

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
 

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15067//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15067//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15067//artifact/patchprocess/checkstyle-aggregate.html

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15067//console

This message is automatically generated.

 remove duplicate code getTableDescriptor in HTable
 --

 Key: HBASE-14203
 URL: https://issues.apache.org/jira/browse/HBASE-14203
 Project: HBase
  Issue Type: Improvement
Reporter: Heng Chen
Priority: Trivial
 Attachments: HBASE-14203.patch, HBASE-14203_v2.patch, 
 HBASE-14203_v3.patch, HBASE-14203_v4.patch


 As TODO in comment said, 
 {{HTable.getTableDescriptor}} is same as {{HAdmin.getTableDescriptor}}. 
 remove the duplicate code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14078) improve error message when HMaster can't bind to port

2015-08-12 Thread stack (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14693668#comment-14693668
 ] 

stack commented on HBASE-14078:
---

Formatting is a little off. Please check.

Do you have example of what the emission looks like now?

Whatever the IOE exception that comes up out of setting up the rpc server, we 
will always have this suffix about how to config port. Will it always be a port 
issue? Perhaps work on BindException only? And only if 'Address in use'?

The changes in HMaster.java do not seem to make for the same emission. Is that 
intentional? For example, before patch, if an Exception, we used to  
e.getCause().getMessage() if non-null but now if I read it right, we do 
e.toString



 improve error message when HMaster can't bind to port
 -

 Key: HBASE-14078
 URL: https://issues.apache.org/jira/browse/HBASE-14078
 Project: HBase
  Issue Type: Improvement
  Components: master
Affects Versions: 2.0.0
Reporter: Sean Busbey
Assignee: Matt Warhaftig
  Labels: beginner
 Fix For: 2.0.0

 Attachments: hbase-14708-v1.patch, hbase-14708-v2.patch


 When the master fails to start becahse hbase.master.port is already taken, 
 the log messages could make it easier to tell.
 {quote}
 2015-07-14 13:10:02,667 INFO  [main] regionserver.RSRpcServices: 
 master/master01.example.com/10.20.188.121:16000 server-side HConnection 
 retries=350
 2015-07-14 13:10:02,879 INFO  [main] ipc.SimpleRpcScheduler: Using deadline 
 as user call queue, count=3
 2015-07-14 13:10:02,895 ERROR [main] master.HMasterCommandLine: Master exiting
 java.lang.RuntimeException: Failed construction of Master: class 
 org.apache.hadoop.hbase.master.HMaster
 at 
 org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2258)
 at 
 org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:234)
 at 
 org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
 at 
 org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
 at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2272)
 Caused by: java.net.BindException: Address already in use
 at sun.nio.ch.Net.bind0(Native Method)
 at sun.nio.ch.Net.bind(Net.java:444)
 at sun.nio.ch.Net.bind(Net.java:436)
 at 
 sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
 at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
 at org.apache.hadoop.hbase.ipc.RpcServer.bind(RpcServer.java:2513)
 at 
 org.apache.hadoop.hbase.ipc.RpcServer$Listener.init(RpcServer.java:599)
 at org.apache.hadoop.hbase.ipc.RpcServer.init(RpcServer.java:2000)
 at 
 org.apache.hadoop.hbase.regionserver.RSRpcServices.init(RSRpcServices.java:919)
 at 
 org.apache.hadoop.hbase.master.MasterRpcServices.init(MasterRpcServices.java:211)
 at 
 org.apache.hadoop.hbase.master.HMaster.createRpcServices(HMaster.java:509)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.init(HRegionServer.java:535)
 at org.apache.hadoop.hbase.master.HMaster.init(HMaster.java:351)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
 Method)
 at 
 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
 at 
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
 at 
 org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2253)
 ... 5 more
 {quote}
 I recognize that the RSRpcServices log message shows port 16000, but I 
 don't know why a new operator would. Additionally, it'd be nice to tell them 
 that the port is controlled by {{hbase.master.port}}. Maybe give a hint on 
 how to see what's using the port. Could be too os-dist specific?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-13838) Fix shared TaskStatusTmpl.jamon issues (coloring, content, etc.)

2015-08-12 Thread stack (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-13838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14693628#comment-14693628
 ] 

stack commented on HBASE-13838:
---

Do you have a gif of what page looks like after your changes [~mwarhaftig] 
including what the JSON toString looks like? Thanks.

 Fix shared TaskStatusTmpl.jamon issues (coloring, content, etc.)
 

 Key: HBASE-13838
 URL: https://issues.apache.org/jira/browse/HBASE-13838
 Project: HBase
  Issue Type: Bug
  Components: UI
Affects Versions: 1.1.0
Reporter: Lars George
Assignee: Matt Warhaftig
  Labels: beginner
 Fix For: 2.0.0, 1.3.0

 Attachments: hbase-13838-v1.patch


 There are a few issues with the shared TaskStatusTmpl:
 - Client operations tab is always empty 
 For Master this is expected, but for RegionServers there is never anything 
 listed either. Fix for RS status page (probably caused by params not 
 containing Operation subclass anymore, but some PB generated classes?)
 - Hide “Client Operations” tab for master UI
 Since operations are RS only. Or we fix this and make other calls show here.
 - The alert-error for aborted tasks is not set in CSS at all
 When a task was aborted it should be amber or red, but the assigned style is 
 not in any of the linked stylesheets (abort-error). Add.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-13708) PE - Add option to specify the range

2015-08-12 Thread stack (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-13708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-13708:
--
Attachment: hbase-13708-v2.patch

Resubmit (You resubmit by reattaching [~mwarhaftig] -- just FYI).

I've always wanted to have this option but was always too lazy to implement it.

 PE - Add option to specify the range
 

 Key: HBASE-13708
 URL: https://issues.apache.org/jira/browse/HBASE-13708
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.0.1
Reporter: Jean-Marc Spaggiari
Assignee: Matt Warhaftig
Priority: Minor
  Labels: beginner
 Fix For: 2.0.0, 1.3.0

 Attachments: HBASE-13708-master_v1.patch, hbase-13708-v2.patch, 
 hbase-13708-v2.patch


 When specifying --rows=YYY for randomReads in PE, it will read YYY rows but 
 only betweem 0 and YYY.
 We might want to read YYY rows randomly between 0 and ZZZ.
 YYY should only be the number of rows we want to ready. Not the high range.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-13708) PE - Add option to specify the range

2015-08-12 Thread stack (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-13708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14693634#comment-14693634
 ] 

stack commented on HBASE-13708:
---

You have output to show this works [~mwarhaftig]? Thanks.

 PE - Add option to specify the range
 

 Key: HBASE-13708
 URL: https://issues.apache.org/jira/browse/HBASE-13708
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.0.1
Reporter: Jean-Marc Spaggiari
Assignee: Matt Warhaftig
Priority: Minor
  Labels: beginner
 Fix For: 2.0.0, 1.3.0

 Attachments: HBASE-13708-master_v1.patch, hbase-13708-v2.patch, 
 hbase-13708-v2.patch


 When specifying --rows=YYY for randomReads in PE, it will read YYY rows but 
 only betweem 0 and YYY.
 We might want to read YYY rows randomly between 0 and ZZZ.
 YYY should only be the number of rows we want to ready. Not the high range.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-13966) Limit column width in table.jsp

2015-08-12 Thread stack (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-13966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14693644#comment-14693644
 ] 

stack commented on HBASE-13966:
---

[~mwarhaftig] You have example of your fix in effect -- images? -- of how it 
addresses [~jmspaggi]'s issue? Thanks.

 Limit column width in table.jsp
 ---

 Key: HBASE-13966
 URL: https://issues.apache.org/jira/browse/HBASE-13966
 Project: HBase
  Issue Type: Bug
  Components: UI
Affects Versions: 1.1.0.1
Reporter: Jean-Marc Spaggiari
Assignee: Matt Warhaftig
Priority: Minor
  Labels: beginner
 Attachments: hbase-13966-v1.patch


 In table.jsp, for tables with very wide keys like URLs, the page can be very 
 wide. On my own cluster, this page is 8 screens wide, almost un-usable.
 Might be good to have a way to resize the columns, or wrap the long keys or 
 truncate them, or anything else. When we want to look at the last colums, 
 this is a big difficult.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-13839) Fix AssgnmentManagerTmpl.jamon issues (coloring, content etc.)

2015-08-12 Thread stack (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-13839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14693638#comment-14693638
 ] 

stack commented on HBASE-13839:
---

You have images that show these fixes in operation [~mwarhaftig]? Or else, I 
could try and apply this and your other UI fixes locally and check they do the 
right thing.

 Fix AssgnmentManagerTmpl.jamon issues (coloring, content etc.)
 --

 Key: HBASE-13839
 URL: https://issues.apache.org/jira/browse/HBASE-13839
 Project: HBase
  Issue Type: Bug
  Components: master, UI
Affects Versions: 1.1.0
Reporter: Lars George
Assignee: Matt Warhaftig
  Labels: beginner
 Fix For: 2.0.0, 1.3.0

 Attachments: hbase-13839-v1.patch


 The template for the RIT in the Master status page, 
 AssignmentManagerTmpl.jamon) has a few issues:
 - The oldest RIT should not be _red_, looks like a failed entry
 The RIT entries should be for example yellow/amber when over the threshold 
 time, and red if 2x the threshold - or red for the oldest once over the 
 threshold.
 - Region count over RIT threshold should only be colored if  0
 The summary line (first of two) should not be colored unless there is a value 
  0 in it.
 - Color is overriden by table-stripped CSS style!
 The Bootstrap stylesheet cancels out the hardcoded coloring! The 
 table-stripped resets the conditional coloring and should be fixed. Best is 
 to use alert-warning etc. that come from the Bootstrap theme stylesheet. 
 That should maybe already work in combination with the table-stripped from 
 the same.
 - Should sort descending by time
 Currently the list of regions is sorted by encoded region name. Better is to 
 have the table sorted by RIT time descending.
 We should also think about a pagination option for the currently hardcoded 
 100 entries max. Maybe a separate issue?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14147) REST Support for Namespaces

2015-08-12 Thread stack (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14693674#comment-14693674
 ] 

stack commented on HBASE-14147:
---

Looks like a nice fat addition. On skim, looks nicely done. Manipulating 
namespace from REST is a little on the exotic side though I'd say. What is the 
use case? Any REST heads want to check this out?



 REST Support for Namespaces
 ---

 Key: HBASE-14147
 URL: https://issues.apache.org/jira/browse/HBASE-14147
 Project: HBase
  Issue Type: Sub-task
  Components: REST
Affects Versions: 1.1.1
Reporter: Rick Kellogg
Assignee: Matt Warhaftig
Priority: Minor
 Attachments: hbase-14147-v1.patch


 Expand REST services to include addition features:
 * Create namespace
 * Alter namespace
 * Describe namespace
 * Drop namespace
 * List tables in a specific namespace
 * List all namespaces.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-14150) Add BulkLoad functionality to HBase-Spark Module

2015-08-12 Thread Ted Yu (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-14150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14150:
---
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Thanks for the patch, Ted.

Thanks for the review, Andrew.

 Add BulkLoad functionality to HBase-Spark Module
 

 Key: HBASE-14150
 URL: https://issues.apache.org/jira/browse/HBASE-14150
 Project: HBase
  Issue Type: New Feature
  Components: spark
Reporter: Ted Malaska
Assignee: Ted Malaska
 Fix For: 2.0.0

 Attachments: HBASE-14150.1.patch, HBASE-14150.2.patch, 
 HBASE-14150.3.patch, HBASE-14150.4.patch, HBASE-14150.5.patch


 Add on to the work done in HBASE-13992 to add functionality to do a bulk load 
 from a given RDD.
 This will do the following:
 1. figure out the number of regions and sort and partition the data correctly 
 to be written out to HFiles
 2. Also unlike the MR bulkload I would like that the columns to be sorted in 
 the shuffle stage and not in the memory of the reducer.  This will allow this 
 design to support super wide records with out going out of memory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14150) Add BulkLoad functionality to HBase-Spark Module

2015-08-12 Thread Ted Malaska (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14693714#comment-14693714
 ] 

Ted Malaska commented on HBASE-14150:
-

Thanks guys,

Super excited.  Now one more big one 14181 and it will be much smaller jira and 
documentation for awhile.

 Add BulkLoad functionality to HBase-Spark Module
 

 Key: HBASE-14150
 URL: https://issues.apache.org/jira/browse/HBASE-14150
 Project: HBase
  Issue Type: New Feature
  Components: spark
Reporter: Ted Malaska
Assignee: Ted Malaska
 Fix For: 2.0.0

 Attachments: HBASE-14150.1.patch, HBASE-14150.2.patch, 
 HBASE-14150.3.patch, HBASE-14150.4.patch, HBASE-14150.5.patch


 Add on to the work done in HBASE-13992 to add functionality to do a bulk load 
 from a given RDD.
 This will do the following:
 1. figure out the number of regions and sort and partition the data correctly 
 to be written out to HFiles
 2. Also unlike the MR bulkload I would like that the columns to be sorted in 
 the shuffle stage and not in the memory of the reducer.  This will allow this 
 design to support super wide records with out going out of memory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14201) hbck should not take a lock unless fixing errors

2015-08-12 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694225#comment-14694225
 ] 

Hudson commented on HBASE-14201:


FAILURE: Integrated in HBase-TRUNK #6719 (See 
[https://builds.apache.org/job/HBase-TRUNK/6719/])
HBASE-14201 hbck should not take a lock unless fixing errors (eclark: rev 
0c63d415d9d23a80ec3b8f454bbf4748de4a1a78)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java


 hbck should not take a lock unless fixing errors
 

 Key: HBASE-14201
 URL: https://issues.apache.org/jira/browse/HBASE-14201
 Project: HBase
  Issue Type: Bug
  Components: hbck, util
Affects Versions: 2.0.0, 1.3.0
Reporter: Simon Law
Assignee: Simon Law
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-14201-v0.patch, HBASE-14201-v1.patch


 By default, hbck is run in a read-only checker mode. In this case, it is
 sensible to let others run. By default, the balancer is left alone,
 which may cause spurious errors, but cannot leave the balancer in a bad
 state. It is dangerous to leave the balancer by accident, so it is only
 ever enabled after fixing, it will never be forced off because of
 racing.
 When hbck is run in fixer mode, it must take an exclusive lock and
 disable the balancer, or all havoc will break loose.
 If you want to stop hbck from running in parallel, the -exclusive flag
 will create the lock file. If you want to force -disableBalancer, that
 option is available too. This makes more semantic sense than -noLock and
 -noSwitchBalancer, respectively.
 This task is related to HBASE-14092.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14209) TestShell visibility tests failing

2015-08-12 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694323#comment-14694323
 ] 

Hudson commented on HBASE-14209:


FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1030 (See 
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1030/])
HBASE-14209 TestShell visibility tests failing (apurtell: rev 
cddb28ef9fae67087280b99c4323945e43fba000)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityLabelsCache.java


 TestShell visibility tests failing
 --

 Key: HBASE-14209
 URL: https://issues.apache.org/jira/browse/HBASE-14209
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.98.14
Reporter: Andrew Purtell
Assignee: Andrew Purtell
 Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2, 1.3.0

 Attachments: HBASE-14209-0.98.patch


 This is after HBASE-14105  but we've seen this earlier where adding ruby 
 units to the shell tests can cause the visibility tests to fail inexplicably. 
 We can't just avoid adding ruby units to TestShell in 0.98 so figure out the 
 root cause and fix or disable these. 
 {noformat}
   1) Error:
 test_The_get/put_methods_should_work_for_data_written_with_Visibility(Hbase::VisibilityLabelsAdminMethodsTest):
 NativeException: junit.framework.AssertionFailedError: Waiting timed out 
 after [10,000] msec
 junit/framework/Assert.java:57:in `fail'
 org/apache/hadoop/hbase/Waiter.java:193:in `waitFor'
 org/apache/hadoop/hbase/Waiter.java:128:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3514:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3576:in 
 `waitLabelAvailable'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:73:in 
 `test_The_get/put_methods_should_work_for_data_written_with_Visibility'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
   2) Error:
 test_The_set/clear_methods_should_work_with_authorizations(Hbase::VisibilityLabelsAdminMethodsTest):
 NativeException: junit.framework.AssertionFailedError: Waiting timed out 
 after [10,000] msec
 junit/framework/Assert.java:57:in `fail'
 org/apache/hadoop/hbase/Waiter.java:193:in `waitFor'
 org/apache/hadoop/hbase/Waiter.java:128:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3514:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3576:in 
 `waitLabelAvailable'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:52:in 
 `test_The_set/clear_methods_should_work_with_authorizations'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-12912) StoreScanner calls Configuration for Boolean Check on each initialization

2015-08-12 Thread Andrew Purtell (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-12912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-12912:
---
Fix Version/s: (was: 1.1.3)
   (was: 1.0.3)
   (was: 1.2.1)
   (was: 0.98.14)
   0.98.15

 StoreScanner calls Configuration for Boolean Check on each initialization
 -

 Key: HBASE-12912
 URL: https://issues.apache.org/jira/browse/HBASE-12912
 Project: HBase
  Issue Type: Bug
Reporter: John Leach
Assignee: John Leach
 Fix For: 2.0.0, 1.3.0, 0.98.15

 Attachments: StoreScannerStall.tiff

   Original Estimate: 1h
  Remaining Estimate: 1h

 There is a clear CPU drain and iterator creation when creating store scanners 
 under high load.  Splice was running a TPCC test of our database and we are 
 seeing object creation and CPU waste on the boolean check
 Code Snippet...
 if (store != null  ((HStore)store).getHRegion() != null
  store.getStorefilesCount()  1) {
   RegionServerServices rsService = 
 ((HStore)store).getHRegion().getRegionServerServices();
   if (rsService == null || !rsService.getConfiguration().getBoolean(
 STORESCANNER_PARALLEL_SEEK_ENABLE, false)) return;
   isParallelSeekEnabled = true;
   executor = rsService.getExecutorService();
 }
 Will attach profile...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-13744) TestCorruptedRegionStoreFile is flaky

2015-08-12 Thread Andrew Purtell (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-13744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-13744:
---
Fix Version/s: (was: 0.98.14)
   0.98.15

 TestCorruptedRegionStoreFile is flaky
 -

 Key: HBASE-13744
 URL: https://issues.apache.org/jira/browse/HBASE-13744
 Project: HBase
  Issue Type: Bug
Reporter: Andrew Purtell
 Fix For: 0.98.15


 TestCorruptedRegionStoreFile#testLosingFileAfterScannerInit is failing on 
 recent Jenkins 0.98 builds and I can reproduce it with a few runs locally, 
 though not every time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-13143) TestCacheOnWrite is flaky and needs a diet

2015-08-12 Thread Andrew Purtell (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-13143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-13143:
---
Fix Version/s: (was: 1.1.3)
   (was: 1.0.3)
   (was: 1.2.1)
   (was: 0.98.14)
   0.98.15

 TestCacheOnWrite is flaky and needs a diet
 --

 Key: HBASE-13143
 URL: https://issues.apache.org/jira/browse/HBASE-13143
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.98.11
Reporter: Andrew Purtell
Assignee: Esteban Gutierrez
Priority: Critical
 Fix For: 2.0.0, 1.3.0, 0.98.15


 TestCacheOnWrite passes locally but has been flaking in 0.98 builds on 
 Jenkins, most recently https://builds.apache.org/job/HBase-0.98/878/
 The test takes a long time to execute (338.492 sec) and is resource intensive 
 (216 tests). Neither of these characteristics endear it to Jenkins.
 When I ran this unit test on a macbook after a minute the fan was running so 
 fast I thought it would take flight. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14122) Client API for determining if server side supports cell level security

2015-08-12 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694449#comment-14694449
 ] 

Hudson commented on HBASE-14122:


FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1031 (See 
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1031/])
HBASE-14122 Client API for determining if server side supports cell level 
security (apurtell: rev 87729ccfacd8684fab141d34ab4776ae7d5c2d79)
* hbase-shell/src/main/ruby/hbase/admin.rb
* 
hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/MasterProtos.java
* hbase-shell/src/main/ruby/shell.rb
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityClient.java
* hbase-shell/src/main/ruby/shell/commands/list_security_capabilities.rb
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabels.java
* hbase-protocol/src/main/protobuf/Master.proto
* hbase-shell/src/main/ruby/hbase/security.rb
* hbase-shell/src/main/ruby/hbase/visibility_labels.rb
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlClient.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityController.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/security/SecurityCapability.java


 Client API for determining if server side supports cell level security
 --

 Key: HBASE-14122
 URL: https://issues.apache.org/jira/browse/HBASE-14122
 Project: HBase
  Issue Type: Improvement
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
 Fix For: 2.0.0, 0.98.14, 1.2.0, 1.3.0

 Attachments: HBASE-14122-0.98.patch, HBASE-14122-0.98.patch, 
 HBASE-14122-branch-1.patch, HBASE-14122-branch-1.patch, HBASE-14122.patch, 
 HBASE-14122.patch, HBASE-14122.patch


 Add a client API for determining if the server side supports cell level 
 security. 
 Ask the master, assuming as we do in many other instances that the master and 
 regionservers all have a consistent view of site configuration.
 Return {{true}} if all features required for cell level security are present, 
 {{false}} otherwise, or throw {{UnsupportedOperationException}} if the master 
 does not have support for the RPC call.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14218) CompareFilter based on part of the key

2015-08-12 Thread Jean-Marc Spaggiari (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694194#comment-14694194
 ] 

Jean-Marc Spaggiari commented on HBASE-14218:
-

I agree, say purpose. Let me close this one.

 CompareFilter based on part of the key
 --

 Key: HBASE-14218
 URL: https://issues.apache.org/jira/browse/HBASE-14218
 Project: HBase
  Issue Type: Bug
  Components: Filters
Affects Versions: 1.1.0.1
Reporter: Jean-Marc Spaggiari
Priority: Minor

 HBase needs to provide a combination of FuzzyFilte rand CompareFilter to be 
 able to compare based on part of the key.
 Something like:
 xxxxx
 And we want to filter all rows where  is greater than a certain value.
 Fuzzy filter can do the test on  but only on exact match
 CompareFilter can to GREATER but on the entire key...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Resolved] (HBASE-14218) CompareFilter based on part of the key

2015-08-12 Thread Jean-Marc Spaggiari (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-14218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Marc Spaggiari resolved HBASE-14218.
-
  Resolution: Duplicate
Release Note: Duplicate of HBASE-6618

 CompareFilter based on part of the key
 --

 Key: HBASE-14218
 URL: https://issues.apache.org/jira/browse/HBASE-14218
 Project: HBase
  Issue Type: Bug
  Components: Filters
Affects Versions: 1.1.0.1
Reporter: Jean-Marc Spaggiari
Priority: Minor

 HBase needs to provide a combination of FuzzyFilte rand CompareFilter to be 
 able to compare based on part of the key.
 Something like:
 xxxxx
 And we want to filter all rows where  is greater than a certain value.
 Fuzzy filter can do the test on  but only on exact match
 CompareFilter can to GREATER but on the entire key...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Comment Edited] (HBASE-14218) CompareFilter based on part of the key

2015-08-12 Thread Jean-Marc Spaggiari (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694194#comment-14694194
 ] 

Jean-Marc Spaggiari edited comment on HBASE-14218 at 8/12/15 9:23 PM:
--

I agree, same purpose. Let me close this one.


was (Author: jmspaggi):
I agree, say purpose. Let me close this one.

 CompareFilter based on part of the key
 --

 Key: HBASE-14218
 URL: https://issues.apache.org/jira/browse/HBASE-14218
 Project: HBase
  Issue Type: Bug
  Components: Filters
Affects Versions: 1.1.0.1
Reporter: Jean-Marc Spaggiari
Priority: Minor

 HBase needs to provide a combination of FuzzyFilte rand CompareFilter to be 
 able to compare based on part of the key.
 Something like:
 xxxxx
 And we want to filter all rows where  is greater than a certain value.
 Fuzzy filter can do the test on  but only on exact match
 CompareFilter can to GREATER but on the entire key...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-14203) remove duplicate code getTableDescriptor in HTable

2015-08-12 Thread Heng Chen (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-14203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Heng Chen updated HBASE-14203:
--
Attachment: HBASE-14203_v4.patch

retry

 remove duplicate code getTableDescriptor in HTable
 --

 Key: HBASE-14203
 URL: https://issues.apache.org/jira/browse/HBASE-14203
 Project: HBase
  Issue Type: Improvement
Reporter: Heng Chen
Priority: Trivial
 Attachments: HBASE-14203.patch, HBASE-14203_v2.patch, 
 HBASE-14203_v3.patch, HBASE-14203_v4.patch


 As TODO in comment said, 
 {{HTable.getTableDescriptor}} is same as {{HAdmin.getTableDescriptor}}. 
 remove the duplicate code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14209) TestShell visibility tests failing

2015-08-12 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694313#comment-14694313
 ] 

Hudson commented on HBASE-14209:


SUCCESS: Integrated in HBase-1.1 #610 (See 
[https://builds.apache.org/job/HBase-1.1/610/])
HBASE-14209 TestShell visibility tests failing (apurtell: rev 
aa646053998a718d13ddaa3d53cac21bf2559951)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityLabelsCache.java


 TestShell visibility tests failing
 --

 Key: HBASE-14209
 URL: https://issues.apache.org/jira/browse/HBASE-14209
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.98.14
Reporter: Andrew Purtell
Assignee: Andrew Purtell
 Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2, 1.3.0

 Attachments: HBASE-14209-0.98.patch


 This is after HBASE-14105  but we've seen this earlier where adding ruby 
 units to the shell tests can cause the visibility tests to fail inexplicably. 
 We can't just avoid adding ruby units to TestShell in 0.98 so figure out the 
 root cause and fix or disable these. 
 {noformat}
   1) Error:
 test_The_get/put_methods_should_work_for_data_written_with_Visibility(Hbase::VisibilityLabelsAdminMethodsTest):
 NativeException: junit.framework.AssertionFailedError: Waiting timed out 
 after [10,000] msec
 junit/framework/Assert.java:57:in `fail'
 org/apache/hadoop/hbase/Waiter.java:193:in `waitFor'
 org/apache/hadoop/hbase/Waiter.java:128:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3514:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3576:in 
 `waitLabelAvailable'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:73:in 
 `test_The_get/put_methods_should_work_for_data_written_with_Visibility'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
   2) Error:
 test_The_set/clear_methods_should_work_with_authorizations(Hbase::VisibilityLabelsAdminMethodsTest):
 NativeException: junit.framework.AssertionFailedError: Waiting timed out 
 after [10,000] msec
 junit/framework/Assert.java:57:in `fail'
 org/apache/hadoop/hbase/Waiter.java:193:in `waitFor'
 org/apache/hadoop/hbase/Waiter.java:128:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3514:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3576:in 
 `waitLabelAvailable'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:52:in 
 `test_The_set/clear_methods_should_work_with_authorizations'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-14098) Allow dropping caches behind compactions

2015-08-12 Thread Elliott Clark (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-14098:
--
Attachment: (was: HBASE-14098-v7-branch1.patch)

 Allow dropping caches behind compactions
 

 Key: HBASE-14098
 URL: https://issues.apache.org/jira/browse/HBASE-14098
 Project: HBase
  Issue Type: Bug
  Components: Compaction, hadoop2, HFile
Affects Versions: 2.0.0, 1.3.0
Reporter: Elliott Clark
Assignee: Elliott Clark
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-14098-v1.patch, HBASE-14098-v2.patch, 
 HBASE-14098-v3.patch, HBASE-14098-v4.patch, HBASE-14098-v5.patch, 
 HBASE-14098-v6.patch, HBASE-14098-v7-branch-1.patch, HBASE-14098-v7.patch, 
 HBASE-14098.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-14098) Allow dropping caches behind compactions

2015-08-12 Thread Elliott Clark (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-14098:
--
Attachment: (was: HBASE-14098-branch1-v2.patch)

 Allow dropping caches behind compactions
 

 Key: HBASE-14098
 URL: https://issues.apache.org/jira/browse/HBASE-14098
 Project: HBase
  Issue Type: Bug
  Components: Compaction, hadoop2, HFile
Affects Versions: 2.0.0, 1.3.0
Reporter: Elliott Clark
Assignee: Elliott Clark
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-14098-v1.patch, HBASE-14098-v2.patch, 
 HBASE-14098-v3.patch, HBASE-14098-v4.patch, HBASE-14098-v5.patch, 
 HBASE-14098-v6.patch, HBASE-14098-v7-branch-1.patch, 
 HBASE-14098-v7-branch1.patch, HBASE-14098-v7.patch, HBASE-14098.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14201) hbck should not take a lock unless fixing errors

2015-08-12 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694334#comment-14694334
 ] 

Hudson commented on HBASE-14201:


FAILURE: Integrated in HBase-1.2 #103 (See 
[https://builds.apache.org/job/HBase-1.2/103/])
HBASE-14201 hbck should not take a lock unless fixing errors (eclark: rev 
25bf721b517b7740821dbffc5f7c7b5c539e9aca)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java


 hbck should not take a lock unless fixing errors
 

 Key: HBASE-14201
 URL: https://issues.apache.org/jira/browse/HBASE-14201
 Project: HBase
  Issue Type: Bug
  Components: hbck, util
Affects Versions: 2.0.0, 1.3.0
Reporter: Simon Law
Assignee: Simon Law
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-14201-v0.patch, HBASE-14201-v1.patch


 By default, hbck is run in a read-only checker mode. In this case, it is
 sensible to let others run. By default, the balancer is left alone,
 which may cause spurious errors, but cannot leave the balancer in a bad
 state. It is dangerous to leave the balancer by accident, so it is only
 ever enabled after fixing, it will never be forced off because of
 racing.
 When hbck is run in fixer mode, it must take an exclusive lock and
 disable the balancer, or all havoc will break loose.
 If you want to stop hbck from running in parallel, the -exclusive flag
 will create the lock file. If you want to force -disableBalancer, that
 option is available too. This makes more semantic sense than -noLock and
 -noSwitchBalancer, respectively.
 This task is related to HBASE-14092.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-14098) Allow dropping caches behind compactions

2015-08-12 Thread Elliott Clark (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-14098:
--
Attachment: HBASE-14098-v7-branch1.patch

 Allow dropping caches behind compactions
 

 Key: HBASE-14098
 URL: https://issues.apache.org/jira/browse/HBASE-14098
 Project: HBase
  Issue Type: Bug
  Components: Compaction, hadoop2, HFile
Affects Versions: 2.0.0, 1.3.0
Reporter: Elliott Clark
Assignee: Elliott Clark
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-14098-v1.patch, HBASE-14098-v2.patch, 
 HBASE-14098-v3.patch, HBASE-14098-v4.patch, HBASE-14098-v5.patch, 
 HBASE-14098-v6.patch, HBASE-14098-v7-branch-1.patch, 
 HBASE-14098-v7-branch1.patch, HBASE-14098-v7.patch, HBASE-14098.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-13857) Slow WAL Append count in ServerMetricsTmpl.jamon is hardcoded to zero

2015-08-12 Thread Andrew Purtell (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-13857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-13857:
---
Fix Version/s: (was: 1.1.3)
   (was: 1.0.3)
   (was: 1.2.1)
   (was: 0.98.14)
   0.98.15

 Slow WAL Append count in ServerMetricsTmpl.jamon is hardcoded to zero
 -

 Key: HBASE-13857
 URL: https://issues.apache.org/jira/browse/HBASE-13857
 Project: HBase
  Issue Type: Bug
  Components: regionserver, UI
Affects Versions: 0.98.0
Reporter: Lars George
  Labels: beginner
 Fix For: 2.0.0, 1.3.0, 0.98.15


 The template has this:
 {noformat}
  tr
 ...
 thSlow WAL Append Count/th
 /tr
 tr
 
 td% 0 %/td
 /tr
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-12890) Provide a way to throttle the number of regions moved by the balancer

2015-08-12 Thread Andrew Purtell (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694404#comment-14694404
 ] 

Andrew Purtell commented on HBASE-12890:


Ok, going to commit shortly.

 Provide a way to throttle the number of regions moved by the balancer
 -

 Key: HBASE-12890
 URL: https://issues.apache.org/jira/browse/HBASE-12890
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.98.10
Reporter: churro morales
Assignee: churro morales
 Fix For: 2.0.0, 0.98.14, 1.3.0

 Attachments: HBASE-12890.patch


 We have a very large cluster and we frequently add remove quite a few 
 regionservers from our cluster.  Whenever we do this the balancer moves 
 thousands of regions at once.  Instead we provide a configuration parameter: 
 hbase.balancer.max.regions.  This limits the number of regions that are 
 balanced per iteration.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-14209) TestShell visibility tests failing

2015-08-12 Thread Andrew Purtell (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-14209:
---
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Pushed to 0.98 and up.
I ran all visibility unit tests once and TestShell for 10 iterations on all 
branches before committing.

 TestShell visibility tests failing
 --

 Key: HBASE-14209
 URL: https://issues.apache.org/jira/browse/HBASE-14209
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.98.14
Reporter: Andrew Purtell
Assignee: Andrew Purtell
 Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2, 1.3.0

 Attachments: HBASE-14209-0.98.patch


 This is after HBASE-14105  but we've seen this earlier where adding ruby 
 units to the shell tests can cause the visibility tests to fail inexplicably. 
 We can't just avoid adding ruby units to TestShell in 0.98 so figure out the 
 root cause and fix or disable these. 
 {noformat}
   1) Error:
 test_The_get/put_methods_should_work_for_data_written_with_Visibility(Hbase::VisibilityLabelsAdminMethodsTest):
 NativeException: junit.framework.AssertionFailedError: Waiting timed out 
 after [10,000] msec
 junit/framework/Assert.java:57:in `fail'
 org/apache/hadoop/hbase/Waiter.java:193:in `waitFor'
 org/apache/hadoop/hbase/Waiter.java:128:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3514:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3576:in 
 `waitLabelAvailable'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:73:in 
 `test_The_get/put_methods_should_work_for_data_written_with_Visibility'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
   2) Error:
 test_The_set/clear_methods_should_work_with_authorizations(Hbase::VisibilityLabelsAdminMethodsTest):
 NativeException: junit.framework.AssertionFailedError: Waiting timed out 
 after [10,000] msec
 junit/framework/Assert.java:57:in `fail'
 org/apache/hadoop/hbase/Waiter.java:193:in `waitFor'
 org/apache/hadoop/hbase/Waiter.java:128:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3514:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3576:in 
 `waitLabelAvailable'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:52:in 
 `test_The_set/clear_methods_should_work_with_authorizations'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-14098) Allow dropping caches behind compactions

2015-08-12 Thread Elliott Clark (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-14098:
--
Attachment: HBASE-14098-branch1-v2.patch

The patch didn't apply cleanly to branch-1. Here's the backport.

 Allow dropping caches behind compactions
 

 Key: HBASE-14098
 URL: https://issues.apache.org/jira/browse/HBASE-14098
 Project: HBase
  Issue Type: Bug
  Components: Compaction, hadoop2, HFile
Affects Versions: 2.0.0, 1.3.0
Reporter: Elliott Clark
Assignee: Elliott Clark
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-14098-branch1-v2.patch, HBASE-14098-v1.patch, 
 HBASE-14098-v2.patch, HBASE-14098-v3.patch, HBASE-14098-v4.patch, 
 HBASE-14098-v5.patch, HBASE-14098-v6.patch, HBASE-14098-v7.patch, 
 HBASE-14098.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14201) hbck should not take a lock unless fixing errors

2015-08-12 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694259#comment-14694259
 ] 

Hudson commented on HBASE-14201:


FAILURE: Integrated in HBase-1.3 #103 (See 
[https://builds.apache.org/job/HBase-1.3/103/])
HBASE-14201 hbck should not take a lock unless fixing errors (eclark: rev 
881d2c1d74419013c10a37789f6203ed0bd05dd5)
* hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java


 hbck should not take a lock unless fixing errors
 

 Key: HBASE-14201
 URL: https://issues.apache.org/jira/browse/HBASE-14201
 Project: HBase
  Issue Type: Bug
  Components: hbck, util
Affects Versions: 2.0.0, 1.3.0
Reporter: Simon Law
Assignee: Simon Law
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-14201-v0.patch, HBASE-14201-v1.patch


 By default, hbck is run in a read-only checker mode. In this case, it is
 sensible to let others run. By default, the balancer is left alone,
 which may cause spurious errors, but cannot leave the balancer in a bad
 state. It is dangerous to leave the balancer by accident, so it is only
 ever enabled after fixing, it will never be forced off because of
 racing.
 When hbck is run in fixer mode, it must take an exclusive lock and
 disable the balancer, or all havoc will break loose.
 If you want to stop hbck from running in parallel, the -exclusive flag
 will create the lock file. If you want to force -disableBalancer, that
 option is available too. This makes more semantic sense than -noLock and
 -noSwitchBalancer, respectively.
 This task is related to HBASE-14092.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-14122) Client API for determining if server side supports cell level security

2015-08-12 Thread Andrew Purtell (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-14122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-14122:
---
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Pushed to 0.98, branch-1, branch-1.2, and master. New unit tests, security unit 
tests, and TestShell pass on all. 

 Client API for determining if server side supports cell level security
 --

 Key: HBASE-14122
 URL: https://issues.apache.org/jira/browse/HBASE-14122
 Project: HBase
  Issue Type: Improvement
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
 Fix For: 2.0.0, 0.98.14, 1.2.0, 1.3.0

 Attachments: HBASE-14122-0.98.patch, HBASE-14122-0.98.patch, 
 HBASE-14122-branch-1.patch, HBASE-14122-branch-1.patch, HBASE-14122.patch, 
 HBASE-14122.patch, HBASE-14122.patch


 Add a client API for determining if the server side supports cell level 
 security. 
 Ask the master, assuming as we do in many other instances that the master and 
 regionservers all have a consistent view of site configuration.
 Return {{true}} if all features required for cell level security are present, 
 {{false}} otherwise, or throw {{UnsupportedOperationException}} if the master 
 does not have support for the RPC call.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-13730) Backport HBASE-13576 (Failure in checking one region should not fail the entire HBCK operation) to 0.98

2015-08-12 Thread Andrew Purtell (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-13730:
---
Attachment: HBASE-13730-0.98.patch

 Backport HBASE-13576 (Failure in checking one region should not fail the 
 entire HBCK operation) to 0.98
 ---

 Key: HBASE-13730
 URL: https://issues.apache.org/jira/browse/HBASE-13730
 Project: HBase
  Issue Type: Sub-task
  Components: hbck
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
 Fix For: 0.98.14

 Attachments: HBASE-13730-0.98.patch


 Backport HBASE-13576 - Failure in checking one region should not fail the 
 entire HBCK operation - to 0.98



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-13730) Backport HBASE-13576 (Failure in checking one region should not fail the entire HBCK operation) to 0.98

2015-08-12 Thread Andrew Purtell (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-13730:
---
Status: Patch Available  (was: Open)

Ready to go. Straightforward backport. Anyone want to look before commit?

/cc [~syuanjiang] [~lhofhansl]

 Backport HBASE-13576 (Failure in checking one region should not fail the 
 entire HBCK operation) to 0.98
 ---

 Key: HBASE-13730
 URL: https://issues.apache.org/jira/browse/HBASE-13730
 Project: HBase
  Issue Type: Sub-task
  Components: hbck
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
 Fix For: 0.98.14

 Attachments: HBASE-13730-0.98.patch


 Backport HBASE-13576 - Failure in checking one region should not fail the 
 entire HBCK operation - to 0.98



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-14129) If any regionserver gets shutdown uncleanly during full cluster restart, locality looks to be lost

2015-08-12 Thread Andrew Purtell (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-14129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-14129:
---
Fix Version/s: (was: 0.98.14)
   0.98.15
   1.3.0
   Status: Open  (was: Patch Available)

{code}
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
index f7f98fe..1c3ceee 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
@@ -539,6 +539,12 @@ public class AssignmentManager {
   LOG.info(Clean cluster startup. Assigning user regions);
   assignAllUserRegions(allRegions);
 }
+
+if (this.server.getConfiguration().getBoolean(hbase.full.cluster.start, 
false)) {
+  // Hint to do a full cluster startup cluster startup.
+  LOG.info(Clean cluster startup forced via parameterized startup. 
Assigning user regions);
+  assignAllUserRegions(allRegions);
+}
 // unassign replicas of the split parents and the merged regions
 // the daughter replicas are opened in assignAllUserRegions if it was
 // not already opened.
{code}

Can someone who knows the AM better take a quick peek if this is sufficient?

 If any regionserver gets shutdown uncleanly during full cluster restart, 
 locality looks to be lost
 --

 Key: HBASE-14129
 URL: https://issues.apache.org/jira/browse/HBASE-14129
 Project: HBase
  Issue Type: Bug
Reporter: churro morales
 Fix For: 2.0.0, 1.3.0, 0.98.15

 Attachments: HBASE-14129.patch


 We were doing a cluster restart the other day.  Some regionservers did not 
 shut down cleanly.  Upon restart our locality went from 99% to 5%.  Upon 
 looking at the AssignmentManager.joinCluster() code it calls 
 AssignmentManager.processDeadServersAndRegionsInTransition().
 If the failover flag gets set for any reason it seems we don't call 
 assignAllUserRegions().  Then it looks like the balancer does the work in 
 assigning those regions, we don't use a locality aware balancer and we lost 
 our region locality.
 I don't have a solid grasp on the reasoning for these checks but there could 
 be some potential workarounds here.
 1. After shutting down your cluster, move your WALs aside (replay later).  
 2. Clean up your zNodes 
 That seems to work, but requires a lot of manual labor.  Another solution 
 which I prefer would be to have a flag for ./start-hbase.sh --clean 
 If we start master with that flag then we do a check in 
 AssignmentManager.processDeadServersAndRegionsInTransition()  thus if this 
 flag is set we call: assignAllUserRegions() regardless of the failover state.
 I have a patch for the later solution, that is if I am understanding the 
 logic correctly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14196) Thrift server idle connection timeout issue

2015-08-12 Thread Andrew Purtell (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694392#comment-14694392
 ] 

Andrew Purtell commented on HBASE-14196:


+1, changes look reasonable, going to commit shortly


 Thrift server idle connection timeout issue
 ---

 Key: HBASE-14196
 URL: https://issues.apache.org/jira/browse/HBASE-14196
 Project: HBase
  Issue Type: Bug
  Components: Thrift
Affects Versions: 0.98.13, 1.1.1, 1.0.1.1, 1.1.0.1
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov
 Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.3.0, 1.1.3

 Attachments: HBASE-14196.patch


 When idle connection get cleaned from Thrift server, underlying table 
 instances are still cached in a thread local cache.
 This is the antipattern. Table objects are lightweight and should not be 
 cached, besides this, underlying connections can be closed by periodic 
 connection cleaner chore (ConnectionCache) and cached table instances may 
 become invalid. This is Thrift1 specific issue. Thrift2 is OK.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-13376) Improvements to Stochastic load balancer

2015-08-12 Thread Vandana Ayyalasomayajula (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vandana Ayyalasomayajula updated HBASE-13376:
-
Attachment: HBASE-13376_4.patch

 Improvements to Stochastic load balancer
 

 Key: HBASE-13376
 URL: https://issues.apache.org/jira/browse/HBASE-13376
 Project: HBase
  Issue Type: Improvement
  Components: Balancer
Affects Versions: 1.0.0, 0.98.12
Reporter: Vandana Ayyalasomayajula
Assignee: Vandana Ayyalasomayajula
Priority: Minor
 Attachments: 13376-v2.txt, HBASE-13376.patch, HBASE-13376_0.98.txt, 
 HBASE-13376_0.txt, HBASE-13376_1.txt, HBASE-13376_1_1.txt, 
 HBASE-13376_2.patch, HBASE-13376_2_branch-1.patch, HBASE-13376_3.patch, 
 HBASE-13376_3.patch, HBASE-13376_4.patch, HBASE-13376_98.patch, 
 HBASE-13376_branch-1.patch


 There are two things this jira tries to address:
 1. The locality picker in the stochastic balancer does not pick regions with 
 least locality as candidates for swap/move. So when any user configures 
 locality cost in the configs, the balancer does not always seems to move 
 regions with bad locality. 
 2. When a cluster has equal number of loaded regions, it always picks the 
 first one. It should pick a random region on one of the equally loaded 
 servers. This improves a chance of finding a good candidate, when load picker 
 is invoked several times. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-14203) remove duplicate code getTableDescriptor in HTable

2015-08-12 Thread Heng Chen (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-14203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Heng Chen updated HBASE-14203:
--
Attachment: (was: HBASE-14203_v4.patch)

 remove duplicate code getTableDescriptor in HTable
 --

 Key: HBASE-14203
 URL: https://issues.apache.org/jira/browse/HBASE-14203
 Project: HBase
  Issue Type: Improvement
Reporter: Heng Chen
Priority: Trivial
 Attachments: HBASE-14203.patch, HBASE-14203_v2.patch, 
 HBASE-14203_v3.patch


 As TODO in comment said, 
 {{HTable.getTableDescriptor}} is same as {{HAdmin.getTableDescriptor}}. 
 remove the duplicate code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14215) Default cost used for PrimaryRegionCountSkewCostFunction is not sufficient

2015-08-12 Thread Nick Dimiduk (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694308#comment-14694308
 ] 

Nick Dimiduk commented on HBASE-14215:
--

Seems alright to me [~gsbiju], though I haven't spent much time in stochastic 
balancer. These replica weights of 1 seem really high compared to the other 
weight values used in other cost functions. Locality, for instance, is only 25, 
and this often has a high impact on multi-tenent environments (so much so that 
we have HBASE-6721). Any chance you can test on a wider variety of cluster 
sizes and rack distributions to give us a better feel for how this will behave 
in practice? I'd be willing to commit with a couple more data points.

I think [~eclark] and [~enis] have spent more time on balancer, maybe they can 
weigh in too.

 Default cost used for PrimaryRegionCountSkewCostFunction is not sufficient 
 ---

 Key: HBASE-14215
 URL: https://issues.apache.org/jira/browse/HBASE-14215
 Project: HBase
  Issue Type: Bug
  Components: Balancer
Reporter: Biju Nair
Priority: Minor
 Attachments: 14215-v1.txt


 Current multiplier of 500 used in the stochastic balancer cost function 
 ``PrimaryRegionCountSkewCostFunction`` to calculate the cost of  total 
 primary replication skew doesn't seem to be sufficient to prevent the skews 
 (Refer HBASE-14110). We would want the default cost to be a higher value so 
 that skews in primary region replica has higher cost. The following is the 
 test result by setting the multiplier value to 1 (same as the region 
 replica rack cost multiplier) on a 3 Rack 9 RS node cluster which seems to 
 get the balancer distribute the primaries uniformly.
 *Initial Primary replica distribution - using the current multiplier* 
  r1n10  102
  r1n11  85
  r1n988
  r2n10  120
  r2n11  120
  r2n9   124
  r3n10  135
  r3n11  124
  r3n9129
 *After long duration of read  writes - using current multiplier* 
  r1n10  102
  r1n11  85
  r1n988
  r2n10  120
  r2n11  120
  r2n9124
  r3n10  135
  r3n11  124
  r3n9129
 *After manual balancing*  
  r1n10  102
  r1n11  85
  r1n988
  r2n10  120
  r2n11  120
  r2n9124
  r3n10  135
  r3n11  124
  r3n9129
 *Increased multiplier for primaryRegionCountSkewCost to 1*
  r1n10  114
  r1n11  113
  r1n9114
  r2n10  114
  r2n11  114
  r2n9113
  r3n10  115
  r3n11  115
  r3n9115 
 Setting the `PrimaryRegionCountSkewCostFunction` multiplier value to 1 
 should help HBase general use.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-13996) Add write sniffing in canary

2015-08-12 Thread Andrew Purtell (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-13996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-13996:
---
Fix Version/s: (was: 0.98.14)
   0.98.15
   1.3.0
   Status: Open  (was: Patch Available)

 Add write sniffing in canary
 

 Key: HBASE-13996
 URL: https://issues.apache.org/jira/browse/HBASE-13996
 Project: HBase
  Issue Type: New Feature
  Components: canary
Affects Versions: 1.1.0.1, 0.98.13
Reporter: Liu Shaohui
Assignee: Liu Shaohui
 Fix For: 2.0.0, 1.3.0, 0.98.15

 Attachments: HBASE-13996-v001.diff


 Currently the canary tool only sniff the read operations, it's hard to 
 finding the problem in write path. 
 To support the write sniffing, we create a system table named '_canary_'  in 
 the canary tool. And the tool will make sure that the region number is large 
 than the number of the regionserver and the regions will be distributed onto 
 all regionservers.
 Periodically, the tool will put data to these regions to calculate the write 
 availability of HBase and send alerts if needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-12890) Provide a way to throttle the number of regions moved by the balancer

2015-08-12 Thread Andrew Purtell (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-12890:
---
Fix Version/s: (was: 0.98.14)
   0.98.15
   Status: Open  (was: Patch Available)

Moving out to 0.98.15

 Provide a way to throttle the number of regions moved by the balancer
 -

 Key: HBASE-12890
 URL: https://issues.apache.org/jira/browse/HBASE-12890
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.98.10
Reporter: churro morales
Assignee: churro morales
 Fix For: 2.0.0, 1.3.0, 0.98.15

 Attachments: HBASE-12890.patch


 We have a very large cluster and we frequently add remove quite a few 
 regionservers from our cluster.  Whenever we do this the balancer moves 
 thousands of regions at once.  Instead we provide a configuration parameter: 
 hbase.balancer.max.regions.  This limits the number of regions that are 
 balanced per iteration.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-13730) Backport HBASE-13576 (Failure in checking one region should not fail the entire HBCK operation) to 0.98

2015-08-12 Thread Andrew Purtell (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-13730:
---
Status: Open  (was: Patch Available)

 Backport HBASE-13576 (Failure in checking one region should not fail the 
 entire HBCK operation) to 0.98
 ---

 Key: HBASE-13730
 URL: https://issues.apache.org/jira/browse/HBASE-13730
 Project: HBase
  Issue Type: Sub-task
  Components: hbck
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
 Fix For: 0.98.14

 Attachments: HBASE-13730-0.98.patch


 Backport HBASE-13576 - Failure in checking one region should not fail the 
 entire HBCK operation - to 0.98



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14215) Default cost used for PrimaryRegionCountSkewCostFunction is not sufficient

2015-08-12 Thread Elliott Clark (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694328#comment-14694328
 ] 

Elliott Clark commented on HBASE-14215:
---

That cost is pretty freaking high and it could lead to some churn (I really 
think we need to do another pass over the weights that are currently there).
If there's an issue with not converging on for primary replica's then we should 
add a candidate generator. That usually has a larger effect on converging then 
the weights.

 Default cost used for PrimaryRegionCountSkewCostFunction is not sufficient 
 ---

 Key: HBASE-14215
 URL: https://issues.apache.org/jira/browse/HBASE-14215
 Project: HBase
  Issue Type: Bug
  Components: Balancer
Reporter: Biju Nair
Priority: Minor
 Attachments: 14215-v1.txt


 Current multiplier of 500 used in the stochastic balancer cost function 
 ``PrimaryRegionCountSkewCostFunction`` to calculate the cost of  total 
 primary replication skew doesn't seem to be sufficient to prevent the skews 
 (Refer HBASE-14110). We would want the default cost to be a higher value so 
 that skews in primary region replica has higher cost. The following is the 
 test result by setting the multiplier value to 1 (same as the region 
 replica rack cost multiplier) on a 3 Rack 9 RS node cluster which seems to 
 get the balancer distribute the primaries uniformly.
 *Initial Primary replica distribution - using the current multiplier* 
  r1n10  102
  r1n11  85
  r1n988
  r2n10  120
  r2n11  120
  r2n9   124
  r3n10  135
  r3n11  124
  r3n9129
 *After long duration of read  writes - using current multiplier* 
  r1n10  102
  r1n11  85
  r1n988
  r2n10  120
  r2n11  120
  r2n9124
  r3n10  135
  r3n11  124
  r3n9129
 *After manual balancing*  
  r1n10  102
  r1n11  85
  r1n988
  r2n10  120
  r2n11  120
  r2n9124
  r3n10  135
  r3n11  124
  r3n9129
 *Increased multiplier for primaryRegionCountSkewCost to 1*
  r1n10  114
  r1n11  113
  r1n9114
  r2n10  114
  r2n11  114
  r2n9113
  r3n10  115
  r3n11  115
  r3n9115 
 Setting the `PrimaryRegionCountSkewCostFunction` multiplier value to 1 
 should help HBase general use.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-13907) Document how to deploy a coprocessor

2015-08-12 Thread Misty Stanley-Jones (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-13907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Misty Stanley-Jones updated HBASE-13907:

Attachment: HBASE-13907-v4.patch

Is this accurate, [~larsgeorge]?

 Document how to deploy a coprocessor
 

 Key: HBASE-13907
 URL: https://issues.apache.org/jira/browse/HBASE-13907
 Project: HBase
  Issue Type: Bug
  Components: documentation
Reporter: Misty Stanley-Jones
Assignee: Misty Stanley-Jones
 Attachments: HBASE-13907-1.patch, HBASE-13907-2.patch, 
 HBASE-13907-v3.patch, HBASE-13907-v4.patch, HBASE-13907.patch


 Capture this information:
  Where are the dependencies located for these classes? Is there a path on 
  HDFS or local disk that dependencies need to be placed so that each 
  RegionServer has access to them?
 It is suggested to bundle them as a single jar so that RS can load the whole 
 jar and resolve dependencies. If you are not able to do that, you need place 
 the dependencies in regionservers class path so that they are loaded during 
 RS startup. Do either of these options work for you? Btw, you can load the 
 coprocessors/filters into path specified by hbase.dynamic.jars.dir [1], so 
 that they are loaded dynamically by regionservers when the class is accessed 
 (or you can place them in the RS class path too, so that they are loaded 
 during RS JVM startup).
  How would one deploy these using an automated system? 
  (puppet/chef/ansible/etc)
 You can probably use these tools to automate shipping the jars to above 
 locations?
  Tests our developers have done suggest that simply disabling a coprocessor, 
  replacing the jar with a different version, and enabling the coprocessor 
  again does not load the newest version. With that in mind how does one know 
  which version is currently deployed and enabled without resorting to 
  parsing `hbase shell` output or restarting hbase?
 Actually this is a design issue with current classloader. You can't reload a 
 class in a JVM unless you delete all the current references to it. Since the 
 current JVM (classloader) has reference to it, you can't overwrite it unless 
 you kill the JVM, which is equivalent to restarting it. So you still have the 
 older class loaded in place. For this to work, classloader design should be 
 changed. If it works for you, you can rename the coprocessor class name and 
 the new version of jar and RS loads it properly.
  Where does logging go, and how does one access it? Does logging need to be 
  configured in a certain way?
 Can you please specify which logging you are referring to?
  Where is a good location to place configuration files?
 Same as above, are these hbase configs or something else? If hbase configs, 
 are these gateway configs/server side? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-11290) Unlock RegionStates

2015-08-12 Thread Andrew Purtell (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-11290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-11290:
---
Fix Version/s: (was: 0.98.14)
   0.98.15

 Unlock RegionStates
 ---

 Key: HBASE-11290
 URL: https://issues.apache.org/jira/browse/HBASE-11290
 Project: HBase
  Issue Type: Sub-task
Reporter: Francis Liu
Assignee: Virag Kothari
 Fix For: 2.0.0, 1.3.0, 0.98.15

 Attachments: HBASE-11290-0.98.patch, HBASE-11290-0.98_v2.patch, 
 HBASE-11290.draft.patch


 Even though RegionStates is a highly accessed data structure in HMaster. Most 
 of it's methods are synchronized. Which limits concurrency. Even simply 
 making some of the getters non-synchronized by using concurrent data 
 structures has helped with region assignments. We can go as simple as this 
 approach or create locks per region or a bucket lock per region bucket.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-13730) Backport HBASE-13576 (Failure in checking one region should not fail the entire HBCK operation) to 0.98

2015-08-12 Thread Stephen Yuan Jiang (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694460#comment-14694460
 ] 

Stephen Yuan Jiang commented on HBASE-13730:


[~apurtell], in my original patch, I had a cap and config to either turn off 
this feature or fail hbck if too many regions are skipped.  It is not in your 
0.98 patch.  Is this your intension? If so, I am ok with it.  Other than that, 
the 0.98 patch looks good to me.

{code}
+// If some regions is skipped during checkRegionConsistencyConcurrently() 
phase, we might
+// not get accurate state of the hbase if continuing. The config here 
allows users to tune
+// the tolerance of number of skipped region.
+// TODO: evaluate the consequence to continue the hbck operation without 
config.
+int terminateThreshold =  
getConf().getInt(hbase.hbck.skipped.regions.limit, 0);
+int numOfSkippedRegions = skippedRegions.size();
+if (numOfSkippedRegions  0  numOfSkippedRegions  terminateThreshold) {
+  throw new IOException(numOfSkippedRegions
++  region(s) could not be checked or repaired.  See logs for 
detail.);
+}
+
{code}

 Backport HBASE-13576 (Failure in checking one region should not fail the 
 entire HBCK operation) to 0.98
 ---

 Key: HBASE-13730
 URL: https://issues.apache.org/jira/browse/HBASE-13730
 Project: HBase
  Issue Type: Sub-task
  Components: hbck
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
 Fix For: 0.98.14

 Attachments: HBASE-13730-0.98.patch


 Backport HBASE-13576 - Failure in checking one region should not fail the 
 entire HBCK operation - to 0.98



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14098) Allow dropping caches behind compactions

2015-08-12 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694468#comment-14694468
 ] 

Hadoop QA commented on HBASE-14098:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12750184/HBASE-14098-v7-branch-1.patch
  against branch-1 branch at commit 5e5bcceb533e6e4ded65bc778f05da213c07b688.
  ATTACHMENT ID: 12750184

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 20 new 
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
 

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15074//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15074//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15074//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15074//console

This message is automatically generated.

 Allow dropping caches behind compactions
 

 Key: HBASE-14098
 URL: https://issues.apache.org/jira/browse/HBASE-14098
 Project: HBase
  Issue Type: Bug
  Components: Compaction, hadoop2, HFile
Affects Versions: 2.0.0, 1.3.0
Reporter: Elliott Clark
Assignee: Elliott Clark
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-14098-v1.patch, HBASE-14098-v2.patch, 
 HBASE-14098-v3.patch, HBASE-14098-v4.patch, HBASE-14098-v5.patch, 
 HBASE-14098-v6.patch, HBASE-14098-v7-branch-1.patch, HBASE-14098-v7.patch, 
 HBASE-14098.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-13730) Backport HBASE-13576 (Failure in checking one region should not fail the entire HBCK operation) to 0.98

2015-08-12 Thread Andrew Purtell (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694475#comment-14694475
 ] 

Andrew Purtell commented on HBASE-13730:


Thanks for looking [~syuanjiang]. Looks like I took that out along with other 
changes git merged in from branch-1, let me put it back. 

 Backport HBASE-13576 (Failure in checking one region should not fail the 
 entire HBCK operation) to 0.98
 ---

 Key: HBASE-13730
 URL: https://issues.apache.org/jira/browse/HBASE-13730
 Project: HBase
  Issue Type: Sub-task
  Components: hbck
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
 Fix For: 0.98.14

 Attachments: HBASE-13730-0.98.patch


 Backport HBASE-13576 - Failure in checking one region should not fail the 
 entire HBCK operation - to 0.98



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-14196) Thrift server idle connection timeout issue

2015-08-12 Thread Andrew Purtell (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-14196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-14196:
---
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Pushed to 0.98 and up. All thrift units pass on all branches. 

 Thrift server idle connection timeout issue
 ---

 Key: HBASE-14196
 URL: https://issues.apache.org/jira/browse/HBASE-14196
 Project: HBase
  Issue Type: Bug
  Components: Thrift
Affects Versions: 0.98.13, 1.1.1, 1.0.1.1, 1.1.0.1
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov
 Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.3.0, 1.1.3

 Attachments: HBASE-14196.patch


 When idle connection get cleaned from Thrift server, underlying table 
 instances are still cached in a thread local cache.
 This is the antipattern. Table objects are lightweight and should not be 
 cached, besides this, underlying connections can be closed by periodic 
 connection cleaner chore (ConnectionCache) and cached table instances may 
 become invalid. This is Thrift1 specific issue. Thrift2 is OK.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14189) BlockCache options should consider CF Level BlockCacheEnabled setting

2015-08-12 Thread Heng Chen (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694221#comment-14694221
 ] 

Heng Chen commented on HBASE-14189:
---

[~ndimiduk] any suggestions?

 BlockCache options should consider CF Level BlockCacheEnabled setting
 -

 Key: HBASE-14189
 URL: https://issues.apache.org/jira/browse/HBASE-14189
 Project: HBase
  Issue Type: Improvement
  Components: BlockCache
Affects Versions: 2.0.0
Reporter: Heng Chen
Assignee: Heng Chen
 Attachments: HBASE-14189.patch


 While using BlockCache,  we use {{cacheDataOnRead}}({{cacheDataOnWrite}}) 
 represents for whether we should cache block after read(write) block from(to) 
 hdfs.  We should honour BC setting and CF Level cache setting while using 
 BlockCache.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14201) hbck should not take a lock unless fixing errors

2015-08-12 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694244#comment-14694244
 ] 

Hudson commented on HBASE-14201:


FAILURE: Integrated in HBase-1.2-IT #86 (See 
[https://builds.apache.org/job/HBase-1.2-IT/86/])
HBASE-14201 hbck should not take a lock unless fixing errors (eclark: rev 
25bf721b517b7740821dbffc5f7c7b5c539e9aca)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java


 hbck should not take a lock unless fixing errors
 

 Key: HBASE-14201
 URL: https://issues.apache.org/jira/browse/HBASE-14201
 Project: HBase
  Issue Type: Bug
  Components: hbck, util
Affects Versions: 2.0.0, 1.3.0
Reporter: Simon Law
Assignee: Simon Law
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-14201-v0.patch, HBASE-14201-v1.patch


 By default, hbck is run in a read-only checker mode. In this case, it is
 sensible to let others run. By default, the balancer is left alone,
 which may cause spurious errors, but cannot leave the balancer in a bad
 state. It is dangerous to leave the balancer by accident, so it is only
 ever enabled after fixing, it will never be forced off because of
 racing.
 When hbck is run in fixer mode, it must take an exclusive lock and
 disable the balancer, or all havoc will break loose.
 If you want to stop hbck from running in parallel, the -exclusive flag
 will create the lock file. If you want to force -disableBalancer, that
 option is available too. This makes more semantic sense than -noLock and
 -noSwitchBalancer, respectively.
 This task is related to HBASE-14092.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14209) TestShell visibility tests failing

2015-08-12 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694245#comment-14694245
 ] 

Hudson commented on HBASE-14209:


FAILURE: Integrated in HBase-1.2-IT #86 (See 
[https://builds.apache.org/job/HBase-1.2-IT/86/])
HBASE-14209 TestShell visibility tests failing (apurtell: rev 
7fc0b814da02187125a033d98e4f51b8f70d53b7)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityLabelsCache.java


 TestShell visibility tests failing
 --

 Key: HBASE-14209
 URL: https://issues.apache.org/jira/browse/HBASE-14209
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.98.14
Reporter: Andrew Purtell
Assignee: Andrew Purtell
 Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2, 1.3.0

 Attachments: HBASE-14209-0.98.patch


 This is after HBASE-14105  but we've seen this earlier where adding ruby 
 units to the shell tests can cause the visibility tests to fail inexplicably. 
 We can't just avoid adding ruby units to TestShell in 0.98 so figure out the 
 root cause and fix or disable these. 
 {noformat}
   1) Error:
 test_The_get/put_methods_should_work_for_data_written_with_Visibility(Hbase::VisibilityLabelsAdminMethodsTest):
 NativeException: junit.framework.AssertionFailedError: Waiting timed out 
 after [10,000] msec
 junit/framework/Assert.java:57:in `fail'
 org/apache/hadoop/hbase/Waiter.java:193:in `waitFor'
 org/apache/hadoop/hbase/Waiter.java:128:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3514:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3576:in 
 `waitLabelAvailable'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:73:in 
 `test_The_get/put_methods_should_work_for_data_written_with_Visibility'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
   2) Error:
 test_The_set/clear_methods_should_work_with_authorizations(Hbase::VisibilityLabelsAdminMethodsTest):
 NativeException: junit.framework.AssertionFailedError: Waiting timed out 
 after [10,000] msec
 junit/framework/Assert.java:57:in `fail'
 org/apache/hadoop/hbase/Waiter.java:193:in `waitFor'
 org/apache/hadoop/hbase/Waiter.java:128:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3514:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3576:in 
 `waitLabelAvailable'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:52:in 
 `test_The_set/clear_methods_should_work_with_authorizations'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14098) Allow dropping caches behind compactions

2015-08-12 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694316#comment-14694316
 ] 

Hadoop QA commented on HBASE-14098:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12750165/HBASE-14098-branch1-v2.patch
  against master branch at commit beb1f1d358e7f786c510c988b09dab0f87945d1f.
  ATTACHMENT ID: 12750165

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 20 new 
or modified tests.

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15072//console

This message is automatically generated.

 Allow dropping caches behind compactions
 

 Key: HBASE-14098
 URL: https://issues.apache.org/jira/browse/HBASE-14098
 Project: HBase
  Issue Type: Bug
  Components: Compaction, hadoop2, HFile
Affects Versions: 2.0.0, 1.3.0
Reporter: Elliott Clark
Assignee: Elliott Clark
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-14098-branch1-v2.patch, HBASE-14098-v1.patch, 
 HBASE-14098-v2.patch, HBASE-14098-v3.patch, HBASE-14098-v4.patch, 
 HBASE-14098-v5.patch, HBASE-14098-v6.patch, HBASE-14098-v7.patch, 
 HBASE-14098.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-14098) Allow dropping caches behind compactions

2015-08-12 Thread Elliott Clark (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-14098:
--
Attachment: HBASE-14098-v7-branch-1.patch

 Allow dropping caches behind compactions
 

 Key: HBASE-14098
 URL: https://issues.apache.org/jira/browse/HBASE-14098
 Project: HBase
  Issue Type: Bug
  Components: Compaction, hadoop2, HFile
Affects Versions: 2.0.0, 1.3.0
Reporter: Elliott Clark
Assignee: Elliott Clark
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-14098-v1.patch, HBASE-14098-v2.patch, 
 HBASE-14098-v3.patch, HBASE-14098-v4.patch, HBASE-14098-v5.patch, 
 HBASE-14098-v6.patch, HBASE-14098-v7-branch-1.patch, 
 HBASE-14098-v7-branch1.patch, HBASE-14098-v7.patch, HBASE-14098.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14209) TestShell visibility tests failing

2015-08-12 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694477#comment-14694477
 ] 

Hudson commented on HBASE-14209:


FAILURE: Integrated in HBase-TRUNK #6720 (See 
[https://builds.apache.org/job/HBase-TRUNK/6720/])
HBASE-14209 TestShell visibility tests failing (apurtell: rev 
beb1f1d358e7f786c510c988b09dab0f87945d1f)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityLabelsCache.java


 TestShell visibility tests failing
 --

 Key: HBASE-14209
 URL: https://issues.apache.org/jira/browse/HBASE-14209
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.98.14
Reporter: Andrew Purtell
Assignee: Andrew Purtell
 Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2, 1.3.0

 Attachments: HBASE-14209-0.98.patch


 This is after HBASE-14105  but we've seen this earlier where adding ruby 
 units to the shell tests can cause the visibility tests to fail inexplicably. 
 We can't just avoid adding ruby units to TestShell in 0.98 so figure out the 
 root cause and fix or disable these. 
 {noformat}
   1) Error:
 test_The_get/put_methods_should_work_for_data_written_with_Visibility(Hbase::VisibilityLabelsAdminMethodsTest):
 NativeException: junit.framework.AssertionFailedError: Waiting timed out 
 after [10,000] msec
 junit/framework/Assert.java:57:in `fail'
 org/apache/hadoop/hbase/Waiter.java:193:in `waitFor'
 org/apache/hadoop/hbase/Waiter.java:128:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3514:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3576:in 
 `waitLabelAvailable'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:73:in 
 `test_The_get/put_methods_should_work_for_data_written_with_Visibility'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
   2) Error:
 test_The_set/clear_methods_should_work_with_authorizations(Hbase::VisibilityLabelsAdminMethodsTest):
 NativeException: junit.framework.AssertionFailedError: Waiting timed out 
 after [10,000] msec
 junit/framework/Assert.java:57:in `fail'
 org/apache/hadoop/hbase/Waiter.java:193:in `waitFor'
 org/apache/hadoop/hbase/Waiter.java:128:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3514:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3576:in 
 `waitLabelAvailable'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:52:in 
 `test_The_set/clear_methods_should_work_with_authorizations'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14209) TestShell visibility tests failing

2015-08-12 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694266#comment-14694266
 ] 

Hudson commented on HBASE-14209:


FAILURE: Integrated in HBase-1.0 #1012 (See 
[https://builds.apache.org/job/HBase-1.0/1012/])
HBASE-14209 TestShell visibility tests failing (apurtell: rev 
b99b56715e2def928615506b2e9dcd73f779f3d9)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityLabelsCache.java


 TestShell visibility tests failing
 --

 Key: HBASE-14209
 URL: https://issues.apache.org/jira/browse/HBASE-14209
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.98.14
Reporter: Andrew Purtell
Assignee: Andrew Purtell
 Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2, 1.3.0

 Attachments: HBASE-14209-0.98.patch


 This is after HBASE-14105  but we've seen this earlier where adding ruby 
 units to the shell tests can cause the visibility tests to fail inexplicably. 
 We can't just avoid adding ruby units to TestShell in 0.98 so figure out the 
 root cause and fix or disable these. 
 {noformat}
   1) Error:
 test_The_get/put_methods_should_work_for_data_written_with_Visibility(Hbase::VisibilityLabelsAdminMethodsTest):
 NativeException: junit.framework.AssertionFailedError: Waiting timed out 
 after [10,000] msec
 junit/framework/Assert.java:57:in `fail'
 org/apache/hadoop/hbase/Waiter.java:193:in `waitFor'
 org/apache/hadoop/hbase/Waiter.java:128:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3514:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3576:in 
 `waitLabelAvailable'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:73:in 
 `test_The_get/put_methods_should_work_for_data_written_with_Visibility'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
   2) Error:
 test_The_set/clear_methods_should_work_with_authorizations(Hbase::VisibilityLabelsAdminMethodsTest):
 NativeException: junit.framework.AssertionFailedError: Waiting timed out 
 after [10,000] msec
 junit/framework/Assert.java:57:in `fail'
 org/apache/hadoop/hbase/Waiter.java:193:in `waitFor'
 org/apache/hadoop/hbase/Waiter.java:128:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3514:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3576:in 
 `waitLabelAvailable'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:52:in 
 `test_The_set/clear_methods_should_work_with_authorizations'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-11819) Unit test for CoprocessorHConnection

2015-08-12 Thread Andrew Purtell (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-11819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694401#comment-14694401
 ] 

Andrew Purtell commented on HBASE-11819:


Any progress here?

 Unit test for CoprocessorHConnection 
 -

 Key: HBASE-11819
 URL: https://issues.apache.org/jira/browse/HBASE-11819
 Project: HBase
  Issue Type: Test
Reporter: Andrew Purtell
Assignee: Talat UYARER
Priority: Minor
  Labels: beginner
 Fix For: 2.0.0, 0.98.14, 1.3.0, 1.2.1, 1.1.3

 Attachments: HBASE-11819v4-master.patch, HBASE-11819v5-0.98 
 (1).patch, HBASE-11819v5-0.98.patch, HBASE-11819v5-master (1).patch, 
 HBASE-11819v5-master.patch, HBASE-11819v5-master.patch, 
 HBASE-11819v5-v0.98.patch, HBASE-11819v5-v1.0.patch


 Add a unit test to hbase-server that exercises CoprocessorHConnection . 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14082) Add replica id to JMX metrics names

2015-08-12 Thread Lei Chen (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694425#comment-14694425
 ] 

Lei Chen commented on HBASE-14082:
--

Thanks for the suggestion, I will upload a patch for the 1.x soon

 Add replica id to JMX metrics names
 ---

 Key: HBASE-14082
 URL: https://issues.apache.org/jira/browse/HBASE-14082
 Project: HBase
  Issue Type: Improvement
  Components: metrics
Reporter: Lei Chen
Assignee: Lei Chen
 Attachments: HBASE-14082-v1.patch, HBASE-14082-v2.patch


 Today, via JMX, one cannot distinguish a primary region from a replica. A 
 possible solution is to add replica id to JMX metrics names. The benefits may 
 include, for example:
 # Knowing the latency of a read request on a replica region means the first 
 attempt to the primary region has timeout.
 # Write requests on replicas are due to the replication process, while the 
 ones on primary are from clients.
 # In case of looking for hot spots of read operations, replicas should be 
 excluded since TIMELINE reads are sent to all replicas.
 To implement, we can change the format of metrics names found at 
 {code}Hadoop-HBase-RegionServer-Regions-Attributes{code}
 from 
 {code}namespace_namespace_table_tablename_region_regionname_metric_metricname{code}
 to
 {code}namespace_namespace_table_tablename_region_regionname_replicaid_replicaid_metric_metricname{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-12890) Provide a way to throttle the number of regions moved by the balancer

2015-08-12 Thread Elliott Clark (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694426#comment-14694426
 ] 

Elliott Clark commented on HBASE-12890:
---

We already have something pretty close to this in the balancer. Why not put 
that in there?

{code}hbase.master.balancer.stochastic.maxMovePercent{code}

That sets the max percent. Then we also will only move Max of the percent and 
600 regions at a time. We could just make the 600 configurable. That would make 
the balancer smarter rather than generating plans that won't do much.

 Provide a way to throttle the number of regions moved by the balancer
 -

 Key: HBASE-12890
 URL: https://issues.apache.org/jira/browse/HBASE-12890
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.98.10
Reporter: churro morales
Assignee: churro morales
 Fix For: 2.0.0, 0.98.14, 1.3.0

 Attachments: HBASE-12890.patch


 We have a very large cluster and we frequently add remove quite a few 
 regionservers from our cluster.  Whenever we do this the balancer moves 
 thousands of regions at once.  Instead we provide a configuration parameter: 
 hbase.balancer.max.regions.  This limits the number of regions that are 
 balanced per iteration.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14209) TestShell visibility tests failing

2015-08-12 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694217#comment-14694217
 ] 

Hudson commented on HBASE-14209:


FAILURE: Integrated in HBase-0.98 #1077 (See 
[https://builds.apache.org/job/HBase-0.98/1077/])
HBASE-14209 TestShell visibility tests failing (apurtell: rev 
cddb28ef9fae67087280b99c4323945e43fba000)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityLabelsCache.java


 TestShell visibility tests failing
 --

 Key: HBASE-14209
 URL: https://issues.apache.org/jira/browse/HBASE-14209
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.98.14
Reporter: Andrew Purtell
Assignee: Andrew Purtell
 Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2, 1.3.0

 Attachments: HBASE-14209-0.98.patch


 This is after HBASE-14105  but we've seen this earlier where adding ruby 
 units to the shell tests can cause the visibility tests to fail inexplicably. 
 We can't just avoid adding ruby units to TestShell in 0.98 so figure out the 
 root cause and fix or disable these. 
 {noformat}
   1) Error:
 test_The_get/put_methods_should_work_for_data_written_with_Visibility(Hbase::VisibilityLabelsAdminMethodsTest):
 NativeException: junit.framework.AssertionFailedError: Waiting timed out 
 after [10,000] msec
 junit/framework/Assert.java:57:in `fail'
 org/apache/hadoop/hbase/Waiter.java:193:in `waitFor'
 org/apache/hadoop/hbase/Waiter.java:128:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3514:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3576:in 
 `waitLabelAvailable'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:73:in 
 `test_The_get/put_methods_should_work_for_data_written_with_Visibility'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
   2) Error:
 test_The_set/clear_methods_should_work_with_authorizations(Hbase::VisibilityLabelsAdminMethodsTest):
 NativeException: junit.framework.AssertionFailedError: Waiting timed out 
 after [10,000] msec
 junit/framework/Assert.java:57:in `fail'
 org/apache/hadoop/hbase/Waiter.java:193:in `waitFor'
 org/apache/hadoop/hbase/Waiter.java:128:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3514:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3576:in 
 `waitLabelAvailable'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:52:in 
 `test_The_set/clear_methods_should_work_with_authorizations'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-2821) Keep young storefiles at lower replication

2015-08-12 Thread Andrew Purtell (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-2821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-2821:
--
   Resolution: Later
 Assignee: (was: Andrew Purtell)
Fix Version/s: (was: 2.0.0)
   Status: Resolved  (was: Patch Available)

No movement. Let's resolve as Later

 Keep young storefiles at lower replication
 --

 Key: HBASE-2821
 URL: https://issues.apache.org/jira/browse/HBASE-2821
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Todd Lipcon
 Attachments: HBASE-2821.patch, HBASE-2821.patch, HBASE-2821.patch, 
 lifetime-distribution.png, storefile_age.pl


 jgray and I were brainstorming some ideas about this:
 In a typical heavy-write scenario, many store files do not last very long. 
 They're flushed and then within a small number of seconds a compaction runs 
 and they get deleted. For these short lifetime store files, it's less 
 likely that a failure will occur during the window in which they're valid. 
 So, I think we can consider some optimizations like the following:
 - Flush files at replication count 2. Scan once a minute for any store files 
 in the region that are older than 2 minutes. If they're found, increase their 
 replication to 3. (alternatively, queue them to avoid scanning)
 - More dangerous: flush files at replication count 1, but don't count them 
 when figuring log expiration. So, if they get lost, we force log splitting to 
 recover.
 The performance gain here is that we avoid the network and disk transfer of 
 writing the third replica for a file that we're just about to delete anyway.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14207) Region was hijacked and remained in transition when RS failed to open a region and later regionplan changed to new RS on retry

2015-08-12 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694379#comment-14694379
 ] 

Hadoop QA commented on HBASE-14207:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12750146/HBASE-14207-0.98.patch
  against 0.98 branch at commit 0c63d415d9d23a80ec3b8f454bbf4748de4a1a78.
  ATTACHMENT ID: 12750146

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 
22 warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

{color:red}-1 site{color}.  The patch appears to cause mvn post-site goal 
to fail.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   org.apache.hadoop.hbase.master.TestZKLessAMOnCluster

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15071//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15071//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15071//artifact/patchprocess/checkstyle-aggregate.html

  Javadoc warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15071//artifact/patchprocess/patchJavadocWarnings.txt
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15071//console

This message is automatically generated.

 Region was hijacked and remained in transition when RS failed to open a 
 region and later regionplan changed to new RS on retry
 --

 Key: HBASE-14207
 URL: https://issues.apache.org/jira/browse/HBASE-14207
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.98.6
Reporter: Pankaj Kumar
Assignee: Pankaj Kumar
Priority: Critical
 Fix For: 0.98.14, 0.98.15

 Attachments: HBASE-14207-0.98.patch


 On production environment, following events happened
 1. Master is trying to assign a region to RS, but due to 
 KeeperException$SessionExpiredException RS failed to open the region.
   In RS log, saw multiple WARN log related to 
 KeeperException$SessionExpiredException 
KeeperErrorCode = Session expired for 
 /hbase/region-in-transition/08f1935d652e5dbdac09b423b8f9401b
Unable to get data of znode 
 /hbase/region-in-transition/08f1935d652e5dbdac09b423b8f9401b
 2. Master retried to assign the region to same RS, but RS again failed.
 3. On second retry new plan formed and this time plan destination (RS) is 
 different, so master send the request to new RS to open the region. But new 
 RS failed to open the region as there was server mismatch in ZNODE than the  
 expected current server name. 
 Logs Snippet:
 {noformat}
 HM
 2015-07-14 03:50:29,759 | INFO  | master:T101PC03VM13:21300 | Processing 
 08f1935d652e5dbdac09b423b8f9401b in state: M_ZK_REGION_OFFLINE | 
 org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:644)
 2015-07-14 03:50:29,759 | INFO  | master:T101PC03VM13:21300 | Transitioned 
 {08f1935d652e5dbdac09b423b8f9401b state=OFFLINE, ts=1436817029679, 
 server=null} to {08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN, 
 ts=1436817029759, server=T101PC03VM13,21302,1436816690692} | 
 org.apache.hadoop.hbase.master.RegionStates.updateRegionState(RegionStates.java:327)
 2015-07-14 03:50:29,760 | INFO  | master:T101PC03VM13:21300 | Processed 
 region 08f1935d652e5dbdac09b423b8f9401b in state M_ZK_REGION_OFFLINE, on 
 server: 

[jira] [Resolved] (HBASE-13391) TestRegionObserverInterface frequently failing on branch-1

2015-08-12 Thread Andrew Purtell (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-13391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell resolved HBASE-13391.

   Resolution: Not A Problem
 Assignee: (was: Andrew Purtell)
Fix Version/s: (was: 1.3.0)
   (was: 2.0.0)

Haven't seen this for a while

 TestRegionObserverInterface frequently failing on branch-1 
 ---

 Key: HBASE-13391
 URL: https://issues.apache.org/jira/browse/HBASE-13391
 Project: HBase
  Issue Type: Bug
Reporter: Andrew Purtell
 Attachments: HBASE-13391-ignore2Tests.v1-branch-1.1.patch, 
 test.log.fail.txt, test.log.pass.txt


 TestRegionObserverInterface is frequently failing on branch-1 .
 Example:
 {noformat}
 java.lang.AssertionError: Result of 
 org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy.getCtPreWALRestore
  is expected to be 1, while we get 0
   at org.junit.Assert.fail(Assert.java:88)
   at org.junit.Assert.assertTrue(Assert.java:41)
   at 
 org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface.verifyMethodResult(TestRegionObserverInterface.java:751)
   at 
 org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface.testLegacyRecovery(TestRegionObserverInterface.java:685)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-13336) Consistent rules for security meta table protections

2015-08-12 Thread Andrew Purtell (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-13336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-13336:
---
Fix Version/s: (was: 0.98.14)
   0.98.15
   Status: Open  (was: Patch Available)

Cancelling stale patch

 Consistent rules for security meta table protections
 

 Key: HBASE-13336
 URL: https://issues.apache.org/jira/browse/HBASE-13336
 Project: HBase
  Issue Type: Improvement
Reporter: Andrew Purtell
Assignee: Srikanth Srungarapu
 Fix For: 2.0.0, 1.3.0, 0.98.15

 Attachments: HBASE-13336.patch, HBASE-13336_v2.patch


 The AccessController and VisibilityController do different things regarding 
 protecting their meta tables. The AC allows schema changes and disable/enable 
 if the user has permission. The VC unconditionally disallows all admin 
 actions. Generally, bad things will happen if these meta tables are damaged, 
 disabled, or dropped. The likely outcome is random frequent (or constant) 
 server side op failures with nasty stack traces. On the other hand some 
 things like column family and table attribute changes can have valid use 
 cases. We should have consistent and sensible rules for protecting security 
 meta tables.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-14207) Region was hijacked and remained in transition when RS failed to open a region and later regionplan changed to new RS on retry

2015-08-12 Thread Andrew Purtell (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-14207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-14207:
---
Fix Version/s: (was: 0.98.14)
   Status: Open  (was: Patch Available)

bq. org.apache.hadoop.hbase.master.TestZKLessAMOnCluster

This looks like a relevant test failure.

 Region was hijacked and remained in transition when RS failed to open a 
 region and later regionplan changed to new RS on retry
 --

 Key: HBASE-14207
 URL: https://issues.apache.org/jira/browse/HBASE-14207
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.98.6
Reporter: Pankaj Kumar
Assignee: Pankaj Kumar
Priority: Critical
 Fix For: 0.98.15

 Attachments: HBASE-14207-0.98.patch


 On production environment, following events happened
 1. Master is trying to assign a region to RS, but due to 
 KeeperException$SessionExpiredException RS failed to open the region.
   In RS log, saw multiple WARN log related to 
 KeeperException$SessionExpiredException 
KeeperErrorCode = Session expired for 
 /hbase/region-in-transition/08f1935d652e5dbdac09b423b8f9401b
Unable to get data of znode 
 /hbase/region-in-transition/08f1935d652e5dbdac09b423b8f9401b
 2. Master retried to assign the region to same RS, but RS again failed.
 3. On second retry new plan formed and this time plan destination (RS) is 
 different, so master send the request to new RS to open the region. But new 
 RS failed to open the region as there was server mismatch in ZNODE than the  
 expected current server name. 
 Logs Snippet:
 {noformat}
 HM
 2015-07-14 03:50:29,759 | INFO  | master:T101PC03VM13:21300 | Processing 
 08f1935d652e5dbdac09b423b8f9401b in state: M_ZK_REGION_OFFLINE | 
 org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:644)
 2015-07-14 03:50:29,759 | INFO  | master:T101PC03VM13:21300 | Transitioned 
 {08f1935d652e5dbdac09b423b8f9401b state=OFFLINE, ts=1436817029679, 
 server=null} to {08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN, 
 ts=1436817029759, server=T101PC03VM13,21302,1436816690692} | 
 org.apache.hadoop.hbase.master.RegionStates.updateRegionState(RegionStates.java:327)
 2015-07-14 03:50:29,760 | INFO  | master:T101PC03VM13:21300 | Processed 
 region 08f1935d652e5dbdac09b423b8f9401b in state M_ZK_REGION_OFFLINE, on 
 server: T101PC03VM13,21302,1436816690692 | 
 org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:768)
 2015-07-14 03:50:29,800 | INFO  | 
 MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Assigning 
 INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to 
 T101PC03VM13,21302,1436816690692 | 
 org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1983)
 2015-07-14 03:50:29,801 | WARN  | 
 MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Failed assignment of 
 INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to 
 T101PC03VM13,21302,1436816690692, trying to assign elsewhere instead; try=1 
 of 10 | 
 org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2077)
 2015-07-14 03:50:29,802 | INFO  | 
 MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Trying to re-assign 
 INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to 
 the same failed server. | 
 org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2123)
 2015-07-14 03:50:31,804 | INFO  | 
 MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Assigning 
 INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to 
 T101PC03VM13,21302,1436816690692 | 
 org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1983)
 2015-07-14 03:50:31,806 | WARN  | 
 MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Failed assignment of 
 INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to 
 T101PC03VM13,21302,1436816690692, trying to assign elsewhere instead; try=2 
 of 10 | 
 org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2077)
 2015-07-14 03:50:31,807 | INFO  | 
 MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Transitioned 
 {08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN, ts=1436817031804, 
 server=T101PC03VM13,21302,1436816690692} to {08f1935d652e5dbdac09b423b8f9401b 
 state=OFFLINE, ts=1436817031807, server=T101PC03VM13,21302,1436816690692} | 
 org.apache.hadoop.hbase.master.RegionStates.updateRegionState(RegionStates.java:327)
 2015-07-14 03:50:31,807 | INFO  | 
 MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Assigning 
 

[jira] [Updated] (HBASE-12748) RegionCoprocessorHost.execOperation creates too many iterator objects

2015-08-12 Thread Andrew Purtell (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-12748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-12748:
---
Fix Version/s: (was: 1.0.3)
   (was: 0.98.14)
   0.98.15
   1.3.0

 RegionCoprocessorHost.execOperation creates too many iterator objects
 -

 Key: HBASE-12748
 URL: https://issues.apache.org/jira/browse/HBASE-12748
 Project: HBase
  Issue Type: Sub-task
Affects Versions: 0.94.25, 0.98.9
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov
 Fix For: 2.0.0, 1.3.0, 0.98.15

 Attachments: HBase-12748.patch


 This is typical pattern of enhanced for ... loop usage in a hot code path. 
 For every HBase operation it instantiates iterator for coprocessor list 
 twice. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14218) CompareFilter based on part of the key

2015-08-12 Thread Vladimir Rodionov (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694187#comment-14694187
 ] 

Vladimir Rodionov commented on HBASE-14218:
---

This HBASE-6618 is more generic approach.

 CompareFilter based on part of the key
 --

 Key: HBASE-14218
 URL: https://issues.apache.org/jira/browse/HBASE-14218
 Project: HBase
  Issue Type: Bug
  Components: Filters
Affects Versions: 1.1.0.1
Reporter: Jean-Marc Spaggiari
Priority: Minor

 HBase needs to provide a combination of FuzzyFilte rand CompareFilter to be 
 able to compare based on part of the key.
 Something like:
 xxxxx
 And we want to filter all rows where  is greater than a certain value.
 Fuzzy filter can do the test on  but only on exact match
 CompareFilter can to GREATER but on the entire key...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14201) hbck should not take a lock unless fixing errors

2015-08-12 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694260#comment-14694260
 ] 

Hudson commented on HBASE-14201:


FAILURE: Integrated in HBase-1.3-IT #85 (See 
[https://builds.apache.org/job/HBase-1.3-IT/85/])
HBASE-14201 hbck should not take a lock unless fixing errors (eclark: rev 
881d2c1d74419013c10a37789f6203ed0bd05dd5)
* hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java


 hbck should not take a lock unless fixing errors
 

 Key: HBASE-14201
 URL: https://issues.apache.org/jira/browse/HBASE-14201
 Project: HBase
  Issue Type: Bug
  Components: hbck, util
Affects Versions: 2.0.0, 1.3.0
Reporter: Simon Law
Assignee: Simon Law
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-14201-v0.patch, HBASE-14201-v1.patch


 By default, hbck is run in a read-only checker mode. In this case, it is
 sensible to let others run. By default, the balancer is left alone,
 which may cause spurious errors, but cannot leave the balancer in a bad
 state. It is dangerous to leave the balancer by accident, so it is only
 ever enabled after fixing, it will never be forced off because of
 racing.
 When hbck is run in fixer mode, it must take an exclusive lock and
 disable the balancer, or all havoc will break loose.
 If you want to stop hbck from running in parallel, the -exclusive flag
 will create the lock file. If you want to force -disableBalancer, that
 option is available too. This makes more semantic sense than -noLock and
 -noSwitchBalancer, respectively.
 This task is related to HBASE-14092.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14209) TestShell visibility tests failing

2015-08-12 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694261#comment-14694261
 ] 

Hudson commented on HBASE-14209:


FAILURE: Integrated in HBase-1.3-IT #85 (See 
[https://builds.apache.org/job/HBase-1.3-IT/85/])
HBASE-14209 TestShell visibility tests failing (apurtell: rev 
ba328d26eab8bc82d6c7e3a11f4254cb036b4d23)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityLabelsCache.java


 TestShell visibility tests failing
 --

 Key: HBASE-14209
 URL: https://issues.apache.org/jira/browse/HBASE-14209
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.98.14
Reporter: Andrew Purtell
Assignee: Andrew Purtell
 Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2, 1.3.0

 Attachments: HBASE-14209-0.98.patch


 This is after HBASE-14105  but we've seen this earlier where adding ruby 
 units to the shell tests can cause the visibility tests to fail inexplicably. 
 We can't just avoid adding ruby units to TestShell in 0.98 so figure out the 
 root cause and fix or disable these. 
 {noformat}
   1) Error:
 test_The_get/put_methods_should_work_for_data_written_with_Visibility(Hbase::VisibilityLabelsAdminMethodsTest):
 NativeException: junit.framework.AssertionFailedError: Waiting timed out 
 after [10,000] msec
 junit/framework/Assert.java:57:in `fail'
 org/apache/hadoop/hbase/Waiter.java:193:in `waitFor'
 org/apache/hadoop/hbase/Waiter.java:128:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3514:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3576:in 
 `waitLabelAvailable'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:73:in 
 `test_The_get/put_methods_should_work_for_data_written_with_Visibility'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
   2) Error:
 test_The_set/clear_methods_should_work_with_authorizations(Hbase::VisibilityLabelsAdminMethodsTest):
 NativeException: junit.framework.AssertionFailedError: Waiting timed out 
 after [10,000] msec
 junit/framework/Assert.java:57:in `fail'
 org/apache/hadoop/hbase/Waiter.java:193:in `waitFor'
 org/apache/hadoop/hbase/Waiter.java:128:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3514:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3576:in 
 `waitLabelAvailable'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:52:in 
 `test_The_set/clear_methods_should_work_with_authorizations'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14209) TestShell visibility tests failing

2015-08-12 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694464#comment-14694464
 ] 

Hudson commented on HBASE-14209:


FAILURE: Integrated in HBase-1.3 #104 (See 
[https://builds.apache.org/job/HBase-1.3/104/])
HBASE-14209 TestShell visibility tests failing (apurtell: rev 
ba328d26eab8bc82d6c7e3a11f4254cb036b4d23)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityLabelsCache.java


 TestShell visibility tests failing
 --

 Key: HBASE-14209
 URL: https://issues.apache.org/jira/browse/HBASE-14209
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.98.14
Reporter: Andrew Purtell
Assignee: Andrew Purtell
 Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2, 1.3.0

 Attachments: HBASE-14209-0.98.patch


 This is after HBASE-14105  but we've seen this earlier where adding ruby 
 units to the shell tests can cause the visibility tests to fail inexplicably. 
 We can't just avoid adding ruby units to TestShell in 0.98 so figure out the 
 root cause and fix or disable these. 
 {noformat}
   1) Error:
 test_The_get/put_methods_should_work_for_data_written_with_Visibility(Hbase::VisibilityLabelsAdminMethodsTest):
 NativeException: junit.framework.AssertionFailedError: Waiting timed out 
 after [10,000] msec
 junit/framework/Assert.java:57:in `fail'
 org/apache/hadoop/hbase/Waiter.java:193:in `waitFor'
 org/apache/hadoop/hbase/Waiter.java:128:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3514:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3576:in 
 `waitLabelAvailable'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:73:in 
 `test_The_get/put_methods_should_work_for_data_written_with_Visibility'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
   2) Error:
 test_The_set/clear_methods_should_work_with_authorizations(Hbase::VisibilityLabelsAdminMethodsTest):
 NativeException: junit.framework.AssertionFailedError: Waiting timed out 
 after [10,000] msec
 junit/framework/Assert.java:57:in `fail'
 org/apache/hadoop/hbase/Waiter.java:193:in `waitFor'
 org/apache/hadoop/hbase/Waiter.java:128:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3514:in `waitFor'
 org/apache/hadoop/hbase/HBaseTestingUtility.java:3576:in 
 `waitLabelAvailable'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:52:in 
 `test_The_set/clear_methods_should_work_with_authorizations'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-13730) Backport HBASE-13576 (Failure in checking one region should not fail the entire HBCK operation) to 0.98

2015-08-12 Thread Andrew Purtell (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-13730:
---
Status: Patch Available  (was: Open)

 Backport HBASE-13576 (Failure in checking one region should not fail the 
 entire HBCK operation) to 0.98
 ---

 Key: HBASE-13730
 URL: https://issues.apache.org/jira/browse/HBASE-13730
 Project: HBase
  Issue Type: Sub-task
  Components: hbck
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
 Fix For: 0.98.14

 Attachments: HBASE-13730-0.98.patch, HBASE-13730-0.98.patch


 Backport HBASE-13576 - Failure in checking one region should not fail the 
 entire HBCK operation - to 0.98



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14122) Client API for determining if server side supports cell level security

2015-08-12 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694520#comment-14694520
 ] 

Hudson commented on HBASE-14122:


SUCCESS: Integrated in HBase-1.2-IT #87 (See 
[https://builds.apache.org/job/HBase-1.2-IT/87/])
HBASE-14122 Client API for determining if server side supports cell level 
security (apurtell: rev 2a5b5c791d0b0a6db02be912708def885d35a404)
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityClient.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
* hbase-shell/src/main/ruby/hbase/visibility_labels.rb
* hbase-shell/src/main/ruby/hbase/security.rb
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java
* 
hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/MasterProtos.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlClient.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabels.java
* hbase-shell/src/main/ruby/shell.rb
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClusterConnection.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityController.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
* hbase-shell/src/main/ruby/hbase/admin.rb
* hbase-shell/src/main/ruby/shell/commands/list_security_capabilities.rb
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/security/SecurityCapability.java
* hbase-protocol/src/main/protobuf/Master.proto


 Client API for determining if server side supports cell level security
 --

 Key: HBASE-14122
 URL: https://issues.apache.org/jira/browse/HBASE-14122
 Project: HBase
  Issue Type: Improvement
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
 Fix For: 2.0.0, 0.98.14, 1.2.0, 1.3.0

 Attachments: HBASE-14122-0.98.patch, HBASE-14122-0.98.patch, 
 HBASE-14122-branch-1.patch, HBASE-14122-branch-1.patch, HBASE-14122.patch, 
 HBASE-14122.patch, HBASE-14122.patch


 Add a client API for determining if the server side supports cell level 
 security. 
 Ask the master, assuming as we do in many other instances that the master and 
 regionservers all have a consistent view of site configuration.
 Return {{true}} if all features required for cell level security are present, 
 {{false}} otherwise, or throw {{UnsupportedOperationException}} if the master 
 does not have support for the RPC call.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-13985) Add configuration to skip validating HFile format when bulk loading millions of HFiles

2015-08-12 Thread Andrew Purtell (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694527#comment-14694527
 ] 

Andrew Purtell commented on HBASE-13985:


Ok, committing shortly


 Add configuration to skip validating HFile format when bulk loading millions 
 of HFiles
 --

 Key: HBASE-13985
 URL: https://issues.apache.org/jira/browse/HBASE-13985
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.98.13
Reporter: Victor Xu
Assignee: Victor Xu
Priority: Minor
  Labels: regionserver
 Fix For: 0.98.14

 Attachments: HBASE-13985-v2.patch, HBASE-13985-v3.patch, 
 HBASE-13985.patch


 When bulk loading millions of HFile into one HTable, checking HFile format is 
 the most time-consuming phase. Maybe we could use a parallel mechanism to 
 increase the speed, but when it comes to millions of HFiles, it may still 
 cost dozens of minutes. So I think it's necessary to add an option for 
 advanced user to bulkload without checking HFile format at all. 
 Of course, the default value of this option should be true.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14203) remove duplicate code getTableDescriptor in HTable

2015-08-12 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694532#comment-14694532
 ] 

Hadoop QA commented on HBASE-14203:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12750163/HBASE-14203_v4.patch
  against master branch at commit beb1f1d358e7f786c510c988b09dab0f87945d1f.
  ATTACHMENT ID: 12750163

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:red}-1 checkstyle{color}.  The applied patch generated 
1861 checkstyle errors (more than the master's current 1857 errors).

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   
org.apache.hadoop.hbase.procedure.TestZKProcedureControllers
  org.apache.hadoop.hbase.procedure.TestZKProcedure
  org.apache.hadoop.hbase.mapreduce.TestImportExport
  org.apache.hadoop.hbase.util.TestProcessBasedCluster
  org.apache.hadoop.hbase.TestAcidGuarantees

 {color:red}-1 core zombie tests{color}.  There are 2 zombie test(s):   
at 
org.apache.phoenix.hbase.index.balancer.IndexLoadBalancerIT.testRoundRobinAssignmentDuringMasterStartUp(IndexLoadBalancerIT.java:420)

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15073//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15073//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15073//artifact/patchprocess/checkstyle-aggregate.html

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15073//console

This message is automatically generated.

 remove duplicate code getTableDescriptor in HTable
 --

 Key: HBASE-14203
 URL: https://issues.apache.org/jira/browse/HBASE-14203
 Project: HBase
  Issue Type: Improvement
Reporter: Heng Chen
Priority: Trivial
 Attachments: HBASE-14203.patch, HBASE-14203_v2.patch, 
 HBASE-14203_v3.patch, HBASE-14203_v4.patch


 As TODO in comment said, 
 {{HTable.getTableDescriptor}} is same as {{HAdmin.getTableDescriptor}}. 
 remove the duplicate code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-13985) Add configuration to skip validating HFile format when bulk loading

2015-08-12 Thread Andrew Purtell (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-13985:
---
Summary: Add configuration to skip validating HFile format when bulk 
loading  (was: Add configuration to skip validating HFile format when bulk 
loading millions of HFiles)

 Add configuration to skip validating HFile format when bulk loading
 ---

 Key: HBASE-13985
 URL: https://issues.apache.org/jira/browse/HBASE-13985
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.98.13
Reporter: Victor Xu
Assignee: Victor Xu
Priority: Minor
  Labels: regionserver
 Fix For: 0.98.14

 Attachments: HBASE-13985-v2.patch, HBASE-13985-v3.patch, 
 HBASE-13985.patch


 When bulk loading millions of HFile into one HTable, checking HFile format is 
 the most time-consuming phase. Maybe we could use a parallel mechanism to 
 increase the speed, but when it comes to millions of HFiles, it may still 
 cost dozens of minutes. So I think it's necessary to add an option for 
 advanced user to bulkload without checking HFile format at all. 
 Of course, the default value of this option should be true.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-13996) Add write sniffing in canary

2015-08-12 Thread Liu Shaohui (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-13996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694559#comment-14694559
 ] 

Liu Shaohui commented on HBASE-13996:
-

[~stack] [~apurtell]
Sorry for leaving the issue here so long.
I will update the patch according stack's review in few days.


 Add write sniffing in canary
 

 Key: HBASE-13996
 URL: https://issues.apache.org/jira/browse/HBASE-13996
 Project: HBase
  Issue Type: Improvement
  Components: canary
Affects Versions: 0.98.13, 1.1.0.1
Reporter: Liu Shaohui
Assignee: Liu Shaohui
 Fix For: 2.0.0, 1.3.0, 0.98.15

 Attachments: HBASE-13996-v001.diff


 Currently the canary tool only sniff the read operations, it's hard to 
 finding the problem in write path. 
 To support the write sniffing, we create a system table named '_canary_'  in 
 the canary tool. And the tool will make sure that the region number is large 
 than the number of the regionserver and the regions will be distributed onto 
 all regionservers.
 Periodically, the tool will put data to these regions to calculate the write 
 availability of HBase and send alerts if needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14196) Thrift server idle connection timeout issue

2015-08-12 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694620#comment-14694620
 ] 

Hudson commented on HBASE-14196:


FAILURE: Integrated in HBase-1.1 #611 (See 
[https://builds.apache.org/job/HBase-1.1/611/])
HBASE-14196 Thrift server idle connection timeout issue (Vladimir Rodionov) 
(apurtell: rev 5627e0020e59d3bf77d1a83b57037d9837c7b99f)
* 
hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java
* 
hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/IncrementCoalescer.java


 Thrift server idle connection timeout issue
 ---

 Key: HBASE-14196
 URL: https://issues.apache.org/jira/browse/HBASE-14196
 Project: HBase
  Issue Type: Bug
  Components: Thrift
Affects Versions: 0.98.13, 1.1.1, 1.0.1.1, 1.1.0.1
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov
 Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.3.0, 1.1.3

 Attachments: HBASE-14196.patch


 When idle connection get cleaned from Thrift server, underlying table 
 instances are still cached in a thread local cache.
 This is the antipattern. Table objects are lightweight and should not be 
 cached, besides this, underlying connections can be closed by periodic 
 connection cleaner chore (ConnectionCache) and cached table instances may 
 become invalid. This is Thrift1 specific issue. Thrift2 is OK.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-13996) Add write sniffing in canary

2015-08-12 Thread Liu Shaohui (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-13996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liu Shaohui updated HBASE-13996:

Issue Type: Improvement  (was: New Feature)

 Add write sniffing in canary
 

 Key: HBASE-13996
 URL: https://issues.apache.org/jira/browse/HBASE-13996
 Project: HBase
  Issue Type: Improvement
  Components: canary
Affects Versions: 0.98.13, 1.1.0.1
Reporter: Liu Shaohui
Assignee: Liu Shaohui
 Fix For: 2.0.0, 1.3.0, 0.98.15

 Attachments: HBASE-13996-v001.diff


 Currently the canary tool only sniff the read operations, it's hard to 
 finding the problem in write path. 
 To support the write sniffing, we create a system table named '_canary_'  in 
 the canary tool. And the tool will make sure that the region number is large 
 than the number of the regionserver and the regions will be distributed onto 
 all regionservers.
 Periodically, the tool will put data to these regions to calculate the write 
 availability of HBase and send alerts if needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-13730) Backport HBASE-13576 (Failure in checking one region should not fail the entire HBCK operation) to 0.98

2015-08-12 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694678#comment-14694678
 ] 

Hadoop QA commented on HBASE-13730:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12750204/HBASE-13730-0.98.patch
  against 0.98 branch at commit 643ba90185f20419016080d6d32adba9fe7019dd.
  ATTACHMENT ID: 12750204

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 4 new 
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 
23 warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

{color:red}-1 site{color}.  The patch appears to cause mvn post-site goal 
to fail.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15078//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15078//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15078//artifact/patchprocess/checkstyle-aggregate.html

  Javadoc warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15078//artifact/patchprocess/patchJavadocWarnings.txt
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15078//console

This message is automatically generated.

 Backport HBASE-13576 (Failure in checking one region should not fail the 
 entire HBCK operation) to 0.98
 ---

 Key: HBASE-13730
 URL: https://issues.apache.org/jira/browse/HBASE-13730
 Project: HBase
  Issue Type: Sub-task
  Components: hbck
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
 Fix For: 0.98.14

 Attachments: HBASE-13730-0.98.patch, HBASE-13730-0.98.patch


 Backport HBASE-13576 - Failure in checking one region should not fail the 
 entire HBCK operation - to 0.98



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-14144) Bloomfilter path to work with Byte buffered cells

2015-08-12 Thread ramkrishna.s.vasudevan (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-14144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-14144:
---
Status: Open  (was: Patch Available)

 Bloomfilter path to work with Byte buffered cells
 -

 Key: HBASE-14144
 URL: https://issues.apache.org/jira/browse/HBASE-14144
 Project: HBase
  Issue Type: Sub-task
  Components: regionserver, Scanners
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 2.0.0

 Attachments: HBASE-14144.patch, HBASE-14144_1.patch, 
 HBASE-14144_2.patch


 This JIRA is to check if there will be a need to make the bloom filters to 
 work with ByteBuffer cells. During POC this path created lot of duplicated 
 code but considering other refactorings done in this path  may lead to less 
 duplication. This JIRA is a placeholder.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-13730) Backport HBASE-13576 (Failure in checking one region should not fail the entire HBCK operation) to 0.98

2015-08-12 Thread Andrew Purtell (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-13730:
---
Attachment: HBASE-13730-0.98.patch

Updated patch with the skipped limit check. 

 Backport HBASE-13576 (Failure in checking one region should not fail the 
 entire HBCK operation) to 0.98
 ---

 Key: HBASE-13730
 URL: https://issues.apache.org/jira/browse/HBASE-13730
 Project: HBase
  Issue Type: Sub-task
  Components: hbck
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
 Fix For: 0.98.14

 Attachments: HBASE-13730-0.98.patch, HBASE-13730-0.98.patch


 Backport HBASE-13576 - Failure in checking one region should not fail the 
 entire HBCK operation - to 0.98



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14196) Thrift server idle connection timeout issue

2015-08-12 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694576#comment-14694576
 ] 

Hudson commented on HBASE-14196:


FAILURE: Integrated in HBase-1.0 #1013 (See 
[https://builds.apache.org/job/HBase-1.0/1013/])
HBASE-14196 Thrift server idle connection timeout issue (Vladimir Rodionov) 
(apurtell: rev a81d0a4a7b9792681b12b5f7a7fc59aff70ae884)
* 
hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java
* 
hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/IncrementCoalescer.java


 Thrift server idle connection timeout issue
 ---

 Key: HBASE-14196
 URL: https://issues.apache.org/jira/browse/HBASE-14196
 Project: HBase
  Issue Type: Bug
  Components: Thrift
Affects Versions: 0.98.13, 1.1.1, 1.0.1.1, 1.1.0.1
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov
 Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.3.0, 1.1.3

 Attachments: HBASE-14196.patch


 When idle connection get cleaned from Thrift server, underlying table 
 instances are still cached in a thread local cache.
 This is the antipattern. Table objects are lightweight and should not be 
 cached, besides this, underlying connections can be closed by periodic 
 connection cleaner chore (ConnectionCache) and cached table instances may 
 become invalid. This is Thrift1 specific issue. Thrift2 is OK.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94

2015-08-12 Thread Sean Busbey (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-14213:

Summary: Ensure ASF policy compliant headers and correct LICENSE and NOTICE 
files in artifacts for 0.94  (was: Backport HBASE-14085 Correct LICENSE and 
NOTICE files in artifacts to 0.94)

 Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in 
 artifacts for 0.94
 --

 Key: HBASE-14213
 URL: https://issues.apache.org/jira/browse/HBASE-14213
 Project: HBase
  Issue Type: Task
  Components: build
Reporter: Nick Dimiduk
Assignee: Sean Busbey
Priority: Blocker
 Fix For: 0.94.28


 From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took 
 the liberty of assigning to [~busbey].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Work started] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94

2015-08-12 Thread Sean Busbey (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HBASE-14213 started by Sean Busbey.
---
 Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in 
 artifacts for 0.94
 --

 Key: HBASE-14213
 URL: https://issues.apache.org/jira/browse/HBASE-14213
 Project: HBase
  Issue Type: Task
  Components: build
Reporter: Nick Dimiduk
Assignee: Sean Busbey
Priority: Blocker
 Fix For: 0.94.28


 From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took 
 the liberty of assigning to [~busbey].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14098) Allow dropping caches behind compactions

2015-08-12 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694649#comment-14694649
 ] 

Hudson commented on HBASE-14098:


FAILURE: Integrated in HBase-TRUNK #6722 (See 
[https://builds.apache.org/job/HBase-TRUNK/6722/])
HBASE-14098 Allow dropping caches behind compactions (eclark: rev 
a399ac9c4f76dccebb3476c418aac68667dee545)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/DefaultCompactor.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/StripeCompactor.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMobStoreCompaction.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestStripeCompactionPolicy.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/mob/compactions/TestPartitionedMobCompactor.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileInfo.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/io/FSDataInputStreamWrapper.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobFile.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/mob/compactions/PartitionedMobCompactor.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StripeStoreFlusher.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStripeCompactor.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFlusher.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java


 Allow dropping caches behind compactions
 

 Key: HBASE-14098
 URL: https://issues.apache.org/jira/browse/HBASE-14098
 Project: HBase
  Issue Type: Bug
  Components: Compaction, hadoop2, HFile
Affects Versions: 2.0.0, 1.3.0
Reporter: Elliott Clark
Assignee: Elliott Clark
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-14098-v1.patch, HBASE-14098-v2.patch, 
 HBASE-14098-v3.patch, HBASE-14098-v4.patch, HBASE-14098-v5.patch, 
 HBASE-14098-v6.patch, HBASE-14098-v7-branch-1.patch, HBASE-14098-v7.patch, 
 HBASE-14098.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14190) Assign system tables ahead of user region assignment

2015-08-12 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694697#comment-14694697
 ] 

Hadoop QA commented on HBASE-14190:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12750213/14190-v12.txt
  against master branch at commit d7178aa27c55ae06cfbd6db94d6839311dfb4089.
  ATTACHMENT ID: 12750213

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 9 new 
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 1 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   
org.apache.hadoop.hbase.replication.TestReplicationDisableInactivePeer
  org.apache.hadoop.hbase.master.TestRollingRestart
  
org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures
  org.apache.hadoop.hbase.namespace.TestNamespaceAuditor
  org.apache.hadoop.hbase.TestAcidGuarantees
  org.apache.hadoop.hbase.replication.TestReplicationSyncUpTool

 {color:red}-1 core zombie tests{color}.  There are 11 zombie test(s):  
at 
org.apache.hadoop.hbase.client.TestHTablePool$TestHTableReusablePool.testCloseTablePool(TestHTablePool.java:263)
at 
org.apache.hadoop.hbase.master.TestMasterOperationsForRegionReplicas.testCreateTableWithMultipleReplicas(TestMasterOperationsForRegionReplicas.java:228)
at 
org.apache.hadoop.hbase.client.TestTimestampsFilter.testWithVersionDeletes(TestTimestampsFilter.java:235)
at 
org.apache.hadoop.hbase.client.TestTimestampsFilter.testWithVersionDeletes(TestTimestampsFilter.java:222)
at 
org.apache.hadoop.hbase.mapred.TestTableInputFormat.testInputFormat(TestTableInputFormat.java:354)
at 
org.apache.hadoop.hbase.mapred.TestTableInputFormat.testExtensionOfTableInputFormatBase(TestTableInputFormat.java:326)
at 
org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite.testStoreFileCacheOnWriteInternals(TestCacheOnWrite.java:270)
at 
org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite.testStoreFileCacheOnWrite(TestCacheOnWrite.java:486)
at 
org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite.testNotCachingDataBlocksDuringCompactionInternals(TestCacheOnWrite.java:489)
at 
org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite.testNotCachingDataBlocksDuringCompaction(TestCacheOnWrite.java:507)
at 
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClient.testRestoreSnapshot(TestRestoreSnapshotFromClient.java:158)

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15079//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15079//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15079//artifact/patchprocess/checkstyle-aggregate.html

  Javadoc warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15079//artifact/patchprocess/patchJavadocWarnings.txt
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15079//console

This message is automatically generated.

 Assign system tables ahead of user region assignment
 

 Key: HBASE-14190
 URL: https://issues.apache.org/jira/browse/HBASE-14190
 Project: HBase
  Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
 Attachments: 14190-v12.txt, 14190-v6.txt, 14190-v7.txt, 14190-v8.txt


 Currently the namespace table region is assigned like user regions.
 I spent several hours working with a customer where master couldn't finish 
 initialization.
 Even though master was restarted 

[jira] [Updated] (HBASE-13480) ShortCircuitConnection doesn't short-circuit all calls as expected

2015-08-12 Thread Josh Elser (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-13480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser updated HBASE-13480:
---
Assignee: Jingcheng Du  (was: Josh Elser)

 ShortCircuitConnection doesn't short-circuit all calls as expected
 --

 Key: HBASE-13480
 URL: https://issues.apache.org/jira/browse/HBASE-13480
 Project: HBase
  Issue Type: Bug
  Components: Client
Affects Versions: 1.0.0, 2.0.0, 1.1.0
Reporter: Josh Elser
Assignee: Jingcheng Du
 Fix For: 2.0.0, 1.3.0, 1.2.1, 1.0.3, 1.1.3


 Noticed the following situation in debugging unexpected unit tests failures 
 in HBASE-13351.
 {{ConnectionUtils#createShortCircuitHConnection(Connection, ServerName, 
 AdminService.BlockingInterface, ClientService.BlockingInterface)}} is 
 intended to avoid the extra RPC by calling the server's instantiation of the 
 protobuf rpc stub directly for the AdminService and ClientService.
 The problem is that this is insufficient to actually avoid extra remote 
 RPCs as all other calls to the Connection are routed to a real Connection 
 instance. As such, any object created by the real Connection (such as an 
 HTable) will use the real Connection, not the SSC.
 The end result is that 
 {{MasterRpcService#reportRegionStateTransition(RpcController, 
 ReportRegionStateTransitionRequest)}} will make additional remote RPCs over 
 what it thinks is an SSC through a {{Get}} on {{HTable}} which was 
 constructed using the SSC, but the {{Get}} itself will use the underlying 
 real Connection instead of the SSC. With insufficiently sized thread pools, 
 this has been observed to result in RPC deadlock in the HMaster where an RPC 
 attempts to make another RPC but there are no more threads available to 
 service the second RPC so the first RPC blocks indefinitely.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-13480) ShortCircuitConnection doesn't short-circuit all calls as expected

2015-08-12 Thread Josh Elser (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-13480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694724#comment-14694724
 ] 

Josh Elser commented on HBASE-13480:


All yours, [~jingcheng...@intel.com]. I haven't found the time to work on it. 
I'm more than happy to pass it off to you.

 ShortCircuitConnection doesn't short-circuit all calls as expected
 --

 Key: HBASE-13480
 URL: https://issues.apache.org/jira/browse/HBASE-13480
 Project: HBase
  Issue Type: Bug
  Components: Client
Affects Versions: 1.0.0, 2.0.0, 1.1.0
Reporter: Josh Elser
Assignee: Jingcheng Du
 Fix For: 2.0.0, 1.3.0, 1.2.1, 1.0.3, 1.1.3


 Noticed the following situation in debugging unexpected unit tests failures 
 in HBASE-13351.
 {{ConnectionUtils#createShortCircuitHConnection(Connection, ServerName, 
 AdminService.BlockingInterface, ClientService.BlockingInterface)}} is 
 intended to avoid the extra RPC by calling the server's instantiation of the 
 protobuf rpc stub directly for the AdminService and ClientService.
 The problem is that this is insufficient to actually avoid extra remote 
 RPCs as all other calls to the Connection are routed to a real Connection 
 instance. As such, any object created by the real Connection (such as an 
 HTable) will use the real Connection, not the SSC.
 The end result is that 
 {{MasterRpcService#reportRegionStateTransition(RpcController, 
 ReportRegionStateTransitionRequest)}} will make additional remote RPCs over 
 what it thinks is an SSC through a {{Get}} on {{HTable}} which was 
 constructed using the SSC, but the {{Get}} itself will use the underlying 
 real Connection instead of the SSC. With insufficiently sized thread pools, 
 this has been observed to result in RPC deadlock in the HMaster where an RPC 
 attempts to make another RPC but there are no more threads available to 
 service the second RPC so the first RPC blocks indefinitely.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Resolved] (HBASE-13985) Add configuration to skip validating HFile format when bulk loading

2015-08-12 Thread Andrew Purtell (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell resolved HBASE-13985.

   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 1.3.0
   1.2.0
   2.0.0

Pushed to 0.98, branch-1.2, branch-1, and master. Bulk load units pass on all 
branches.

 Add configuration to skip validating HFile format when bulk loading
 ---

 Key: HBASE-13985
 URL: https://issues.apache.org/jira/browse/HBASE-13985
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.98.13
Reporter: Victor Xu
Assignee: Victor Xu
Priority: Minor
  Labels: regionserver
 Fix For: 2.0.0, 0.98.14, 1.2.0, 1.3.0

 Attachments: HBASE-13985-v2.patch, HBASE-13985-v3.patch, 
 HBASE-13985.patch


 When bulk loading millions of HFile into one HTable, checking HFile format is 
 the most time-consuming phase. Maybe we could use a parallel mechanism to 
 increase the speed, but when it comes to millions of HFiles, it may still 
 cost dozens of minutes. So I think it's necessary to add an option for 
 advanced user to bulkload without checking HFile format at all. 
 Of course, the default value of this option should be true.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-14203) remove duplicate code getTableDescriptor in HTable

2015-08-12 Thread Heng Chen (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-14203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Heng Chen updated HBASE-14203:
--
Attachment: HBASE-14203_v4.patch

testcase success in my local environment.
retry!

 remove duplicate code getTableDescriptor in HTable
 --

 Key: HBASE-14203
 URL: https://issues.apache.org/jira/browse/HBASE-14203
 Project: HBase
  Issue Type: Improvement
Reporter: Heng Chen
Priority: Trivial
 Attachments: HBASE-14203.patch, HBASE-14203_v2.patch, 
 HBASE-14203_v3.patch, HBASE-14203_v4.patch


 As TODO in comment said, 
 {{HTable.getTableDescriptor}} is same as {{HAdmin.getTableDescriptor}}. 
 remove the duplicate code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-14098) Allow dropping caches behind compactions

2015-08-12 Thread Elliott Clark (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-14098:
--
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Pushed to master, branch-1, and branch-1.2

 Allow dropping caches behind compactions
 

 Key: HBASE-14098
 URL: https://issues.apache.org/jira/browse/HBASE-14098
 Project: HBase
  Issue Type: Bug
  Components: Compaction, hadoop2, HFile
Affects Versions: 2.0.0, 1.3.0
Reporter: Elliott Clark
Assignee: Elliott Clark
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-14098-v1.patch, HBASE-14098-v2.patch, 
 HBASE-14098-v3.patch, HBASE-14098-v4.patch, HBASE-14098-v5.patch, 
 HBASE-14098-v6.patch, HBASE-14098-v7-branch-1.patch, HBASE-14098-v7.patch, 
 HBASE-14098.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14122) Client API for determining if server side supports cell level security

2015-08-12 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694630#comment-14694630
 ] 

Hudson commented on HBASE-14122:


FAILURE: Integrated in HBase-1.3 #105 (See 
[https://builds.apache.org/job/HBase-1.3/105/])
HBASE-14122 Client API for determining if server side supports cell level 
security (apurtell: rev ba4cc6321b9ed209393dfbc01d2c802062bb7437)
* hbase-protocol/src/main/protobuf/Master.proto
* hbase-shell/src/main/ruby/hbase/security.rb
* hbase-shell/src/main/ruby/shell.rb
* hbase-shell/src/main/ruby/hbase/visibility_labels.rb
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlClient.java
* hbase-shell/src/main/ruby/shell/commands/list_security_capabilities.rb
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabels.java
* 
hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/MasterProtos.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityClient.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
* hbase-shell/src/main/ruby/hbase/admin.rb
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityController.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClusterConnection.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/security/SecurityCapability.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java


 Client API for determining if server side supports cell level security
 --

 Key: HBASE-14122
 URL: https://issues.apache.org/jira/browse/HBASE-14122
 Project: HBase
  Issue Type: Improvement
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
 Fix For: 2.0.0, 0.98.14, 1.2.0, 1.3.0

 Attachments: HBASE-14122-0.98.patch, HBASE-14122-0.98.patch, 
 HBASE-14122-branch-1.patch, HBASE-14122-branch-1.patch, HBASE-14122.patch, 
 HBASE-14122.patch, HBASE-14122.patch


 Add a client API for determining if the server side supports cell level 
 security. 
 Ask the master, assuming as we do in many other instances that the master and 
 regionservers all have a consistent view of site configuration.
 Return {{true}} if all features required for cell level security are present, 
 {{false}} otherwise, or throw {{UnsupportedOperationException}} if the master 
 does not have support for the RPC call.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14122) Client API for determining if server side supports cell level security

2015-08-12 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694640#comment-14694640
 ] 

Hudson commented on HBASE-14122:


FAILURE: Integrated in HBase-TRUNK #6721 (See 
[https://builds.apache.org/job/HBase-TRUNK/6721/])
HBASE-14122 Client API for determining if server side supports cell level 
security (apurtell: rev 5e5bcceb533e6e4ded65bc778f05da213c07b688)
* hbase-shell/src/main/ruby/shell/commands/list_security_capabilities.rb
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
* hbase-protocol/src/main/protobuf/Master.proto
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java
* hbase-shell/src/main/ruby/hbase/visibility_labels.rb
* 
hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/MasterProtos.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlClient.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClusterConnection.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityClient.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabels.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityController.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
* hbase-shell/src/main/ruby/hbase/admin.rb
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/security/SecurityCapability.java
* hbase-shell/src/main/ruby/hbase/security.rb
* hbase-shell/src/main/ruby/shell.rb
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java


 Client API for determining if server side supports cell level security
 --

 Key: HBASE-14122
 URL: https://issues.apache.org/jira/browse/HBASE-14122
 Project: HBase
  Issue Type: Improvement
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
 Fix For: 2.0.0, 0.98.14, 1.2.0, 1.3.0

 Attachments: HBASE-14122-0.98.patch, HBASE-14122-0.98.patch, 
 HBASE-14122-branch-1.patch, HBASE-14122-branch-1.patch, HBASE-14122.patch, 
 HBASE-14122.patch, HBASE-14122.patch


 Add a client API for determining if the server side supports cell level 
 security. 
 Ask the master, assuming as we do in many other instances that the master and 
 regionservers all have a consistent view of site configuration.
 Return {{true}} if all features required for cell level security are present, 
 {{false}} otherwise, or throw {{UnsupportedOperationException}} if the master 
 does not have support for the RPC call.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14196) Thrift server idle connection timeout issue

2015-08-12 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694639#comment-14694639
 ] 

Hudson commented on HBASE-14196:


FAILURE: Integrated in HBase-TRUNK #6721 (See 
[https://builds.apache.org/job/HBase-TRUNK/6721/])
HBASE-14196 Thrift server idle connection timeout issue (Vladimir Rodionov) 
(apurtell: rev 643ba90185f20419016080d6d32adba9fe7019dd)
* 
hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/IncrementCoalescer.java
* 
hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java


 Thrift server idle connection timeout issue
 ---

 Key: HBASE-14196
 URL: https://issues.apache.org/jira/browse/HBASE-14196
 Project: HBase
  Issue Type: Bug
  Components: Thrift
Affects Versions: 0.98.13, 1.1.1, 1.0.1.1, 1.1.0.1
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov
 Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.3.0, 1.1.3

 Attachments: HBASE-14196.patch


 When idle connection get cleaned from Thrift server, underlying table 
 instances are still cached in a thread local cache.
 This is the antipattern. Table objects are lightweight and should not be 
 cached, besides this, underlying connections can be closed by periodic 
 connection cleaner chore (ConnectionCache) and cached table instances may 
 become invalid. This is Thrift1 specific issue. Thrift2 is OK.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-14144) Bloomfilter path to work with Byte buffered cells

2015-08-12 Thread ramkrishna.s.vasudevan (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-14144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-14144:
---
Status: Patch Available  (was: Open)

 Bloomfilter path to work with Byte buffered cells
 -

 Key: HBASE-14144
 URL: https://issues.apache.org/jira/browse/HBASE-14144
 Project: HBase
  Issue Type: Sub-task
  Components: regionserver, Scanners
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 2.0.0

 Attachments: HBASE-14144.patch, HBASE-14144_1.patch, 
 HBASE-14144_2.patch, HBASE-14144_3.patch


 This JIRA is to check if there will be a need to make the bloom filters to 
 work with ByteBuffer cells. During POC this path created lot of duplicated 
 code but considering other refactorings done in this path  may lead to less 
 duplication. This JIRA is a placeholder.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-14144) Bloomfilter path to work with Byte buffered cells

2015-08-12 Thread ramkrishna.s.vasudevan (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-14144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-14144:
---
Attachment: HBASE-14144_3.patch

Updated patch. Changes all the fake cells with 'column' to 'col'. The 
ByteBufferedCell is changed to BBCell to make it small.
The term fake represents that we try to create the first on row or last on row 
type of Cells using the existing cell's components. Previously these type of 
Kvs where created by copying the contents every time and now as we are having 
the concept of cell such copies are not required. Anyway these things are just 
used for doing sort of seek and reseeks and they are not going to be returned 
to the clients.

 Bloomfilter path to work with Byte buffered cells
 -

 Key: HBASE-14144
 URL: https://issues.apache.org/jira/browse/HBASE-14144
 Project: HBase
  Issue Type: Sub-task
  Components: regionserver, Scanners
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 2.0.0

 Attachments: HBASE-14144.patch, HBASE-14144_1.patch, 
 HBASE-14144_2.patch, HBASE-14144_3.patch


 This JIRA is to check if there will be a need to make the bloom filters to 
 work with ByteBuffer cells. During POC this path created lot of duplicated 
 code but considering other refactorings done in this path  may lead to less 
 duplication. This JIRA is a placeholder.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-14190) Assign system tables ahead of user region assignment

2015-08-12 Thread Ted Yu (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-14190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14190:
---
Attachment: 14190-v12.txt

I have been working on making TestRestartCluster pass.
This patch is work in progress.

 Assign system tables ahead of user region assignment
 

 Key: HBASE-14190
 URL: https://issues.apache.org/jira/browse/HBASE-14190
 Project: HBase
  Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
 Attachments: 14190-v12.txt, 14190-v6.txt, 14190-v7.txt, 14190-v8.txt


 Currently the namespace table region is assigned like user regions.
 I spent several hours working with a customer where master couldn't finish 
 initialization.
 Even though master was restarted quite a few times, it went down with the 
 following:
 {code}
 2015-08-05 17:16:57,530 FATAL [hdpmaster1:6.activeMasterManager] 
 master.HMaster: Master server abort: loaded coprocessors are: []
 2015-08-05 17:16:57,530 FATAL [hdpmaster1:6.activeMasterManager] 
 master.HMaster: Unhandled exception. Starting shutdown.
 java.io.IOException: Timedout 30ms waiting for namespace table to be 
 assigned
   at 
 org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:104)
   at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:985)
   at 
 org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:779)
   at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182)
   at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646)
   at java.lang.Thread.run(Thread.java:744)
 {code}
 During previous run(s), namespace table was created, hence leaving an entry 
 in hbase:meta.
 The following if block in TableNamespaceManager#start() was skipped:
 {code}
 if (!MetaTableAccessor.tableExists(masterServices.getConnection(),
   TableName.NAMESPACE_TABLE_NAME)) {
 {code}
 TableNamespaceManager#start() spins, waiting for namespace region to be 
 assigned.
 There was issue in master assigning user regions.
 We tried issuing 'assign' command from hbase shell which didn't work because 
 of the following check in MasterRpcServices#assignRegion():
 {code}
   master.checkInitialized();
 {code}
 This scenario can be avoided if we assign hbase:namespace table after 
 hbase:meta is assigned but before user table region assignment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HBASE-14190) Assign system tables ahead of user region assignment

2015-08-12 Thread Ted Yu (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-14190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14190:
---
Status: Open  (was: Patch Available)

 Assign system tables ahead of user region assignment
 

 Key: HBASE-14190
 URL: https://issues.apache.org/jira/browse/HBASE-14190
 Project: HBase
  Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
 Attachments: 14190-v6.txt, 14190-v7.txt, 14190-v8.txt


 Currently the namespace table region is assigned like user regions.
 I spent several hours working with a customer where master couldn't finish 
 initialization.
 Even though master was restarted quite a few times, it went down with the 
 following:
 {code}
 2015-08-05 17:16:57,530 FATAL [hdpmaster1:6.activeMasterManager] 
 master.HMaster: Master server abort: loaded coprocessors are: []
 2015-08-05 17:16:57,530 FATAL [hdpmaster1:6.activeMasterManager] 
 master.HMaster: Unhandled exception. Starting shutdown.
 java.io.IOException: Timedout 30ms waiting for namespace table to be 
 assigned
   at 
 org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:104)
   at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:985)
   at 
 org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:779)
   at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182)
   at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646)
   at java.lang.Thread.run(Thread.java:744)
 {code}
 During previous run(s), namespace table was created, hence leaving an entry 
 in hbase:meta.
 The following if block in TableNamespaceManager#start() was skipped:
 {code}
 if (!MetaTableAccessor.tableExists(masterServices.getConnection(),
   TableName.NAMESPACE_TABLE_NAME)) {
 {code}
 TableNamespaceManager#start() spins, waiting for namespace region to be 
 assigned.
 There was issue in master assigning user regions.
 We tried issuing 'assign' command from hbase shell which didn't work because 
 of the following check in MasterRpcServices#assignRegion():
 {code}
   master.checkInitialized();
 {code}
 This scenario can be avoided if we assign hbase:namespace table after 
 hbase:meta is assigned but before user table region assignment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-14122) Client API for determining if server side supports cell level security

2015-08-12 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694581#comment-14694581
 ] 

Hudson commented on HBASE-14122:


SUCCESS: Integrated in HBase-0.98 #1078 (See 
[https://builds.apache.org/job/HBase-0.98/1078/])
HBASE-14122 Client API for determining if server side supports cell level 
security (apurtell: rev 87729ccfacd8684fab141d34ab4776ae7d5c2d79)
* hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlClient.java
* hbase-protocol/src/main/protobuf/Master.proto
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityController.java
* hbase-shell/src/main/ruby/shell/commands/list_security_capabilities.rb
* hbase-shell/src/main/ruby/hbase/visibility_labels.rb
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabels.java
* hbase-shell/src/main/ruby/hbase/admin.rb
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityClient.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
* hbase-shell/src/main/ruby/hbase/security.rb
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
* hbase-shell/src/main/ruby/shell.rb
* 
hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/MasterProtos.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/security/SecurityCapability.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java


 Client API for determining if server side supports cell level security
 --

 Key: HBASE-14122
 URL: https://issues.apache.org/jira/browse/HBASE-14122
 Project: HBase
  Issue Type: Improvement
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
 Fix For: 2.0.0, 0.98.14, 1.2.0, 1.3.0

 Attachments: HBASE-14122-0.98.patch, HBASE-14122-0.98.patch, 
 HBASE-14122-branch-1.patch, HBASE-14122-branch-1.patch, HBASE-14122.patch, 
 HBASE-14122.patch, HBASE-14122.patch


 Add a client API for determining if the server side supports cell level 
 security. 
 Ask the master, assuming as we do in many other instances that the master and 
 regionservers all have a consistent view of site configuration.
 Return {{true}} if all features required for cell level security are present, 
 {{false}} otherwise, or throw {{UnsupportedOperationException}} if the master 
 does not have support for the RPC call.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-13907) Document how to deploy a coprocessor

2015-08-12 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-13907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694599#comment-14694599
 ] 

Hadoop QA commented on HBASE-13907:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12750189/HBASE-13907-v4.patch
  against master branch at commit 5e5bcceb533e6e4ded65bc778f05da213c07b688.
  ATTACHMENT ID: 12750189

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+0 tests included{color}.  The patch appears to be a 
documentation patch that doesn't require tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 1 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15075//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15075//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15075//artifact/patchprocess/checkstyle-aggregate.html

  Javadoc warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15075//artifact/patchprocess/patchJavadocWarnings.txt
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15075//console

This message is automatically generated.

 Document how to deploy a coprocessor
 

 Key: HBASE-13907
 URL: https://issues.apache.org/jira/browse/HBASE-13907
 Project: HBase
  Issue Type: Bug
  Components: documentation
Reporter: Misty Stanley-Jones
Assignee: Misty Stanley-Jones
 Attachments: HBASE-13907-1.patch, HBASE-13907-2.patch, 
 HBASE-13907-v3.patch, HBASE-13907-v4.patch, HBASE-13907.patch


 Capture this information:
  Where are the dependencies located for these classes? Is there a path on 
  HDFS or local disk that dependencies need to be placed so that each 
  RegionServer has access to them?
 It is suggested to bundle them as a single jar so that RS can load the whole 
 jar and resolve dependencies. If you are not able to do that, you need place 
 the dependencies in regionservers class path so that they are loaded during 
 RS startup. Do either of these options work for you? Btw, you can load the 
 coprocessors/filters into path specified by hbase.dynamic.jars.dir [1], so 
 that they are loaded dynamically by regionservers when the class is accessed 
 (or you can place them in the RS class path too, so that they are loaded 
 during RS JVM startup).
  How would one deploy these using an automated system? 
  (puppet/chef/ansible/etc)
 You can probably use these tools to automate shipping the jars to above 
 locations?
  Tests our developers have done suggest that simply disabling a coprocessor, 
  replacing the jar with a different version, and enabling the coprocessor 
  again does not load the newest version. With that in mind how does one know 
  which version is currently deployed and enabled without resorting to 
  parsing `hbase shell` output or restarting hbase?
 Actually this is a design issue with current classloader. You can't reload a 
 class in a JVM unless you delete all the current references to it. Since the 
 current JVM (classloader) has reference to it, you can't overwrite it unless 
 you kill the JVM, which is equivalent to restarting it. So you still have the 
 older class loaded in place. For this to work, classloader design should be 
 changed. If it works for you, you can rename the coprocessor class name and 
 the new version of jar and RS loads it properly.
  Where does logging go, and how does one access it? Does logging need to be 
  configured in a certain way?
 Can you please specify which logging you are referring to?
  Where is a good location to place configuration files?
 Same as above, are 

[jira] [Commented] (HBASE-13376) Improvements to Stochastic load balancer

2015-08-12 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694645#comment-14694645
 ] 

Hadoop QA commented on HBASE-13376:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12750202/HBASE-13376_4.patch
  against master branch at commit 643ba90185f20419016080d6d32adba9fe7019dd.
  ATTACHMENT ID: 12750202

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 9 new 
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 1 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   org.apache.hadoop.hbase.util.TestProcessBasedCluster
  org.apache.hadoop.hbase.mapreduce.TestImportExport

 {color:red}-1 core zombie tests{color}.  There are 1 zombie test(s): 

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15077//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15077//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15077//artifact/patchprocess/checkstyle-aggregate.html

  Javadoc warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15077//artifact/patchprocess/patchJavadocWarnings.txt
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15077//console

This message is automatically generated.

 Improvements to Stochastic load balancer
 

 Key: HBASE-13376
 URL: https://issues.apache.org/jira/browse/HBASE-13376
 Project: HBase
  Issue Type: Improvement
  Components: Balancer
Affects Versions: 1.0.0, 0.98.12
Reporter: Vandana Ayyalasomayajula
Assignee: Vandana Ayyalasomayajula
Priority: Minor
 Attachments: 13376-v2.txt, HBASE-13376.patch, HBASE-13376_0.98.txt, 
 HBASE-13376_0.txt, HBASE-13376_1.txt, HBASE-13376_1_1.txt, 
 HBASE-13376_2.patch, HBASE-13376_2_branch-1.patch, HBASE-13376_3.patch, 
 HBASE-13376_3.patch, HBASE-13376_4.patch, HBASE-13376_98.patch, 
 HBASE-13376_branch-1.patch


 There are two things this jira tries to address:
 1. The locality picker in the stochastic balancer does not pick regions with 
 least locality as candidates for swap/move. So when any user configures 
 locality cost in the configs, the balancer does not always seems to move 
 regions with bad locality. 
 2. When a cluster has equal number of loaded regions, it always picks the 
 first one. It should pick a random region on one of the equally loaded 
 servers. This improves a chance of finding a good candidate, when load picker 
 is invoked several times. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


  1   2   >