date:20150227


[ 
https://issues.apache.org/jira/browse/HBASE-13120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341244#comment-14341244
 ] 

Hudson commented on HBASE-13120:


FAILURE: Integrated in HBase-1.1 #223 (See 
[https://builds.apache.org/job/HBase-1.1/223/])
HBASE-13120 Allow disabling hadoop classpath and native library lookup 
(Siddharth Wagle) (enis: rev 287d08447f5de0fd2d7d5c8bc15d7221b1be8c5e)
* bin/hbase


 Allow disabling hadoop classpath and native library lookup
 --

 Key: HBASE-13120
 URL: https://issues.apache.org/jira/browse/HBASE-13120
 Project: HBase
  Issue Type: Improvement
  Components: hbase
Affects Versions: 0.98.0
Reporter: Siddharth Wagle
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-13120-1.patch, HBASE-13120.patch


 - Current bin/hbase script sets the java.library.path to include hadoop 
 native libs based on what version of hadoop is installed on the box.
 {code}
 HADOOP_IN_PATH=$(PATH=${HADOOP_HOME:-${HADOOP_PREFIX}}/bin:$PATH which 
 hadoop 2/dev/null)
 {code}
 - This effectively means that a self-contained HBase running with a different 
 version of embedded hadoop jars will fail to work in case of version 
 incompatibilities.
 - Example: AMBARI-5707, runs a embedded HBase on local FS side-by-side with 
 cluster Hadoop installation.
 - It would be good to have a hbase-env variable to completely override native 
 lib path or a config to disable native lib path lookup, in which case user 
 has to provide it during start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13097) Use same EventLoopGroup for different AsyncRpcClients if possible


[ 
https://issues.apache.org/jira/browse/HBASE-13097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341243#comment-14341243
 ] 

Hudson commented on HBASE-13097:


FAILURE: Integrated in HBase-1.1 #223 (See 
[https://builds.apache.org/job/HBase-1.1/223/])
HBASE-13097 addendum use default ByteBufAllocator (stack: rev 
95d290bc5ebd789c24eda1793266b5976520527a)
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AsyncRpcClient.java


 Use same EventLoopGroup for different AsyncRpcClients if possible
 -

 Key: HBASE-13097
 URL: https://issues.apache.org/jira/browse/HBASE-13097
 Project: HBase
  Issue Type: Bug
  Components: IPC/RPC, test
Affects Versions: 2.0.0, 1.1.0
Reporter: zhangduo
Assignee: zhangduo
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-13097-addendum.patch, HBASE-13097.patch, 
 HBASE-13097_1.patch, HBASE-13097_2.patch


 In some unit tests(such as TestAcidGuarantees) we create multiple Connection 
 instance. If we use AsyncRpcClient, then there will be multiple netty 
 Bootstrap and every Bootstrap has its own PooledByteBufAllocator.
 I haven't read the code clearly but it uses some threadlocal technics and 
 jmap shows io.netty.buffer.PoolThreadCache$MemoryRegionCache$Entry is the 
 biggest things on Heap.
 See 
 https://builds.apache.org/job/HBase-TRUNK/6168/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.TestAcidGuarantees-output.txt
 {noformat}
 2015-02-24 23:50:29,704 WARN  [JvmPauseMonitor] 
 util.JvmPauseMonitor$Monitor(167): Detected pause in JVM or host machine (eg 
 GC): pause of approximately 20133ms
 GC pool 'PS MarkSweep' had collection(s): count=15 time=55525ms
 {noformat}
 Update: We use a singleton PooledByteBufAllocator so the reason should be too 
 many threads. So we will work on reduce the connections and rpclients in unit 
 tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13122) Improve efficiency for return codes of some filters


[ 
https://issues.apache.org/jira/browse/HBASE-13122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341257#comment-14341257
 ] 

Ted Yu commented on HBASE-13122:


lgtm

Test failure was not related.

 Improve efficiency for return codes of some filters
 ---

 Key: HBASE-13122
 URL: https://issues.apache.org/jira/browse/HBASE-13122
 Project: HBase
  Issue Type: Improvement
  Components: Filters
Affects Versions: 0.94.24, 1.0.1, 0.98.10.1
Reporter: Shuaifeng Zhou
 Attachments: 13122-master.patch, 13122.patch


 ColumnRangeFilter:
  when minColumnInclusive is false, it means all the cells at the current 
 rowcolumn not fit the condition, so it should skip to next column, return 
 code should be NEXT_COL, not SKIP.
 FamilyFilter is the similar sitution.
 Currently, SKIP will not causing error, but not efficent.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13115) Fixing the usage of remote user in thrift doAs implementation.

[
https://issues.apache.org/jira/browse/HBASE-13115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341345#comment-14341345
]

Hadoop QA commented on HBASE-13115:
---

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12701533/HBASE-13115.patch
against master branch at commit bec2b0d320554b0af8c891fddc147a953f35765f.
ATTACHMENT ID: 12701533

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)

{color:red}-1 javac{color}. The applied patch generated 113 javac compiler
warnings (more than the master's current 111 warnings).

{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.

{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.

{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors

{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.

{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100

{color:green}+1 site{color}. The mvn site goal succeeds with this patch.

{color:red}-1 core tests{color}. The patch failed these unit tests:

org.apache.hadoop.hbase.regionserver.compactions.TestCompactionWithThroughputController

{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.coprocessor.TestMasterObserver.testRegionTransitionOperations(TestMasterObserver.java:1604)

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13010//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13010//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13010//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13010//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13010//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13010//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13010//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13010//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13010//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13010//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13010//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13010//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13010//artifact/patchprocess/checkstyle-aggregate.html

Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13010//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13010//console

This message is automatically generated.

Fixing the usage of remote user in thrift doAs implementation.
--

Key: HBASE-13115
URL: https://issues.apache.org/jira/browse/HBASE-13115
Project: HBase
Issue Type: Bug
Reporter: Srikanth Srungarapu
Assignee: Srikanth Srungarapu
Attachments: HBASE-13115.patch, HBASE-13115.patch

This issue is similar to HBASE-13085 fixed in REST gateway recently. This
change factors in the usage of remote user. Besides this, I also added the
following changes.
* Adding response headers to strictly adhere to

[jira] [Commented] (HBASE-13084) Add labels to VisibilityLabelsCache asynchronously causes TestShell flakey


[ 
https://issues.apache.org/jira/browse/HBASE-13084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341362#comment-14341362
 ] 

Hudson commented on HBASE-13084:


SUCCESS: Integrated in HBase-1.1 #225 (See 
[https://builds.apache.org/job/HBase-1.1/225/])
HBASE-13084 addendum disable info server in shell test (stack: rev 
3862b30a479fe6d30301dc27c9d3fa973c3f90c1)
* 
hbase-shell/src/test/java/org/apache/hadoop/hbase/client/AbstractTestShell.java


 Add labels to VisibilityLabelsCache asynchronously causes TestShell flakey
 --

 Key: HBASE-13084
 URL: https://issues.apache.org/jira/browse/HBASE-13084
 Project: HBase
  Issue Type: Bug
  Components: test
Reporter: zhangduo
Assignee: zhangduo
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-13084-addendum.patch, HBASE-13084-addendum2.patch, 
 HBASE-13084.patch, HBASE-13084_1.patch, HBASE-13084_2.patch, 
 HBASE-13084_2.patch, HBASE-13084_2.patch, HBASE-13084_2.patch, 
 HBASE-13084_2_disable_test.patch


 As discussed in HBASE-12953, we found this error in PreCommit log
 https://builds.apache.org/job/PreCommit-HBASE-Build/12918/artifact/hbase-shell/target/surefire-reports/org.apache.hadoop.hbase.client.TestShell-output.txt
 {noformat}
   1) Error:
 test_The_get/put_methods_should_work_for_data_written_with_Visibility(Hbase::VisibilityLabelsAdminMethodsTest):
 ArgumentError: org.apache.hadoop.hbase.DoNotRetryIOException: 
 org.apache.hadoop.hbase.security.visibility.InvalidLabelException: Label 
 'TEST_VISIBILITY' doesn't exists
   at 
 org.apache.hadoop.hbase.security.visibility.VisibilityController.setAuths(VisibilityController.java:808)
   at 
 org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService$1.setAuths(VisibilityLabelsProtos.java:6036)
   at 
 org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService.callMethod(VisibilityLabelsProtos.java:6219)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:6867)
   at 
 org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1707)
   at 
 org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1689)
   at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31309)
   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2038)
   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
   at 
 org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
   at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
   at java.lang.Thread.run(Thread.java:744)
 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-shell/src/main/ruby/hbase/visibility_labels.rb:84:in
  `set_auths'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:77:in 
 `test_The_get/put_methods_should_work_for_data_written_with_Visibility'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
   2) Error:
 test_The_set/clear_methods_should_work_with_authorizations(Hbase::VisibilityLabelsAdminMethodsTest):
 ArgumentError: No authentication set for the given user jenkins
 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-shell/src/main/ruby/hbase/visibility_labels.rb:97:in
  `get_auths'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:57:in 
 `test_The_set/clear_methods_should_work_with_authorizations'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
 {noformat}
 This is the test code
 {code:title=visibility_labels_admin_test.rb}
   label = 'TEST_VISIBILITY'
   user = org.apache.hadoop.hbase.security.User.getCurrent().getName();
   visibility_admin.add_labels(label)
   visibility_admin.set_auths(user, label)
 {code}
 It says 'label does not exists' when calling set_auths.
 Then I add some ugly logs in  DefaultVisibilityLabelServiceImpl and 
 VisibilityLabelsCache.
 {code:title=DefaultVisibilityLabelServiceImpl.java}
   public OperationStatus[] addLabels(Listbyte[] labels) throws IOException {
 ...
 if (mutateLabelsRegion(puts, finalOpStatus)) {
   updateZk(true);
 }
 for (byte[] label : labels) {
   String labelStr = Bytes.toString(label);
   LOG.info(labelStr + = + 
 this.labelsCache.getLabelOrdinal(labelStr));
 }
 ...
   }
 {code}
 {code:title=VisibilityLabelsCache.java}
   public void refreshLabelsCache(byte[] data) throws IOException {

[jira] [Commented] (HBASE-12311) Version stats in HFiles?

[
https://issues.apache.org/jira/browse/HBASE-12311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341363#comment-14341363
]

Lars Hofhansl commented on HBASE-12311:
---

I've thought of another approach. StorefileScanners have the notion of the
next indexed key, that is next known key to seek to (i.e. beginning of a
block). What if we took the next indexed key of the scanner that is on top of
the heap and only issue a seek if we would seek past that key? It's only a
heuristic and that check would not come free, but assuming it likely that
chunks of the Cells will come from the same file, we'd have a fairly good
indicator whether the seek will help. I have a 0.98 patch for that, and it
improves things. As an example I've used a range with the timerange. If the
range is before all Cells (except one so that the files isn't ruled out) it's
takes about 3.1s (we SKIP in that case) if the timerange fall after all Cells
(again except one) it 10.2s (we're seeking this time).

With the patch the first case is unchanged (3.1s), but the 2nd case it reduced
to 4.5s, since can avoid the unnecessary in many cases.

Version stats in HFiles?

Key: HBASE-12311
URL: https://issues.apache.org/jira/browse/HBASE-12311
Project: HBase
Issue Type: Brainstorming
Reporter: Lars Hofhansl
Attachments: 12311-v2.txt, 12311-v3.txt, 12311.txt,
CellStatTracker.java

In HBASE-9778 I basically punted the decision on whether doing repeated
scanner.next() called instead of the issueing (re)seeks to the user.
I think we can do better.
One way do that is maintain simple stats of what the maximum number of
versions we've seen for any row/col combination and store these in the
HFile's metadata (just like the timerange, oldest Put, etc).
Then we estimate fairly accurately whether we have to expect lots of versions
(i.e. seek between columns is better) or not (in which case we'd issue
repeated next()'s).

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13087) branch-1 isn't rolling upgradable from 0.98


[ 
https://issues.apache.org/jira/browse/HBASE-13087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341377#comment-14341377
 ] 

Lars Hofhansl commented on HBASE-13087:
---

Seems very reasonable to me put a change into 0.98 now to enable a rolling 
upgrade to 1.1.0 (I assume we have to do the same for 1.0.x).


 branch-1 isn't rolling upgradable from 0.98
 ---

 Key: HBASE-13087
 URL: https://issues.apache.org/jira/browse/HBASE-13087
 Project: HBase
  Issue Type: Bug
Reporter: Elliott Clark
Assignee: Rajesh Nishtala
Priority: Blocker
 Fix For: 2.0.0, 1.1.0


 {code}org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: 
 Failed 1 action: 
 org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column 
 family table does not exist in region hbase:meta,,1.1588230740 in table 
 'hbase:meta', {TABLE_ATTRIBUTES = {IS_META = 'true', coprocessor$1 = 
 '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, 
 {NAME = 'info', BLOOMFILTER = 'NONE', VERSIONS = '10', IN_MEMORY = 
 'true', KEEP_DELETED_CELLS = 'FALSE', DATA_BLOCK_ENCODING = 'NONE', TTL = 
 'FOREVER', COMPRESSION = 'NONE', MIN_VERSIONS = '0', BLOCKCACHE = 'true', 
 BLOCKSIZE = '8192', REPLICATION_SCOPE = '0'}
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4513)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3687)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3576)
   at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:30816)
   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2029)
   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
   at 
 org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
   at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
   at java.lang.Thread.run(Thread.java:745)
 : 1 time, 
   at 
 org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:228)
   at 
 org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(AsyncProcess.java:208)
   at 
 org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1689)
   at 
 org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
   at 
 org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
   at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1404)
   at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1017)
   at 
 org.apache.hadoop.hbase.MetaTableAccessor.put(MetaTableAccessor.java:1123)
   at 
 org.apache.hadoop.hbase.MetaTableAccessor.putToMetaTable(MetaTableAccessor.java:1113)
   at 
 org.apache.hadoop.hbase.MetaTableAccessor.updateTableState(MetaTableAccessor.java:1436)
   at 
 org.apache.hadoop.hbase.MetaTableAccessor.updateTableState(MetaTableAccessor.java:948)
   at 
 org.apache.hadoop.hbase.master.TableStateManager.writeMetaState(TableStateManager.java:195)
   at 
 org.apache.hadoop.hbase.master.TableStateManager.setTableState(TableStateManager.java:69)
   at 
 org.apache.hadoop.hbase.master.AssignmentManager.setEnabledTable(AssignmentManager.java:3427)
   at org.apache.hadoop.hbase.master.HMaster.assignMeta(HMaster.java:903)
   at 
 org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:698)
   at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:166)
   at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1494)
   at java.lang.Thread.run(Thread.java:745)
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13084) Add labels to VisibilityLabelsCache asynchronously causes TestShell flakey


 [ 
https://issues.apache.org/jira/browse/HBASE-13084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhangduo updated HBASE-13084:
-
Attachment: HBASE-13084-addendum2.patch

Disable info server when running shell tests.

InfoServer is disabled in src/test/resources/hbase-site.xml when running 
hbase-server unit tests, but hbase-shell does not have a src/test/resources, so 
disable it manually.

 Add labels to VisibilityLabelsCache asynchronously causes TestShell flakey
 --

 Key: HBASE-13084
 URL: https://issues.apache.org/jira/browse/HBASE-13084
 Project: HBase
  Issue Type: Bug
  Components: test
Reporter: zhangduo
Assignee: zhangduo
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-13084-addendum.patch, HBASE-13084-addendum2.patch, 
 HBASE-13084.patch, HBASE-13084_1.patch, HBASE-13084_2.patch, 
 HBASE-13084_2.patch, HBASE-13084_2.patch, HBASE-13084_2.patch, 
 HBASE-13084_2_disable_test.patch


 As discussed in HBASE-12953, we found this error in PreCommit log
 https://builds.apache.org/job/PreCommit-HBASE-Build/12918/artifact/hbase-shell/target/surefire-reports/org.apache.hadoop.hbase.client.TestShell-output.txt
 {noformat}
   1) Error:
 test_The_get/put_methods_should_work_for_data_written_with_Visibility(Hbase::VisibilityLabelsAdminMethodsTest):
 ArgumentError: org.apache.hadoop.hbase.DoNotRetryIOException: 
 org.apache.hadoop.hbase.security.visibility.InvalidLabelException: Label 
 'TEST_VISIBILITY' doesn't exists
   at 
 org.apache.hadoop.hbase.security.visibility.VisibilityController.setAuths(VisibilityController.java:808)
   at 
 org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService$1.setAuths(VisibilityLabelsProtos.java:6036)
   at 
 org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService.callMethod(VisibilityLabelsProtos.java:6219)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:6867)
   at 
 org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1707)
   at 
 org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1689)
   at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31309)
   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2038)
   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
   at 
 org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
   at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
   at java.lang.Thread.run(Thread.java:744)
 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-shell/src/main/ruby/hbase/visibility_labels.rb:84:in
  `set_auths'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:77:in 
 `test_The_get/put_methods_should_work_for_data_written_with_Visibility'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
   2) Error:
 test_The_set/clear_methods_should_work_with_authorizations(Hbase::VisibilityLabelsAdminMethodsTest):
 ArgumentError: No authentication set for the given user jenkins
 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-shell/src/main/ruby/hbase/visibility_labels.rb:97:in
  `get_auths'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:57:in 
 `test_The_set/clear_methods_should_work_with_authorizations'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
 {noformat}
 This is the test code
 {code:title=visibility_labels_admin_test.rb}
   label = 'TEST_VISIBILITY'
   user = org.apache.hadoop.hbase.security.User.getCurrent().getName();
   visibility_admin.add_labels(label)
   visibility_admin.set_auths(user, label)
 {code}
 It says 'label does not exists' when calling set_auths.
 Then I add some ugly logs in  DefaultVisibilityLabelServiceImpl and 
 VisibilityLabelsCache.
 {code:title=DefaultVisibilityLabelServiceImpl.java}
   public OperationStatus[] addLabels(Listbyte[] labels) throws IOException {
 ...
 if (mutateLabelsRegion(puts, finalOpStatus)) {
   updateZk(true);
 }
 for (byte[] label : labels) {
   String labelStr = Bytes.toString(label);
   LOG.info(labelStr + = + 
 this.labelsCache.getLabelOrdinal(labelStr));
 }
 ...
   }
 {code}
 {code:title=VisibilityLabelsCache.java}
   public void refreshLabelsCache(byte[] data) throws IOException {
 LOG.info(refresh, new Exception());
 ...
   }
 {code}
 And I

[jira] [Commented] (HBASE-13115) Fixing the usage of remote user in thrift doAs implementation.


[ 
https://issues.apache.org/jira/browse/HBASE-13115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341351#comment-14341351
 ] 

Ted Yu commented on HBASE-13115:


Here're the new javac warnings:
{code}
[WARNING] 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-examples/src/main/java/org/apache/hadoop/hbase/thrift/HttpDoAsClient.java:[21,16]
 sun.misc.   BASE64Encoder is internal proprietary API and may be 
removed in a future release
[WARNING] 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-examples/src/main/java/org/apache/hadoop/hbase/thrift/HttpDoAsClient.java:[222,29]
 sun.misc.  BASE64Encoder is internal proprietary API and may be 
removed in a future release
{code}
But BASE64Encoder has already been used:
{code}
[WARNING] 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftHttpServlet.java:[21,16]
 sun.misc.  BASE64Encoder is internal proprietary API and may be 
removed in a future release
{code}
Here is the javadoc warning:
{code}
[WARNING] 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftHttpServlet.java:21:
 warning: BASE64Encoder is internal proprietary API and may be removed in a 
future release
{code}

 Fixing the usage of remote user in thrift doAs implementation.
 --

 Key: HBASE-13115
 URL: https://issues.apache.org/jira/browse/HBASE-13115
 Project: HBase
  Issue Type: Bug
Reporter: Srikanth Srungarapu
Assignee: Srikanth Srungarapu
 Attachments: HBASE-13115.patch, HBASE-13115.patch


 This issue is similar to HBASE-13085 fixed in REST gateway recently. This 
 change factors in the usage of remote user. Besides this, I also added the 
 following changes.
 * Adding response headers to strictly adhere to RFC specifcations.
 * Made changes to demo client to get rid of hard codings.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-13115) Fixing the usage of remote user in thrift doAs implementation.


[ 
https://issues.apache.org/jira/browse/HBASE-13115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341351#comment-14341351
 ] 

Ted Yu edited comment on HBASE-13115 at 2/28/15 5:22 AM:
-

Here're the new javac warnings:
{code}
[WARNING] 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-examples/src/main/java/org/apache/hadoop/hbase/thrift/HttpDoAsClient.java:[21,16]
 sun.misc.   BASE64Encoder is internal proprietary API and may be 
removed in a future release
[WARNING] 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-examples/src/main/java/org/apache/hadoop/hbase/thrift/HttpDoAsClient.java:[222,29]
 sun.misc.  BASE64Encoder is internal proprietary API and may be 
removed in a future release
{code}
Here is the javadoc warning:
{code}
[WARNING] 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftHttpServlet.java:21:
 warning: BASE64Encoder is internal proprietary API and may be removed in a 
future release
{code}


was (Author: yuzhih...@gmail.com):
Here're the new javac warnings:
{code}
[WARNING] 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-examples/src/main/java/org/apache/hadoop/hbase/thrift/HttpDoAsClient.java:[21,16]
 sun.misc.   BASE64Encoder is internal proprietary API and may be 
removed in a future release
[WARNING] 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-examples/src/main/java/org/apache/hadoop/hbase/thrift/HttpDoAsClient.java:[222,29]
 sun.misc.  BASE64Encoder is internal proprietary API and may be 
removed in a future release
{code}
But BASE64Encoder has already been used:
{code}
[WARNING] 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftHttpServlet.java:[21,16]
 sun.misc.  BASE64Encoder is internal proprietary API and may be 
removed in a future release
{code}
Here is the javadoc warning:
{code}
[WARNING] 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftHttpServlet.java:21:
 warning: BASE64Encoder is internal proprietary API and may be removed in a 
future release
{code}

 Fixing the usage of remote user in thrift doAs implementation.
 --

 Key: HBASE-13115
 URL: https://issues.apache.org/jira/browse/HBASE-13115
 Project: HBase
  Issue Type: Bug
Reporter: Srikanth Srungarapu
Assignee: Srikanth Srungarapu
 Attachments: HBASE-13115.patch, HBASE-13115.patch


 This issue is similar to HBASE-13085 fixed in REST gateway recently. This 
 change factors in the usage of remote user. Besides this, I also added the 
 following changes.
 * Adding response headers to strictly adhere to RFC specifcations.
 * Made changes to demo client to get rid of hard codings.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13128) Make HBCK's lock file retry creation and deletion


[ 
https://issues.apache.org/jira/browse/HBASE-13128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341358#comment-14341358
 ] 

Ted Yu commented on HBASE-13128:


bq. working with the older version of hadoop
Which version of hadoop are you using ?
{code}
387   } catch (InterruptedException ie) {
388 throw (InterruptedIOException) new InterruptedIOException(
389 Can't create lock file).initCause(ie);
{code}
Please add path to lock file in exception message.


 Make HBCK's lock file retry creation and deletion
 -

 Key: HBASE-13128
 URL: https://issues.apache.org/jira/browse/HBASE-13128
 Project: HBase
  Issue Type: Improvement
  Components: hbck
Reporter: Victoria
Assignee: Victoria
Priority: Minor
 Attachments: hbck_files_fix.patch


 When hbck runs it creates a lock file to ensure that no two hbck instances 
 are running. We've been seeing creating and removing that file fail sometimes.
 This improvement should make the creation, closing of the file, and the 
 deletion retry multiple times. This should allow our alerting which uses this 
 command to be more reliable and have fewer false positives.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11544) [Ergonomics] hbase.client.scanner.caching is dogged and will try to return batch even if it means OOME


[ 
https://issues.apache.org/jira/browse/HBASE-11544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341378#comment-14341378
 ] 

stack commented on HBASE-11544:
---

Currently run cluster test to ensure no breakage and to look for perf 
degradation (hopefully we see opposite).

[~jonathan.lawlor] which of the tests in TestPartialResultsFromClientSide 
verifies that a client who is getting partial results has same view as a client 
who is asking for full rows at a time; i.e. which tests readpoint is not 
changed under a scanner that is returning a wide row in multiple partials? 
Thanks boss.

 [Ergonomics] hbase.client.scanner.caching is dogged and will try to return 
 batch even if it means OOME
 --

 Key: HBASE-11544
 URL: https://issues.apache.org/jira/browse/HBASE-11544
 Project: HBase
  Issue Type: Bug
Reporter: stack
Assignee: Jonathan Lawlor
Priority: Critical
  Labels: beginner
 Attachments: HBASE-11544-v1.patch, HBASE-11544-v2.patch, 
 HBASE-11544-v3.patch, HBASE-11544-v4.patch, HBASE-11544-v5.patch, 
 HBASE-11544-v6.patch, HBASE-11544-v6.patch, HBASE-11544-v6.patch


 Running some tests, I set hbase.client.scanner.caching=1000.  Dataset has 
 large cells.  I kept OOME'ing.
 Serverside, we should measure how much we've accumulated and return to the 
 client whatever we've gathered once we pass out a certain size threshold 
 rather than keep accumulating till we OOME.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-11571) Bulk load handling from secondary region replicas

2015-02-27 Thread Jeffrey Zhong (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-11571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeffrey Zhong updated HBASE-11571:
--
Attachment: HBASE-11571-rebase.patch

Rebase patch. [~enis] Please review it. Thanks.

 Bulk load handling from secondary region replicas
 -

 Key: HBASE-11571
 URL: https://issues.apache.org/jira/browse/HBASE-11571
 Project: HBase
  Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Jeffrey Zhong
 Attachments: HBASE-11571-rebase.patch, hbase-11571.patch


 We should be replaying the bulk load events from the primary region replica 
 in the secondary region replica so that the bulk loaded files will be made 
 visible in the secondaries. 
 This will depend on HBASE-11567 and HBASE-11568



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-11571) Bulk load handling from secondary region replicas

2015-02-27 Thread Jeffrey Zhong (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-11571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeffrey Zhong updated HBASE-11571:
--
Status: Patch Available  (was: Open)

 Bulk load handling from secondary region replicas
 -

 Key: HBASE-11571
 URL: https://issues.apache.org/jira/browse/HBASE-11571
 Project: HBase
  Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Jeffrey Zhong
 Attachments: HBASE-11571-rebase.patch, hbase-11571.patch


 We should be replaying the bulk load events from the primary region replica 
 in the secondary region replica so that the bulk loaded files will be made 
 visible in the secondaries. 
 This will depend on HBASE-11567 and HBASE-11568



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13115) Fixing the usage of remote user in thrift doAs implementation.


 [ 
https://issues.apache.org/jira/browse/HBASE-13115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-13115:
---
Attachment: HBASE-13115.patch

Re-attaching to see if warnings are re-producible.

 Fixing the usage of remote user in thrift doAs implementation.
 --

 Key: HBASE-13115
 URL: https://issues.apache.org/jira/browse/HBASE-13115
 Project: HBase
  Issue Type: Bug
Reporter: Srikanth Srungarapu
Assignee: Srikanth Srungarapu
 Attachments: HBASE-13115.patch, HBASE-13115.patch


 This issue is similar to HBASE-13085 fixed in REST gateway recently. This 
 change factors in the usage of remote user. Besides this, I also added the 
 following changes.
 * Adding response headers to strictly adhere to RFC specifcations.
 * Made changes to demo client to get rid of hard codings.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13128) Make HBCK's lock file retry creation and deletion

2015-02-27 Thread Victoria (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-13128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Victoria updated HBASE-13128:
-
Status: Patch Available  (was: Open)

Added retries for creation and deletion of the lock file.

 Make HBCK's lock file retry creation and deletion
 -

 Key: HBASE-13128
 URL: https://issues.apache.org/jira/browse/HBASE-13128
 Project: HBase
  Issue Type: Improvement
  Components: hbck
Reporter: Victoria
Assignee: Victoria
Priority: Minor

 When hbck runs it creates a lock file to ensure that no two hbck instances 
 are running. We've been seeing creating and removing that file fail sometimes.
 This improvement should make the creation, closing of the file, and the 
 deletion retry multiple times. This should allow our alerting which uses this 
 command to be more reliable and have fewer false positives.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12975) Supportable SplitTransaction and RegionMergeTransaction interfaces


[ 
https://issues.apache.org/jira/browse/HBASE-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341372#comment-14341372
 ] 

Lars Hofhansl commented on HBASE-12975:
---

Sorry a bit late here. Why can't we add a *new* stable interface in a 1.0.x 
release? Where's the compatibility issue?

 Supportable SplitTransaction and RegionMergeTransaction interfaces
 --

 Key: HBASE-12975
 URL: https://issues.apache.org/jira/browse/HBASE-12975
 Project: HBase
  Issue Type: Improvement
Reporter: Rajeshbabu Chintaguntla
Assignee: Andrew Purtell
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-12975.patch, HBASE-12975.patch


 Making SplitTransaction, RegionMergeTransaction limited private is required 
 to support local indexing feature in Phoenix to ensure regions colocation. 
 We can ensure region split, regions merge in the coprocessors in few method 
 calls without touching internals like creating zk's, file layout changes or 
 assignments.
 1) stepsBeforePONR, stepsAfterPONR we can ensure split.
 2) meta entries can pass through coprocessors to atomically update with the 
 normal split/merge.
 3) rollback on failure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13119) FileLink should implement equals

2015-02-27 Thread Enis Soztutar (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-13119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Enis Soztutar updated HBASE-13119:
--
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

I've committed this. Thanks for reviews. 

 FileLink should implement equals
 

 Key: HBASE-13119
 URL: https://issues.apache.org/jira/browse/HBASE-13119
 Project: HBase
  Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Fix For: 2.0.0, 1.1.0

 Attachments: hbase-13119_v1.patch


 StoreFileInfo used to implement Comparable, but got changed in HBASE-12749. 
 Now comparing StoreFileInfo's with links results in false, since 
 FileLink/HFileLink does not implement equals(). 
 We should do both equals and hashCode() it seems. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13119) FileLink should implement equals


[ 
https://issues.apache.org/jira/browse/HBASE-13119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341303#comment-14341303
 ] 

Hudson commented on HBASE-13119:


FAILURE: Integrated in HBase-1.1 #224 (See 
[https://builds.apache.org/job/HBase-1.1/224/])
HBASE-13119 FileLink should implement equals (enis: rev 
b09c4feade0c86c284c814322971c3b8e0279a5e)
* hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestFileLink.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileInfo.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/FileLink.java


 FileLink should implement equals
 

 Key: HBASE-13119
 URL: https://issues.apache.org/jira/browse/HBASE-13119
 Project: HBase
  Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Fix For: 2.0.0, 1.1.0

 Attachments: hbase-13119_v1.patch


 StoreFileInfo used to implement Comparable, but got changed in HBASE-12749. 
 Now comparing StoreFileInfo's with links results in false, since 
 FileLink/HFileLink does not implement equals(). 
 We should do both equals and hashCode() it seems. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13119) FileLink should implement equals


[ 
https://issues.apache.org/jira/browse/HBASE-13119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341305#comment-14341305
 ] 

Hudson commented on HBASE-13119:


FAILURE: Integrated in HBase-TRUNK #6181 (See 
[https://builds.apache.org/job/HBase-TRUNK/6181/])
HBASE-13119 FileLink should implement equals (enis: rev 
bec2b0d320554b0af8c891fddc147a953f35765f)
* hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestFileLink.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileInfo.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/FileLink.java


 FileLink should implement equals
 

 Key: HBASE-13119
 URL: https://issues.apache.org/jira/browse/HBASE-13119
 Project: HBase
  Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Fix For: 2.0.0, 1.1.0

 Attachments: hbase-13119_v1.patch


 StoreFileInfo used to implement Comparable, but got changed in HBASE-12749. 
 Now comparing StoreFileInfo's with links results in false, since 
 FileLink/HFileLink does not implement equals(). 
 We should do both equals and hashCode() it seems. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13097) Use same EventLoopGroup for different AsyncRpcClients if possible


[ 
https://issues.apache.org/jira/browse/HBASE-13097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341306#comment-14341306
 ] 

Hudson commented on HBASE-13097:


FAILURE: Integrated in HBase-TRUNK #6181 (See 
[https://builds.apache.org/job/HBase-TRUNK/6181/])
HBASE-13097 addendum use default ByteBufAllocator (stack: rev 
21f12ce8e5e6bc33a858f4586fe48c44b28f4a1b)
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AsyncRpcClient.java


 Use same EventLoopGroup for different AsyncRpcClients if possible
 -

 Key: HBASE-13097
 URL: https://issues.apache.org/jira/browse/HBASE-13097
 Project: HBase
  Issue Type: Bug
  Components: IPC/RPC, test
Affects Versions: 2.0.0, 1.1.0
Reporter: zhangduo
Assignee: zhangduo
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-13097-addendum.patch, HBASE-13097.patch, 
 HBASE-13097_1.patch, HBASE-13097_2.patch


 In some unit tests(such as TestAcidGuarantees) we create multiple Connection 
 instance. If we use AsyncRpcClient, then there will be multiple netty 
 Bootstrap and every Bootstrap has its own PooledByteBufAllocator.
 I haven't read the code clearly but it uses some threadlocal technics and 
 jmap shows io.netty.buffer.PoolThreadCache$MemoryRegionCache$Entry is the 
 biggest things on Heap.
 See 
 https://builds.apache.org/job/HBase-TRUNK/6168/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.TestAcidGuarantees-output.txt
 {noformat}
 2015-02-24 23:50:29,704 WARN  [JvmPauseMonitor] 
 util.JvmPauseMonitor$Monitor(167): Detected pause in JVM or host machine (eg 
 GC): pause of approximately 20133ms
 GC pool 'PS MarkSweep' had collection(s): count=15 time=55525ms
 {noformat}
 Update: We use a singleton PooledByteBufAllocator so the reason should be too 
 many threads. So we will work on reduce the connections and rpclients in unit 
 tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13128) Make HBCK's lock file retry creation and deletion

2015-02-27 Thread Victoria (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-13128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Victoria updated HBASE-13128:
-
Attachment: hbck_files_fix.patch

I have added retries for both creation and deletion of the lock file.
The default number of attempts is set to 5 with the initial sleep time between 
the attempts set to 200 msec.


 Make HBCK's lock file retry creation and deletion
 -

 Key: HBASE-13128
 URL: https://issues.apache.org/jira/browse/HBASE-13128
 Project: HBase
  Issue Type: Improvement
  Components: hbck
Reporter: Victoria
Assignee: Victoria
Priority: Minor
 Attachments: hbck_files_fix.patch


 When hbck runs it creates a lock file to ensure that no two hbck instances 
 are running. We've been seeing creating and removing that file fail sometimes.
 This improvement should make the creation, closing of the file, and the 
 deletion retry multiple times. This should allow our alerting which uses this 
 command to be more reliable and have fewer false positives.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12975) Supportable SplitTransaction and RegionMergeTransaction interfaces


[ 
https://issues.apache.org/jira/browse/HBASE-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341164#comment-14341164
 ] 

Andrew Purtell commented on HBASE-12975:


Here's what I have for SplitTransaction. RegionMergeTransaction will be 
similar, differing in ways you can infer. 
{code:java}
/**
 * Executes region split as a transaction.  Call {@link #prepare()} to setup
 * the transaction, {@link #execute(Server, RegionServerServices)} to run the
 * transaction and {@link #rollback(Server, RegionServerServices)} to cleanup 
if execute fails.
 *
 * pHere is an example of how you would use this interface:
 * pre
 *  SplitTransactionFactory factory = new SplitTransactionFactory(conf);
 *  SplitTransaction st = factory.create(parent, midKey)
 *.registerTransactionListener(new TransactionListener() {
 *   public void transition(SplitTransaction transaction, 
SplitTransactionPhase from,
 *   SplitTransactionPhase to) throws IOException {
 * // ...
 *   }
 *   public void rollback(SplitTransaction transaction, 
SplitTransactionPhase from,
 *   SplitTransactionPhase to) {
 * // ...
 *   }
 *});
 *  if (!st.prepare()) return;
 *  try {
 *st.execute(server, services);
 *  } catch (IOException e) {
 *try {
 *  st.rollback(server, services);
 *  return;
 *} catch (RuntimeException e) {
 *  // abort the server
 *}
 *  }
 * /Pre
 * pA split transaction is not thread safe.  Callers must ensure a split is 
run by
 * one thread only.
 */
@InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.COPROC)
@InterfaceStability.Evolving
public interface SplitTransaction {

  /**
   * Each enum is a step in the split transaction.
   */
  public enum SplitTransactionPhase {
/**
 * Started
 */
STARTED,
/**
 * Prepared (after table lock)
 */
PREPARED,
/**
 * Before preSplit coprocessor hook
 */
BEFORE_PRE_SPLIT_HOOK,
/**
 * After preSplit coprocessor hook
 */
AFTER_PRE_SPLIT_HOOK,
/**
 * Set region as in transition, set it into SPLITTING state.
 */
SET_SPLITTING,
/**
 * We created the temporary split data directory.
 */
CREATE_SPLIT_DIR,
/**
 * Closed the parent region.
 */
CLOSED_PARENT_REGION,
/**
 * The parent has been taken out of the server's online regions list.
 */
OFFLINED_PARENT,
/**
 * Started in on creation of the first daughter region.
 */
STARTED_REGION_A_CREATION,
/**
 * Started in on the creation of the second daughter region.
 */
STARTED_REGION_B_CREATION,
/**
 * Opened the first daughter region
 */
OPENED_REGION_A,
/**
 * Opened the second daughter region
 */
OPENED_REGION_B,
/**
 * Point of no return.
 * If we got here, then transaction is not recoverable other than by
 * crashing out the regionserver.
 */
PONR,
/**
 * Before postSplit coprocessor hook
 */
BEFORE_POST_SPLIT_HOOK,
/**
 * After postSplit coprocessor hook
 */
AFTER_POST_SPLIT_HOOK,
/**
 * Completed
 */
COMPLETED
  }

  /**
   * Split transaction journal entry
   */
  public interface JournalEntry {

/** @return the completed phase marked by this journal entry */
SplitTransactionPhase getPhase();

/** @return the time of phase completion */
long getTimeStamp();
  }

  /**
   * Split transaction listener
   */
  public interface TransactionListener {

/**
 * Invoked when transitioning forward from one transaction phase to another
 * @param transaction the transaction
 * @param from the current phase
 * @param to the next phase
 * @throws IOException listener can throw this to abort
 */
void transition(SplitTransaction transaction, SplitTransactionPhase from,
SplitTransactionPhase to) throws IOException;

/**
 * Invoked when rolling back a transaction from one transaction phase to the
 * previous
 * @param transaction the transaction
 * @param from the current phase
 * @param to the previous phase
 */
void rollback(SplitTransaction transaction, SplitTransactionPhase from,
SplitTransactionPhase to);
  }

  /**
   * Check split inputs and prepare the transaction.
   * @return codetrue/code if the region is splittable else
   * codefalse/code if it is not (e.g. its already closed, etc.).
   * @throws IOException 
   */
  boolean prepare() throws IOException;

  /**
   * Run the transaction.
   * @param server Hosting server instance.  Can be null when testing.
   * @param services Used to online/offline regions.
   * @throws IOException If thrown, transaction failed.
   *  Call {@link #rollback(Server, RegionServerServices)}
   * @return Regions created
   * @throws IOException
   * @see #rollback(Server,

[jira] [Commented] (HBASE-12975) Supportable SplitTransaction and RegionMergeTransaction interfaces

2015-02-27 Thread Rajeshbabu Chintaguntla (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341253#comment-14341253
 ] 

Rajeshbabu Chintaguntla commented on HBASE-12975:
-

bq. Since I'm working on a patch I'm going to take this Rajeshbabu 
Chintaguntla, but please let me know if you plan on actively working on it 
right away, we can figure something out.
You can continue [~apurtell]. I can help you by parallely testing and let you 
know any pain points to make use for local indexes case in Phoenix.
Thanks.

 Supportable SplitTransaction and RegionMergeTransaction interfaces
 --

 Key: HBASE-12975
 URL: https://issues.apache.org/jira/browse/HBASE-12975
 Project: HBase
  Issue Type: Improvement
Reporter: Rajeshbabu Chintaguntla
Assignee: Andrew Purtell
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-12975.patch, HBASE-12975.patch


 Making SplitTransaction, RegionMergeTransaction limited private is required 
 to support local indexing feature in Phoenix to ensure regions colocation. 
 We can ensure region split, regions merge in the coprocessors in few method 
 calls without touching internals like creating zk's, file layout changes or 
 assignments.
 1) stepsBeforePONR, stepsAfterPONR we can ensure split.
 2) meta entries can pass through coprocessors to atomically update with the 
 normal split/merge.
 3) rollback on failure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13097) Use same EventLoopGroup for different AsyncRpcClients if possible

[
https://issues.apache.org/jira/browse/HBASE-13097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341269#comment-14341269
]

Hadoop QA commented on HBASE-13097:
---

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment

http://issues.apache.org/jira/secure/attachment/12701509/HBASE-13097-addendum.patch
against master branch at commit ec877959d7611ed8edfb15d67f5a662080c2e6da.
ATTACHMENT ID: 12701509

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.

{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.

{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors

{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.

{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100

{color:green}+1 site{color}. The mvn site goal succeeds with this patch.

{color:red}-1 core tests{color}. The patch failed these unit tests:

org.apache.hadoop.hbase.replication.TestReplicationKillSlaveRS

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13008//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13008//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13008//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13008//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13008//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13008//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13008//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13008//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13008//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13008//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13008//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13008//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13008//artifact/patchprocess/checkstyle-aggregate.html

Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13008//console

This message is automatically generated.

Use same EventLoopGroup for different AsyncRpcClients if possible
-

Key: HBASE-13097
URL: https://issues.apache.org/jira/browse/HBASE-13097
Project: HBase
Issue Type: Bug
Components: IPC/RPC, test
Affects Versions: 2.0.0, 1.1.0
Reporter: zhangduo
Assignee: zhangduo
Fix For: 2.0.0, 1.1.0

Attachments: HBASE-13097-addendum.patch, HBASE-13097.patch,
HBASE-13097_1.patch, HBASE-13097_2.patch

In some unit tests(such as TestAcidGuarantees) we create multiple Connection
instance. If we use AsyncRpcClient, then there will be multiple netty
Bootstrap and every Bootstrap has its own PooledByteBufAllocator.
I haven't read the code clearly but it uses some threadlocal technics and
jmap shows io.netty.buffer.PoolThreadCache$MemoryRegionCache$Entry is the
biggest things on Heap.
See

[jira] [Commented] (HBASE-12975) Supportable SplitTransaction and RegionMergeTransaction interfaces


[ 
https://issues.apache.org/jira/browse/HBASE-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341295#comment-14341295
 ] 

Hadoop QA commented on HBASE-12975:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12701528/HBASE-12975.patch
  against master branch at commit bec2b0d320554b0af8c891fddc147a953f35765f.
  ATTACHMENT ID: 12701528

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 36 new 
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.1 2.5.2 2.6.0)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 1 
warning messages.

{color:red}-1 checkstyle{color}.  The applied patch generated 
1941 checkstyle errors (more than the master's current 1938 errors).

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 lineLengths{color}.  The patch introduces the following lines 
longer than 100:
+  public static final String MERGE_TRANSACTION_IMPL_KEY = 
hbase.regionserver.merge.transaction.impl;
+  public RegionMergeTransactionImpl create(final HRegion a, final HRegion b, 
final boolean forcible) {
+  public static final String SPLIT_TRANSACTION_IMPL_KEY = 
hbase.regionserver.split.transaction.impl;
+expectedReferenceFileCount != 
FSUtils.getRegionReferenceFileCount(this.parent.getFilesystem(), dir)) {
+  private PairPath, Path splitStoreFile(final byte[] family, final StoreFile 
sf) throws IOException {
+  private SplitTransactionImpl prepareGOOD_SPLIT_ROW(final HRegion 
parentRegion) throws IOException {

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   org.apache.hadoop.hbase.regionserver.TestSplitTransaction
  org.apache.hadoop.hbase.coprocessor.TestCoprocessorInterface
  
org.apache.hadoop.hbase.regionserver.TestRegionMergeTransaction

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13009//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13009//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13009//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13009//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13009//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13009//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13009//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13009//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13009//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13009//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13009//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13009//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13009//artifact/patchprocess/checkstyle-aggregate.html

Javadoc warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13009//artifact/patchprocess/patchJavadocWarnings.txt
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13009//console

This message is automatically generated.

 Supportable SplitTransaction and RegionMergeTransaction interfaces
 --

 Key: HBASE-12975
 URL: https://issues.apache.org/jira/browse/HBASE-12975

[jira] [Commented] (HBASE-13084) Add labels to VisibilityLabelsCache asynchronously causes TestShell flakey


[ 
https://issues.apache.org/jira/browse/HBASE-13084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341373#comment-14341373
 ] 

stack commented on HBASE-13084:
---

[~Apache9] you are making progress https://builds.apache.org/job/HBase-1.1/225/

 Add labels to VisibilityLabelsCache asynchronously causes TestShell flakey
 --

 Key: HBASE-13084
 URL: https://issues.apache.org/jira/browse/HBASE-13084
 Project: HBase
  Issue Type: Bug
  Components: test
Reporter: zhangduo
Assignee: zhangduo
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-13084-addendum.patch, HBASE-13084-addendum2.patch, 
 HBASE-13084.patch, HBASE-13084_1.patch, HBASE-13084_2.patch, 
 HBASE-13084_2.patch, HBASE-13084_2.patch, HBASE-13084_2.patch, 
 HBASE-13084_2_disable_test.patch


 As discussed in HBASE-12953, we found this error in PreCommit log
 https://builds.apache.org/job/PreCommit-HBASE-Build/12918/artifact/hbase-shell/target/surefire-reports/org.apache.hadoop.hbase.client.TestShell-output.txt
 {noformat}
   1) Error:
 test_The_get/put_methods_should_work_for_data_written_with_Visibility(Hbase::VisibilityLabelsAdminMethodsTest):
 ArgumentError: org.apache.hadoop.hbase.DoNotRetryIOException: 
 org.apache.hadoop.hbase.security.visibility.InvalidLabelException: Label 
 'TEST_VISIBILITY' doesn't exists
   at 
 org.apache.hadoop.hbase.security.visibility.VisibilityController.setAuths(VisibilityController.java:808)
   at 
 org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService$1.setAuths(VisibilityLabelsProtos.java:6036)
   at 
 org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService.callMethod(VisibilityLabelsProtos.java:6219)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:6867)
   at 
 org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1707)
   at 
 org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1689)
   at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31309)
   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2038)
   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
   at 
 org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
   at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
   at java.lang.Thread.run(Thread.java:744)
 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-shell/src/main/ruby/hbase/visibility_labels.rb:84:in
  `set_auths'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:77:in 
 `test_The_get/put_methods_should_work_for_data_written_with_Visibility'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
   2) Error:
 test_The_set/clear_methods_should_work_with_authorizations(Hbase::VisibilityLabelsAdminMethodsTest):
 ArgumentError: No authentication set for the given user jenkins
 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-shell/src/main/ruby/hbase/visibility_labels.rb:97:in
  `get_auths'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:57:in 
 `test_The_set/clear_methods_should_work_with_authorizations'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
 {noformat}
 This is the test code
 {code:title=visibility_labels_admin_test.rb}
   label = 'TEST_VISIBILITY'
   user = org.apache.hadoop.hbase.security.User.getCurrent().getName();
   visibility_admin.add_labels(label)
   visibility_admin.set_auths(user, label)
 {code}
 It says 'label does not exists' when calling set_auths.
 Then I add some ugly logs in  DefaultVisibilityLabelServiceImpl and 
 VisibilityLabelsCache.
 {code:title=DefaultVisibilityLabelServiceImpl.java}
   public OperationStatus[] addLabels(Listbyte[] labels) throws IOException {
 ...
 if (mutateLabelsRegion(puts, finalOpStatus)) {
   updateZk(true);
 }
 for (byte[] label : labels) {
   String labelStr = Bytes.toString(label);
   LOG.info(labelStr + = + 
 this.labelsCache.getLabelOrdinal(labelStr));
 }
 ...
   }
 {code}
 {code:title=VisibilityLabelsCache.java}
   public void refreshLabelsCache(byte[] data) throws IOException {
 LOG.info(refresh, new Exception());
 ...
   }
 {code}
 And I modified TestVisibilityLabelsWithCustomVisLabService to use 
 DefaultVisibilityLabelServiceImpl, then collected the logs of

[jira] [Commented] (HBASE-12311) Version stats in HFiles?


[ 
https://issues.apache.org/jira/browse/HBASE-12311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341400#comment-14341400
 ] 

Lars Hofhansl commented on HBASE-12311:
---

Tested some more with KVs with 10, 100, and 1 versions. The more version we 
have the more likely it becomes that we would seek past the next indexed key of 
the top scanner in the heap and hence keep the seek. So this appears to work 
fine with few and many versions, as well as few and many columns, and all cases 
it will estimate whether a SKIP or a SEEK is better.


 Version stats in HFiles?
 

 Key: HBASE-12311
 URL: https://issues.apache.org/jira/browse/HBASE-12311
 Project: HBase
  Issue Type: Brainstorming
Reporter: Lars Hofhansl
 Attachments: 12311-indexed-0.98.txt, 12311-v2.txt, 12311-v3.txt, 
 12311.txt, CellStatTracker.java


 In HBASE-9778 I basically punted the decision on whether doing repeated 
 scanner.next() called instead of the issueing (re)seeks to the user.
 I think we can do better.
 One way do that is maintain simple stats of what the maximum number of 
 versions we've seen for any row/col combination and store these in the 
 HFile's metadata (just like the timerange, oldest Put, etc).
 Then we estimate fairly accurately whether we have to expect lots of versions 
 (i.e. seek between columns is better) or not (in which case we'd issue 
 repeated next()'s).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12311) Version stats in HFiles?


[ 
https://issues.apache.org/jira/browse/HBASE-12311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341401#comment-14341401
 ] 

Lars Hofhansl commented on HBASE-12311:
---

I might file a separate issue for this.

 Version stats in HFiles?
 

 Key: HBASE-12311
 URL: https://issues.apache.org/jira/browse/HBASE-12311
 Project: HBase
  Issue Type: Brainstorming
Reporter: Lars Hofhansl
 Attachments: 12311-indexed-0.98.txt, 12311-v2.txt, 12311-v3.txt, 
 12311.txt, CellStatTracker.java


 In HBASE-9778 I basically punted the decision on whether doing repeated 
 scanner.next() called instead of the issueing (re)seeks to the user.
 I think we can do better.
 One way do that is maintain simple stats of what the maximum number of 
 versions we've seen for any row/col combination and store these in the 
 HFile's metadata (just like the timerange, oldest Put, etc).
 Then we estimate fairly accurately whether we have to expect lots of versions 
 (i.e. seek between columns is better) or not (in which case we'd issue 
 repeated next()'s).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12975) Supportable SplitTransaction and RegionMergeTransaction interfaces


 [ 
https://issues.apache.org/jira/browse/HBASE-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-12975:
---
Attachment: HBASE-12975.patch

Dropping WIP patch. Not tested yet.

 Supportable SplitTransaction and RegionMergeTransaction interfaces
 --

 Key: HBASE-12975
 URL: https://issues.apache.org/jira/browse/HBASE-12975
 Project: HBase
  Issue Type: Improvement
Reporter: Rajeshbabu Chintaguntla
Assignee: Andrew Purtell
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-12975.patch, HBASE-12975.patch


 Making SplitTransaction, RegionMergeTransaction limited private is required 
 to support local indexing feature in Phoenix to ensure regions colocation. 
 We can ensure region split, regions merge in the coprocessors in few method 
 calls without touching internals like creating zk's, file layout changes or 
 assignments.
 1) stepsBeforePONR, stepsAfterPONR we can ensure split.
 2) meta entries can pass through coprocessors to atomically update with the 
 normal split/merge.
 3) rollback on failure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13120) Allow disabling hadoop classpath and native library lookup


[ 
https://issues.apache.org/jira/browse/HBASE-13120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341235#comment-14341235
 ] 

Hudson commented on HBASE-13120:


FAILURE: Integrated in HBase-TRUNK #6180 (See 
[https://builds.apache.org/job/HBase-TRUNK/6180/])
HBASE-13120 Allow disabling hadoop classpath and native library lookup 
(Siddharth Wagle) (enis: rev ec877959d7611ed8edfb15d67f5a662080c2e6da)
* bin/hbase


 Allow disabling hadoop classpath and native library lookup
 --

 Key: HBASE-13120
 URL: https://issues.apache.org/jira/browse/HBASE-13120
 Project: HBase
  Issue Type: Improvement
  Components: hbase
Affects Versions: 0.98.0
Reporter: Siddharth Wagle
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-13120-1.patch, HBASE-13120.patch


 - Current bin/hbase script sets the java.library.path to include hadoop 
 native libs based on what version of hadoop is installed on the box.
 {code}
 HADOOP_IN_PATH=$(PATH=${HADOOP_HOME:-${HADOOP_PREFIX}}/bin:$PATH which 
 hadoop 2/dev/null)
 {code}
 - This effectively means that a self-contained HBase running with a different 
 version of embedded hadoop jars will fail to work in case of version 
 incompatibilities.
 - Example: AMBARI-5707, runs a embedded HBase on local FS side-by-side with 
 cluster Hadoop installation.
 - It would be good to have a hbase-env variable to completely override native 
 lib path or a config to disable native lib path lookup, in which case user 
 has to provide it during start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13084) Add labels to VisibilityLabelsCache asynchronously causes TestShell flakey


[ 
https://issues.apache.org/jira/browse/HBASE-13084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341246#comment-14341246
 ] 

zhangduo commented on HBASE-13084:
--

https://builds.apache.org/job/HBase-TRUNK/6180/testReport/junit/org.apache.hadoop.hbase.client/TestReplicationShell/org_apache_hadoop_hbase_client_TestReplicationShell/

{noformat}
2015-02-28 02:28:36,085 WARN  [main] mortbay.log (Slf4jLog.java:warn(76)) - 
failed SelectChannelConnector@0.0.0.0:16010: java.net.BindException: Address 
already in use
2015-02-28 02:28:36,085 WARN  [main] mortbay.log (Slf4jLog.java:warn(76)) - 
failed Server@5de44bb4: java.net.BindException: Address already in use
2015-02-28 02:28:36,086 ERROR [main] hbase.MiniHBaseCluster 
(MiniHBaseCluster.java:init(229)) - Error starting cluster
java.lang.RuntimeException: Failed construction of Master: class 
org.apache.hadoop.hbase.master.HMasterAddress already in use
at 
org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:143)
at 
org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:216)
at 
org.apache.hadoop.hbase.LocalHBaseCluster.init(LocalHBaseCluster.java:154)
at 
org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:213)
at 
org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:93)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:996)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:956)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:830)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:824)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:768)
at 
org.apache.hadoop.hbase.client.AbstractTestShell.setUpBeforeClass(AbstractTestShell.java:50)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runners.Suite.runChild(Suite.java:127)
at org.junit.runners.Suite.runChild(Suite.java:26)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
at org.junit.runner.JUnitCore.run(JUnitCore.java:138)
at 
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:107)
at 
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:77)
at 
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:53)
at 
org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:144)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Caused by: java.io.IOException: Failed to start redirecting jetty server
at 
org.apache.hadoop.hbase.master.HMaster.putUpJettyServer(HMaster.java:428)
at org.apache.hadoop.hbase.master.HMaster.init(HMaster.java:391)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at

[jira] [Comment Edited] (HBASE-12975) Supportable SplitTransaction and RegionMergeTransaction interfaces


[ 
https://issues.apache.org/jira/browse/HBASE-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341239#comment-14341239
 ] 

Andrew Purtell edited comment on HBASE-12975 at 2/28/15 2:39 AM:
-

Dropping WIP patch. Not tested yet. Needs new tests for the transaction 
listeners too


was (Author: apurtell):
Dropping WIP patch. Not tested yet.

 Supportable SplitTransaction and RegionMergeTransaction interfaces
 --

 Key: HBASE-12975
 URL: https://issues.apache.org/jira/browse/HBASE-12975
 Project: HBase
  Issue Type: Improvement
Reporter: Rajeshbabu Chintaguntla
Assignee: Andrew Purtell
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-12975.patch, HBASE-12975.patch


 Making SplitTransaction, RegionMergeTransaction limited private is required 
 to support local indexing feature in Phoenix to ensure regions colocation. 
 We can ensure region split, regions merge in the coprocessors in few method 
 calls without touching internals like creating zk's, file layout changes or 
 assignments.
 1) stepsBeforePONR, stepsAfterPONR we can ensure split.
 2) meta entries can pass through coprocessors to atomically update with the 
 normal split/merge.
 3) rollback on failure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-11580) Failover handling for secondary region replicas

2015-02-27 Thread Enis Soztutar (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Enis Soztutar updated HBASE-11580:
--
Attachment: hbase-11580_v2.patch

Thanks Jeffrey for the review. Attaching updated patch from RB.

Failover handling for secondary region replicas
---

Key: HBASE-11580
URL: https://issues.apache.org/jira/browse/HBASE-11580
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Attachments: hbase-11580_v2.patch

With the async wal approach (HBASE-11568), the edits are not persisted (to
wal) in the secondary region replicas. However this means that we have to
deal with secondary region replica failures.
We can seek to re-replicate the edits from primary to the secondary when the
secondary region is opened in another server but this would mean to setup a
replication queue again, and holding on to the wals for longer.
Instead, we can design it so that the edits form the secondaries are not
persisted to wal, and if the secondary replica fails over, it will not start
serving reads until it has guaranteed that it has all the past data.
For guaranteeing that the secondary replica has all the edits before serving
reads, we can use flush and region opening markers. Whenever a region open
event is seen, it writes all the files at the time of opening to wal
(HBASE-11512). In case of flush, the flushed file is written as well, and the
secondary replica can do a ls for the store files and pick up all the files
before the seqId of the flushed file. So, in this design, the secodary
replica will wait until it sees and replays a flush or region open marker
from wal from primary. and then start serving. For speeding up replica
opening time, we can trigger a flush to the primary whenever the secondary
replica opens as an optimization.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-11580) Failover handling for secondary region replicas

2015-02-27 Thread Enis Soztutar (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Enis Soztutar updated HBASE-11580:
--
Status: Patch Available (was: Open)

Failover handling for secondary region replicas
---

Key: HBASE-11580
URL: https://issues.apache.org/jira/browse/HBASE-11580
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Attachments: hbase-11580_v2.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13084) Add labels to VisibilityLabelsCache asynchronously causes TestShell flakey

[
https://issues.apache.org/jira/browse/HBASE-13084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341354#comment-14341354
]

Hadoop QA commented on HBASE-13084:
---

{color:green}+1 overall{color}. Here are the results of testing the latest
attachment

http://issues.apache.org/jira/secure/attachment/12701536/HBASE-13084-addendum2.patch
against master branch at commit bec2b0d320554b0af8c891fddc147a953f35765f.
ATTACHMENT ID: 12701536

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)

{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.

{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.

{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors

{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.

{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100

{color:green}+1 site{color}. The mvn site goal succeeds with this patch.

{color:green}+1 core tests{color}. The patch passed unit tests in .

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13012//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13012//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13012//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13012//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13012//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13012//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13012//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13012//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13012//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13012//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13012//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13012//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13012//artifact/patchprocess/checkstyle-aggregate.html

Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13012//console

This message is automatically generated.

Add labels to VisibilityLabelsCache asynchronously causes TestShell flakey
--

Key: HBASE-13084
URL: https://issues.apache.org/jira/browse/HBASE-13084
Project: HBase
Issue Type: Bug
Components: test
Reporter: zhangduo
Assignee: zhangduo
Fix For: 2.0.0, 1.1.0

Attachments: HBASE-13084-addendum.patch, HBASE-13084-addendum2.patch,
HBASE-13084.patch, HBASE-13084_1.patch, HBASE-13084_2.patch,
HBASE-13084_2.patch, HBASE-13084_2.patch, HBASE-13084_2.patch,
HBASE-13084_2_disable_test.patch

As discussed in HBASE-12953, we found this error in PreCommit log
https://builds.apache.org/job/PreCommit-HBASE-Build/12918/artifact/hbase-shell/target/surefire-reports/org.apache.hadoop.hbase.client.TestShell-output.txt
{noformat}
1) Error:
test_The_get/put_methods_should_work_for_data_written_with_Visibility(Hbase::VisibilityLabelsAdminMethodsTest):
ArgumentError: org.apache.hadoop.hbase.DoNotRetryIOException:
org.apache.hadoop.hbase.security.visibility.InvalidLabelException: Label
'TEST_VISIBILITY' doesn't exists
at

[jira] [Comment Edited] (HBASE-12311) Version stats in HFiles?

[
https://issues.apache.org/jira/browse/HBASE-12311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341363#comment-14341363
]

Lars Hofhansl edited comment on HBASE-12311 at 2/28/15 6:03 AM:

I've thought of another approach. StorefileScanners have the notion of a next
indexed key, that is next known key to seek to (i.e. beginning of a block).
What if we took the next indexed key of the scanner that is on top of the
(StoreFileScanner/MemstoreScanner) heap and only issue a seek if we would seek
past that key? It's only a heuristic and that check would not come free, but
assuming it likely that chunks of Cells will come from the same file, we'd have
a fairly good indicator whether the seek will help. I have a 0.98 patch for
that, and it improves things. As an example I've used a scan with a timerange.
If the range is before all Cells (except one so that the files isn't ruled out)
it's takes about 3.1s (we SKIP in that case) if the timerange falls after all
Cells (again except one) it's 10.2s (we're seeking this time - see SQM.match).

With the patch the first case is unchanged (3.1s), but the 2nd case it reduced
to 4.5s, since can avoid the unnecessary seek in many cases.

was (Author: lhofhansl):
I've thought of another approach. StorefileScanners have the notion of the
next indexed key, that is next known key to seek to (i.e. beginning of a
block). What if we took the next indexed key of the scanner that is on top of
the heap and only issue a seek if we would seek past that key? It's only a
heuristic and that check would not come free, but assuming it likely that
chunks of the Cells will come from the same file, we'd have a fairly good
indicator whether the seek will help. I have a 0.98 patch for that, and it
improves things. As an example I've used a range with the timerange. If the
range is before all Cells (except one so that the files isn't ruled out) it's
takes about 3.1s (we SKIP in that case) if the timerange fall after all Cells
(again except one) it 10.2s (we're seeking this time).

With the patch the first case is unchanged (3.1s), but the 2nd case it reduced
to 4.5s, since can avoid the unnecessary in many cases.

Version stats in HFiles?

Key: HBASE-12311
URL: https://issues.apache.org/jira/browse/HBASE-12311
Project: HBase
Issue Type: Brainstorming
Reporter: Lars Hofhansl
Attachments: 12311-indexed-0.98.txt, 12311-v2.txt, 12311-v3.txt,
12311.txt, CellStatTracker.java

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13087) branch-1 isn't rolling upgradable from 0.98


[ 
https://issues.apache.org/jira/browse/HBASE-13087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341383#comment-14341383
 ] 

stack commented on HBASE-13087:
---

+1 on #1.

#2 would be against the usual upgrade grain and it would be missed by all but 
those paying close attention.

Thanks.

 branch-1 isn't rolling upgradable from 0.98
 ---

 Key: HBASE-13087
 URL: https://issues.apache.org/jira/browse/HBASE-13087
 Project: HBase
  Issue Type: Bug
Reporter: Elliott Clark
Assignee: Rajesh Nishtala
Priority: Blocker
 Fix For: 2.0.0, 1.1.0


 {code}org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: 
 Failed 1 action: 
 org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column 
 family table does not exist in region hbase:meta,,1.1588230740 in table 
 'hbase:meta', {TABLE_ATTRIBUTES = {IS_META = 'true', coprocessor$1 = 
 '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, 
 {NAME = 'info', BLOOMFILTER = 'NONE', VERSIONS = '10', IN_MEMORY = 
 'true', KEEP_DELETED_CELLS = 'FALSE', DATA_BLOCK_ENCODING = 'NONE', TTL = 
 'FOREVER', COMPRESSION = 'NONE', MIN_VERSIONS = '0', BLOCKCACHE = 'true', 
 BLOCKSIZE = '8192', REPLICATION_SCOPE = '0'}
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4513)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3687)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3576)
   at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:30816)
   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2029)
   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
   at 
 org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
   at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
   at java.lang.Thread.run(Thread.java:745)
 : 1 time, 
   at 
 org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:228)
   at 
 org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(AsyncProcess.java:208)
   at 
 org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1689)
   at 
 org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
   at 
 org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
   at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1404)
   at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1017)
   at 
 org.apache.hadoop.hbase.MetaTableAccessor.put(MetaTableAccessor.java:1123)
   at 
 org.apache.hadoop.hbase.MetaTableAccessor.putToMetaTable(MetaTableAccessor.java:1113)
   at 
 org.apache.hadoop.hbase.MetaTableAccessor.updateTableState(MetaTableAccessor.java:1436)
   at 
 org.apache.hadoop.hbase.MetaTableAccessor.updateTableState(MetaTableAccessor.java:948)
   at 
 org.apache.hadoop.hbase.master.TableStateManager.writeMetaState(TableStateManager.java:195)
   at 
 org.apache.hadoop.hbase.master.TableStateManager.setTableState(TableStateManager.java:69)
   at 
 org.apache.hadoop.hbase.master.AssignmentManager.setEnabledTable(AssignmentManager.java:3427)
   at org.apache.hadoop.hbase.master.HMaster.assignMeta(HMaster.java:903)
   at 
 org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:698)
   at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:166)
   at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1494)
   at java.lang.Thread.run(Thread.java:745)
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13084) Add labels to VisibilityLabelsCache asynchronously causes TestShell flakey


[ 
https://issues.apache.org/jira/browse/HBASE-13084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341307#comment-14341307
 ] 

stack commented on HBASE-13084:
---

Pushed addendum to branch-1+.  Thanks [~Apache9]


 Add labels to VisibilityLabelsCache asynchronously causes TestShell flakey
 --

 Key: HBASE-13084
 URL: https://issues.apache.org/jira/browse/HBASE-13084
 Project: HBase
  Issue Type: Bug
  Components: test
Reporter: zhangduo
Assignee: zhangduo
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-13084-addendum.patch, HBASE-13084-addendum2.patch, 
 HBASE-13084.patch, HBASE-13084_1.patch, HBASE-13084_2.patch, 
 HBASE-13084_2.patch, HBASE-13084_2.patch, HBASE-13084_2.patch, 
 HBASE-13084_2_disable_test.patch


 As discussed in HBASE-12953, we found this error in PreCommit log
 https://builds.apache.org/job/PreCommit-HBASE-Build/12918/artifact/hbase-shell/target/surefire-reports/org.apache.hadoop.hbase.client.TestShell-output.txt
 {noformat}
   1) Error:
 test_The_get/put_methods_should_work_for_data_written_with_Visibility(Hbase::VisibilityLabelsAdminMethodsTest):
 ArgumentError: org.apache.hadoop.hbase.DoNotRetryIOException: 
 org.apache.hadoop.hbase.security.visibility.InvalidLabelException: Label 
 'TEST_VISIBILITY' doesn't exists
   at 
 org.apache.hadoop.hbase.security.visibility.VisibilityController.setAuths(VisibilityController.java:808)
   at 
 org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService$1.setAuths(VisibilityLabelsProtos.java:6036)
   at 
 org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService.callMethod(VisibilityLabelsProtos.java:6219)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:6867)
   at 
 org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1707)
   at 
 org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1689)
   at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31309)
   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2038)
   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
   at 
 org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
   at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
   at java.lang.Thread.run(Thread.java:744)
 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-shell/src/main/ruby/hbase/visibility_labels.rb:84:in
  `set_auths'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:77:in 
 `test_The_get/put_methods_should_work_for_data_written_with_Visibility'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
   2) Error:
 test_The_set/clear_methods_should_work_with_authorizations(Hbase::VisibilityLabelsAdminMethodsTest):
 ArgumentError: No authentication set for the given user jenkins
 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-shell/src/main/ruby/hbase/visibility_labels.rb:97:in
  `get_auths'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:57:in 
 `test_The_set/clear_methods_should_work_with_authorizations'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
 {noformat}
 This is the test code
 {code:title=visibility_labels_admin_test.rb}
   label = 'TEST_VISIBILITY'
   user = org.apache.hadoop.hbase.security.User.getCurrent().getName();
   visibility_admin.add_labels(label)
   visibility_admin.set_auths(user, label)
 {code}
 It says 'label does not exists' when calling set_auths.
 Then I add some ugly logs in  DefaultVisibilityLabelServiceImpl and 
 VisibilityLabelsCache.
 {code:title=DefaultVisibilityLabelServiceImpl.java}
   public OperationStatus[] addLabels(Listbyte[] labels) throws IOException {
 ...
 if (mutateLabelsRegion(puts, finalOpStatus)) {
   updateZk(true);
 }
 for (byte[] label : labels) {
   String labelStr = Bytes.toString(label);
   LOG.info(labelStr + = + 
 this.labelsCache.getLabelOrdinal(labelStr));
 }
 ...
   }
 {code}
 {code:title=VisibilityLabelsCache.java}
   public void refreshLabelsCache(byte[] data) throws IOException {
 LOG.info(refresh, new Exception());
 ...
   }
 {code}
 And I modified TestVisibilityLabelsWithCustomVisLabService to use 
 DefaultVisibilityLabelServiceImpl, then collected the logs of setupBeforeClass
 {noformat}
 2015-02-21

[jira] [Comment Edited] (HBASE-13115) Fixing the usage of remote user in thrift doAs implementation.


[ 
https://issues.apache.org/jira/browse/HBASE-13115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341351#comment-14341351
 ] 

Ted Yu edited comment on HBASE-13115 at 2/28/15 5:24 AM:
-

Here're the new javac warnings:
{code}
[WARNING] 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftHttpServlet.java:[21,16]
 sun.misc.  BASE64Encoder is internal proprietary API and may be 
removed in a future release
[WARNING] 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftHttpServlet.java:[182,26]
 sun.misc. BASE64Encoder is internal proprietary API and may be removed 
in a future release
{code}
Here is the javadoc warning:
{code}
[WARNING] 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftHttpServlet.java:21:
 warning: BASE64Encoder is internal proprietary API and may be removed in a 
future release
{code}


was (Author: yuzhih...@gmail.com):
Here're the new javac warnings:
{code}
[WARNING] 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-examples/src/main/java/org/apache/hadoop/hbase/thrift/HttpDoAsClient.java:[21,16]
 sun.misc.   BASE64Encoder is internal proprietary API and may be 
removed in a future release
[WARNING] 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-examples/src/main/java/org/apache/hadoop/hbase/thrift/HttpDoAsClient.java:[222,29]
 sun.misc.  BASE64Encoder is internal proprietary API and may be 
removed in a future release
{code}
Here is the javadoc warning:
{code}
[WARNING] 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftHttpServlet.java:21:
 warning: BASE64Encoder is internal proprietary API and may be removed in a 
future release
{code}

 Fixing the usage of remote user in thrift doAs implementation.
 --

 Key: HBASE-13115
 URL: https://issues.apache.org/jira/browse/HBASE-13115
 Project: HBase
  Issue Type: Bug
Reporter: Srikanth Srungarapu
Assignee: Srikanth Srungarapu
 Attachments: HBASE-13115.patch, HBASE-13115.patch


 This issue is similar to HBASE-13085 fixed in REST gateway recently. This 
 change factors in the usage of remote user. Besides this, I also added the 
 following changes.
 * Adding response headers to strictly adhere to RFC specifcations.
 * Made changes to demo client to get rid of hard codings.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13128) Make HBCK's lock file retry creation and deletion


[ 
https://issues.apache.org/jira/browse/HBASE-13128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341365#comment-14341365
 ] 

Hadoop QA commented on HBASE-13128:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12701541/hbck_files_fix.patch
  against master branch at commit fdb48a7bbeecf891a0fc74171961901fbfdb5664.
  ATTACHMENT ID: 12701541

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.1 2.5.2 2.6.0)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:red}-1 checkstyle{color}.  The applied patch generated 
1941 checkstyle errors (more than the master's current 1938 errors).

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 lineLengths{color}.  The patch introduces the following lines 
longer than 100:
+   
getConf().getInt(hbase.hbck.lockfile.attempt.sleep.interval, 
LOCK_FILE_ATTEMPT_SLEEP_INTERVAL));
+   getConf().getInt(hbase.hbck.lockfile.attempt.sleep.interval, 
LOCK_FILE_ATTEMPT_SLEEP_INTERVAL));

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   org.apache.hadoop.hbase.util.TestHBaseFsck

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13013//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13013//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13013//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13013//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13013//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13013//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13013//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13013//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13013//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13013//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13013//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13013//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13013//artifact/patchprocess/checkstyle-aggregate.html

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13013//console

This message is automatically generated.

 Make HBCK's lock file retry creation and deletion
 -

 Key: HBASE-13128
 URL: https://issues.apache.org/jira/browse/HBASE-13128
 Project: HBase
  Issue Type: Improvement
  Components: hbck
Reporter: Victoria
Assignee: Victoria
Priority: Minor
 Attachments: hbck_files_fix.patch


 When hbck runs it creates a lock file to ensure that no two hbck instances 
 are running. We've been seeing creating and removing that file fail sometimes.
 This improvement should make the creation, closing of the file, and the 
 deletion retry multiple times. This should allow our alerting which uses this 
 command to be

[jira] [Commented] (HBASE-13084) Add labels to VisibilityLabelsCache asynchronously causes TestShell flakey


[ 
https://issues.apache.org/jira/browse/HBASE-13084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341376#comment-14341376
 ] 

Hudson commented on HBASE-13084:


FAILURE: Integrated in HBase-TRUNK #6183 (See 
[https://builds.apache.org/job/HBase-TRUNK/6183/])
HBASE-13084 addendum disable info server in shell test (stack: rev 
fdb48a7bbeecf891a0fc74171961901fbfdb5664)
* 
hbase-shell/src/test/java/org/apache/hadoop/hbase/client/AbstractTestShell.java


 Add labels to VisibilityLabelsCache asynchronously causes TestShell flakey
 --

 Key: HBASE-13084
 URL: https://issues.apache.org/jira/browse/HBASE-13084
 Project: HBase
  Issue Type: Bug
  Components: test
Reporter: zhangduo
Assignee: zhangduo
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-13084-addendum.patch, HBASE-13084-addendum2.patch, 
 HBASE-13084.patch, HBASE-13084_1.patch, HBASE-13084_2.patch, 
 HBASE-13084_2.patch, HBASE-13084_2.patch, HBASE-13084_2.patch, 
 HBASE-13084_2_disable_test.patch


 As discussed in HBASE-12953, we found this error in PreCommit log
 https://builds.apache.org/job/PreCommit-HBASE-Build/12918/artifact/hbase-shell/target/surefire-reports/org.apache.hadoop.hbase.client.TestShell-output.txt
 {noformat}
   1) Error:
 test_The_get/put_methods_should_work_for_data_written_with_Visibility(Hbase::VisibilityLabelsAdminMethodsTest):
 ArgumentError: org.apache.hadoop.hbase.DoNotRetryIOException: 
 org.apache.hadoop.hbase.security.visibility.InvalidLabelException: Label 
 'TEST_VISIBILITY' doesn't exists
   at 
 org.apache.hadoop.hbase.security.visibility.VisibilityController.setAuths(VisibilityController.java:808)
   at 
 org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService$1.setAuths(VisibilityLabelsProtos.java:6036)
   at 
 org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService.callMethod(VisibilityLabelsProtos.java:6219)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:6867)
   at 
 org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1707)
   at 
 org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1689)
   at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31309)
   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2038)
   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
   at 
 org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
   at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
   at java.lang.Thread.run(Thread.java:744)
 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-shell/src/main/ruby/hbase/visibility_labels.rb:84:in
  `set_auths'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:77:in 
 `test_The_get/put_methods_should_work_for_data_written_with_Visibility'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
   2) Error:
 test_The_set/clear_methods_should_work_with_authorizations(Hbase::VisibilityLabelsAdminMethodsTest):
 ArgumentError: No authentication set for the given user jenkins
 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-shell/src/main/ruby/hbase/visibility_labels.rb:97:in
  `get_auths'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:57:in 
 `test_The_set/clear_methods_should_work_with_authorizations'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
 {noformat}
 This is the test code
 {code:title=visibility_labels_admin_test.rb}
   label = 'TEST_VISIBILITY'
   user = org.apache.hadoop.hbase.security.User.getCurrent().getName();
   visibility_admin.add_labels(label)
   visibility_admin.set_auths(user, label)
 {code}
 It says 'label does not exists' when calling set_auths.
 Then I add some ugly logs in  DefaultVisibilityLabelServiceImpl and 
 VisibilityLabelsCache.
 {code:title=DefaultVisibilityLabelServiceImpl.java}
   public OperationStatus[] addLabels(Listbyte[] labels) throws IOException {
 ...
 if (mutateLabelsRegion(puts, finalOpStatus)) {
   updateZk(true);
 }
 for (byte[] label : labels) {
   String labelStr = Bytes.toString(label);
   LOG.info(labelStr + = + 
 this.labelsCache.getLabelOrdinal(labelStr));
 }
 ...
   }
 {code}
 {code:title=VisibilityLabelsCache.java}
   public void refreshLabelsCache(byte[] data) throws IOException {

[jira] [Commented] (HBASE-12975) Supportable SplitTransaction and RegionMergeTransaction interfaces


[ 
https://issues.apache.org/jira/browse/HBASE-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341387#comment-14341387
 ] 

Sean Busbey commented on HBASE-12975:
-

semver says we only add features in minor releases.

 Supportable SplitTransaction and RegionMergeTransaction interfaces
 --

 Key: HBASE-12975
 URL: https://issues.apache.org/jira/browse/HBASE-12975
 Project: HBase
  Issue Type: Improvement
Reporter: Rajeshbabu Chintaguntla
Assignee: Andrew Purtell
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-12975.patch, HBASE-12975.patch


 Making SplitTransaction, RegionMergeTransaction limited private is required 
 to support local indexing feature in Phoenix to ensure regions colocation. 
 We can ensure region split, regions merge in the coprocessors in few method 
 calls without touching internals like creating zk's, file layout changes or 
 assignments.
 1) stepsBeforePONR, stepsAfterPONR we can ensure split.
 2) meta entries can pass through coprocessors to atomically update with the 
 normal split/merge.
 3) rollback on failure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11580) Failover handling for secondary region replicas


[ 
https://issues.apache.org/jira/browse/HBASE-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341308#comment-14341308
 ] 

Hadoop QA commented on HBASE-11580:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12701535/hbase-11580_v2.patch
  against master branch at commit bec2b0d320554b0af8c891fddc147a953f35765f.
  ATTACHMENT ID: 12701535

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 23 new 
or modified tests.

{color:red}-1 Anti-pattern{color}.  The patch appears to 
have anti-pattern where BYTES_COMPARATOR was omitted:
 +getRegionInfo(), -1, new TreeMapbyte[], ListPath());.
{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.1 2.5.2 2.6.0)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 3 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   org.apache.hadoop.hbase.TestInterfaceAudienceAnnotations

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13011//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/checkstyle-aggregate.html

  Javadoc warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/patchJavadocWarnings.txt
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13011//console

This message is automatically generated.

 Failover handling for secondary region replicas
 ---

 Key: HBASE-11580
 URL: https://issues.apache.org/jira/browse/HBASE-11580
 Project: HBase
  Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Attachments: hbase-11580_v2.patch


 With the async wal approach (HBASE-11568), the edits are not persisted (to 
 wal) in the secondary region replicas. However this means that we have to 
 deal with secondary region replica failures. 
 We can seek to re-replicate the edits from primary to the secondary when the 
 secondary region is opened in another server but this would mean to setup a 
 replication queue again, and holding on to the wals for longer. 
 Instead, we can design it so that the edits form the secondaries are not

[jira] [Commented] (HBASE-13097) Use same EventLoopGroup for different AsyncRpcClients if possible


[ 
https://issues.apache.org/jira/browse/HBASE-13097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341310#comment-14341310
 ] 

zhangduo commented on HBASE-13097:
--

https://builds.apache.org/job/HBase-trunk/6181/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestClientScannerRPCTimeout-output.txt
{noformat}
2015-02-28 03:32:26,444 INFO  [MASTER_TABLE_OPERATIONS-asf900:37322-0] 
hbase.MetaTableAccessor(1306): Added 2
2015-02-28 03:32:26,445 WARN  [MASTER_TABLE_OPERATIONS-asf900:37322-0] 
balancer.BaseLoadBalancer(1039): Wanted to do round robin assignment but no 
servers to assign to
2015-02-28 03:32:26,445 ERROR [MASTER_TABLE_OPERATIONS-asf900:37322-0] 
handler.CreateTableHandler(168): Error trying to create the table 
testScannerNextRPCTimesout
java.io.IOException: Unable to determine a plan to assign region(s)
at 
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1446)
at 
org.apache.hadoop.hbase.util.ModifyRegionUtils.assignRegions(ModifyRegionUtils.java:241)
at 
org.apache.hadoop.hbase.master.handler.CreateTableHandler.handleCreateTable(CreateTableHandler.java:247)
at 
org.apache.hadoop.hbase.master.handler.CreateTableHandler.process(CreateTableHandler.java:156)
at 
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:103)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
{noformat}

Seems failed to create table?

 Use same EventLoopGroup for different AsyncRpcClients if possible
 -

 Key: HBASE-13097
 URL: https://issues.apache.org/jira/browse/HBASE-13097
 Project: HBase
  Issue Type: Bug
  Components: IPC/RPC, test
Affects Versions: 2.0.0, 1.1.0
Reporter: zhangduo
Assignee: zhangduo
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-13097-addendum.patch, HBASE-13097.patch, 
 HBASE-13097_1.patch, HBASE-13097_2.patch


 In some unit tests(such as TestAcidGuarantees) we create multiple Connection 
 instance. If we use AsyncRpcClient, then there will be multiple netty 
 Bootstrap and every Bootstrap has its own PooledByteBufAllocator.
 I haven't read the code clearly but it uses some threadlocal technics and 
 jmap shows io.netty.buffer.PoolThreadCache$MemoryRegionCache$Entry is the 
 biggest things on Heap.
 See 
 https://builds.apache.org/job/HBase-TRUNK/6168/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.TestAcidGuarantees-output.txt
 {noformat}
 2015-02-24 23:50:29,704 WARN  [JvmPauseMonitor] 
 util.JvmPauseMonitor$Monitor(167): Detected pause in JVM or host machine (eg 
 GC): pause of approximately 20133ms
 GC pool 'PS MarkSweep' had collection(s): count=15 time=55525ms
 {noformat}
 Update: We use a singleton PooledByteBufAllocator so the reason should be too 
 many threads. So we will work on reduce the connections and rpclients in unit 
 tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13115) Fixing the usage of remote user in thrift doAs implementation.


[ 
https://issues.apache.org/jira/browse/HBASE-13115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341338#comment-14341338
 ] 

Ted Yu commented on HBASE-13115:


Ping [~apurtell], [~enis]

 Fixing the usage of remote user in thrift doAs implementation.
 --

 Key: HBASE-13115
 URL: https://issues.apache.org/jira/browse/HBASE-13115
 Project: HBase
  Issue Type: Bug
Reporter: Srikanth Srungarapu
Assignee: Srikanth Srungarapu
 Attachments: HBASE-13115.patch, HBASE-13115.patch


 This issue is similar to HBASE-13085 fixed in REST gateway recently. This 
 change factors in the usage of remote user. Besides this, I also added the 
 following changes.
 * Adding response headers to strictly adhere to RFC specifcations.
 * Made changes to demo client to get rid of hard codings.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13071) Hbase Streaming Scan Feature

[
https://issues.apache.org/jira/browse/HBASE-13071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341359#comment-14341359
]

Ted Yu commented on HBASE-13071:

There has been effort to stabilize trunk build.

Please prepare patch for trunk.

Hbase Streaming Scan Feature

Key: HBASE-13071
URL: https://issues.apache.org/jira/browse/HBASE-13071
Project: HBase
Issue Type: New Feature
Affects Versions: 0.98.11
Reporter: Eshcar Hillel
Attachments: HBASE-13071-v1.patch, HBASE-13071-v2.patch,
HBaseStreamingScanDesign.pdf, HbaseStreamingScanEvaluation.pdf

A scan operation iterates over all rows of a table or a subrange of the
table. The synchronous nature in which the data is served at the client side
hinders the speed the application traverses the data: it increases the
overall processing time, and may cause a great variance in the times the
application waits for the next piece of data.
The scanner next() method at the client side invokes an RPC to the
regionserver and then stores the results in a cache. The application can
specify how many rows will be transmitted per RPC; by default this is set to
100 rows.
The cache can be considered as a producer-consumer queue, where the hbase
client pushes the data to the queue and the application consumes it.
Currently this queue is synchronous, i.e., blocking. More specifically, when
the application consumed all the data from the cache --- so the cache is
empty --- the hbase client retrieves additional data from the server and
re-fills the cache with new data. During this time the application is blocked.
Under the assumption that the application processing time can be balanced by
the time it takes to retrieve the data, an asynchronous approach can reduce
the time the application is waiting for data.
We attach a design document.
We also have a patch that is based on a private branch, and some evaluation
results of this code.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12311) Version stats in HFiles?


 [ 
https://issues.apache.org/jira/browse/HBASE-12311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-12311:
--
Attachment: 12311-indexed-0.98-v2.txt

Can now remove the lookahead hack I put in before.
(Will test more, to make sure this is true in all cases)

 Version stats in HFiles?
 

 Key: HBASE-12311
 URL: https://issues.apache.org/jira/browse/HBASE-12311
 Project: HBase
  Issue Type: Brainstorming
Reporter: Lars Hofhansl
 Attachments: 12311-indexed-0.98-v2.txt, 12311-indexed-0.98.txt, 
 12311-v2.txt, 12311-v3.txt, 12311.txt, CellStatTracker.java


 In HBASE-9778 I basically punted the decision on whether doing repeated 
 scanner.next() called instead of the issueing (re)seeks to the user.
 I think we can do better.
 One way do that is maintain simple stats of what the maximum number of 
 versions we've seen for any row/col combination and store these in the 
 HFile's metadata (just like the timerange, oldest Put, etc).
 Then we estimate fairly accurately whether we have to expect lots of versions 
 (i.e. seek between columns is better) or not (in which case we'd issue 
 repeated next()'s).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13127) Add timeouts on all tests so less zombie sightings


[ 
https://issues.apache.org/jira/browse/HBASE-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341199#comment-14341199
 ] 

Hadoop QA commented on HBASE-13127:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12701488/13127.alternate.txt
  against master branch at commit d1619bceb3142f3ab8c134365e18a150fbd5b9bf.
  ATTACHMENT ID: 12701488

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 4 new 
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.1 2.5.2 2.6.0)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:red}-1 release audit{color}.  The applied patch generated 1 release 
audit warnings (more than the master's current 0 warnings).

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   org.apache.hadoop.hbase.TestMultiVersions

 {color:red}-1 core zombie tests{color}.  There are 3 zombie test(s):   
at org.apache.sqoop.common.test.kafka.KafkaLocal.stop(KafkaLocal.java:52)
at 
org.apache.sqoop.common.test.kafka.TestUtil.tearDown(TestUtil.java:149)
at 
org.apache.sqoop.test.testcases.KafkaConnectorTestCase.stopKafka(KafkaConnectorTestCase.java:49)
at 
org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
at 
org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:564)
at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:213)
at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:138)
at 
org.testng.internal.TestMethodWorker.invokeAfterClassMethods(TestMethodWorker.java:225)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:114)
at org.testng.TestRunner.privateRun(TestRunner.java:767)
at org.testng.TestRunner.run(TestRunner.java:617)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:343)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:305)
at org.testng.SuiteRunner.run(SuiteRunner.java:254)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1149)
at org.testng.TestNG.run(TestNG.java:1057)
at 
org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:77)
at 
org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:126)
at 
org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:110)
at 
org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:117)
at 
org.apache.sqoop.test.testcases.ConnectorTestCase.executeJob(ConnectorTestCase.java:265)
at 
org.apache.sqoop.test.testcases.ConnectorTestCase.executeJob(ConnectorTestCase.java:281)
at 
org.apache.sqoop.integration.connector.jdbc.generic.PartitionerTest.testSplitter(PartitionerTest.java:109)
at 
org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
at 
org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
at org.testng.TestRunner.privateRun(TestRunner.java:767)
at org.testng.TestRunner.run(TestRunner.java:617)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:343)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:305)
at

[jira] [Updated] (HBASE-12311) Version stats in HFiles?

[
https://issues.apache.org/jira/browse/HBASE-12311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Lars Hofhansl updated HBASE-12311:
--
Attachment: 12311-indexed-0.98.txt

Here's a patch that illustrates the idea for 0.98. In store scanner when the
SQM indicated we should seek, we check the nextIndexedKey (if available) and we
would seek before that, we simply SKIP and let the SQM try again.

The only annoying part is that we only an indexed *key* (i.e. row, family,
column), which we are trying to get rid of.
HFileReaderV2.AbstractScannerV2.reseekTo performs the same check to decide
whether to seek or to retry on the same block, this just pulls the check up. We
can probably remove that optimization from the AbstractScannerV2 now (and save
a few more compares).

Version stats in HFiles?

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12924) HRegionServer#MovedRegionsCleaner Chore does not start

2015-02-27 Thread Elliott Clark (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341368#comment-14341368
 ] 

Elliott Clark commented on HBASE-12924:
---

+1 from me. I'll commit unless there are any objections


 HRegionServer#MovedRegionsCleaner Chore does not start
 --

 Key: HBASE-12924
 URL: https://issues.apache.org/jira/browse/HBASE-12924
 Project: HBase
  Issue Type: Bug
Reporter: Jonathan Lawlor
Assignee: Sameet Agarwal
Priority: Minor
 Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.11

 Attachments: first-task-v2.patch, first-task-v3.patch, 
 first-task-v4.patch, first-task.patch


 This issue is very similar to the one described in HBASE-11354. The method 
 createAndStart in MovedRegionsCleaner creates an instance of the chore but 
 never starts the underlying thread:
 {code}
 static MovedRegionsCleaner createAndStart(HRegionServer rs){
   Stoppable stoppable = new Stoppable() {
 private volatile boolean isStopped = false;
 @Override public void stop(String why) { isStopped = true;}
 @Override public boolean isStopped() {return isStopped;}
   };
   return new MovedRegionsCleaner(rs, stoppable);
 }
 {code}
 Nobody has noticed this issue so far so the Chore may not be that important



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13100) Shell command to retrieve table splits


[ 
https://issues.apache.org/jira/browse/HBASE-13100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14340207#comment-14340207
 ] 

Sean Busbey commented on HBASE-13100:
-

This looks good Ashish. Could you add a test similar to the first patch, where 
you get the splits of table that has some? 

 Shell command to retrieve table splits
 --

 Key: HBASE-13100
 URL: https://issues.apache.org/jira/browse/HBASE-13100
 Project: HBase
  Issue Type: Improvement
  Components: shell
Reporter: Sean Busbey
Assignee: Ashish Singhi
Priority: Minor
  Labels: beginner
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-13100-v1.patch, HBASE-13100-v2.patch, 
 HBASE-13100.patch


 Add a shell command that returns the splits for a table.
 Doing this yourself is currently possible, but involves going outside of the 
 public api.
 {code}
 jruby-1.7.3 :012  create 'example_table', 'f1', SPLITS = [10, 20, 30, 
 40]
 0 row(s) in 0.5500 seconds
  = Hbase::Table - example_table 
 jruby-1.7.3 :013  
 get_table('example_table').table.get_all_region_locations.map do |location| 
 org.apache.hadoop.hbase.util.Bytes::toStringBinary(location.get_region_info.get_start_key)
  end
 0 row(s) in 0.0130 seconds
  = [, 10, 20, 30, 40] 
 jruby-1.7.3 :014  
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13100) Shell command to retrieve table splits

2015-02-27 Thread Ashish Singhi (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14340224#comment-14340224
 ] 

Ashish Singhi commented on HBASE-13100:
---

I can do. I will add a new method in test_helper.rb which creates a table with 
split.
will attach a new patch with that change.

 Shell command to retrieve table splits
 --

 Key: HBASE-13100
 URL: https://issues.apache.org/jira/browse/HBASE-13100
 Project: HBase
  Issue Type: Improvement
  Components: shell
Reporter: Sean Busbey
Assignee: Ashish Singhi
Priority: Minor
  Labels: beginner
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-13100-v1.patch, HBASE-13100-v2.patch, 
 HBASE-13100.patch


 Add a shell command that returns the splits for a table.
 Doing this yourself is currently possible, but involves going outside of the 
 public api.
 {code}
 jruby-1.7.3 :012  create 'example_table', 'f1', SPLITS = [10, 20, 30, 
 40]
 0 row(s) in 0.5500 seconds
  = Hbase::Table - example_table 
 jruby-1.7.3 :013  
 get_table('example_table').table.get_all_region_locations.map do |location| 
 org.apache.hadoop.hbase.util.Bytes::toStringBinary(location.get_region_info.get_start_key)
  end
 0 row(s) in 0.0130 seconds
  = [, 10, 20, 30, 40] 
 jruby-1.7.3 :014  
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13097) Use same EventLoopGroup for different AsyncRpcClients if possible


[ 
https://issues.apache.org/jira/browse/HBASE-13097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14340129#comment-14340129
 ] 

zhangduo commented on HBASE-13097:
--

[~stack] The only two blue builds in this page 
https://builds.apache.org/job/PreCommit-HBASE-Build/

 Use same EventLoopGroup for different AsyncRpcClients if possible
 -

 Key: HBASE-13097
 URL: https://issues.apache.org/jira/browse/HBASE-13097
 Project: HBase
  Issue Type: Bug
  Components: IPC/RPC, test
Affects Versions: 2.0.0, 1.1.0
Reporter: zhangduo
Assignee: zhangduo
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-13097.patch, HBASE-13097_1.patch, 
 HBASE-13097_2.patch


 In some unit tests(such as TestAcidGuarantees) we create multiple Connection 
 instance. If we use AsyncRpcClient, then there will be multiple netty 
 Bootstrap and every Bootstrap has its own PooledByteBufAllocator.
 I haven't read the code clearly but it uses some threadlocal technics and 
 jmap shows io.netty.buffer.PoolThreadCache$MemoryRegionCache$Entry is the 
 biggest things on Heap.
 See 
 https://builds.apache.org/job/HBase-TRUNK/6168/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.TestAcidGuarantees-output.txt
 {noformat}
 2015-02-24 23:50:29,704 WARN  [JvmPauseMonitor] 
 util.JvmPauseMonitor$Monitor(167): Detected pause in JVM or host machine (eg 
 GC): pause of approximately 20133ms
 GC pool 'PS MarkSweep' had collection(s): count=15 time=55525ms
 {noformat}
 Update: We use a singleton PooledByteBufAllocator so the reason should be too 
 many threads. So we will work on reduce the connections and rpclients in unit 
 tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12975) SplitTranaction, RegionMergeTransaction to should have InterfaceAudience of LimitedPrivate(Coproc,Phoenix)

[
https://issues.apache.org/jira/browse/HBASE-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14340432#comment-14340432
]

Andrew Purtell commented on HBASE-12975:

bq. The reason to use it is that we'd label it Evolving or Stable in the next
minor release. We could even add a note in the javadoc explaining that it's
Unstable just to flag that folks have to keep in mind that it wasn't around at
the start of the API cut off for 1.0. Like the compat guide says, just because
we have the option to break things at a particular version (in this case at
patches sine it's Unstable) doesn't mean we will.

Ok, will do that and add the suggested javadoc

SplitTranaction, RegionMergeTransaction to should have InterfaceAudience of
LimitedPrivate(Coproc,Phoenix)
--

Key: HBASE-12975
URL: https://issues.apache.org/jira/browse/HBASE-12975
Project: HBase
Issue Type: Improvement
Reporter: Rajeshbabu Chintaguntla
Assignee: Rajeshbabu Chintaguntla
Fix For: 2.0.0, 1.0.1, 1.1.0

Attachments: HBASE-12975.patch

Making SplitTransaction, RegionMergeTransaction limited private is required
to support local indexing feature in Phoenix to ensure regions colocation.
We can ensure region split, regions merge in the coprocessors in few method
calls without touching internals like creating zk's, file layout changes or
assignments.
1) stepsBeforePONR, stepsAfterPONR we can ensure split.
2) meta entries can pass through coprocessors to atomically update with the
normal split/merge.
3) rollback on failure.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-13125) Purge hbase.bucketcache.sizes from the refguide; no such config (should be hbase.bucketcache.size?)

stack created HBASE-13125:
-

 Summary: Purge hbase.bucketcache.sizes from the refguide; no such 
config (should be hbase.bucketcache.size?)
 Key: HBASE-13125
 URL: https://issues.apache.org/jira/browse/HBASE-13125
 Project: HBase
  Issue Type: Bug
  Components: documentation
Reporter: stack
Priority: Trivial


Came up on mailing list today... a gentleman saw hbase.bucketcache.sizes and 
didn't know what it was about.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12972) Region, a supportable public/evolving subset of HRegion


 [ 
https://issues.apache.org/jira/browse/HBASE-12972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-12972:
---
Attachment: (was: HBASE-12972-0.98.patch)

 Region, a supportable public/evolving subset of HRegion
 ---

 Key: HBASE-12972
 URL: https://issues.apache.org/jira/browse/HBASE-12972
 Project: HBase
  Issue Type: New Feature
Reporter: Andrew Purtell
Assignee: Andrew Purtell
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-12972-0.98.patch, HBASE-12972-0.98.patch, 
 HBASE-12972-0.98.patch


 On HBASE-12566, [~lhofhansl] proposed:
 {quote}
 Maybe we can have a {{Region}} interface that is to {{HRegion}} is what 
 {{Store}} is to {{HStore}}. Store marked with {{@InterfaceAudience.Private}} 
 but used in some coprocessor hooks.
 {quote}
 By example, now coprocessors have to reach into HRegion in order to 
 participate in row and region locking protocols, this is one area where the 
 functionality is legitimate for coprocessors but not for users, so an 
 in-between interface make sense.
 In addition we should promote {{Store}}'s interface audience to 
 LimitedPrivate(COPROC).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13126) Clean up API for unintended methods within non-private classes.


[ 
https://issues.apache.org/jira/browse/HBASE-13126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14340443#comment-14340443
 ] 

Sean Busbey commented on HBASE-13126:
-

* HBTU#createAndForceNodeToOpenedState

 Clean up API for unintended methods within non-private classes.
 ---

 Key: HBASE-13126
 URL: https://issues.apache.org/jira/browse/HBASE-13126
 Project: HBase
  Issue Type: Task
  Components: API
Affects Versions: 2.0.0
Reporter: Sean Busbey
Priority: Blocker
 Fix For: 2.0.0


 Over in the review for HBASE-12972, [~enis] mentioned that one of the HBTU 
 methods wasn't intended for public consumption.
 Can we build a list of such methods across the API, appropriately annotate 
 them for 2.0, and deprecate them in earlier versions with a warning that 
 they're going to be restricted?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12972) Region, a supportable public/evolving subset of HRegion


[ 
https://issues.apache.org/jira/browse/HBASE-12972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14340450#comment-14340450
 ] 

Andrew Purtell commented on HBASE-12972:


[~jamestaylor] Would something like this be fine?
- Phoenix 4.x - HBase 0.98.x
- Phoenix 5.0.x - HBase 1.0.x
- Phoenix 5.1.x - HBase 1.1.x (not compatible with HBase 1.0.x or earlier)


 Region, a supportable public/evolving subset of HRegion
 ---

 Key: HBASE-12972
 URL: https://issues.apache.org/jira/browse/HBASE-12972
 Project: HBase
  Issue Type: New Feature
Reporter: Andrew Purtell
Assignee: Andrew Purtell
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-12972-0.98.patch


 On HBASE-12566, [~lhofhansl] proposed:
 {quote}
 Maybe we can have a {{Region}} interface that is to {{HRegion}} is what 
 {{Store}} is to {{HStore}}. Store marked with {{@InterfaceAudience.Private}} 
 but used in some coprocessor hooks.
 {quote}
 By example, now coprocessors have to reach into HRegion in order to 
 participate in row and region locking protocols, this is one area where the 
 functionality is legitimate for coprocessors but not for users, so an 
 in-between interface make sense.
 In addition we should promote {{Store}}'s interface audience to 
 LimitedPrivate(COPROC).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12924) HRegionServer#MovedRegionsCleaner Chore does not start

2015-02-27 Thread Jonathan Lawlor (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14340445#comment-14340445
 ] 

Jonathan Lawlor commented on HBASE-12924:
-

Looks like it ran into the flaky test TestAcidGuarantees, looks unrelated. We 
can retry the QA build to be sure. 

The checkstyle error is coming from an unused import in HRegionServer.java 
(there also seem to be some ununsed imports inside TestMovedRegionsCleaner that 
could be removed). The checkstyle also recommends we declare 
MovedRegionsCleaner as a final class.

 HRegionServer#MovedRegionsCleaner Chore does not start
 --

 Key: HBASE-12924
 URL: https://issues.apache.org/jira/browse/HBASE-12924
 Project: HBase
  Issue Type: Bug
Reporter: Jonathan Lawlor
Assignee: Sameet Agarwal
Priority: Minor
 Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.11

 Attachments: first-task-v2.patch, first-task-v3.patch, 
 first-task.patch


 This issue is very similar to the one described in HBASE-11354. The method 
 createAndStart in MovedRegionsCleaner creates an instance of the chore but 
 never starts the underlying thread:
 {code}
 static MovedRegionsCleaner createAndStart(HRegionServer rs){
   Stoppable stoppable = new Stoppable() {
 private volatile boolean isStopped = false;
 @Override public void stop(String why) { isStopped = true;}
 @Override public boolean isStopped() {return isStopped;}
   };
   return new MovedRegionsCleaner(rs, stoppable);
 }
 {code}
 Nobody has noticed this issue so far so the Chore may not be that important



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13097) Use same EventLoopGroup for different AsyncRpcClients if possible


[ 
https://issues.apache.org/jira/browse/HBASE-13097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14340364#comment-14340364
 ] 

stack commented on HBASE-13097:
---

Very nice test refactor. Very nice figuring out root problem.  Patch looks 
great (though I have only a notion of what is going on in here).  Lets try it 
and see how it does.

Leaving issue open to see how next few trunk and branch-1 builds go.  Thanks 
[~Apache9] (and [~jurmous] for input).

 Use same EventLoopGroup for different AsyncRpcClients if possible
 -

 Key: HBASE-13097
 URL: https://issues.apache.org/jira/browse/HBASE-13097
 Project: HBase
  Issue Type: Bug
  Components: IPC/RPC, test
Affects Versions: 2.0.0, 1.1.0
Reporter: zhangduo
Assignee: zhangduo
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-13097.patch, HBASE-13097_1.patch, 
 HBASE-13097_2.patch


 In some unit tests(such as TestAcidGuarantees) we create multiple Connection 
 instance. If we use AsyncRpcClient, then there will be multiple netty 
 Bootstrap and every Bootstrap has its own PooledByteBufAllocator.
 I haven't read the code clearly but it uses some threadlocal technics and 
 jmap shows io.netty.buffer.PoolThreadCache$MemoryRegionCache$Entry is the 
 biggest things on Heap.
 See 
 https://builds.apache.org/job/HBase-TRUNK/6168/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.TestAcidGuarantees-output.txt
 {noformat}
 2015-02-24 23:50:29,704 WARN  [JvmPauseMonitor] 
 util.JvmPauseMonitor$Monitor(167): Detected pause in JVM or host machine (eg 
 GC): pause of approximately 20133ms
 GC pool 'PS MarkSweep' had collection(s): count=15 time=55525ms
 {noformat}
 Update: We use a singleton PooledByteBufAllocator so the reason should be too 
 many threads. So we will work on reduce the connections and rpclients in unit 
 tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13125) Purge hbase.bucketcache.sizes from the refguide; no such config (should be hbase.bucketcache.size?)


[ 
https://issues.apache.org/jira/browse/HBASE-13125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14340428#comment-14340428
 ] 

stack commented on HBASE-13125:
---

Ditto hfile.block.cache.sizes (should be hfile.block.cache.size).

 Purge hbase.bucketcache.sizes from the refguide; no such config (should be 
 hbase.bucketcache.size?)
 ---

 Key: HBASE-13125
 URL: https://issues.apache.org/jira/browse/HBASE-13125
 Project: HBase
  Issue Type: Bug
  Components: documentation
Reporter: stack
Priority: Trivial

 Came up on mailing list today... a gentleman saw hbase.bucketcache.sizes and 
 didn't know what it was about.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12972) Region, a supportable public/evolving subset of HRegion


 [ 
https://issues.apache.org/jira/browse/HBASE-12972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-12972:
---
Attachment: (was: HBASE-12972-0.98.patch)

 Region, a supportable public/evolving subset of HRegion
 ---

 Key: HBASE-12972
 URL: https://issues.apache.org/jira/browse/HBASE-12972
 Project: HBase
  Issue Type: New Feature
Reporter: Andrew Purtell
Assignee: Andrew Purtell
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-12972-0.98.patch


 On HBASE-12566, [~lhofhansl] proposed:
 {quote}
 Maybe we can have a {{Region}} interface that is to {{HRegion}} is what 
 {{Store}} is to {{HStore}}. Store marked with {{@InterfaceAudience.Private}} 
 but used in some coprocessor hooks.
 {quote}
 By example, now coprocessors have to reach into HRegion in order to 
 participate in row and region locking protocols, this is one area where the 
 functionality is legitimate for coprocessors but not for users, so an 
 in-between interface make sense.
 In addition we should promote {{Store}}'s interface audience to 
 LimitedPrivate(COPROC).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13100) Shell command to retrieve table splits


[ 
https://issues.apache.org/jira/browse/HBASE-13100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14340438#comment-14340438
 ] 

Sean Busbey commented on HBASE-13100:
-

{code}
+  create_test_table_with_splits(@testTableName)
+  @table = table(@testTableName)
+  splits = @table._get_splits_internal()
+  #Total splits is 5 but here count is 4 as we ignore implicit empty split.
+  assert_equal(4, splits.size)
+  assert_equal([10, 20, 30, 40], splits)
+  drop_test_table(@testTableName)
+end
+
   end
 end
diff --git a/hbase-shell/src/test/ruby/test_helper.rb 
b/hbase-shell/src/test/ruby/test_helper.rb
index e2ab921..c75242d 100644
--- a/hbase-shell/src/test/ruby/test_helper.rb
+++ b/hbase-shell/src/test/ruby/test_helper.rb
@@ -85,6 +85,19 @@ module Hbase
   end
 end
 
+   def create_test_table_with_splits(name)
+  # Create the table if needed
+  unless admin.exists?(name)
+admin.create name, 'f1', SPLITS = ['10', '20', '30', '40']
+return
+  end
{code}

You should pass the split array as an argument to 
create_test_table_with_splits. That way it's easier to understand why the test 
looks for those particular splits and also makes the helper more widely useful.

 Shell command to retrieve table splits
 --

 Key: HBASE-13100
 URL: https://issues.apache.org/jira/browse/HBASE-13100
 Project: HBase
  Issue Type: Improvement
  Components: shell
Reporter: Sean Busbey
Assignee: Ashish Singhi
Priority: Minor
  Labels: beginner
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-13100-v1.patch, HBASE-13100-v2.patch, 
 HBASE-13100-v3.patch, HBASE-13100.patch


 Add a shell command that returns the splits for a table.
 Doing this yourself is currently possible, but involves going outside of the 
 public api.
 {code}
 jruby-1.7.3 :012  create 'example_table', 'f1', SPLITS = [10, 20, 30, 
 40]
 0 row(s) in 0.5500 seconds
  = Hbase::Table - example_table 
 jruby-1.7.3 :013  
 get_table('example_table').table.get_all_region_locations.map do |location| 
 org.apache.hadoop.hbase.util.Bytes::toStringBinary(location.get_region_info.get_start_key)
  end
 0 row(s) in 0.0130 seconds
  = [, 10, 20, 30, 40] 
 jruby-1.7.3 :014  
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12972) Region, a supportable public/evolving subset of HRegion


 [ 
https://issues.apache.org/jira/browse/HBASE-12972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-12972:
---
Attachment: (was: HBASE-12972-0.98.patch)

 Region, a supportable public/evolving subset of HRegion
 ---

 Key: HBASE-12972
 URL: https://issues.apache.org/jira/browse/HBASE-12972
 Project: HBase
  Issue Type: New Feature
Reporter: Andrew Purtell
Assignee: Andrew Purtell
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-12972-0.98.patch


 On HBASE-12566, [~lhofhansl] proposed:
 {quote}
 Maybe we can have a {{Region}} interface that is to {{HRegion}} is what 
 {{Store}} is to {{HStore}}. Store marked with {{@InterfaceAudience.Private}} 
 but used in some coprocessor hooks.
 {quote}
 By example, now coprocessors have to reach into HRegion in order to 
 participate in row and region locking protocols, this is one area where the 
 functionality is legitimate for coprocessors but not for users, so an 
 in-between interface make sense.
 In addition we should promote {{Store}}'s interface audience to 
 LimitedPrivate(COPROC).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13084) Add labels to VisibilityLabelsCache asynchronously causes TestShell flakey


[ 
https://issues.apache.org/jira/browse/HBASE-13084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14340442#comment-14340442
 ] 

Hudson commented on HBASE-13084:


FAILURE: Integrated in HBase-1.1 #218 (See 
[https://builds.apache.org/job/HBase-1.1/218/])
HBASE-13084 addendum move replication_admin_test.rb to individual test (stack: 
rev 6331bc2afb2d0d62a1b749ee8e1772305b4665bc)
* 
hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestReplicationShell.java
* hbase-shell/src/test/ruby/tests_runner.rb
* 
hbase-shell/src/test/java/org/apache/hadoop/hbase/client/AbstractTestShell.java
* hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestShell.java


 Add labels to VisibilityLabelsCache asynchronously causes TestShell flakey
 --

 Key: HBASE-13084
 URL: https://issues.apache.org/jira/browse/HBASE-13084
 Project: HBase
  Issue Type: Bug
  Components: test
Reporter: zhangduo
Assignee: zhangduo
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-13084-addendum.patch, HBASE-13084.patch, 
 HBASE-13084_1.patch, HBASE-13084_2.patch, HBASE-13084_2.patch, 
 HBASE-13084_2.patch, HBASE-13084_2.patch, HBASE-13084_2_disable_test.patch


 As discussed in HBASE-12953, we found this error in PreCommit log
 https://builds.apache.org/job/PreCommit-HBASE-Build/12918/artifact/hbase-shell/target/surefire-reports/org.apache.hadoop.hbase.client.TestShell-output.txt
 {noformat}
   1) Error:
 test_The_get/put_methods_should_work_for_data_written_with_Visibility(Hbase::VisibilityLabelsAdminMethodsTest):
 ArgumentError: org.apache.hadoop.hbase.DoNotRetryIOException: 
 org.apache.hadoop.hbase.security.visibility.InvalidLabelException: Label 
 'TEST_VISIBILITY' doesn't exists
   at 
 org.apache.hadoop.hbase.security.visibility.VisibilityController.setAuths(VisibilityController.java:808)
   at 
 org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService$1.setAuths(VisibilityLabelsProtos.java:6036)
   at 
 org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService.callMethod(VisibilityLabelsProtos.java:6219)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:6867)
   at 
 org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1707)
   at 
 org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1689)
   at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31309)
   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2038)
   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
   at 
 org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
   at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
   at java.lang.Thread.run(Thread.java:744)
 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-shell/src/main/ruby/hbase/visibility_labels.rb:84:in
  `set_auths'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:77:in 
 `test_The_get/put_methods_should_work_for_data_written_with_Visibility'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
   2) Error:
 test_The_set/clear_methods_should_work_with_authorizations(Hbase::VisibilityLabelsAdminMethodsTest):
 ArgumentError: No authentication set for the given user jenkins
 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-shell/src/main/ruby/hbase/visibility_labels.rb:97:in
  `get_auths'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:57:in 
 `test_The_set/clear_methods_should_work_with_authorizations'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
 {noformat}
 This is the test code
 {code:title=visibility_labels_admin_test.rb}
   label = 'TEST_VISIBILITY'
   user = org.apache.hadoop.hbase.security.User.getCurrent().getName();
   visibility_admin.add_labels(label)
   visibility_admin.set_auths(user, label)
 {code}
 It says 'label does not exists' when calling set_auths.
 Then I add some ugly logs in  DefaultVisibilityLabelServiceImpl and 
 VisibilityLabelsCache.
 {code:title=DefaultVisibilityLabelServiceImpl.java}
   public OperationStatus[] addLabels(Listbyte[] labels) throws IOException {
 ...
 if (mutateLabelsRegion(puts, finalOpStatus)) {
   updateZk(true);
 }
 for (byte[] label : labels) {
   String labelStr = Bytes.toString(label);
   LOG.info(labelStr + = +

[jira] [Created] (HBASE-13126) Clean up API for unintended methods within non-private classes.

Sean Busbey created HBASE-13126:
---

 Summary: Clean up API for unintended methods within non-private 
classes.
 Key: HBASE-13126
 URL: https://issues.apache.org/jira/browse/HBASE-13126
 Project: HBase
  Issue Type: Task
  Components: API
Affects Versions: 2.0.0
Reporter: Sean Busbey
Priority: Blocker
 Fix For: 2.0.0


Over in the review for HBASE-12972, [~enis] mentioned that one of the HBTU 
methods wasn't intended for public consumption.

Can we build a list of such methods across the API, appropriately annotate them 
for 2.0, and deprecate them in earlier versions with a warning that they're 
going to be restricted?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13100) Shell command to retrieve table splits


[ 
https://issues.apache.org/jira/browse/HBASE-13100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14340422#comment-14340422
 ] 

Hadoop QA commented on HBASE-13100:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12701371/HBASE-13100-v3.patch
  against master branch at commit 458846ef7b0528cb7952c413694eaf55c5d94342.
  ATTACHMENT ID: 12701371

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 8 new 
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.1 2.5.2 2.6.0)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
 

 {color:red}-1 core zombie tests{color}.  There are 1 zombie test(s):   
at 
org.apache.hadoop.hbase.TestAcidGuarantees.testScanAtomicity(TestAcidGuarantees.java:354)

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13001//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13001//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13001//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13001//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13001//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13001//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13001//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13001//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13001//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13001//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13001//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13001//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13001//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13001//console

This message is automatically generated.

 Shell command to retrieve table splits
 --

 Key: HBASE-13100
 URL: https://issues.apache.org/jira/browse/HBASE-13100
 Project: HBase
  Issue Type: Improvement
  Components: shell
Reporter: Sean Busbey
Assignee: Ashish Singhi
Priority: Minor
  Labels: beginner
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-13100-v1.patch, HBASE-13100-v2.patch, 
 HBASE-13100-v3.patch, HBASE-13100.patch


 Add a shell command that returns the splits for a table.
 Doing this yourself is currently possible, but involves going outside of the 
 public api.
 {code}
 jruby-1.7.3 :012  create 'example_table', 'f1', SPLITS = [10, 20, 30, 
 40]
 0 row(s) in 0.5500 seconds
  = Hbase::Table - example_table 
 jruby-1.7.3 :013  
 get_table('example_table').table.get_all_region_locations.map do |location| 
 org.apache.hadoop.hbase.util.Bytes::toStringBinary(location.get_region_info.get_start_key)
  end
 0 row(s) in 0.0130 seconds
  = [, 10, 20, 30, 40] 
 jruby-1.7.3 :014

[jira] [Commented] (HBASE-13123) Minor bug in ROW bloom filter

2015-02-27 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14340423#comment-14340423
 ] 

ramkrishna.s.vasudevan commented on HBASE-13123:


Will commit this unless objections.

 Minor bug in ROW bloom filter
 -

 Key: HBASE-13123
 URL: https://issues.apache.org/jira/browse/HBASE-13123
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.0.0, 2.0.0, 1.1.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Minor
 Attachments: HBASE-13163.patch


 While checking the code for Bloom filter found that while checking if a key 
 passes the ROW bloom check we try to create a bloom key. The bloom key should 
 be constructed only with the row part of the key. But try to form the bloom 
 key including the meta data part of the key.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13084) Add labels to VisibilityLabelsCache asynchronously causes TestShell flakey


[ 
https://issues.apache.org/jira/browse/HBASE-13084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14340343#comment-14340343
 ] 

stack commented on HBASE-13084:
---

[~Apache9] I applied your patch on top of the added timeout. Lets see how it 
does. We can back out the ten second wait later when all stable again.  I 
applied your patch to branch-1+.  Lets see how builds do from here on before 
closing.  You are pretty amazing the way you are digging in here on these ugly 
issues. Thanks for doing it.

 Add labels to VisibilityLabelsCache asynchronously causes TestShell flakey
 --

 Key: HBASE-13084
 URL: https://issues.apache.org/jira/browse/HBASE-13084
 Project: HBase
  Issue Type: Bug
  Components: test
Reporter: zhangduo
Assignee: zhangduo
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-13084-addendum.patch, HBASE-13084.patch, 
 HBASE-13084_1.patch, HBASE-13084_2.patch, HBASE-13084_2.patch, 
 HBASE-13084_2.patch, HBASE-13084_2.patch, HBASE-13084_2_disable_test.patch


 As discussed in HBASE-12953, we found this error in PreCommit log
 https://builds.apache.org/job/PreCommit-HBASE-Build/12918/artifact/hbase-shell/target/surefire-reports/org.apache.hadoop.hbase.client.TestShell-output.txt
 {noformat}
   1) Error:
 test_The_get/put_methods_should_work_for_data_written_with_Visibility(Hbase::VisibilityLabelsAdminMethodsTest):
 ArgumentError: org.apache.hadoop.hbase.DoNotRetryIOException: 
 org.apache.hadoop.hbase.security.visibility.InvalidLabelException: Label 
 'TEST_VISIBILITY' doesn't exists
   at 
 org.apache.hadoop.hbase.security.visibility.VisibilityController.setAuths(VisibilityController.java:808)
   at 
 org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService$1.setAuths(VisibilityLabelsProtos.java:6036)
   at 
 org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService.callMethod(VisibilityLabelsProtos.java:6219)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:6867)
   at 
 org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1707)
   at 
 org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1689)
   at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31309)
   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2038)
   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
   at 
 org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
   at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
   at java.lang.Thread.run(Thread.java:744)
 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-shell/src/main/ruby/hbase/visibility_labels.rb:84:in
  `set_auths'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:77:in 
 `test_The_get/put_methods_should_work_for_data_written_with_Visibility'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
   2) Error:
 test_The_set/clear_methods_should_work_with_authorizations(Hbase::VisibilityLabelsAdminMethodsTest):
 ArgumentError: No authentication set for the given user jenkins
 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-shell/src/main/ruby/hbase/visibility_labels.rb:97:in
  `get_auths'
 ./src/test/ruby/hbase/visibility_labels_admin_test.rb:57:in 
 `test_The_set/clear_methods_should_work_with_authorizations'
 org/jruby/RubyProc.java:270:in `call'
 org/jruby/RubyKernel.java:2105:in `send'
 org/jruby/RubyArray.java:1620:in `each'
 org/jruby/RubyArray.java:1620:in `each'
 {noformat}
 This is the test code
 {code:title=visibility_labels_admin_test.rb}
   label = 'TEST_VISIBILITY'
   user = org.apache.hadoop.hbase.security.User.getCurrent().getName();
   visibility_admin.add_labels(label)
   visibility_admin.set_auths(user, label)
 {code}
 It says 'label does not exists' when calling set_auths.
 Then I add some ugly logs in  DefaultVisibilityLabelServiceImpl and 
 VisibilityLabelsCache.
 {code:title=DefaultVisibilityLabelServiceImpl.java}
   public OperationStatus[] addLabels(Listbyte[] labels) throws IOException {
 ...
 if (mutateLabelsRegion(puts, finalOpStatus)) {
   updateZk(true);
 }
 for (byte[] label : labels) {
   String labelStr = Bytes.toString(label);
   LOG.info(labelStr + = + 
 this.labelsCache.getLabelOrdinal(labelStr));
 }
 ...
   }
 {code}
 {code:title=VisibilityLabelsCache.java}
   public void refreshLabelsCache(byte[] data) throws

[jira] [Commented] (HBASE-13122) Improve efficiency for return codes of some filters