date:20150309


[ 
https://issues.apache.org/jira/browse/HBASE-13172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352877#comment-14352877
 ] 

Hadoop QA commented on HBASE-13172:
---

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12703399/HBASE-13172-branch-1.patch
  against branch-1 branch at commit 0fba20471e66b8cc1b1152a6ae5965e09ebbe6ce.
  ATTACHMENT ID: 12703399

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 4 new 
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.1 2.5.2 2.6.0)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13142//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13142//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13142//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13142//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13142//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13142//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13142//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13142//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13142//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13142//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13142//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13142//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13142//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13142//console

This message is automatically generated.

 TestDistributedLogSplitting.testThreeRSAbort fails several times on branch-1
 

 Key: HBASE-13172
 URL: https://issues.apache.org/jira/browse/HBASE-13172
 Project: HBase
  Issue Type: Bug
  Components: test
Affects Versions: 1.1.0
Reporter: zhangduo
Assignee: zhangduo
 Attachments: HBASE-13172-branch-1.patch


 The direct reason is we are stuck in ServerManager.isServerReachable.
 https://builds.apache.org/job/HBase-1.1/253/testReport/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testThreeRSAbort/
 {noformat}
 2015-03-06 04:06:19,430 DEBUG [AM.-pool300-t1] master.ServerManager(855): 
 Couldn't reach asf906.gq1.ygridcore.net,59366,1425614770146, try=0 of 10
 2015-03-06 04:07:10,545 DEBUG [AM.-pool300-t1] master.ServerManager(855): 
 Couldn't reach asf906.gq1.ygridcore.net,59366,1425614770146, try=9 of 10
 {noformat}
 The interval between first and last retry log is about 1 minute, and we only 
 wait 1 minute so the test is timeout.
 Still do not know why this happen.
 And at last there are lots of this 
 {noformat}
 2015-03-06 04:07:21,529 DEBUG [AM.-pool300-t1] master.ServerManager(855):

[jira] [Commented] (HBASE-12990) MetaScanner should be replaced by MetaTableAccessor


[ 
https://issues.apache.org/jira/browse/HBASE-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352879#comment-14352879
 ] 

Hudson commented on HBASE-12990:


FAILURE: Integrated in HBase-TRUNK #6224 (See 
[https://builds.apache.org/job/HBase-TRUNK/6224/])
HBASE-12990 MetaScanner should be replaced by MetaTableAccessor (octo47: rev 
948746ce4ed3bd174927c41bd4884cad70d693ef)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/HRegionLocator.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
* conf/log4j.properties
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionSizeCalculator.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRegionPlacement.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMetaScanner.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java
* hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RegionsResource.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexerFlushCache.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/namespace/NamespaceStateManager.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/handler/TestEnableTableHandler.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ModifyTableHandler.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/TestMetaTableAccessor.java
* 
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientNoCluster.java
* bin/region_status.rb
* hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java


 MetaScanner should be replaced by MetaTableAccessor
 ---

 Key: HBASE-12990
 URL: https://issues.apache.org/jira/browse/HBASE-12990
 Project: HBase
  Issue Type: Improvement
  Components: Client
Affects Versions: 2.0.0
Reporter: Andrey Stepachev
Assignee: Andrey Stepachev
 Attachments: HBASE-12990.patch, HBASE-12990.v2.patch, 
 HBASE-12990.v3.patch, HBASE-12990.v4.patch, HBASE-12990.v5.patch, 
 HBASE-12990.v5.patch, HBASE-12990.v5.patch, HBASE-12990.v6.patch, 
 HBASE-12990.v7.patch, HBASE-12990.v7.patch, HBASE-12990.v7.patch, 
 HBASE-12990.v8.patch


 MetaScanner and MetaTableAccessor do similar things, but seems they tend to 
 diverge. Let's have only one thing to enquiry META.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-6877) Coprocessor exec result is incorrect when region is in splitting

2015-03-09 Thread Ploder (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352976#comment-14352976
 ] 

Ploder commented on HBASE-6877:
---

We ran into exactly the same issue. It doesn't affect Scans, because they run 
server-side. It also does not affect Gets, because they issue multiple 
requests. The faulty code inside the rpc channel of the hbase client resubmits 
a *single* call to a region if the region can not be found anymore on the 
region server (because it was splitted in this case). However when a region has 
been splitted a single call to the region server will not suffice, hence this 
problem. 

 Coprocessor exec result is incorrect when region is in splitting 
 -

 Key: HBASE-6877
 URL: https://issues.apache.org/jira/browse/HBASE-6877
 Project: HBase
  Issue Type: Bug
  Components: Coprocessors
Affects Versions: 0.94.1
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Attachments: HBASE-6877.patch


 When we execute the coprocessor, we will called HTable#getStartKeysInRange 
 first and get the Keys to exec coprocessor,
 if then some regions are split before execCoprocessor RPC, the Keys are 
 something wrong now, and the result we get is not integrated, 
 for example:
 parent region is split into daughter region A and daughter region B,
 we executed coprocessor on the parent region, but the result data is only 
 daughter region A or daughter region B



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-11425) Cell/DBB end-to-end on the read-path

2015-03-09 Thread ramkrishna.s.vasudevan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-11425:
---
Attachment: Offheap reads in HBase using BBs_final.pdf

A document explaining the motive, the design considerations, the reason for 
arriving at BB and the Cell level APIS changes required for supporting offheap 
memory in HBase's read path.
We will be uploading a patch ported to trunk shortly by the end of this week or 
early next week. Along with some perf results. 
Request feedback/comments on the doc and the approach.

 Cell/DBB end-to-end on the read-path
 

 Key: HBASE-11425
 URL: https://issues.apache.org/jira/browse/HBASE-11425
 Project: HBase
  Issue Type: Umbrella
  Components: regionserver, Scanners
Affects Versions: 0.99.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
 Attachments: Offheap reads in HBase using BBs_final.pdf


 Umbrella jira to make sure we can have blocks cached in offheap backed cache. 
 In the entire read path, we can refer to this offheap buffer and avoid onheap 
 copying.
 The high level items I can identify as of now are
 1. Avoid the array() call on BB in read path.. (This is there in many 
 classes. We can handle class by class)
 2. Support Buffer based getter APIs in cell.  In read path we will create a 
 new Cell with backed by BB. Will need in CellComparator, Filter (like SCVF), 
 CPs etc.
 3. Avoid KeyValue.ensureKeyValue() calls in read path - This make byte copy.
 4. Remove all CP hooks (which are already deprecated) which deal with KVs.  
 (In read path)
 Will add subtasks under this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11425) Cell/DBB end-to-end on the read-path

2015-03-09 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352996#comment-14352996
 ] 

Anoop Sam John commented on HBASE-11425:


The ref count and increment/decrement happens in one place.  You can get more 
when seeing code.
Yes it should be HBASE-13412 yet. Thanks for correcting.

 Cell/DBB end-to-end on the read-path
 

 Key: HBASE-11425
 URL: https://issues.apache.org/jira/browse/HBASE-11425
 Project: HBase
  Issue Type: Umbrella
  Components: regionserver, Scanners
Affects Versions: 0.99.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
 Attachments: Offheap reads in HBase using BBs_final.pdf


 Umbrella jira to make sure we can have blocks cached in offheap backed cache. 
 In the entire read path, we can refer to this offheap buffer and avoid onheap 
 copying.
 The high level items I can identify as of now are
 1. Avoid the array() call on BB in read path.. (This is there in many 
 classes. We can handle class by class)
 2. Support Buffer based getter APIs in cell.  In read path we will create a 
 new Cell with backed by BB. Will need in CellComparator, Filter (like SCVF), 
 CPs etc.
 3. Avoid KeyValue.ensureKeyValue() calls in read path - This make byte copy.
 4. Remove all CP hooks (which are already deprecated) which deal with KVs.  
 (In read path)
 Will add subtasks under this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11425) Cell/DBB end-to-end on the read-path

2015-03-09 Thread zhangduo (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353001#comment-14353001
 ] 

zhangduo commented on HBASE-11425:
--

[~anoopsamjohn] increment/decrement in one place sounds good to be.

 Cell/DBB end-to-end on the read-path
 

 Key: HBASE-11425
 URL: https://issues.apache.org/jira/browse/HBASE-11425
 Project: HBase
  Issue Type: Umbrella
  Components: regionserver, Scanners
Affects Versions: 0.99.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
 Attachments: Offheap reads in HBase using BBs_final.pdf


 Umbrella jira to make sure we can have blocks cached in offheap backed cache. 
 In the entire read path, we can refer to this offheap buffer and avoid onheap 
 copying.
 The high level items I can identify as of now are
 1. Avoid the array() call on BB in read path.. (This is there in many 
 classes. We can handle class by class)
 2. Support Buffer based getter APIs in cell.  In read path we will create a 
 new Cell with backed by BB. Will need in CellComparator, Filter (like SCVF), 
 CPs etc.
 3. Avoid KeyValue.ensureKeyValue() calls in read path - This make byte copy.
 4. Remove all CP hooks (which are already deprecated) which deal with KVs.  
 (In read path)
 Will add subtasks under this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13179) TestMasterObserver deleteTable is flaky

[
https://issues.apache.org/jira/browse/HBASE-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Matteo Bertozzi updated HBASE-13179:

Attachment: HBASE-13179-v0.patch

TestMasterObserver deleteTable is flaky
---

Key: HBASE-13179
URL: https://issues.apache.org/jira/browse/HBASE-13179
Project: HBase
Issue Type: Test
Components: test
Affects Versions: 1.0.0, 0.98.10.1
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
Attachments: HBASE-13179-v0.patch

TestMasterObserver fails when deleteTable() takes more time to complete the
last steps.
{code}
java.lang.AssertionError: Delete table handler should be called.
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.hbase.coprocessor.TestMasterObserver.testTableOperations(TestMasterObserver.java:1283)
{code}
The problem is the same as the one in createTable() and it is because the
sync version of the Admin operation is not sync, but relies on the last
operation on the server e.g. delete meta.
but the post-operation method of the coprocessor is called after meta is
deleted..
long story short, the client is not really sync and it will be fixed by
HBASE-12439. so for now we should have the same fix we have for createTable
which is using a latch.
(note: there are other tests failing for this reason e.g. AccessController,
NamespaceAuditor, ... but I'll fix them in another patch since we have
already a workaround in TestMasterObserver)

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-13179) TestMasterObserver deleteTable is flaky

Matteo Bertozzi created HBASE-13179:
---

 Summary: TestMasterObserver deleteTable is flaky
 Key: HBASE-13179
 URL: https://issues.apache.org/jira/browse/HBASE-13179
 Project: HBase
  Issue Type: Test
  Components: test
Affects Versions: 0.98.10.1, 1.0.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
 Attachments: HBASE-13179-v0.patch

TestMasterObserver fails when deleteTable() takes more time to complete the 
last steps.
{code}
java.lang.AssertionError: Delete table handler should be called.
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at 
org.apache.hadoop.hbase.coprocessor.TestMasterObserver.testTableOperations(TestMasterObserver.java:1283)
{code}
The problem is the same as the one in createTable() and it is because the sync 
version of the Admin operation is not sync, but relies on the last operation on 
the server e.g. delete meta. 
but the post-operation method of the coprocessor is called after meta is 
deleted.. 
long story short, the client is not really sync and it will be fixed by 
HBASE-12439. so for now we should have the same fix we have for createTable 
which is using a latch.

(note: there are other tests failing for this reason e.g. AccessController, 
NamespaceAuditor, ... but I'll fix them in another patch since we have already 
a workaround in TestMasterObserver)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13179) TestMasterObserver deleteTable is flaky

2015-03-09 Thread Ted Yu (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353065#comment-14353065
]

Ted Yu commented on HBASE-13179:

lgtm

TestMasterObserver deleteTable is flaky
---

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13173) Azure Storage FileSystem rename operations are throttled too aggressively to complete HBase WAL archiving.

2015-03-09 Thread Chris Nauroth (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-13173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Nauroth updated HBASE-13173:
--
Summary: Azure Storage FileSystem rename operations are throttled too 
aggressively to complete HBase WAL archiving.  (was: HBase gives up retries 
after being throttled by Azure storage)

[~apurtell], although the issue title mentions HBase, the root cause for this 
problem actually resides in Hadoop Common, specifically the Azure Storage 
{{FileSystem}} implementation.  I have updated the issue title to try to make 
this clearer.  It looks like I don't have access to move HBASE issues, so would 
you mind moving this back to HADOOP?  Thanks!

 Azure Storage FileSystem rename operations are throttled too aggressively to 
 complete HBase WAL archiving.
 --

 Key: HBASE-13173
 URL: https://issues.apache.org/jira/browse/HBASE-13173
 Project: HBase
  Issue Type: Bug
Reporter: Duo Xu
 Attachments: HADOOP-11681.01.patch


 One of our customers' production HBase clusters was periodically throttled by 
 Azure storage, when HBase was archiving old WALs. HMaster aborted the region 
 server and tried to restart it.
 However, since the cluster was still being throttled by Azure storage, the 
 upcoming distributed log splitting also failed. Sometimes hbase:meta table 
 was on this region server and finally showed offline, which cause the whole 
 cluster in bad state.
 {code}
 2015-03-01 18:36:45,623 ERROR org.apache.hadoop.hbase.master.HMaster: Region 
 server 
 workernode4.hbaseproddb4001.f5.internal.cloudapp.net,60020,1424845421044 
 reported a fatal error:
 ABORTING region server 
 workernode4.hbaseproddb4001.f5.internal.cloudapp.net,60020,1424845421044: IOE 
 in log roller
 Cause:
 org.apache.hadoop.fs.azure.AzureException: 
 com.microsoft.windowsazure.storage.StorageException: The server is busy.
   at 
 org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2446)
   at 
 org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2367)
   at 
 org.apache.hadoop.fs.azurenative.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1960)
   at 
 org.apache.hadoop.hbase.util.FSUtils.renameAndSetModifyTime(FSUtils.java:1719)
   at 
 org.apache.hadoop.hbase.regionserver.wal.FSHLog.archiveLogFile(FSHLog.java:798)
   at 
 org.apache.hadoop.hbase.regionserver.wal.FSHLog.cleanOldLogs(FSHLog.java:656)
   at 
 org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:593)
   at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:97)
   at java.lang.Thread.run(Thread.java:745)
 Caused by: com.microsoft.windowsazure.storage.StorageException: The server is 
 busy.
   at 
 com.microsoft.windowsazure.storage.StorageException.translateException(StorageException.java:163)
   at 
 com.microsoft.windowsazure.storage.core.StorageRequest.materializeException(StorageRequest.java:306)
   at 
 com.microsoft.windowsazure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:229)
   at 
 com.microsoft.windowsazure.storage.blob.CloudBlob.startCopyFromBlob(CloudBlob.java:762)
   at 
 org.apache.hadoop.fs.azurenative.StorageInterfaceImpl$CloudBlobWrapperImpl.startCopyFromBlob(StorageInterfaceImpl.java:350)
   at 
 org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2439)
   ... 8 more
 2015-03-01 18:43:29,072 ERROR org.apache.hadoop.hbase.executor.EventHandler: 
 Caught throwable while processing event M_META_SERVER_SHUTDOWN
 java.io.IOException: failed log splitting for 
 workernode13.hbaseproddb4001.f5.internal.cloudapp.net,60020,1424845307901, 
 will retry
   at 
 org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.process(MetaServerShutdownHandler.java:71)
   at 
 org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
   at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:745)
 Caused by: org.apache.hadoop.fs.azure.AzureException: 
 com.microsoft.windowsazure.storage.StorageException: The server is busy.
   at 
 org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2446)
   at 
 org.apache.hadoop.fs.azurenative.NativeAzureFileSystem$FolderRenamePending.execute(NativeAzureFileSystem.java:393)
   at 
 org.apache.hadoop.fs.azurenative.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1973)
   at

[jira] [Updated] (HBASE-13179) TestMasterObserver deleteTable is flaky

[
https://issues.apache.org/jira/browse/HBASE-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Matteo Bertozzi updated HBASE-13179:

Status: Patch Available (was: Open)

TestMasterObserver deleteTable is flaky
---

Key: HBASE-13179
URL: https://issues.apache.org/jira/browse/HBASE-13179
Project: HBase
Issue Type: Test
Components: test
Affects Versions: 0.98.10.1, 1.0.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
Attachments: HBASE-13179-v0.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13172) TestDistributedLogSplitting.testThreeRSAbort fails several times on branch-1

2015-03-09 Thread Ted Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353038#comment-14353038
 ] 

Ted Yu commented on HBASE-13172:


lgtm

 TestDistributedLogSplitting.testThreeRSAbort fails several times on branch-1
 

 Key: HBASE-13172
 URL: https://issues.apache.org/jira/browse/HBASE-13172
 Project: HBase
  Issue Type: Bug
  Components: test
Affects Versions: 1.1.0
Reporter: zhangduo
Assignee: zhangduo
 Attachments: HBASE-13172-branch-1.patch


 The direct reason is we are stuck in ServerManager.isServerReachable.
 https://builds.apache.org/job/HBase-1.1/253/testReport/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testThreeRSAbort/
 {noformat}
 2015-03-06 04:06:19,430 DEBUG [AM.-pool300-t1] master.ServerManager(855): 
 Couldn't reach asf906.gq1.ygridcore.net,59366,1425614770146, try=0 of 10
 2015-03-06 04:07:10,545 DEBUG [AM.-pool300-t1] master.ServerManager(855): 
 Couldn't reach asf906.gq1.ygridcore.net,59366,1425614770146, try=9 of 10
 {noformat}
 The interval between first and last retry log is about 1 minute, and we only 
 wait 1 minute so the test is timeout.
 Still do not know why this happen.
 And at last there are lots of this 
 {noformat}
 2015-03-06 04:07:21,529 DEBUG [AM.-pool300-t1] master.ServerManager(855): 
 Couldn't reach asf906.gq1.ygridcore.net,59366,1425614770146, try=9 of 10
 org.apache.hadoop.hbase.ipc.StoppedRpcClientException
   at 
 org.apache.hadoop.hbase.ipc.RpcClientImpl.getConnection(RpcClientImpl.java:1261)
   at 
 org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1146)
   at 
 org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
   at 
 org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
   at 
 org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.getServerInfo(AdminProtos.java:22031)
   at 
 org.apache.hadoop.hbase.protobuf.ProtobufUtil.getServerInfo(ProtobufUtil.java:1797)
   at 
 org.apache.hadoop.hbase.master.ServerManager.isServerReachable(ServerManager.java:850)
   at 
 org.apache.hadoop.hbase.master.RegionStates.isServerDeadAndNotProcessed(RegionStates.java:843)
   at 
 org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1969)
   at 
 org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1576)
   at 
 org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48)
   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:744)
 {noformat}
 I think the problem is here
 {code:title=ServerManager.java}
 while (retryCounter.shouldRetry()) {
 ...
 try {
   retryCounter.sleepUntilNextRetry();
 } catch(InterruptedException ie) {
   Thread.currentThread().interrupt();
 }
 ...
 }
 {code}
 We need to break out of the while loop when getting InterruptedException, not 
 just mark current thread as interrupted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13168) Backport HBASE-12590 A solution for data skew in HBase-Mapreduce Job

2015-03-09 Thread Nick Dimiduk (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353149#comment-14353149
 ] 

Nick Dimiduk commented on HBASE-13168:
--

Thanks for the back port [~tedyu]. Anything significant change from the master 
patch, or was is pretty straightforward cherry-pick apply?

nit: reading the patch, indentation appears off:

{noformat}
+} else {
+  return splits;
+}
 } finally {
{noformat}

+1 for fixup on commit.

Do you have time for a 0.98 back port also? I can pick it up no problem.

FYI [~enis], [~apurtell]

 Backport HBASE-12590 A solution for data skew in HBase-Mapreduce Job
 --

 Key: HBASE-13168
 URL: https://issues.apache.org/jira/browse/HBASE-13168
 Project: HBase
  Issue Type: Improvement
  Components: mapreduce
Reporter: Nick Dimiduk
Assignee: Ted Yu
 Fix For: 1.1.0

 Attachments: 13168-branch-1.patch, 13168-branch-1.patch


 We should bring this back into active release branches. Seems like it 
 shouldn't break any API compatibility, but should be checked more closely.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13168) Backport HBASE-12590 A solution for data skew in HBase-Mapreduce Job

2015-03-09 Thread Nick Dimiduk (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-13168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk updated HBASE-13168:
-
Fix Version/s: 0.98.13

 Backport HBASE-12590 A solution for data skew in HBase-Mapreduce Job
 --

 Key: HBASE-13168
 URL: https://issues.apache.org/jira/browse/HBASE-13168
 Project: HBase
  Issue Type: Improvement
  Components: mapreduce
Reporter: Nick Dimiduk
Assignee: Ted Yu
 Fix For: 1.1.0, 0.98.13

 Attachments: 13168-branch-1.patch, 13168-branch-1.patch


 We should bring this back into active release branches. Seems like it 
 shouldn't break any API compatibility, but should be checked more closely.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13168) Backport HBASE-12590 A solution for data skew in HBase-Mapreduce Job

2015-03-09 Thread Nick Dimiduk (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353151#comment-14353151
 ] 

Nick Dimiduk commented on HBASE-13168:
--

[~tedyu] -- actually, any reason this cannot go to 1.0.x ?

 Backport HBASE-12590 A solution for data skew in HBase-Mapreduce Job
 --

 Key: HBASE-13168
 URL: https://issues.apache.org/jira/browse/HBASE-13168
 Project: HBase
  Issue Type: Improvement
  Components: mapreduce
Reporter: Nick Dimiduk
Assignee: Ted Yu
 Fix For: 1.1.0, 0.98.13

 Attachments: 13168-branch-1.patch, 13168-branch-1.patch


 We should bring this back into active release branches. Seems like it 
 shouldn't break any API compatibility, but should be checked more closely.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13168) Backport HBASE-12590 A solution for data skew in HBase-Mapreduce Job

2015-03-09 Thread Ted Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353158#comment-14353158
 ] 

Ted Yu commented on HBASE-13168:


This is straight forward port.

bq. indentation appears off
This was to keep the large code block in try block intact.

bq. any reason this cannot go to 1.0.x ?
Nothing I can think of.

 Backport HBASE-12590 A solution for data skew in HBase-Mapreduce Job
 --

 Key: HBASE-13168
 URL: https://issues.apache.org/jira/browse/HBASE-13168
 Project: HBase
  Issue Type: Improvement
  Components: mapreduce
Reporter: Nick Dimiduk
Assignee: Ted Yu
 Fix For: 1.1.0, 0.98.13

 Attachments: 13168-branch-1.patch, 13168-branch-1.patch


 We should bring this back into active release branches. Seems like it 
 shouldn't break any API compatibility, but should be checked more closely.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13179) TestMasterObserver deleteTable is flaky


[ 
https://issues.apache.org/jira/browse/HBASE-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353239#comment-14353239
 ] 

Hadoop QA commented on HBASE-13179:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12703433/HBASE-13179-v0.patch
  against master branch at commit 948746ce4ed3bd174927c41bd4884cad70d693ef.
  ATTACHMENT ID: 12703433

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 3 new 
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.1 2.5.2 2.6.0)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

{color:red}-1 site{color}.  The patch appears to cause mvn site goal to 
fail.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13143//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13143//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13143//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13143//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13143//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13143//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13143//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13143//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13143//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13143//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13143//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13143//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13143//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/13143//console

This message is automatically generated.

 TestMasterObserver deleteTable is flaky
 ---

 Key: HBASE-13179
 URL: https://issues.apache.org/jira/browse/HBASE-13179
 Project: HBase
  Issue Type: Test
  Components: test
Affects Versions: 1.0.0, 0.98.10.1
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
 Attachments: HBASE-13179-v0.patch


 TestMasterObserver fails when deleteTable() takes more time to complete the 
 last steps.
 {code}
 java.lang.AssertionError: Delete table handler should be called.
   at org.junit.Assert.fail(Assert.java:88)
   at org.junit.Assert.assertTrue(Assert.java:41)
   at 
 org.apache.hadoop.hbase.coprocessor.TestMasterObserver.testTableOperations(TestMasterObserver.java:1283)
 {code}
 The problem is the same as the one in createTable() and it is because the 
 sync version of the Admin operation is not sync, but relies on the last 
 operation on the server e.g. delete meta. 
 but the post-operation method of the coprocessor is called after meta is 
 deleted.. 
 long story short, the client is not really sync and it will be fixed by 
 HBASE-12439. so for now we should have the same fix we have for createTable 
 which is

[jira] [Commented] (HBASE-13174) Apply HBASE-11804 to Windows scripts


[ 
https://issues.apache.org/jira/browse/HBASE-13174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353338#comment-14353338
 ] 

Enis Soztutar commented on HBASE-13174:
---

+1. 

 Apply HBASE-11804 to Windows scripts
 

 Key: HBASE-13174
 URL: https://issues.apache.org/jira/browse/HBASE-13174
 Project: HBase
  Issue Type: Bug
  Components: scripts
Affects Versions: 1.0.0
Reporter: Lars George
 Attachments: HBASE-13174.patch


 HBASE-11804 was only applied to the Linux scripts. Do the same for Windows. 
 Need a +1 here from the Windows supporters, since this patch is a one liner 
 but I need to know if it is OK to do on Windows. I'd say yes, but better ask 
 first. cc [~enis]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13173) Azure Storage FileSystem rename operations are throttled too aggressively to complete HBase WAL archiving.


[ 
https://issues.apache.org/jira/browse/HBASE-13173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353180#comment-14353180
 ] 

Andrew Purtell commented on HBASE-13173:


Sure, moving it back! Thanks [~cnauroth]


 Azure Storage FileSystem rename operations are throttled too aggressively to 
 complete HBase WAL archiving.
 --

 Key: HBASE-13173
 URL: https://issues.apache.org/jira/browse/HBASE-13173
 Project: HBase
  Issue Type: Bug
Reporter: Duo Xu
 Attachments: HADOOP-11681.01.patch


 One of our customers' production HBase clusters was periodically throttled by 
 Azure storage, when HBase was archiving old WALs. HMaster aborted the region 
 server and tried to restart it.
 However, since the cluster was still being throttled by Azure storage, the 
 upcoming distributed log splitting also failed. Sometimes hbase:meta table 
 was on this region server and finally showed offline, which cause the whole 
 cluster in bad state.
 {code}
 2015-03-01 18:36:45,623 ERROR org.apache.hadoop.hbase.master.HMaster: Region 
 server 
 workernode4.hbaseproddb4001.f5.internal.cloudapp.net,60020,1424845421044 
 reported a fatal error:
 ABORTING region server 
 workernode4.hbaseproddb4001.f5.internal.cloudapp.net,60020,1424845421044: IOE 
 in log roller
 Cause:
 org.apache.hadoop.fs.azure.AzureException: 
 com.microsoft.windowsazure.storage.StorageException: The server is busy.
   at 
 org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2446)
   at 
 org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2367)
   at 
 org.apache.hadoop.fs.azurenative.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1960)
   at 
 org.apache.hadoop.hbase.util.FSUtils.renameAndSetModifyTime(FSUtils.java:1719)
   at 
 org.apache.hadoop.hbase.regionserver.wal.FSHLog.archiveLogFile(FSHLog.java:798)
   at 
 org.apache.hadoop.hbase.regionserver.wal.FSHLog.cleanOldLogs(FSHLog.java:656)
   at 
 org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:593)
   at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:97)
   at java.lang.Thread.run(Thread.java:745)
 Caused by: com.microsoft.windowsazure.storage.StorageException: The server is 
 busy.
   at 
 com.microsoft.windowsazure.storage.StorageException.translateException(StorageException.java:163)
   at 
 com.microsoft.windowsazure.storage.core.StorageRequest.materializeException(StorageRequest.java:306)
   at 
 com.microsoft.windowsazure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:229)
   at 
 com.microsoft.windowsazure.storage.blob.CloudBlob.startCopyFromBlob(CloudBlob.java:762)
   at 
 org.apache.hadoop.fs.azurenative.StorageInterfaceImpl$CloudBlobWrapperImpl.startCopyFromBlob(StorageInterfaceImpl.java:350)
   at 
 org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2439)
   ... 8 more
 2015-03-01 18:43:29,072 ERROR org.apache.hadoop.hbase.executor.EventHandler: 
 Caught throwable while processing event M_META_SERVER_SHUTDOWN
 java.io.IOException: failed log splitting for 
 workernode13.hbaseproddb4001.f5.internal.cloudapp.net,60020,1424845307901, 
 will retry
   at 
 org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.process(MetaServerShutdownHandler.java:71)
   at 
 org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
   at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:745)
 Caused by: org.apache.hadoop.fs.azure.AzureException: 
 com.microsoft.windowsazure.storage.StorageException: The server is busy.
   at 
 org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2446)
   at 
 org.apache.hadoop.fs.azurenative.NativeAzureFileSystem$FolderRenamePending.execute(NativeAzureFileSystem.java:393)
   at 
 org.apache.hadoop.fs.azurenative.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1973)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.getLogDirs(MasterFileSystem.java:319)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:406)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.splitMetaLog(MasterFileSystem.java:302)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.splitMetaLog(MasterFileSystem.java:293)
   at 
 org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.process(MetaServerShutdownHandler.java:64)
   ... 4 more

[jira] [Created] (HBASE-13180) TestRegionReplicas.testGetOnTargetRegionReplica fails frequently.

2015-03-09 Thread Srikanth Srungarapu (JIRA)

Srikanth Srungarapu created HBASE-13180:
---

 Summary: TestRegionReplicas.testGetOnTargetRegionReplica fails 
frequently.
 Key: HBASE-13180
 URL: https://issues.apache.org/jira/browse/HBASE-13180
 Project: HBase
  Issue Type: Bug
Reporter: Srikanth Srungarapu
Priority: Minor


Noticing the test failure frequently on internal test rig. 
{code}
FAILED:  
org.apache.hadoop.hbase.regionserver.TestRegionReplicas.testGetOnTargetRegionReplicaError
 Message:
1

Stack Trace:
java.lang.ArrayIndexOutOfBoundsException: 1
at 
org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas$ResultBoundedCompletionService.submit(RpcRetryingCallerWithReadReplicas.java:420)
at 
org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.addCallsForReplica(RpcRetryingCallerWithReadReplicas.java:280)
at 
org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.call(RpcRetryingCallerWithReadReplicas.java:199)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:890)
at 
org.apache.hadoop.hbase.regionserver.TestRegionReplicas.testGetOnTargetRegionReplica(TestRegionReplicas.java:193)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-13181) TestHRegionReplayEvents.testReplayBulkLoadEvent fails frequently.

2015-03-09 Thread Srikanth Srungarapu (JIRA)

Srikanth Srungarapu created HBASE-13181:
---

 Summary: TestHRegionReplayEvents.testReplayBulkLoadEvent fails 
frequently.
 Key: HBASE-13181
 URL: https://issues.apache.org/jira/browse/HBASE-13181
 Project: HBase
  Issue Type: Bug
Reporter: Srikanth Srungarapu
Priority: Minor


Noticing the test failures frequently on internal test rig:
{code}
FAILED:  
org.apache.hadoop.hbase.regionserver.TestHRegionReplayEvents.testReplayBulkLoadEvent

Error Message:
3 exceptions [org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem 
reading HFile Trailer from file 
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/15e393f3-1a5c-45e4-b048-fd647956f1f7,
 org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading HFile 
Trailer from file 
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/9bdfbfb1-8a72-4907-9f91-5570d5accfcd,
 org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading HFile 
Trailer from file 
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/fe63364b-906a-4085-85b6-605aee4d41c2]

Stack Trace:
org.apache.hadoop.io.MultipleIOException: 3 exceptions 
[org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading HFile 
Trailer from file 
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/15e393f3-1a5c-45e4-b048-fd647956f1f7,
 org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading HFile 
Trailer from file 
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/9bdfbfb1-8a72-4907-9f91-5570d5accfcd,
 org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading HFile 
Trailer from file 
/var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/fe63364b-906a-4085-85b6-605aee4d41c2]
at 
org.apache.hadoop.io.MultipleIOException.createIOException(MultipleIOException.java:52)
at 
org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:4910)
at 
org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:4855)
at 
org.apache.hadoop.hbase.regionserver.TestHRegionReplayEvents.testReplayBulkLoadEvent(TestHRegionReplayEvents.java:1159)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner

2015-03-09 Thread Lars Hofhansl (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353288#comment-14353288
 ] 

Lars Hofhansl commented on HBASE-13082:
---

To fix TestFromClientSideWithCoprocessor I need to change the coprocessor API 
for preStoreScannerOpen in order to pass the correct lock object in.


 Coarsen StoreScanner locks to RegionScanner
 ---

 Key: HBASE-13082
 URL: https://issues.apache.org/jira/browse/HBASE-13082
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Attachments: 13082-test.txt, 13082-v2.txt, 13082-v3.txt, 
 13082-v4.txt, 13082.txt, 13082.txt, gc.png, gc.png, gc.png, hits.png, 
 next.png, next.png


 Continuing where HBASE-10015 left of.
 We can avoid locking (and memory fencing) inside StoreScanner by deferring to 
 the lock already held by the RegionScanner.
 In tests this shows quite a scan improvement and reduced CPU (the fences make 
 the cores wait for memory fetches).
 There are some drawbacks too:
 * All calls to RegionScanner need to be remain synchronized
 * Implementors of coprocessors need to be diligent in following the locking 
 contract. For example Phoenix does not lock RegionScanner.nextRaw() and 
 required in the documentation (not picking on Phoenix, this one is my fault 
 as I told them it's OK)
 * possible starving of flushes and compaction with heavy read load. 
 RegionScanner operations would keep getting the locks and the 
 flushes/compactions would not be able finalize the set of files.
 I'll have a patch soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13179) TestMasterObserver deleteTable is flaky

[
https://issues.apache.org/jira/browse/HBASE-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353287#comment-14353287
]

stack commented on HBASE-13179:
---

Go for it [~mbertozzi] Site failure not related.

TestMasterObserver deleteTable is flaky
---

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13172) TestDistributedLogSplitting.testThreeRSAbort fails several times on branch-1


[ 
https://issues.apache.org/jira/browse/HBASE-13172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353323#comment-14353323
 ] 

stack commented on HBASE-13172:
---

Patch makes sense to me. Asking [~jxiang] for input.


 TestDistributedLogSplitting.testThreeRSAbort fails several times on branch-1
 

 Key: HBASE-13172
 URL: https://issues.apache.org/jira/browse/HBASE-13172
 Project: HBase
  Issue Type: Bug
  Components: test
Affects Versions: 1.1.0
Reporter: zhangduo
Assignee: zhangduo
 Attachments: HBASE-13172-branch-1.patch


 The direct reason is we are stuck in ServerManager.isServerReachable.
 https://builds.apache.org/job/HBase-1.1/253/testReport/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testThreeRSAbort/
 {noformat}
 2015-03-06 04:06:19,430 DEBUG [AM.-pool300-t1] master.ServerManager(855): 
 Couldn't reach asf906.gq1.ygridcore.net,59366,1425614770146, try=0 of 10
 2015-03-06 04:07:10,545 DEBUG [AM.-pool300-t1] master.ServerManager(855): 
 Couldn't reach asf906.gq1.ygridcore.net,59366,1425614770146, try=9 of 10
 {noformat}
 The interval between first and last retry log is about 1 minute, and we only 
 wait 1 minute so the test is timeout.
 Still do not know why this happen.
 And at last there are lots of this 
 {noformat}
 2015-03-06 04:07:21,529 DEBUG [AM.-pool300-t1] master.ServerManager(855): 
 Couldn't reach asf906.gq1.ygridcore.net,59366,1425614770146, try=9 of 10
 org.apache.hadoop.hbase.ipc.StoppedRpcClientException
   at 
 org.apache.hadoop.hbase.ipc.RpcClientImpl.getConnection(RpcClientImpl.java:1261)
   at 
 org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1146)
   at 
 org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
   at 
 org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
   at 
 org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.getServerInfo(AdminProtos.java:22031)
   at 
 org.apache.hadoop.hbase.protobuf.ProtobufUtil.getServerInfo(ProtobufUtil.java:1797)
   at 
 org.apache.hadoop.hbase.master.ServerManager.isServerReachable(ServerManager.java:850)
   at 
 org.apache.hadoop.hbase.master.RegionStates.isServerDeadAndNotProcessed(RegionStates.java:843)
   at 
 org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1969)
   at 
 org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1576)
   at 
 org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48)
   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:744)
 {noformat}
 I think the problem is here
 {code:title=ServerManager.java}
 while (retryCounter.shouldRetry()) {
 ...
 try {
   retryCounter.sleepUntilNextRetry();
 } catch(InterruptedException ie) {
   Thread.currentThread().interrupt();
 }
 ...
 }
 {code}
 We need to break out of the while loop when getting InterruptedException, not 
 just mark current thread as interrupted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-12987) HBCK should print status while scanning over many regions


 [ 
https://issues.apache.org/jira/browse/HBASE-12987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Antonov reassigned HBASE-12987:
---

Assignee: Mikhail Antonov

Grabbing this one

 HBCK should print status while scanning over many regions
 -

 Key: HBASE-12987
 URL: https://issues.apache.org/jira/browse/HBASE-12987
 Project: HBase
  Issue Type: Improvement
  Components: hbck, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
  Labels: beginner

 Running simple commands like hbck -summary on a large table can take some 
 time. We should print some information to let it be known how things are 
 progressing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12586) Task 6 7 from HBASE-9117, delete all public HTable constructors and delete ConnectionManager#{delete,get}Connection


[ 
https://issues.apache.org/jira/browse/HBASE-12586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353638#comment-14353638
 ] 

stack commented on HBASE-12586:
---

[~mantonov] Sounds good. Suggest check how long hadoopqa takes to run.  Compare 
it to your patch that swaps in a bunch of unmanaged uses. Is it much longer? If 
not radically different, can address bad cases in a followup I'd say.

 Task 6  7 from HBASE-9117,  delete all public HTable constructors and delete 
 ConnectionManager#{delete,get}Connection
 --

 Key: HBASE-12586
 URL: https://issues.apache.org/jira/browse/HBASE-12586
 Project: HBase
  Issue Type: Task
Affects Versions: 2.0.0
Reporter: stack
Assignee: Mikhail Antonov
  Labels: beginner, beginners
 Attachments: HBASE-12586.patch, HBASE-12586.patch, HBASE-12586.patch


 Finish cleanup from HBASE-9117 removing old API.  This issue covers tasks 6 
 and 7 from the list here: 
 https://issues.apache.org/jira/browse/HBASE-9117?focusedCommentId=13919716page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13919716
 To be done in master branch only.
 Marked as beginner task.  The idea is straight-forward.  It is just a lot of 
 work going through all tests converting to use new API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13172) TestDistributedLogSplitting.testThreeRSAbort fails several times on branch-1


 [ 
https://issues.apache.org/jira/browse/HBASE-13172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-13172:
--
   Resolution: Fixed
Fix Version/s: 0.98.12
   1.1.0
   1.0.1
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Thanks for all the expert input lads ([~jeffreyz] and [~jxiang]). Thanks for 
patch [~Apache9]  Pushed to 0.98-branch-1.

 TestDistributedLogSplitting.testThreeRSAbort fails several times on branch-1
 

 Key: HBASE-13172
 URL: https://issues.apache.org/jira/browse/HBASE-13172
 Project: HBase
  Issue Type: Bug
  Components: test
Affects Versions: 1.1.0
Reporter: zhangduo
Assignee: zhangduo
 Fix For: 1.0.1, 1.1.0, 0.98.12

 Attachments: HBASE-13172-branch-1.patch


 The direct reason is we are stuck in ServerManager.isServerReachable.
 https://builds.apache.org/job/HBase-1.1/253/testReport/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testThreeRSAbort/
 {noformat}
 2015-03-06 04:06:19,430 DEBUG [AM.-pool300-t1] master.ServerManager(855): 
 Couldn't reach asf906.gq1.ygridcore.net,59366,1425614770146, try=0 of 10
 2015-03-06 04:07:10,545 DEBUG [AM.-pool300-t1] master.ServerManager(855): 
 Couldn't reach asf906.gq1.ygridcore.net,59366,1425614770146, try=9 of 10
 {noformat}
 The interval between first and last retry log is about 1 minute, and we only 
 wait 1 minute so the test is timeout.
 Still do not know why this happen.
 And at last there are lots of this 
 {noformat}
 2015-03-06 04:07:21,529 DEBUG [AM.-pool300-t1] master.ServerManager(855): 
 Couldn't reach asf906.gq1.ygridcore.net,59366,1425614770146, try=9 of 10
 org.apache.hadoop.hbase.ipc.StoppedRpcClientException
   at 
 org.apache.hadoop.hbase.ipc.RpcClientImpl.getConnection(RpcClientImpl.java:1261)
   at 
 org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1146)
   at 
 org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
   at 
 org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
   at 
 org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.getServerInfo(AdminProtos.java:22031)
   at 
 org.apache.hadoop.hbase.protobuf.ProtobufUtil.getServerInfo(ProtobufUtil.java:1797)
   at 
 org.apache.hadoop.hbase.master.ServerManager.isServerReachable(ServerManager.java:850)
   at 
 org.apache.hadoop.hbase.master.RegionStates.isServerDeadAndNotProcessed(RegionStates.java:843)
   at 
 org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1969)
   at 
 org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1576)
   at 
 org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48)
   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:744)
 {noformat}
 I think the problem is here
 {code:title=ServerManager.java}
 while (retryCounter.shouldRetry()) {
 ...
 try {
   retryCounter.sleepUntilNextRetry();
 } catch(InterruptedException ie) {
   Thread.currentThread().interrupt();
 }
 ...
 }
 {code}
 We need to break out of the while loop when getting InterruptedException, not 
 just mark current thread as interrupted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13179) TestMasterObserver deleteTable is flaky


[ 
https://issues.apache.org/jira/browse/HBASE-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353727#comment-14353727
 ] 

Hudson commented on HBASE-13179:


SUCCESS: Integrated in HBase-1.1 #262 (See 
[https://builds.apache.org/job/HBase-1.1/262/])
HBASE-13179 TestMasterObserver deleteTable is flaky (matteo.bertozzi: rev 
5197640c3095c2a0d4ca0b78361fa5645f54a0e2)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java


 TestMasterObserver deleteTable is flaky
 ---

 Key: HBASE-13179
 URL: https://issues.apache.org/jira/browse/HBASE-13179
 Project: HBase
  Issue Type: Test
  Components: test
Affects Versions: 1.0.0, 0.98.10.1
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-13179-v0.patch


 TestMasterObserver fails when deleteTable() takes more time to complete the 
 last steps.
 {code}
 java.lang.AssertionError: Delete table handler should be called.
   at org.junit.Assert.fail(Assert.java:88)
   at org.junit.Assert.assertTrue(Assert.java:41)
   at 
 org.apache.hadoop.hbase.coprocessor.TestMasterObserver.testTableOperations(TestMasterObserver.java:1283)
 {code}
 The problem is the same as the one in createTable() and it is because the 
 sync version of the Admin operation is not sync, but relies on the last 
 operation on the server e.g. delete meta. 
 but the post-operation method of the coprocessor is called after meta is 
 deleted.. 
 long story short, the client is not really sync and it will be fixed by 
 HBASE-12439. so for now we should have the same fix we have for createTable 
 which is using a latch.
 (note: there are other tests failing for this reason e.g. AccessController, 
 NamespaceAuditor, ... but I'll fix them in another patch since we have 
 already a workaround in TestMasterObserver)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13174) Apply HBASE-11804 to Windows scripts

2015-03-09 Thread Lars George (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-13174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars George updated HBASE-13174:

Fix Version/s: 1.1.0
   1.0.1
   2.0.0

 Apply HBASE-11804 to Windows scripts
 

 Key: HBASE-13174
 URL: https://issues.apache.org/jira/browse/HBASE-13174
 Project: HBase
  Issue Type: Bug
  Components: scripts
Affects Versions: 1.0.0
Reporter: Lars George
 Fix For: 2.0.0, 1.0.1, 1.1.0

 Attachments: HBASE-13174.patch


 HBASE-11804 was only applied to the Linux scripts. Do the same for Windows. 
 Need a +1 here from the Windows supporters, since this patch is a one liner 
 but I need to know if it is OK to do on Windows. I'd say yes, but better ask 
 first. cc [~enis]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12586) Task 6 7 from HBASE-9117, delete all public HTable constructors and delete ConnectionManager#{delete,get}Connection

[
https://issues.apache.org/jira/browse/HBASE-12586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353730#comment-14353730
]

Hadoop QA commented on HBASE-12586:
---

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12703480/HBASE-12586.patch
against master branch at commit 948746ce4ed3bd174927c41bd4884cad70d693ef.
ATTACHMENT ID: 12703480

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 33 new
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)

{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.

{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.

{color:red}-1 checkstyle{color}. The applied patch generated
1928 checkstyle errors (more than the master's current 1927 errors).

{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.

{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100

{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.

{color:green}+1 core tests{color}. The patch passed unit tests in .

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13144//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13144//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13144//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13144//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13144//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13144//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13144//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13144//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13144//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13144//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13144//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13144//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13144//artifact/patchprocess/checkstyle-aggregate.html

Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13144//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13144//console

This message is automatically generated.

Task 6 7 from HBASE-9117, delete all public HTable constructors and delete
ConnectionManager#{delete,get}Connection
--

Key: HBASE-12586
URL: https://issues.apache.org/jira/browse/HBASE-12586
Project: HBase
Issue Type: Task
Affects Versions: 2.0.0
Reporter: stack
Assignee: Mikhail Antonov
Labels: beginner, beginners
Attachments: HBASE-12586.patch, HBASE-12586.patch, HBASE-12586.patch

Finish cleanup from HBASE-9117 removing old API. This issue covers tasks 6
and 7 from the list here:
https://issues.apache.org/jira/browse/HBASE-9117?focusedCommentId=13919716page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13919716
To be done in master branch only.
Marked as beginner task. The idea is straight-forward. It is just a lot of
work going through all tests converting to use new API.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13149) HBase MR is broken on Hadoop 2.5+ Yarn


[ 
https://issues.apache.org/jira/browse/HBASE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353623#comment-14353623
 ] 

Enis Soztutar commented on HBASE-13149:
---

+1 for changing the jackson version in master and branch-1. For 1.0 and 0.98 we 
can document this in the book (there is a section about hadoop versions, we can 
add it there). 

 HBase MR is broken on Hadoop 2.5+ Yarn
 --

 Key: HBASE-13149
 URL: https://issues.apache.org/jira/browse/HBASE-13149
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.0.0, 2.0.0, 0.98.10.1
Reporter: Jerry He
Priority: Critical
 Attachments: HBASE-13149-0.98.patch, 
 jackson-core-asl-compat_report.html, jackson-jaxrs-compat_report.html, 
 jackson-mapper-asl-compat_report.html, jackson-xc-compat_report.html


 Running the server MR tools is not working on Yarn version 2.5+.
 Running org.apache.hadoop.hbase.mapreduce.Export:
 {noformat}
 Exception in thread main java.lang.NoSuchMethodError: 
 org.codehaus.jackson.map.ObjectMapper.setSerializationInclusion(Lorg/codehaus/jackson/map/annotate/JsonSerialize$Inclusion;)Lorg/codehaus/jackson/map/ObjectMapper;
 at 
 org.apache.hadoop.yarn.webapp.YarnJacksonJaxbJsonProvider.configObjectMapper(YarnJacksonJaxbJsonProvider.java:59)
 at 
 org.apache.hadoop.yarn.util.timeline.TimelineUtils.clinit(TimelineUtils.java:47)
 at 
 org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:166)
 at 
 org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
 at 
 org.apache.hadoop.mapred.ResourceMgrDelegate.serviceInit(ResourceMgrDelegate.java:102)
 at 
 org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
 at 
 org.apache.hadoop.mapred.ResourceMgrDelegate.init(ResourceMgrDelegate.java:96)
 at org.apache.hadoop.mapred.YARNRunner.init(YARNRunner.java:112)
 at 
 org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34)
 at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:95)
 at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:82)
 at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:75)
 at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1266)
 at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1262)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:415)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
 at org.apache.hadoop.mapreduce.Job.connect(Job.java:1261)
 at org.apache.hadoop.mapreduce.Job.submit(Job.java:1290)
 at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
 at org.apache.hadoop.hbase.mapreduce.Export.main(Export.java:189)
 {noformat}
 The problem seems to be the jackson jar version.  HADOOP-10104 updated 
 jackson version to 1.9.13.  YARN-2092 reported a problem as well.
 HBase is using jackson 1.8.8. This version of the jar in the classpath seem 
 to cause the problem.
 Should we upgrade to jackson 1.9.13? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12586) Task 6 7 from HBASE-9117, delete all public HTable constructors and delete ConnectionManager#{delete,get}Connection


[ 
https://issues.apache.org/jira/browse/HBASE-12586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353741#comment-14353741
 ] 

Mikhail Antonov commented on HBASE-12586:
-

The build took 2 hr 15 min, no measurable difference from previous runs on that 
executor.

 Task 6  7 from HBASE-9117,  delete all public HTable constructors and delete 
 ConnectionManager#{delete,get}Connection
 --

 Key: HBASE-12586
 URL: https://issues.apache.org/jira/browse/HBASE-12586
 Project: HBase
  Issue Type: Task
Affects Versions: 2.0.0
Reporter: stack
Assignee: Mikhail Antonov
  Labels: beginner, beginners
 Attachments: HBASE-12586.patch, HBASE-12586.patch, HBASE-12586.patch


 Finish cleanup from HBASE-9117 removing old API.  This issue covers tasks 6 
 and 7 from the list here: 
 https://issues.apache.org/jira/browse/HBASE-9117?focusedCommentId=13919716page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13919716
 To be done in master branch only.
 Marked as beginner task.  The idea is straight-forward.  It is just a lot of 
 work going through all tests converting to use new API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13170) Allow block cache to be external

2015-03-09 Thread Elliott Clark (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353618#comment-14353618
 ] 

Elliott Clark commented on HBASE-13170:
---

[~stack] No I don't think that this will be the default deployment for 
everyone. Sitting next to some of the people with direct knowledge of Memcached 
does make this a little easier to tolerate the added complexity.

bq.I've been thinking we should implement our multi-layer caching properly, 
supporting an arbitrary number of layers and with ability to specify how we 
want blocks to cascade from layer to layer.
[~ndimiduk] That would be really cool.

 Allow block cache to be external
 

 Key: HBASE-13170
 URL: https://issues.apache.org/jira/browse/HBASE-13170
 Project: HBase
  Issue Type: New Feature
  Components: io
Affects Versions: 2.0.0
Reporter: Elliott Clark
Assignee: Elliott Clark
 Fix For: 1.1.0


 Allow an external service to provide the block cache. This has the nice 
 property of allowing failover/upgrades to happen without causing a fully cold 
 cache.
 Additionally this allows read replicas to share some of the same memory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13149) HBase MR is broken on Hadoop 2.5+ Yarn

2015-03-09 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353740#comment-14353740
 ] 

Sean Busbey commented on HBASE-13149:
-

From reading the compatibility comparison between Jackson 1.8 and 1.9, 
updating it in branch-1 would violate our promise not to break any 
dependencies on minor version releases.

Can we find a version of jackson that doesn't break yarn, us, and our 
compatibility promise for dependencies on branch-1?

 HBase MR is broken on Hadoop 2.5+ Yarn
 --

 Key: HBASE-13149
 URL: https://issues.apache.org/jira/browse/HBASE-13149
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.0.0, 2.0.0, 0.98.10.1
Reporter: Jerry He
Priority: Critical
 Attachments: HBASE-13149-0.98.patch, 
 jackson-core-asl-compat_report.html, jackson-jaxrs-compat_report.html, 
 jackson-mapper-asl-compat_report.html, jackson-xc-compat_report.html


 Running the server MR tools is not working on Yarn version 2.5+.
 Running org.apache.hadoop.hbase.mapreduce.Export:
 {noformat}
 Exception in thread main java.lang.NoSuchMethodError: 
 org.codehaus.jackson.map.ObjectMapper.setSerializationInclusion(Lorg/codehaus/jackson/map/annotate/JsonSerialize$Inclusion;)Lorg/codehaus/jackson/map/ObjectMapper;
 at 
 org.apache.hadoop.yarn.webapp.YarnJacksonJaxbJsonProvider.configObjectMapper(YarnJacksonJaxbJsonProvider.java:59)
 at 
 org.apache.hadoop.yarn.util.timeline.TimelineUtils.clinit(TimelineUtils.java:47)
 at 
 org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:166)
 at 
 org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
 at 
 org.apache.hadoop.mapred.ResourceMgrDelegate.serviceInit(ResourceMgrDelegate.java:102)
 at 
 org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
 at 
 org.apache.hadoop.mapred.ResourceMgrDelegate.init(ResourceMgrDelegate.java:96)
 at org.apache.hadoop.mapred.YARNRunner.init(YARNRunner.java:112)
 at 
 org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34)
 at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:95)
 at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:82)
 at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:75)
 at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1266)
 at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1262)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:415)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
 at org.apache.hadoop.mapreduce.Job.connect(Job.java:1261)
 at org.apache.hadoop.mapreduce.Job.submit(Job.java:1290)
 at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
 at org.apache.hadoop.hbase.mapreduce.Export.main(Export.java:189)
 {noformat}
 The problem seems to be the jackson jar version.  HADOOP-10104 updated 
 jackson version to 1.9.13.  YARN-2092 reported a problem as well.
 HBase is using jackson 1.8.8. This version of the jar in the classpath seem 
 to cause the problem.
 Should we upgrade to jackson 1.9.13? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12586) Task 6 7 from HBASE-9117, delete all public HTable constructors and delete ConnectionManager#{delete,get}Connection


[ 
https://issues.apache.org/jira/browse/HBASE-12586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353438#comment-14353438
 ] 

Mikhail Antonov commented on HBASE-12586:
-

Turns out the vast majority of broken tests (all but 3, which are easy to fix 
separately) are fixable with changing ConnectionManager#getConnectionInternal() 
to use unmanaged connections. That somewhat diminishes  the role of internal 
connection cache, but I believe connection caching going away anyway (as stated 
in javadoc)?

Thoughts?

 Task 6  7 from HBASE-9117,  delete all public HTable constructors and delete 
 ConnectionManager#{delete,get}Connection
 --

 Key: HBASE-12586
 URL: https://issues.apache.org/jira/browse/HBASE-12586
 Project: HBase
  Issue Type: Task
Affects Versions: 2.0.0
Reporter: stack
Assignee: Mikhail Antonov
  Labels: beginner, beginners
 Attachments: HBASE-12586.patch, HBASE-12586.patch


 Finish cleanup from HBASE-9117 removing old API.  This issue covers tasks 6 
 and 7 from the list here: 
 https://issues.apache.org/jira/browse/HBASE-9117?focusedCommentId=13919716page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13919716
 To be done in master branch only.
 Marked as beginner task.  The idea is straight-forward.  It is just a lot of 
 work going through all tests converting to use new API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13179) TestMasterObserver deleteTable is flaky

[
https://issues.apache.org/jira/browse/HBASE-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Matteo Bertozzi updated HBASE-13179:

Resolution: Fixed
Fix Version/s: 1.1.0
2.0.0
Status: Resolved (was: Patch Available)

TestMasterObserver deleteTable is flaky
---

Attachments: HBASE-13179-v0.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13149) HBase MR is broken on Hadoop 2.5+ Yarn

2015-03-09 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353496#comment-14353496
 ] 

Sean Busbey commented on HBASE-13149:
-

+1 on updating it in 2.0, with a release note.

Shouldn't the RunJar isolation protect us on the client side when launching our 
tools? Do we do something that prevents the use of runjar?

IIRC the runjar isolation is only on 2.6. For versions that don't have it, we 
can give folks instructions on manually replacing the impacted library (as we 
do for replacing the hadoop jars) with a note that they'll need to update their 
application to ensure it also uses the correct version.

 HBase MR is broken on Hadoop 2.5+ Yarn
 --

 Key: HBASE-13149
 URL: https://issues.apache.org/jira/browse/HBASE-13149
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.0.0, 2.0.0, 0.98.10.1
Reporter: Jerry He
Priority: Critical
 Attachments: HBASE-13149-0.98.patch, 
 jackson-core-asl-compat_report.html, jackson-jaxrs-compat_report.html, 
 jackson-mapper-asl-compat_report.html, jackson-xc-compat_report.html


 Running the server MR tools is not working on Yarn version 2.5+.
 Running org.apache.hadoop.hbase.mapreduce.Export:
 {noformat}
 Exception in thread main java.lang.NoSuchMethodError: 
 org.codehaus.jackson.map.ObjectMapper.setSerializationInclusion(Lorg/codehaus/jackson/map/annotate/JsonSerialize$Inclusion;)Lorg/codehaus/jackson/map/ObjectMapper;
 at 
 org.apache.hadoop.yarn.webapp.YarnJacksonJaxbJsonProvider.configObjectMapper(YarnJacksonJaxbJsonProvider.java:59)
 at 
 org.apache.hadoop.yarn.util.timeline.TimelineUtils.clinit(TimelineUtils.java:47)
 at 
 org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:166)
 at 
 org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
 at 
 org.apache.hadoop.mapred.ResourceMgrDelegate.serviceInit(ResourceMgrDelegate.java:102)
 at 
 org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
 at 
 org.apache.hadoop.mapred.ResourceMgrDelegate.init(ResourceMgrDelegate.java:96)
 at org.apache.hadoop.mapred.YARNRunner.init(YARNRunner.java:112)
 at 
 org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34)
 at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:95)
 at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:82)
 at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:75)
 at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1266)
 at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1262)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:415)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
 at org.apache.hadoop.mapreduce.Job.connect(Job.java:1261)
 at org.apache.hadoop.mapreduce.Job.submit(Job.java:1290)
 at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
 at org.apache.hadoop.hbase.mapreduce.Export.main(Export.java:189)
 {noformat}
 The problem seems to be the jackson jar version.  HADOOP-10104 updated 
 jackson version to 1.9.13.  YARN-2092 reported a problem as well.
 HBase is using jackson 1.8.8. This version of the jar in the classpath seem 
 to cause the problem.
 Should we upgrade to jackson 1.9.13? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12586) Task 6 7 from HBASE-9117, delete all public HTable constructors and delete ConnectionManager#{delete,get}Connection


 [ 
https://issues.apache.org/jira/browse/HBASE-12586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Antonov updated HBASE-12586:

Attachment: HBASE-12586.patch

updated patch

 Task 6  7 from HBASE-9117,  delete all public HTable constructors and delete 
 ConnectionManager#{delete,get}Connection
 --

 Key: HBASE-12586
 URL: https://issues.apache.org/jira/browse/HBASE-12586
 Project: HBase
  Issue Type: Task
Affects Versions: 2.0.0
Reporter: stack
Assignee: Mikhail Antonov
  Labels: beginner, beginners
 Attachments: HBASE-12586.patch, HBASE-12586.patch, HBASE-12586.patch


 Finish cleanup from HBASE-9117 removing old API.  This issue covers tasks 6 
 and 7 from the list here: 
 https://issues.apache.org/jira/browse/HBASE-9117?focusedCommentId=13919716page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13919716
 To be done in master branch only.
 Marked as beginner task.  The idea is straight-forward.  It is just a lot of 
 work going through all tests converting to use new API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-13180) TestRegionReplicas.testGetOnTargetRegionReplica fails frequently.


 [ 
https://issues.apache.org/jira/browse/HBASE-13180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi resolved HBASE-13180.
-
Resolution: Invalid
  Assignee: Matteo Bertozzi

 TestRegionReplicas.testGetOnTargetRegionReplica fails frequently.
 -

 Key: HBASE-13180
 URL: https://issues.apache.org/jira/browse/HBASE-13180
 Project: HBase
  Issue Type: Bug
Reporter: Srikanth Srungarapu
Assignee: Matteo Bertozzi
Priority: Minor

 Noticing the test failure frequently on internal test rig. 
 {code}
 FAILED:  
 org.apache.hadoop.hbase.regionserver.TestRegionReplicas.testGetOnTargetRegionReplicaError
  Message:
 1
 Stack Trace:
 java.lang.ArrayIndexOutOfBoundsException: 1
 at 
 org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas$ResultBoundedCompletionService.submit(RpcRetryingCallerWithReadReplicas.java:420)
 at 
 org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.addCallsForReplica(RpcRetryingCallerWithReadReplicas.java:280)
 at 
 org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.call(RpcRetryingCallerWithReadReplicas.java:199)
 at org.apache.hadoop.hbase.client.HTable.get(HTable.java:890)
 at 
 org.apache.hadoop.hbase.regionserver.TestRegionReplicas.testGetOnTargetRegionReplica(TestRegionReplicas.java:193)
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-13181) TestHRegionReplayEvents.testReplayBulkLoadEvent fails frequently.


 [ 
https://issues.apache.org/jira/browse/HBASE-13181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi resolved HBASE-13181.
-
Resolution: Invalid
  Assignee: Matteo Bertozzi

 TestHRegionReplayEvents.testReplayBulkLoadEvent fails frequently.
 -

 Key: HBASE-13181
 URL: https://issues.apache.org/jira/browse/HBASE-13181
 Project: HBase
  Issue Type: Bug
Reporter: Srikanth Srungarapu
Assignee: Matteo Bertozzi
Priority: Minor

 Noticing the test failures frequently on internal test rig:
 {code}
 FAILED:  
 org.apache.hadoop.hbase.regionserver.TestHRegionReplayEvents.testReplayBulkLoadEvent
 Error Message:
 3 exceptions [org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem 
 reading HFile Trailer from file 
 /var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/15e393f3-1a5c-45e4-b048-fd647956f1f7,
  org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading 
 HFile Trailer from file 
 /var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/9bdfbfb1-8a72-4907-9f91-5570d5accfcd,
  org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading 
 HFile Trailer from file 
 /var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/fe63364b-906a-4085-85b6-605aee4d41c2]
 Stack Trace:
 org.apache.hadoop.io.MultipleIOException: 3 exceptions 
 [org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading 
 HFile Trailer from file 
 /var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/15e393f3-1a5c-45e4-b048-fd647956f1f7,
  org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading 
 HFile Trailer from file 
 /var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/9bdfbfb1-8a72-4907-9f91-5570d5accfcd,
  org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading 
 HFile Trailer from file 
 /var/lib/jenkins/workspace/CDH5-HBase-1.0.0/hbase-server/target/test-data/a68c6f22-f578-44d2-b17b-6314ea7f6b97/fe63364b-906a-4085-85b6-605aee4d41c2]
 at 
 org.apache.hadoop.io.MultipleIOException.createIOException(MultipleIOException.java:52)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:4910)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:4855)
 at 
 org.apache.hadoop.hbase.regionserver.TestHRegionReplayEvents.testReplayBulkLoadEvent(TestHRegionReplayEvents.java:1159)
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13174) Apply HBASE-11804 to Windows scripts

2015-03-09 Thread Lars George (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353545#comment-14353545
 ] 

Lars George commented on HBASE-13174:
-

Thanks [~enis], I will commit tomorrow.

 Apply HBASE-11804 to Windows scripts
 

 Key: HBASE-13174
 URL: https://issues.apache.org/jira/browse/HBASE-13174
 Project: HBase
  Issue Type: Bug
  Components: scripts
Affects Versions: 1.0.0
Reporter: Lars George
 Attachments: HBASE-13174.patch


 HBASE-11804 was only applied to the Linux scripts. Do the same for Windows. 
 Need a +1 here from the Windows supporters, since this patch is a one liner 
 but I need to know if it is OK to do on Windows. I'd say yes, but better ask 
 first. cc [~enis]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13149) HBase MR is broken on Hadoop 2.5+ Yarn

2015-03-09 Thread Jerry He (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353433#comment-14353433
 ] 

Jerry He commented on HBASE-13149:
--

Hi, folks

We need a fix here. 
Should we at least upgrade to jackson 1.9.13 on the 2.0 branch?  And maybe even 
the 1.1+ branch?   We build on Hadoop 2.5.1 by default on these branches.
We'll document clearly for 0.98.

Any suggestions? 



 HBase MR is broken on Hadoop 2.5+ Yarn
 --

 Key: HBASE-13149
 URL: https://issues.apache.org/jira/browse/HBASE-13149
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.0.0, 2.0.0, 0.98.10.1
Reporter: Jerry He
Priority: Critical
 Attachments: HBASE-13149-0.98.patch, 
 jackson-core-asl-compat_report.html, jackson-jaxrs-compat_report.html, 
 jackson-mapper-asl-compat_report.html, jackson-xc-compat_report.html


 Running the server MR tools is not working on Yarn version 2.5+.
 Running org.apache.hadoop.hbase.mapreduce.Export:
 {noformat}
 Exception in thread main java.lang.NoSuchMethodError: 
 org.codehaus.jackson.map.ObjectMapper.setSerializationInclusion(Lorg/codehaus/jackson/map/annotate/JsonSerialize$Inclusion;)Lorg/codehaus/jackson/map/ObjectMapper;
 at 
 org.apache.hadoop.yarn.webapp.YarnJacksonJaxbJsonProvider.configObjectMapper(YarnJacksonJaxbJsonProvider.java:59)
 at 
 org.apache.hadoop.yarn.util.timeline.TimelineUtils.clinit(TimelineUtils.java:47)
 at 
 org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:166)
 at 
 org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
 at 
 org.apache.hadoop.mapred.ResourceMgrDelegate.serviceInit(ResourceMgrDelegate.java:102)
 at 
 org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
 at 
 org.apache.hadoop.mapred.ResourceMgrDelegate.init(ResourceMgrDelegate.java:96)
 at org.apache.hadoop.mapred.YARNRunner.init(YARNRunner.java:112)
 at 
 org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34)
 at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:95)
 at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:82)
 at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:75)
 at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1266)
 at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1262)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:415)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
 at org.apache.hadoop.mapreduce.Job.connect(Job.java:1261)
 at org.apache.hadoop.mapreduce.Job.submit(Job.java:1290)
 at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
 at org.apache.hadoop.hbase.mapreduce.Export.main(Export.java:189)
 {noformat}
 The problem seems to be the jackson jar version.  HADOOP-10104 updated 
 jackson version to 1.9.13.  YARN-2092 reported a problem as well.
 HBase is using jackson 1.8.8. This version of the jar in the classpath seem 
 to cause the problem.
 Should we upgrade to jackson 1.9.13? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13172) TestDistributedLogSplitting.testThreeRSAbort fails several times on branch-1

2015-03-09 Thread Jimmy Xiang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353452#comment-14353452
 ] 

Jimmy Xiang commented on HBASE-13172:
-

+1. Looks good to me. As to the issue [~jeffreyz] pointed out, that part is 
needed.  It is preferred that a RS dies naturally (means per ZK) instead of 
marked dead by AM. Call isServerReachable should not return false info after 
retries since we check the start code, if the retries take longer the ZK 
session time-out time.

 TestDistributedLogSplitting.testThreeRSAbort fails several times on branch-1
 

 Key: HBASE-13172
 URL: https://issues.apache.org/jira/browse/HBASE-13172
 Project: HBase
  Issue Type: Bug
  Components: test
Affects Versions: 1.1.0
Reporter: zhangduo
Assignee: zhangduo
 Attachments: HBASE-13172-branch-1.patch


 The direct reason is we are stuck in ServerManager.isServerReachable.
 https://builds.apache.org/job/HBase-1.1/253/testReport/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testThreeRSAbort/
 {noformat}
 2015-03-06 04:06:19,430 DEBUG [AM.-pool300-t1] master.ServerManager(855): 
 Couldn't reach asf906.gq1.ygridcore.net,59366,1425614770146, try=0 of 10
 2015-03-06 04:07:10,545 DEBUG [AM.-pool300-t1] master.ServerManager(855): 
 Couldn't reach asf906.gq1.ygridcore.net,59366,1425614770146, try=9 of 10
 {noformat}
 The interval between first and last retry log is about 1 minute, and we only 
 wait 1 minute so the test is timeout.
 Still do not know why this happen.
 And at last there are lots of this 
 {noformat}
 2015-03-06 04:07:21,529 DEBUG [AM.-pool300-t1] master.ServerManager(855): 
 Couldn't reach asf906.gq1.ygridcore.net,59366,1425614770146, try=9 of 10
 org.apache.hadoop.hbase.ipc.StoppedRpcClientException
   at 
 org.apache.hadoop.hbase.ipc.RpcClientImpl.getConnection(RpcClientImpl.java:1261)
   at 
 org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1146)
   at 
 org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
   at 
 org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
   at 
 org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.getServerInfo(AdminProtos.java:22031)
   at 
 org.apache.hadoop.hbase.protobuf.ProtobufUtil.getServerInfo(ProtobufUtil.java:1797)
   at 
 org.apache.hadoop.hbase.master.ServerManager.isServerReachable(ServerManager.java:850)
   at 
 org.apache.hadoop.hbase.master.RegionStates.isServerDeadAndNotProcessed(RegionStates.java:843)
   at 
 org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1969)
   at 
 org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1576)
   at 
 org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48)
   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:744)
 {noformat}
 I think the problem is here
 {code:title=ServerManager.java}
 while (retryCounter.shouldRetry()) {
 ...
 try {
   retryCounter.sleepUntilNextRetry();
 } catch(InterruptedException ie) {
   Thread.currentThread().interrupt();
 }
 ...
 }
 {code}
 We need to break out of the while loop when getting InterruptedException, not 
 just mark current thread as interrupted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12586) Task 6 7 from HBASE-9117, delete all public HTable constructors and delete ConnectionManager#{delete,get}Connection


 [ 
https://issues.apache.org/jira/browse/HBASE-12586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Antonov updated HBASE-12586:

Status: Patch Available  (was: Open)

 Task 6  7 from HBASE-9117,  delete all public HTable constructors and delete 
 ConnectionManager#{delete,get}Connection
 --

 Key: HBASE-12586
 URL: https://issues.apache.org/jira/browse/HBASE-12586
 Project: HBase
  Issue Type: Task
Affects Versions: 2.0.0
Reporter: stack
Assignee: Mikhail Antonov
  Labels: beginner, beginners
 Attachments: HBASE-12586.patch, HBASE-12586.patch, HBASE-12586.patch


 Finish cleanup from HBASE-9117 removing old API.  This issue covers tasks 6 
 and 7 from the list here: 
 https://issues.apache.org/jira/browse/HBASE-9117?focusedCommentId=13919716page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13919716
 To be done in master branch only.
 Marked as beginner task.  The idea is straight-forward.  It is just a lot of 
 work going through all tests converting to use new API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12586) Task 6 7 from HBASE-9117, delete all public HTable constructors and delete ConnectionManager#{delete,get}Connection


 [ 
https://issues.apache.org/jira/browse/HBASE-12586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Antonov updated HBASE-12586:

Status: Open  (was: Patch Available)

 Task 6  7 from HBASE-9117,  delete all public HTable constructors and delete 
 ConnectionManager#{delete,get}Connection
 --

 Key: HBASE-12586
 URL: https://issues.apache.org/jira/browse/HBASE-12586
 Project: HBase
  Issue Type: Task
Affects Versions: 2.0.0
Reporter: stack
Assignee: Mikhail Antonov
  Labels: beginner, beginners
 Attachments: HBASE-12586.patch, HBASE-12586.patch, HBASE-12586.patch


 Finish cleanup from HBASE-9117 removing old API.  This issue covers tasks 6 
 and 7 from the list here: 
 https://issues.apache.org/jira/browse/HBASE-9117?focusedCommentId=13919716page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13919716
 To be done in master branch only.
 Marked as beginner task.  The idea is straight-forward.  It is just a lot of 
 work going through all tests converting to use new API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-13174) Apply HBASE-11804 to Windows scripts

2015-03-09 Thread Lars George (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-13174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars George reassigned HBASE-13174:
---

Assignee: Lars George

 Apply HBASE-11804 to Windows scripts
 

 Key: HBASE-13174
 URL: https://issues.apache.org/jira/browse/HBASE-13174
 Project: HBase
  Issue Type: Bug
  Components: scripts
Affects Versions: 1.0.0
Reporter: Lars George
Assignee: Lars George
 Fix For: 2.0.0, 1.0.1, 1.1.0

 Attachments: HBASE-13174.patch


 HBASE-11804 was only applied to the Linux scripts. Do the same for Windows. 
 Need a +1 here from the Windows supporters, since this patch is a one liner 
 but I need to know if it is OK to do on Windows. I'd say yes, but better ask 
 first. cc [~enis]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13182) Test NamespaceAuditor/AccessController create/delete table is flaky


[ 
https://issues.apache.org/jira/browse/HBASE-13182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353811#comment-14353811
 ] 

Enis Soztutar commented on HBASE-13182:
---

+1. 

 Test NamespaceAuditor/AccessController create/delete table is flaky
 ---

 Key: HBASE-13182
 URL: https://issues.apache.org/jira/browse/HBASE-13182
 Project: HBase
  Issue Type: Test
  Components: test
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
 Attachments: HBASE-13182-v0.patch


 Similar to HBASE-13179
 TestNamespaceAuditor and the two test AccessController fail when create or 
 delete table takes more time.and it is because the sync version of the Admin 
 operation is not sync, but relies on the last operation on the server e.g. 
 delete meta. 
 but the post-operation method of the coprocessor is called after meta is 
 deleted.. 
 long story short, the client is not really sync and it will be fixed by 
 HBASE-12439. 
 similar to what we do in TestMasterObserver we can add an observer with a 
 countDownLatch to wait the postOpHandler call.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13183) Make ZK tickTime configurable in standalone HBase


 [ 
https://issues.apache.org/jira/browse/HBASE-13183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-13183:
---
Attachment: HBASE-13183.patch

Patch for trunk. Actually the 0.98 patch applies cleanly to all later branches.

 Make ZK tickTime configurable in standalone HBase
 -

 Key: HBASE-13183
 URL: https://issues.apache.org/jira/browse/HBASE-13183
 Project: HBase
  Issue Type: Improvement
  Components: master
Affects Versions: 0.98.10.1
Reporter: Alex Araujo
Assignee: Alex Araujo
Priority: Minor
 Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.12

 Attachments: HBASE-13183-0.98.patch, HBASE-13183.patch


 Standalone HBase does not allow the ZooKeeper tickTime to be configured for 
 the MiniZooKeeperCluster. The default tickTime for MiniZooKeeperCluster is 
 2000 ms, which is too low for some use cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13183) Make ZK tickTime configurable in standalone HBase


 [ 
https://issues.apache.org/jira/browse/HBASE-13183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-13183:
---
Fix Version/s: 0.98.12
   1.1.0
   1.0.1
   2.0.0
   Status: Patch Available  (was: Open)

 Make ZK tickTime configurable in standalone HBase
 -

 Key: HBASE-13183
 URL: https://issues.apache.org/jira/browse/HBASE-13183
 Project: HBase
  Issue Type: Improvement
  Components: master
Affects Versions: 0.98.10.1
Reporter: Alex Araujo
Assignee: Alex Araujo
Priority: Minor
 Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.12

 Attachments: HBASE-13183-0.98.patch, HBASE-13183.patch


 Standalone HBase does not allow the ZooKeeper tickTime to be configured for 
 the MiniZooKeeperCluster. The default tickTime for MiniZooKeeperCluster is 
 2000 ms, which is too low for some use cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13169) ModifyTable increasing the region replica count should also auto-setup RRRE


[ 
https://issues.apache.org/jira/browse/HBASE-13169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353859#comment-14353859
 ] 

Mikhail Antonov commented on HBASE-13169:
-

bq. We do not have a global lock on all the table operations in master so doing 
an atomic check for is this the last table with replicas is hard to do.

I see, thanks. I guess we may have a chore on master watching for obsolete 
peers, but that'd be separate thing..

 ModifyTable increasing the region replica count should also auto-setup RRRE
 ---

 Key: HBASE-13169
 URL: https://issues.apache.org/jira/browse/HBASE-13169
 Project: HBase
  Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Fix For: 2.0.0, 1.1.0

 Attachments: hbase-13169_v1.patch


 This is a follow up to Async WAL replication jira (HBASE-11568). In 
 HBASE-11568 we setup replication automatically in create table if the 
 configuration is enabled. 
 We should do the same in case a table with region replication = 1 is 
 modified, and region replication is set to 1. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12723) Update ACL matrix to reflect reality

2015-03-09 Thread Misty Stanley-Jones (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-12723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353761#comment-14353761
 ] 

Misty Stanley-Jones commented on HBASE-12723:
-

+1 committed. Thanks [~srikanth235]

 Update ACL matrix to reflect reality
 

 Key: HBASE-12723
 URL: https://issues.apache.org/jira/browse/HBASE-12723
 Project: HBase
  Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Srikanth Srungarapu
 Fix For: 2.0.0, 1.0.1, 1.1.0

 Attachments: HBASE-12723.patch, HBASE-12723_v2.patch, 
 HBASE-12723_v3.patch, book.html


 The ACL matrix in the book should be updated with the recent changes.  
 https://hbase.apache.org/book/appendix_acl_matrix.html
 Also the format is not optimal. There is a hierarchy relation between scopes 
 (GLOBAL  NS  TABLE), but not so much between Permissions (A,C,R)
 Some things to do:
 - {{Minimum Permission}} column does not make sense. We should replace it. 
 - Add information about superuser 
 - grant is a multi level thing. Required permissions depend on the scope.
 - See HBASE-12511 and others changed some of the permissions 
 What I would like to see at the end is something like:
 {code}
 createNamespace: superuser | global(A)
 deleteNamespace: superuser | global(A) | NS(A)
 modifyNamespace: superuser | global(A) | NS(A)
 getNamespaceDescriptor : superuser | global(A) | NS(A)
 listNamespaces : All access*
 createTable: superuser | global(C) | NS(C)
 grant 
   NS Perm  : superuser | global(A) | NS(A)
   Table Perm   : ...
 revoke 
   NS Perm  : superuser | global(A) | NS(A)
   Table Perm   : ...
 getPerms 
   NS perm  : superuser | global(A) | NS(A)
   Table Perm   : ...
 {code}
 See HBASE-12511. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13182) Test NamespaceAuditor/AccessController create/delete table is flaky

2015-03-09 Thread Srikanth Srungarapu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353802#comment-14353802
 ] 

Srikanth Srungarapu commented on HBASE-13182:
-

+1(non-binding).

 Test NamespaceAuditor/AccessController create/delete table is flaky
 ---

 Key: HBASE-13182
 URL: https://issues.apache.org/jira/browse/HBASE-13182
 Project: HBase
  Issue Type: Test
  Components: test
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
 Attachments: HBASE-13182-v0.patch


 Similar to HBASE-13179
 TestNamespaceAuditor and the two test AccessController fail when create or 
 delete table takes more time.and it is because the sync version of the Admin 
 operation is not sync, but relies on the last operation on the server e.g. 
 delete meta. 
 but the post-operation method of the coprocessor is called after meta is 
 deleted.. 
 long story short, the client is not really sync and it will be fixed by 
 HBASE-12439. 
 similar to what we do in TestMasterObserver we can add an observer with a 
 countDownLatch to wait the postOpHandler call.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13071) Hbase Streaming Scan Feature

2015-03-09 Thread Eshcar Hillel (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-13071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353865#comment-14353865
]

Eshcar Hillel commented on HBASE-13071:
---

A new patch is attached following the comments by [~jonathan.lawlor] and
[~stack].

Some notes on implementation and design:
* The default value is now set to async. (btw, this means async scanner is
used in multiple tests, which used to have sync scan.)
* The responsibility to invoke super.close() is now shifted to the pending
prefetch thread, so it is not missed.
* In case of sync scanner, the caching parameter indicates both the size of
the buffer and the chunk size (#rows fetched). In case of async scanner, the
parameter only indicates the later, while the buffer size is doubled. This
should now be clear from the documentation, as well as from the new methods
getCacheCapacity() and getThresholdSize().
* cache and caching were members of ClientScanner even before this patch. I
only added the abstract initCache() method. I agree that having two abstract
classes is not the cleanest solution, but neither is having initCache() in a
class where not all subclasses have a cache. As I said before, this hierarchy
can benefit from some re-factoring (the right design might use composition like
in the strategy pattern instead of inheritance, but all these decisions should
not be in the scope of the current Jira).

Some notes on performance:
* This feature is a client side feature and therefore should be tested in
terms of client side latency.
* This feature should reduce the latency, and in worse case scenario should
not increase it (at least not significantly)
* On the server side I would expect the same behavior as in sync scanner,
since the same RPC calls are invoked, they only shift earlier in time to have
the data ready at the client side before the user needs it.
* I cannot explain the behavior of the low humps in your test. Do you see
this consistently? What is the exact setting? Is it a fixed number of scans or
a fixed time?

Hbase Streaming Scan Feature

Key: HBASE-13071
URL: https://issues.apache.org/jira/browse/HBASE-13071
Project: HBase
Issue Type: New Feature
Affects Versions: 0.98.11
Reporter: Eshcar Hillel
Attachments: 99.eshcar.png, HBASE-13071_98_1.patch,
HBASE-13071_trunk_1.patch, HBASE-13071_trunk_2.patch,
HBASE-13071_trunk_3.patch, HBASE-13071_trunk_4.patch,
HBASE-13071_trunk_5.patch, HBaseStreamingScanDesign.pdf,
HbaseStreamingScanEvaluation.pdf, gc.eshcar.png, hits.eshcar.png, network.png

A scan operation iterates over all rows of a table or a subrange of the
table. The synchronous nature in which the data is served at the client side
hinders the speed the application traverses the data: it increases the
overall processing time, and may cause a great variance in the times the
application waits for the next piece of data.
The scanner next() method at the client side invokes an RPC to the
regionserver and then stores the results in a cache. The application can
specify how many rows will be transmitted per RPC; by default this is set to
100 rows.
The cache can be considered as a producer-consumer queue, where the hbase
client pushes the data to the queue and the application consumes it.
Currently this queue is synchronous, i.e., blocking. More specifically, when
the application consumed all the data from the cache --- so the cache is
empty --- the hbase client retrieves additional data from the server and
re-fills the cache with new data. During this time the application is blocked.
Under the assumption that the application processing time can be balanced by
the time it takes to retrieve the data, an asynchronous approach can reduce
the time the application is waiting for data.
We attach a design document.
We also have a patch that is based on a private branch, and some evaluation
results of this code.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13172) TestDistributedLogSplitting.testThreeRSAbort fails several times on branch-1


[ 
https://issues.apache.org/jira/browse/HBASE-13172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353870#comment-14353870
 ] 

Hudson commented on HBASE-13172:


SUCCESS: Integrated in HBase-1.1 #263 (See 
[https://builds.apache.org/job/HBase-1.1/263/])
HBASE-13172 TestDistributedLogSplitting.testThreeRSAbort fails several times on 
branch-1 (stack: rev c40d880a3e199d04c632212a1e51abca9973a4be)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java


 TestDistributedLogSplitting.testThreeRSAbort fails several times on branch-1
 

 Key: HBASE-13172
 URL: https://issues.apache.org/jira/browse/HBASE-13172
 Project: HBase
  Issue Type: Bug
  Components: test
Affects Versions: 1.1.0
Reporter: zhangduo
Assignee: zhangduo
 Fix For: 1.0.1, 1.1.0, 0.98.12

 Attachments: HBASE-13172-branch-1.patch


 The direct reason is we are stuck in ServerManager.isServerReachable.
 https://builds.apache.org/job/HBase-1.1/253/testReport/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testThreeRSAbort/
 {noformat}
 2015-03-06 04:06:19,430 DEBUG [AM.-pool300-t1] master.ServerManager(855): 
 Couldn't reach asf906.gq1.ygridcore.net,59366,1425614770146, try=0 of 10
 2015-03-06 04:07:10,545 DEBUG [AM.-pool300-t1] master.ServerManager(855): 
 Couldn't reach asf906.gq1.ygridcore.net,59366,1425614770146, try=9 of 10
 {noformat}
 The interval between first and last retry log is about 1 minute, and we only 
 wait 1 minute so the test is timeout.
 Still do not know why this happen.
 And at last there are lots of this 
 {noformat}
 2015-03-06 04:07:21,529 DEBUG [AM.-pool300-t1] master.ServerManager(855): 
 Couldn't reach asf906.gq1.ygridcore.net,59366,1425614770146, try=9 of 10
 org.apache.hadoop.hbase.ipc.StoppedRpcClientException
   at 
 org.apache.hadoop.hbase.ipc.RpcClientImpl.getConnection(RpcClientImpl.java:1261)
   at 
 org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1146)
   at 
 org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
   at 
 org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
   at 
 org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.getServerInfo(AdminProtos.java:22031)
   at 
 org.apache.hadoop.hbase.protobuf.ProtobufUtil.getServerInfo(ProtobufUtil.java:1797)
   at 
 org.apache.hadoop.hbase.master.ServerManager.isServerReachable(ServerManager.java:850)
   at 
 org.apache.hadoop.hbase.master.RegionStates.isServerDeadAndNotProcessed(RegionStates.java:843)
   at 
 org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1969)
   at 
 org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1576)
   at 
 org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48)
   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:744)
 {noformat}
 I think the problem is here
 {code:title=ServerManager.java}
 while (retryCounter.shouldRetry()) {
 ...
 try {
   retryCounter.sleepUntilNextRetry();
 } catch(InterruptedException ie) {
   Thread.currentThread().interrupt();
 }
 ...
 }
 {code}
 We need to break out of the while loop when getting InterruptedException, not 
 just mark current thread as interrupted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13149) HBase MR is broken on Hadoop 2.5+ Yarn


[ 
https://issues.apache.org/jira/browse/HBASE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353914#comment-14353914
 ] 

Enis Soztutar commented on HBASE-13149:
---

Running Hbase with jackson 1.8 with Hadoop-2.5+ does not work at all, so I 
don't know how compatibility is in the picture. 

 HBase MR is broken on Hadoop 2.5+ Yarn
 --

 Key: HBASE-13149
 URL: https://issues.apache.org/jira/browse/HBASE-13149
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.0.0, 2.0.0, 0.98.10.1
Reporter: Jerry He
Priority: Critical
 Attachments: HBASE-13149-0.98.patch, 
 jackson-core-asl-compat_report.html, jackson-jaxrs-compat_report.html, 
 jackson-mapper-asl-compat_report.html, jackson-xc-compat_report.html


 Running the server MR tools is not working on Yarn version 2.5+.
 Running org.apache.hadoop.hbase.mapreduce.Export:
 {noformat}
 Exception in thread main java.lang.NoSuchMethodError: 
 org.codehaus.jackson.map.ObjectMapper.setSerializationInclusion(Lorg/codehaus/jackson/map/annotate/JsonSerialize$Inclusion;)Lorg/codehaus/jackson/map/ObjectMapper;
 at 
 org.apache.hadoop.yarn.webapp.YarnJacksonJaxbJsonProvider.configObjectMapper(YarnJacksonJaxbJsonProvider.java:59)
 at 
 org.apache.hadoop.yarn.util.timeline.TimelineUtils.clinit(TimelineUtils.java:47)
 at 
 org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:166)
 at 
 org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
 at 
 org.apache.hadoop.mapred.ResourceMgrDelegate.serviceInit(ResourceMgrDelegate.java:102)
 at 
 org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
 at 
 org.apache.hadoop.mapred.ResourceMgrDelegate.init(ResourceMgrDelegate.java:96)
 at org.apache.hadoop.mapred.YARNRunner.init(YARNRunner.java:112)
 at 
 org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34)
 at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:95)
 at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:82)
 at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:75)
 at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1266)
 at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1262)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:415)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
 at org.apache.hadoop.mapreduce.Job.connect(Job.java:1261)
 at org.apache.hadoop.mapreduce.Job.submit(Job.java:1290)
 at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
 at org.apache.hadoop.hbase.mapreduce.Export.main(Export.java:189)
 {noformat}
 The problem seems to be the jackson jar version.  HADOOP-10104 updated 
 jackson version to 1.9.13.  YARN-2092 reported a problem as well.
 HBase is using jackson 1.8.8. This version of the jar in the classpath seem 
 to cause the problem.
 Should we upgrade to jackson 1.9.13? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13169) ModifyTable increasing the region replica count should also auto-setup RRRE

[
https://issues.apache.org/jira/browse/HBASE-13169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353785#comment-14353785
]

Enis Soztutar commented on HBASE-13169:
---

Thanks for taking a look.
bq. Wondering if any special handling is needed if modification goes in
opposite direction, say from 3 to 1 num of reps?
We auto-add the replication peer when a table with region replicas is created
or a table is modified to have replicas (with this patch). However, there is
not yet support for auto-removing the replication peer if the last table with
region replicas is deleted or replication is reduced to 1 via modify table. We
do not have a global lock on all the table operations in master so doing an
atomic check for is this the last table with replicas is hard to do.
bq. does this work for the online table modification case?
Right now, online modifying a table for increasing region replicas is not
supported (see ModifyTableHandler.prepareWithTableLock()).
bq. seems like the remove and add replicas would be good to have in one method,
or pull the ifNeeded check into handleTableOperation to make it look parallel.
I see your point. These two cases are doing different things (removing from
meta vs adding a replication peer). Having separate methods should be good.

ModifyTable increasing the region replica count should also auto-setup RRRE
---

Key: HBASE-13169
URL: https://issues.apache.org/jira/browse/HBASE-13169
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 2.0.0, 1.1.0

Attachments: hbase-13169_v1.patch

This is a follow up to Async WAL replication jira (HBASE-11568). In
HBASE-11568 we setup replication automatically in create table if the
configuration is enabled.
We should do the same in case a table with region replication = 1 is
modified, and region replication is set to 1.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13183) Make ZK tickTime configurable in standalone HBase


[ 
https://issues.apache.org/jira/browse/HBASE-13183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353814#comment-14353814
 ] 

Andrew Purtell commented on HBASE-13183:


[~alexaraujo] says he tested this and it worked. Let me put up a patch for 
master, check, and commit shortly.

 Make ZK tickTime configurable in standalone HBase
 -

 Key: HBASE-13183
 URL: https://issues.apache.org/jira/browse/HBASE-13183
 Project: HBase
  Issue Type: Improvement
  Components: master
Affects Versions: 0.98.10.1
Reporter: Alex Araujo
Assignee: Alex Araujo
Priority: Minor
 Attachments: HBASE-13183-0.98.patch


 Standalone HBase does not allow the ZooKeeper tickTime to be configured for 
 the MiniZooKeeperCluster. The default tickTime for MiniZooKeeperCluster is 
 2000 ms, which is too low for some use cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13121) Async wal replication for region replicas and dist log replay does not work together

[
https://issues.apache.org/jira/browse/HBASE-13121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Enis Soztutar updated HBASE-13121:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)

I have committed this. Thanks Jeffrey for the review.

Async wal replication for region replicas and dist log replay does not work
together

Key: HBASE-13121
URL: https://issues.apache.org/jira/browse/HBASE-13121
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 2.0.0, 1.1.0

Attachments: hbase-13121_v1.patch, hbase-13121_v2.patch

We had not tested dist log replay while testing async wal replication for
region replicas. There seems to be a couple of issues, but fixable.
The distinction for dist log replay is that, the region will be opened for
recovery and regular writes when a primary fails over. This causes the region
open event marker to be written to WAL, but at this time, the region actually
does not contain all the edits flushed (since it is still recovering). If
secondary regions see this event, and picks up all the files in the region
open event marker, then they can drop edits.
The solution is:
- Only write the region open event marker to WAL when region is out of
recovering mode.
- Force a flush out of recovering mode. This ensures that all data is force
flushed in this case. Before the region open event marker is written, we
guarantee that all data in the region is flushed, so the list of files in the
event marker is complete.
- Edits coming from recovery are re-written to WAL when recovery is in
action. These edits will have a larger seqId then their original seqId. If
this is the case, we do not replicate these edits to the secondary replicas.
Since the dist log replay recovers edits out of order (coming from parallel
replays from WAL file split tasks), this ensures that TIMELINE consistency is
respected and edits are not seen out of order in secondaries. These edits are
seen from secondaries via the forced flush event.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13071) Hbase Streaming Scan Feature

2015-03-09 Thread Eshcar Hillel (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-13071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Eshcar Hillel updated HBASE-13071:
--
Attachment: HBASE-13071_trunk_5.patch

Hbase Streaming Scan Feature

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12723) Update ACL matrix to reflect reality


[ 
https://issues.apache.org/jira/browse/HBASE-12723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353952#comment-14353952
 ] 

Hudson commented on HBASE-12723:


FAILURE: Integrated in HBase-TRUNK #6226 (See 
[https://builds.apache.org/job/HBase-TRUNK/6226/])
HBASE-12723 Update ACL matrix to reflect reality Srikanth Srungarapu 
(mstanleyjones: rev 61cc8e0de12987b1d64fd06fa27a55c89c4a742f)
* src/main/asciidoc/_chapters/appendix_acl_matrix.adoc


 Update ACL matrix to reflect reality
 

 Key: HBASE-12723
 URL: https://issues.apache.org/jira/browse/HBASE-12723
 Project: HBase
  Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Srikanth Srungarapu
 Fix For: 2.0.0, 1.0.1, 1.1.0

 Attachments: HBASE-12723.patch, HBASE-12723_v2.patch, 
 HBASE-12723_v3.patch, book.html


 The ACL matrix in the book should be updated with the recent changes.  
 https://hbase.apache.org/book/appendix_acl_matrix.html
 Also the format is not optimal. There is a hierarchy relation between scopes 
 (GLOBAL  NS  TABLE), but not so much between Permissions (A,C,R)
 Some things to do:
 - {{Minimum Permission}} column does not make sense. We should replace it. 
 - Add information about superuser 
 - grant is a multi level thing. Required permissions depend on the scope.
 - See HBASE-12511 and others changed some of the permissions 
 What I would like to see at the end is something like:
 {code}
 createNamespace: superuser | global(A)
 deleteNamespace: superuser | global(A) | NS(A)
 modifyNamespace: superuser | global(A) | NS(A)
 getNamespaceDescriptor : superuser | global(A) | NS(A)
 listNamespaces : All access*
 createTable: superuser | global(C) | NS(C)
 grant 
   NS Perm  : superuser | global(A) | NS(A)
   Table Perm   : ...
 revoke 
   NS Perm  : superuser | global(A) | NS(A)
   Table Perm   : ...
 getPerms 
   NS perm  : superuser | global(A) | NS(A)
   Table Perm   : ...
 {code}
 See HBASE-12511. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13183) Make ZK tickTime configurable in standalone HBase

2015-03-09 Thread Alex Araujo (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-13183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Araujo updated HBASE-13183:

Attachment: HBASE-13183-0.98.patch

Attaching 0.98 patch

 Make ZK tickTime configurable in standalone HBase
 -

 Key: HBASE-13183
 URL: https://issues.apache.org/jira/browse/HBASE-13183
 Project: HBase
  Issue Type: Improvement
  Components: master
Affects Versions: 0.98.10.1
Reporter: Alex Araujo
Assignee: Alex Araujo
Priority: Minor
 Attachments: HBASE-13183-0.98.patch


 Standalone HBase does not allow the ZooKeeper tickTime to be configured for 
 the MiniZooKeeperCluster. The default tickTime for MiniZooKeeperCluster is 
 2000 ms, which is too low for some use cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13179) TestMasterObserver deleteTable is flaky


[ 
https://issues.apache.org/jira/browse/HBASE-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353779#comment-14353779
 ] 

Hudson commented on HBASE-13179:


SUCCESS: Integrated in HBase-TRUNK #6225 (See 
[https://builds.apache.org/job/HBase-TRUNK/6225/])
HBASE-13179 TestMasterObserver deleteTable is flaky (matteo.bertozzi: rev 
fb5e6b3f75ad29ffebeaad1fac4c4fdebd69684f)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java


 TestMasterObserver deleteTable is flaky
 ---

 Key: HBASE-13179
 URL: https://issues.apache.org/jira/browse/HBASE-13179
 Project: HBase
  Issue Type: Test
  Components: test
Affects Versions: 1.0.0, 0.98.10.1
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-13179-v0.patch


 TestMasterObserver fails when deleteTable() takes more time to complete the 
 last steps.
 {code}
 java.lang.AssertionError: Delete table handler should be called.
   at org.junit.Assert.fail(Assert.java:88)
   at org.junit.Assert.assertTrue(Assert.java:41)
   at 
 org.apache.hadoop.hbase.coprocessor.TestMasterObserver.testTableOperations(TestMasterObserver.java:1283)
 {code}
 The problem is the same as the one in createTable() and it is because the 
 sync version of the Admin operation is not sync, but relies on the last 
 operation on the server e.g. delete meta. 
 but the post-operation method of the coprocessor is called after meta is 
 deleted.. 
 long story short, the client is not really sync and it will be fixed by 
 HBASE-12439. so for now we should have the same fix we have for createTable 
 which is using a latch.
 (note: there are other tests failing for this reason e.g. AccessController, 
 NamespaceAuditor, ... but I'll fix them in another patch since we have 
 already a workaround in TestMasterObserver)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13172) TestDistributedLogSplitting.testThreeRSAbort fails several times on branch-1


[ 
https://issues.apache.org/jira/browse/HBASE-13172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353798#comment-14353798
 ] 

Hudson commented on HBASE-13172:


FAILURE: Integrated in HBase-1.0 #793 (See 
[https://builds.apache.org/job/HBase-1.0/793/])
HBASE-13172 TestDistributedLogSplitting.testThreeRSAbort fails several times on 
branch-1 (stack: rev 5af7bf149f297af3c132993eab6de5e1f78de41a)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java


 TestDistributedLogSplitting.testThreeRSAbort fails several times on branch-1
 

 Key: HBASE-13172
 URL: https://issues.apache.org/jira/browse/HBASE-13172
 Project: HBase
  Issue Type: Bug
  Components: test
Affects Versions: 1.1.0
Reporter: zhangduo
Assignee: zhangduo
 Fix For: 1.0.1, 1.1.0, 0.98.12

 Attachments: HBASE-13172-branch-1.patch


 The direct reason is we are stuck in ServerManager.isServerReachable.
 https://builds.apache.org/job/HBase-1.1/253/testReport/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testThreeRSAbort/
 {noformat}
 2015-03-06 04:06:19,430 DEBUG [AM.-pool300-t1] master.ServerManager(855): 
 Couldn't reach asf906.gq1.ygridcore.net,59366,1425614770146, try=0 of 10
 2015-03-06 04:07:10,545 DEBUG [AM.-pool300-t1] master.ServerManager(855): 
 Couldn't reach asf906.gq1.ygridcore.net,59366,1425614770146, try=9 of 10
 {noformat}
 The interval between first and last retry log is about 1 minute, and we only 
 wait 1 minute so the test is timeout.
 Still do not know why this happen.
 And at last there are lots of this 
 {noformat}
 2015-03-06 04:07:21,529 DEBUG [AM.-pool300-t1] master.ServerManager(855): 
 Couldn't reach asf906.gq1.ygridcore.net,59366,1425614770146, try=9 of 10
 org.apache.hadoop.hbase.ipc.StoppedRpcClientException
   at 
 org.apache.hadoop.hbase.ipc.RpcClientImpl.getConnection(RpcClientImpl.java:1261)
   at 
 org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1146)
   at 
 org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
   at 
 org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
   at 
 org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.getServerInfo(AdminProtos.java:22031)
   at 
 org.apache.hadoop.hbase.protobuf.ProtobufUtil.getServerInfo(ProtobufUtil.java:1797)
   at 
 org.apache.hadoop.hbase.master.ServerManager.isServerReachable(ServerManager.java:850)
   at 
 org.apache.hadoop.hbase.master.RegionStates.isServerDeadAndNotProcessed(RegionStates.java:843)
   at 
 org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1969)
   at 
 org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1576)
   at 
 org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48)
   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:744)
 {noformat}
 I think the problem is here
 {code:title=ServerManager.java}
 while (retryCounter.shouldRetry()) {
 ...
 try {
   retryCounter.sleepUntilNextRetry();
 } catch(InterruptedException ie) {
   Thread.currentThread().interrupt();
 }
 ...
 }
 {code}
 We need to break out of the while loop when getting InterruptedException, not 
 just mark current thread as interrupted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12586) Task 6 7 from HBASE-9117, delete all public HTable constructors and delete ConnectionManager#{delete,get}Connection


 [ 
https://issues.apache.org/jira/browse/HBASE-12586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Antonov updated HBASE-12586:

Status: Patch Available  (was: Open)

 Task 6  7 from HBASE-9117,  delete all public HTable constructors and delete 
 ConnectionManager#{delete,get}Connection
 --

 Key: HBASE-12586
 URL: https://issues.apache.org/jira/browse/HBASE-12586
 Project: HBase
  Issue Type: Task
Affects Versions: 2.0.0
Reporter: stack
Assignee: Mikhail Antonov
  Labels: beginner, beginners
 Attachments: HBASE-12586.patch, HBASE-12586.patch, HBASE-12586.patch, 
 HBASE-12586.patch


 Finish cleanup from HBASE-9117 removing old API.  This issue covers tasks 6 
 and 7 from the list here: 
 https://issues.apache.org/jira/browse/HBASE-9117?focusedCommentId=13919716page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13919716
 To be done in master branch only.
 Marked as beginner task.  The idea is straight-forward.  It is just a lot of 
 work going through all tests converting to use new API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12586) Task 6 7 from HBASE-9117, delete all public HTable constructors and delete ConnectionManager#{delete,get}Connection


 [ 
https://issues.apache.org/jira/browse/HBASE-12586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Antonov updated HBASE-12586:

Attachment: HBASE-12586.patch

Updated (fixing javadoc and checkstyle warnings)

 Task 6  7 from HBASE-9117,  delete all public HTable constructors and delete 
 ConnectionManager#{delete,get}Connection
 --

 Key: HBASE-12586
 URL: https://issues.apache.org/jira/browse/HBASE-12586
 Project: HBase
  Issue Type: Task
Affects Versions: 2.0.0
Reporter: stack
Assignee: Mikhail Antonov
  Labels: beginner, beginners
 Attachments: HBASE-12586.patch, HBASE-12586.patch, HBASE-12586.patch, 
 HBASE-12586.patch


 Finish cleanup from HBASE-9117 removing old API.  This issue covers tasks 6 
 and 7 from the list here: 
 https://issues.apache.org/jira/browse/HBASE-9117?focusedCommentId=13919716page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13919716
 To be done in master branch only.
 Marked as beginner task.  The idea is straight-forward.  It is just a lot of 
 work going through all tests converting to use new API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12586) Task 6 7 from HBASE-9117, delete all public HTable constructors and delete ConnectionManager#{delete,get}Connection


 [ 
https://issues.apache.org/jira/browse/HBASE-12586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Antonov updated HBASE-12586:

Status: Open  (was: Patch Available)

 Task 6  7 from HBASE-9117,  delete all public HTable constructors and delete 
 ConnectionManager#{delete,get}Connection
 --

 Key: HBASE-12586
 URL: https://issues.apache.org/jira/browse/HBASE-12586
 Project: HBase
  Issue Type: Task
Affects Versions: 2.0.0
Reporter: stack
Assignee: Mikhail Antonov
  Labels: beginner, beginners
 Attachments: HBASE-12586.patch, HBASE-12586.patch, HBASE-12586.patch, 
 HBASE-12586.patch


 Finish cleanup from HBASE-9117 removing old API.  This issue covers tasks 6 
 and 7 from the list here: 
 https://issues.apache.org/jira/browse/HBASE-9117?focusedCommentId=13919716page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13919716
 To be done in master branch only.
 Marked as beginner task.  The idea is straight-forward.  It is just a lot of 
 work going through all tests converting to use new API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13182) Test NamespaceAuditor/AccessController create/delete table is flaky

[
https://issues.apache.org/jira/browse/HBASE-13182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353977#comment-14353977
]

Hadoop QA commented on HBASE-13182:
---

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12703510/HBASE-13182-v0.patch
against master branch at commit 61cc8e0de12987b1d64fd06fa27a55c89c4a742f.
ATTACHMENT ID: 12703510

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)

{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.

{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.

{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors

{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.

{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100

{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.

{color:green}+1 core tests{color}. The patch passed unit tests in .

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13145//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13145//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13145//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13145//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13145//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13145//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13145//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13145//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13145//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13145//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13145//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13145//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13145//artifact/patchprocess/checkstyle-aggregate.html

Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13145//console

This message is automatically generated.

Test NamespaceAuditor/AccessController create/delete table is flaky
---

Key: HBASE-13182
URL: https://issues.apache.org/jira/browse/HBASE-13182
Project: HBase
Issue Type: Test
Components: test
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
Attachments: HBASE-13182-v0.patch

Similar to HBASE-13179
TestNamespaceAuditor and the two test AccessController fail when create or
delete table takes more time.and it is because the sync version of the Admin
operation is not sync, but relies on the last operation on the server e.g.
delete meta.
but the post-operation method of the coprocessor is called after meta is
deleted..
long story short, the client is not really sync and it will be fixed by
HBASE-12439.
similar to what we do in TestMasterObserver we can add an observer with a
countDownLatch to wait the postOpHandler call.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13182) Test NamespaceAuditor/AccessController create/delete table is flaky


 [ 
https://issues.apache.org/jira/browse/HBASE-13182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi updated HBASE-13182:

Status: Patch Available  (was: Open)

 Test NamespaceAuditor/AccessController create/delete table is flaky
 ---

 Key: HBASE-13182
 URL: https://issues.apache.org/jira/browse/HBASE-13182
 Project: HBase
  Issue Type: Test
  Components: test
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
 Attachments: HBASE-13182-v0.patch


 Similar to HBASE-13179
 TestNamespaceAuditor and the two test AccessController fail when create or 
 delete table takes more time.and it is because the sync version of the Admin 
 operation is not sync, but relies on the last operation on the server e.g. 
 delete meta. 
 but the post-operation method of the coprocessor is called after meta is 
 deleted.. 
 long story short, the client is not really sync and it will be fixed by 
 HBASE-12439. 
 similar to what we do in TestMasterObserver we can add an observer with a 
 countDownLatch to wait the postOpHandler call.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-13182) Test NamespaceAuditor/AccessController create/delete table is flaky

Matteo Bertozzi created HBASE-13182:
---

 Summary: Test NamespaceAuditor/AccessController create/delete 
table is flaky
 Key: HBASE-13182
 URL: https://issues.apache.org/jira/browse/HBASE-13182
 Project: HBase
  Issue Type: Test
  Components: test
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
 Attachments: HBASE-13182-v0.patch

Similar to HBASE-13179
TestNamespaceAuditor and the two test AccessController fail when create or 
delete table takes more time.and it is because the sync version of the Admin 
operation is not sync, but relies on the last operation on the server e.g. 
delete meta. 
but the post-operation method of the coprocessor is called after meta is 
deleted.. 
long story short, the client is not really sync and it will be fixed by 
HBASE-12439. 

similar to what we do in TestMasterObserver we can add an observer with a 
countDownLatch to wait the postOpHandler call.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13182) Test NamespaceAuditor/AccessController create/delete table is flaky


 [ 
https://issues.apache.org/jira/browse/HBASE-13182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi updated HBASE-13182:

Attachment: HBASE-13182-v0.patch

 Test NamespaceAuditor/AccessController create/delete table is flaky
 ---

 Key: HBASE-13182
 URL: https://issues.apache.org/jira/browse/HBASE-13182
 Project: HBase
  Issue Type: Test
  Components: test
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
 Attachments: HBASE-13182-v0.patch


 Similar to HBASE-13179
 TestNamespaceAuditor and the two test AccessController fail when create or 
 delete table takes more time.and it is because the sync version of the Admin 
 operation is not sync, but relies on the last operation on the server e.g. 
 delete meta. 
 but the post-operation method of the coprocessor is called after meta is 
 deleted.. 
 long story short, the client is not really sync and it will be fixed by 
 HBASE-12439. 
 similar to what we do in TestMasterObserver we can add an observer with a 
 countDownLatch to wait the postOpHandler call.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-13183) Make ZK tickTime configurable in standalone HBase

2015-03-09 Thread Alex Araujo (JIRA)

Alex Araujo created HBASE-13183:
---

 Summary: Make ZK tickTime configurable in standalone HBase
 Key: HBASE-13183
 URL: https://issues.apache.org/jira/browse/HBASE-13183
 Project: HBase
  Issue Type: Improvement
  Components: master
Affects Versions: 0.98.10.1
Reporter: Alex Araujo
Assignee: Alex Araujo
Priority: Minor


Standalone HBase does not allow the ZooKeeper tickTime to be configured for the 
MiniZooKeeperCluster. The default tickTime for MiniZooKeeperCluster is 2000 ms, 
which is too low for some use cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13183) Make ZK tickTime configurable in standalone HBase


[ 
https://issues.apache.org/jira/browse/HBASE-13183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353810#comment-14353810
 ] 

Andrew Purtell commented on HBASE-13183:


+1, thanks

 Make ZK tickTime configurable in standalone HBase
 -

 Key: HBASE-13183
 URL: https://issues.apache.org/jira/browse/HBASE-13183
 Project: HBase
  Issue Type: Improvement
  Components: master
Affects Versions: 0.98.10.1
Reporter: Alex Araujo
Assignee: Alex Araujo
Priority: Minor
 Attachments: HBASE-13183-0.98.patch


 Standalone HBase does not allow the ZooKeeper tickTime to be configured for 
 the MiniZooKeeperCluster. The default tickTime for MiniZooKeeperCluster is 
 2000 ms, which is too low for some use cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13167) The check for balancerCutoffTime is buggy


 [ 
https://issues.apache.org/jira/browse/HBASE-13167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Antonov updated HBASE-13167:

Attachment: HBASE-13167.patch

Trivial patch.

A note..if max cutoff time isn't set, we're currently defaulting to period 
time. I sort of remember it used to be defaulted to half of balancer period 
time (2.5 min by default). Do we want to stick with cutoff time == period time 
behavior?

 The check for balancerCutoffTime is buggy
 -

 Key: HBASE-13167
 URL: https://issues.apache.org/jira/browse/HBASE-13167
 Project: HBase
  Issue Type: Bug
  Components: Balancer
Affects Versions: 0.98.8
Reporter: Tianyin Xu
 Attachments: HBASE-13167.patch


 In HMaster, the checking logic for balancerCutoffTime in buggy.
 See the following code:
 {code:title=HMaster.java|borderStyle=solid}
 1419   private int getBalancerCutoffTime() {
 1420 int balancerCutoffTime =
 1421   getConfiguration().getInt(hbase.balancer.max.balancing, -1);
 1422 if (balancerCutoffTime == -1) {
 1423   // No time period set so create one
 1424   int balancerPeriod =
 1425 getConfiguration().getInt(hbase.balancer.period, 30);
 1426   balancerCutoffTime = balancerPeriod;
 1427   // If nonsense period, set it to balancerPeriod
 1428   if (balancerCutoffTime = 0) balancerCutoffTime = balancerPeriod;
 1429 }
 1430 return balancerCutoffTime;
 1431   }
 {code}
 Basically, the check for nonsense period is no effect, because  
 balancerCutoffTime is always assigned to be balancerPeriod whatever the value 
 is. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13121) Async wal replication for region replicas and dist log replay does not work together


[ 
https://issues.apache.org/jira/browse/HBASE-13121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353951#comment-14353951
 ] 

Hudson commented on HBASE-13121:


FAILURE: Integrated in HBase-TRUNK #6226 (See 
[https://builds.apache.org/job/HBase-TRUNK/6226/])
HBASE-13121 Async wal replication for region replicas and dist log replay does 
not work together (enis: rev 5025d3aa91d18310fc4d738114ee2b58e48c46c2)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/FinishRegionRecoveringHandler.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicaFailover.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALPrettyPrinter.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoveringRegionWatcher.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RegionReplicaReplicationEndpoint.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/BaseWALEntryFilter.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpointNoMaster.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RpcClientImpl.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZkSplitLogWorkerCoordination.java


 Async wal replication for region replicas and dist log replay does not work 
 together
 

 Key: HBASE-13121
 URL: https://issues.apache.org/jira/browse/HBASE-13121
 Project: HBase
  Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Fix For: 2.0.0, 1.1.0

 Attachments: hbase-13121_v1.patch, hbase-13121_v2.patch


 We had not tested dist log replay while testing async wal replication for 
 region replicas. There seems to be a couple of issues, but fixable. 
 The distinction for dist log replay is that, the region will be opened for 
 recovery and regular writes when a primary fails over. This causes the region 
 open event marker to be written to WAL, but at this time, the region actually 
 does not contain all the edits flushed (since it is still recovering). If 
 secondary regions see this event, and picks up all the files in the region 
 open event marker, then they can drop edits. 
 The solution is: 
  - Only write the region open event marker to WAL when region is out of 
 recovering mode. 
  - Force a flush out of recovering mode. This ensures that all data is force 
 flushed in this case. Before the region open event marker is written, we 
 guarantee that all data in the region is flushed, so the list of files in the 
 event marker is complete.  
  - Edits coming from recovery are re-written to WAL when recovery is in 
 action. These edits will have a larger seqId then their original seqId. If 
 this is the case, we do not replicate these edits to the secondary replicas. 
 Since the dist log replay recovers edits out of order (coming from parallel 
 replays from WAL file split tasks), this ensures that TIMELINE consistency is 
 respected and edits are not seen out of order in secondaries. These edits are 
 seen from secondaries via the forced flush event.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12990) MetaScanner should be replaced by MetaTableAccessor

2015-03-09 Thread Andrey Stepachev (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352774#comment-14352774
 ] 

Andrey Stepachev commented on HBASE-12990:
--

pushed to master, thanks [~stack] for helping with this patch.
btw, this patch will not apply cleanly on branch1 (there some 
differences). So I'll work that out for branch-1 and post shortly.


 MetaScanner should be replaced by MetaTableAccessor
 ---

 Key: HBASE-12990
 URL: https://issues.apache.org/jira/browse/HBASE-12990
 Project: HBase
  Issue Type: Improvement
  Components: Client
Affects Versions: 2.0.0
Reporter: Andrey Stepachev
Assignee: Andrey Stepachev
 Attachments: HBASE-12990.patch, HBASE-12990.v2.patch, 
 HBASE-12990.v3.patch, HBASE-12990.v4.patch, HBASE-12990.v5.patch, 
 HBASE-12990.v5.patch, HBASE-12990.v5.patch, HBASE-12990.v6.patch, 
 HBASE-12990.v7.patch, HBASE-12990.v7.patch, HBASE-12990.v7.patch, 
 HBASE-12990.v8.patch


 MetaScanner and MetaTableAccessor do similar things, but seems they tend to 
 diverge. Let's have only one thing to enquiry META.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13176) Flakey TestZooKeeper test.

2015-03-09 Thread Andrey Stepachev (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352779#comment-14352779
 ] 

Andrey Stepachev commented on HBASE-13176:
--

[~stack] sure, but that renders this test flakey. Need some other measure for 
assertation.
I'll find how that can be fixed to ensure that all actually ciritical listeners 
are there upon
failover. stay tuned.

 Flakey TestZooKeeper test.
 --

 Key: HBASE-13176
 URL: https://issues.apache.org/jira/browse/HBASE-13176
 Project: HBase
  Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Andrey Stepachev
Assignee: Andrey Stepachev

 Test fails with wrong number of listeners:
 {code}
 Tests run: 11, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 202.248 sec 
  FAILURE! - in org.apache.hadoop.hbase.TestZooKeeper
 testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(org.apache.hadoop.hbase.TestZooKeeper)
  Time elapsed: 48.687 sec  FAILURE!
 java.lang.AssertionError: expected:16 but was:15
 at org.junit.Assert.fail(Assert.java:88)
 at org.junit.Assert.failNotEquals(Assert.java:743)
 at org.junit.Assert.assertEquals(Assert.java:118)
 at org.junit.Assert.assertEquals(Assert.java:555)
 at org.junit.Assert.assertEquals(Assert.java:542)
 at 
 org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:518)
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13183) Make ZK tickTime configurable in standalone HBase


[ 
https://issues.apache.org/jira/browse/HBASE-13183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354230#comment-14354230
 ] 

Hudson commented on HBASE-13183:


FAILURE: Integrated in HBase-1.1 #265 (See 
[https://builds.apache.org/job/HBase-1.1/265/])
HBASE-13183 Make ZK tickTime configurable in standalone HBase (Alex Araujo) 
(apurtell: rev 4afae59cfaf4e10dc97e35ce4c6d2aa24e40f532)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMasterCommandLine.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java


 Make ZK tickTime configurable in standalone HBase
 -

 Key: HBASE-13183
 URL: https://issues.apache.org/jira/browse/HBASE-13183
 Project: HBase
  Issue Type: Improvement
  Components: master
Affects Versions: 0.98.10.1
Reporter: Alex Araujo
Assignee: Alex Araujo
Priority: Minor
 Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.12

 Attachments: HBASE-13183-0.98.patch, HBASE-13183.patch


 Standalone HBase does not allow the ZooKeeper tickTime to be configured for 
 the MiniZooKeeperCluster. The default tickTime for MiniZooKeeperCluster is 
 2000 ms, which is too low for some use cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12405) WAL accounting by Store


[ 
https://issues.apache.org/jira/browse/HBASE-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354274#comment-14354274
 ] 

stack commented on HBASE-12405:
---

[~Apache9] Do it yourself (smile)!

Suggest off in 1.1 and on in 2.0.

I made a subissue to do up an ITBLL that verifies no data loss. If that passes, 
lets argue that by CF is on by default in 1.1 too.

 WAL accounting by Store
 ---

 Key: HBASE-12405
 URL: https://issues.apache.org/jira/browse/HBASE-12405
 Project: HBase
  Issue Type: Improvement
  Components: wal
Affects Versions: 2.0.0, 1.1.0
Reporter: zhangduo
Assignee: zhangduo
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-12405.patch, HBASE-12405_1.patch, 
 HBASE-12405_2.patch, HBASE-12405_3.patch, HBASE-12405_4.patch, 
 HBASE-12405_5.patch


 HBASE-10201 has made flush decisions per Store, but has not done enough work 
 on HLog, so there are two problems:
 1. We record minSeqId both in HRegion and FSHLog, which is a duplication.
 2. There maybe holes in WAL accounting.
 For example, assume family A with sequence id 1 and 3, family B with 
 seqId 2. If we flush family A, we can only record that WAL before sequence id 
 1 can be removed safely. If we do a replay at this point, sequence id 3 will 
 also be replayed which is unnecessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-7126) Update website with info on how to report security bugs


[ 
https://issues.apache.org/jira/browse/HBASE-7126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354271#comment-14354271
 ] 

stack commented on HBASE-7126:
--

+1 as a start [~misty]

 Update website with info on how to report security bugs 
 

 Key: HBASE-7126
 URL: https://issues.apache.org/jira/browse/HBASE-7126
 Project: HBase
  Issue Type: Task
  Components: documentation
Reporter: Eli Collins
Assignee: Misty Stanley-Jones
Priority: Critical
  Labels: website
 Attachments: HBASE-7126.patch


 The HBase website should be updated with information on how to report 
 potential security vulnerabilities. In Hadoop land we have a private security 
 list that anyone case post to that we point to on our list page: Hadoop 
 example http://hadoop.apache.org/general_lists.html#Security.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-7126) Update website with info on how to report security bugs

2015-03-09 Thread Misty Stanley-Jones (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354277#comment-14354277
 ] 

Misty Stanley-Jones commented on HBASE-7126:


What else would you like to see, besides a start? :)

 Update website with info on how to report security bugs 
 

 Key: HBASE-7126
 URL: https://issues.apache.org/jira/browse/HBASE-7126
 Project: HBase
  Issue Type: Task
  Components: documentation
Reporter: Eli Collins
Assignee: Misty Stanley-Jones
Priority: Critical
  Labels: website
 Attachments: HBASE-7126.patch


 The HBase website should be updated with information on how to report 
 potential security vulnerabilities. In Hadoop land we have a private security 
 list that anyone case post to that we point to on our list page: Hadoop 
 example http://hadoop.apache.org/general_lists.html#Security.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12586) Task 6 7 from HBASE-9117, delete all public HTable constructors and delete ConnectionManager#{delete,get}Connection


[ 
https://issues.apache.org/jira/browse/HBASE-12586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354284#comment-14354284
 ] 

stack commented on HBASE-12586:
---

Site failure is very likely unrelated. Misty just fixed it. No harm in letting 
next hadoopqa run complete before commit. Good on you [~mantonov]


 Task 6  7 from HBASE-9117,  delete all public HTable constructors and delete 
 ConnectionManager#{delete,get}Connection
 --

 Key: HBASE-12586
 URL: https://issues.apache.org/jira/browse/HBASE-12586
 Project: HBase
  Issue Type: Task
Affects Versions: 2.0.0
Reporter: stack
Assignee: Mikhail Antonov
  Labels: beginner, beginners
 Attachments: HBASE-12586.patch, HBASE-12586.patch, HBASE-12586.patch, 
 HBASE-12586.patch


 Finish cleanup from HBASE-9117 removing old API.  This issue covers tasks 6 
 and 7 from the list here: 
 https://issues.apache.org/jira/browse/HBASE-9117?focusedCommentId=13919716page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13919716
 To be done in master branch only.
 Marked as beginner task.  The idea is straight-forward.  It is just a lot of 
 work going through all tests converting to use new API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-7126) Update website with info on how to report security bugs

2015-03-09 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354285#comment-14354285
 ] 

Sean Busbey commented on HBASE-7126:


a link to the ASF policy on reported vulnerabilities would be a nice addition 
(ref http://apache.org/security/), perhaps with a note that folks who wish to 
send an encrypted report can use the GPG details provided for the general ASF 
security list (with a warning that responses will take longer).

 Update website with info on how to report security bugs 
 

 Key: HBASE-7126
 URL: https://issues.apache.org/jira/browse/HBASE-7126
 Project: HBase
  Issue Type: Task
  Components: documentation
Reporter: Eli Collins
Assignee: Misty Stanley-Jones
Priority: Critical
  Labels: website
 Attachments: HBASE-7126.patch


 The HBase website should be updated with information on how to report 
 potential security vulnerabilities. In Hadoop land we have a private security 
 list that anyone case post to that we point to on our list page: Hadoop 
 example http://hadoop.apache.org/general_lists.html#Security.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13071) Hbase Streaming Scan Feature

[
https://issues.apache.org/jira/browse/HBASE-13071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354312#comment-14354312
]

stack commented on HBASE-13071:
---

[~eshcar] How should I test so your patch shines?

Hbase Streaming Scan Feature

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12405) WAL accounting by Store

2015-03-09 Thread zhangduo (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354311#comment-14354311
 ] 

zhangduo commented on HBASE-12405:
--

Yes sir!

 WAL accounting by Store
 ---

 Key: HBASE-12405
 URL: https://issues.apache.org/jira/browse/HBASE-12405
 Project: HBase
  Issue Type: Improvement
  Components: wal
Affects Versions: 2.0.0, 1.1.0
Reporter: zhangduo
Assignee: zhangduo
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-12405.patch, HBASE-12405_1.patch, 
 HBASE-12405_2.patch, HBASE-12405_3.patch, HBASE-12405_4.patch, 
 HBASE-12405_5.patch


 HBASE-10201 has made flush decisions per Store, but has not done enough work 
 on HLog, so there are two problems:
 1. We record minSeqId both in HRegion and FSHLog, which is a duplication.
 2. There maybe holes in WAL accounting.
 For example, assume family A with sequence id 1 and 3, family B with 
 seqId 2. If we flush family A, we can only record that WAL before sequence id 
 1 can be removed safely. If we do a replay at this point, sequence id 3 will 
 also be replayed which is unnecessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13169) ModifyTable increasing the region replica count should also auto-setup RRRE


[ 
https://issues.apache.org/jira/browse/HBASE-13169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354327#comment-14354327
 ] 

Hudson commented on HBASE-13169:


FAILURE: Integrated in HBase-TRUNK #6229 (See 
[https://builds.apache.org/job/HBase-TRUNK/6229/])
Revert HBASE-13169 ModifyTable increasing the region replica count should also 
auto-setup RRRE (enis: rev 21b27c865087580d75f08819144067308c4fd80c)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ModifyTableHandler.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java


 ModifyTable increasing the region replica count should also auto-setup RRRE
 ---

 Key: HBASE-13169
 URL: https://issues.apache.org/jira/browse/HBASE-13169
 Project: HBase
  Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Fix For: 2.0.0, 1.1.0

 Attachments: hbase-13169_v1.patch


 This is a follow up to Async WAL replication jira (HBASE-11568). In 
 HBASE-11568 we setup replication automatically in create table if the 
 configuration is enabled. 
 We should do the same in case a table with region replication = 1 is 
 modified, and region replication is set to 1. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13186) HBase build error due to checkstyle