[jira] [Updated] (HBASE-17336) get/update replication peer config requests should be routed through master

[Guanghao Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-17336:
---
Release Note: Get/update replication peer config requests will be routed 
through master.

> get/update replication peer config requests should be routed through master
> ---
>
> Key: HBASE-17336
> URL: https://issues.apache.org/jira/browse/HBASE-17336
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17336-v1.patch, HBASE-17336-v2.patch, 
> HBASE-17336-v3.patch, HBASE-17336-v4.patch, HBASE-17336-v5.patch
>
>
> As HBASE-11392 description says, we should move replication operations to be 
> routed through master.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-10367) RegionServer graceful stop / decommissioning

[Lars George (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-10367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784771#comment-15784771
 ] 

Lars George commented on HBASE-10367:
-

The state has not changed and I have users I talk to struggle with this mess 
way too often. We need to have a clear story on this. Voting up!

> RegionServer graceful stop / decommissioning
> 
>
> Key: HBASE-10367
> URL: https://issues.apache.org/jira/browse/HBASE-10367
> Project: HBase
>  Issue Type: Improvement
>Reporter: Enis Soztutar
>
> Right now, we have a weird way of node decommissioning / graceful stop, which 
> is a graceful_stop.sh bash script, and a region_mover ruby script, and some 
> draining server support which you have to manually write to a znode 
> (really!). Also draining servers is only partially supported in LB operations 
> (LB does take that into account for roundRobin assignment, but not for normal 
> balance) 
> See 
> http://hbase.apache.org/book/node.management.html and HBASE-3071
> I think we should support graceful stop as a first class citizen. Thinking 
> about it, it seems that the difference between regionserver stop and graceful 
> stop is that regionserver stop will close the regions, but the master will 
> only assign them after the znode is deleted. 
> In the new master design (or even before), if we allow RS to be able to close 
> regions on its own (without master initiating it), then graceful stop becomes 
> regular stop. The RS already closes the regions cleanly, and will reject new 
> region assignments, so that we don't need much of the balancer or draining 
> server trickery. 
> This ties into the new master/AM redesign (HBASE-5487), but still deserves 
> it's own jira. Let's use this to brainstorm on the design. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17374) ZKPermissionWatcher crashed when grant after region close

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784770#comment-15784770
 ] 

Hudson commented on HBASE-17374:


SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2219 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/2219/])
HBASE-17374 ZKPermissionWatcher crashed when grant after region close (tedyu: 
rev 001a26d404ca39ab6dbb9efeb59c08f20938f112)
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/ZKPermissionWatcher.java


> ZKPermissionWatcher crashed when grant after region close
> -
>
> Key: HBASE-17374
> URL: https://issues.apache.org/jira/browse/HBASE-17374
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.98.15
>Reporter: Liu Junhong
>Assignee: Liu Junhong
>Priority: Critical
> Attachments: 0001-fix-for-HBASE-17374-20161228.patch, 
> 0001-fix-for-HBASE-17374.patch
>
>
> It was occurred many time that  I granted some permission,  but few of some 
> regionservers was not token effect and must be restart . When I look up logs, 
>  I found that :
> 2016-12-08 15:00:26,238 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] handler.CloseRegionHandler 
> (CloseRegionHandler.java:process(128)) - Processing close of 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> {color:red} 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1163)) - Closing 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.: disabling 
> compactions & flushes {color}
> 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1190)) - Updates disabled for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> 2016-12-08 15:00:26,242 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1753)) - Started memstore flush for 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14., current 
> region memstore size 160
> 2016-12-08 15:00:26,284 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.DefaultStoreFlusher (DefaultStoreFlusher.java:flushSnapshot(95)) 
> - Flushed, sequenceid=6, memsize=160, hasBloomFilter=true, into tmp file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,303 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.HRegionFileSystem (HRegionFileSystem.java:commitStoreFile(370)) 
> - Committing store file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
>  as 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,318 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HStore 
> (HStore.java:commitFile(877)) - Added 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3,
>  entries=1, sequenceid=6, filesize=985
> 2016-12-08 15:00:26,319 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1920)) - Finished memstore flush of 
> ~160/160, currentsize=0/0 for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14. in 77ms, 
> sequenceid=6, compaction requested=false
> 2016-12-08 15:00:26,323 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf1
> 2016-12-08 15:00:26,325 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf2
> 2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.token.TokenProvider
> {color:red}  2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.AccessController  {color}
> 2016-12-08 15:00:26,327 DEBUG 
> [R

[jira] [Commented] (HBASE-17320) Add inclusive/exclusive support for startRow and endRow of scan

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784769#comment-15784769
 ] 

Hudson commented on HBASE-17320:


SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2219 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/2219/])
HBASE-17320 Add inclusive/exclusive support for startRow and endRow of 
(zhangduo: rev 05b1d918b0a845ced066a66b187823c357ed673d)
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/RawScanQueryMatcher.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/AbstractTestAsyncTableScan.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileManager.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncClientScanner.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncSmallScanRpcRetryingCaller.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestRawAsyncTableScan.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncTableResultScanner.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRpcRetryingCallerFactory.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StripeStoreFileManager.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionUtils.java
* (edit) 
hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/ScanQueryMatcher.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/UserScanQueryMatcher.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncScanSingleRegionRpcRetryingCaller.java
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/io/TimeRange.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/LegacyScanQueryMatcher.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/NormalUserScanQueryMatcher.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStripeStoreFileManager.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Scan.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* (edit) hbase-protocol/src/main/protobuf/Client.proto
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/CompactionScanQueryMatcher.java
* (edit) hbase-protocol-shaded/src/main/protobuf/Client.proto
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReversedRegionScannerImpl.java
* (edit) 
hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos.java


> Add inclusive/exclusive support for startRow and endRow of scan
> ---
>
> Key: HBASE-17320
> URL: https://issues.apache.org/jira/browse/HBASE-17320
> Project: HBase
>  Issue Type: Sub-task
>  Components: Client, scan
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17320-v1.patch, HBASE-17320-v2.patch, 
> HBASE-17320-v3.patch, HBASE-17320-v4.patch, HBASE-17320-v5.patch, 
> HBASE-17320-v6.patch, HBASE-17320.patch
>
>
> This is especially useful when doing reverse scan. HBASE-17168 maybe a more 
> powerful solution but we need to be careful about the atomicity, and I do not 
> think we will provide the feature to end user. But I think it is OK to 
> provide inclusive/exclusive option to end user.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17238) Wrong in-memory hbase:meta location causing SSH failure

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784761#comment-15784761
 ] 

Hudson commented on HBASE-17238:


SUCCESS: Integrated in Jenkins build HBase-1.1-JDK8 #1914 (See 
[https://builds.apache.org/job/HBase-1.1-JDK8/1914/])
HBASE-17238 Wrong in-memory hbase:meta location causing SSH failure 
(syuanjiangdev: rev a1d900db44cf7834f714b2ad4720795ddbc45e12)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMetaWithReplicas.java


> Wrong in-memory hbase:meta location causing SSH failure
> ---
>
> Key: HBASE-17238
> URL: https://issues.apache.org/jira/browse/HBASE-17238
> Project: HBase
>  Issue Type: Bug
>  Components: Region Assignment
>Affects Versions: 1.1.0
>Reporter: Stephen Yuan Jiang
>Assignee: Stephen Yuan Jiang
>Priority: Critical
> Fix For: 1.3.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: HBASE-17238.v1-branch-1.1.patch, 
> HBASE-17238.v1-branch-1.patch, HBASE-17238.v2-branch-1.1.patch
>
>
> In HBase 1.x, if HMaster#assignMeta() assigns a non-DEFAULT_REPLICA_ID 
> hbase:meta region, it would wrongly update the DEFAULT_REPLICA_ID hbase:meta 
> region in-memory.  This caused the in-memory region state has wrong RS 
> information for default replica hbase:meta region.  One of the problem we saw 
> is a wrong type of SSH could be chosen and causing problems.
> {code}
> void assignMeta(MonitoredTask status, Set 
> previouslyFailedMetaRSs, int replicaId)
>   throws InterruptedException, IOException, KeeperException {
> // Work on meta region
> ...
> if (replicaId == HRegionInfo.DEFAULT_REPLICA_ID) {
>   status.setStatus("Assigning hbase:meta region");
> } else {
>   status.setStatus("Assigning hbase:meta region, replicaId " + replicaId);
> }
> // Get current meta state from zk.
> RegionStates regionStates = assignmentManager.getRegionStates();
> RegionState metaState = 
> MetaTableLocator.getMetaRegionState(getZooKeeper(), replicaId);
> HRegionInfo hri = 
> RegionReplicaUtil.getRegionInfoForReplica(HRegionInfo.FIRST_META_REGIONINFO,
> replicaId);
> ServerName currentMetaServer = metaState.getServerName();
> ...
> boolean rit = this.assignmentManager.
>   processRegionInTransitionAndBlockUntilAssigned(hri);
> boolean metaRegionLocation = metaTableLocator.verifyMetaRegionLocation(
>   this.getConnection(), this.getZooKeeper(), timeout, replicaId);
> ...
> } else {
>   // Region already assigned. We didn't assign it. Add to in-memory state.
>   regionStates.updateRegionState(
> HRegionInfo.FIRST_META_REGIONINFO, State.OPEN, currentMetaServer); 
> <<--- Wrong region to update -->>
>   this.assignmentManager.regionOnline(
> HRegionInfo.FIRST_META_REGIONINFO, currentMetaServer); <<--- Wrong 
> region to update -->>
> }
> ...
> {code}
> Here is the problem scenario:
> Step 1: master failovers (or starts could have the same issue) and find 
> default replica of hbase:meta is in rs1.
> {noformat}
> 2016-11-26 00:06:36,590 INFO org.apache.hadoop.hbase.master.ServerManager: 
> AssignmentManager hasn't finished failover cleanup; waiting
> 2016-11-26 00:06:36,591 INFO org.apache.hadoop.hbase.master.HMaster: 
> hbase:meta with replicaId 0 assigned=0, rit=false, 
> location=rs1,60200,1480103147220
> {noformat}
> Step 2: master finds that replica 1 of hbase:meta is unassigned, therefore, 
> HMaster#assignMeta() is called and assign the replica 1 region to rs2, but at 
> the end, it wrongly updates the in-memory state of default replica to rs2
> {noformat}
> 2016-11-26 00:08:21,741 DEBUG org.apache.hadoop.hbase.master.RegionStates: 
> Onlined 1588230740 on rs2,60200,1480102993815 {ENCODED => 1588230740, NAME => 
> 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}
> 2016-11-26 00:08:21,741 INFO org.apache.hadoop.hbase.master.RegionStates: 
> Offlined 1588230740 from rs1,60200,1480103147220
> 2016-11-26 00:08:21,741 INFO org.apache.hadoop.hbase.master.HMaster: 
> hbase:meta with replicaId 1 assigned=0, rit=false, 
> location=rs2,60200,1480102993815
> {noformat}
> Step 3: now rs1 is down, master needs to choose which SSH to call 
> (MetaServerShutdownHandler or normal ServerShutdownHandler) - in this case, 
> MetaServerShutdownHandler should be chosen; however, due to wrong in-memory 
> location, normal ServerShutdownHandler was chosen:
> {noformat}
> 2016-11-26 00:08:33,995 INFO 
> org.apache.hadoop.hbase.zookeeper.RegionServerTracker: RegionServer ephemeral 
> node deleted, processing expiration [rs1,60200,1480103147220]
> 2016-11-26 00:08:33,998 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: based on AM, 

[jira] [Commented] (HBASE-17336) get/update replication peer config requests should be routed through master

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784759#comment-15784759
 ] 

Hadoop QA commented on HBASE-17336:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m 0s 
{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m 0s 
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 4 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 17s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 18m 
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m 
9s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s 
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 11s 
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 24s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 46s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 46s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 46s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 18m 
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m 
3s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 52 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
28m 22s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 3m 
4s {color} | {color:green} the patch passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s 
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 7s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 52s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 27s 
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 58s 
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 86m 38s 
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 41s 
{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 103m 50s 
{color} | {color:green} root in the patch passed. 

[jira] [Commented] (HBASE-17372) Make AsyncTable thread safe

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784638#comment-15784638
 ] 

stack commented on HBASE-17372:
---

My comment comes from seeing how much you are able to cleanup when you 
aggregate all timeouts in a single class (RetryConfig). Having a generic 
operationconfig object instead of a particular retryconfig seems like a safer 
bet.

If RetyConfig is in classes marked private and it seems to be all client-side 
-- no protos involved -- then we can change it later if we need to add config 
beyond retry/timeout. In this case, +1 on the patch.

> Make AsyncTable thread safe
> ---
>
> Key: HBASE-17372
> URL: https://issues.apache.org/jira/browse/HBASE-17372
> Project: HBase
>  Issue Type: Sub-task
>  Components: asyncclient, Client
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17372-v1.patch, HBASE-17372.patch
>
>
> The most methods are already thread safe. The problem is that we have some 
> methods that used to set timeout, we need to remove these methods and add a 
> parameter for each call to specific timeout settings.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-9465) Push entries to peer clusters serially

[Phil Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784611#comment-15784611
 ] 

Phil Yang commented on HBASE-9465:
--

Thanks [~stack] for your comment.

bq. The latter keeps only the latest opening? Could it not have been amended to 
keep all?
info:seqnumDuringOpen saves sequence id in its value so it can only saves one 
value. But for replication we need all sequence id for each time a region being 
opened. For compatibility I didn't want to change this column so I create a new 
one. And a independent family(rep_barrier) can prevent read too much data when 
we only want to read info family.

bq. Do we have the position in two places now? Do we need to update zk if we've 
updated meta?
We save position for each WAL files in ZK(of course we have HBASE-15867 now). 
For serial replication we save position for each region in meta. Two different 
positions.

bq. Because? It is not continuous against a peer? Seqid is 'continuous' within 
a region?
If I am not wrong, openSeq is the max sequence +1 , and the first log's 
sequence after opening is openSeq+1, so in fact we will not have a log in WAL 
whose seq is openSeq.

bq. Why the -1 in the above? Because we add 1 when we open a region?
Yes

bq. We need to write this assumption into the code around where splits bring up 
daughters on same RS as parent. This policy could change (Y! have put up a 
patch to make it so splits do not bring up daughters on same servers as parent 
region).
Yes, the doc is old now. We have a special logic for split/merge in master 
branch now.

bq. This is another assumption of the design that needs to be marked in code so 
when this changes, we'll accommodate the fix here.
OK. And in fact I have a plan to improve this. We can use only one thread to 
read the WAL for non-recovery sources to reduce I/O pressure and should have 
some logic when one of peers being blocked, will file a issue when I am going 
to do this.

bq. We do not write to the hbase:meta state of WALs, unless REPLICATION_SCOPE 
is set to 2?
Yes

bq. Can you say more on this?
WAL is server-level and replication source is peer level. So if in a peer a 
region's log can not be pushed because of serial replication. All logs for this 
peer after this log are also blocked. To prevent this we have to split these 
tables/cfs into different peers.

bq. When would an Entry not be ready to push?
When the region this entry belongs to has some logs whose seq is smaller than 
this entry and they are not pushed to peer cluster, this entry can not be 
pushed.

bq. Do we have an idea of how much extra load this fix puts on hbase:meta?
Puts to rep_barrier and rep_meta are in batch mutation when region opens, so I 
think it is not a big extra load. rep_position is updated frequently whose QPS 
is same as positions logging on ZK. They only happen when some families enable 
this feature.

bq. How do we get insight on say delay that is happening because another RS's 
thread is (we think) replaying a WAL?
As long as we think this log can not be pushed, normally something is delayed, 
maybe failover or region moved. But we can not know the reason, we can only 
wait the work done. Unless there is a bug finally they can be pushed.

These days I am working on this to enable this feature in our production 
cluster, maybe there will be something need to improved. Hope I can say this 
feature is stable when 1.4 release :)

> Push entries to peer clusters serially
> --
>
> Key: HBASE-9465
> URL: https://issues.apache.org/jira/browse/HBASE-9465
> Project: HBase
>  Issue Type: New Feature
>  Components: regionserver, Replication
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Honghua Feng
>Assignee: Phil Yang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-9465-branch-1-v1.patch, 
> HBASE-9465-branch-1-v1.patch, HBASE-9465-branch-1-v2.patch, 
> HBASE-9465-branch-1-v3.patch, HBASE-9465-branch-1-v4.patch, 
> HBASE-9465-branch-1-v4.patch, HBASE-9465-v1.patch, HBASE-9465-v2.patch, 
> HBASE-9465-v2.patch, HBASE-9465-v3.patch, HBASE-9465-v4.patch, 
> HBASE-9465-v5.patch, HBASE-9465-v6.patch, HBASE-9465-v6.patch, 
> HBASE-9465-v7.patch, HBASE-9465-v7.patch, HBASE-9465.pdf
>
>
> When region-move or RS failure occurs in master cluster, the hlog entries 
> that are not pushed before region-move or RS-failure will be pushed by 
> original RS(for region move) or another RS which takes over the remained hlog 
> of dead RS(for RS failure), and the new entries for the same region(s) will 
> be pushed by the RS which now serves the region(s), but they push the hlog 
> entries of a same region concurrently without coordination.
> This treatment can possibly lead to data inconsistency between master and 
> peer clusters:
> 1. there are put and then delete written to master 

[jira] [Comment Edited] (HBASE-17382) Give RegionLocateType a better name

[Weizhan Zeng (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784566#comment-15784566
 ] 

Weizhan Zeng edited comment on HBASE-17382 at 12/29/16 5:48 AM:


Sorry ! Since "It is a type of the locating operation", what about 
"LocateRegionOperationType"? 


was (Author: weizhan zeng):
Sorry ! Since "It is a type of the locating operation", what about 
"RegionLocateOperationType"? 

> Give RegionLocateType a better name
> ---
>
> Key: HBASE-17382
> URL: https://issues.apache.org/jira/browse/HBASE-17382
> Project: HBase
>  Issue Type: Improvement
>Reporter: Duo Zhang
>Assignee: Jan Hentschel
> Attachments: HBASE-17382.master.001.patch
>
>
> Pointed out by [~tedyu] that 'Locate' is a verb and usually we need a noun 
> here. 'Locating' or 'Location'?
> Suggestion are welcomed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17382) Give RegionLocateType a better name

[Weizhan Zeng (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784566#comment-15784566
 ] 

Weizhan Zeng commented on HBASE-17382:
--

Sorry ! Since "It is a type of the locating operation", what about 
"RegionLocateOperationType"? 

> Give RegionLocateType a better name
> ---
>
> Key: HBASE-17382
> URL: https://issues.apache.org/jira/browse/HBASE-17382
> Project: HBase
>  Issue Type: Improvement
>Reporter: Duo Zhang
>Assignee: Jan Hentschel
> Attachments: HBASE-17382.master.001.patch
>
>
> Pointed out by [~tedyu] that 'Locate' is a verb and usually we need a noun 
> here. 'Locating' or 'Location'?
> Suggestion are welcomed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17149) Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor multiple times

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784564#comment-15784564
 ] 

Hudson commented on HBASE-17149:


SUCCESS: Integrated in Jenkins build HBase-1.3-JDK8 #86 (See 
[https://builds.apache.org/job/HBase-1.3-JDK8/86/])
HBASE-17149 Procedure V2 - Fix nonce submission to avoid unnecessary 
(syuanjiangdev: rev e06c3249046f33aab5ff74d1bdbad34e1055326f)
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestTruncateTableProcedure.java
* (edit) 
hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/TestProcedureRecovery.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockNoopMasterServices.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestAddColumnFamilyProcedure.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestCreateTableProcedure.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestDeleteNamespaceProcedure.java
* (add) 
hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/TestProcedureNonce.java
* (edit) 
hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/ProcedureTestingUtility.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestDisableTableProcedure.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestMasterProcedureEvents.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestProcedureAdmin.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestCreateNamespaceProcedure.java
* (edit) 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureUtil.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestDeleteColumnFamilyProcedure.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestModifyNamespaceProcedure.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestModifyColumnFamilyProcedure.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableNamespaceManager.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestModifyTableProcedure.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestDeleteTableProcedure.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestEnableTableProcedure.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java


> Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor 
> multiple times
> ---
>
> Key: HBASE-17149
> URL: https://issues.apache.org/jira/browse/HBASE-17149
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: 17149.branch-1.incomplete.txt, 
> HBASE-17149-addendum.v1-master.patch, HBASE-17149.master.001.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.003.patch, 
> HBASE-17149.v1-branch-1.2.patch, HBASE-17149.v1-branch-1.patch, nonce.patch
>
>
> instead of having all the logic in submitProcedure(), split in 
> registerNonce() + submitProcedure().
> In this case we can avoid calling the coprocessor twice and having a clean 
> submit logic knowing that there will only be one submission.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16421) Introducing the CellChunkMap as a new additional index variant in the MemStore

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784518#comment-15784518
 ] 

ramkrishna.s.vasudevan commented on HBASE-16421:


Sure. I will try using this option. But what I observed was that with 100 
threads and 0.7 as the update/insert proportion still it wasn't enough to 
generate the load. I will try once again then.

> Introducing the CellChunkMap as a new additional index variant in the MemStore
> --
>
> Key: HBASE-16421
> URL: https://issues.apache.org/jira/browse/HBASE-16421
> Project: HBase
>  Issue Type: Umbrella
>Reporter: Anastasia Braginsky
> Attachments: CellChunkMapRevived.pdf, ChunkCell_creation.png, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Follow up for HBASE-14921. This is going to be the umbrella JIRA to include 
> all the parts of integration of the CellChunkMap to the MemStore.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17149) Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor multiple times

[Stephen Yuan Jiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784496#comment-15784496
 ] 

Stephen Yuan Jiang commented on HBASE-17149:


Failed test has nothing to do with the change.

> Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor 
> multiple times
> ---
>
> Key: HBASE-17149
> URL: https://issues.apache.org/jira/browse/HBASE-17149
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: 17149.branch-1.incomplete.txt, 
> HBASE-17149-addendum.v1-master.patch, HBASE-17149.master.001.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.003.patch, 
> HBASE-17149.v1-branch-1.2.patch, HBASE-17149.v1-branch-1.patch, nonce.patch
>
>
> instead of having all the logic in submitProcedure(), split in 
> registerNonce() + submitProcedure().
> In this case we can avoid calling the coprocessor twice and having a clean 
> submit logic knowing that there will only be one submission.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-9465) Push entries to peer clusters serially

[Phil Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784438#comment-15784438
 ] 

Phil Yang commented on HBASE-9465:
--

Sorry for missing this comment, sounds reasonable, I'll check it and file a new 
issue. Thanks.

> Push entries to peer clusters serially
> --
>
> Key: HBASE-9465
> URL: https://issues.apache.org/jira/browse/HBASE-9465
> Project: HBase
>  Issue Type: New Feature
>  Components: regionserver, Replication
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Honghua Feng
>Assignee: Phil Yang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-9465-branch-1-v1.patch, 
> HBASE-9465-branch-1-v1.patch, HBASE-9465-branch-1-v2.patch, 
> HBASE-9465-branch-1-v3.patch, HBASE-9465-branch-1-v4.patch, 
> HBASE-9465-branch-1-v4.patch, HBASE-9465-v1.patch, HBASE-9465-v2.patch, 
> HBASE-9465-v2.patch, HBASE-9465-v3.patch, HBASE-9465-v4.patch, 
> HBASE-9465-v5.patch, HBASE-9465-v6.patch, HBASE-9465-v6.patch, 
> HBASE-9465-v7.patch, HBASE-9465-v7.patch, HBASE-9465.pdf
>
>
> When region-move or RS failure occurs in master cluster, the hlog entries 
> that are not pushed before region-move or RS-failure will be pushed by 
> original RS(for region move) or another RS which takes over the remained hlog 
> of dead RS(for RS failure), and the new entries for the same region(s) will 
> be pushed by the RS which now serves the region(s), but they push the hlog 
> entries of a same region concurrently without coordination.
> This treatment can possibly lead to data inconsistency between master and 
> peer clusters:
> 1. there are put and then delete written to master cluster
> 2. due to region-move / RS-failure, they are pushed by different 
> replication-source threads to peer cluster
> 3. if delete is pushed to peer cluster before put, and flush and 
> major-compact occurs in peer cluster before put is pushed to peer cluster, 
> the delete is collected and the put remains in peer cluster
> In this scenario, the put remains in peer cluster, but in master cluster the 
> put is masked by the delete, hence data inconsistency between master and peer 
> clusters



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17382) Give RegionLocateType a better name

[Duo Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784431#comment-15784431
 ] 

Duo Zhang commented on HBASE-17382:
---

I think Type, Direction, Orientation are all OK. The problem is the word in the 
middle. I prefer 'Locating'.

> Give RegionLocateType a better name
> ---
>
> Key: HBASE-17382
> URL: https://issues.apache.org/jira/browse/HBASE-17382
> Project: HBase
>  Issue Type: Improvement
>Reporter: Duo Zhang
>Assignee: Jan Hentschel
> Attachments: HBASE-17382.master.001.patch
>
>
> Pointed out by [~tedyu] that 'Locate' is a verb and usually we need a noun 
> here. 'Locating' or 'Location'?
> Suggestion are welcomed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-17374) ZKPermissionWatcher crashed when grant after region close

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu reassigned HBASE-17374:
--

Assignee: Liu Junhong

> ZKPermissionWatcher crashed when grant after region close
> -
>
> Key: HBASE-17374
> URL: https://issues.apache.org/jira/browse/HBASE-17374
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.98.15
>Reporter: Liu Junhong
>Assignee: Liu Junhong
>Priority: Critical
> Attachments: 0001-fix-for-HBASE-17374-20161228.patch, 
> 0001-fix-for-HBASE-17374.patch
>
>
> It was occurred many time that  I granted some permission,  but few of some 
> regionservers was not token effect and must be restart . When I look up logs, 
>  I found that :
> 2016-12-08 15:00:26,238 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] handler.CloseRegionHandler 
> (CloseRegionHandler.java:process(128)) - Processing close of 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> {color:red} 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1163)) - Closing 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.: disabling 
> compactions & flushes {color}
> 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1190)) - Updates disabled for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> 2016-12-08 15:00:26,242 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1753)) - Started memstore flush for 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14., current 
> region memstore size 160
> 2016-12-08 15:00:26,284 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.DefaultStoreFlusher (DefaultStoreFlusher.java:flushSnapshot(95)) 
> - Flushed, sequenceid=6, memsize=160, hasBloomFilter=true, into tmp file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,303 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.HRegionFileSystem (HRegionFileSystem.java:commitStoreFile(370)) 
> - Committing store file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
>  as 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,318 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HStore 
> (HStore.java:commitFile(877)) - Added 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3,
>  entries=1, sequenceid=6, filesize=985
> 2016-12-08 15:00:26,319 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1920)) - Finished memstore flush of 
> ~160/160, currentsize=0/0 for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14. in 77ms, 
> sequenceid=6, compaction requested=false
> 2016-12-08 15:00:26,323 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf1
> 2016-12-08 15:00:26,325 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf2
> 2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.token.TokenProvider
> {color:red}  2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.AccessController  {color}
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) 

[jira] [Commented] (HBASE-17238) Wrong in-memory hbase:meta location causing SSH failure

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784425#comment-15784425
 ] 

Hudson commented on HBASE-17238:


SUCCESS: Integrated in Jenkins build HBase-1.3-JDK7 #75 (See 
[https://builds.apache.org/job/HBase-1.3-JDK7/75/])
HBASE-17238 Wrong in-memory hbase:meta location causing SSH failure 
(syuanjiangdev: rev 5d86ab50095963ea04df90a02e02bc3fdab0499c)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMetaWithReplicas.java


> Wrong in-memory hbase:meta location causing SSH failure
> ---
>
> Key: HBASE-17238
> URL: https://issues.apache.org/jira/browse/HBASE-17238
> Project: HBase
>  Issue Type: Bug
>  Components: Region Assignment
>Affects Versions: 1.1.0
>Reporter: Stephen Yuan Jiang
>Assignee: Stephen Yuan Jiang
>Priority: Critical
> Fix For: 1.3.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: HBASE-17238.v1-branch-1.1.patch, 
> HBASE-17238.v1-branch-1.patch, HBASE-17238.v2-branch-1.1.patch
>
>
> In HBase 1.x, if HMaster#assignMeta() assigns a non-DEFAULT_REPLICA_ID 
> hbase:meta region, it would wrongly update the DEFAULT_REPLICA_ID hbase:meta 
> region in-memory.  This caused the in-memory region state has wrong RS 
> information for default replica hbase:meta region.  One of the problem we saw 
> is a wrong type of SSH could be chosen and causing problems.
> {code}
> void assignMeta(MonitoredTask status, Set 
> previouslyFailedMetaRSs, int replicaId)
>   throws InterruptedException, IOException, KeeperException {
> // Work on meta region
> ...
> if (replicaId == HRegionInfo.DEFAULT_REPLICA_ID) {
>   status.setStatus("Assigning hbase:meta region");
> } else {
>   status.setStatus("Assigning hbase:meta region, replicaId " + replicaId);
> }
> // Get current meta state from zk.
> RegionStates regionStates = assignmentManager.getRegionStates();
> RegionState metaState = 
> MetaTableLocator.getMetaRegionState(getZooKeeper(), replicaId);
> HRegionInfo hri = 
> RegionReplicaUtil.getRegionInfoForReplica(HRegionInfo.FIRST_META_REGIONINFO,
> replicaId);
> ServerName currentMetaServer = metaState.getServerName();
> ...
> boolean rit = this.assignmentManager.
>   processRegionInTransitionAndBlockUntilAssigned(hri);
> boolean metaRegionLocation = metaTableLocator.verifyMetaRegionLocation(
>   this.getConnection(), this.getZooKeeper(), timeout, replicaId);
> ...
> } else {
>   // Region already assigned. We didn't assign it. Add to in-memory state.
>   regionStates.updateRegionState(
> HRegionInfo.FIRST_META_REGIONINFO, State.OPEN, currentMetaServer); 
> <<--- Wrong region to update -->>
>   this.assignmentManager.regionOnline(
> HRegionInfo.FIRST_META_REGIONINFO, currentMetaServer); <<--- Wrong 
> region to update -->>
> }
> ...
> {code}
> Here is the problem scenario:
> Step 1: master failovers (or starts could have the same issue) and find 
> default replica of hbase:meta is in rs1.
> {noformat}
> 2016-11-26 00:06:36,590 INFO org.apache.hadoop.hbase.master.ServerManager: 
> AssignmentManager hasn't finished failover cleanup; waiting
> 2016-11-26 00:06:36,591 INFO org.apache.hadoop.hbase.master.HMaster: 
> hbase:meta with replicaId 0 assigned=0, rit=false, 
> location=rs1,60200,1480103147220
> {noformat}
> Step 2: master finds that replica 1 of hbase:meta is unassigned, therefore, 
> HMaster#assignMeta() is called and assign the replica 1 region to rs2, but at 
> the end, it wrongly updates the in-memory state of default replica to rs2
> {noformat}
> 2016-11-26 00:08:21,741 DEBUG org.apache.hadoop.hbase.master.RegionStates: 
> Onlined 1588230740 on rs2,60200,1480102993815 {ENCODED => 1588230740, NAME => 
> 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}
> 2016-11-26 00:08:21,741 INFO org.apache.hadoop.hbase.master.RegionStates: 
> Offlined 1588230740 from rs1,60200,1480103147220
> 2016-11-26 00:08:21,741 INFO org.apache.hadoop.hbase.master.HMaster: 
> hbase:meta with replicaId 1 assigned=0, rit=false, 
> location=rs2,60200,1480102993815
> {noformat}
> Step 3: now rs1 is down, master needs to choose which SSH to call 
> (MetaServerShutdownHandler or normal ServerShutdownHandler) - in this case, 
> MetaServerShutdownHandler should be chosen; however, due to wrong in-memory 
> location, normal ServerShutdownHandler was chosen:
> {noformat}
> 2016-11-26 00:08:33,995 INFO 
> org.apache.hadoop.hbase.zookeeper.RegionServerTracker: RegionServer ephemeral 
> node deleted, processing expiration [rs1,60200,1480103147220]
> 2016-11-26 00:08:33,998 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: based on AM, 

[jira] [Commented] (HBASE-17149) Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor multiple times

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784426#comment-15784426
 ] 

Hudson commented on HBASE-17149:


SUCCESS: Integrated in Jenkins build HBase-1.3-JDK7 #75 (See 
[https://builds.apache.org/job/HBase-1.3-JDK7/75/])
HBASE-17149 Procedure V2 - Fix nonce submission to avoid unnecessary 
(syuanjiangdev: rev e06c3249046f33aab5ff74d1bdbad34e1055326f)
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestModifyColumnFamilyProcedure.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableNamespaceManager.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestMasterProcedureEvents.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestCreateTableProcedure.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestDeleteNamespaceProcedure.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestTruncateTableProcedure.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestProcedureAdmin.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestDeleteColumnFamilyProcedure.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockNoopMasterServices.java
* (edit) 
hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/TestProcedureRecovery.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestAddColumnFamilyProcedure.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestDeleteTableProcedure.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestModifyNamespaceProcedure.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestEnableTableProcedure.java
* (edit) 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestDisableTableProcedure.java
* (add) 
hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/TestProcedureNonce.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestModifyTableProcedure.java
* (edit) 
hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/ProcedureTestingUtility.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureUtil.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestCreateNamespaceProcedure.java


> Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor 
> multiple times
> ---
>
> Key: HBASE-17149
> URL: https://issues.apache.org/jira/browse/HBASE-17149
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: 17149.branch-1.incomplete.txt, 
> HBASE-17149-addendum.v1-master.patch, HBASE-17149.master.001.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.003.patch, 
> HBASE-17149.v1-branch-1.2.patch, HBASE-17149.v1-branch-1.patch, nonce.patch
>
>
> instead of having all the logic in submitProcedure(), split in 
> registerNonce() + submitProcedure().
> In this case we can avoid calling the coprocessor twice and having a clean 
> submit logic knowing that there will only be one submission.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17374) ZKPermissionWatcher crashed when grant after region close

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-17374:
---
Summary: ZKPermissionWatcher crashed when grant after region close  (was: 
ZKPermissionWatcher crashed when grant after close region )

> ZKPermissionWatcher crashed when grant after region close
> -
>
> Key: HBASE-17374
> URL: https://issues.apache.org/jira/browse/HBASE-17374
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.98.15
>Reporter: Liu Junhong
>Priority: Critical
> Attachments: 0001-fix-for-HBASE-17374-20161228.patch, 
> 0001-fix-for-HBASE-17374.patch
>
>
> It was occurred many time that  I granted some permission,  but few of some 
> regionservers was not token effect and must be restart . When I look up logs, 
>  I found that :
> 2016-12-08 15:00:26,238 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] handler.CloseRegionHandler 
> (CloseRegionHandler.java:process(128)) - Processing close of 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> {color:red} 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1163)) - Closing 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.: disabling 
> compactions & flushes {color}
> 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1190)) - Updates disabled for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> 2016-12-08 15:00:26,242 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1753)) - Started memstore flush for 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14., current 
> region memstore size 160
> 2016-12-08 15:00:26,284 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.DefaultStoreFlusher (DefaultStoreFlusher.java:flushSnapshot(95)) 
> - Flushed, sequenceid=6, memsize=160, hasBloomFilter=true, into tmp file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,303 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.HRegionFileSystem (HRegionFileSystem.java:commitStoreFile(370)) 
> - Committing store file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
>  as 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,318 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HStore 
> (HStore.java:commitFile(877)) - Added 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3,
>  entries=1, sequenceid=6, filesize=985
> 2016-12-08 15:00:26,319 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1920)) - Finished memstore flush of 
> ~160/160, currentsize=0/0 for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14. in 77ms, 
> sequenceid=6, compaction requested=false
> 2016-12-08 15:00:26,323 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf1
> 2016-12-08 15:00:26,325 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf2
> 2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.token.TokenProvider
> {color:red}  2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.AccessController  {color}
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:6

[jira] [Commented] (HBASE-17382) Give RegionLocateType a better name

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784423#comment-15784423
 ] 

Ted Yu commented on HBASE-17382:


How about RegionLocatorOrientation ?

> Give RegionLocateType a better name
> ---
>
> Key: HBASE-17382
> URL: https://issues.apache.org/jira/browse/HBASE-17382
> Project: HBase
>  Issue Type: Improvement
>Reporter: Duo Zhang
>Assignee: Jan Hentschel
> Attachments: HBASE-17382.master.001.patch
>
>
> Pointed out by [~tedyu] that 'Locate' is a verb and usually we need a noun 
> here. 'Locating' or 'Location'?
> Suggestion are welcomed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17375) PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784401#comment-15784401
 ] 

Ted Yu commented on HBASE-17375:


{code}
43  public class TestHBASE17375 {
{code}
Can you give the test descriptive name ?

Or, fold the (small) test into existing prefix trie test.

> PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly
> ---
>
> Key: HBASE-17375
> URL: https://issues.apache.org/jira/browse/HBASE-17375
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.1.7, 0.98.23, 0.98.24
>Reporter: Chang chen
>Assignee: Chang chen
> Fix For: 2.0.0
>
> Attachments: HBASE_17375_master_v1.patch, row trie example.PNG
>
>
> Recently, we find our hbase compaction thread never end.  Assume we have 
> following cells:
> {quote}
>  1
>  1
>  1
>  1
>  1
>  1
>  1
>  1
> {quote}
> If we encode above datas into prefix tree block, then it looks like:
> !row trie example.PNG!
> Assume the current row is {color:red}Abc{color} (e.g. the current row node is 
> 4), then the previous row should be *Aa* (e.g. 2). However 
> previousRowInternal return {color:red}A{color}(e.g. 1)
> After investigation, I believe it's the bug of 
> PrefixTreeArrayReversibleScanner#previousRowInternal.
> {code}
>   private boolean previousRowInternal() {
> //...
> while (!beforeFirst) {
>   //
>   // what if currentRowNode is nub?
>   if (currentRowNode.hasOccurrences()) {// escape clause
> currentRowNode.resetFanIndex();
> return true;// found some values
>   }
> }
> {code}
> currentRowNode.hasOccurrences() only test whether it has cell or not. But in 
> the case of  currentRowNode.isNub() is true, previousRowInternal should 
> follow the previous fan instead of return.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17149) Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor multiple times

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784393#comment-15784393
 ] 

Hadoop QA commented on HBASE-17149:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 14 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
36s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s 
{color} | {color:green} branch-1.2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s 
{color} | {color:green} branch-1.2 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
24s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
27s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
19s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s 
{color} | {color:green} branch-1.2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s 
{color} | {color:green} branch-1.2 passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s 
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
12m 50s {color} | {color:green} The patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s 
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 21s 
{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 13s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
29s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 110m 25s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.hbase.regionserver.compactions.TestCompactionWithThroughputController |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:d6626eb |
| JIRA Patch URL | 

[jira] [Commented] (HBASE-17372) Make AsyncTable thread safe

[Duo Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784343#comment-15784343
 ] 

Duo Zhang commented on HBASE-17372:
---

What do you think of the rpc timeout config? Now we have read rpc timeout and 
write rpc timeout, but as [~yangzhe1991] said above, there is no rule for 
operations like append, increment, batch, etc. And is it necessary to have 
different rpc timeout configs as now we can set rpc timeout per call?

> Make AsyncTable thread safe
> ---
>
> Key: HBASE-17372
> URL: https://issues.apache.org/jira/browse/HBASE-17372
> Project: HBase
>  Issue Type: Sub-task
>  Components: asyncclient, Client
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17372-v1.patch, HBASE-17372.patch
>
>
> The most methods are already thread safe. The problem is that we have some 
> methods that used to set timeout, we need to remove these methods and add a 
> parameter for each call to specific timeout settings.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17383) Improve log msg when offheap memstore breaches higher water mark

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784341#comment-15784341
 ] 

Anoop Sam John commented on HBASE-17383:


Noticed this while doing patch for HBASE-17338..  Latest patch there having 
this fix already. Pls have a look..  I did not check the test failures.. Will 
do that soon.

> Improve log msg when offheap memstore breaches higher water mark
> 
>
> Key: HBASE-17383
> URL: https://issues.apache.org/jira/browse/HBASE-17383
> Project: HBase
>  Issue Type: Improvement
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Trivial
> Fix For: 2.0.0
>
>
> Currently we get this log
> {code}
> 2016-12-28 21:11:14,349 INFO  
> [RpcServer.deafult.FPBQ.Fifo.handler=39,queue=9,port=16041] 
> regionserver.MemStoreFlusher: Blocking updates on 
> stobdtserver5,16041,1482938527980: the global offheap memstore size 12.6 G + 
> global memstore heap overhead 4.0 G is >= than blocking 12.6 G size
> {code}
> Here the global offheap memstore size is greater than the blocking size. The 
> memstore heap overhead need not be included in this log unless the higher 
> water mark breach is only due to the heap overhead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17081) Flush the entire CompactingMemStore content to disk

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784339#comment-15784339
 ] 

Ted Yu commented on HBASE-17081:


Looks like there is non-trivial amount of work on top of the latest patch.

How about creating HBASE-17081 branch and commit the latest patch there ?
Follow on work can be reviewed and committed afterwards.

When we are confident for the final solution, we can merge the branch to master.

> Flush the entire CompactingMemStore content to disk
> ---
>
> Key: HBASE-17081
> URL: https://issues.apache.org/jira/browse/HBASE-17081
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Eshcar Hillel
> Fix For: 2.0.0
>
> Attachments: HBASE-15787_8.patch, HBASE-17081-V01.patch, 
> HBASE-17081-V02.patch, HBASE-17081-V03.patch, HBASE-17081-V04.patch, 
> HBASE-17081-V05.patch, HBASE-17081-V06.patch, HBASE-17081-V06.patch, 
> HBASE-17081-V07.patch, HBASE-17081-V10.patch, 
> HBaseMeetupDecember2016-V02.pptx, Pipelinememstore_fortrunk_3.patch
>
>
> Part of CompactingMemStore's memory is held by an active segment, and another 
> part is divided between immutable segments in the compacting pipeline. Upon 
> flush-to-disk request we want to flush all of it to disk, in contrast to 
> flushing only tail of the compacting pipeline.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17382) Give RegionLocateType a better name

[Duo Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784303#comment-15784303
 ] 

Duo Zhang commented on HBASE-17382:
---

Why? RegionLocateType is used to guide the locating direction for 
AsyncRegionLocator. It is a type of the locating operation, not a type of 
region location...

> Give RegionLocateType a better name
> ---
>
> Key: HBASE-17382
> URL: https://issues.apache.org/jira/browse/HBASE-17382
> Project: HBase
>  Issue Type: Improvement
>Reporter: Duo Zhang
>Assignee: Jan Hentschel
> Attachments: HBASE-17382.master.001.patch
>
>
> Pointed out by [~tedyu] that 'Locate' is a verb and usually we need a noun 
> here. 'Locating' or 'Location'?
> Suggestion are welcomed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17336) get/update replication peer config requests should be routed through master

[Guanghao Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784299#comment-15784299
 ] 

Guanghao Zhang commented on HBASE-17336:


Thanks for your suggestion. I will do it in HBASE-17389.

> get/update replication peer config requests should be routed through master
> ---
>
> Key: HBASE-17336
> URL: https://issues.apache.org/jira/browse/HBASE-17336
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17336-v1.patch, HBASE-17336-v2.patch, 
> HBASE-17336-v3.patch, HBASE-17336-v4.patch, HBASE-17336-v5.patch
>
>
> As HBASE-11392 description says, we should move replication operations to be 
> routed through master.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17346) Add coprocessor service support

[Duo Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784298#comment-15784298
 ] 

Duo Zhang commented on HBASE-17346:
---

So we do not want to expose the AggregationClient to user?

And for retrying, if we want the user to do it, then I think we need some 
modifications on the current API. If the call to one region fails, we will fail 
the whole operation which means the user can only redo the whole operation when 
retrying.

I will also test the ServerRpcController usage at client side to see how it 
works.

Thanks.

> Add coprocessor service support
> ---
>
> Key: HBASE-17346
> URL: https://issues.apache.org/jira/browse/HBASE-17346
> Project: HBase
>  Issue Type: Sub-task
>  Components: Client
>Reporter: Duo Zhang
> Fix For: 2.0.0
>
>
> I think we need to redesign the API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17336) get/update replication peer config requests should be routed through master

[Guanghao Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784294#comment-15784294
 ] 

Guanghao Zhang commented on HBASE-17336:


Attach a v5 patch fix the copy-paste error.

bq. move ReplicationPeer and other replication related PB messages to the 
replication.proto from zookeeper.proto.
bq. Maybe after all methods moved to Admin, we can do a refactor patch to 
convert internal usages from RA to Admin.
Open new issue HBASE-17388 and HBASE-17389 for these.

> get/update replication peer config requests should be routed through master
> ---
>
> Key: HBASE-17336
> URL: https://issues.apache.org/jira/browse/HBASE-17336
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17336-v1.patch, HBASE-17336-v2.patch, 
> HBASE-17336-v3.patch, HBASE-17336-v4.patch, HBASE-17336-v5.patch
>
>
> As HBASE-11392 description says, we should move replication operations to be 
> routed through master.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-17389) Convert all internal usages from ReplicationAdmin to Admin

[Guanghao Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang reassigned HBASE-17389:
--

Assignee: Guanghao Zhang

> Convert all internal usages from ReplicationAdmin to Admin
> --
>
> Key: HBASE-17389
> URL: https://issues.apache.org/jira/browse/HBASE-17389
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17336) get/update replication peer config requests should be routed through master

[Guanghao Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-17336:
---
Attachment: HBASE-17336-v5.patch

> get/update replication peer config requests should be routed through master
> ---
>
> Key: HBASE-17336
> URL: https://issues.apache.org/jira/browse/HBASE-17336
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17336-v1.patch, HBASE-17336-v2.patch, 
> HBASE-17336-v3.patch, HBASE-17336-v4.patch, HBASE-17336-v5.patch
>
>
> As HBASE-11392 description says, we should move replication operations to be 
> routed through master.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17238) Wrong in-memory hbase:meta location causing SSH failure

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784284#comment-15784284
 ] 

Hudson commented on HBASE-17238:


SUCCESS: Integrated in Jenkins build HBase-1.2-JDK7 #83 (See 
[https://builds.apache.org/job/HBase-1.2-JDK7/83/])
HBASE-17238 Wrong in-memory hbase:meta location causing SSH failure 
(syuanjiangdev: rev 64f41715d51f2e37bcd67f6b479da83d45c0c15e)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMetaWithReplicas.java


> Wrong in-memory hbase:meta location causing SSH failure
> ---
>
> Key: HBASE-17238
> URL: https://issues.apache.org/jira/browse/HBASE-17238
> Project: HBase
>  Issue Type: Bug
>  Components: Region Assignment
>Affects Versions: 1.1.0
>Reporter: Stephen Yuan Jiang
>Assignee: Stephen Yuan Jiang
>Priority: Critical
> Fix For: 1.3.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: HBASE-17238.v1-branch-1.1.patch, 
> HBASE-17238.v1-branch-1.patch, HBASE-17238.v2-branch-1.1.patch
>
>
> In HBase 1.x, if HMaster#assignMeta() assigns a non-DEFAULT_REPLICA_ID 
> hbase:meta region, it would wrongly update the DEFAULT_REPLICA_ID hbase:meta 
> region in-memory.  This caused the in-memory region state has wrong RS 
> information for default replica hbase:meta region.  One of the problem we saw 
> is a wrong type of SSH could be chosen and causing problems.
> {code}
> void assignMeta(MonitoredTask status, Set 
> previouslyFailedMetaRSs, int replicaId)
>   throws InterruptedException, IOException, KeeperException {
> // Work on meta region
> ...
> if (replicaId == HRegionInfo.DEFAULT_REPLICA_ID) {
>   status.setStatus("Assigning hbase:meta region");
> } else {
>   status.setStatus("Assigning hbase:meta region, replicaId " + replicaId);
> }
> // Get current meta state from zk.
> RegionStates regionStates = assignmentManager.getRegionStates();
> RegionState metaState = 
> MetaTableLocator.getMetaRegionState(getZooKeeper(), replicaId);
> HRegionInfo hri = 
> RegionReplicaUtil.getRegionInfoForReplica(HRegionInfo.FIRST_META_REGIONINFO,
> replicaId);
> ServerName currentMetaServer = metaState.getServerName();
> ...
> boolean rit = this.assignmentManager.
>   processRegionInTransitionAndBlockUntilAssigned(hri);
> boolean metaRegionLocation = metaTableLocator.verifyMetaRegionLocation(
>   this.getConnection(), this.getZooKeeper(), timeout, replicaId);
> ...
> } else {
>   // Region already assigned. We didn't assign it. Add to in-memory state.
>   regionStates.updateRegionState(
> HRegionInfo.FIRST_META_REGIONINFO, State.OPEN, currentMetaServer); 
> <<--- Wrong region to update -->>
>   this.assignmentManager.regionOnline(
> HRegionInfo.FIRST_META_REGIONINFO, currentMetaServer); <<--- Wrong 
> region to update -->>
> }
> ...
> {code}
> Here is the problem scenario:
> Step 1: master failovers (or starts could have the same issue) and find 
> default replica of hbase:meta is in rs1.
> {noformat}
> 2016-11-26 00:06:36,590 INFO org.apache.hadoop.hbase.master.ServerManager: 
> AssignmentManager hasn't finished failover cleanup; waiting
> 2016-11-26 00:06:36,591 INFO org.apache.hadoop.hbase.master.HMaster: 
> hbase:meta with replicaId 0 assigned=0, rit=false, 
> location=rs1,60200,1480103147220
> {noformat}
> Step 2: master finds that replica 1 of hbase:meta is unassigned, therefore, 
> HMaster#assignMeta() is called and assign the replica 1 region to rs2, but at 
> the end, it wrongly updates the in-memory state of default replica to rs2
> {noformat}
> 2016-11-26 00:08:21,741 DEBUG org.apache.hadoop.hbase.master.RegionStates: 
> Onlined 1588230740 on rs2,60200,1480102993815 {ENCODED => 1588230740, NAME => 
> 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}
> 2016-11-26 00:08:21,741 INFO org.apache.hadoop.hbase.master.RegionStates: 
> Offlined 1588230740 from rs1,60200,1480103147220
> 2016-11-26 00:08:21,741 INFO org.apache.hadoop.hbase.master.HMaster: 
> hbase:meta with replicaId 1 assigned=0, rit=false, 
> location=rs2,60200,1480102993815
> {noformat}
> Step 3: now rs1 is down, master needs to choose which SSH to call 
> (MetaServerShutdownHandler or normal ServerShutdownHandler) - in this case, 
> MetaServerShutdownHandler should be chosen; however, due to wrong in-memory 
> location, normal ServerShutdownHandler was chosen:
> {noformat}
> 2016-11-26 00:08:33,995 INFO 
> org.apache.hadoop.hbase.zookeeper.RegionServerTracker: RegionServer ephemeral 
> node deleted, processing expiration [rs1,60200,1480103147220]
> 2016-11-26 00:08:33,998 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: based on AM, 

[jira] [Created] (HBASE-17389) Convert all internal usages from ReplicationAdmin to Admin

[Guanghao Zhang (JIRA)]

Guanghao Zhang created HBASE-17389:
--

 Summary: Convert all internal usages from ReplicationAdmin to Admin
 Key: HBASE-17389
 URL: https://issues.apache.org/jira/browse/HBASE-17389
 Project: HBase
  Issue Type: Sub-task
Affects Versions: 2.0.0
Reporter: Guanghao Zhang
 Fix For: 2.0.0






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17149) Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor multiple times

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784274#comment-15784274
 ] 

Hudson commented on HBASE-17149:


FAILURE: Integrated in Jenkins build HBase-1.4 #579 (See 
[https://builds.apache.org/job/HBase-1.4/579/])
HBASE-17149 Procedure V2 - Fix nonce submission to avoid unnecessary 
(syuanjiangdev: rev ce33cf2d3d3044bc4a24180325830fc767745d6f)
* (edit) 
hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/ProcedureTestingUtility.java
* (edit) 
hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/TestProcedureRecovery.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestDeleteColumnFamilyProcedure.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestDisableTableProcedure.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestCreateTableProcedure.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureUtil.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestEnableTableProcedure.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestAddColumnFamilyProcedure.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestDeleteTableProcedure.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestProcedureAdmin.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestDeleteNamespaceProcedure.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestCreateNamespaceProcedure.java
* (add) 
hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/TestProcedureNonce.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockNoopMasterServices.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestMasterProcedureEvents.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableNamespaceManager.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestTruncateTableProcedure.java
* (edit) 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestModifyNamespaceProcedure.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestModifyColumnFamilyProcedure.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestModifyTableProcedure.java


> Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor 
> multiple times
> ---
>
> Key: HBASE-17149
> URL: https://issues.apache.org/jira/browse/HBASE-17149
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: 17149.branch-1.incomplete.txt, 
> HBASE-17149-addendum.v1-master.patch, HBASE-17149.master.001.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.003.patch, 
> HBASE-17149.v1-branch-1.2.patch, HBASE-17149.v1-branch-1.patch, nonce.patch
>
>
> instead of having all the logic in submitProcedure(), split in 
> registerNonce() + submitProcedure().
> In this case we can avoid calling the coprocessor twice and having a clean 
> submit logic knowing that there will only be one submission.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17238) Wrong in-memory hbase:meta location causing SSH failure

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784273#comment-15784273
 ] 

Hudson commented on HBASE-17238:


FAILURE: Integrated in Jenkins build HBase-1.4 #579 (See 
[https://builds.apache.org/job/HBase-1.4/579/])
HBASE-17238 Wrong in-memory hbase:meta location causing SSH failure 
(syuanjiangdev: rev d4b2627916efa90c284dfc5c828d27ac4294ecc5)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMetaWithReplicas.java


> Wrong in-memory hbase:meta location causing SSH failure
> ---
>
> Key: HBASE-17238
> URL: https://issues.apache.org/jira/browse/HBASE-17238
> Project: HBase
>  Issue Type: Bug
>  Components: Region Assignment
>Affects Versions: 1.1.0
>Reporter: Stephen Yuan Jiang
>Assignee: Stephen Yuan Jiang
>Priority: Critical
> Fix For: 1.3.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: HBASE-17238.v1-branch-1.1.patch, 
> HBASE-17238.v1-branch-1.patch, HBASE-17238.v2-branch-1.1.patch
>
>
> In HBase 1.x, if HMaster#assignMeta() assigns a non-DEFAULT_REPLICA_ID 
> hbase:meta region, it would wrongly update the DEFAULT_REPLICA_ID hbase:meta 
> region in-memory.  This caused the in-memory region state has wrong RS 
> information for default replica hbase:meta region.  One of the problem we saw 
> is a wrong type of SSH could be chosen and causing problems.
> {code}
> void assignMeta(MonitoredTask status, Set 
> previouslyFailedMetaRSs, int replicaId)
>   throws InterruptedException, IOException, KeeperException {
> // Work on meta region
> ...
> if (replicaId == HRegionInfo.DEFAULT_REPLICA_ID) {
>   status.setStatus("Assigning hbase:meta region");
> } else {
>   status.setStatus("Assigning hbase:meta region, replicaId " + replicaId);
> }
> // Get current meta state from zk.
> RegionStates regionStates = assignmentManager.getRegionStates();
> RegionState metaState = 
> MetaTableLocator.getMetaRegionState(getZooKeeper(), replicaId);
> HRegionInfo hri = 
> RegionReplicaUtil.getRegionInfoForReplica(HRegionInfo.FIRST_META_REGIONINFO,
> replicaId);
> ServerName currentMetaServer = metaState.getServerName();
> ...
> boolean rit = this.assignmentManager.
>   processRegionInTransitionAndBlockUntilAssigned(hri);
> boolean metaRegionLocation = metaTableLocator.verifyMetaRegionLocation(
>   this.getConnection(), this.getZooKeeper(), timeout, replicaId);
> ...
> } else {
>   // Region already assigned. We didn't assign it. Add to in-memory state.
>   regionStates.updateRegionState(
> HRegionInfo.FIRST_META_REGIONINFO, State.OPEN, currentMetaServer); 
> <<--- Wrong region to update -->>
>   this.assignmentManager.regionOnline(
> HRegionInfo.FIRST_META_REGIONINFO, currentMetaServer); <<--- Wrong 
> region to update -->>
> }
> ...
> {code}
> Here is the problem scenario:
> Step 1: master failovers (or starts could have the same issue) and find 
> default replica of hbase:meta is in rs1.
> {noformat}
> 2016-11-26 00:06:36,590 INFO org.apache.hadoop.hbase.master.ServerManager: 
> AssignmentManager hasn't finished failover cleanup; waiting
> 2016-11-26 00:06:36,591 INFO org.apache.hadoop.hbase.master.HMaster: 
> hbase:meta with replicaId 0 assigned=0, rit=false, 
> location=rs1,60200,1480103147220
> {noformat}
> Step 2: master finds that replica 1 of hbase:meta is unassigned, therefore, 
> HMaster#assignMeta() is called and assign the replica 1 region to rs2, but at 
> the end, it wrongly updates the in-memory state of default replica to rs2
> {noformat}
> 2016-11-26 00:08:21,741 DEBUG org.apache.hadoop.hbase.master.RegionStates: 
> Onlined 1588230740 on rs2,60200,1480102993815 {ENCODED => 1588230740, NAME => 
> 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}
> 2016-11-26 00:08:21,741 INFO org.apache.hadoop.hbase.master.RegionStates: 
> Offlined 1588230740 from rs1,60200,1480103147220
> 2016-11-26 00:08:21,741 INFO org.apache.hadoop.hbase.master.HMaster: 
> hbase:meta with replicaId 1 assigned=0, rit=false, 
> location=rs2,60200,1480102993815
> {noformat}
> Step 3: now rs1 is down, master needs to choose which SSH to call 
> (MetaServerShutdownHandler or normal ServerShutdownHandler) - in this case, 
> MetaServerShutdownHandler should be chosen; however, due to wrong in-memory 
> location, normal ServerShutdownHandler was chosen:
> {noformat}
> 2016-11-26 00:08:33,995 INFO 
> org.apache.hadoop.hbase.zookeeper.RegionServerTracker: RegionServer ephemeral 
> node deleted, processing expiration [rs1,60200,1480103147220]
> 2016-11-26 00:08:33,998 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: based on AM, current 
> 

[jira] [Created] (HBASE-17388) Move ReplicationPeer and other replication related PB messages to the replication.proto

[Guanghao Zhang (JIRA)]

Guanghao Zhang created HBASE-17388:
--

 Summary: Move ReplicationPeer and other replication related PB 
messages to the replication.proto
 Key: HBASE-17388
 URL: https://issues.apache.org/jira/browse/HBASE-17388
 Project: HBase
  Issue Type: Sub-task
  Components: Replication
Affects Versions: 2.0.0
Reporter: Guanghao Zhang
Assignee: Guanghao Zhang
 Fix For: 2.0.0






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17320) Add inclusive/exclusive support for startRow and endRow of scan

[Duo Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang updated HBASE-17320:
--
Release Note: 
Now you can specific the inclusive of startRow and stopRow for a scan using the 
new methods withStartRow(byte[] startRow, boolean inclusive) and 
withStopRow(byte[] stopRow, boolean inclusive). The old setStartRow and 
setStopRow methods, and the constructors are marked as deprecated because of an 
strange behavior that we will include the stopRow implicitly if startRow equals 
to stopRow. This is used to support get scan in the old time. Use withStartRow 
and withStopRow instead.

For developers, the ConnectionUtils.createClosestRowBefore is also marked as 
deprecated as the row returned by this method is only very very close to the 
current row, not closest. Avoid using this method in the future.

> Add inclusive/exclusive support for startRow and endRow of scan
> ---
>
> Key: HBASE-17320
> URL: https://issues.apache.org/jira/browse/HBASE-17320
> Project: HBase
>  Issue Type: Sub-task
>  Components: Client, scan
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17320-v1.patch, HBASE-17320-v2.patch, 
> HBASE-17320-v3.patch, HBASE-17320-v4.patch, HBASE-17320-v5.patch, 
> HBASE-17320-v6.patch, HBASE-17320.patch
>
>
> This is especially useful when doing reverse scan. HBASE-17168 maybe a more 
> powerful solution but we need to be careful about the atomicity, and I do not 
> think we will provide the feature to end user. But I think it is OK to 
> provide inclusive/exclusive option to end user.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17339) Scan-Memory-First Optimization for Get Operation

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784266#comment-15784266
 ] 

Enis Soztutar commented on HBASE-17339:
---

The main problem is how to determine " ONLY if result is not complete". When 
you get the result of a row from memory, it can happen that some other store 
file contains a higher version, and you will miss it unless we have the 
monotonically increasing timestamps guarantee. 

However, we already have min - max timestamps per store file tracked, and we 
have logic to eliminate scanners based on min/max timestamps. We can do this 
algorithm for correctness: 
{code}
 1. open all relevant  *memory* scanners
 2. get results
 3. If get returns a result
   check the timestamp against all remaining scanners 
(KeyValueScanner.shouldUseScanner()).  if all (hfile) scanners have less 
timestamps, return results. 
 else 
  open all scanners 
  return results 
{code}

This will ensure correctness without having to rely on a promise from the user. 

> Scan-Memory-First Optimization for Get Operation
> 
>
> Key: HBASE-17339
> URL: https://issues.apache.org/jira/browse/HBASE-17339
> Project: HBase
>  Issue Type: Improvement
>Reporter: Eshcar Hillel
> Attachments: HBASE-17339-V01.patch
>
>
> The current implementation of a get operation (to retrieve values for a 
> specific key) scans through all relevant stores of the region; for each store 
> both memory components (memstores segments) and disk components (hfiles) are 
> scanned in parallel.
> We suggest to apply an optimization that speculatively scans memory-only 
> components first and only if the result is incomplete scans both memory and 
> disk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17320) Add inclusive/exclusive support for startRow and endRow of scan

[Duo Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang updated HBASE-17320:
--
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Pushed to master.

Thanks all for reviewing.

Will add a release note soon.

> Add inclusive/exclusive support for startRow and endRow of scan
> ---
>
> Key: HBASE-17320
> URL: https://issues.apache.org/jira/browse/HBASE-17320
> Project: HBase
>  Issue Type: Sub-task
>  Components: Client, scan
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17320-v1.patch, HBASE-17320-v2.patch, 
> HBASE-17320-v3.patch, HBASE-17320-v4.patch, HBASE-17320-v5.patch, 
> HBASE-17320-v6.patch, HBASE-17320.patch
>
>
> This is especially useful when doing reverse scan. HBASE-17168 maybe a more 
> powerful solution but we need to be careful about the atomicity, and I do not 
> think we will provide the feature to end user. But I think it is OK to 
> provide inclusive/exclusive option to end user.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17238) Wrong in-memory hbase:meta location causing SSH failure

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784233#comment-15784233
 ] 

Hudson commented on HBASE-17238:


SUCCESS: Integrated in Jenkins build HBase-1.3-JDK8 #85 (See 
[https://builds.apache.org/job/HBase-1.3-JDK8/85/])
HBASE-17238 Wrong in-memory hbase:meta location causing SSH failure 
(syuanjiangdev: rev 5d86ab50095963ea04df90a02e02bc3fdab0499c)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMetaWithReplicas.java


> Wrong in-memory hbase:meta location causing SSH failure
> ---
>
> Key: HBASE-17238
> URL: https://issues.apache.org/jira/browse/HBASE-17238
> Project: HBase
>  Issue Type: Bug
>  Components: Region Assignment
>Affects Versions: 1.1.0
>Reporter: Stephen Yuan Jiang
>Assignee: Stephen Yuan Jiang
>Priority: Critical
> Fix For: 1.3.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: HBASE-17238.v1-branch-1.1.patch, 
> HBASE-17238.v1-branch-1.patch, HBASE-17238.v2-branch-1.1.patch
>
>
> In HBase 1.x, if HMaster#assignMeta() assigns a non-DEFAULT_REPLICA_ID 
> hbase:meta region, it would wrongly update the DEFAULT_REPLICA_ID hbase:meta 
> region in-memory.  This caused the in-memory region state has wrong RS 
> information for default replica hbase:meta region.  One of the problem we saw 
> is a wrong type of SSH could be chosen and causing problems.
> {code}
> void assignMeta(MonitoredTask status, Set 
> previouslyFailedMetaRSs, int replicaId)
>   throws InterruptedException, IOException, KeeperException {
> // Work on meta region
> ...
> if (replicaId == HRegionInfo.DEFAULT_REPLICA_ID) {
>   status.setStatus("Assigning hbase:meta region");
> } else {
>   status.setStatus("Assigning hbase:meta region, replicaId " + replicaId);
> }
> // Get current meta state from zk.
> RegionStates regionStates = assignmentManager.getRegionStates();
> RegionState metaState = 
> MetaTableLocator.getMetaRegionState(getZooKeeper(), replicaId);
> HRegionInfo hri = 
> RegionReplicaUtil.getRegionInfoForReplica(HRegionInfo.FIRST_META_REGIONINFO,
> replicaId);
> ServerName currentMetaServer = metaState.getServerName();
> ...
> boolean rit = this.assignmentManager.
>   processRegionInTransitionAndBlockUntilAssigned(hri);
> boolean metaRegionLocation = metaTableLocator.verifyMetaRegionLocation(
>   this.getConnection(), this.getZooKeeper(), timeout, replicaId);
> ...
> } else {
>   // Region already assigned. We didn't assign it. Add to in-memory state.
>   regionStates.updateRegionState(
> HRegionInfo.FIRST_META_REGIONINFO, State.OPEN, currentMetaServer); 
> <<--- Wrong region to update -->>
>   this.assignmentManager.regionOnline(
> HRegionInfo.FIRST_META_REGIONINFO, currentMetaServer); <<--- Wrong 
> region to update -->>
> }
> ...
> {code}
> Here is the problem scenario:
> Step 1: master failovers (or starts could have the same issue) and find 
> default replica of hbase:meta is in rs1.
> {noformat}
> 2016-11-26 00:06:36,590 INFO org.apache.hadoop.hbase.master.ServerManager: 
> AssignmentManager hasn't finished failover cleanup; waiting
> 2016-11-26 00:06:36,591 INFO org.apache.hadoop.hbase.master.HMaster: 
> hbase:meta with replicaId 0 assigned=0, rit=false, 
> location=rs1,60200,1480103147220
> {noformat}
> Step 2: master finds that replica 1 of hbase:meta is unassigned, therefore, 
> HMaster#assignMeta() is called and assign the replica 1 region to rs2, but at 
> the end, it wrongly updates the in-memory state of default replica to rs2
> {noformat}
> 2016-11-26 00:08:21,741 DEBUG org.apache.hadoop.hbase.master.RegionStates: 
> Onlined 1588230740 on rs2,60200,1480102993815 {ENCODED => 1588230740, NAME => 
> 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}
> 2016-11-26 00:08:21,741 INFO org.apache.hadoop.hbase.master.RegionStates: 
> Offlined 1588230740 from rs1,60200,1480103147220
> 2016-11-26 00:08:21,741 INFO org.apache.hadoop.hbase.master.HMaster: 
> hbase:meta with replicaId 1 assigned=0, rit=false, 
> location=rs2,60200,1480102993815
> {noformat}
> Step 3: now rs1 is down, master needs to choose which SSH to call 
> (MetaServerShutdownHandler or normal ServerShutdownHandler) - in this case, 
> MetaServerShutdownHandler should be chosen; however, due to wrong in-memory 
> location, normal ServerShutdownHandler was chosen:
> {noformat}
> 2016-11-26 00:08:33,995 INFO 
> org.apache.hadoop.hbase.zookeeper.RegionServerTracker: RegionServer ephemeral 
> node deleted, processing expiration [rs1,60200,1480103147220]
> 2016-11-26 00:08:33,998 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: based on AM, 

[jira] [Commented] (HBASE-17238) Wrong in-memory hbase:meta location causing SSH failure

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784228#comment-15784228
 ] 

Ted Yu commented on HBASE-17238:


>From 
>https://builds.apache.org/job/HBase-1.2-JDK8/org.apache.hbase$hbase-server/77/testReport/junit/org.apache.hadoop.hbase.client/TestMetaWithReplicas/testMetaTableReplicaAssignment/
> , it seems the new test is flaky.

> Wrong in-memory hbase:meta location causing SSH failure
> ---
>
> Key: HBASE-17238
> URL: https://issues.apache.org/jira/browse/HBASE-17238
> Project: HBase
>  Issue Type: Bug
>  Components: Region Assignment
>Affects Versions: 1.1.0
>Reporter: Stephen Yuan Jiang
>Assignee: Stephen Yuan Jiang
>Priority: Critical
> Fix For: 1.3.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: HBASE-17238.v1-branch-1.1.patch, 
> HBASE-17238.v1-branch-1.patch, HBASE-17238.v2-branch-1.1.patch
>
>
> In HBase 1.x, if HMaster#assignMeta() assigns a non-DEFAULT_REPLICA_ID 
> hbase:meta region, it would wrongly update the DEFAULT_REPLICA_ID hbase:meta 
> region in-memory.  This caused the in-memory region state has wrong RS 
> information for default replica hbase:meta region.  One of the problem we saw 
> is a wrong type of SSH could be chosen and causing problems.
> {code}
> void assignMeta(MonitoredTask status, Set 
> previouslyFailedMetaRSs, int replicaId)
>   throws InterruptedException, IOException, KeeperException {
> // Work on meta region
> ...
> if (replicaId == HRegionInfo.DEFAULT_REPLICA_ID) {
>   status.setStatus("Assigning hbase:meta region");
> } else {
>   status.setStatus("Assigning hbase:meta region, replicaId " + replicaId);
> }
> // Get current meta state from zk.
> RegionStates regionStates = assignmentManager.getRegionStates();
> RegionState metaState = 
> MetaTableLocator.getMetaRegionState(getZooKeeper(), replicaId);
> HRegionInfo hri = 
> RegionReplicaUtil.getRegionInfoForReplica(HRegionInfo.FIRST_META_REGIONINFO,
> replicaId);
> ServerName currentMetaServer = metaState.getServerName();
> ...
> boolean rit = this.assignmentManager.
>   processRegionInTransitionAndBlockUntilAssigned(hri);
> boolean metaRegionLocation = metaTableLocator.verifyMetaRegionLocation(
>   this.getConnection(), this.getZooKeeper(), timeout, replicaId);
> ...
> } else {
>   // Region already assigned. We didn't assign it. Add to in-memory state.
>   regionStates.updateRegionState(
> HRegionInfo.FIRST_META_REGIONINFO, State.OPEN, currentMetaServer); 
> <<--- Wrong region to update -->>
>   this.assignmentManager.regionOnline(
> HRegionInfo.FIRST_META_REGIONINFO, currentMetaServer); <<--- Wrong 
> region to update -->>
> }
> ...
> {code}
> Here is the problem scenario:
> Step 1: master failovers (or starts could have the same issue) and find 
> default replica of hbase:meta is in rs1.
> {noformat}
> 2016-11-26 00:06:36,590 INFO org.apache.hadoop.hbase.master.ServerManager: 
> AssignmentManager hasn't finished failover cleanup; waiting
> 2016-11-26 00:06:36,591 INFO org.apache.hadoop.hbase.master.HMaster: 
> hbase:meta with replicaId 0 assigned=0, rit=false, 
> location=rs1,60200,1480103147220
> {noformat}
> Step 2: master finds that replica 1 of hbase:meta is unassigned, therefore, 
> HMaster#assignMeta() is called and assign the replica 1 region to rs2, but at 
> the end, it wrongly updates the in-memory state of default replica to rs2
> {noformat}
> 2016-11-26 00:08:21,741 DEBUG org.apache.hadoop.hbase.master.RegionStates: 
> Onlined 1588230740 on rs2,60200,1480102993815 {ENCODED => 1588230740, NAME => 
> 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}
> 2016-11-26 00:08:21,741 INFO org.apache.hadoop.hbase.master.RegionStates: 
> Offlined 1588230740 from rs1,60200,1480103147220
> 2016-11-26 00:08:21,741 INFO org.apache.hadoop.hbase.master.HMaster: 
> hbase:meta with replicaId 1 assigned=0, rit=false, 
> location=rs2,60200,1480102993815
> {noformat}
> Step 3: now rs1 is down, master needs to choose which SSH to call 
> (MetaServerShutdownHandler or normal ServerShutdownHandler) - in this case, 
> MetaServerShutdownHandler should be chosen; however, due to wrong in-memory 
> location, normal ServerShutdownHandler was chosen:
> {noformat}
> 2016-11-26 00:08:33,995 INFO 
> org.apache.hadoop.hbase.zookeeper.RegionServerTracker: RegionServer ephemeral 
> node deleted, processing expiration [rs1,60200,1480103147220]
> 2016-11-26 00:08:33,998 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: based on AM, current 
> region=hbase:meta,,1.1588230740 is on server=rs2,60200,1480102993815 server 
> being checked: rs1,60200,1480103147220
> 2016-11-26 00:08:34,001 DEBUG 

[jira] [Commented] (HBASE-17320) Add inclusive/exclusive support for startRow and endRow of scan

[Duo Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784212#comment-15784212
 ] 

Duo Zhang commented on HBASE-17320:
---

Will commit shortly.

> Add inclusive/exclusive support for startRow and endRow of scan
> ---
>
> Key: HBASE-17320
> URL: https://issues.apache.org/jira/browse/HBASE-17320
> Project: HBase
>  Issue Type: Sub-task
>  Components: Client, scan
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17320-v1.patch, HBASE-17320-v2.patch, 
> HBASE-17320-v3.patch, HBASE-17320-v4.patch, HBASE-17320-v5.patch, 
> HBASE-17320-v6.patch, HBASE-17320.patch
>
>
> This is especially useful when doing reverse scan. HBASE-17168 maybe a more 
> powerful solution but we need to be careful about the atomicity, and I do not 
> think we will provide the feature to end user. But I think it is OK to 
> provide inclusive/exclusive option to end user.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17149) Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor multiple times

[Stephen Yuan Jiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Yuan Jiang updated HBASE-17149:
---
Attachment: HBASE-17149.v1-branch-1.2.patch

> Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor 
> multiple times
> ---
>
> Key: HBASE-17149
> URL: https://issues.apache.org/jira/browse/HBASE-17149
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: 17149.branch-1.incomplete.txt, 
> HBASE-17149-addendum.v1-master.patch, HBASE-17149.master.001.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.003.patch, 
> HBASE-17149.v1-branch-1.2.patch, HBASE-17149.v1-branch-1.patch, nonce.patch
>
>
> instead of having all the logic in submitProcedure(), split in 
> registerNonce() + submitProcedure().
> In this case we can avoid calling the coprocessor twice and having a clean 
> submit logic knowing that there will only be one submission.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17238) Wrong in-memory hbase:meta location causing SSH failure

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784178#comment-15784178
 ] 

Hudson commented on HBASE-17238:


FAILURE: Integrated in Jenkins build HBase-1.2-JDK8 #77 (See 
[https://builds.apache.org/job/HBase-1.2-JDK8/77/])
HBASE-17238 Wrong in-memory hbase:meta location causing SSH failure 
(syuanjiangdev: rev 64f41715d51f2e37bcd67f6b479da83d45c0c15e)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMetaWithReplicas.java


> Wrong in-memory hbase:meta location causing SSH failure
> ---
>
> Key: HBASE-17238
> URL: https://issues.apache.org/jira/browse/HBASE-17238
> Project: HBase
>  Issue Type: Bug
>  Components: Region Assignment
>Affects Versions: 1.1.0
>Reporter: Stephen Yuan Jiang
>Assignee: Stephen Yuan Jiang
>Priority: Critical
> Fix For: 1.3.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: HBASE-17238.v1-branch-1.1.patch, 
> HBASE-17238.v1-branch-1.patch, HBASE-17238.v2-branch-1.1.patch
>
>
> In HBase 1.x, if HMaster#assignMeta() assigns a non-DEFAULT_REPLICA_ID 
> hbase:meta region, it would wrongly update the DEFAULT_REPLICA_ID hbase:meta 
> region in-memory.  This caused the in-memory region state has wrong RS 
> information for default replica hbase:meta region.  One of the problem we saw 
> is a wrong type of SSH could be chosen and causing problems.
> {code}
> void assignMeta(MonitoredTask status, Set 
> previouslyFailedMetaRSs, int replicaId)
>   throws InterruptedException, IOException, KeeperException {
> // Work on meta region
> ...
> if (replicaId == HRegionInfo.DEFAULT_REPLICA_ID) {
>   status.setStatus("Assigning hbase:meta region");
> } else {
>   status.setStatus("Assigning hbase:meta region, replicaId " + replicaId);
> }
> // Get current meta state from zk.
> RegionStates regionStates = assignmentManager.getRegionStates();
> RegionState metaState = 
> MetaTableLocator.getMetaRegionState(getZooKeeper(), replicaId);
> HRegionInfo hri = 
> RegionReplicaUtil.getRegionInfoForReplica(HRegionInfo.FIRST_META_REGIONINFO,
> replicaId);
> ServerName currentMetaServer = metaState.getServerName();
> ...
> boolean rit = this.assignmentManager.
>   processRegionInTransitionAndBlockUntilAssigned(hri);
> boolean metaRegionLocation = metaTableLocator.verifyMetaRegionLocation(
>   this.getConnection(), this.getZooKeeper(), timeout, replicaId);
> ...
> } else {
>   // Region already assigned. We didn't assign it. Add to in-memory state.
>   regionStates.updateRegionState(
> HRegionInfo.FIRST_META_REGIONINFO, State.OPEN, currentMetaServer); 
> <<--- Wrong region to update -->>
>   this.assignmentManager.regionOnline(
> HRegionInfo.FIRST_META_REGIONINFO, currentMetaServer); <<--- Wrong 
> region to update -->>
> }
> ...
> {code}
> Here is the problem scenario:
> Step 1: master failovers (or starts could have the same issue) and find 
> default replica of hbase:meta is in rs1.
> {noformat}
> 2016-11-26 00:06:36,590 INFO org.apache.hadoop.hbase.master.ServerManager: 
> AssignmentManager hasn't finished failover cleanup; waiting
> 2016-11-26 00:06:36,591 INFO org.apache.hadoop.hbase.master.HMaster: 
> hbase:meta with replicaId 0 assigned=0, rit=false, 
> location=rs1,60200,1480103147220
> {noformat}
> Step 2: master finds that replica 1 of hbase:meta is unassigned, therefore, 
> HMaster#assignMeta() is called and assign the replica 1 region to rs2, but at 
> the end, it wrongly updates the in-memory state of default replica to rs2
> {noformat}
> 2016-11-26 00:08:21,741 DEBUG org.apache.hadoop.hbase.master.RegionStates: 
> Onlined 1588230740 on rs2,60200,1480102993815 {ENCODED => 1588230740, NAME => 
> 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}
> 2016-11-26 00:08:21,741 INFO org.apache.hadoop.hbase.master.RegionStates: 
> Offlined 1588230740 from rs1,60200,1480103147220
> 2016-11-26 00:08:21,741 INFO org.apache.hadoop.hbase.master.HMaster: 
> hbase:meta with replicaId 1 assigned=0, rit=false, 
> location=rs2,60200,1480102993815
> {noformat}
> Step 3: now rs1 is down, master needs to choose which SSH to call 
> (MetaServerShutdownHandler or normal ServerShutdownHandler) - in this case, 
> MetaServerShutdownHandler should be chosen; however, due to wrong in-memory 
> location, normal ServerShutdownHandler was chosen:
> {noformat}
> 2016-11-26 00:08:33,995 INFO 
> org.apache.hadoop.hbase.zookeeper.RegionServerTracker: RegionServer ephemeral 
> node deleted, processing expiration [rs1,60200,1480103147220]
> 2016-11-26 00:08:33,998 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: based on AM, 

[jira] [Updated] (HBASE-17149) Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor multiple times

[Stephen Yuan Jiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Yuan Jiang updated HBASE-17149:
---
Attachment: (was: HBASE-17149.v1-branch-1.2.patch)

> Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor 
> multiple times
> ---
>
> Key: HBASE-17149
> URL: https://issues.apache.org/jira/browse/HBASE-17149
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: 17149.branch-1.incomplete.txt, 
> HBASE-17149-addendum.v1-master.patch, HBASE-17149.master.001.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.003.patch, 
> HBASE-17149.v1-branch-1.patch, nonce.patch
>
>
> instead of having all the logic in submitProcedure(), split in 
> registerNonce() + submitProcedure().
> In this case we can avoid calling the coprocessor twice and having a clean 
> submit logic knowing that there will only be one submission.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784122#comment-15784122
 ] 

Hadoop QA commented on HBASE-14123:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 5s 
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 23 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 25s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m 
5s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s 
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 52s 
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 9s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 28s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 28s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 28s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m 
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m 
4s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
25m 54s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 3m 
15s {color} | {color:green} the patch passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s 
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 55s 
{color} | {color:red} hbase-server generated 2 new + 0 unchanged - 0 fixed = 2 
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 14s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 43s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 27s 
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 57s 

[jira] [Updated] (HBASE-17149) Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor multiple times

[Stephen Yuan Jiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Yuan Jiang updated HBASE-17149:
---
Attachment: HBASE-17149.v1-branch-1.2.patch

> Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor 
> multiple times
> ---
>
> Key: HBASE-17149
> URL: https://issues.apache.org/jira/browse/HBASE-17149
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: 17149.branch-1.incomplete.txt, 
> HBASE-17149-addendum.v1-master.patch, HBASE-17149.master.001.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.003.patch, 
> HBASE-17149.v1-branch-1.2.patch, HBASE-17149.v1-branch-1.patch, nonce.patch
>
>
> instead of having all the logic in submitProcedure(), split in 
> registerNonce() + submitProcedure().
> In this case we can avoid calling the coprocessor twice and having a clean 
> submit logic knowing that there will only be one submission.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17387) Reduce the overhead of exception report in RegionActionResult for multi()

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-17387:
---
Priority: Minor  (was: Major)

> Reduce the overhead of exception report in RegionActionResult for multi()
> -
>
> Key: HBASE-17387
> URL: https://issues.apache.org/jira/browse/HBASE-17387
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>
> For RSRpcServices#doNonAtomicRegionMutation() :
> {code}
> for (ClientProtos.Action action: actions.getActionList()) {
> ...
>   } catch (IOException ie) {
> rpcServer.getMetrics().exception(ie);
> resultOrExceptionBuilder = ResultOrException.newBuilder().
>   setException(ResponseConverter.buildException(ie));
>   }
>   if (resultOrExceptionBuilder != null) {
> // Propagate index.
> resultOrExceptionBuilder.setIndex(action.getIndex());
> builder.addResultOrException(resultOrExceptionBuilder.build());
>   }
> {code}
> The exceptions are added to builder in the for loop.
> The ClientProtos.ResultOrException.Builder instance is created within the for 
> loop.
> For large multi call, this may incur non-trivial overhead for garbage 
> collector if there're many exceptions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17238) Wrong in-memory hbase:meta location causing SSH failure

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784053#comment-15784053
 ] 

Hudson commented on HBASE-17238:


FAILURE: Integrated in Jenkins build HBase-1.1-JDK7 #1830 (See 
[https://builds.apache.org/job/HBase-1.1-JDK7/1830/])
HBASE-17238 Wrong in-memory hbase:meta location causing SSH failure 
(syuanjiangdev: rev a1d900db44cf7834f714b2ad4720795ddbc45e12)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMetaWithReplicas.java


> Wrong in-memory hbase:meta location causing SSH failure
> ---
>
> Key: HBASE-17238
> URL: https://issues.apache.org/jira/browse/HBASE-17238
> Project: HBase
>  Issue Type: Bug
>  Components: Region Assignment
>Affects Versions: 1.1.0
>Reporter: Stephen Yuan Jiang
>Assignee: Stephen Yuan Jiang
>Priority: Critical
> Fix For: 1.3.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: HBASE-17238.v1-branch-1.1.patch, 
> HBASE-17238.v1-branch-1.patch, HBASE-17238.v2-branch-1.1.patch
>
>
> In HBase 1.x, if HMaster#assignMeta() assigns a non-DEFAULT_REPLICA_ID 
> hbase:meta region, it would wrongly update the DEFAULT_REPLICA_ID hbase:meta 
> region in-memory.  This caused the in-memory region state has wrong RS 
> information for default replica hbase:meta region.  One of the problem we saw 
> is a wrong type of SSH could be chosen and causing problems.
> {code}
> void assignMeta(MonitoredTask status, Set 
> previouslyFailedMetaRSs, int replicaId)
>   throws InterruptedException, IOException, KeeperException {
> // Work on meta region
> ...
> if (replicaId == HRegionInfo.DEFAULT_REPLICA_ID) {
>   status.setStatus("Assigning hbase:meta region");
> } else {
>   status.setStatus("Assigning hbase:meta region, replicaId " + replicaId);
> }
> // Get current meta state from zk.
> RegionStates regionStates = assignmentManager.getRegionStates();
> RegionState metaState = 
> MetaTableLocator.getMetaRegionState(getZooKeeper(), replicaId);
> HRegionInfo hri = 
> RegionReplicaUtil.getRegionInfoForReplica(HRegionInfo.FIRST_META_REGIONINFO,
> replicaId);
> ServerName currentMetaServer = metaState.getServerName();
> ...
> boolean rit = this.assignmentManager.
>   processRegionInTransitionAndBlockUntilAssigned(hri);
> boolean metaRegionLocation = metaTableLocator.verifyMetaRegionLocation(
>   this.getConnection(), this.getZooKeeper(), timeout, replicaId);
> ...
> } else {
>   // Region already assigned. We didn't assign it. Add to in-memory state.
>   regionStates.updateRegionState(
> HRegionInfo.FIRST_META_REGIONINFO, State.OPEN, currentMetaServer); 
> <<--- Wrong region to update -->>
>   this.assignmentManager.regionOnline(
> HRegionInfo.FIRST_META_REGIONINFO, currentMetaServer); <<--- Wrong 
> region to update -->>
> }
> ...
> {code}
> Here is the problem scenario:
> Step 1: master failovers (or starts could have the same issue) and find 
> default replica of hbase:meta is in rs1.
> {noformat}
> 2016-11-26 00:06:36,590 INFO org.apache.hadoop.hbase.master.ServerManager: 
> AssignmentManager hasn't finished failover cleanup; waiting
> 2016-11-26 00:06:36,591 INFO org.apache.hadoop.hbase.master.HMaster: 
> hbase:meta with replicaId 0 assigned=0, rit=false, 
> location=rs1,60200,1480103147220
> {noformat}
> Step 2: master finds that replica 1 of hbase:meta is unassigned, therefore, 
> HMaster#assignMeta() is called and assign the replica 1 region to rs2, but at 
> the end, it wrongly updates the in-memory state of default replica to rs2
> {noformat}
> 2016-11-26 00:08:21,741 DEBUG org.apache.hadoop.hbase.master.RegionStates: 
> Onlined 1588230740 on rs2,60200,1480102993815 {ENCODED => 1588230740, NAME => 
> 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}
> 2016-11-26 00:08:21,741 INFO org.apache.hadoop.hbase.master.RegionStates: 
> Offlined 1588230740 from rs1,60200,1480103147220
> 2016-11-26 00:08:21,741 INFO org.apache.hadoop.hbase.master.HMaster: 
> hbase:meta with replicaId 1 assigned=0, rit=false, 
> location=rs2,60200,1480102993815
> {noformat}
> Step 3: now rs1 is down, master needs to choose which SSH to call 
> (MetaServerShutdownHandler or normal ServerShutdownHandler) - in this case, 
> MetaServerShutdownHandler should be chosen; however, due to wrong in-memory 
> location, normal ServerShutdownHandler was chosen:
> {noformat}
> 2016-11-26 00:08:33,995 INFO 
> org.apache.hadoop.hbase.zookeeper.RegionServerTracker: RegionServer ephemeral 
> node deleted, processing expiration [rs1,60200,1480103147220]
> 2016-11-26 00:08:33,998 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: based on AM, 

[jira] [Created] (HBASE-17387) Reduce the overhead of exception report in RegionActionResult for multi()

[Ted Yu (JIRA)]

Ted Yu created HBASE-17387:
--

 Summary: Reduce the overhead of exception report in 
RegionActionResult for multi()
 Key: HBASE-17387
 URL: https://issues.apache.org/jira/browse/HBASE-17387
 Project: HBase
  Issue Type: Bug
Reporter: Ted Yu


For RSRpcServices#doNonAtomicRegionMutation() :
{code}
for (ClientProtos.Action action: actions.getActionList()) {
...
  } catch (IOException ie) {
rpcServer.getMetrics().exception(ie);
resultOrExceptionBuilder = ResultOrException.newBuilder().
  setException(ResponseConverter.buildException(ie));
  }
  if (resultOrExceptionBuilder != null) {
// Propagate index.
resultOrExceptionBuilder.setIndex(action.getIndex());
builder.addResultOrException(resultOrExceptionBuilder.build());
  }
{code}
The exceptions are added to builder in the for loop.
The ClientProtos.ResultOrException.Builder instance is created within the for 
loop.

For large multi call, this may incur non-trivial overhead for garbage collector 
if there're many exceptions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12439) Procedure V2

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-12439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-12439:
--
Attachment: Procedurev2Notification-BusRoadmap.pdf

Updated Roadmap Document from Matteo

> Procedure V2
> 
>
> Key: HBASE-12439
> URL: https://issues.apache.org/jira/browse/HBASE-12439
> Project: HBase
>  Issue Type: New Feature
>  Components: master
>Affects Versions: 2.0.0
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>  Labels: reliability
> Fix For: 2.0.0
>
> Attachments: ProcedureV2.pdf, Procedurev2Notification-Bus.pdf, 
> Procedurev2Notification-BusRoadmap.pdf
>
>
> Procedure v2 (aka Notification Bus) aims to provide a unified way to build:
> * multi-steps procedure with a rollback/rollforward ability in case of 
> failure (e.g. create/delete table)
> ** HBASE-12070
> * notifications across multiple machines (e.g. ACLs/Labels/Quotas cache 
> updates)
> ** Make sure that every machine has the grant/revoke/label
> ** Enforce "space limit" quota across the namespace
> ** HBASE-10295 eliminate permanent replication zk node
> * procedures across multiple machines (e.g. Snapshots)
> * coordinated long-running procedures (e.g. compactions, splits, ...)
> * Synchronous calls, with the ability to see the state/result in case of 
> failure.
> ** HBASE-11608 sync split
> still work in progress/initial prototype: https://reviews.apache.org/r/27703/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12439) Procedure V2

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-12439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-12439:
--
Attachment: (was: Procedure v2 roadmap.pdf)

> Procedure V2
> 
>
> Key: HBASE-12439
> URL: https://issues.apache.org/jira/browse/HBASE-12439
> Project: HBase
>  Issue Type: New Feature
>  Components: master
>Affects Versions: 2.0.0
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>  Labels: reliability
> Fix For: 2.0.0
>
> Attachments: ProcedureV2.pdf, Procedurev2Notification-Bus.pdf
>
>
> Procedure v2 (aka Notification Bus) aims to provide a unified way to build:
> * multi-steps procedure with a rollback/rollforward ability in case of 
> failure (e.g. create/delete table)
> ** HBASE-12070
> * notifications across multiple machines (e.g. ACLs/Labels/Quotas cache 
> updates)
> ** Make sure that every machine has the grant/revoke/label
> ** Enforce "space limit" quota across the namespace
> ** HBASE-10295 eliminate permanent replication zk node
> * procedures across multiple machines (e.g. Snapshots)
> * coordinated long-running procedures (e.g. compactions, splits, ...)
> * Synchronous calls, with the ability to see the state/result in case of 
> failure.
> ** HBASE-11608 sync split
> still work in progress/initial prototype: https://reviews.apache.org/r/27703/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17081) Flush the entire CompactingMemStore content to disk

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784008#comment-15784008
 ] 

Hudson commented on HBASE-17081:


FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2217 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/2217/])
HBASE-17081 Flush the entire CompactingMemStore content to disk - revert 
(tedyu: rev 79e5efd35c9f3660b8c58364f25816581fb84d7a)
* (delete) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompositeImmutableSegment.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemstoreSize.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/AbstractMemStore.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWalAndCompactingMemStoreFlush.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactingMemStore.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultMemStore.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionPipeline.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultMemStore.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactingMemStore.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreCompactor.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ImmutableSegment.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SegmentFactory.java


> Flush the entire CompactingMemStore content to disk
> ---
>
> Key: HBASE-17081
> URL: https://issues.apache.org/jira/browse/HBASE-17081
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Eshcar Hillel
> Fix For: 2.0.0
>
> Attachments: HBASE-15787_8.patch, HBASE-17081-V01.patch, 
> HBASE-17081-V02.patch, HBASE-17081-V03.patch, HBASE-17081-V04.patch, 
> HBASE-17081-V05.patch, HBASE-17081-V06.patch, HBASE-17081-V06.patch, 
> HBASE-17081-V07.patch, HBASE-17081-V10.patch, 
> HBaseMeetupDecember2016-V02.pptx, Pipelinememstore_fortrunk_3.patch
>
>
> Part of CompactingMemStore's memory is held by an active segment, and another 
> part is divided between immutable segments in the compacting pipeline. Upon 
> flush-to-disk request we want to flush all of it to disk, in contrast to 
> flushing only tail of the compacting pipeline.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17149) Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor multiple times

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784009#comment-15784009
 ] 

Hudson commented on HBASE-17149:


FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2217 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/2217/])
HBASE-17149 Procedure V2 - Fix nonce submission to avoid unnecessary (stack: 
rev a3e0e0df0d0957fc02723aa349f96ba45bda3c7f)
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestCreateNamespaceProcedure.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestModifyNamespaceProcedure.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestDeleteNamespaceProcedure.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestTableDDLProcedureBase.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestMergeTableRegionsProcedure.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestTruncateTableProcedure.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestDeleteColumnFamilyProcedure.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestRestoreSnapshotProcedure.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestCloneSnapshotProcedure.java


> Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor 
> multiple times
> ---
>
> Key: HBASE-17149
> URL: https://issues.apache.org/jira/browse/HBASE-17149
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: 17149.branch-1.incomplete.txt, 
> HBASE-17149-addendum.v1-master.patch, HBASE-17149.master.001.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.003.patch, 
> HBASE-17149.v1-branch-1.patch, nonce.patch
>
>
> instead of having all the logic in submitProcedure(), split in 
> registerNonce() + submitProcedure().
> In this case we can avoid calling the coprocessor twice and having a clean 
> submit logic knowing that there will only be one submission.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17386) Adding FN documentation to reference guide

[Thiruvel Thirumoolan (JIRA)]

Thiruvel Thirumoolan created HBASE-17386:


 Summary: Adding FN documentation to reference guide
 Key: HBASE-17386
 URL: https://issues.apache.org/jira/browse/HBASE-17386
 Project: HBase
  Issue Type: Sub-task
Reporter: Thiruvel Thirumoolan
Assignee: Thiruvel Thirumoolan






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15141) Procedure V2 - Web UI

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15141:
--
Priority: Blocker  (was: Trivial)

> Procedure V2 - Web UI
> -
>
> Key: HBASE-15141
> URL: https://issues.apache.org/jira/browse/HBASE-15141
> Project: HBase
>  Issue Type: Task
>  Components: proc-v2, UI
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Blocker
>
> The procedures should be displayed in the webui. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17372) Make AsyncTable thread safe

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783945#comment-15783945
 ] 

stack commented on HBASE-17372:
---

Looking at patch, I like the aggregating of all the timeout into a single 
class. It is a very nice improvement/cleanup.

I like the [~yangzhe1991] suggestion of operationconfig rather than retryconfig 
so could add in to it other per-request-configs... (as you suggest above 
...non-timeout related config):

  private final RetryConfig retryConfig;

This is very nice.





> Make AsyncTable thread safe
> ---
>
> Key: HBASE-17372
> URL: https://issues.apache.org/jira/browse/HBASE-17372
> Project: HBase
>  Issue Type: Sub-task
>  Components: asyncclient, Client
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17372-v1.patch, HBASE-17372.patch
>
>
> The most methods are already thread safe. The problem is that we have some 
> methods that used to set timeout, we need to remove these methods and add a 
> parameter for each call to specific timeout settings.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17149) Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor multiple times

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783944#comment-15783944
 ] 

Hadoop QA commented on HBASE-17149:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 19 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
46s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s 
{color} | {color:green} branch-1 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s 
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
23s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
28s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 5s 
{color} | {color:red} hbase-server in branch-1 has 2 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s 
{color} | {color:green} branch-1 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s 
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s 
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
29s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
15m 18s {color} | {color:green} The patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 49s 
{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 86m 39s 
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
30s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 124m 1s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:e01ee2f |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12844979/HBASE-17149.v1-branch-1.patch
 |
| 

[jira] [Commented] (HBASE-17346) Add coprocessor service support

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783892#comment-15783892
 ] 

stack commented on HBASE-17346:
---

bq. I think we need to redesign the API.

Which API boss?

bq. The old implementation is a little confusing to me.

Nod. Understood. But in the end I found it coheres it makes some sense.

bq. We have an AggregationClient but marked as InterfaceAudience.Private?

Yes. It looks like the below cleanup project changed it from pubilc, evolving, 
which makes more sense:

commit cd63f2055e001e9b3063676cf75b471eae9b3d75
Author: Jonathan Hsieh 
Date:   Mon Sep 16 05:18:27 2013 +

HBASE-9529 Audit of hbase-client @InterfaceAudience.Public apis

This client used to live inside in hbase-client (or hbase-server, I don't 
remember which) so the intent was probably to shutdown access generally and in 
this case, on a little used endpoint.

Now this client has been moved out to hbase-endpoint and it is a canonical 
illustration of how to do a coprocessor, we could change the audience back to 
public?

On use of ServerRpcController, that looks like lazyness (on my part I think).

Attaching a patch that cleans up the audience and that makes a local 
RpcController for use by this Endpoint. Looking at the endpoint utility 
methods, we seem to do the right thing if it is a base RpcController and not a 
ServerRpcController.

On retrying, yeah, looks like none implemented.

Thanks [~Apache9]




> Add coprocessor service support
> ---
>
> Key: HBASE-17346
> URL: https://issues.apache.org/jira/browse/HBASE-17346
> Project: HBase
>  Issue Type: Sub-task
>  Components: Client
>Reporter: Duo Zhang
> Fix For: 2.0.0
>
>
> I think we need to redesign the API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17238) Wrong in-memory hbase:meta location causing SSH failure

[Stephen Yuan Jiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Yuan Jiang updated HBASE-17238:
---
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 1.1.8
   1.2.5
   1.4.0
   1.3.0
   Status: Resolved  (was: Patch Available)

> Wrong in-memory hbase:meta location causing SSH failure
> ---
>
> Key: HBASE-17238
> URL: https://issues.apache.org/jira/browse/HBASE-17238
> Project: HBase
>  Issue Type: Bug
>  Components: Region Assignment
>Affects Versions: 1.1.0
>Reporter: Stephen Yuan Jiang
>Assignee: Stephen Yuan Jiang
>Priority: Critical
> Fix For: 1.3.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: HBASE-17238.v1-branch-1.1.patch, 
> HBASE-17238.v1-branch-1.patch, HBASE-17238.v2-branch-1.1.patch
>
>
> In HBase 1.x, if HMaster#assignMeta() assigns a non-DEFAULT_REPLICA_ID 
> hbase:meta region, it would wrongly update the DEFAULT_REPLICA_ID hbase:meta 
> region in-memory.  This caused the in-memory region state has wrong RS 
> information for default replica hbase:meta region.  One of the problem we saw 
> is a wrong type of SSH could be chosen and causing problems.
> {code}
> void assignMeta(MonitoredTask status, Set 
> previouslyFailedMetaRSs, int replicaId)
>   throws InterruptedException, IOException, KeeperException {
> // Work on meta region
> ...
> if (replicaId == HRegionInfo.DEFAULT_REPLICA_ID) {
>   status.setStatus("Assigning hbase:meta region");
> } else {
>   status.setStatus("Assigning hbase:meta region, replicaId " + replicaId);
> }
> // Get current meta state from zk.
> RegionStates regionStates = assignmentManager.getRegionStates();
> RegionState metaState = 
> MetaTableLocator.getMetaRegionState(getZooKeeper(), replicaId);
> HRegionInfo hri = 
> RegionReplicaUtil.getRegionInfoForReplica(HRegionInfo.FIRST_META_REGIONINFO,
> replicaId);
> ServerName currentMetaServer = metaState.getServerName();
> ...
> boolean rit = this.assignmentManager.
>   processRegionInTransitionAndBlockUntilAssigned(hri);
> boolean metaRegionLocation = metaTableLocator.verifyMetaRegionLocation(
>   this.getConnection(), this.getZooKeeper(), timeout, replicaId);
> ...
> } else {
>   // Region already assigned. We didn't assign it. Add to in-memory state.
>   regionStates.updateRegionState(
> HRegionInfo.FIRST_META_REGIONINFO, State.OPEN, currentMetaServer); 
> <<--- Wrong region to update -->>
>   this.assignmentManager.regionOnline(
> HRegionInfo.FIRST_META_REGIONINFO, currentMetaServer); <<--- Wrong 
> region to update -->>
> }
> ...
> {code}
> Here is the problem scenario:
> Step 1: master failovers (or starts could have the same issue) and find 
> default replica of hbase:meta is in rs1.
> {noformat}
> 2016-11-26 00:06:36,590 INFO org.apache.hadoop.hbase.master.ServerManager: 
> AssignmentManager hasn't finished failover cleanup; waiting
> 2016-11-26 00:06:36,591 INFO org.apache.hadoop.hbase.master.HMaster: 
> hbase:meta with replicaId 0 assigned=0, rit=false, 
> location=rs1,60200,1480103147220
> {noformat}
> Step 2: master finds that replica 1 of hbase:meta is unassigned, therefore, 
> HMaster#assignMeta() is called and assign the replica 1 region to rs2, but at 
> the end, it wrongly updates the in-memory state of default replica to rs2
> {noformat}
> 2016-11-26 00:08:21,741 DEBUG org.apache.hadoop.hbase.master.RegionStates: 
> Onlined 1588230740 on rs2,60200,1480102993815 {ENCODED => 1588230740, NAME => 
> 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}
> 2016-11-26 00:08:21,741 INFO org.apache.hadoop.hbase.master.RegionStates: 
> Offlined 1588230740 from rs1,60200,1480103147220
> 2016-11-26 00:08:21,741 INFO org.apache.hadoop.hbase.master.HMaster: 
> hbase:meta with replicaId 1 assigned=0, rit=false, 
> location=rs2,60200,1480102993815
> {noformat}
> Step 3: now rs1 is down, master needs to choose which SSH to call 
> (MetaServerShutdownHandler or normal ServerShutdownHandler) - in this case, 
> MetaServerShutdownHandler should be chosen; however, due to wrong in-memory 
> location, normal ServerShutdownHandler was chosen:
> {noformat}
> 2016-11-26 00:08:33,995 INFO 
> org.apache.hadoop.hbase.zookeeper.RegionServerTracker: RegionServer ephemeral 
> node deleted, processing expiration [rs1,60200,1480103147220]
> 2016-11-26 00:08:33,998 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: based on AM, current 
> region=hbase:meta,,1.1588230740 is on server=rs2,60200,1480102993815 server 
> being checked: rs1,60200,1480103147220
> 2016-11-26 00:08:34,001 DEBUG org.apache.hadoop.hbase.master.ServerManager: 
> 

[jira] [Commented] (HBASE-17385) Change usage documentation from bin/hbase to hbase in various tools

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783765#comment-15783765
 ] 

Enis Soztutar commented on HBASE-17385:
---

Should be an easy task. Do a grep of the code as a starter: 
{code}
$ grep -R "bin/hbase" . 
{code}

> Change usage documentation from bin/hbase to hbase in various tools 
> 
>
> Key: HBASE-17385
> URL: https://issues.apache.org/jira/browse/HBASE-17385
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>  Labels: beginner, beginners
>
> Some tools print bin/hbase in their usage documentation.{{bin/hbase}} is only 
> applicable to development environments. Typical deployments always should  
> just refer to {{hbase}}. 
> For example, CopyTable usage is like: 
> {code}
> Usage: CopyTable [general options] [--starttime=X] [--endtime=Y] 
> [--new.name=NEW] [--peer.adr=ADR] 
> Options:
>  rs.class hbase.regionserver.class of the peer cluster
>   specify if different from current cluster
>  rs.impl  hbase.regionserver.impl of the peer cluster
>  startrow the start row
>  stoprow  the stop row
>  starttimebeginning of the time range (unixtime in millis)
>   without endtime means from starttime to forever
>  endtime  end of the time range.  Ignored if no starttime specified.
>  versions number of cell versions to copy
>  new.name new table's name
>  peer.adr Address of the peer cluster given in the format
>   
> hbase.zookeeper.quorum:hbase.zookeeper.client.port:zookeeper.znode.parent
>  families comma-separated list of families to copy
>   To copy from cf1 to cf2, give sourceCfName:destCfName. 
>   To keep the same name, just give "cfName"
>  all.cellsalso copy delete markers and deleted cells
>  bulkload Write input into HFiles and bulk load to the destination table
> Args:
>  tablenameName of the table to copy
> Examples:
>  To copy 'TestTable' to a cluster that uses replication for a 1 hour window:
>  $ bin/hbase org.apache.hadoop.hbase.mapreduce.CopyTable 
> --starttime=1265875194289 --endtime=1265878794289 
> --peer.adr=server1,server2,server3:2181:/hbase 
> --families=myOldCf:myNewCf,cf2,cf3 TestTable 
> For performance consider the following general option:
>   It is recommended that you set the following to >=100. A higher value uses 
> more memory but
>   decreases the round trip time to the server and may increase performance.
> -Dhbase.client.scanner.caching=100
>   The following should always be set to false, to prevent writing data twice, 
> which may produce 
>   inaccurate results.
> -Dmapreduce.map.speculative=false
> {code}
> in above, it should be: 
> {code}
>  To copy 'TestTable' to a cluster that uses replication for a 1 hour window:
>  $ hbase org.apache.hadoop.hbase.mapreduce.CopyTable 
> --starttime=1265875194289 --endtime=1265878794289 
> --peer.adr=server1,server2,server3:2181:/hbase --
> families=myOldCf:myNewCf,cf2,cf3 TestTable 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17385) Change usage documentation from bin/hbase to hbase in various tools

[Enis Soztutar (JIRA)]

Enis Soztutar created HBASE-17385:
-

 Summary: Change usage documentation from bin/hbase to hbase in 
various tools 
 Key: HBASE-17385
 URL: https://issues.apache.org/jira/browse/HBASE-17385
 Project: HBase
  Issue Type: Bug
Reporter: Enis Soztutar


Some tools print bin/hbase in their usage documentation.{{bin/hbase}} is only 
applicable to development environments. Typical deployments always should  just 
refer to {{hbase}}. 

For example, CopyTable usage is like: 
{code}
Usage: CopyTable [general options] [--starttime=X] [--endtime=Y] 
[--new.name=NEW] [--peer.adr=ADR] 

Options:
 rs.class hbase.regionserver.class of the peer cluster
  specify if different from current cluster
 rs.impl  hbase.regionserver.impl of the peer cluster
 startrow the start row
 stoprow  the stop row
 starttimebeginning of the time range (unixtime in millis)
  without endtime means from starttime to forever
 endtime  end of the time range.  Ignored if no starttime specified.
 versions number of cell versions to copy
 new.name new table's name
 peer.adr Address of the peer cluster given in the format
  
hbase.zookeeper.quorum:hbase.zookeeper.client.port:zookeeper.znode.parent
 families comma-separated list of families to copy
  To copy from cf1 to cf2, give sourceCfName:destCfName. 
  To keep the same name, just give "cfName"
 all.cellsalso copy delete markers and deleted cells
 bulkload Write input into HFiles and bulk load to the destination table

Args:
 tablenameName of the table to copy

Examples:
 To copy 'TestTable' to a cluster that uses replication for a 1 hour window:
 $ bin/hbase org.apache.hadoop.hbase.mapreduce.CopyTable 
--starttime=1265875194289 --endtime=1265878794289 
--peer.adr=server1,server2,server3:2181:/hbase 
--families=myOldCf:myNewCf,cf2,cf3 TestTable 
For performance consider the following general option:
  It is recommended that you set the following to >=100. A higher value uses 
more memory but
  decreases the round trip time to the server and may increase performance.
-Dhbase.client.scanner.caching=100
  The following should always be set to false, to prevent writing data twice, 
which may produce 
  inaccurate results.
-Dmapreduce.map.speculative=false
{code}

in above, it should be: 
{code}
 To copy 'TestTable' to a cluster that uses replication for a 1 hour window:
 $ hbase org.apache.hadoop.hbase.mapreduce.CopyTable --starttime=1265875194289 
--endtime=1265878794289 --peer.adr=server1,server2,server3:2181:/hbase --
families=myOldCf:myNewCf,cf2,cf3 TestTable 
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17238) Wrong in-memory hbase:meta location causing SSH failure

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783753#comment-15783753
 ] 

Enis Soztutar commented on HBASE-17238:
---

+1. Nice test. 

BTW, for the future, you can use HBaseTestingUtil.waitFor() instead of these 
kind of loops: 
{code}
+do {
+  Thread.sleep(100);
+  newMeta1SN = rl.getRegionLocation(1).getServerName();
+  i++;
+} while (meta1SN.equals(newMeta1SN) & i < 600); // wait for 60 seconds
{code}
Feel free to fix on commit if you want. 

Commit this to all branches except master? 

> Wrong in-memory hbase:meta location causing SSH failure
> ---
>
> Key: HBASE-17238
> URL: https://issues.apache.org/jira/browse/HBASE-17238
> Project: HBase
>  Issue Type: Bug
>  Components: Region Assignment
>Affects Versions: 1.1.0
>Reporter: Stephen Yuan Jiang
>Assignee: Stephen Yuan Jiang
>Priority: Critical
> Attachments: HBASE-17238.v1-branch-1.1.patch, 
> HBASE-17238.v1-branch-1.patch, HBASE-17238.v2-branch-1.1.patch
>
>
> In HBase 1.x, if HMaster#assignMeta() assigns a non-DEFAULT_REPLICA_ID 
> hbase:meta region, it would wrongly update the DEFAULT_REPLICA_ID hbase:meta 
> region in-memory.  This caused the in-memory region state has wrong RS 
> information for default replica hbase:meta region.  One of the problem we saw 
> is a wrong type of SSH could be chosen and causing problems.
> {code}
> void assignMeta(MonitoredTask status, Set 
> previouslyFailedMetaRSs, int replicaId)
>   throws InterruptedException, IOException, KeeperException {
> // Work on meta region
> ...
> if (replicaId == HRegionInfo.DEFAULT_REPLICA_ID) {
>   status.setStatus("Assigning hbase:meta region");
> } else {
>   status.setStatus("Assigning hbase:meta region, replicaId " + replicaId);
> }
> // Get current meta state from zk.
> RegionStates regionStates = assignmentManager.getRegionStates();
> RegionState metaState = 
> MetaTableLocator.getMetaRegionState(getZooKeeper(), replicaId);
> HRegionInfo hri = 
> RegionReplicaUtil.getRegionInfoForReplica(HRegionInfo.FIRST_META_REGIONINFO,
> replicaId);
> ServerName currentMetaServer = metaState.getServerName();
> ...
> boolean rit = this.assignmentManager.
>   processRegionInTransitionAndBlockUntilAssigned(hri);
> boolean metaRegionLocation = metaTableLocator.verifyMetaRegionLocation(
>   this.getConnection(), this.getZooKeeper(), timeout, replicaId);
> ...
> } else {
>   // Region already assigned. We didn't assign it. Add to in-memory state.
>   regionStates.updateRegionState(
> HRegionInfo.FIRST_META_REGIONINFO, State.OPEN, currentMetaServer); 
> <<--- Wrong region to update -->>
>   this.assignmentManager.regionOnline(
> HRegionInfo.FIRST_META_REGIONINFO, currentMetaServer); <<--- Wrong 
> region to update -->>
> }
> ...
> {code}
> Here is the problem scenario:
> Step 1: master failovers (or starts could have the same issue) and find 
> default replica of hbase:meta is in rs1.
> {noformat}
> 2016-11-26 00:06:36,590 INFO org.apache.hadoop.hbase.master.ServerManager: 
> AssignmentManager hasn't finished failover cleanup; waiting
> 2016-11-26 00:06:36,591 INFO org.apache.hadoop.hbase.master.HMaster: 
> hbase:meta with replicaId 0 assigned=0, rit=false, 
> location=rs1,60200,1480103147220
> {noformat}
> Step 2: master finds that replica 1 of hbase:meta is unassigned, therefore, 
> HMaster#assignMeta() is called and assign the replica 1 region to rs2, but at 
> the end, it wrongly updates the in-memory state of default replica to rs2
> {noformat}
> 2016-11-26 00:08:21,741 DEBUG org.apache.hadoop.hbase.master.RegionStates: 
> Onlined 1588230740 on rs2,60200,1480102993815 {ENCODED => 1588230740, NAME => 
> 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}
> 2016-11-26 00:08:21,741 INFO org.apache.hadoop.hbase.master.RegionStates: 
> Offlined 1588230740 from rs1,60200,1480103147220
> 2016-11-26 00:08:21,741 INFO org.apache.hadoop.hbase.master.HMaster: 
> hbase:meta with replicaId 1 assigned=0, rit=false, 
> location=rs2,60200,1480102993815
> {noformat}
> Step 3: now rs1 is down, master needs to choose which SSH to call 
> (MetaServerShutdownHandler or normal ServerShutdownHandler) - in this case, 
> MetaServerShutdownHandler should be chosen; however, due to wrong in-memory 
> location, normal ServerShutdownHandler was chosen:
> {noformat}
> 2016-11-26 00:08:33,995 INFO 
> org.apache.hadoop.hbase.zookeeper.RegionServerTracker: RegionServer ephemeral 
> node deleted, processing expiration [rs1,60200,1480103147220]
> 2016-11-26 00:08:33,998 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: based on AM, current 
> region=hbase:meta,,1.1588230740 is on 

[jira] [Commented] (HBASE-16851) User-facing documentation for the In-Memory Compaction feature

[Edward Bortnikov (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783747#comment-15783747
 ] 

Edward Bortnikov commented on HBASE-16851:
--

Thanks Michael. Unlocked the doc for commenting. 

> User-facing documentation for the In-Memory Compaction feature
> --
>
> Key: HBASE-16851
> URL: https://issues.apache.org/jira/browse/HBASE-16851
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Edward Bortnikov
>Assignee: Edward Bortnikov
> Attachments: Accordion HBase In-Memory Compaction - Nov 1 .pdf, 
> Accordion HBase In-Memory Compaction - Nov 23.pdf, Accordion_ HBase In-Memory 
> Compaction - Oct 27.pdf, HBaseAcceleratedHbaseConf-final.pptx
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17149) Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor multiple times

[Stephen Yuan Jiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783736#comment-15783736
 ] 

Stephen Yuan Jiang commented on HBASE-17149:


the addendum has already committed.  No need to run pre-commit job.

> Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor 
> multiple times
> ---
>
> Key: HBASE-17149
> URL: https://issues.apache.org/jira/browse/HBASE-17149
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: 17149.branch-1.incomplete.txt, 
> HBASE-17149-addendum.v1-master.patch, HBASE-17149.master.001.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.003.patch, 
> HBASE-17149.v1-branch-1.patch, nonce.patch
>
>
> instead of having all the logic in submitProcedure(), split in 
> registerNonce() + submitProcedure().
> In this case we can avoid calling the coprocessor twice and having a clean 
> submit logic knowing that there will only be one submission.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17379) Lack of synchronization in CompactionPipeline#getScanners()

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783732#comment-15783732
 ] 

Ted Yu commented on HBASE-17379:


TestHRegionWithInMemoryFlush passed:
{code}
Tests run: 106, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 97.854 sec - 
in org.apache.hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush
{code}

TestMasterProcedureWalLease failure was not related.

> Lack of synchronization in CompactionPipeline#getScanners()
> ---
>
> Key: HBASE-17379
> URL: https://issues.apache.org/jira/browse/HBASE-17379
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 17379.v1.txt, 17379.v2.txt, 17379.v3.txt, 17379.v4.txt, 
> 17379.v5.txt, 17379.v6.txt
>
>
> From 
> https://builds.apache.org/job/PreCommit-HBASE-Build/5053/testReport/org.apache.hadoop.hbase.regionserver/TestHRegionWithInMemoryFlush/testWritesWhileGetting/
>  :
> {code}
> java.io.IOException: java.util.ConcurrentModificationException
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.handleException(HRegion.java:5886)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.initializeScanners(HRegion.java:5856)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5819)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2786)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2766)
>   at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7036)
>   at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7015)
>   at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6994)
>   at 
> org.apache.hadoop.hbase.regionserver.TestHRegion.testWritesWhileGetting(TestHRegion.java:4141)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.ConcurrentModificationException: null
>   at 
> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
>   at java.util.LinkedList$ListItr.next(LinkedList.java:888)
>   at 
> org.apache.hadoop.hbase.regionserver.CompactionPipeline.getScanners(CompactionPipeline.java:220)
>   at 
> org.apache.hadoop.hbase.regionserver.CompactingMemStore.getScanners(CompactingMemStore.java:298)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore.getScanners(HStore.java:1154)
>   at org.apache.hadoop.hbase.regionserver.Store.getScanners(Store.java:97)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreScanner.getScannersNoCompaction(StoreScanner.java:353)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:210)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:1892)
>   at 

[jira] [Updated] (HBASE-17149) Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor multiple times

[Stephen Yuan Jiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Yuan Jiang updated HBASE-17149:
---
Attachment: (was: HBASE-17149.v1-branch-1.patch)

> Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor 
> multiple times
> ---
>
> Key: HBASE-17149
> URL: https://issues.apache.org/jira/browse/HBASE-17149
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: 17149.branch-1.incomplete.txt, 
> HBASE-17149-addendum.v1-master.patch, HBASE-17149.master.001.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.003.patch, 
> HBASE-17149.v1-branch-1.patch, nonce.patch
>
>
> instead of having all the logic in submitProcedure(), split in 
> registerNonce() + submitProcedure().
> In this case we can avoid calling the coprocessor twice and having a clean 
> submit logic knowing that there will only be one submission.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17149) Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor multiple times

[Stephen Yuan Jiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Yuan Jiang updated HBASE-17149:
---
Attachment: HBASE-17149.v1-branch-1.patch

> Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor 
> multiple times
> ---
>
> Key: HBASE-17149
> URL: https://issues.apache.org/jira/browse/HBASE-17149
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: 17149.branch-1.incomplete.txt, 
> HBASE-17149-addendum.v1-master.patch, HBASE-17149.master.001.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.003.patch, 
> HBASE-17149.v1-branch-1.patch, nonce.patch
>
>
> instead of having all the logic in submitProcedure(), split in 
> registerNonce() + submitProcedure().
> In this case we can avoid calling the coprocessor twice and having a clean 
> submit logic knowing that there will only be one submission.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17149) Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor multiple times

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783691#comment-15783691
 ] 

Hadoop QA commented on HBASE-17149:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s {color} 
| {color:red} HBASE-17149 does not apply to master. Rebase required? Wrong 
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames 
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12844963/HBASE-17149-addendum.v1-master.patch
 |
| JIRA Issue | HBASE-17149 |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5079/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor 
> multiple times
> ---
>
> Key: HBASE-17149
> URL: https://issues.apache.org/jira/browse/HBASE-17149
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: 17149.branch-1.incomplete.txt, 
> HBASE-17149-addendum.v1-master.patch, HBASE-17149.master.001.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.003.patch, 
> HBASE-17149.v1-branch-1.patch, nonce.patch
>
>
> instead of having all the logic in submitProcedure(), split in 
> registerNonce() + submitProcedure().
> In this case we can avoid calling the coprocessor twice and having a clean 
> submit logic knowing that there will only be one submission.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17149) Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor multiple times

[Stephen Yuan Jiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Yuan Jiang updated HBASE-17149:
---
Status: Patch Available  (was: Reopened)

> Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor 
> multiple times
> ---
>
> Key: HBASE-17149
> URL: https://issues.apache.org/jira/browse/HBASE-17149
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 1.2.4, 1.1.7, 2.0.0, 1.3.0, 1.4.0
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: 17149.branch-1.incomplete.txt, 
> HBASE-17149-addendum.v1-master.patch, HBASE-17149.master.001.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.003.patch, 
> HBASE-17149.v1-branch-1.patch, nonce.patch
>
>
> instead of having all the logic in submitProcedure(), split in 
> registerNonce() + submitProcedure().
> In this case we can avoid calling the coprocessor twice and having a clean 
> submit logic knowing that there will only be one submission.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17379) Lack of synchronization in CompactionPipeline#getScanners()

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783661#comment-15783661
 ] 

Hadoop QA commented on HBASE-17379:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 1s 
{color} | {color:blue} The patch file was not named according to hbase's naming 
conventions. Please see 
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for 
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
27m 18s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 3s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 88m 23s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
16s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 128m 36s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.hbase.master.procedure.TestMasterProcedureWalLease |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12844961/17379.v6.txt |
| JIRA Issue | HBASE-17379 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux c1663e9cee53 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / da97569 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5077/artifact/patchprocess/patch-unit-hbase-server.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-HBASE-Build/5077/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results | 

[jira] [Commented] (HBASE-17149) Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor multiple times

[Stephen Yuan Jiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783659#comment-15783659
 ] 

Stephen Yuan Jiang commented on HBASE-17149:


yeah, every 1.x branch.  (note: branch-1.1 is a little different in 
HMaster.java; but this should be a piece of cake comparing master->branch-1 :-).

> Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor 
> multiple times
> ---
>
> Key: HBASE-17149
> URL: https://issues.apache.org/jira/browse/HBASE-17149
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: 17149.branch-1.incomplete.txt, 
> HBASE-17149-addendum.v1-master.patch, HBASE-17149.master.001.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.003.patch, 
> HBASE-17149.v1-branch-1.patch, nonce.patch
>
>
> instead of having all the logic in submitProcedure(), split in 
> registerNonce() + submitProcedure().
> In this case we can avoid calling the coprocessor twice and having a clean 
> submit logic knowing that there will only be one submission.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16851) User-facing documentation for the In-Memory Compaction feature

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783643#comment-15783643
 ] 

stack commented on HBASE-16851:
---

I like the way this doc reads. The justification flows nicely. Helpful 
diagrams. It is quality.   Enable comments on the doc (I'd suggest) for minor 
edit suggestions. Its great.



> User-facing documentation for the In-Memory Compaction feature
> --
>
> Key: HBASE-16851
> URL: https://issues.apache.org/jira/browse/HBASE-16851
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Edward Bortnikov
>Assignee: Edward Bortnikov
> Attachments: Accordion HBase In-Memory Compaction - Nov 1 .pdf, 
> Accordion HBase In-Memory Compaction - Nov 23.pdf, Accordion_ HBase In-Memory 
> Compaction - Oct 27.pdf, HBaseAcceleratedHbaseConf-final.pptx
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14123) HBase Backup/Restore Phase 2

[Vladimir Rodionov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov updated HBASE-14123:
--
Attachment: 14123.master.v46.patch

v46. Fixed integration test initialization.

> HBase Backup/Restore Phase 2
> 
>
> Key: HBASE-14123
> URL: https://issues.apache.org/jira/browse/HBASE-14123
> Project: HBase
>  Issue Type: Umbrella
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: 14123-master.v14.txt, 14123-master.v15.txt, 
> 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, 
> 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v20.txt, 
> 14123-master.v21.txt, 14123-master.v24.txt, 14123-master.v25.txt, 
> 14123-master.v27.txt, 14123-master.v28.txt, 14123-master.v29.full.txt, 
> 14123-master.v3.txt, 14123-master.v30.txt, 14123-master.v31.txt, 
> 14123-master.v32.txt, 14123-master.v33.txt, 14123-master.v34.txt, 
> 14123-master.v35.txt, 14123-master.v36.txt, 14123-master.v37.txt, 
> 14123-master.v38.txt, 14123-master.v5.txt, 14123-master.v6.txt, 
> 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt, 
> 14123.master.v39.patch, 14123.master.v40.patch, 14123.master.v41.patch, 
> 14123.master.v42.patch, 14123.master.v44.patch, 14123.master.v45.patch, 
> 14123.master.v46.patch, HBASE-14123-for-7912-v1.patch, 
> HBASE-14123-for-7912-v6.patch, HBASE-14123-v1.patch, HBASE-14123-v10.patch, 
> HBASE-14123-v11.patch, HBASE-14123-v12.patch, HBASE-14123-v13.patch, 
> HBASE-14123-v15.patch, HBASE-14123-v16.patch, HBASE-14123-v2.patch, 
> HBASE-14123-v3.patch, HBASE-14123-v4.patch, HBASE-14123-v5.patch, 
> HBASE-14123-v6.patch, HBASE-14123-v7.patch, HBASE-14123-v9.patch
>
>
> Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17336) get/update replication peer config requests should be routed through master

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783614#comment-15783614
 ] 

Enis Soztutar commented on HBASE-17336:
---

There is a copy-paste error below, should call the post method. 
{code}
+  public void postGetReplicationPeerConfig(final String peerId) throws 
IOException {
+execOperation(coprocessors.isEmpty() ? null : new CoprocessorOperation() {
+  @Override
+  public void call(MasterObserver observer, 
ObserverContext ctx)
+  throws IOException {
+observer.preGetReplicationPeerConfig(ctx, peerId);
+  }
+});
+  }
{code}

Same thing in the postUpdateReplicationPeerConfig. 

This is not in this patch, but maybe we should do a follow up patch in the 
parent to move ReplicationPeer and other replication related PB messages to the 
replication.proto from zookeeper.proto. 

In ServerRegionReplicaUtil, you could have also switched to using Admin, rather 
than RA. Not critical for this patch though. Maybe after all methods moved to 
Admin, we can do a refactor patch to convert internal usages from RA to Admin. 

Looks great otherwise. 

> get/update replication peer config requests should be routed through master
> ---
>
> Key: HBASE-17336
> URL: https://issues.apache.org/jira/browse/HBASE-17336
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17336-v1.patch, HBASE-17336-v2.patch, 
> HBASE-17336-v3.patch, HBASE-17336-v4.patch
>
>
> As HBASE-11392 description says, we should move replication operations to be 
> routed through master.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17382) Give RegionLocateType a better name

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783596#comment-15783596
 ] 

Hadoop QA commented on HBASE-17382:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 4 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
25m 15s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 58s 
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 78m 42s 
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
25s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 120m 24s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12844956/HBASE-17382.master.001.patch
 |
| JIRA Issue | HBASE-17382 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 789da80ac36d 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / da97569 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5076/testReport/ |
| modules | C: hbase-client hbase-server U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5076/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Give RegionLocateType a better name
> 

[jira] [Commented] (HBASE-17149) Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor multiple times

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783582#comment-15783582
 ] 

stack commented on HBASE-17149:
---

+1 on patch (I see procedure taking a latch is in branch-1...).

Lets see what hadoopqa says. If all passes, lets commit. Should be in 1.3 and 
1.2?

> Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor 
> multiple times
> ---
>
> Key: HBASE-17149
> URL: https://issues.apache.org/jira/browse/HBASE-17149
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: 17149.branch-1.incomplete.txt, 
> HBASE-17149-addendum.v1-master.patch, HBASE-17149.master.001.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.003.patch, 
> HBASE-17149.v1-branch-1.patch, nonce.patch
>
>
> instead of having all the logic in submitProcedure(), split in 
> registerNonce() + submitProcedure().
> In this case we can avoid calling the coprocessor twice and having a clean 
> submit logic knowing that there will only be one submission.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17339) Scan-Memory-First Optimization for Get Operation

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783549#comment-15783549
 ] 

stack commented on HBASE-17339:
---

Sounds good. The '... is applicable...' in the above is that the table was 
marked that it is a monotonically increasing table? Would be cool if the 
attribute were done in alignment w/ scheme described over in HBASE-10247 
(though there'd be only this option). Thanks [~eshcar]

> Scan-Memory-First Optimization for Get Operation
> 
>
> Key: HBASE-17339
> URL: https://issues.apache.org/jira/browse/HBASE-17339
> Project: HBase
>  Issue Type: Improvement
>Reporter: Eshcar Hillel
> Attachments: HBASE-17339-V01.patch
>
>
> The current implementation of a get operation (to retrieve values for a 
> specific key) scans through all relevant stores of the region; for each store 
> both memory components (memstores segments) and disk components (hfiles) are 
> scanned in parallel.
> We suggest to apply an optimization that speculatively scans memory-only 
> components first and only if the result is incomplete scans both memory and 
> disk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17339) Scan-Memory-First Optimization for Get Operation

[Eshcar Hillel (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783540#comment-15783540
 ] 

Eshcar Hillel commented on HBASE-17339:
---

We can use existing TRT, no need for a new one. Only need to remember the 
maximal flushed TS so far and make sure the range in TRT is beyond this TS.

The protocol including step-0 is for each step:

{code}
 0. check monotonicity in all relevant stores
if monotonicity is maintained and the optimization is applicable (given list of 
columns) then
 1. open all relevant  *memory* scanners
 2. get results
 ONLY if result is not complete then
  3. open all scanners
  4. get results
else
 5. open all scanners
 6. get results
{code}

possible executions are 0-1-2, 0-1-2-3-4, 0-5-6, the operation returns the 
result of step 2 (access only memory) or steps 4/6 (access memory and HFiles)

> Scan-Memory-First Optimization for Get Operation
> 
>
> Key: HBASE-17339
> URL: https://issues.apache.org/jira/browse/HBASE-17339
> Project: HBase
>  Issue Type: Improvement
>Reporter: Eshcar Hillel
> Attachments: HBASE-17339-V01.patch
>
>
> The current implementation of a get operation (to retrieve values for a 
> specific key) scans through all relevant stores of the region; for each store 
> both memory components (memstores segments) and disk components (hfiles) are 
> scanned in parallel.
> We suggest to apply an optimization that speculatively scans memory-only 
> components first and only if the result is incomplete scans both memory and 
> disk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-17339) Scan-Memory-First Optimization for Get Operation

[Eshcar Hillel (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783540#comment-15783540
 ] 

Eshcar Hillel edited comment on HBASE-17339 at 12/28/16 7:19 PM:
-

We can use existing TRT, no need for a new one. Only need to remember the 
maximal flushed TS so far and make sure the range in TRT is beyond this TS.

The protocol including step-0 is for each operation:

{code}
 0. check monotonicity in all relevant stores
if monotonicity is maintained and the optimization is applicable (given list of 
columns) then
 1. open all relevant  *memory* scanners
 2. get results
 ONLY if result is not complete then
  3. open all scanners
  4. get results
else
 5. open all scanners
 6. get results
{code}

possible executions are 0-1-2, 0-1-2-3-4, 0-5-6, the operation returns the 
result of step 2 (access only memory) or steps 4/6 (access memory and HFiles)


was (Author: eshcar):
We can use existing TRT, no need for a new one. Only need to remember the 
maximal flushed TS so far and make sure the range in TRT is beyond this TS.

The protocol including step-0 is for each step:

{code}
 0. check monotonicity in all relevant stores
if monotonicity is maintained and the optimization is applicable (given list of 
columns) then
 1. open all relevant  *memory* scanners
 2. get results
 ONLY if result is not complete then
  3. open all scanners
  4. get results
else
 5. open all scanners
 6. get results
{code}

possible executions are 0-1-2, 0-1-2-3-4, 0-5-6, the operation returns the 
result of step 2 (access only memory) or steps 4/6 (access memory and HFiles)

> Scan-Memory-First Optimization for Get Operation
> 
>
> Key: HBASE-17339
> URL: https://issues.apache.org/jira/browse/HBASE-17339
> Project: HBase
>  Issue Type: Improvement
>Reporter: Eshcar Hillel
> Attachments: HBASE-17339-V01.patch
>
>
> The current implementation of a get operation (to retrieve values for a 
> specific key) scans through all relevant stores of the region; for each store 
> both memory components (memstores segments) and disk components (hfiles) are 
> scanned in parallel.
> We suggest to apply an optimization that speculatively scans memory-only 
> components first and only if the result is incomplete scans both memory and 
> disk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17149) Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor multiple times

[Stephen Yuan Jiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783541#comment-15783541
 ] 

Stephen Yuan Jiang commented on HBASE-17149:


thank you, [~stack]!

> Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor 
> multiple times
> ---
>
> Key: HBASE-17149
> URL: https://issues.apache.org/jira/browse/HBASE-17149
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: 17149.branch-1.incomplete.txt, 
> HBASE-17149-addendum.v1-master.patch, HBASE-17149.master.001.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.003.patch, 
> HBASE-17149.v1-branch-1.patch, nonce.patch
>
>
> instead of having all the logic in submitProcedure(), split in 
> registerNonce() + submitProcedure().
> In this case we can avoid calling the coprocessor twice and having a clean 
> submit logic knowing that there will only be one submission.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-9465) Push entries to peer clusters serially

[Vincent Poon (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530835#comment-15530835
 ] 

Vincent Poon edited comment on HBASE-9465 at 12/28/16 7:12 PM:
---

I don't think you need the code addition above 'continue" here - if 
readAllEntries() returns true, then lastPositionsForSerialScope should always 
be empty?

{code}
if (readAllEntriesToReplicateOrNextFile(currentWALisBeingWrittenTo, entries,
  lastPositionsForSerialScope)) {
for (Map.Entry entry : 
lastPositionsForSerialScope.entrySet()) {
  waitingUntilCanPush(entry);
}
try {
  MetaTableAccessor
  .updateReplicationPositions(manager.getConnection(), 
actualPeerId,
  lastPositionsForSerialScope);
} catch (IOException e) {
  LOG.error("updateReplicationPositions fail", e);
  stopper.stop("updateReplicationPositions fail");
}

continue;
  }
{code}


was (Author: vincentpoon):
I think you need the code addition above 'continue" here - if readAllEntries... 
returns true, then lastPositionsForSerialScope should always be empty?

{code}
if (readAllEntriesToReplicateOrNextFile(currentWALisBeingWrittenTo, entries,
  lastPositionsForSerialScope)) {
for (Map.Entry entry : 
lastPositionsForSerialScope.entrySet()) {
  waitingUntilCanPush(entry);
}
try {
  MetaTableAccessor
  .updateReplicationPositions(manager.getConnection(), 
actualPeerId,
  lastPositionsForSerialScope);
} catch (IOException e) {
  LOG.error("updateReplicationPositions fail", e);
  stopper.stop("updateReplicationPositions fail");
}

continue;
  }
{code}

> Push entries to peer clusters serially
> --
>
> Key: HBASE-9465
> URL: https://issues.apache.org/jira/browse/HBASE-9465
> Project: HBase
>  Issue Type: New Feature
>  Components: regionserver, Replication
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Honghua Feng
>Assignee: Phil Yang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-9465-branch-1-v1.patch, 
> HBASE-9465-branch-1-v1.patch, HBASE-9465-branch-1-v2.patch, 
> HBASE-9465-branch-1-v3.patch, HBASE-9465-branch-1-v4.patch, 
> HBASE-9465-branch-1-v4.patch, HBASE-9465-v1.patch, HBASE-9465-v2.patch, 
> HBASE-9465-v2.patch, HBASE-9465-v3.patch, HBASE-9465-v4.patch, 
> HBASE-9465-v5.patch, HBASE-9465-v6.patch, HBASE-9465-v6.patch, 
> HBASE-9465-v7.patch, HBASE-9465-v7.patch, HBASE-9465.pdf
>
>
> When region-move or RS failure occurs in master cluster, the hlog entries 
> that are not pushed before region-move or RS-failure will be pushed by 
> original RS(for region move) or another RS which takes over the remained hlog 
> of dead RS(for RS failure), and the new entries for the same region(s) will 
> be pushed by the RS which now serves the region(s), but they push the hlog 
> entries of a same region concurrently without coordination.
> This treatment can possibly lead to data inconsistency between master and 
> peer clusters:
> 1. there are put and then delete written to master cluster
> 2. due to region-move / RS-failure, they are pushed by different 
> replication-source threads to peer cluster
> 3. if delete is pushed to peer cluster before put, and flush and 
> major-compact occurs in peer cluster before put is pushed to peer cluster, 
> the delete is collected and the put remains in peer cluster
> In this scenario, the put remains in peer cluster, but in master cluster the 
> put is masked by the delete, hence data inconsistency between master and peer 
> clusters



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17149) Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor multiple times

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783521#comment-15783521
 ] 

stack commented on HBASE-17149:
---

I pushed the nice cleanup addendum to master branch. Thanks [~syuanjiang]

> Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor 
> multiple times
> ---
>
> Key: HBASE-17149
> URL: https://issues.apache.org/jira/browse/HBASE-17149
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: 17149.branch-1.incomplete.txt, 
> HBASE-17149-addendum.v1-master.patch, HBASE-17149.master.001.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.003.patch, 
> HBASE-17149.v1-branch-1.patch, nonce.patch
>
>
> instead of having all the logic in submitProcedure(), split in 
> registerNonce() + submitProcedure().
> In this case we can avoid calling the coprocessor twice and having a clean 
> submit logic knowing that there will only be one submission.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-10247) Client promises about timestamps

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-10247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783508#comment-15783508
 ] 

stack commented on HBASE-10247:
---

For completeness, an obstacle is removed by HBASE-9465 but then a new one 
presents itself (Client reading and writing against two replicating clusters) 
courtesy of [~yangzhe1991] comment at 
https://issues.apache.org/jira/browse/HBASE-17339?focusedCommentId=15780751=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15780751

> Client promises about timestamps
> 
>
> Key: HBASE-10247
> URL: https://issues.apache.org/jira/browse/HBASE-10247
> Project: HBase
>  Issue Type: Brainstorming
>Reporter: Lars Hofhansl
>Priority: Minor
> Attachments: 10247-do-not-try-may-eat-your-first-born-v2.txt, 
> 10247.txt
>
>
> This is to start a discussion about timestamp promises declared per table of 
> CF.
> For example if a client promises only monotonically increasing timestamps (or 
> no custom set timestamps) and VERSIONS=1, we can aggressively and easily 
> remove old versions of the same row/fam/col from the memstore before we 
> flush, just by supplying a comparator that ignores the timestamp (i.e. two KV 
> just differing by TS would be considered equal).
> That would increase the performance of counters significantly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-17081) Flush the entire CompactingMemStore content to disk

[Eshcar Hillel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eshcar Hillel reassigned HBASE-17081:
-

Assignee: Eshcar Hillel  (was: Anastasia Braginsky)

> Flush the entire CompactingMemStore content to disk
> ---
>
> Key: HBASE-17081
> URL: https://issues.apache.org/jira/browse/HBASE-17081
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Eshcar Hillel
> Fix For: 2.0.0
>
> Attachments: HBASE-15787_8.patch, HBASE-17081-V01.patch, 
> HBASE-17081-V02.patch, HBASE-17081-V03.patch, HBASE-17081-V04.patch, 
> HBASE-17081-V05.patch, HBASE-17081-V06.patch, HBASE-17081-V06.patch, 
> HBASE-17081-V07.patch, HBASE-17081-V10.patch, 
> HBaseMeetupDecember2016-V02.pptx, Pipelinememstore_fortrunk_3.patch
>
>
> Part of CompactingMemStore's memory is held by an active segment, and another 
> part is divided between immutable segments in the compacting pipeline. Upon 
> flush-to-disk request we want to flush all of it to disk, in contrast to 
> flushing only tail of the compacting pipeline.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17081) Flush the entire CompactingMemStore content to disk

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783493#comment-15783493
 ] 

Ted Yu commented on HBASE-17081:


Reverted so that we can have ample time figuring how to deal with the 
concurrency issue exposed by TestHRegionWithInMemoryFlush

> Flush the entire CompactingMemStore content to disk
> ---
>
> Key: HBASE-17081
> URL: https://issues.apache.org/jira/browse/HBASE-17081
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Anastasia Braginsky
> Fix For: 2.0.0
>
> Attachments: HBASE-15787_8.patch, HBASE-17081-V01.patch, 
> HBASE-17081-V02.patch, HBASE-17081-V03.patch, HBASE-17081-V04.patch, 
> HBASE-17081-V05.patch, HBASE-17081-V06.patch, HBASE-17081-V06.patch, 
> HBASE-17081-V07.patch, HBASE-17081-V10.patch, 
> HBaseMeetupDecember2016-V02.pptx, Pipelinememstore_fortrunk_3.patch
>
>
> Part of CompactingMemStore's memory is held by an active segment, and another 
> part is divided between immutable segments in the compacting pipeline. Upon 
> flush-to-disk request we want to flush all of it to disk, in contrast to 
> flushing only tail of the compacting pipeline.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17339) Scan-Memory-First Optimization for Get Operation

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783473#comment-15783473
 ] 

stack commented on HBASE-17339:
---

bq. We can add a step-0 where the region queries all relevant stores to 
determine if they all maintain monotonicity and if so apply the optimization. 
Sounds reasonable?

On each Get or once at startup?

bq. As Phil suggested we can use the TimeRangeTracker at each store to 
understand whether or not monotonicity is maintained.

This an existing TRT or a new one (TRT is expensive -- it is our main 
contention point writing... FYI).



> Scan-Memory-First Optimization for Get Operation
> 
>
> Key: HBASE-17339
> URL: https://issues.apache.org/jira/browse/HBASE-17339
> Project: HBase
>  Issue Type: Improvement
>Reporter: Eshcar Hillel
> Attachments: HBASE-17339-V01.patch
>
>
> The current implementation of a get operation (to retrieve values for a 
> specific key) scans through all relevant stores of the region; for each store 
> both memory components (memstores segments) and disk components (hfiles) are 
> scanned in parallel.
> We suggest to apply an optimization that speculatively scans memory-only 
> components first and only if the result is incomplete scans both memory and 
> disk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17149) Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor multiple times

[Stephen Yuan Jiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783449#comment-15783449
 ] 

Stephen Yuan Jiang commented on HBASE-17149:


[~stack], while doing porting the patch to branch-1 from master, I found some 
typo and missed cleanup in the master patch, attached the addendum patch for 
master branch.  Please help review.

> Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor 
> multiple times
> ---
>
> Key: HBASE-17149
> URL: https://issues.apache.org/jira/browse/HBASE-17149
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: 17149.branch-1.incomplete.txt, 
> HBASE-17149-addendum.v1-master.patch, HBASE-17149.master.001.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.003.patch, 
> HBASE-17149.v1-branch-1.patch, nonce.patch
>
>
> instead of having all the logic in submitProcedure(), split in 
> registerNonce() + submitProcedure().
> In this case we can avoid calling the coprocessor twice and having a clean 
> submit logic knowing that there will only be one submission.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-9465) Push entries to peer clusters serially

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783447#comment-15783447
 ] 

stack commented on HBASE-9465:
--

[~yangzhe1991] You see above by [~vincentpoon]?

The design doc is great. I had trouble on this section: "rep_barrier:{seqid}, 
in each time a RS opens a region, it saves the max sequence id in this region. 
info:seqnumDuringOpen also save a seqid and it only save the latest one, but we 
need all." Did we already have a info:seqnumDuringOpen ? How does it relate to  
rep_barrier:{seqid}?  The latter keeps only the latest opening? Could it not 
have been amended to keep all?

bq.  This record is saved after we ship the logs to peer and before we update 
the log position in ZK. 

Do we have the position in two places now? Do we need to update zk if we've 
updated meta?

bq. However, sequence id is not continuous...

Because?  It is not continuous against a peer? Seqid is 'continuous' within a 
region?

bq. ...should not be pushed to this peer until rep_position:{peerid}>=barrier_b 
­- 1.

Why the  -1 in the above? Because we add 1 when we open a region?

bq. There are special cases: region spliting and merging. However, three 
related regions must be in the same region server, so the order of pushing logs 
from parent to daughter can be guaranteed.

We need to write this assumption into the code around where splits bring up 
daughters on same RS as parent. This policy could change (Y! have put up a 
patch to make it so splits do not bring up daughters on same servers as parent 
region).

bq. ...Because we have only one thread for each peer in a RS

This is another assumption of the design that needs to be marked in code so 
when this changes, we'll accommodate the fix here.

bq. Set REPLICATION_SCOPE=2

Not your fault but I was trying to find doc on what REPLICATION_SCOPE meant and 
there seems little. There is some in HConstants. REPLICATION_SCOPE seems to be 
0 for no replication, 1 for replicating to all peers and 2 if we want this 
feature which is all peers but with attempt at guaranteeing sequential play.

We do not write to the hbase:meta state of WALs, unless REPLICATION_SCOPE is 
set to 2?

bq. Because we have only one thread for each peer in a RS, as long as a cf’s 
REPLICATION_SCOPE is 2, all regions’ logs may be delayed but the order is not 
garenteed.

Can you say more on this?

bq. so if an Entry is not ready to push, all logs after it will be blocked.

When would an Entry not be ready to push?

Do we have an idea of how much extra load this fix puts on hbase:meta?

How do we get insight on say delay that is happening because another RS's 
thread is (we think) replaying a WAL?

Thanks for this fixup [~yangzhe1991] Its great.


> Push entries to peer clusters serially
> --
>
> Key: HBASE-9465
> URL: https://issues.apache.org/jira/browse/HBASE-9465
> Project: HBase
>  Issue Type: New Feature
>  Components: regionserver, Replication
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Honghua Feng
>Assignee: Phil Yang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-9465-branch-1-v1.patch, 
> HBASE-9465-branch-1-v1.patch, HBASE-9465-branch-1-v2.patch, 
> HBASE-9465-branch-1-v3.patch, HBASE-9465-branch-1-v4.patch, 
> HBASE-9465-branch-1-v4.patch, HBASE-9465-v1.patch, HBASE-9465-v2.patch, 
> HBASE-9465-v2.patch, HBASE-9465-v3.patch, HBASE-9465-v4.patch, 
> HBASE-9465-v5.patch, HBASE-9465-v6.patch, HBASE-9465-v6.patch, 
> HBASE-9465-v7.patch, HBASE-9465-v7.patch, HBASE-9465.pdf
>
>
> When region-move or RS failure occurs in master cluster, the hlog entries 
> that are not pushed before region-move or RS-failure will be pushed by 
> original RS(for region move) or another RS which takes over the remained hlog 
> of dead RS(for RS failure), and the new entries for the same region(s) will 
> be pushed by the RS which now serves the region(s), but they push the hlog 
> entries of a same region concurrently without coordination.
> This treatment can possibly lead to data inconsistency between master and 
> peer clusters:
> 1. there are put and then delete written to master cluster
> 2. due to region-move / RS-failure, they are pushed by different 
> replication-source threads to peer cluster
> 3. if delete is pushed to peer cluster before put, and flush and 
> major-compact occurs in peer cluster before put is pushed to peer cluster, 
> the delete is collected and the put remains in peer cluster
> In this scenario, the put remains in peer cluster, but in master cluster the 
> put is masked by the delete, hence data inconsistency between master and peer 
> clusters



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17149) Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor multiple times

[Stephen Yuan Jiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Yuan Jiang updated HBASE-17149:
---
Attachment: HBASE-17149-addendum.v1-master.patch

> Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor 
> multiple times
> ---
>
> Key: HBASE-17149
> URL: https://issues.apache.org/jira/browse/HBASE-17149
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: 17149.branch-1.incomplete.txt, 
> HBASE-17149-addendum.v1-master.patch, HBASE-17149.master.001.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.003.patch, 
> HBASE-17149.v1-branch-1.patch, nonce.patch
>
>
> instead of having all the logic in submitProcedure(), split in 
> registerNonce() + submitProcedure().
> In this case we can avoid calling the coprocessor twice and having a clean 
> submit logic knowing that there will only be one submission.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17149) Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor multiple times

[Stephen Yuan Jiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Yuan Jiang updated HBASE-17149:
---
Attachment: HBASE-17149.v1-branch-1.patch

> Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor 
> multiple times
> ---
>
> Key: HBASE-17149
> URL: https://issues.apache.org/jira/browse/HBASE-17149
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: 17149.branch-1.incomplete.txt, 
> HBASE-17149.master.001.patch, HBASE-17149.master.002.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, 
> HBASE-17149.master.003.patch, HBASE-17149.v1-branch-1.patch, nonce.patch
>
>
> instead of having all the logic in submitProcedure(), split in 
> registerNonce() + submitProcedure().
> In this case we can avoid calling the coprocessor twice and having a clean 
> submit logic knowing that there will only be one submission.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17149) Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor multiple times

[Stephen Yuan Jiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Yuan Jiang updated HBASE-17149:
---
Attachment: (was: HBASE-17149.v1-branch-1.patch)

> Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor 
> multiple times
> ---
>
> Key: HBASE-17149
> URL: https://issues.apache.org/jira/browse/HBASE-17149
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: 17149.branch-1.incomplete.txt, 
> HBASE-17149.master.001.patch, HBASE-17149.master.002.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, 
> HBASE-17149.master.003.patch, nonce.patch
>
>
> instead of having all the logic in submitProcedure(), split in 
> registerNonce() + submitProcedure().
> In this case we can avoid calling the coprocessor twice and having a clean 
> submit logic knowing that there will only be one submission.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (HBASE-17149) Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor multiple times

[Stephen Yuan Jiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Yuan Jiang reopened HBASE-17149:


> Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor 
> multiple times
> ---
>
> Key: HBASE-17149
> URL: https://issues.apache.org/jira/browse/HBASE-17149
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: 17149.branch-1.incomplete.txt, 
> HBASE-17149.master.001.patch, HBASE-17149.master.002.patch, 
> HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, 
> HBASE-17149.master.003.patch, nonce.patch
>
>
> instead of having all the logic in submitProcedure(), split in 
> registerNonce() + submitProcedure().
> In this case we can avoid calling the coprocessor twice and having a clean 
> submit logic knowing that there will only be one submission.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17379) Lack of synchronization in CompactionPipeline#getScanners()

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-17379:
---
Attachment: 17379.v6.txt

Patch v6 fixes a bug in patch v5 where pullTail() should have taken the write 
lock.

I understand there is more work (copy-on-write, etc) to be done for future 
patches.

> Lack of synchronization in CompactionPipeline#getScanners()
> ---
>
> Key: HBASE-17379
> URL: https://issues.apache.org/jira/browse/HBASE-17379
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 17379.v1.txt, 17379.v2.txt, 17379.v3.txt, 17379.v4.txt, 
> 17379.v5.txt, 17379.v6.txt
>
>
> From 
> https://builds.apache.org/job/PreCommit-HBASE-Build/5053/testReport/org.apache.hadoop.hbase.regionserver/TestHRegionWithInMemoryFlush/testWritesWhileGetting/
>  :
> {code}
> java.io.IOException: java.util.ConcurrentModificationException
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.handleException(HRegion.java:5886)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.initializeScanners(HRegion.java:5856)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5819)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2786)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2766)
>   at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7036)
>   at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7015)
>   at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6994)
>   at 
> org.apache.hadoop.hbase.regionserver.TestHRegion.testWritesWhileGetting(TestHRegion.java:4141)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.ConcurrentModificationException: null
>   at 
> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
>   at java.util.LinkedList$ListItr.next(LinkedList.java:888)
>   at 
> org.apache.hadoop.hbase.regionserver.CompactionPipeline.getScanners(CompactionPipeline.java:220)
>   at 
> org.apache.hadoop.hbase.regionserver.CompactingMemStore.getScanners(CompactingMemStore.java:298)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore.getScanners(HStore.java:1154)
>   at org.apache.hadoop.hbase.regionserver.Store.getScanners(Store.java:97)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreScanner.getScannersNoCompaction(StoreScanner.java:353)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:210)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:1892)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1880)
>   at 
> 

[jira] [Updated] (HBASE-17382) Give RegionLocateType a better name

[Jan Hentschel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Hentschel updated HBASE-17382:
--
Status: Patch Available  (was: In Progress)

> Give RegionLocateType a better name
> ---
>
> Key: HBASE-17382
> URL: https://issues.apache.org/jira/browse/HBASE-17382
> Project: HBase
>  Issue Type: Improvement
>Reporter: Duo Zhang
>Assignee: Jan Hentschel
> Attachments: HBASE-17382.master.001.patch
>
>
> Pointed out by [~tedyu] that 'Locate' is a verb and usually we need a noun 
> here. 'Locating' or 'Location'?
> Suggestion are welcomed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (HBASE-17382) Give RegionLocateType a better name

[Jan Hentschel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HBASE-17382 started by Jan Hentschel.
-
> Give RegionLocateType a better name
> ---
>
> Key: HBASE-17382
> URL: https://issues.apache.org/jira/browse/HBASE-17382
> Project: HBase
>  Issue Type: Improvement
>Reporter: Duo Zhang
>Assignee: Jan Hentschel
> Attachments: HBASE-17382.master.001.patch
>
>
> Pointed out by [~tedyu] that 'Locate' is a verb and usually we need a noun 
> here. 'Locating' or 'Location'?
> Suggestion are welcomed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (HBASE-17382) Give RegionLocateType a better name

[Jan Hentschel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HBASE-17382 started by Jan Hentschel.
-
> Give RegionLocateType a better name
> ---
>
> Key: HBASE-17382
> URL: https://issues.apache.org/jira/browse/HBASE-17382
> Project: HBase
>  Issue Type: Improvement
>Reporter: Duo Zhang
>Assignee: Jan Hentschel
> Attachments: HBASE-17382.master.001.patch
>
>
> Pointed out by [~tedyu] that 'Locate' is a verb and usually we need a noun 
> here. 'Locating' or 'Location'?
> Suggestion are welcomed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work stopped] (HBASE-17382) Give RegionLocateType a better name

[Jan Hentschel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HBASE-17382 stopped by Jan Hentschel.
-
> Give RegionLocateType a better name
> ---
>
> Key: HBASE-17382
> URL: https://issues.apache.org/jira/browse/HBASE-17382
> Project: HBase
>  Issue Type: Improvement
>Reporter: Duo Zhang
>Assignee: Jan Hentschel
> Attachments: HBASE-17382.master.001.patch
>
>
> Pointed out by [~tedyu] that 'Locate' is a verb and usually we need a noun 
> here. 'Locating' or 'Location'?
> Suggestion are welcomed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17382) Give RegionLocateType a better name

[Jan Hentschel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Hentschel updated HBASE-17382:
--
Attachment: HBASE-17382.master.001.patch

> Give RegionLocateType a better name
> ---
>
> Key: HBASE-17382
> URL: https://issues.apache.org/jira/browse/HBASE-17382
> Project: HBase
>  Issue Type: Improvement
>Reporter: Duo Zhang
>Assignee: Jan Hentschel
> Attachments: HBASE-17382.master.001.patch
>
>
> Pointed out by [~tedyu] that 'Locate' is a verb and usually we need a noun 
> here. 'Locating' or 'Location'?
> Suggestion are welcomed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17379) Lack of synchronization in CompactionPipeline#getScanners()

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783353#comment-15783353
 ] 

stack commented on HBASE-17379:
---

bq. stack I agree we should minimize synchronization. But we do need some 
synchronization to avoid ConcurrentModificationException in such cases.

We have worked with this pattern elsewhere [~eshcar]. Lets be careful adding in 
new locking. Crossing the synchronization may be fast when uncontended but when 
many threads,  it requires coordination which throttles throughput.

In other places we've done copy-on-write or we've done explicit versioned views 
(changes in pipeline happen orders of magnitude less often than gets... we 
should make use of this fact). I can try dig up a soln. that would work here if 
needed if it'd help.

Meantime, let us revert the aggravating patch, HBASE-17081 so you can have the 
weekend free.

> Lack of synchronization in CompactionPipeline#getScanners()
> ---
>
> Key: HBASE-17379
> URL: https://issues.apache.org/jira/browse/HBASE-17379
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 17379.v1.txt, 17379.v2.txt, 17379.v3.txt, 17379.v4.txt, 
> 17379.v5.txt
>
>
> From 
> https://builds.apache.org/job/PreCommit-HBASE-Build/5053/testReport/org.apache.hadoop.hbase.regionserver/TestHRegionWithInMemoryFlush/testWritesWhileGetting/
>  :
> {code}
> java.io.IOException: java.util.ConcurrentModificationException
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.handleException(HRegion.java:5886)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.initializeScanners(HRegion.java:5856)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5819)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2786)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2766)
>   at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7036)
>   at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7015)
>   at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6994)
>   at 
> org.apache.hadoop.hbase.regionserver.TestHRegion.testWritesWhileGetting(TestHRegion.java:4141)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.ConcurrentModificationException: null
>   at 
> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
>   at java.util.LinkedList$ListItr.next(LinkedList.java:888)
>   at 
> org.apache.hadoop.hbase.regionserver.CompactionPipeline.getScanners(CompactionPipeline.java:220)
>   at 
> org.apache.hadoop.hbase.regionserver.CompactingMemStore.getScanners(CompactingMemStore.java:298)
>