[jira] [Resolved] (HBASE-19145) Look into hbase-2 client going to hbase-1 server

2018-08-07 Thread Jerry He (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-19145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He resolved HBASE-19145.
--
Resolution: Done

> Look into hbase-2 client going to hbase-1 server
> 
>
> Key: HBASE-19145
> URL: https://issues.apache.org/jira/browse/HBASE-19145
> Project: HBase
>  Issue Type: Task
>Affects Versions: 2.0.0-beta-1
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Major
>
> From the "[DISCUSS] hbase-2.0.0 compatibility expectations" thread.
> Do we support hbase-2 client going against hbase-1 server?
> We seem to be fine mix-and-match the clients and servers within the
> hbase-1 releases.   And IIRC hbase-1 client is ok against 0.98 server.
> Suppose I have a product  that depends and bundles HBase client. I
> want to upgrade the dependency to hbase-2 so that it can take
> advantages of and claim support of hbase-2.
> But does it mean that I will need drop the claims that the new version
> of the product support any hbase-1 backend?
> It has not been an objective. It might work doing basic Client API on a
> later branch-1 but will fail doing Admin functions (and figuring if a Table
> is online).  If it was a small thing to make it
> work, lets get it in.
> Let's look into it to see what works and what not.  Have a statement at least.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-19145) Look into hbase-2 client going to hbase-1 server

2018-08-07 Thread Jerry He (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-19145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572648#comment-16572648
 ] 

Jerry He commented on HBASE-19145:
--

Trying to put some result here with the latest HBase 2.1.0 client going to 
1.4.6 server.

Simple put, delete, scan work ok.

But some of the Admin APIs will fail.  The main reason is in HBase 2.x, the 
table state is kept in meta. Therefore 2.x client will always go ask for table 
state from meta on the server, but 1.x server does not have it in meta.
{code:java}
org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family 
table does not exist in region hbase:meta,,1.1588230740 in table 'hbase:meta', 
{TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => 
'|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, 
{NAME => 'info', BLOOMFILTER => 'NONE', VERSIONS => '3', IN_MEMORY => 'true', 
KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', 
COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'true', MIN_VERSIONS => '0', 
BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}
at org.apache.hadoop.hbase.regionserver.HRegion.checkFamily(HRegion.java:8298)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7306)
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2259)
at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:36609)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2354){code}
See HBASE-12035.

Any API asking for the status of the table will fail: disable, enable, flush, 
alter, exists, clone_snapshot, etc.

It is not an extensive testing.  There does not seem to be a need or 
requirement to make it work. So I am closing this task for now.

> Look into hbase-2 client going to hbase-1 server
> 
>
> Key: HBASE-19145
> URL: https://issues.apache.org/jira/browse/HBASE-19145
> Project: HBase
>  Issue Type: Task
>Affects Versions: 2.0.0-beta-1
>Reporter: Jerry He
>Priority: Major
>
> From the "[DISCUSS] hbase-2.0.0 compatibility expectations" thread.
> Do we support hbase-2 client going against hbase-1 server?
> We seem to be fine mix-and-match the clients and servers within the
> hbase-1 releases.   And IIRC hbase-1 client is ok against 0.98 server.
> Suppose I have a product  that depends and bundles HBase client. I
> want to upgrade the dependency to hbase-2 so that it can take
> advantages of and claim support of hbase-2.
> But does it mean that I will need drop the claims that the new version
> of the product support any hbase-1 backend?
> It has not been an objective. It might work doing basic Client API on a
> later branch-1 but will fail doing Admin functions (and figuring if a Table
> is online).  If it was a small thing to make it
> work, lets get it in.
> Let's look into it to see what works and what not.  Have a statement at least.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (HBASE-19145) Look into hbase-2 client going to hbase-1 server

2018-08-07 Thread Jerry He (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-19145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He reassigned HBASE-19145:


Assignee: Jerry He

> Look into hbase-2 client going to hbase-1 server
> 
>
> Key: HBASE-19145
> URL: https://issues.apache.org/jira/browse/HBASE-19145
> Project: HBase
>  Issue Type: Task
>Affects Versions: 2.0.0-beta-1
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Major
>
> From the "[DISCUSS] hbase-2.0.0 compatibility expectations" thread.
> Do we support hbase-2 client going against hbase-1 server?
> We seem to be fine mix-and-match the clients and servers within the
> hbase-1 releases.   And IIRC hbase-1 client is ok against 0.98 server.
> Suppose I have a product  that depends and bundles HBase client. I
> want to upgrade the dependency to hbase-2 so that it can take
> advantages of and claim support of hbase-2.
> But does it mean that I will need drop the claims that the new version
> of the product support any hbase-1 backend?
> It has not been an objective. It might work doing basic Client API on a
> later branch-1 but will fail doing Admin functions (and figuring if a Table
> is online).  If it was a small thing to make it
> work, lets get it in.
> Let's look into it to see what works and what not.  Have a statement at least.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-21008) HBase 1.x can not read HBase2 hfiles due to TimeRangeTracker

2018-08-05 Thread Jerry He (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569741#comment-16569741
 ] 

Jerry He commented on HBASE-21008:
--

I had the same question for you :) 

But go ahead. You will be faster than me. Thanks for the quick response on this 
issue!

> HBase 1.x can not read HBase2 hfiles due to TimeRangeTracker
> 
>
> Key: HBASE-21008
> URL: https://issues.apache.org/jira/browse/HBASE-21008
> Project: HBase
>  Issue Type: Bug
>  Components: compatibility, HFile
>Affects Versions: 2.1.0, 1.4.6
>Reporter: Jerry He
>Priority: Critical
>
> It looks like HBase 1.x can not open hfiiles written by HBase2 still.
> I tested the latest HBase 1.4.6 and 2.1.0.  1.4.6 tried to read and open 
> regions written by 2.1.0.
> {code}
> 2018-07-30 16:01:31,274 ERROR [StoreFileOpenerThread-info-1] 
> regionserver.StoreFile: Error reading timestamp range data from meta -- 
> proceeding without
> java.lang.IllegalArgumentException: Timestamp cannot be negative. 
> minStamp:5783278630776778969, maxStamp:-4698050386518222402
> at org.apache.hadoop.hbase.io.TimeRange.check(TimeRange.java:112)
> at org.apache.hadoop.hbase.io.TimeRange.(TimeRange.java:100)
> at 
> org.apache.hadoop.hbase.regionserver.TimeRangeTracker.toTimeRange(TimeRangeTracker.java:214)
> at 
> org.apache.hadoop.hbase.regionserver.TimeRangeTracker.getTimeRange(TimeRangeTracker.java:198)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:507)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:531)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:521)
> at 
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:679)
> at 
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:122)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:538)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:535)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> {code}
> Or:
> {code}
> 2018-07-30 16:01:31,305 ERROR [RS_OPEN_REGION-throb1:34004-0] 
> handler.OpenRegionHandler: Failed open of 
> region=janusgraph,,1532630557542.b0fa15cb0bf1b0bf740997b7056c., starting 
> to roll back the global memstore size.
> java.io.IOException: java.io.IOException: java.io.EOFException
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initializeStores(HRegion.java:1033)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:908)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:876)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6995)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6956)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6927)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6883)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6834)
> at 
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:364)
> at 
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:131)
> at 
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: java.io.EOFException
> at 
> org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(HStore.java:564)
> at 
> org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:518)
> at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:281)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:5378)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1007)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1004)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> Caused by: java.io.EOFException
> at 

[jira] [Commented] (HBASE-21008) HBase 1.x can not read HBase2 hfiles due to TimeRangeTracker

2018-08-03 Thread Jerry He (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569078#comment-16569078
 ] 

Jerry He commented on HBASE-21008:
--

This is good with me!

> HBase 1.x can not read HBase2 hfiles due to TimeRangeTracker
> 
>
> Key: HBASE-21008
> URL: https://issues.apache.org/jira/browse/HBASE-21008
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.1.0, 1.4.6
>Reporter: Jerry He
>Priority: Major
>
> It looks like HBase 1.x can not open hfiiles written by HBase2 still.
> I tested the latest HBase 1.4.6 and 2.1.0.  1.4.6 tried to read and open 
> regions written by 2.1.0.
> {code}
> 2018-07-30 16:01:31,274 ERROR [StoreFileOpenerThread-info-1] 
> regionserver.StoreFile: Error reading timestamp range data from meta -- 
> proceeding without
> java.lang.IllegalArgumentException: Timestamp cannot be negative. 
> minStamp:5783278630776778969, maxStamp:-4698050386518222402
> at org.apache.hadoop.hbase.io.TimeRange.check(TimeRange.java:112)
> at org.apache.hadoop.hbase.io.TimeRange.(TimeRange.java:100)
> at 
> org.apache.hadoop.hbase.regionserver.TimeRangeTracker.toTimeRange(TimeRangeTracker.java:214)
> at 
> org.apache.hadoop.hbase.regionserver.TimeRangeTracker.getTimeRange(TimeRangeTracker.java:198)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:507)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:531)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:521)
> at 
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:679)
> at 
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:122)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:538)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:535)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> {code}
> Or:
> {code}
> 2018-07-30 16:01:31,305 ERROR [RS_OPEN_REGION-throb1:34004-0] 
> handler.OpenRegionHandler: Failed open of 
> region=janusgraph,,1532630557542.b0fa15cb0bf1b0bf740997b7056c., starting 
> to roll back the global memstore size.
> java.io.IOException: java.io.IOException: java.io.EOFException
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initializeStores(HRegion.java:1033)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:908)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:876)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6995)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6956)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6927)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6883)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6834)
> at 
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:364)
> at 
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:131)
> at 
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: java.io.EOFException
> at 
> org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(HStore.java:564)
> at 
> org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:518)
> at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:281)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:5378)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1007)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1004)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:197)
> at java.io.DataInputStream.readLong(DataInputStream.java:416)
> at 
> 

[jira] [Commented] (HBASE-21008) HBase 1.x can not read HBase2 hfiles due to TimeRangeTracker

2018-08-03 Thread Jerry He (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569045#comment-16569045
 ] 

Jerry He commented on HBASE-21008:
--

{quote}Perhaps we can backport a part of HBASE-18754 to all active 1.x branch 
in order to make them "can" read the hfiles generated by 2.x
{quote}
Yeah, only the 'read' part needs to be put in 1.x. This approach is similarly 
used in HBASE-16189 and HBASE-19052. However, in HBASE-19116, [~stack] made 
changes in 2.x so that 1.x deployment no longer needs to upgrade to the latest 
1.x to work.  What is preferred here?

> HBase 1.x can not read HBase2 hfiles due to TimeRangeTracker
> 
>
> Key: HBASE-21008
> URL: https://issues.apache.org/jira/browse/HBASE-21008
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.1.0, 1.4.6
>Reporter: Jerry He
>Priority: Major
>
> It looks like HBase 1.x can not open hfiiles written by HBase2 still.
> I tested the latest HBase 1.4.6 and 2.1.0.  1.4.6 tried to read and open 
> regions written by 2.1.0.
> {code}
> 2018-07-30 16:01:31,274 ERROR [StoreFileOpenerThread-info-1] 
> regionserver.StoreFile: Error reading timestamp range data from meta -- 
> proceeding without
> java.lang.IllegalArgumentException: Timestamp cannot be negative. 
> minStamp:5783278630776778969, maxStamp:-4698050386518222402
> at org.apache.hadoop.hbase.io.TimeRange.check(TimeRange.java:112)
> at org.apache.hadoop.hbase.io.TimeRange.(TimeRange.java:100)
> at 
> org.apache.hadoop.hbase.regionserver.TimeRangeTracker.toTimeRange(TimeRangeTracker.java:214)
> at 
> org.apache.hadoop.hbase.regionserver.TimeRangeTracker.getTimeRange(TimeRangeTracker.java:198)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:507)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:531)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:521)
> at 
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:679)
> at 
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:122)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:538)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:535)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> {code}
> Or:
> {code}
> 2018-07-30 16:01:31,305 ERROR [RS_OPEN_REGION-throb1:34004-0] 
> handler.OpenRegionHandler: Failed open of 
> region=janusgraph,,1532630557542.b0fa15cb0bf1b0bf740997b7056c., starting 
> to roll back the global memstore size.
> java.io.IOException: java.io.IOException: java.io.EOFException
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initializeStores(HRegion.java:1033)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:908)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:876)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6995)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6956)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6927)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6883)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6834)
> at 
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:364)
> at 
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:131)
> at 
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: java.io.EOFException
> at 
> org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(HStore.java:564)
> at 
> org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:518)
> at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:281)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:5378)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1007)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1004)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> 

[jira] [Commented] (HBASE-21008) HBase 1.x can not read HBase2 hfiles due to TimeRangeTracker

2018-08-03 Thread Jerry He (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568532#comment-16568532
 ] 

Jerry He commented on HBASE-21008:
--

FYI [~chia7712].

> HBase 1.x can not read HBase2 hfiles due to TimeRangeTracker
> 
>
> Key: HBASE-21008
> URL: https://issues.apache.org/jira/browse/HBASE-21008
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.1.0, 1.4.6
>Reporter: Jerry He
>Priority: Major
>
> It looks like HBase 1.x can not open hfiiles written by HBase2 still.
> I tested the latest HBase 1.4.6 and 2.1.0.  1.4.6 tried to read and open 
> regions written by 2.1.0.
> {code}
> 2018-07-30 16:01:31,274 ERROR [StoreFileOpenerThread-info-1] 
> regionserver.StoreFile: Error reading timestamp range data from meta -- 
> proceeding without
> java.lang.IllegalArgumentException: Timestamp cannot be negative. 
> minStamp:5783278630776778969, maxStamp:-4698050386518222402
> at org.apache.hadoop.hbase.io.TimeRange.check(TimeRange.java:112)
> at org.apache.hadoop.hbase.io.TimeRange.(TimeRange.java:100)
> at 
> org.apache.hadoop.hbase.regionserver.TimeRangeTracker.toTimeRange(TimeRangeTracker.java:214)
> at 
> org.apache.hadoop.hbase.regionserver.TimeRangeTracker.getTimeRange(TimeRangeTracker.java:198)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:507)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:531)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:521)
> at 
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:679)
> at 
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:122)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:538)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:535)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> {code}
> Or:
> {code}
> 2018-07-30 16:01:31,305 ERROR [RS_OPEN_REGION-throb1:34004-0] 
> handler.OpenRegionHandler: Failed open of 
> region=janusgraph,,1532630557542.b0fa15cb0bf1b0bf740997b7056c., starting 
> to roll back the global memstore size.
> java.io.IOException: java.io.IOException: java.io.EOFException
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initializeStores(HRegion.java:1033)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:908)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:876)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6995)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6956)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6927)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6883)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6834)
> at 
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:364)
> at 
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:131)
> at 
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: java.io.EOFException
> at 
> org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(HStore.java:564)
> at 
> org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:518)
> at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:281)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:5378)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1007)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1004)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:197)
> at java.io.DataInputStream.readLong(DataInputStream.java:416)
> at 
> 

[jira] [Commented] (HBASE-21008) HBase 1.x can not read HBase2 hfiles due to TimeRangeTracker

2018-08-03 Thread Jerry He (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-21008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568513#comment-16568513
 ] 

Jerry He commented on HBASE-21008:
--

The problem seems to come from HBASE-18754, which removed the TimeRangeTracker 
Writable, but added a protobuf HBaseProtos.TimeRangeTracker. HBase 1.x will not 
be able to read the protobuf serialized TimeRangeTracker in hfiles.

> HBase 1.x can not read HBase2 hfiles due to TimeRangeTracker
> 
>
> Key: HBASE-21008
> URL: https://issues.apache.org/jira/browse/HBASE-21008
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.1.0, 1.4.6
>Reporter: Jerry He
>Priority: Major
>
> It looks like HBase 1.x can not open hfiiles written by HBase2 still.
> I tested the latest HBase 1.4.6 and 2.1.0.  1.4.6 tried to read and open 
> regions written by 2.1.0.
> {code}
> 2018-07-30 16:01:31,274 ERROR [StoreFileOpenerThread-info-1] 
> regionserver.StoreFile: Error reading timestamp range data from meta -- 
> proceeding without
> java.lang.IllegalArgumentException: Timestamp cannot be negative. 
> minStamp:5783278630776778969, maxStamp:-4698050386518222402
> at org.apache.hadoop.hbase.io.TimeRange.check(TimeRange.java:112)
> at org.apache.hadoop.hbase.io.TimeRange.(TimeRange.java:100)
> at 
> org.apache.hadoop.hbase.regionserver.TimeRangeTracker.toTimeRange(TimeRangeTracker.java:214)
> at 
> org.apache.hadoop.hbase.regionserver.TimeRangeTracker.getTimeRange(TimeRangeTracker.java:198)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:507)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:531)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:521)
> at 
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:679)
> at 
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:122)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:538)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:535)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> {code}
> Or:
> {code}
> 2018-07-30 16:01:31,305 ERROR [RS_OPEN_REGION-throb1:34004-0] 
> handler.OpenRegionHandler: Failed open of 
> region=janusgraph,,1532630557542.b0fa15cb0bf1b0bf740997b7056c., starting 
> to roll back the global memstore size.
> java.io.IOException: java.io.IOException: java.io.EOFException
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initializeStores(HRegion.java:1033)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:908)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:876)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6995)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6956)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6927)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6883)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6834)
> at 
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:364)
> at 
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:131)
> at 
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: java.io.EOFException
> at 
> org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(HStore.java:564)
> at 
> org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:518)
> at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:281)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:5378)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1007)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1004)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> Caused by: java.io.EOFException
>

[jira] [Created] (HBASE-21008) HBase 1.x can not read HBase2 hfiles due to TimeRangeTracker

2018-08-03 Thread Jerry He (JIRA)
Jerry He created HBASE-21008:


 Summary: HBase 1.x can not read HBase2 hfiles due to 
TimeRangeTracker
 Key: HBASE-21008
 URL: https://issues.apache.org/jira/browse/HBASE-21008
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.4.6, 2.1.0
Reporter: Jerry He


It looks like HBase 1.x can not open hfiiles written by HBase2 still.

I tested the latest HBase 1.4.6 and 2.1.0.  1.4.6 tried to read and open 
regions written by 2.1.0.

{code}
2018-07-30 16:01:31,274 ERROR [StoreFileOpenerThread-info-1] 
regionserver.StoreFile: Error reading timestamp range data from meta -- 
proceeding without
java.lang.IllegalArgumentException: Timestamp cannot be negative. 
minStamp:5783278630776778969, maxStamp:-4698050386518222402
at org.apache.hadoop.hbase.io.TimeRange.check(TimeRange.java:112)
at org.apache.hadoop.hbase.io.TimeRange.(TimeRange.java:100)
at 
org.apache.hadoop.hbase.regionserver.TimeRangeTracker.toTimeRange(TimeRangeTracker.java:214)
at 
org.apache.hadoop.hbase.regionserver.TimeRangeTracker.getTimeRange(TimeRangeTracker.java:198)
at 
org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:507)
at 
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:531)
at 
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:521)
at 
org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:679)
at 
org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:122)
at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:538)
at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:535)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
{code}
Or:
{code}
2018-07-30 16:01:31,305 ERROR [RS_OPEN_REGION-throb1:34004-0] 
handler.OpenRegionHandler: Failed open of 
region=janusgraph,,1532630557542.b0fa15cb0bf1b0bf740997b7056c., starting to 
roll back the global memstore size.
java.io.IOException: java.io.IOException: java.io.EOFException
at 
org.apache.hadoop.hbase.regionserver.HRegion.initializeStores(HRegion.java:1033)
at 
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:908)
at 
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:876)
at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6995)
at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6956)
at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6927)
at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6883)
at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6834)
at 
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:364)
at 
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:131)
at 
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: java.io.EOFException
at 
org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(HStore.java:564)
at 
org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:518)
at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:281)
at 
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:5378)
at 
org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1007)
at 
org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1004)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:197)
at java.io.DataInputStream.readLong(DataInputStream.java:416)
at 
org.apache.hadoop.hbase.regionserver.TimeRangeTracker.readFields(TimeRangeTracker.java:170)
at 
org.apache.hadoop.hbase.util.Writables.copyWritable(Writables.java:161)
at 
org.apache.hadoop.hbase.regionserver.TimeRangeTracker.getTimeRangeTracker(TimeRangeTracker.java:187)
at 
org.apache.hadoop.hbase.regionserver.TimeRangeTracker.getTimeRange(TimeRangeTracker.java:197)
at 
org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:507)
at 

[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4

2018-07-20 Thread Jerry He (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550281#comment-16550281
 ] 

Jerry He commented on HBASE-20565:
--

{quote}place the count-related filters at the last position
{quote}
{quote}ColumnPaginationFilter is order dependence filter
{quote}
Makes sense. I would think people use it last anyways. Thanks for the fix and 
explanation.

+1

> ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect 
> result since 1.4
> -
>
> Key: HBASE-20565
> URL: https://issues.apache.org/jira/browse/HBASE-20565
> Project: HBase
>  Issue Type: Bug
>  Components: Filters
>Affects Versions: 2.1.0, 1.4.4, 2.0.1
>Reporter: Jerry He
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 2.1.0, 1.5.0, 1.4.6, 2.0.2
>
> Attachments: HBASE-20565.v1.patch, HBASE-20565.v2.patch, debug.diff, 
> debug.log, test-branch-1.4.patch
>
>
> When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see 
> incorrect result.
> Here is a simple example.
> One row with 10 columns c0, c1, c2, .., c9.  I have a ColumnRangeFilter for 
> range c2 to c9.  Then I have a ColumnPaginationFilter with limit 5 and offset 
> 0.  FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, 
> ColumnPaginationFilter).
> We expect 5 columns being returned.  But in HBase 1.4 and after, 4 columns 
> are returned.
> In 1.2.x, the correct 5 columns are returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4

2018-07-15 Thread Jerry He (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544805#comment-16544805
 ] 

Jerry He commented on HBASE-20565:
--

[~openinx] Thanks for the update.
I can twist the application to use ColumnRangeFilter (startColumn, endColumn), 
and ColumnPaginationFilter(limit, column-name-offset). It is ok to do it, and 
easy when starting from the beginning since column-name-offset is the 
startColumn.  But then we will have to know the last retrieved column name to 
set the next round.

> ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect 
> result since 1.4
> -
>
> Key: HBASE-20565
> URL: https://issues.apache.org/jira/browse/HBASE-20565
> Project: HBase
>  Issue Type: Bug
>  Components: Filters
>Affects Versions: 1.4.4
>Reporter: Jerry He
>Assignee: Zheng Hu
>Priority: Major
> Attachments: debug.diff, debug.log, test-branch-1.4.patch
>
>
> When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see 
> incorrect result.
> Here is a simple example.
> One row with 10 columns c0, c1, c2, .., c9.  I have a ColumnRangeFilter for 
> range c2 to c9.  Then I have a ColumnPaginationFilter with limit 5 and offset 
> 0.  FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, 
> ColumnPaginationFilter).
> We expect 5 columns being returned.  But in HBase 1.4 and after, 4 columns 
> are returned.
> In 1.2.x, the correct 5 columns are returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4

2018-07-12 Thread Jerry He (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541187#comment-16541187
 ] 

Jerry He commented on HBASE-20565:
--

[~openinx] Is there any good news on this one?  I think we need to fix it.  
Thanks.

> ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect 
> result since 1.4
> -
>
> Key: HBASE-20565
> URL: https://issues.apache.org/jira/browse/HBASE-20565
> Project: HBase
>  Issue Type: Bug
>  Components: Filters
>Affects Versions: 1.4.4
>Reporter: Jerry He
>Assignee: Zheng Hu
>Priority: Major
> Attachments: debug.diff, debug.log, test-branch-1.4.patch
>
>
> When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see 
> incorrect result.
> Here is a simple example.
> One row with 10 columns c0, c1, c2, .., c9.  I have a ColumnRangeFilter for 
> range c2 to c9.  Then I have a ColumnPaginationFilter with limit 5 and offset 
> 0.  FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, 
> ColumnPaginationFilter).
> We expect 5 columns being returned.  But in HBase 1.4 and after, 4 columns 
> are returned.
> In 1.2.x, the correct 5 columns are returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4

2018-05-11 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472775#comment-16472775
 ] 

Jerry He commented on HBASE-20565:
--

You can see the results from branch-1.2 makes more sense and have correctly 
identify the ColumnPaginationFilter's offset of either 0 or 1 relative to the 
ColumnRangeFilter's range.

> ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect 
> result since 1.4
> -
>
> Key: HBASE-20565
> URL: https://issues.apache.org/jira/browse/HBASE-20565
> Project: HBase
>  Issue Type: Bug
>  Components: Filters
>Affects Versions: 1.4.4
>Reporter: Jerry He
>Assignee: Zheng Hu
>Priority: Major
> Attachments: debug.diff, debug.log, test-branch-1.4.patch
>
>
> When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see 
> incorrect result.
> Here is a simple example.
> One row with 10 columns c0, c1, c2, .., c9.  I have a ColumnRangeFilter for 
> range c2 to c9.  Then I have a ColumnPaginationFilter with limit 5 and offset 
> 0.  FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, 
> ColumnPaginationFilter).
> We expect 5 columns being returned.  But in HBase 1.4 and after, 4 columns 
> are returned.
> In 1.2.x, the correct 5 columns are returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4

2018-05-11 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472774#comment-16472774
 ] 

Jerry He commented on HBASE-20565:
--

Hi, [~openinx] Thanks for taking a look!

i did a little more testing with the test patch.  Here is what I got:

Branch 1.4:
---
ColumnRangeFilter StringRange(“1”, true, "9", false), ColumnPaginationFilter(5, 
0)
Result:  0, 1, 2, 3, 4 

StringRange(“1”, true, "9", false), ColumnPaginationFilter(5, 0)
Result:  1, 2, 3, 4 

StringRange(“3”, true, "9", false), ColumnPaginationFilter(5, 0)
Result:  3, 4, 5, 6

StringRange(“0”, true, "9", false), ColumnPaginationFilter(5, 1)
Result:  1, 2, 3, 4, 5 

StringRange(“1”, true, "9", false), ColumnPaginationFilter(5, 1)
Result:  1, 2, 3, 4 , 5

StringRange(“3”, true, "9", false), ColumnPaginationFilter(5, 1)
Result:  3, 4, 5, 6, 7

Branch 1.2
---
ColumnRangeFilter StringRange(“0”, true, "9", false), ColumnPaginationFilter(5, 
0)
Result:  0, 1, 2, 3, 4 

StringRange(“1”, true, "9", false), ColumnPaginationFilter(5, 0)
Result:  1, 2, 3, 4, 5

StringRange(“3”, true, "9", false), ColumnPaginationFilter(5, 0)
Result:  3, 4, 5, 6, 7

StringRange(“0”, true, "9", false), ColumnPaginationFilter(5, 1)
Result:  1, 2, 3, 4, 5 

StringRange(“1”, true, "9", false), ColumnPaginationFilter(5, 1)
Result:  2, 3, 4 , 5, 6

StringRange(“3”, true, "9", false), ColumnPaginationFilter(5, 1)
Result:  4, 5, 6, 7, 8

> ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect 
> result since 1.4
> -
>
> Key: HBASE-20565
> URL: https://issues.apache.org/jira/browse/HBASE-20565
> Project: HBase
>  Issue Type: Bug
>  Components: Filters
>Affects Versions: 1.4.4
>Reporter: Jerry He
>Assignee: Zheng Hu
>Priority: Major
> Attachments: debug.diff, debug.log, test-branch-1.4.patch
>
>
> When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see 
> incorrect result.
> Here is a simple example.
> One row with 10 columns c0, c1, c2, .., c9.  I have a ColumnRangeFilter for 
> range c2 to c9.  Then I have a ColumnPaginationFilter with limit 5 and offset 
> 0.  FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, 
> ColumnPaginationFilter).
> We expect 5 columns being returned.  But in HBase 1.4 and after, 4 columns 
> are returned.
> In 1.2.x, the correct 5 columns are returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4

2018-05-10 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471024#comment-16471024
 ] 

Jerry He commented on HBASE-20565:
--

Some research shows it is caused by HBASE-18993. 
FYI . [~openinx], [~apurtell]

> ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect 
> result since 1.4
> -
>
> Key: HBASE-20565
> URL: https://issues.apache.org/jira/browse/HBASE-20565
> Project: HBase
>  Issue Type: Bug
>  Components: Filters
>Affects Versions: 1.4.4
>Reporter: Jerry He
>Priority: Major
> Attachments: test-branch-1.4.patch
>
>
> When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see 
> incorrect result.
> Here is a simple example.
> One row with 10 columns c0, c1, c2, .., c9.  I have a ColumnRangeFilter for 
> range c2 to c9.  Then I have a ColumnPaginationFilter with limit 5 and offset 
> 0.  FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, 
> ColumnPaginationFilter).
> We expect 5 columns being returned.  But in HBase 1.4 and after, 4 columns 
> are returned.
> In 1.2.x, the correct 5 columns are returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4

2018-05-10 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471010#comment-16471010
 ] 

Jerry He commented on HBASE-20565:
--

I attached a test case that can be used to re-create the problem.  The test 
passes in branch 1.2, but fails in 1.4 and later.

> ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect 
> result since 1.4
> -
>
> Key: HBASE-20565
> URL: https://issues.apache.org/jira/browse/HBASE-20565
> Project: HBase
>  Issue Type: Bug
>  Components: Filters
>Affects Versions: 1.4.4
>Reporter: Jerry He
>Priority: Major
> Attachments: test-branch-1.4.patch
>
>
> When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see 
> incorrect result.
> Here is a simple example.
> One row with 10 columns c0, c1, c2, .., c9.  I have a ColumnRangeFilter for 
> range c2 to c9.  Then I have a ColumnPaginationFilter with limit 5 and offset 
> 0.  FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, 
> ColumnPaginationFilter).
> We expect 5 columns being returned.  But in HBase 1.4 and after, 4 columns 
> are returned.
> In 1.2.x, the correct 5 columns are returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4

2018-05-10 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-20565:
-
Attachment: test-branch-1.4.patch

> ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect 
> result since 1.4
> -
>
> Key: HBASE-20565
> URL: https://issues.apache.org/jira/browse/HBASE-20565
> Project: HBase
>  Issue Type: Bug
>  Components: Filters
>Affects Versions: 1.4.4
>Reporter: Jerry He
>Priority: Major
> Attachments: test-branch-1.4.patch
>
>
> When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see 
> incorrect result.
> Here is a simple example.
> One row with 10 columns c0, c1, c2, .., c9.  I have a ColumnRangeFilter for 
> range c2 to c9.  Then I have a ColumnPaginationFilter with limit 5 and offset 
> 0.  FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, 
> ColumnPaginationFilter).
> We expect 5 columns being returned.  But in HBase 1.4 and after, 4 columns 
> are returned.
> In 1.2.x, the correct 5 columns are returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4

2018-05-10 Thread Jerry He (JIRA)
Jerry He created HBASE-20565:


 Summary: ColumnRangeFilter combined with ColumnPaginationFilter 
can produce incorrect result since 1.4
 Key: HBASE-20565
 URL: https://issues.apache.org/jira/browse/HBASE-20565
 Project: HBase
  Issue Type: Bug
  Components: Filters
Affects Versions: 1.4.4
Reporter: Jerry He


When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see 
incorrect result.

Here is a simple example.

One row with 10 columns c0, c1, c2, .., c9.  I have a ColumnRangeFilter for 
range c2 to c9.  Then I have a ColumnPaginationFilter with limit 5 and offset 
0.  FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, 
ColumnPaginationFilter).
We expect 5 columns being returned.  But in HBase 1.4 and after, 4 columns are 
returned.
In 1.2.x, the correct 5 columns are returned.





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-19557) Build and release source jars for hbase-shaded-client and others

2018-03-29 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-19557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419860#comment-16419860
 ] 

Jerry He commented on HBASE-19557:
--

[~busbey] They are empty with only meta info in the artifacts, IIRC.

> Build and release source jars for hbase-shaded-client and others
> 
>
> Key: HBASE-19557
> URL: https://issues.apache.org/jira/browse/HBASE-19557
> Project: HBase
>  Issue Type: Sub-task
>  Components: shading
>Affects Versions: 1.3.1, 1.2.6
>Reporter: Jerry He
>Priority: Major
>
> It seems that currently we don't build and release source jars for 
> hbase-shaded-client (and server or mapreduce).  IDEs will complain from the 
> dependent users. We should provide them.
> http://central.maven.org/maven2/org/apache/hbase/hbase-shaded-client/1.3.1/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout

2017-12-20 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299280#comment-16299280
 ] 

Jerry He commented on HBASE-19486:
--

I try to get a better feel for the word 'linger' to see if it fits better here.
Still like {{write buffer periodic flush interva}} better at the moment to fit 
into hbase.
Thanks for the effort to explain it.

> Automatically flush BufferedMutator after a timeout 
> 
>
> Key: HBASE-19486
> URL: https://issues.apache.org/jira/browse/HBASE-19486
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Reporter: Niels Basjes
>Assignee: Niels Basjes
> Attachments: HBASE-19486-20171212-2117.patch, 
> HBASE-19486-20171218-1229.patch, HBASE-19486-20171218-1300.patch, 
> HBASE-19486-20171219-0933.patch, HBASE-19486-20171219-1026.patch, 
> HBASE-19486-20171219-1122-trigger-qa-run.patch, 
> HBASE-19486-20171220-1612-trigger-qa-run.patch, 
> HBASE-19486-20171220-2228-trigger-qa-run.patch
>
>
> I'm working on several projects where we are doing stream / event type 
> processing instead of batch type processing. We mostly use Apache Flink and 
> Apache Beam for these projects.
> When we ingest a continuous stream of events and feed that into HBase via a 
> BufferedMutator this all works fine. The buffer fills up at a predictable 
> rate and we can make sure it flushes several times per second into HBase by 
> tuning the buffer size.
> We also have situations where the event rate is unpredictable. Some times 
> because the source is in reality a batch job that puts records into Kafka, 
> sometimes because it is the "predictable in production" application in our 
> testing environment (where only the dev triggers a handful of events).
> For these kinds of use cases we need a way to 'force' the BufferedMutator to 
> automatically flush any records in the buffer even if the buffer is not full.
> I'll put up a pull request with a proposed implementation for review against 
> the master (i.e. 3.0.0).
> When approved I would like to backport this to the 1.x and 2.x versions of 
> the client in the same (as close as possible) way.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout

2017-12-20 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16298975#comment-16298975
 ] 

Jerry He commented on HBASE-19486:
--

Maybe similar to {{HRegion.MEMSTORE_PERIODIC_FLUSH_INTERVAL}}?

> Automatically flush BufferedMutator after a timeout 
> 
>
> Key: HBASE-19486
> URL: https://issues.apache.org/jira/browse/HBASE-19486
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Reporter: Niels Basjes
>Assignee: Niels Basjes
> Attachments: HBASE-19486-20171212-2117.patch, 
> HBASE-19486-20171218-1229.patch, HBASE-19486-20171218-1300.patch, 
> HBASE-19486-20171219-0933.patch, HBASE-19486-20171219-1026.patch, 
> HBASE-19486-20171219-1122-trigger-qa-run.patch, 
> HBASE-19486-20171220-1612-trigger-qa-run.patch
>
>
> I'm working on several projects where we are doing stream / event type 
> processing instead of batch type processing. We mostly use Apache Flink and 
> Apache Beam for these projects.
> When we ingest a continuous stream of events and feed that into HBase via a 
> BufferedMutator this all works fine. The buffer fills up at a predictable 
> rate and we can make sure it flushes several times per second into HBase by 
> tuning the buffer size.
> We also have situations where the event rate is unpredictable. Some times 
> because the source is in reality a batch job that puts records into Kafka, 
> sometimes because it is the "predictable in production" application in our 
> testing environment (where only the dev triggers a handful of events).
> For these kinds of use cases we need a way to 'force' the BufferedMutator to 
> automatically flush any records in the buffer even if the buffer is not full.
> I'll put up a pull request with a proposed implementation for review against 
> the master (i.e. 3.0.0).
> When approved I would like to backport this to the 1.x and 2.x versions of 
> the client in the same (as close as possible) way.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout

2017-12-20 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16298969#comment-16298969
 ] 

Jerry He commented on HBASE-19486:
--

+1  Nice!  Good tests too.

nit.  Could you rename it to periodic flush or similar? If I recall, there is 
somewhere we use auto flush for a different meaning.

> Automatically flush BufferedMutator after a timeout 
> 
>
> Key: HBASE-19486
> URL: https://issues.apache.org/jira/browse/HBASE-19486
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Reporter: Niels Basjes
>Assignee: Niels Basjes
> Attachments: HBASE-19486-20171212-2117.patch, 
> HBASE-19486-20171218-1229.patch, HBASE-19486-20171218-1300.patch, 
> HBASE-19486-20171219-0933.patch, HBASE-19486-20171219-1026.patch, 
> HBASE-19486-20171219-1122-trigger-qa-run.patch, 
> HBASE-19486-20171220-1612-trigger-qa-run.patch
>
>
> I'm working on several projects where we are doing stream / event type 
> processing instead of batch type processing. We mostly use Apache Flink and 
> Apache Beam for these projects.
> When we ingest a continuous stream of events and feed that into HBase via a 
> BufferedMutator this all works fine. The buffer fills up at a predictable 
> rate and we can make sure it flushes several times per second into HBase by 
> tuning the buffer size.
> We also have situations where the event rate is unpredictable. Some times 
> because the source is in reality a batch job that puts records into Kafka, 
> sometimes because it is the "predictable in production" application in our 
> testing environment (where only the dev triggers a handful of events).
> For these kinds of use cases we need a way to 'force' the BufferedMutator to 
> automatically flush any records in the buffer even if the buffer is not full.
> I'll put up a pull request with a proposed implementation for review against 
> the master (i.e. 3.0.0).
> When approved I would like to backport this to the 1.x and 2.x versions of 
> the client in the same (as close as possible) way.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-19557) Build and release source jars for hbase-shaded-client and others

2017-12-19 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-19557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-19557:
-
Issue Type: Sub-task  (was: Improvement)
Parent: HBASE-18800

> Build and release source jars for hbase-shaded-client and others
> 
>
> Key: HBASE-19557
> URL: https://issues.apache.org/jira/browse/HBASE-19557
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 1.3.1, 1.2.6
>Reporter: Jerry He
>
> It seems that currently we don't build and release source jars for 
> hbase-shaded-client (and server or mapreduce).  IDEs will complain from the 
> dependent users. We should provide them.
> http://central.maven.org/maven2/org/apache/hbase/hbase-shaded-client/1.3.1/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (HBASE-19557) Build and release source jars for hbase-shaded-client and others

2017-12-19 Thread Jerry He (JIRA)
Jerry He created HBASE-19557:


 Summary: Build and release source jars for hbase-shaded-client and 
others
 Key: HBASE-19557
 URL: https://issues.apache.org/jira/browse/HBASE-19557
 Project: HBase
  Issue Type: Improvement
Affects Versions: 1.2.6, 1.3.1
Reporter: Jerry He


It seems that currently we don't build and release source jars for 
hbase-shaded-client (and server or mapreduce).  IDEs will complain from the 
dependent users. We should provide them.

http://central.maven.org/maven2/org/apache/hbase/hbase-shaded-client/1.3.1/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-19552) update hbase to use new thirdparty libs

2017-12-19 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-19552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297487#comment-16297487
 ] 

Jerry He commented on HBASE-19552:
--

Glanced through.  Looks good.

> update hbase to use new thirdparty libs
> ---
>
> Key: HBASE-19552
> URL: https://issues.apache.org/jira/browse/HBASE-19552
> Project: HBase
>  Issue Type: Task
>  Components: dependencies, thirdparty
>Reporter: Mike Drob
>Assignee: Mike Drob
> Attachments: HBASE-19552.patch, HBASE-19552.v2.patch
>
>
> When we release hbase-thirdparty next release, we will need to update our 
> code for the packaging change.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Resolved] (HBASE-19171) Update package references to match new shaded offset in hbase-thirdparty

2017-12-19 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-19171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He resolved HBASE-19171.
--
   Resolution: Duplicate
Fix Version/s: (was: 2.0.0-beta-1)

Resolve as dup of the other task [~mdrob] is working on.

> Update package references to match new shaded offset in hbase-thirdparty
> 
>
> Key: HBASE-19171
> URL: https://issues.apache.org/jira/browse/HBASE-19171
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Jerry He
>Priority: Critical
>
> This has dependency on the parent task, and can only be done after a new 
> hbase-thirdparty release.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-15482) Provide an option to skip calculating block locations for SnapshotInputFormat

2017-12-19 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297478#comment-16297478
 ] 

Jerry He commented on HBASE-15482:
--

+1 on the latest patch.

> Provide an option to skip calculating block locations for SnapshotInputFormat
> -
>
> Key: HBASE-15482
> URL: https://issues.apache.org/jira/browse/HBASE-15482
> Project: HBase
>  Issue Type: Improvement
>  Components: mapreduce
>Reporter: Liyin Tang
>Assignee: Xiang Li
>Priority: Minor
> Fix For: 2.1.0
>
> Attachments: 15482.v3.txt, HBASE-15482.master.000.patch, 
> HBASE-15482.master.001.patch, HBASE-15482.master.002.patch, 
> HBASE-15482.master.003.patch
>
>
> When a MR job is reading from SnapshotInputFormat, it needs to calculate the 
> splits based on the block locations in order to get best locality. However, 
> this process may take a long time for large snapshots. 
> In some setup, the computing layer, Spark, Hive or Presto could run out side 
> of HBase cluster. In these scenarios, the block locality doesn't matter. 
> Therefore, it will be great to have an option to skip calculating the block 
> locations for every job. That will super useful for the Hive/Presto/Spark 
> connectors.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-15482) Provide an option to skip calculating block locations for SnapshotInputFormat

2017-12-10 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16285528#comment-16285528
 ] 

Jerry He commented on HBASE-15482:
--

Hi, [~water]

In the 002 patch, you added 'numTopsAtMost' in getBestLocations.  You will need 
another 'break' in the loop? Like:
If numTopsAtMost is met, then break out.

But again, the new code with this 'numTopsAtMost' is probably unnecessary.  The 
comment for the method getBestLocations has explained that it is not very 
likely you will get more than 3 hosts with at least 80% 
(hbase.tablesnapshotinputformat.locality.cutoff.multiplier) as much block 
locality as the top host with the best locality.  So you will break out early 
anyway with the filterWeight check. 
Your first patch's logic is good enough.
The added comment is good.
{code}
// As hostAndWeights is in descending order,
// we could break the loop as long as we meet a weight which is less than 
filterWeight
{code}

> Provide an option to skip calculating block locations for SnapshotInputFormat
> -
>
> Key: HBASE-15482
> URL: https://issues.apache.org/jira/browse/HBASE-15482
> Project: HBase
>  Issue Type: Improvement
>  Components: mapreduce
>Reporter: Liyin Tang
>Assignee: Xiang Li
>Priority: Minor
> Fix For: 2.1.0
>
> Attachments: HBASE-15482.master.000.patch, 
> HBASE-15482.master.001.patch, HBASE-15482.master.002.patch
>
>
> When a MR job is reading from SnapshotInputFormat, it needs to calculate the 
> splits based on the block locations in order to get best locality. However, 
> this process may take a long time for large snapshots. 
> In some setup, the computing layer, Spark, Hive or Presto could run out side 
> of HBase cluster. In these scenarios, the block locality doesn't matter. 
> Therefore, it will be great to have an option to skip calculating the block 
> locations for every job. That will super useful for the Hive/Presto/Spark 
> connectors.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-15482) Provide an option to skip calculating block locations for SnapshotInputFormat

2017-12-09 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16285062#comment-16285062
 ] 

Jerry He commented on HBASE-15482:
--

The patch looks good!
I just think the first patch 000 is cleaner.  But, as Ted suggested, change 
hbase.TableSnapshotInputFormat.locality to 
hbase.TableSnapshotInputFormat.locality.enable.  (Change the name 
SNAPSHOT_INPUTFORMAT_CARE_BLOCK_LOCALITY_KEY too).  The other changes look 
unnecessary except making it more complicated.
{code}
if (careBlockLocality) {
  Assert.assertTrue(split.getLocations() != null && split.getLocations().length 
!= 0);
} else {
  Assert.assertTrue(split.getLocations() != null && split.getLocations().length 
== 0);
}
{code}
This is ok too.  The first test is an existing test, and it has not failed 
previously.

> Provide an option to skip calculating block locations for SnapshotInputFormat
> -
>
> Key: HBASE-15482
> URL: https://issues.apache.org/jira/browse/HBASE-15482
> Project: HBase
>  Issue Type: Improvement
>  Components: mapreduce
>Reporter: Liyin Tang
>Assignee: Xiang Li
>Priority: Minor
> Fix For: 2.1.0
>
> Attachments: HBASE-15482.master.000.patch, 
> HBASE-15482.master.001.patch, HBASE-15482.master.002.patch
>
>
> When a MR job is reading from SnapshotInputFormat, it needs to calculate the 
> splits based on the block locations in order to get best locality. However, 
> this process may take a long time for large snapshots. 
> In some setup, the computing layer, Spark, Hive or Presto could run out side 
> of HBase cluster. In these scenarios, the block locality doesn't matter. 
> Therefore, it will be great to have an option to skip calculating the block 
> locations for every job. That will super useful for the Hive/Presto/Spark 
> connectors.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-19096) Add RowMutions batch support in AsyncTable

2017-11-28 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-19096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-19096:
-
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: (was: 2.0.0)
   2.0.0-beta-1
   Status: Resolved  (was: Patch Available)

Pushed to branch-2 and master. Thanks for the review!

> Add RowMutions batch support in AsyncTable
> --
>
> Key: HBASE-19096
> URL: https://issues.apache.org/jira/browse/HBASE-19096
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19096-master-v2.patch, 
> HBASE-19096-master-v3.patch, HBASE-19096-master-v4.patch, 
> HBASE-19096-master.patch
>
>
> Batch support for RowMutations has been added in the Table interface, but is 
> not in AsyncTable. This JIRA will add it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-19096) Add RowMutions batch support in AsyncTable

2017-11-28 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-19096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-19096:
-
Attachment: HBASE-19096-master-v4.patch

> Add RowMutions batch support in AsyncTable
> --
>
> Key: HBASE-19096
> URL: https://issues.apache.org/jira/browse/HBASE-19096
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-19096-master-v2.patch, 
> HBASE-19096-master-v3.patch, HBASE-19096-master-v4.patch, 
> HBASE-19096-master.patch
>
>
> Batch support for RowMutations has been added in the Table interface, but is 
> not in AsyncTable. This JIRA will add it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-19096) Add RowMutions batch support in AsyncTable

2017-11-28 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-19096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-19096:
-
Attachment: (was: HBASE-19096-master-v4.patch)

> Add RowMutions batch support in AsyncTable
> --
>
> Key: HBASE-19096
> URL: https://issues.apache.org/jira/browse/HBASE-19096
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-19096-master-v2.patch, 
> HBASE-19096-master-v3.patch, HBASE-19096-master.patch
>
>
> Batch support for RowMutations has been added in the Table interface, but is 
> not in AsyncTable. This JIRA will add it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-19096) Add RowMutions batch support in AsyncTable

2017-11-28 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-19096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-19096:
-
Attachment: HBASE-19096-master-v4.patch

v4 updated the method names.

> Add RowMutions batch support in AsyncTable
> --
>
> Key: HBASE-19096
> URL: https://issues.apache.org/jira/browse/HBASE-19096
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-19096-master-v2.patch, 
> HBASE-19096-master-v3.patch, HBASE-19096-master-v4.patch, 
> HBASE-19096-master.patch
>
>
> Batch support for RowMutations has been added in the Table interface, but is 
> not in AsyncTable. This JIRA will add it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-19096) Add RowMutions batch support in AsyncTable

2017-11-27 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-19096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268204#comment-16268204
 ] 

Jerry He commented on HBASE-19096:
--

The regionActionBuilder, actionBuilder, and mutationBuilder have been there as 
parameters originally. I guess they are passed in so that they are re-used to 
save some new allocation of the builders.
Add a 's' to the name. Like 'buildRegionActions'? 

> Add RowMutions batch support in AsyncTable
> --
>
> Key: HBASE-19096
> URL: https://issues.apache.org/jira/browse/HBASE-19096
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-19096-master-v2.patch, 
> HBASE-19096-master-v3.patch, HBASE-19096-master.patch
>
>
> Batch support for RowMutations has been added in the Table interface, but is 
> not in AsyncTable. This JIRA will add it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-19096) Add RowMutions batch support in AsyncTable

2017-11-27 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-19096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267997#comment-16267997
 ] 

Jerry He commented on HBASE-19096:
--

v3 fixes checkstyle warnings.

> Add RowMutions batch support in AsyncTable
> --
>
> Key: HBASE-19096
> URL: https://issues.apache.org/jira/browse/HBASE-19096
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-19096-master-v2.patch, 
> HBASE-19096-master-v3.patch, HBASE-19096-master.patch
>
>
> Batch support for RowMutations has been added in the Table interface, but is 
> not in AsyncTable. This JIRA will add it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-19096) Add RowMutions batch support in AsyncTable

2017-11-27 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-19096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-19096:
-
Attachment: HBASE-19096-master-v3.patch

> Add RowMutions batch support in AsyncTable
> --
>
> Key: HBASE-19096
> URL: https://issues.apache.org/jira/browse/HBASE-19096
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-19096-master-v2.patch, 
> HBASE-19096-master-v3.patch, HBASE-19096-master.patch
>
>
> Batch support for RowMutations has been added in the Table interface, but is 
> not in AsyncTable. This JIRA will add it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-19333) Consider exposing ExportSnapshot#getSnapshotFiles through POJO class

2017-11-27 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-19333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267738#comment-16267738
 ] 

Jerry He commented on HBASE-19333:
--

I agree here that it is probably better to have a clean-up tool. We can ask if 
the person who asked/responded can contribute the tool.

> Consider exposing ExportSnapshot#getSnapshotFiles through POJO class
> 
>
> Key: HBASE-19333
> URL: https://issues.apache.org/jira/browse/HBASE-19333
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>
> In the thread, 
> http://search-hadoop.com/m/HBase/YGbbUxY9FyU74X?subj=Re+Deleting+and+cleaning+old+snapshots+exported+to+S3
>  , Timothy mentioned that he used reflection to get to 
> ExportSnapshot#getSnapshotFiles().
> {code}
>   private static List> getSnapshotFiles(final 
> Configuration conf,
>   final FileSystem fs, final Path snapshotDir) throws IOException {
> {code}
> SnapshotFileInfo is protobuf.
> We should consider exposing the API by replacing the protobuf class with POJO 
> class.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-19096) Add RowMutions batch support in AsyncTable

2017-11-26 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-19096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266285#comment-16266285
 ] 

Jerry He commented on HBASE-19096:
--

Attached v2 to address comments from [~zghaobac] and [~stack].
Pushed down the looping to RequestConverter and refactored.

> Add RowMutions batch support in AsyncTable
> --
>
> Key: HBASE-19096
> URL: https://issues.apache.org/jira/browse/HBASE-19096
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-19096-master-v2.patch, HBASE-19096-master.patch
>
>
> Batch support for RowMutations has been added in the Table interface, but is 
> not in AsyncTable. This JIRA will add it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-19096) Add RowMutions batch support in AsyncTable

2017-11-26 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-19096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-19096:
-
Attachment: HBASE-19096-master-v2.patch

> Add RowMutions batch support in AsyncTable
> --
>
> Key: HBASE-19096
> URL: https://issues.apache.org/jira/browse/HBASE-19096
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-19096-master-v2.patch, HBASE-19096-master.patch
>
>
> Batch support for RowMutations has been added in the Table interface, but is 
> not in AsyncTable. This JIRA will add it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-19257) Document tool to dump information from MasterProcWALs file

2017-11-21 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-19257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261435#comment-16261435
 ] 

Jerry He commented on HBASE-19257:
--

The shortname means the shortcut in the 'hbase' command?
{noformat}
Usage: hbase []  []
...
Commands:
Some commands take arguments. Pass no args or -h for usage.
  shell   Run the HBase shell
  hbckRun the hbase 'fsck' tool
  snapshotCreate a new snapshot of a table
  snapshotinfoTool for dumping snapshot information
  wal Write-ahead-log analyzer
  hfile   Store file analyzer
...
  pe  Run PerformanceEvaluation
  ltt Run LoadTestTool
  version Print the version
  CLASSNAME   Run the class named CLASSNAME
{noformat}

Then 'procwal'?  To be similar to the 'wal' and 'hfile'.

> Document tool to dump information from MasterProcWALs file
> --
>
> Key: HBASE-19257
> URL: https://issues.apache.org/jira/browse/HBASE-19257
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>
> I was troubleshooting customer case where high number of files piled up under 
> MasterProcWALs directory.
> Gaining insight into (sample) file from MasterProcWALs dir would help find 
> the root cause.
> This JIRA is to document ProcedureWALPrettyPrinter which reads proc wal file 
> and prints (selected) information.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-19096) Add RowMutions batch support in AsyncTable

2017-11-19 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-19096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258600#comment-16258600
 ] 

Jerry He commented on HBASE-19096:
--

[~zghaobac] Thanks for chiming in.
bq. Seems we will iterate RegionAction's actions twice. One is in buildReq. 
Another one is in buildNoDataRegionAction. Can we move this to one loop?
That is a good idea. There is an old TODO there.
Probably need some refracting around buildNoDataRegionAction.  Let me see what 
I can do.
bq. If a RegionAction's actions have a RowMutations and some put/delete, then 
there are 2 RegionAction will be added to MutliRequest?
Yes, each RowMutations will be a separate RegionAction.
bq. About buildReq, there are some code same with MultiServerCallable, can we 
do some refactor to avoid this?
Again, need some refactoring. Let me see if it is too much.

> Add RowMutions batch support in AsyncTable
> --
>
> Key: HBASE-19096
> URL: https://issues.apache.org/jira/browse/HBASE-19096
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-19096-master.patch
>
>
> Batch support for RowMutations has been added in the Table interface, but is 
> not in AsyncTable. This JIRA will add it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-19096) Add RowMutions batch support in AsyncTable

2017-11-19 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-19096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258597#comment-16258597
 ] 

Jerry He commented on HBASE-19096:
--

bq. We do this if (row instanceof RowMutations) { ... but if the row is 
anything else, we do nothing? Could it be something else?
All other cases handled in the code below that.

I will add more comment.


> Add RowMutions batch support in AsyncTable
> --
>
> Key: HBASE-19096
> URL: https://issues.apache.org/jira/browse/HBASE-19096
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-19096-master.patch
>
>
> Batch support for RowMutations has been added in the Table interface, but is 
> not in AsyncTable. This JIRA will add it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-19257) Tool to dump information from MasterProcWALs file

2017-11-14 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-19257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251937#comment-16251937
 ] 

Jerry He commented on HBASE-19257:
--

HBASE-15592?

> Tool to dump information from MasterProcWALs file
> -
>
> Key: HBASE-19257
> URL: https://issues.apache.org/jira/browse/HBASE-19257
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>
> I was troubleshooting customer case where high number of files piled up under 
> MasterProcWALs directory.
> Gaining insight into (sample) file from MasterProcWALs dir would help find 
> the root cause.
> This JIRA is to add such tool which reads proc wal file and prints (selected) 
> information.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (HBASE-19171) Update package references to match new shaded offset in hbase-thirdparty

2017-11-03 Thread Jerry He (JIRA)
Jerry He created HBASE-19171:


 Summary: Update package references to match new shaded offset in 
hbase-thirdparty
 Key: HBASE-19171
 URL: https://issues.apache.org/jira/browse/HBASE-19171
 Project: HBase
  Issue Type: Sub-task
Reporter: Jerry He
Priority: Critical
 Fix For: 2.0.0


This has dependency on the parent task, and can only be done after a new 
hbase-thirdparty release.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (HBASE-19170) [hbase-thirdparty] Change the relocation offset of shaded artifacts

2017-11-03 Thread Jerry He (JIRA)
Jerry He created HBASE-19170:


 Summary: [hbase-thirdparty] Change the relocation offset of shaded 
artifacts
 Key: HBASE-19170
 URL: https://issues.apache.org/jira/browse/HBASE-19170
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.0.1
Reporter: Jerry He
Priority: Critical
 Fix For: 1.0.2


On the dev@hbase list, we conclude that we need to change the relocation offset 
in hbase-thirdparty to avoid shading conflicts with the other hbase shaded 
components (hbase-shaded-client and hbase-shaded-mapreduce components).
https://lists.apache.org/thread.html/1aa5d1d7f6d176df49e72096926b011cafe1315932515346d06e8342@%3Cdev.hbase.apache.org%3E
The suggestion is to use "o.a.h.hbase.thirdparty" in hbase-thirdparty to 
differentiate between "shaded" for downstream of us vs "thirdparty" for our 
internal use.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-19127) Set State.SPLITTING, MERGING, MERGING_NEW, SPLITTING_NEW properly in RegionStatesNode

2017-11-01 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-19127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-19127:
-
Issue Type: Sub-task  (was: Improvement)
Parent: HBASE-19126

> Set State.SPLITTING, MERGING, MERGING_NEW, SPLITTING_NEW properly in 
> RegionStatesNode
> -
>
> Key: HBASE-19127
> URL: https://issues.apache.org/jira/browse/HBASE-19127
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Yi Liang
>Assignee: Yi Liang
>Priority: Major
> Attachments: state.patch
>
>
> In current code, we did not set above states to a region node at all, but we 
> still have statements like below to check if node have above states.
> {code}
> else if (!regionNode.isInState(State.CLOSING, State.SPLITTING)) {
> 
> }
> {code}
> We need to set above states in a correct place.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-19127) Set State.SPLITTING, MERGING, MERGING_NEW, SPLITTING_NEW properly in RegionStatesNode

2017-10-31 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-19127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16233629#comment-16233629
 ] 

Jerry He commented on HBASE-19127:
--

Will the change/shuffle of the state codes affect backward compatibility? For 
example, old proc wals can not be read correctly anymore?
If this is new code, it should be fine.  
Just asking.

> Set State.SPLITTING, MERGING, MERGING_NEW, SPLITTING_NEW properly in 
> RegionStatesNode
> -
>
> Key: HBASE-19127
> URL: https://issues.apache.org/jira/browse/HBASE-19127
> Project: HBase
>  Issue Type: Improvement
>Reporter: Yi Liang
>Assignee: Yi Liang
>Priority: Major
> Attachments: state.patch
>
>
> In current code, we did not set above states to a region node at all, but we 
> still have statements like below to check if node have above states.
> {code}
> else if (!regionNode.isInState(State.CLOSING, State.SPLITTING)) {
> 
> }
> {code}
> We need to set above states in a correct place.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (HBASE-19145) Look into hbase-2 client going to hbase-1 server

2017-10-31 Thread Jerry He (JIRA)
Jerry He created HBASE-19145:


 Summary: Look into hbase-2 client going to hbase-1 server
 Key: HBASE-19145
 URL: https://issues.apache.org/jira/browse/HBASE-19145
 Project: HBase
  Issue Type: Task
Affects Versions: 2.0.0-beta-1
Reporter: Jerry He
Priority: Major


>From the "[DISCUSS] hbase-2.0.0 compatibility expectations" thread.

Do we support hbase-2 client going against hbase-1 server?
We seem to be fine mix-and-match the clients and servers within the
hbase-1 releases.   And IIRC hbase-1 client is ok against 0.98 server.
Suppose I have a product  that depends and bundles HBase client. I
want to upgrade the dependency to hbase-2 so that it can take
advantages of and claim support of hbase-2.
But does it mean that I will need drop the claims that the new version
of the product support any hbase-1 backend?

It has not been an objective. It might work doing basic Client API on a
later branch-1 but will fail doing Admin functions (and figuring if a Table
is online).  If it was a small thing to make it
work, lets get it in.

Let's look into it to see what works and what not.  Have a statement at least.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Resolved] (HBASE-19003) Make sure all balancer actions respect decommissioned server

2017-10-30 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-19003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He resolved HBASE-19003.
--
Resolution: Done

> Make sure all balancer actions respect decommissioned server
> 
>
> Key: HBASE-19003
> URL: https://issues.apache.org/jira/browse/HBASE-19003
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0-beta-1
>
>
> There have been questions raised in HBASE-10367 and other related JIRAs. We 
> want to make sure all aspects of the balancer respect the draining flag. We 
> will have a good look, and fix if any violation is found.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Assigned] (HBASE-19003) Make sure all balancer actions respect decommissioned server

2017-10-30 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-19003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He reassigned HBASE-19003:


Assignee: Jerry He

> Make sure all balancer actions respect decommissioned server
> 
>
> Key: HBASE-19003
> URL: https://issues.apache.org/jira/browse/HBASE-19003
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0-beta-1
>
>
> There have been questions raised in HBASE-10367 and other related JIRAs. We 
> want to make sure all aspects of the balancer respect the draining flag. We 
> will have a good look, and fix if any violation is found.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-19003) Make sure all balancer actions respect decommissioned server

2017-10-30 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-19003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16225510#comment-16225510
 ] 

Jerry He commented on HBASE-19003:
--

It looks good overall. 
The balancer's retainAssignment, randomAssignment and roundRobinAssignment all 
take a list of servers as parameter. 
We always call ServerManager.createDestinationServersList() to get the server 
list. This is a good list, with consideration of only online servers and 
avoiding the drainning servers.
The balancer's balanceCluster call has the draining servers removed from 
consideration to begin with.
Moreover, the assign phase will check the plan against the list obtained by 
ServerManager.createDestinationServersList(), which doubly make it unlikely
that a region is assigned to a wrong server.

> Make sure all balancer actions respect decommissioned server
> 
>
> Key: HBASE-19003
> URL: https://issues.apache.org/jira/browse/HBASE-19003
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Jerry He
> Fix For: 2.0.0-beta-1
>
>
> There have been questions raised in HBASE-10367 and other related JIRAs. We 
> want to make sure all aspects of the balancer respect the draining flag. We 
> will have a good look, and fix if any violation is found.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-19096) Add RowMutions batch support in AsyncTable

2017-10-30 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-19096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16225500#comment-16225500
 ] 

Jerry He commented on HBASE-19096:
--

Ping [~Apache9] [~zghaobac], or [~stack]  Can one of you give a quick review?

> Add RowMutions batch support in AsyncTable
> --
>
> Key: HBASE-19096
> URL: https://issues.apache.org/jira/browse/HBASE-19096
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-19096-master.patch
>
>
> Batch support for RowMutations has been added in the Table interface, but is 
> not in AsyncTable. This JIRA will add it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-19120) IllegalArgumentException from ZNodeClearer when master shuts down

2017-10-30 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-19120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16225487#comment-16225487
 ] 

Jerry He commented on HBASE-19120:
--

+1

> IllegalArgumentException from ZNodeClearer when master shuts down
> -
>
> Key: HBASE-19120
> URL: https://issues.apache.org/jira/browse/HBASE-19120
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 19120.v1.txt
>
>
> Found the following in master log (build as of commit eee3b0) :
> {code}
> 2017-10-30 15:40:24,383 ERROR [main] util.ServerCommandLine: Failed to run
> java.lang.IllegalArgumentException: Path must start with / character
> at 
> org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:51)
> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:851)
> at 
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:182)
> at 
> org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeFailSilent(ZKUtil.java:1266)
> at 
> org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeFailSilent(ZKUtil.java:1258)
> at org.apache.hadoop.hbase.ZNodeClearer.clear(ZNodeClearer.java:186)
> at 
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:143)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at 
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:127)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2873)
> {code}
> Looking at ZNodeClearer, it seems that intention was to remove znode under 
> /rs subtree.
> However, the znode name was passed without path.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-19096) Add RowMutions batch support in AsyncTable

2017-10-26 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-19096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-19096:
-
Attachment: (was: HBASE-19096-master.patch)

> Add RowMutions batch support in AsyncTable
> --
>
> Key: HBASE-19096
> URL: https://issues.apache.org/jira/browse/HBASE-19096
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-19096-master.patch
>
>
> Batch support for RowMutations has been added in the Table interface, but is 
> not in AsyncTable. This JIRA will add it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-19096) Add RowMutions batch support in AsyncTable

2017-10-26 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-19096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-19096:
-
Attachment: HBASE-19096-master.patch

> Add RowMutions batch support in AsyncTable
> --
>
> Key: HBASE-19096
> URL: https://issues.apache.org/jira/browse/HBASE-19096
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-19096-master.patch, HBASE-19096-master.patch
>
>
> Batch support for RowMutations has been added in the Table interface, but is 
> not in AsyncTable. This JIRA will add it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-19096) Add RowMutions batch support in AsyncTable

2017-10-26 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-19096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-19096:
-
Status: Patch Available  (was: Open)

> Add RowMutions batch support in AsyncTable
> --
>
> Key: HBASE-19096
> URL: https://issues.apache.org/jira/browse/HBASE-19096
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-19096-master.patch
>
>
> Batch support for RowMutations has been added in the Table interface, but is 
> not in AsyncTable. This JIRA will add it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-19096) Add RowMutions batch support in AsyncTable

2017-10-26 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-19096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-19096:
-
Attachment: HBASE-19096-master.patch

> Add RowMutions batch support in AsyncTable
> --
>
> Key: HBASE-19096
> URL: https://issues.apache.org/jira/browse/HBASE-19096
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-19096-master.patch
>
>
> Batch support for RowMutations has been added in the Table interface, but is 
> not in AsyncTable. This JIRA will add it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-19096) Add RowMutions batch support in AsyncTable

2017-10-26 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-19096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-19096:
-
Fix Version/s: (was: 1.5.0)
   (was: 1.4.0)
   (was: 3.0.0)

> Add RowMutions batch support in AsyncTable
> --
>
> Key: HBASE-19096
> URL: https://issues.apache.org/jira/browse/HBASE-19096
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
>
> Batch support for RowMutations has been added in the Table interface, but is 
> not in AsyncTable. This JIRA will add it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (HBASE-19096) Add RowMutions batch support in AsyncTable

2017-10-26 Thread Jerry He (JIRA)
Jerry He created HBASE-19096:


 Summary: Add RowMutions batch support in AsyncTable
 Key: HBASE-19096
 URL: https://issues.apache.org/jira/browse/HBASE-19096
 Project: HBase
  Issue Type: Sub-task
Reporter: Jerry He
Assignee: Jerry He


Batch support for RowMutations has been added in the Table interface, but is 
not in AsyncTable. This JIRA will add it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-19021) Restore a few important missing logics for balancer in 2.0

2017-10-24 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16217353#comment-16217353
 ] 

Jerry He commented on HBASE-19021:
--

Thanks [~stack]. I got interrupted after committing. 
Added Release Notes as well.

> Restore a few important missing logics for balancer in 2.0
> --
>
> Key: HBASE-19021
> URL: https://issues.apache.org/jira/browse/HBASE-19021
> Project: HBase
>  Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Critical
> Fix For: 2.0.0-alpha-4
>
> Attachments: HBASE-19021-master-v2.patch, HBASE-19021-master.patch, 
> HBASE-19021-master.patch
>
>
> After looking at the code, and some testing, I see the following things are 
> missing for balancer to work properly after AMv2.
> # hbase.master.loadbalance.bytable is not respected. It is always 'bytable'. 
> Previous default is cluster wide, not by table.
> # Servers with no assignments is not added for balance consideration.
> # Crashed server is not removed from the in-memory server map in 
> RegionStates, which affects balance.
> # Draining marker is not respected when balance.
> Also try to re-enable {{TestRegionRebalancing}}, which has a 
> {{testRebalanceOnRegionServerNumberChange}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-19021) Restore a few important missing logics for balancer in 2.0

2017-10-24 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-19021:
-
Release Note: 
Re-enabled 'hbase.master.loadbalance.bytable', default 'false'.
Draining servers are removed from consideration by blancer.balanceCluster() 
call.

> Restore a few important missing logics for balancer in 2.0
> --
>
> Key: HBASE-19021
> URL: https://issues.apache.org/jira/browse/HBASE-19021
> Project: HBase
>  Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Critical
> Fix For: 2.0.0-alpha-4
>
> Attachments: HBASE-19021-master-v2.patch, HBASE-19021-master.patch, 
> HBASE-19021-master.patch
>
>
> After looking at the code, and some testing, I see the following things are 
> missing for balancer to work properly after AMv2.
> # hbase.master.loadbalance.bytable is not respected. It is always 'bytable'. 
> Previous default is cluster wide, not by table.
> # Servers with no assignments is not added for balance consideration.
> # Crashed server is not removed from the in-memory server map in 
> RegionStates, which affects balance.
> # Draining marker is not respected when balance.
> Also try to re-enable {{TestRegionRebalancing}}, which has a 
> {{testRebalanceOnRegionServerNumberChange}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-19072) Missing beak in catch block of InterruptedException in HRegion#waitForFlushes()

2017-10-23 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-19072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16215885#comment-16215885
 ] 

Jerry He commented on HBASE-19072:
--

+1

> Missing beak in catch block of InterruptedException in 
> HRegion#waitForFlushes() 
> 
>
> Key: HBASE-19072
> URL: https://issues.apache.org/jira/browse/HBASE-19072
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 19072.v1.txt
>
>
> During review of HBASE-18358, [~elserj] found that there was missing break in 
> the catch block:
> {code}
>   } catch (InterruptedException iex) {
> // essentially ignore and propagate the interrupt back up
> LOG.warn("Interrupted while waiting");
> interrupted = true;
>   }
> {code}
> When thread is interrupted, we shouldn't wait on writestate.flushing anymore.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-19021) Restore a few important missing logics for balancer in 2.0

2017-10-22 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214636#comment-16214636
 ] 

Jerry He commented on HBASE-19021:
--

Are you good with the patch?  [~stack], [~yuzhih...@gmail.com]
Any other comment from others?

> Restore a few important missing logics for balancer in 2.0
> --
>
> Key: HBASE-19021
> URL: https://issues.apache.org/jira/browse/HBASE-19021
> Project: HBase
>  Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Critical
> Attachments: HBASE-19021-master-v2.patch, HBASE-19021-master.patch, 
> HBASE-19021-master.patch
>
>
> After looking at the code, and some testing, I see the following things are 
> missing for balancer to work properly after AMv2.
> # hbase.master.loadbalance.bytable is not respected. It is always 'bytable'. 
> Previous default is cluster wide, not by table.
> # Servers with no assignments is not added for balance consideration.
> # Crashed server is not removed from the in-memory server map in 
> RegionStates, which affects balance.
> # Draining marker is not respected when balance.
> Also try to re-enable {{TestRegionRebalancing}}, which has a 
> {{testRebalanceOnRegionServerNumberChange}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-19021) Restore a few important missing logics for balancer in 2.0

2017-10-22 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-19021:
-
Attachment: HBASE-19021-master-v2.patch

v2 addressed [~yuzhih...@gmail.com]'s earlier comment.

> Restore a few important missing logics for balancer in 2.0
> --
>
> Key: HBASE-19021
> URL: https://issues.apache.org/jira/browse/HBASE-19021
> Project: HBase
>  Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Critical
> Attachments: HBASE-19021-master-v2.patch, HBASE-19021-master.patch, 
> HBASE-19021-master.patch
>
>
> After looking at the code, and some testing, I see the following things are 
> missing for balancer to work properly after AMv2.
> # hbase.master.loadbalance.bytable is not respected. It is always 'bytable'. 
> Previous default is cluster wide, not by table.
> # Servers with no assignments is not added for balance consideration.
> # Crashed server is not removed from the in-memory server map in 
> RegionStates, which affects balance.
> # Draining marker is not respected when balance.
> Also try to re-enable {{TestRegionRebalancing}}, which has a 
> {{testRebalanceOnRegionServerNumberChange}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-10367) RegionServer graceful stop / decommissioning

2017-10-20 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-10367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212231#comment-16212231
 ] 

Jerry He commented on HBASE-10367:
--

Yes, the AC hooks are added.

For the recommission, the region list is optional. (I tried to use the 
Optional, but in the other JIRA we try to avoid using Optional as parameter.)  
It is up to the user/caller.  If it is not provided, then no region move.  
decommission and recommission need to pair up.
But a normal graceful stop sequence would be:
0. get regions for the region server.
1. decommission call with offloading of the regions.
2. Stop the region server. 
3. Start the region server (after patching, e.g.).
4. recommission call with the last list of regions.

The current graceful-stop script does similar steps, but with manual region 
mover.  We can get it to use the new APIs.

> RegionServer graceful stop / decommissioning
> 
>
> Key: HBASE-10367
> URL: https://issues.apache.org/jira/browse/HBASE-10367
> Project: HBase
>  Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Jerry He
> Fix For: 3.0.0, 2.0.0-alpha-4
>
> Attachments: HBASE-10367-master-2.patch, HBASE-10367-master.patch, 
> HBASE-10367-master.patch
>
>
> Right now, we have a weird way of node decommissioning / graceful stop, which 
> is a graceful_stop.sh bash script, and a region_mover ruby script, and some 
> draining server support which you have to manually write to a znode 
> (really!). Also draining servers is only partially supported in LB operations 
> (LB does take that into account for roundRobin assignment, but not for normal 
> balance) 
> See 
> http://hbase.apache.org/book/node.management.html and HBASE-3071
> I think we should support graceful stop as a first class citizen. Thinking 
> about it, it seems that the difference between regionserver stop and graceful 
> stop is that regionserver stop will close the regions, but the master will 
> only assign them after the znode is deleted. 
> In the new master design (or even before), if we allow RS to be able to close 
> regions on its own (without master initiating it), then graceful stop becomes 
> regular stop. The RS already closes the regions cleanly, and will reject new 
> region assignments, so that we don't need much of the balancer or draining 
> server trickery. 
> This ties into the new master/AM redesign (HBASE-5487), but still deserves 
> it's own jira. Let's use this to brainstorm on the design. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Resolved] (HBASE-17369) Add ACL to the new region server drain related API

2017-10-19 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-17369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He resolved HBASE-17369.
--
Resolution: Implemented

Implemented as part of HBASE-10367.

> Add ACL to the new region server drain related API
> --
>
> Key: HBASE-17369
> URL: https://issues.apache.org/jira/browse/HBASE-17369
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Jerry He
>Priority: Critical
>
> Add ACL to the new region server drain related API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-10367) RegionServer graceful stop / decommissioning

2017-10-19 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-10367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-10367:
-
   Resolution: Fixed
Fix Version/s: (was: 2.0.0-beta-1)
   2.0.0-alpha-4
   3.0.0
   Status: Resolved  (was: Patch Available)

Committed to branch-2 and master.  Thanks for the review, [~stack]

> RegionServer graceful stop / decommissioning
> 
>
> Key: HBASE-10367
> URL: https://issues.apache.org/jira/browse/HBASE-10367
> Project: HBase
>  Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Jerry He
> Fix For: 3.0.0, 2.0.0-alpha-4
>
> Attachments: HBASE-10367-master-2.patch, HBASE-10367-master.patch, 
> HBASE-10367-master.patch
>
>
> Right now, we have a weird way of node decommissioning / graceful stop, which 
> is a graceful_stop.sh bash script, and a region_mover ruby script, and some 
> draining server support which you have to manually write to a znode 
> (really!). Also draining servers is only partially supported in LB operations 
> (LB does take that into account for roundRobin assignment, but not for normal 
> balance) 
> See 
> http://hbase.apache.org/book/node.management.html and HBASE-3071
> I think we should support graceful stop as a first class citizen. Thinking 
> about it, it seems that the difference between regionserver stop and graceful 
> stop is that regionserver stop will close the regions, but the master will 
> only assign them after the znode is deleted. 
> In the new master design (or even before), if we allow RS to be able to close 
> regions on its own (without master initiating it), then graceful stop becomes 
> regular stop. The RS already closes the regions cleanly, and will reject new 
> region assignments, so that we don't need much of the balancer or draining 
> server trickery. 
> This ties into the new master/AM redesign (HBASE-5487), but still deserves 
> it's own jira. Let's use this to brainstorm on the design. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-10367) RegionServer graceful stop / decommissioning

2017-10-19 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-10367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-10367:
-
Hadoop Flags: Reviewed
Release Note: 
Added three top level Admin APIs to help decommissioning and graceful stop of 
region servers.

{code}
  /**
   * Mark region server(s) as decommissioned to prevent additional regions from 
getting
   * assigned to them. Optionally unload the regions on the servers. If there 
are multiple servers
   * to be decommissioned, decommissioning them at the same time can prevent 
wasteful region
   * movements. Region unloading is asynchronous.
   * @param servers The list of servers to decommission.
   * @param offload True to offload the regions from the decommissioned servers
   */
  void decommissionRegionServers(List servers, boolean offload) 
throws IOException;

  /**
   * List region servers marked as decommissioned, which can not be assigned 
regions.
   * @return List of decommissioned region servers.
   */
  List listDecommissionedRegionServers() throws IOException;

  /**
   * Remove decommission marker from a region server to allow regions 
assignments.
   * Load regions onto the server if a list of regions is given. Region loading 
is
   * asynchronous.
   * @param server The server to recommission.
   * @param encodedRegionNames Regions to load onto the server.
   */
  void recommissionRegionServer(ServerName server, List 
encodedRegionNames)
  throws IOException;
{code}



> RegionServer graceful stop / decommissioning
> 
>
> Key: HBASE-10367
> URL: https://issues.apache.org/jira/browse/HBASE-10367
> Project: HBase
>  Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Jerry He
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-10367-master-2.patch, HBASE-10367-master.patch, 
> HBASE-10367-master.patch
>
>
> Right now, we have a weird way of node decommissioning / graceful stop, which 
> is a graceful_stop.sh bash script, and a region_mover ruby script, and some 
> draining server support which you have to manually write to a znode 
> (really!). Also draining servers is only partially supported in LB operations 
> (LB does take that into account for roundRobin assignment, but not for normal 
> balance) 
> See 
> http://hbase.apache.org/book/node.management.html and HBASE-3071
> I think we should support graceful stop as a first class citizen. Thinking 
> about it, it seems that the difference between regionserver stop and graceful 
> stop is that regionserver stop will close the regions, but the master will 
> only assign them after the znode is deleted. 
> In the new master design (or even before), if we allow RS to be able to close 
> regions on its own (without master initiating it), then graceful stop becomes 
> regular stop. The RS already closes the regions cleanly, and will reject new 
> region assignments, so that we don't need much of the balancer or draining 
> server trickery. 
> This ties into the new master/AM redesign (HBASE-5487), but still deserves 
> it's own jira. Let's use this to brainstorm on the design. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-10367) RegionServer graceful stop / decommissioning

2017-10-19 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-10367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-10367:
-
Release Note: 
Added three top level Admin APIs to help decommissioning and graceful stop of 
region servers.

  /**
   * Mark region server(s) as decommissioned to prevent additional regions from 
getting
   * assigned to them. Optionally unload the regions on the servers. If there 
are multiple servers
   * to be decommissioned, decommissioning them at the same time can prevent 
wasteful region
   * movements. Region unloading is asynchronous.
   * @param servers The list of servers to decommission.
   * @param offload True to offload the regions from the decommissioned servers
   */
  void decommissionRegionServers(List servers, boolean offload) 
throws IOException;

  /**
   * List region servers marked as decommissioned, which can not be assigned 
regions.
   * @return List of decommissioned region servers.
   */
  List listDecommissionedRegionServers() throws IOException;

  /**
   * Remove decommission marker from a region server to allow regions 
assignments.
   * Load regions onto the server if a list of regions is given. Region loading 
is
   * asynchronous.
   * @param server The server to recommission.
   * @param encodedRegionNames Regions to load onto the server.
   */
  void recommissionRegionServer(ServerName server, List 
encodedRegionNames)  throws IOException;



  was:
Added three top level Admin APIs to help decommissioning and graceful stop of 
region servers.

  /**
   * Mark region server(s) as decommissioned to prevent additional regions from 
getting
   * assigned to them. Optionally unload the regions on the servers. If there 
are multiple servers
   * to be decommissioned, decommissioning them at the same time can prevent 
wasteful region
   * movements. Region unloading is asynchronous.
   * @param servers The list of servers to decommission.
   * @param offload True to offload the regions from the decommissioned servers
   */
  void decommissionRegionServers(List servers, boolean offload) 
throws IOException;

  /**
   * List region servers marked as decommissioned, which can not be assigned 
regions.
   * @return List of decommissioned region servers.
   */
  List listDecommissionedRegionServers() throws 
 IOException;

  /**
   * Remove decommission marker from a region server to allow regions 
assignments.
   * Load regions onto the server if a list of regions is given. Region loading 
is
   * asynchronous.
   * @param server The server to recommission.
   * @param encodedRegionNames Regions to load onto the server.
   */
  void recommissionRegionServer(ServerName server, List 
encodedRegionNames)  throws IOException;




> RegionServer graceful stop / decommissioning
> 
>
> Key: HBASE-10367
> URL: https://issues.apache.org/jira/browse/HBASE-10367
> Project: HBase
>  Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Jerry He
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-10367-master-2.patch, HBASE-10367-master.patch, 
> HBASE-10367-master.patch
>
>
> Right now, we have a weird way of node decommissioning / graceful stop, which 
> is a graceful_stop.sh bash script, and a region_mover ruby script, and some 
> draining server support which you have to manually write to a znode 
> (really!). Also draining servers is only partially supported in LB operations 
> (LB does take that into account for roundRobin assignment, but not for normal 
> balance) 
> See 
> http://hbase.apache.org/book/node.management.html and HBASE-3071
> I think we should support graceful stop as a first class citizen. Thinking 
> about it, it seems that the difference between regionserver stop and graceful 
> stop is that regionserver stop will close the regions, but the master will 
> only assign them after the znode is deleted. 
> In the new master design (or even before), if we allow RS to be able to close 
> regions on its own (without master initiating it), then graceful stop becomes 
> regular stop. The RS already closes the regions cleanly, and will reject new 
> region assignments, so that we don't need much of the balancer or draining 
> server trickery. 
> This ties into the new master/AM redesign (HBASE-5487), but still deserves 
> it's own jira. Let's use this to brainstorm on the design. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-10367) RegionServer graceful stop / decommissioning

2017-10-19 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-10367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-10367:
-
Release Note: 
Added three top level Admin APIs to help decommissioning and graceful stop of 
region servers.

  /**
   * Mark region server(s) as decommissioned to prevent additional regions from 
getting
   * assigned to them. Optionally unload the regions on the servers. If there 
are multiple servers
   * to be decommissioned, decommissioning them at the same time can prevent 
wasteful region
   * movements. Region unloading is asynchronous.
   * @param servers The list of servers to decommission.
   * @param offload True to offload the regions from the decommissioned servers
   */
  void decommissionRegionServers(List servers, boolean offload) 
throws IOException;

  /**
   * List region servers marked as decommissioned, which can not be assigned 
regions.
   * @return List of decommissioned region servers.
   */
  List listDecommissionedRegionServers() throws 
 IOException;

  /**
   * Remove decommission marker from a region server to allow regions 
assignments.
   * Load regions onto the server if a list of regions is given. Region loading 
is
   * asynchronous.
   * @param server The server to recommission.
   * @param encodedRegionNames Regions to load onto the server.
   */
  void recommissionRegionServer(ServerName server, List 
encodedRegionNames)  throws IOException;



  was:
Added three top level Admin APIs to help decommissioning and graceful stop of 
region servers.

{code}
  /**
   * Mark region server(s) as decommissioned to prevent additional regions from 
getting
   * assigned to them. Optionally unload the regions on the servers. If there 
are multiple servers
   * to be decommissioned, decommissioning them at the same time can prevent 
wasteful region
   * movements. Region unloading is asynchronous.
   * @param servers The list of servers to decommission.
   * @param offload True to offload the regions from the decommissioned servers
   */
  void decommissionRegionServers(List servers, boolean offload) 
throws IOException;

  /**
   * List region servers marked as decommissioned, which can not be assigned 
regions.
   * @return List of decommissioned region servers.
   */
  List listDecommissionedRegionServers() throws IOException;

  /**
   * Remove decommission marker from a region server to allow regions 
assignments.
   * Load regions onto the server if a list of regions is given. Region loading 
is
   * asynchronous.
   * @param server The server to recommission.
   * @param encodedRegionNames Regions to load onto the server.
   */
  void recommissionRegionServer(ServerName server, List 
encodedRegionNames)
  throws IOException;
{code}




> RegionServer graceful stop / decommissioning
> 
>
> Key: HBASE-10367
> URL: https://issues.apache.org/jira/browse/HBASE-10367
> Project: HBase
>  Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Jerry He
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-10367-master-2.patch, HBASE-10367-master.patch, 
> HBASE-10367-master.patch
>
>
> Right now, we have a weird way of node decommissioning / graceful stop, which 
> is a graceful_stop.sh bash script, and a region_mover ruby script, and some 
> draining server support which you have to manually write to a znode 
> (really!). Also draining servers is only partially supported in LB operations 
> (LB does take that into account for roundRobin assignment, but not for normal 
> balance) 
> See 
> http://hbase.apache.org/book/node.management.html and HBASE-3071
> I think we should support graceful stop as a first class citizen. Thinking 
> about it, it seems that the difference between regionserver stop and graceful 
> stop is that regionserver stop will close the regions, but the master will 
> only assign them after the znode is deleted. 
> In the new master design (or even before), if we allow RS to be able to close 
> regions on its own (without master initiating it), then graceful stop becomes 
> regular stop. The RS already closes the regions cleanly, and will reject new 
> region assignments, so that we don't need much of the balancer or draining 
> server trickery. 
> This ties into the new master/AM redesign (HBASE-5487), but still deserves 
> it's own jira. Let's use this to brainstorm on the design. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Comment Edited] (HBASE-19021) Restore a few important missing logics for balancer in 2.0

2017-10-19 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211831#comment-16211831
 ] 

Jerry He edited comment on HBASE-19021 at 10/19/17 10:09 PM:
-

More explanation.

In the branch-1 RegionStates.getAssignmentsByTable()
https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java#L1115
there is a part to deal with servers w/o assignments and draining mode.  This 
is missing after AMv2.

But the draining mode is actually ok after a 'detour' in AMv2.
The balancer's balanceCluster() can pick a plan to move regions to the draining 
servers. The regions will be 'unassigned'. But in the 'assign' phase, when 
going thru retainAssignment check, the plan is checked against the server list 
obtained from ServerManager.createDestinationServersList().  This list is a 
good list without the draining servers. So it is like a detour, but the end 
result is ok.
But I restored the branch-1 behavior, which is to take the draining servers out 
of consideration from the beginning.

The balancer's retainAssignment, randomAssignment and roundRobinAssignment all 
take a server list as parameter.  We seem to be always calling 
ServerManager.createDestinationServersList() to pass the server list. They are 
all good.  Only the big balanceCluster() call has the issue.


was (Author: jinghe):
More explanation.

In the branch-1 RegionStates.getAssignmentsByTable()
https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java#L1115
there is a part to deal with servers w/o assignments and draining mode.  This 
is missing after AMv2.

But the draining mode is actually ok after a 'detour' in AMv2.
The balancer's balanceCluster() can pick a plan to move regions to the draining 
servers. The regions will be 'unassigned'. But in the 'assign' phase, when 
going thru retainAssignment check, the plan is checked against the server list 
obtained from ServerManager.createDestinationServersList().  This list is a 
good list without the draining servers. So it is like a detour, but the end 
result is ok.
But I restored the branch-1 behavior, which is to take the draining servers out 
of consideration from the beginning.

The balancer's retainAssignment, randomAssignment and roundRobinAssignment all 
take a server list an parameters.  We seem to be always calling 
ServerManager.createDestinationServersList() to pass the server list. They are 
all good.  Only the big balanceCluster() call has the issue.

> Restore a few important missing logics for balancer in 2.0
> --
>
> Key: HBASE-19021
> URL: https://issues.apache.org/jira/browse/HBASE-19021
> Project: HBase
>  Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Critical
> Attachments: HBASE-19021-master.patch, HBASE-19021-master.patch
>
>
> After looking at the code, and some testing, I see the following things are 
> missing for balancer to work properly after AMv2.
> # hbase.master.loadbalance.bytable is not respected. It is always 'bytable'. 
> Previous default is cluster wide, not by table.
> # Servers with no assignments is not added for balance consideration.
> # Crashed server is not removed from the in-memory server map in 
> RegionStates, which affects balance.
> # Draining marker is not respected when balance.
> Also try to re-enable {{TestRegionRebalancing}}, which has a 
> {{testRebalanceOnRegionServerNumberChange}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-19021) Restore a few important missing logics for balancer in 2.0

2017-10-19 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-19021:
-
Attachment: HBASE-19021-master.patch

> Restore a few important missing logics for balancer in 2.0
> --
>
> Key: HBASE-19021
> URL: https://issues.apache.org/jira/browse/HBASE-19021
> Project: HBase
>  Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Critical
> Attachments: HBASE-19021-master.patch, HBASE-19021-master.patch
>
>
> After looking at the code, and some testing, I see the following things are 
> missing for balancer to work properly after AMv2.
> # hbase.master.loadbalance.bytable is not respected. It is always 'bytable'. 
> Previous default is cluster wide, not by table.
> # Servers with no assignments is not added for balance consideration.
> # Crashed server is not removed from the in-memory server map in 
> RegionStates, which affects balance.
> # Draining marker is not respected when balance.
> Also try to re-enable {{TestRegionRebalancing}}, which has a 
> {{testRebalanceOnRegionServerNumberChange}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-19021) Restore a few important missing logics for balancer in 2.0

2017-10-19 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211835#comment-16211835
 ] 

Jerry He commented on HBASE-19021:
--

All the timeout tests passed locally for me.  There is already a successful QA 
run earlier.
But retry QA again.

> Restore a few important missing logics for balancer in 2.0
> --
>
> Key: HBASE-19021
> URL: https://issues.apache.org/jira/browse/HBASE-19021
> Project: HBase
>  Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Critical
> Attachments: HBASE-19021-master.patch
>
>
> After looking at the code, and some testing, I see the following things are 
> missing for balancer to work properly after AMv2.
> # hbase.master.loadbalance.bytable is not respected. It is always 'bytable'. 
> Previous default is cluster wide, not by table.
> # Servers with no assignments is not added for balance consideration.
> # Crashed server is not removed from the in-memory server map in 
> RegionStates, which affects balance.
> # Draining marker is not respected when balance.
> Also try to re-enable {{TestRegionRebalancing}}, which has a 
> {{testRebalanceOnRegionServerNumberChange}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-19021) Restore a few important missing logics for balancer in 2.0

2017-10-19 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211831#comment-16211831
 ] 

Jerry He commented on HBASE-19021:
--

More explanation.

In the branch-1 RegionStates.getAssignmentsByTable()
https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java#L1115
there is a part to deal with servers w/o assignments and draining mode.  This 
is missing after AMv2.

But the draining mode is actually ok after a 'detour' in AMv2.
The balancer's balanceCluster() can pick a plan to move regions to the draining 
servers. The regions will be 'unassigned'. But in the 'assign' phase, when 
going thru retainAssignment check, the plan is checked against the server list 
obtained from ServerManager.createDestinationServersList().  This list is a 
good list without the draining servers. So it is like a detour, but the end 
result is ok.
But I restored the branch-1 behavior, which is to take the draining servers out 
of consideration from the beginning.

The balancer's retainAssignment, randomAssignment and roundRobinAssignment all 
take a server list an parameters.  We seem to be always calling 
ServerManager.createDestinationServersList() to pass the server list. They are 
all good.  Only the big balanceCluster() call has the issue.

> Restore a few important missing logics for balancer in 2.0
> --
>
> Key: HBASE-19021
> URL: https://issues.apache.org/jira/browse/HBASE-19021
> Project: HBase
>  Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Critical
> Attachments: HBASE-19021-master.patch
>
>
> After looking at the code, and some testing, I see the following things are 
> missing for balancer to work properly after AMv2.
> # hbase.master.loadbalance.bytable is not respected. It is always 'bytable'. 
> Previous default is cluster wide, not by table.
> # Servers with no assignments is not added for balance consideration.
> # Crashed server is not removed from the in-memory server map in 
> RegionStates, which affects balance.
> # Draining marker is not respected when balance.
> Also try to re-enable {{TestRegionRebalancing}}, which has a 
> {{testRebalanceOnRegionServerNumberChange}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-19021) Restore a few important missing logics for balancer in 2.0

2017-10-19 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211787#comment-16211787
 ] 

Jerry He commented on HBASE-19021:
--

Hi, [~stack] 
bq.  bytable is not implemented in AMv2 IIRC
It is implemented RegionStates,  but just missing a call to it. 
 {code}
   * @param forceByCluster a flag to force to aggregate the server-load to the 
cluster level
   * @return A clone of current assignments by table.
   */
  public Map> 
getAssignmentsByTable(
  final boolean forceByCluster) {
if (!forceByCluster) return getAssignmentsByTable();
{code}

bq. You saying when you add a Server, it doesn't get Regions? (I don't recall 
this in testing but perhaps so).
Yes.  From looking at the code, and some testing.  Tested it again today.
{noformat}
ServerName  Start time  Last contactVersion 
Requests Per Second Num. Regions
throb1,16020,1508446443086  Thu Oct 19 13:54:03 PDT 20172 s 
3.0.0-SNAPSHOT  0   0
throb1,45403,1508440236028  Thu Oct 19 12:10:36 PDT 20170 s 
3.0.0-SNAPSHOT  0   6
Total:2 0   6
{noformat}
bq. You are fixing this over in another issue?
The draining issue is dealt with in this patch too.

> Restore a few important missing logics for balancer in 2.0
> --
>
> Key: HBASE-19021
> URL: https://issues.apache.org/jira/browse/HBASE-19021
> Project: HBase
>  Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Critical
> Attachments: HBASE-19021-master.patch
>
>
> After looking at the code, and some testing, I see the following things are 
> missing for balancer to work properly after AMv2.
> # hbase.master.loadbalance.bytable is not respected. It is always 'bytable'. 
> Previous default is cluster wide, not by table.
> # Servers with no assignments is not added for balance consideration.
> # Crashed server is not removed from the in-memory server map in 
> RegionStates, which affects balance.
> # Draining marker is not respected when balance.
> Also try to re-enable {{TestRegionRebalancing}}, which has a 
> {{testRebalanceOnRegionServerNumberChange}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-10367) RegionServer graceful stop / decommissioning

2017-10-16 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-10367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206970#comment-16206970
 ] 

Jerry He commented on HBASE-10367:
--

[~stack]  are you good with the patch?  Any more comments?
Comments from others?

> RegionServer graceful stop / decommissioning
> 
>
> Key: HBASE-10367
> URL: https://issues.apache.org/jira/browse/HBASE-10367
> Project: HBase
>  Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Jerry He
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-10367-master-2.patch, HBASE-10367-master.patch, 
> HBASE-10367-master.patch
>
>
> Right now, we have a weird way of node decommissioning / graceful stop, which 
> is a graceful_stop.sh bash script, and a region_mover ruby script, and some 
> draining server support which you have to manually write to a znode 
> (really!). Also draining servers is only partially supported in LB operations 
> (LB does take that into account for roundRobin assignment, but not for normal 
> balance) 
> See 
> http://hbase.apache.org/book/node.management.html and HBASE-3071
> I think we should support graceful stop as a first class citizen. Thinking 
> about it, it seems that the difference between regionserver stop and graceful 
> stop is that regionserver stop will close the regions, but the master will 
> only assign them after the znode is deleted. 
> In the new master design (or even before), if we allow RS to be able to close 
> regions on its own (without master initiating it), then graceful stop becomes 
> regular stop. The RS already closes the regions cleanly, and will reject new 
> region assignments, so that we don't need much of the balancer or draining 
> server trickery. 
> This ties into the new master/AM redesign (HBASE-5487), but still deserves 
> it's own jira. Let's use this to brainstorm on the design. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-19021) Restore a few important missing logics for balancer in 2.0

2017-10-16 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-19021:
-
Attachment: HBASE-19021-master.patch

> Restore a few important missing logics for balancer in 2.0
> --
>
> Key: HBASE-19021
> URL: https://issues.apache.org/jira/browse/HBASE-19021
> Project: HBase
>  Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Critical
> Attachments: HBASE-19021-master.patch
>
>
> After looking at the code, and some testing, I see the following things are 
> missing for balancer to work properly after AMv2.
> # hbase.master.loadbalance.bytable is not respected. It is always 'bytable'. 
> Previous default is cluster wide, not by table.
> # Servers with no assignments is not added for balance consideration.
> # Crashed server is not removed from the in-memory server map in 
> RegionStates, which affects balance.
> # Draining marker is not respected when balance.
> Also try to re-enable {{TestRegionRebalancing}}, which has a 
> {{testRebalanceOnRegionServerNumberChange}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-19021) Restore a few important missing logics for balancer in 2.0

2017-10-16 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206963#comment-16206963
 ] 

Jerry He commented on HBASE-19021:
--

bq. Why the negation ?
{code}
   * @param forceByCluster a flag to force to aggregate the server-load to the 
cluster level
   * @return A clone of current assignments by table.
   */
  public Map> 
getAssignmentsByTable(
  final boolean forceByCluster) {
if (!forceByCluster) return getAssignmentsByTable();
{code}
!isByTable will be cluster level.

> Restore a few important missing logics for balancer in 2.0
> --
>
> Key: HBASE-19021
> URL: https://issues.apache.org/jira/browse/HBASE-19021
> Project: HBase
>  Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Critical
>
> After looking at the code, and some testing, I see the following things are 
> missing for balancer to work properly after AMv2.
> # hbase.master.loadbalance.bytable is not respected. It is always 'bytable'. 
> Previous default is cluster wide, not by table.
> # Servers with no assignments is not added for balance consideration.
> # Crashed server is not removed from the in-memory server map in 
> RegionStates, which affects balance.
> # Draining marker is not respected when balance.
> Also try to re-enable {{TestRegionRebalancing}}, which has a 
> {{testRebalanceOnRegionServerNumberChange}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-19021) Restore a few important missing logics for balancer in 2.0

2017-10-16 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-19021:
-
Attachment: (was: HBASE-19021-master.patch)

> Restore a few important missing logics for balancer in 2.0
> --
>
> Key: HBASE-19021
> URL: https://issues.apache.org/jira/browse/HBASE-19021
> Project: HBase
>  Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Critical
>
> After looking at the code, and some testing, I see the following things are 
> missing for balancer to work properly after AMv2.
> # hbase.master.loadbalance.bytable is not respected. It is always 'bytable'. 
> Previous default is cluster wide, not by table.
> # Servers with no assignments is not added for balance consideration.
> # Crashed server is not removed from the in-memory server map in 
> RegionStates, which affects balance.
> # Draining marker is not respected when balance.
> Also try to re-enable {{TestRegionRebalancing}}, which has a 
> {{testRebalanceOnRegionServerNumberChange}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-19021) Restore a few important missing logics for balancer in 2.0

2017-10-16 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-19021:
-
Attachment: HBASE-19021-master.patch

> Restore a few important missing logics for balancer in 2.0
> --
>
> Key: HBASE-19021
> URL: https://issues.apache.org/jira/browse/HBASE-19021
> Project: HBase
>  Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Critical
> Attachments: HBASE-19021-master.patch
>
>
> After looking at the code, and some testing, I see the following things are 
> missing for balancer to work properly after AMv2.
> # hbase.master.loadbalance.bytable is not respected. It is always 'bytable'. 
> Previous default is cluster wide, not by table.
> # Servers with no assignments is not added for balance consideration.
> # Crashed server is not removed from the in-memory server map in 
> RegionStates, which affects balance.
> # Draining marker is not respected when balance.
> Also try to re-enable {{TestRegionRebalancing}}, which has a 
> {{testRebalanceOnRegionServerNumberChange}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-19021) Restore a few important missing logics for balancer in 2.0

2017-10-16 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-19021:
-
Status: Patch Available  (was: Open)

> Restore a few important missing logics for balancer in 2.0
> --
>
> Key: HBASE-19021
> URL: https://issues.apache.org/jira/browse/HBASE-19021
> Project: HBase
>  Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Critical
> Attachments: HBASE-19021-master.patch
>
>
> After looking at the code, and some testing, I see the following things are 
> missing for balancer to work properly after AMv2.
> # hbase.master.loadbalance.bytable is not respected. It is always 'bytable'. 
> Previous default is cluster wide, not by table.
> # Servers with no assignments is not added for balance consideration.
> # Crashed server is not removed from the in-memory server map in 
> RegionStates, which affects balance.
> # Draining marker is not respected when balance.
> Also try to re-enable {{TestRegionRebalancing}}, which has a 
> {{testRebalanceOnRegionServerNumberChange}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (HBASE-19021) Restore a few important missing logics for balancer in 2.0

2017-10-16 Thread Jerry He (JIRA)
Jerry He created HBASE-19021:


 Summary: Restore a few important missing logics for balancer in 2.0
 Key: HBASE-19021
 URL: https://issues.apache.org/jira/browse/HBASE-19021
 Project: HBase
  Issue Type: Bug
Reporter: Jerry He
Assignee: Jerry He
Priority: Critical


After looking at the code, and some testing, I see the following things are 
missing for balancer to work properly after AMv2.

# hbase.master.loadbalance.bytable is not respected. It is always 'bytable'. 
Previous default is cluster wide, not by table.
# Servers with no assignments is not added for balance consideration.
# Crashed server is not removed from the in-memory server map in RegionStates, 
which affects balance.
# Draining marker is not respected when balance.

Also try to re-enable {{TestRegionRebalancing}}, which has a 
{{testRebalanceOnRegionServerNumberChange}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-18986) Remove unnecessary null check after CellUtil.cloneQualifier()

2017-10-15 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-18986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-18986:
-
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.0.0-alpha-4
   3.0.0
   Status: Resolved  (was: Patch Available)

+1

> Remove unnecessary null check after CellUtil.cloneQualifier()
> -
>
> Key: HBASE-18986
> URL: https://issues.apache.org/jira/browse/HBASE-18986
> Project: HBase
>  Issue Type: Improvement
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Fix For: 3.0.0, 2.0.0-alpha-4
>
> Attachments: HBASE-18986.master.000.patch
>
>
> In master branch,
> {code:title=hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java|borderStyle=solid}
> // From line 2858
> public void prepareDeleteTimestamps(Mutation mutation, Map List> familyMap,
>   byte[] byteNow) throws IOException {
> for (Map.Entry e : familyMap.entrySet()) {
>   // ...
>   for (int i=0; i < listSize; i++) {
> // ...
> if (cell.getTimestamp() == HConstants.LATEST_TIMESTAMP && 
> CellUtil.isDeleteType(cell)) {
>   byte[] qual = CellUtil.cloneQualifier(cell);
>   if (qual == null) qual = HConstants.EMPTY_BYTE_ARRAY; // <-- here
>   ...
> {code}
> Might {{if (qual == null) qual = HConstants.EMPTY_BYTE_ARRAY;}} be removed?
> Could it be null after CellUtil.cloneQualifier()?
> {code:title=hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java|borderStyle=solid}
> public static byte[] cloneQualifier(Cell cell){
>   byte[] output = new byte[cell.getQualifierLength()];
>   copyQualifierTo(cell, output, 0);
>   return output;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-18986) Remove unnecessary null check after CellUtil.cloneQualifier()

2017-10-13 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-18986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-18986:
-
Attachment: HBASE-18986.master.000.patch

Yup, after the clone, we will get a byte[0] -- EMPTY_BYTE_ARRAY anyway if no 
qual.
Retry.

> Remove unnecessary null check after CellUtil.cloneQualifier()
> -
>
> Key: HBASE-18986
> URL: https://issues.apache.org/jira/browse/HBASE-18986
> Project: HBase
>  Issue Type: Improvement
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Attachments: HBASE-18986.master.000.patch, 
> HBASE-18986.master.000.patch
>
>
> In master branch,
> {code:title=hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java|borderStyle=solid}
> // From line 2858
> public void prepareDeleteTimestamps(Mutation mutation, Map List> familyMap,
>   byte[] byteNow) throws IOException {
> for (Map.Entry e : familyMap.entrySet()) {
>   // ...
>   for (int i=0; i < listSize; i++) {
> // ...
> if (cell.getTimestamp() == HConstants.LATEST_TIMESTAMP && 
> CellUtil.isDeleteType(cell)) {
>   byte[] qual = CellUtil.cloneQualifier(cell);
>   if (qual == null) qual = HConstants.EMPTY_BYTE_ARRAY; // <-- here
>   ...
> {code}
> Might {{if (qual == null) qual = HConstants.EMPTY_BYTE_ARRAY;}} be removed?
> Could it be null after CellUtil.cloneQualifier()?
> {code:title=hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java|borderStyle=solid}
> public static byte[] cloneQualifier(Cell cell){
>   byte[] output = new byte[cell.getQualifierLength()];
>   copyQualifierTo(cell, output, 0);
>   return output;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (HBASE-19003) Make sure all balancer actions respect decommissioned server

2017-10-12 Thread Jerry He (JIRA)
Jerry He created HBASE-19003:


 Summary: Make sure all balancer actions respect decommissioned 
server
 Key: HBASE-19003
 URL: https://issues.apache.org/jira/browse/HBASE-19003
 Project: HBase
  Issue Type: Sub-task
Reporter: Jerry He


There have been questions raised in HBASE-10367 and other related JIRAs. We 
want to make sure all aspects of the balancer respect the draining flag. We 
will have a good look, and fix if any violation is found.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-10367) RegionServer graceful stop / decommissioning

2017-10-12 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-10367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-10367:
-
Attachment: HBASE-10367-master-2.patch

> RegionServer graceful stop / decommissioning
> 
>
> Key: HBASE-10367
> URL: https://issues.apache.org/jira/browse/HBASE-10367
> Project: HBase
>  Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Jerry He
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-10367-master-2.patch, HBASE-10367-master.patch, 
> HBASE-10367-master.patch
>
>
> Right now, we have a weird way of node decommissioning / graceful stop, which 
> is a graceful_stop.sh bash script, and a region_mover ruby script, and some 
> draining server support which you have to manually write to a znode 
> (really!). Also draining servers is only partially supported in LB operations 
> (LB does take that into account for roundRobin assignment, but not for normal 
> balance) 
> See 
> http://hbase.apache.org/book/node.management.html and HBASE-3071
> I think we should support graceful stop as a first class citizen. Thinking 
> about it, it seems that the difference between regionserver stop and graceful 
> stop is that regionserver stop will close the regions, but the master will 
> only assign them after the znode is deleted. 
> In the new master design (or even before), if we allow RS to be able to close 
> regions on its own (without master initiating it), then graceful stop becomes 
> regular stop. The RS already closes the regions cleanly, and will reject new 
> region assignments, so that we don't need much of the balancer or draining 
> server trickery. 
> This ties into the new master/AM redesign (HBASE-5487), but still deserves 
> it's own jira. Let's use this to brainstorm on the design. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-10367) RegionServer graceful stop / decommissioning

2017-10-12 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-10367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-10367:
-
Attachment: (was: HBASE-10367-master-2.patch)

> RegionServer graceful stop / decommissioning
> 
>
> Key: HBASE-10367
> URL: https://issues.apache.org/jira/browse/HBASE-10367
> Project: HBase
>  Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Jerry He
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-10367-master.patch, HBASE-10367-master.patch
>
>
> Right now, we have a weird way of node decommissioning / graceful stop, which 
> is a graceful_stop.sh bash script, and a region_mover ruby script, and some 
> draining server support which you have to manually write to a znode 
> (really!). Also draining servers is only partially supported in LB operations 
> (LB does take that into account for roundRobin assignment, but not for normal 
> balance) 
> See 
> http://hbase.apache.org/book/node.management.html and HBASE-3071
> I think we should support graceful stop as a first class citizen. Thinking 
> about it, it seems that the difference between regionserver stop and graceful 
> stop is that regionserver stop will close the regions, but the master will 
> only assign them after the znode is deleted. 
> In the new master design (or even before), if we allow RS to be able to close 
> regions on its own (without master initiating it), then graceful stop becomes 
> regular stop. The RS already closes the regions cleanly, and will reject new 
> region assignments, so that we don't need much of the balancer or draining 
> server trickery. 
> This ties into the new master/AM redesign (HBASE-5487), but still deserves 
> it's own jira. Let's use this to brainstorm on the design. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Comment Edited] (HBASE-17369) Add ACL to the new region server drain related API

2017-10-12 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201381#comment-16201381
 ] 

Jerry He edited comment on HBASE-17369 at 10/12/17 11:18 PM:
-

The patch on HBASE-10367 has added the ACL.


was (Author: jinghe):
The patch on HBASe-10367 has added the ACL.

> Add ACL to the new region server drain related API
> --
>
> Key: HBASE-17369
> URL: https://issues.apache.org/jira/browse/HBASE-17369
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Jerry He
>Priority: Critical
>
> Add ACL to the new region server drain related API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Comment Edited] (HBASE-10367) RegionServer graceful stop / decommissioning

2017-10-12 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-10367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202369#comment-16202369
 ] 

Jerry He edited comment on HBASE-10367 at 10/12/17 11:09 PM:
-

These Admin APIs are only in 2.0 alpha's releases so far. We are still ok to 
change them, I assume?
Good point on passing servers being decommissioned to the coprocessor. I did 
not think about it.  It sounds reasonable.



was (Author: jinghe):
These Admin APIs are only in 2.0 alpha's releases so far. We are still ok to 
change them, I assume?
Good point on passing servers being decommissioned to the coprocessor. I did 
not thought about it.  It sounds reasonable.


> RegionServer graceful stop / decommissioning
> 
>
> Key: HBASE-10367
> URL: https://issues.apache.org/jira/browse/HBASE-10367
> Project: HBase
>  Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Jerry He
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-10367-master-2.patch, HBASE-10367-master.patch, 
> HBASE-10367-master.patch
>
>
> Right now, we have a weird way of node decommissioning / graceful stop, which 
> is a graceful_stop.sh bash script, and a region_mover ruby script, and some 
> draining server support which you have to manually write to a znode 
> (really!). Also draining servers is only partially supported in LB operations 
> (LB does take that into account for roundRobin assignment, but not for normal 
> balance) 
> See 
> http://hbase.apache.org/book/node.management.html and HBASE-3071
> I think we should support graceful stop as a first class citizen. Thinking 
> about it, it seems that the difference between regionserver stop and graceful 
> stop is that regionserver stop will close the regions, but the master will 
> only assign them after the znode is deleted. 
> In the new master design (or even before), if we allow RS to be able to close 
> regions on its own (without master initiating it), then graceful stop becomes 
> regular stop. The RS already closes the regions cleanly, and will reject new 
> region assignments, so that we don't need much of the balancer or draining 
> server trickery. 
> This ties into the new master/AM redesign (HBASE-5487), but still deserves 
> it's own jira. Let's use this to brainstorm on the design. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-10367) RegionServer graceful stop / decommissioning

2017-10-12 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-10367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-10367:
-
Attachment: HBASE-10367-master-2.patch

v2. updated the coprocessor hooks.  Also added a boolean to the decommission 
API to tell if to offload the regions from the decommissioned server or not.  
There is use case where we just want to 'isolate' the server, not offload the 
regions.

> RegionServer graceful stop / decommissioning
> 
>
> Key: HBASE-10367
> URL: https://issues.apache.org/jira/browse/HBASE-10367
> Project: HBase
>  Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Jerry He
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-10367-master-2.patch, HBASE-10367-master.patch, 
> HBASE-10367-master.patch
>
>
> Right now, we have a weird way of node decommissioning / graceful stop, which 
> is a graceful_stop.sh bash script, and a region_mover ruby script, and some 
> draining server support which you have to manually write to a znode 
> (really!). Also draining servers is only partially supported in LB operations 
> (LB does take that into account for roundRobin assignment, but not for normal 
> balance) 
> See 
> http://hbase.apache.org/book/node.management.html and HBASE-3071
> I think we should support graceful stop as a first class citizen. Thinking 
> about it, it seems that the difference between regionserver stop and graceful 
> stop is that regionserver stop will close the regions, but the master will 
> only assign them after the znode is deleted. 
> In the new master design (or even before), if we allow RS to be able to close 
> regions on its own (without master initiating it), then graceful stop becomes 
> regular stop. The RS already closes the regions cleanly, and will reject new 
> region assignments, so that we don't need much of the balancer or draining 
> server trickery. 
> This ties into the new master/AM redesign (HBASE-5487), but still deserves 
> it's own jira. Let's use this to brainstorm on the design. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-10367) RegionServer graceful stop / decommissioning

2017-10-12 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-10367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202369#comment-16202369
 ] 

Jerry He commented on HBASE-10367:
--

These Admin APIs are only in 2.0 alpha's releases so far. We are still ok to 
change them, I assume?
Good point on passing servers being decommissioned to the coprocessor. I did 
not thought about it.  It sounds reasonable.


> RegionServer graceful stop / decommissioning
> 
>
> Key: HBASE-10367
> URL: https://issues.apache.org/jira/browse/HBASE-10367
> Project: HBase
>  Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Jerry He
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-10367-master.patch, HBASE-10367-master.patch
>
>
> Right now, we have a weird way of node decommissioning / graceful stop, which 
> is a graceful_stop.sh bash script, and a region_mover ruby script, and some 
> draining server support which you have to manually write to a znode 
> (really!). Also draining servers is only partially supported in LB operations 
> (LB does take that into account for roundRobin assignment, but not for normal 
> balance) 
> See 
> http://hbase.apache.org/book/node.management.html and HBASE-3071
> I think we should support graceful stop as a first class citizen. Thinking 
> about it, it seems that the difference between regionserver stop and graceful 
> stop is that regionserver stop will close the regions, but the master will 
> only assign them after the znode is deleted. 
> In the new master design (or even before), if we allow RS to be able to close 
> regions on its own (without master initiating it), then graceful stop becomes 
> regular stop. The RS already closes the regions cleanly, and will reject new 
> region assignments, so that we don't need much of the balancer or draining 
> server trickery. 
> This ties into the new master/AM redesign (HBASE-5487), but still deserves 
> it's own jira. Let's use this to brainstorm on the design. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-10367) RegionServer graceful stop / decommissioning

2017-10-11 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-10367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-10367:
-
Fix Version/s: 2.0.0-alpha-4
   3.0.0
   Status: Patch Available  (was: Open)

> RegionServer graceful stop / decommissioning
> 
>
> Key: HBASE-10367
> URL: https://issues.apache.org/jira/browse/HBASE-10367
> Project: HBase
>  Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Jerry He
> Fix For: 3.0.0, 2.0.0-alpha-4
>
> Attachments: HBASE-10367-master.patch
>
>
> Right now, we have a weird way of node decommissioning / graceful stop, which 
> is a graceful_stop.sh bash script, and a region_mover ruby script, and some 
> draining server support which you have to manually write to a znode 
> (really!). Also draining servers is only partially supported in LB operations 
> (LB does take that into account for roundRobin assignment, but not for normal 
> balance) 
> See 
> http://hbase.apache.org/book/node.management.html and HBASE-3071
> I think we should support graceful stop as a first class citizen. Thinking 
> about it, it seems that the difference between regionserver stop and graceful 
> stop is that regionserver stop will close the regions, but the master will 
> only assign them after the znode is deleted. 
> In the new master design (or even before), if we allow RS to be able to close 
> regions on its own (without master initiating it), then graceful stop becomes 
> regular stop. The RS already closes the regions cleanly, and will reject new 
> region assignments, so that we don't need much of the balancer or draining 
> server trickery. 
> This ties into the new master/AM redesign (HBASE-5487), but still deserves 
> it's own jira. Let's use this to brainstorm on the design. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-17369) Add ACL to the new region server drain related API

2017-10-11 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201381#comment-16201381
 ] 

Jerry He commented on HBASE-17369:
--

The patch on HBASe-10367 has added the ACL.

> Add ACL to the new region server drain related API
> --
>
> Key: HBASE-17369
> URL: https://issues.apache.org/jira/browse/HBASE-17369
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Jerry He
>Priority: Critical
>
> Add ACL to the new region server drain related API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-10367) RegionServer graceful stop / decommissioning

2017-10-11 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-10367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201377#comment-16201377
 ] 

Jerry He commented on HBASE-10367:
--

Continue the work from HBASE-16010.  See  related comments from that issue.
Attached a patch to do 'decommission'.

> RegionServer graceful stop / decommissioning
> 
>
> Key: HBASE-10367
> URL: https://issues.apache.org/jira/browse/HBASE-10367
> Project: HBase
>  Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Jerry He
>
> Right now, we have a weird way of node decommissioning / graceful stop, which 
> is a graceful_stop.sh bash script, and a region_mover ruby script, and some 
> draining server support which you have to manually write to a znode 
> (really!). Also draining servers is only partially supported in LB operations 
> (LB does take that into account for roundRobin assignment, but not for normal 
> balance) 
> See 
> http://hbase.apache.org/book/node.management.html and HBASE-3071
> I think we should support graceful stop as a first class citizen. Thinking 
> about it, it seems that the difference between regionserver stop and graceful 
> stop is that regionserver stop will close the regions, but the master will 
> only assign them after the znode is deleted. 
> In the new master design (or even before), if we allow RS to be able to close 
> regions on its own (without master initiating it), then graceful stop becomes 
> regular stop. The RS already closes the regions cleanly, and will reject new 
> region assignments, so that we don't need much of the balancer or draining 
> server trickery. 
> This ties into the new master/AM redesign (HBASE-5487), but still deserves 
> it's own jira. Let's use this to brainstorm on the design. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (HBASE-10367) RegionServer graceful stop / decommissioning

2017-10-11 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-10367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-10367:
-
Attachment: HBASE-10367-master.patch

> RegionServer graceful stop / decommissioning
> 
>
> Key: HBASE-10367
> URL: https://issues.apache.org/jira/browse/HBASE-10367
> Project: HBase
>  Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Jerry He
> Attachments: HBASE-10367-master.patch
>
>
> Right now, we have a weird way of node decommissioning / graceful stop, which 
> is a graceful_stop.sh bash script, and a region_mover ruby script, and some 
> draining server support which you have to manually write to a znode 
> (really!). Also draining servers is only partially supported in LB operations 
> (LB does take that into account for roundRobin assignment, but not for normal 
> balance) 
> See 
> http://hbase.apache.org/book/node.management.html and HBASE-3071
> I think we should support graceful stop as a first class citizen. Thinking 
> about it, it seems that the difference between regionserver stop and graceful 
> stop is that regionserver stop will close the regions, but the master will 
> only assign them after the znode is deleted. 
> In the new master design (or even before), if we allow RS to be able to close 
> regions on its own (without master initiating it), then graceful stop becomes 
> regular stop. The RS already closes the regions cleanly, and will reject new 
> region assignments, so that we don't need much of the balancer or draining 
> server trickery. 
> This ties into the new master/AM redesign (HBASE-5487), but still deserves 
> it's own jira. Let's use this to brainstorm on the design. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Assigned] (HBASE-10367) RegionServer graceful stop / decommissioning

2017-10-11 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-10367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He reassigned HBASE-10367:


Assignee: Jerry He

> RegionServer graceful stop / decommissioning
> 
>
> Key: HBASE-10367
> URL: https://issues.apache.org/jira/browse/HBASE-10367
> Project: HBase
>  Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Jerry He
>
> Right now, we have a weird way of node decommissioning / graceful stop, which 
> is a graceful_stop.sh bash script, and a region_mover ruby script, and some 
> draining server support which you have to manually write to a znode 
> (really!). Also draining servers is only partially supported in LB operations 
> (LB does take that into account for roundRobin assignment, but not for normal 
> balance) 
> See 
> http://hbase.apache.org/book/node.management.html and HBASE-3071
> I think we should support graceful stop as a first class citizen. Thinking 
> about it, it seems that the difference between regionserver stop and graceful 
> stop is that regionserver stop will close the regions, but the master will 
> only assign them after the znode is deleted. 
> In the new master design (or even before), if we allow RS to be able to close 
> regions on its own (without master initiating it), then graceful stop becomes 
> regular stop. The RS already closes the regions cleanly, and will reject new 
> region assignments, so that we don't need much of the balancer or draining 
> server trickery. 
> This ties into the new master/AM redesign (HBASE-5487), but still deserves 
> it's own jira. Let's use this to brainstorm on the design. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Resolved] (HBASE-16010) Put draining function through Admin API

2017-10-11 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-16010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He resolved HBASE-16010.
--
Resolution: Fixed

> Put draining function through Admin API
> ---
>
> Key: HBASE-16010
> URL: https://issues.apache.org/jira/browse/HBASE-16010
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Matt Warhaftig
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16010-v3.patch, hbase-16010-v1.patch, 
> hbase-16010-v2.patch
>
>
> Currently, there is no Amdin API for draining function. Client has to 
> interact directly with Zookeeper draining node to add and remove draining 
> servers.
> For example, in draining_servers.rb:
> {code}
>   zkw = org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.new(config, 
> "draining_servers", nil)
>   parentZnode = zkw.drainingZNode
>   begin
> for server in servers
>   node = ZKUtil.joinZNode(parentZnode, server)
>   ZKUtil.createAndFailSilent(zkw, node)
> end
>   ensure
> zkw.close()
>   end
> {code}
> This is not good in cases like secure clusters with protected Zookeeper nodes.
> Let's put draining function through Admin API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Assigned] (HBASE-16010) Put draining function through Admin API

2017-10-11 Thread Jerry He (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-16010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He reassigned HBASE-16010:


Assignee: Matt Warhaftig  (was: Jerry He)

> Put draining function through Admin API
> ---
>
> Key: HBASE-16010
> URL: https://issues.apache.org/jira/browse/HBASE-16010
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Matt Warhaftig
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16010-v3.patch, hbase-16010-v1.patch, 
> hbase-16010-v2.patch
>
>
> Currently, there is no Amdin API for draining function. Client has to 
> interact directly with Zookeeper draining node to add and remove draining 
> servers.
> For example, in draining_servers.rb:
> {code}
>   zkw = org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.new(config, 
> "draining_servers", nil)
>   parentZnode = zkw.drainingZNode
>   begin
> for server in servers
>   node = ZKUtil.joinZNode(parentZnode, server)
>   ZKUtil.createAndFailSilent(zkw, node)
> end
>   ensure
> zkw.close()
>   end
> {code}
> This is not good in cases like secure clusters with protected Zookeeper nodes.
> Let's put draining function through Admin API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HBASE-16010) Put draining function through Admin API

2017-10-11 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201166#comment-16201166
 ] 

Jerry He commented on HBASE-16010:
--

I am going to close this ticket and post a patch on HBASE-10367, which will 
modify the API to decommission region servers (mark the drain and move off the 
regions).

> Put draining function through Admin API
> ---
>
> Key: HBASE-16010
> URL: https://issues.apache.org/jira/browse/HBASE-16010
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16010-v3.patch, hbase-16010-v1.patch, 
> hbase-16010-v2.patch
>
>
> Currently, there is no Amdin API for draining function. Client has to 
> interact directly with Zookeeper draining node to add and remove draining 
> servers.
> For example, in draining_servers.rb:
> {code}
>   zkw = org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.new(config, 
> "draining_servers", nil)
>   parentZnode = zkw.drainingZNode
>   begin
> for server in servers
>   node = ZKUtil.joinZNode(parentZnode, server)
>   ZKUtil.createAndFailSilent(zkw, node)
> end
>   ensure
> zkw.close()
>   end
> {code}
> This is not good in cases like secure clusters with protected Zookeeper nodes.
> Let's put draining function through Admin API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Comment Edited] (HBASE-16010) Put draining function through Admin API

2017-09-27 Thread Jerry He (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183340#comment-16183340
 ] 

Jerry He edited comment on HBASE-16010 at 9/27/17 10:06 PM:


I'd like to get a closure on this ticket. I am ready to add the 'decommission' 
logic as suggested by the folks here.
Decommission:  Mark the region servers as 'draining' so that no regions will be 
added. Also unload the regions on them.
What about 'recommission'?  Remove the 'draining' mode.  Then load the regions? 
 What regions to load?  API accepts a list of regions to region servers? Or 
only remove the 'draining' mode, and let the balancer takes care?


was (Author: jinghe):
I'd like to get a closure on this ticket. I am ready to add the 'decommission' 
logic as suggested by the folks here.
Decommission:  Mark the region servers as 'draining' so that no regions will be 
added. Also unload the regions on them.
What about 'recommission'?  Remove the 'draining' mode.  Then load the regions? 
 What regions to load?  API accepts a list of regions to region servers? Or 
only remove the 'draining' mode, and let the balancer take care?

> Put draining function through Admin API
> ---
>
> Key: HBASE-16010
> URL: https://issues.apache.org/jira/browse/HBASE-16010
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: hbase-16010-v1.patch, hbase-16010-v2.patch, 
> HBASE-16010-v3.patch
>
>
> Currently, there is no Amdin API for draining function. Client has to 
> interact directly with Zookeeper draining node to add and remove draining 
> servers.
> For example, in draining_servers.rb:
> {code}
>   zkw = org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.new(config, 
> "draining_servers", nil)
>   parentZnode = zkw.drainingZNode
>   begin
> for server in servers
>   node = ZKUtil.joinZNode(parentZnode, server)
>   ZKUtil.createAndFailSilent(zkw, node)
> end
>   ensure
> zkw.close()
>   end
> {code}
> This is not good in cases like secure clusters with protected Zookeeper nodes.
> Let's put draining function through Admin API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


  1   2   3   4   5   6   7   8   9   10   >