[jira] [Commented] (HBASE-6036) Add Cluster-level PB-based calls to HMasterInterface (minus file-format related calls)

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13279276#comment-13279276
 ] 

jirapos...@reviews.apache.org commented on HBASE-6036:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5157/#review7982
---


Patch looks good.  Minor nits below.  See what you think.


src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
https://reviews.apache.org/r/5157/#comment17326

So, its ok changing the public-facing API because 0.96 is going to be the 
singularity?



src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
https://reviews.apache.org/r/5157/#comment17327

FYI, convention is space after the comma -- its easier to read.



src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
https://reviews.apache.org/r/5157/#comment17328

You should write this as

LOG.info(Checking master connection, e);

Should it be warn?



src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java
https://reviews.apache.org/r/5157/#comment17329

Why we take it if unused?



src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java
https://reviews.apache.org/r/5157/#comment17330

Nit: You should look at the javadoc generated by this markup.  You'll see 
that it comes out nothing like how you have it here formatted.  For future.



src/main/java/org/apache/hadoop/hbase/master/HMaster.java
https://reviews.apache.org/r/5157/#comment17331

Two spaces in hbase and hadoop for 'tab' .  This 'return' is 4 or 6 spaces 
over?



src/main/java/org/apache/hadoop/hbase/master/HMaster.java
https://reviews.apache.org/r/5157/#comment17332

White space



src/main/java/org/apache/hadoop/hbase/master/HMaster.java
https://reviews.apache.org/r/5157/#comment17333

No need of the controller?  Would we ever need it?  If not, don't pass it?



src/main/java/org/apache/hadoop/hbase/master/HMaster.java
https://reviews.apache.org/r/5157/#comment17334

ditto



src/main/protobuf/Master.proto
https://reviews.apache.org/r/5157/#comment17335

Should this be MasterRunningRequest rather than IsMasterRunningRequest?

Or, is it just that you have a pattern going here where the Messages match 
the rpc in name?

If the latter, thats good enough for me.



src/main/protobuf/Master.proto
https://reviews.apache.org/r/5157/#comment17336

This method and message name is awkward.  To match your IsMasterRunning, 
this should be IsBalancer?


- Michael


On 2012-05-17 20:33:52, Gregory Chanan wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5157/
bq.  ---
bq.  
bq.  (Updated 2012-05-17 20:33:52)
bq.  
bq.  
bq.  Review request for hbase and Michael Stack.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Convert the cluster-level calls that do not touch the file-format related 
calls (see HBASE-5453). These are:
bq.  IsMasterRunning
bq.  Shutdown
bq.  StopMaster
bq.  Balance
bq.  LoadBalancerIs (was synchronousBalanceSwitch/balanceSwitch)
bq.  
bq.  
bq.  This addresses bug HBASE-6036.
bq.  https://issues.apache.org/jira/browse/HBASE-6036
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 007d90b 
bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 
5cac9af 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 80c2165 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 16ac781 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java 
4348d20 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/MasterProtos.java 
944e403 
bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 87a7a06 
bq.src/main/protobuf/Master.proto PRE-CREATION 
bq.src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java be52644 
bq.
src/test/java/org/apache/hadoop/hbase/master/TestHMasterRPCException.java 
9ff83c5 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
 a24f937 
bq.  
bq.  Diff: https://reviews.apache.org/r/5157/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Ran unit tests, all passed.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Gregory
bq.  
bq.



 Add Cluster-level PB-based calls to HMasterInterface (minus file-format 
 related calls)
 --

 Key: HBASE-6036
 URL: https://issues.apache.org/jira/browse/HBASE-6036
 Project: HBase
  

[jira] [Commented] (HBASE-6036) Add Cluster-level PB-based calls to HMasterInterface (minus file-format related calls)

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13279355#comment-13279355
 ] 

jirapos...@reviews.apache.org commented on HBASE-6036:
--



bq.  On 2012-05-18 22:43:10, Michael Stack wrote:
bq.   Patch looks good.  Minor nits below.  See what you think.

Thanks for the review.


bq.  On 2012-05-18 22:43:10, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java, 
line 704
bq.   https://reviews.apache.org/r/5157/diff/1/?file=109492#file109492line704
bq.  
bq.   FYI, convention is space after the comma -- its easier to read.

Fixed.


bq.  On 2012-05-18 22:43:10, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java, 
line 1657
bq.   
https://reviews.apache.org/r/5157/diff/1/?file=109492#file109492line1657
bq.  
bq.   You should write this as
bq.   
bq.   LOG.info(Checking master connection, e);
bq.   
bq.   Should it be warn?

Changed.


bq.  On 2012-05-18 22:43:10, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/master/HMaster.java, line 1208
bq.   
https://reviews.apache.org/r/5157/diff/1/?file=109494#file109494line1208
bq.  
bq.   White space

Fixed.


bq.  On 2012-05-18 22:43:10, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java, line 82
bq.   https://reviews.apache.org/r/5157/diff/1/?file=109493#file109493line82
bq.  
bq.   Why we take it if unused?

protobuf generates the function signature like that.  When I implement 
HBASE-6039, I'm going to just take the functions as they are generated by 
protobufs (see RegionServerStatusProtocol for an example).


bq.  On 2012-05-18 22:43:10, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/master/HMaster.java, line 1761
bq.   
https://reviews.apache.org/r/5157/diff/1/?file=109494#file109494line1761
bq.  
bq.   No need of the controller?  Would we ever need it?  If not, don't 
pass it?

Covered above.


bq.  On 2012-05-18 22:43:10, Michael Stack wrote:
bq.   src/main/protobuf/Master.proto, line 56
bq.   https://reviews.apache.org/r/5157/diff/1/?file=109498#file109498line56
bq.  
bq.   Should this be MasterRunningRequest rather than 
IsMasterRunningRequest?
bq.   
bq.   Or, is it just that you have a pattern going here where the Messages 
match the rpc in name?
bq.   
bq.   If the latter, thats good enough for me.

It's the latter.


bq.  On 2012-05-18 22:43:10, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/master/HMaster.java, line 1077
bq.   
https://reviews.apache.org/r/5157/diff/1/?file=109494#file109494line1077
bq.  
bq.   Two spaces in hbase and hadoop for 'tab' .  This 'return' is 4 or 6 
spaces over?

Fixed.


bq.  On 2012-05-18 22:43:10, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java, line 236
bq.   https://reviews.apache.org/r/5157/diff/1/?file=109493#file109493line236
bq.  
bq.   Nit: You should look at the javadoc generated by this markup.  
You'll see that it comes out nothing like how you have it here formatted.  For 
future.

Fixed, thanks for pointing that out.


bq.  On 2012-05-18 22:43:10, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java, line 1433
bq.   
https://reviews.apache.org/r/5157/diff/1/?file=109491#file109491line1433
bq.  
bq.   So, its ok changing the public-facing API because 0.96 is going to 
be the singularity?

That's true, but no point in breaking clients of this class if we don't need to.


bq.  On 2012-05-18 22:43:10, Michael Stack wrote:
bq.   src/main/protobuf/Master.proto, line 133
bq.   https://reviews.apache.org/r/5157/diff/1/?file=109498#file109498line133
bq.  
bq.   This method and message name is awkward.  To match your 
IsMasterRunning, this should be IsBalancer?

Agree that it is awkward.  IsMasterRunning isn't a good match, because that is 
a question, whereas this controls whether the load balancer should be on or off 
(that is, an action).  This is the old balanceSwitch(boolean).

How about setBalancerRunning(boolean)? 


- Gregory


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5157/#review7982
---


On 2012-05-17 20:33:52, Gregory Chanan wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5157/
bq.  ---
bq.  
bq.  (Updated 2012-05-17 20:33:52)
bq.  
bq.  
bq.  Review request for hbase and Michael Stack.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Convert the cluster-level calls that do not touch the 

[jira] [Commented] (HBASE-6036) Add Cluster-level PB-based calls to HMasterInterface (minus file-format related calls)

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13279363#comment-13279363
 ] 

jirapos...@reviews.apache.org commented on HBASE-6036:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5157/
---

(Updated 2012-05-19 00:24:57.479662)


Review request for hbase and Michael Stack.


Changes
---

Update for Stack's comments and latest trunk.


Summary
---

Convert the cluster-level calls that do not touch the file-format related calls 
(see HBASE-5453). These are:
IsMasterRunning
Shutdown
StopMaster
Balance
LoadBalancerIs (was synchronousBalanceSwitch/balanceSwitch)


This addresses bug HBASE-6036.
https://issues.apache.org/jira/browse/HBASE-6036


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 007d90b 
  src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 07334f8 
  src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 80c2165 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java a49651b 
  src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java 90cb53d 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/MasterProtos.java 
944e403 
  src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java dd18ed9 
  src/main/protobuf/Master.proto PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java be52644 
  src/test/java/org/apache/hadoop/hbase/master/TestHMasterRPCException.java 
9ff83c5 
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
 a24f937 

Diff: https://reviews.apache.org/r/5157/diff


Testing
---

Ran unit tests, all passed.


Thanks,

Gregory



 Add Cluster-level PB-based calls to HMasterInterface (minus file-format 
 related calls)
 --

 Key: HBASE-6036
 URL: https://issues.apache.org/jira/browse/HBASE-6036
 Project: HBase
  Issue Type: Task
  Components: ipc, master, migration
Reporter: Gregory Chanan
Assignee: Gregory Chanan
 Fix For: 0.96.0

 Attachments: HBASE-6036.patch


 This should be a subtask of HBASE-5445, but since that is a subtask, I can't 
 also make this a subtask (apparently).
 Convert the cluster-level calls that do not touch the file-format related 
 calls (see HBASE-5453).  These are:
 IsMasterRunning
 Shutdown
 StopMaster
 Balance
 LoadBalancerIs (was synchronousBalanceSwitch/balanceSwitch)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6036) Add Cluster-level PB-based calls to HMasterInterface (minus file-format related calls)

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13279367#comment-13279367
 ] 

jirapos...@reviews.apache.org commented on HBASE-6036:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5157/#review7985
---



src/main/protobuf/Master.proto
https://reviews.apache.org/r/5157/#comment17347

BalancerSwitchRequest, balancerSwitch the method name?

Do whatever you think G. I'm not going to hold up the patch over this 
naming.


- Michael


On 2012-05-19 00:24:57, Gregory Chanan wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5157/
bq.  ---
bq.  
bq.  (Updated 2012-05-19 00:24:57)
bq.  
bq.  
bq.  Review request for hbase and Michael Stack.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Convert the cluster-level calls that do not touch the file-format related 
calls (see HBASE-5453). These are:
bq.  IsMasterRunning
bq.  Shutdown
bq.  StopMaster
bq.  Balance
bq.  LoadBalancerIs (was synchronousBalanceSwitch/balanceSwitch)
bq.  
bq.  
bq.  This addresses bug HBASE-6036.
bq.  https://issues.apache.org/jira/browse/HBASE-6036
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 007d90b 
bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 
07334f8 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 80c2165 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java a49651b 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java 
90cb53d 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/MasterProtos.java 
944e403 
bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java dd18ed9 
bq.src/main/protobuf/Master.proto PRE-CREATION 
bq.src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java be52644 
bq.
src/test/java/org/apache/hadoop/hbase/master/TestHMasterRPCException.java 
9ff83c5 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
 a24f937 
bq.  
bq.  Diff: https://reviews.apache.org/r/5157/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Ran unit tests, all passed.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Gregory
bq.  
bq.



 Add Cluster-level PB-based calls to HMasterInterface (minus file-format 
 related calls)
 --

 Key: HBASE-6036
 URL: https://issues.apache.org/jira/browse/HBASE-6036
 Project: HBase
  Issue Type: Task
  Components: ipc, master, migration
Reporter: Gregory Chanan
Assignee: Gregory Chanan
 Fix For: 0.96.0

 Attachments: HBASE-6036.patch


 This should be a subtask of HBASE-5445, but since that is a subtask, I can't 
 also make this a subtask (apparently).
 Convert the cluster-level calls that do not touch the file-format related 
 calls (see HBASE-5453).  These are:
 IsMasterRunning
 Shutdown
 StopMaster
 Balance
 LoadBalancerIs (was synchronousBalanceSwitch/balanceSwitch)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277596#comment-13277596
 ] 

jirapos...@reviews.apache.org commented on HBASE-5453:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/
---

(Updated 2012-05-17 05:58:11.354772)


Review request for hbase.


Changes
---

Address Gregory's two review comments.


Summary
---

A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
  New  class to hold clusterid in.
M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
  Make it so can do pb serialization.  Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
  Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
  Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
  Make it so can do pb serialization.  Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
  ClusterId under ZK got renamed as ZKClusterId
M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
  Hide the Reference#Range enums.  Don't let them out of this class.
  Make it so can do pb serialization.
M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
  Use new methods on Reference for getting top and bottom.
M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
  ClusterId under zk has been renamed ZKClusterId.
  Use new ClusterId class too.
M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
  Use new clusterid class.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
  Move the RegionInfo convertion up into HRegionInfo instead of here.
  Added generic toDelimitedByteArray helper.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
  Use HRegionInfo convertions instead.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
  Use HRegionInfo convertions instead.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
  Use new utility writing out .regioninfo files.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
  Formatting.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
  Range in Reference is no longer public.
  Range in Reference is no longer public.
M 
b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
M 
b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
  ClusterId got renamed ZKClusterId
M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
  Use new serialization utlity in HTD.
M  b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
  Generic method for writing dot file content.
M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
  Reference#Range is not public any more
M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java
  Deprecated getHRegionInfo, etc.
D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java
A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java
  Rename
A b/src/main/protobuf/ClusterId.proto
  Added file for ClusterId only since its written to fs and to zk.
A b/src/main/protobuf/FS.proto
  Protos for fs files.
M b/src/main/protobuf/ZooKeeper.proto
  Moved ClusterId out to own proto file
M b/src/main/protobuf/hbase.proto
  Added TableSchema and ColumnFamilySchema


This addresses bug hbase-5453.
https://issues.apache.org/jira/browse/hbase-5453


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/ClusterId.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 5862f15 
  src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 8d83ff3 
  src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java af89e3e 
  src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5cac9af 
  src/main/java/org/apache/hadoop/hbase/io/Reference.java 6360059 
  src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 
9e4ada9 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java 947ec5f 
  src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 5052878 
  src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java ccc964e 
  src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java dabfbab 
  src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java 45cb6cf 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClusterIdProtos.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/FSProtos.java 
PRE-CREATION 
  

[jira] [Commented] (HBASE-6036) Add Cluster-level PB-based calls to HMasterInterface (minus file-format related calls)

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13278229#comment-13278229
 ] 

jirapos...@reviews.apache.org commented on HBASE-6036:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5157/
---

Review request for hbase and Michael Stack.


Summary
---

Convert the cluster-level calls that do not touch the file-format related calls 
(see HBASE-5453). These are:
IsMasterRunning
Shutdown
StopMaster
Balance
LoadBalancerIs (was synchronousBalanceSwitch/balanceSwitch)


This addresses bug HBASE-6036.
https://issues.apache.org/jira/browse/HBASE-6036


Diffs
-

  src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 007d90b 
  src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5cac9af 
  src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 80c2165 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java 16ac781 
  src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java 4348d20 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/MasterProtos.java 
944e403 
  src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 87a7a06 
  src/main/protobuf/Master.proto PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java be52644 
  src/test/java/org/apache/hadoop/hbase/master/TestHMasterRPCException.java 
9ff83c5 
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
 a24f937 

Diff: https://reviews.apache.org/r/5157/diff


Testing
---

Ran unit tests, all passed.


Thanks,

Gregory



 Add Cluster-level PB-based calls to HMasterInterface (minus file-format 
 related calls)
 --

 Key: HBASE-6036
 URL: https://issues.apache.org/jira/browse/HBASE-6036
 Project: HBase
  Issue Type: Task
  Components: ipc, master, migration
Reporter: Gregory Chanan
Assignee: Gregory Chanan
 Fix For: 0.96.0


 This should be a subtask of HBASE-5445, but since that is a subtask, I can't 
 also make this a subtask (apparently).
 Convert the cluster-level calls that do not touch the file-format related 
 calls (see HBASE-5453).  These are:
 IsMasterRunning
 Shutdown
 StopMaster
 Balance
 LoadBalancerIs (was synchronousBalanceSwitch/balanceSwitch)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13278263#comment-13278263
 ] 

jirapos...@reviews.apache.org commented on HBASE-5453:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/#review7963
---

Ship it!



src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
https://reviews.apache.org/r/5130/#comment17303

Your logic sounds correct and I agree we should avoid having to read the 
file each time.

It would be cool to just see an example of the old writable vs new pb size 
to ensure our intuition is correct.

Worst case (let's say they happen to be the same) we could write out the 
new pb stuff concatenated with the old way of writing and then they are 
guaranteed to be different sizes.  Hopefully we don't have to do something like 
that, though.


- Gregory


On 2012-05-17 05:58:11, Michael Stack wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5130/
bq.  ---
bq.  
bq.  (Updated 2012-05-17 05:58:11)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
bq.New  class to hold clusterid in.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
bq.Make it so can do pb serialization.  Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
bq.Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
bq.Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
bq.Make it so can do pb serialization.  Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
bq.ClusterId under ZK got renamed as ZKClusterId
bq.  M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
bq.Hide the Reference#Range enums.  Don't let them out of this class.
bq.Make it so can do pb serialization.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
bq.Use new methods on Reference for getting top and bottom.
bq.  M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
bq.ClusterId under zk has been renamed ZKClusterId.
bq.Use new ClusterId class too.
bq.  M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
bq.Use new clusterid class.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
bq.Move the RegionInfo convertion up into HRegionInfo instead of here.
bq.Added generic toDelimitedByteArray helper.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
bq.Use HRegionInfo convertions instead.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
bq.Use HRegionInfo convertions instead.
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
bq.Use new utility writing out .regioninfo files.
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
bq.Formatting.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
bq.Range in Reference is no longer public.
bq.Range in Reference is no longer public.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
bq.ClusterId got renamed ZKClusterId
bq.  M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
bq.Use new serialization utlity in HTD.
bq.  M  b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
bq.Generic method for writing dot file content.
bq.  M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
bq.Reference#Range is not public any more
bq.  M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java
bq.Deprecated getHRegionInfo, etc.
bq.  D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java
bq.  A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java
bq.Rename
bq.  A b/src/main/protobuf/ClusterId.proto
bq.Added file for ClusterId only since its written to fs and to zk.
bq.  A b/src/main/protobuf/FS.proto
bq.Protos for fs files.
bq.  M b/src/main/protobuf/ZooKeeper.proto
bq.Moved ClusterId out to own proto file
bq.  M b/src/main/protobuf/hbase.proto
bq.Added TableSchema and ColumnFamilySchema
bq.  
bq.  
bq.  This 

[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13276887#comment-13276887
 ] 

jirapos...@reviews.apache.org commented on HBASE-5453:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/
---

(Updated 2012-05-16 17:02:35.527708)


Review request for hbase.


Changes
---

Gregory.  I addressed your comments.

I've changed the format of .regioninfo and .tableinfo. Now instead of 
serialized Writable followed by toString of the serialized object, instead its 
just the serialized pb. This removes our having a human readable 
.regioninfo/.tablinfo file but my guess no one relied on this anyways.

Having just serialized content in the file means a check of file length should 
be enough figuring whether the file properly serialized. If ever a chance that 
a Writable + its toString + two '\n' characters was equal to a serialized pb, 
I'd think this a pathological state. If this state is not cleared up 
'naturally' by splits or a schema change, then lets deal if it happens.

I only need this length-checking in one place on region open. I want to avoid 
reading the .regioninfo file on region open. The alternative means more load on 
NN and DNs at region open time which could be problematic at big-bang cluster 
start (Thinking 500 nodes w/ 80k regions, an actual known case -- this is the 
case I have in mind when I'm trying to avoid more load on NN/DNs).

Otherwise, Gregory's comments led to me to check and I was missing convertion 
of fs files to pb in all cases (I was just reading the clusterid and 
hbase.version files, not converting if still Writable). This should be fixed 
now.

There are some failing tests still but running by hadoopqa to see what it says 
anyways. Also putting up on rb to get feedback if problem w/ this approach.


Summary
---

A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
  New  class to hold clusterid in.
M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
  Make it so can do pb serialization.  Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
  Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
  Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
  Make it so can do pb serialization.  Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
  ClusterId under ZK got renamed as ZKClusterId
M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
  Hide the Reference#Range enums.  Don't let them out of this class.
  Make it so can do pb serialization.
M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
  Use new methods on Reference for getting top and bottom.
M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
  ClusterId under zk has been renamed ZKClusterId.
  Use new ClusterId class too.
M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
  Use new clusterid class.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
  Move the RegionInfo convertion up into HRegionInfo instead of here.
  Added generic toDelimitedByteArray helper.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
  Use HRegionInfo convertions instead.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
  Use HRegionInfo convertions instead.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
  Use new utility writing out .regioninfo files.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
  Formatting.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
  Range in Reference is no longer public.
  Range in Reference is no longer public.
M 
b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
M 
b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
  ClusterId got renamed ZKClusterId
M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
  Use new serialization utlity in HTD.
M  b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
  Generic method for writing dot file content.
M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
  Reference#Range is not public any more
M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java
  Deprecated getHRegionInfo, etc.
D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java
A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java
  Rename
A b/src/main/protobuf/ClusterId.proto
  Added file for ClusterId only since 

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277025#comment-13277025
 ] 

jirapos...@reviews.apache.org commented on HBASE-5986:
--



bq.  On 2012-05-16 05:20:29, ramkrishna vasudevan wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 402
bq.   https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line402
bq.  
bq.   Blocking time out is one factor on which we will be waiting.  So 
this blockingtimeout should be in lieu with 'fileSplitTimeout'?

bq. From my understanding, fileSplitTimeout is a regionserver property, while 
the timeout parameter in this patch is a client side property, since the 
blocking will happen on the client. I think hbase.client.operation.timeout 
can serve us well. HConstants state that it is Default HBase client operation 
timeout, which is tantamount to a blocking call.


- enis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5133/#review7928
---


On 2012-05-16 01:53:09, enis wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5133/
bq.  ---
bq.  
bq.  (Updated 2012-05-16 01:53:09)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  We found this issue when running large scale ingestion tests for 
HBASE-5754. The problem is that the .META. table updates are not atomic while 
splitting a region. In SplitTransaction, there is a time lap between the 
marking the parent offline, and adding of daughters to the META table. This can 
result in clients using MetaScanner, of HTable.getStartEndKeys (used by the 
TableInputFormat) missing regions which are made just offline, but the 
daughters are not added yet.
bq.  
bq.  This patch is the approach 2 mentioned in the issue comments, mainly 
during META scan, if we detect that the region is split, we block until the 
information for the child regions are available in META and manually feed those 
rows to the MetaScanner. Although approach 3 (using local region transactions) 
seems cleaner, they are not available under branch 0.92, which I think should 
also incorporate this fix. I'll provide ports once we are clear for trunk. 
bq.  
bq.  Also this patch does not fix MetaReader (see 
https://issues.apache.org/jira/browse/HBASE-3475). 
bq.  
bq.  
bq.  This addresses bug HBASE-5986.
bq.  https://issues.apache.org/jira/browse/HBASE-5986
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512 
bq.src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4 
bq.src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
 a8091e6 
bq.  
bq.  Diff: https://reviews.apache.org/r/5133/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  added extensive tests under TestEndToEndSplitTranscation, and ran existing 
unit tests. 
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  enis
bq.  
bq.



 Clients can see holes in the META table when regions are being split
 

 Key: HBASE-5986
 URL: https://issues.apache.org/jira/browse/HBASE-5986
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Attachments: HBASE-5986-test_v1.patch


 We found this issue when running large scale ingestion tests for HBASE-5754. 
 The problem is that the .META. table updates are not atomic while splitting a 
 region. In SplitTransaction, there is a time lap between the marking the 
 parent offline, and adding of daughters to the META table. This can result in 
 clients using MetaScanner, of HTable.getStartEndKeys (used by the 
 TableInputFormat) missing regions which are made just offline, but the 
 daughters are not added yet. 
 This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277024#comment-13277024
 ] 

jirapos...@reviews.apache.org commented on HBASE-5986:
--



bq.  On 2012-05-16 02:52:04, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 391
bq.   https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line391
bq.  
bq.   null is always returned by getRegionResultBlocking() in case of time 
out.
bq.   How do we deal with that ?
bq.  

we are checking for null return value for resultA, and resultB. But I'll also 
add a LOG warning. 


bq.  On 2012-05-16 02:52:04, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 390
bq.   https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line390
bq.  
bq.   Do we have to instantiate HTable every time ?

This is a bit tricky. Ideally we should not. But there is no close() on 
MetaScannerVisitor, so we cannot close the HTable properly if we reuse the 
HTable across calls to processRow(). We can add a close() method and call it, 
or obtain the HTable from the context, but that would imply changing the class 
signature for MetaScannerVisitor. I assumed since we are reusing HConnection's 
HTable creation is cheap, is that not the case, wdyt? 


bq.  On 2012-05-16 02:52:04, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 360
bq.   https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line360
bq.  
bq.   Can we add a config param for blockingTimeout ?

please see below


bq.  On 2012-05-16 02:52:04, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 398
bq.   https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line398
bq.  
bq.   We shouldn't be passing blockingTimeout here.
bq.   We need to consider the amount of time spent in the call @ line 388.

agreed


bq.  On 2012-05-16 02:52:04, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 430
bq.   https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line430
bq.  
bq.   Please restore interrupted state of the thread.

agreed


- enis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5133/#review7925
---


On 2012-05-16 01:53:09, enis wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5133/
bq.  ---
bq.  
bq.  (Updated 2012-05-16 01:53:09)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  We found this issue when running large scale ingestion tests for 
HBASE-5754. The problem is that the .META. table updates are not atomic while 
splitting a region. In SplitTransaction, there is a time lap between the 
marking the parent offline, and adding of daughters to the META table. This can 
result in clients using MetaScanner, of HTable.getStartEndKeys (used by the 
TableInputFormat) missing regions which are made just offline, but the 
daughters are not added yet.
bq.  
bq.  This patch is the approach 2 mentioned in the issue comments, mainly 
during META scan, if we detect that the region is split, we block until the 
information for the child regions are available in META and manually feed those 
rows to the MetaScanner. Although approach 3 (using local region transactions) 
seems cleaner, they are not available under branch 0.92, which I think should 
also incorporate this fix. I'll provide ports once we are clear for trunk. 
bq.  
bq.  Also this patch does not fix MetaReader (see 
https://issues.apache.org/jira/browse/HBASE-3475). 
bq.  
bq.  
bq.  This addresses bug HBASE-5986.
bq.  https://issues.apache.org/jira/browse/HBASE-5986
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512 
bq.src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4 
bq.src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
 a8091e6 
bq.  
bq.  Diff: https://reviews.apache.org/r/5133/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  added extensive tests under TestEndToEndSplitTranscation, and ran existing 
unit tests. 
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  enis
bq.  
bq.



 Clients can see holes in the META table when regions are being split
 

 Key: HBASE-5986
 URL: 

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277031#comment-13277031
 ] 

jirapos...@reviews.apache.org commented on HBASE-5986:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5133/#review7938
---



src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
https://reviews.apache.org/r/5133/#comment17261

I was asking how do we convey the fact that MetaScanner has timed out to 
client.

I don't see exception being thrown which contains timeout information.


- Ted


On 2012-05-16 01:53:09, enis wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5133/
bq.  ---
bq.  
bq.  (Updated 2012-05-16 01:53:09)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  We found this issue when running large scale ingestion tests for 
HBASE-5754. The problem is that the .META. table updates are not atomic while 
splitting a region. In SplitTransaction, there is a time lap between the 
marking the parent offline, and adding of daughters to the META table. This can 
result in clients using MetaScanner, of HTable.getStartEndKeys (used by the 
TableInputFormat) missing regions which are made just offline, but the 
daughters are not added yet.
bq.  
bq.  This patch is the approach 2 mentioned in the issue comments, mainly 
during META scan, if we detect that the region is split, we block until the 
information for the child regions are available in META and manually feed those 
rows to the MetaScanner. Although approach 3 (using local region transactions) 
seems cleaner, they are not available under branch 0.92, which I think should 
also incorporate this fix. I'll provide ports once we are clear for trunk. 
bq.  
bq.  Also this patch does not fix MetaReader (see 
https://issues.apache.org/jira/browse/HBASE-3475). 
bq.  
bq.  
bq.  This addresses bug HBASE-5986.
bq.  https://issues.apache.org/jira/browse/HBASE-5986
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512 
bq.src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4 
bq.src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
 a8091e6 
bq.  
bq.  Diff: https://reviews.apache.org/r/5133/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  added extensive tests under TestEndToEndSplitTranscation, and ran existing 
unit tests. 
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  enis
bq.  
bq.



 Clients can see holes in the META table when regions are being split
 

 Key: HBASE-5986
 URL: https://issues.apache.org/jira/browse/HBASE-5986
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Attachments: HBASE-5986-test_v1.patch


 We found this issue when running large scale ingestion tests for HBASE-5754. 
 The problem is that the .META. table updates are not atomic while splitting a 
 region. In SplitTransaction, there is a time lap between the marking the 
 parent offline, and adding of daughters to the META table. This can result in 
 clients using MetaScanner, of HTable.getStartEndKeys (used by the 
 TableInputFormat) missing regions which are made just offline, but the 
 daughters are not added yet. 
 This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6004) Adding more logging to help debugging MR job

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277140#comment-13277140
 ] 

jirapos...@reviews.apache.org commented on HBASE-6004:
--



bq.  On 2012-05-15 23:11:04, Andrew Purtell wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java, line 
54
bq.   https://reviews.apache.org/r/5131/diff/1/?file=109074#file109074line54
bq.  
bq.   We control logging via log4j.properties files. Here also new 
configuration in another file to switch some additional logging on and off. 
Would it make more sense to create a logger class e.g. 
ScannerCallable.ActivityLog that logs at TRACE level and update 
log4j.properties with 
log4j.logger.org.apache.hadoop.hbase.client.ScannerCallable.ActivityLog = TRACE 
(default is INFO, i.e. disabled). Just a thought. We did something like this 
for security audit logging.
bq.  
bq.  Jimmy Xiang wrote:
bq.  It is a good idea.  Because of other parameters introduced, it's 
better to put them in the same place for now.

Fair enough, there isn't a way to get the value of properties in the 
log4j.properties file using either commons logging API nor java.util.logging 
that I can see.


- Andrew


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5131/#review7918
---


On 2012-05-16 02:56:16, Jimmy Xiang wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5131/
bq.  ---
bq.  
bq.  (Updated 2012-05-16 02:56:16)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Added some logging for MR debugging in case scanner times out.  The 
logging is disabled by default.
bq.  It will be helpful to know how much time spent in the scanner and, how 
much in the mapper task.
bq.  In case scanner issue, it is helpful to know the region server id, last 
successful rows and so on.
bq.  
bq.  
bq.  This addresses bug HBASE-6004.
bq.  https://issues.apache.org/jira/browse/HBASE-6004
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 
46b1c56 
bq.src/main/java/org/apache/hadoop/hbase/mapred/TableRecordReaderImpl.java 
42569fb 
bq.
src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java 
1c8a393 
bq.  
bq.  Diff: https://reviews.apache.org/r/5131/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Jimmy
bq.  
bq.



 Adding more logging to help debugging MR job
 

 Key: HBASE-6004
 URL: https://issues.apache.org/jira/browse/HBASE-6004
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.94.0, 0.96.0
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
 Fix For: 0.94.0, 0.96.0

 Attachments: hbase-6004.patch


 MR job sometime fails because scanner expired. In this case, it will be 
 helpful to know the last successful row, the ip of the region sever, and so 
 on.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277161#comment-13277161
 ] 

jirapos...@reviews.apache.org commented on HBASE-5986:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5133/#review7939
---


Looks great Enis.  Thanks for having a go at this.  Loads of white space 
introduced which you might fix in v2.  Otherwise a few little items below


src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java
https://reviews.apache.org/r/5133/#comment17262

Bit of javadoc to say this is best-effort.

Also, does this belong in MetaReader (Won't hold you to it... these two 
classes, a MetaReader vs MetaEditor are kinda silly... this whole catalog 
package needs killing).



src/main/java/org/apache/hadoop/hbase/client/HTable.java
https://reviews.apache.org/r/5133/#comment17264

Why you remove this?  We don't return these any more?  Offline I think is 
'dead', unused now.  Split not.



src/main/java/org/apache/hadoop/hbase/client/HTable.java
https://reviews.apache.org/r/5133/#comment17265

This seems like a good refactoring.



src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
https://reviews.apache.org/r/5133/#comment17266

Good



src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
https://reviews.apache.org/r/5133/#comment17267

Should this be public?  Should it remain internal to the HTable hidden?



src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
https://reviews.apache.org/r/5133/#comment17268

Yeah, it could be a long time before a region comes on line if its in a GC 
or some such.



src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
https://reviews.apache.org/r/5133/#comment17269

Yeah, what Ted says... Can you close when done?  See MetaEditor/MetaReader. 
 They do this a bunch.  Closing means for sure the zk and connection resources 
will be cleaned up afterward.  Its reference counting so keepign around an 
HTable could mess it up.



src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
https://reviews.apache.org/r/5133/#comment17270

Yeah, so, what happens if daughter has split by the time I get here?



src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
https://reviews.apache.org/r/5133/#comment17271

So if interrupted or we don't find it by the time the blocking time has 
passed, we just return null?   What you reckon?  We should at least complain?



src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
https://reviews.apache.org/r/5133/#comment17272

Does the scan of meta start at first table region?


- Michael


On 2012-05-16 01:53:09, enis wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5133/
bq.  ---
bq.  
bq.  (Updated 2012-05-16 01:53:09)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  We found this issue when running large scale ingestion tests for 
HBASE-5754. The problem is that the .META. table updates are not atomic while 
splitting a region. In SplitTransaction, there is a time lap between the 
marking the parent offline, and adding of daughters to the META table. This can 
result in clients using MetaScanner, of HTable.getStartEndKeys (used by the 
TableInputFormat) missing regions which are made just offline, but the 
daughters are not added yet.
bq.  
bq.  This patch is the approach 2 mentioned in the issue comments, mainly 
during META scan, if we detect that the region is split, we block until the 
information for the child regions are available in META and manually feed those 
rows to the MetaScanner. Although approach 3 (using local region transactions) 
seems cleaner, they are not available under branch 0.92, which I think should 
also incorporate this fix. I'll provide ports once we are clear for trunk. 
bq.  
bq.  Also this patch does not fix MetaReader (see 
https://issues.apache.org/jira/browse/HBASE-3475). 
bq.  
bq.  
bq.  This addresses bug HBASE-5986.
bq.  https://issues.apache.org/jira/browse/HBASE-5986
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512 
bq.src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4 
bq.src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
 a8091e6 
bq.  
bq.  Diff: https://reviews.apache.org/r/5133/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  added extensive tests under 

[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277200#comment-13277200
 ] 

jirapos...@reviews.apache.org commented on HBASE-5453:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/#review7944
---


In a previous comment you said (about the HTableDescriptor/HColumnDesriptor pb 
stuff):
Well, Andrew beat us both to it over in his REST pb stuff.  We need to 
reconcile his w/ ours too

Where is his stuff?  I couldn't find it.  Should we create a JIRA about 
reconciling? It would be nice to have something, however imperfect, up in trunk 
to work against, then we could fix up later.


src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
https://reviews.apache.org/r/5130/#comment17274

This isn't ever read?



src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
https://reviews.apache.org/r/5130/#comment17273

This constructor sets pbMade to false, but it should be true in this case, 
right?


- Gregory


On 2012-05-16 17:02:35, Michael Stack wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5130/
bq.  ---
bq.  
bq.  (Updated 2012-05-16 17:02:35)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
bq.New  class to hold clusterid in.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
bq.Make it so can do pb serialization.  Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
bq.Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
bq.Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
bq.Make it so can do pb serialization.  Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
bq.ClusterId under ZK got renamed as ZKClusterId
bq.  M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
bq.Hide the Reference#Range enums.  Don't let them out of this class.
bq.Make it so can do pb serialization.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
bq.Use new methods on Reference for getting top and bottom.
bq.  M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
bq.ClusterId under zk has been renamed ZKClusterId.
bq.Use new ClusterId class too.
bq.  M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
bq.Use new clusterid class.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
bq.Move the RegionInfo convertion up into HRegionInfo instead of here.
bq.Added generic toDelimitedByteArray helper.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
bq.Use HRegionInfo convertions instead.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
bq.Use HRegionInfo convertions instead.
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
bq.Use new utility writing out .regioninfo files.
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
bq.Formatting.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
bq.Range in Reference is no longer public.
bq.Range in Reference is no longer public.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
bq.ClusterId got renamed ZKClusterId
bq.  M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
bq.Use new serialization utlity in HTD.
bq.  M  b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
bq.Generic method for writing dot file content.
bq.  M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
bq.Reference#Range is not public any more
bq.  M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java
bq.Deprecated getHRegionInfo, etc.
bq.  D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java
bq.  A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java
bq.Rename
bq.  A b/src/main/protobuf/ClusterId.proto
bq.Added file for ClusterId only since its written to fs and to zk.
bq.  A b/src/main/protobuf/FS.proto
bq.Protos for fs files.
bq.  M b/src/main/protobuf/ZooKeeper.proto
bq.Moved ClusterId 

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277236#comment-13277236
 ] 

jirapos...@reviews.apache.org commented on HBASE-5986:
--



bq.  On 2012-05-16 21:32:56, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/HTable.java, line 473
bq.   https://reviews.apache.org/r/5133/diff/1/?file=109116#file109116line473
bq.  
bq.   Why you remove this?  We don't return these any more?  Offline I 
think is 'dead', unused now.  Split not.

If we discover that a region has been split in META, than, it is past 
point-of-no-return and the region cannot be seen un-split anymore, even though 
concurrent rs failures.  for getStartEndKeys() we are returning whatever the 
getRegionLocation() provides. getRegionLocations() ignores offline regions, but 
returns daughter regions for split-parents (which are offline as well).


bq.  On 2012-05-16 21:32:56, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 351
bq.   https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line351
bq.  
bq.   Should this be public?  Should it remain internal to the HTable 
hidden?

MetaScanner is confusing in its visibility. It's javadoc states Although 
public visibility, this is not a public-facing API and may evolve in minor 
releases., but it is annotated with @InterfaceAudience.Public 
@InterfaceStability.Evolving. I think for BlockingMetaScannerVisitor, we should 
set the visibility the same as MetaScannerVisitor. I think, MetaScannerVisitor 
itself is of little use without the BlockingMetaScannerVisitor functionality 
due to this issue. 
Shall we change MetaScanner, MSV and BMSV to be @InterfaceAudience.Private, 
wdyt? 


bq.  On 2012-05-16 21:32:56, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java, line 310
bq.   https://reviews.apache.org/r/5133/diff/1/?file=109115#file109115line310
bq.  
bq.   Bit of javadoc to say this is best-effort.
bq.   
bq.   Also, does this belong in MetaReader (Won't hold you to it... these 
two classes, a MetaReader vs MetaEditor are kinda silly... this whole catalog 
package needs killing).

agreed


bq.  On 2012-05-16 21:32:56, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 395
bq.   https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line395
bq.  
bq.   Yeah, so, what happens if daughter has split by the time I get here?

The scanner provides the regions results in sorted order, and since the 
daughters are sorted after the parent, we always process parent first. When we 
process the daughter regions manually (processRow(resultA)), we also add them 
to the daughterRegions set. If the scanner also sees them, they are just 
skipped (line 373)


bq.  On 2012-05-16 21:32:56, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 434
bq.   https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line434
bq.  
bq.   So if interrupted or we don't find it by the time the blocking time 
has passed, we just return null?   What you reckon?  We should at least 
complain?

yeah, I think we can throw RegionOfflineException upon timeout and interrupt. 
The operation can be retried by the client upon timeout.


bq.  On 2012-05-16 21:32:56, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 465
bq.   https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line465
bq.  
bq.   Does the scan of meta start at first table region?

MetaScanner.metaScan() ensures that the scan starts at the first table region. 


bq.  On 2012-05-16 21:32:56, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 390
bq.   https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line390
bq.  
bq.   Yeah, what Ted says... Can you close when done?  See 
MetaEditor/MetaReader.  They do this a bunch.  Closing means for sure the zk 
and connection resources will be cleaned up afterward.  Its reference counting 
so keepign around an HTable could mess it up.

We are already closing the HTable's. But let me see how we can best reuse 
Htable instances. 


- enis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5133/#review7939
---


On 2012-05-16 01:53:09, enis wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5133/
bq.  ---
bq.  
bq.  (Updated 2012-05-16 01:53:09)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  

[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277273#comment-13277273
 ] 

jirapos...@reviews.apache.org commented on HBASE-5453:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/
---

(Updated 2012-05-16 23:56:18.496225)


Review request for hbase.


Changes
---

v10... the last thing I added to JIRA


Summary
---

A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
  New  class to hold clusterid in.
M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
  Make it so can do pb serialization.  Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
  Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
  Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
  Make it so can do pb serialization.  Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
  ClusterId under ZK got renamed as ZKClusterId
M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
  Hide the Reference#Range enums.  Don't let them out of this class.
  Make it so can do pb serialization.
M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
  Use new methods on Reference for getting top and bottom.
M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
  ClusterId under zk has been renamed ZKClusterId.
  Use new ClusterId class too.
M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
  Use new clusterid class.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
  Move the RegionInfo convertion up into HRegionInfo instead of here.
  Added generic toDelimitedByteArray helper.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
  Use HRegionInfo convertions instead.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
  Use HRegionInfo convertions instead.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
  Use new utility writing out .regioninfo files.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
  Formatting.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
  Range in Reference is no longer public.
  Range in Reference is no longer public.
M 
b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
M 
b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
  ClusterId got renamed ZKClusterId
M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
  Use new serialization utlity in HTD.
M  b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
  Generic method for writing dot file content.
M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
  Reference#Range is not public any more
M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java
  Deprecated getHRegionInfo, etc.
D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java
A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java
  Rename
A b/src/main/protobuf/ClusterId.proto
  Added file for ClusterId only since its written to fs and to zk.
A b/src/main/protobuf/FS.proto
  Protos for fs files.
M b/src/main/protobuf/ZooKeeper.proto
  Moved ClusterId out to own proto file
M b/src/main/protobuf/hbase.proto
  Added TableSchema and ColumnFamilySchema


This addresses bug hbase-5453.
https://issues.apache.org/jira/browse/hbase-5453


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/ClusterId.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 5862f15 
  src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 8d83ff3 
  src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java af89e3e 
  src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5cac9af 
  src/main/java/org/apache/hadoop/hbase/io/Reference.java 6360059 
  src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 
9e4ada9 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java 947ec5f 
  src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 5052878 
  src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java ccc964e 
  src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java dabfbab 
  src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java 45cb6cf 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClusterIdProtos.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/FSProtos.java 
PRE-CREATION 
  

[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277272#comment-13277272
 ] 

jirapos...@reviews.apache.org commented on HBASE-5453:
--



bq.  On 2012-05-16 22:12:10, Gregory Chanan wrote:
bq.   In a previous comment you said (about the 
HTableDescriptor/HColumnDesriptor pb stuff):
bq.   Well, Andrew beat us both to it over in his REST pb stuff.  We need to 
reconcile his w/ ours too
bq.   
bq.   Where is his stuff?  I couldn't find it.  Should we create a JIRA about 
reconciling? It would be nice to have something, however imperfect, up in trunk 
to work against, then we could fix up later.

Its under src/main/resources/org  I just tripped over it myself yesterday.  
 I made HBASE-6026 to do the reconcile (after HBASE-6000 goes in).

Are you down w/ the change in .regioninfo Gregory as means of 'ensuring' we 
don't have a Writable and pb serialization end up as same size?

Thanks for review.


bq.  On 2012-05-16 22:12:10, Gregory Chanan wrote:
bq.   src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java, line 276
bq.   
https://reviews.apache.org/r/5130/diff/1-2/?file=109037#file109037line276
bq.  
bq.   This isn't ever read?

This is gone now.  I used another technique figuring if object serialized -- 
read file fully into byte array and test for the pb prefix -- rather than this 
hacky setting attribute on class.


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/#review7944
---


On 2012-05-16 17:02:35, Michael Stack wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5130/
bq.  ---
bq.  
bq.  (Updated 2012-05-16 17:02:35)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
bq.New  class to hold clusterid in.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
bq.Make it so can do pb serialization.  Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
bq.Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
bq.Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
bq.Make it so can do pb serialization.  Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
bq.ClusterId under ZK got renamed as ZKClusterId
bq.  M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
bq.Hide the Reference#Range enums.  Don't let them out of this class.
bq.Make it so can do pb serialization.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
bq.Use new methods on Reference for getting top and bottom.
bq.  M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
bq.ClusterId under zk has been renamed ZKClusterId.
bq.Use new ClusterId class too.
bq.  M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
bq.Use new clusterid class.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
bq.Move the RegionInfo convertion up into HRegionInfo instead of here.
bq.Added generic toDelimitedByteArray helper.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
bq.Use HRegionInfo convertions instead.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
bq.Use HRegionInfo convertions instead.
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
bq.Use new utility writing out .regioninfo files.
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
bq.Formatting.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
bq.Range in Reference is no longer public.
bq.Range in Reference is no longer public.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
bq.ClusterId got renamed ZKClusterId
bq.  M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
bq.Use new serialization utlity in HTD.
bq.  M  b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
bq.Generic method for writing dot file content.
bq.  M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
bq.Reference#Range is not public any more
bq.  M 

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277306#comment-13277306
 ] 

jirapos...@reviews.apache.org commented on HBASE-5986:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5133/
---

(Updated 2012-05-17 00:58:21.965488)


Review request for hbase.


Changes
---

v2: 
 - addressed review comments
 - when timeout and interrupt throws RegionOfflineException. 
 - reuse HTable instances in BMSV. 
 - adds close() to MSV assuming it is indeed not public, evolving but private. 
we should decide the visibility. 


Summary
---

We found this issue when running large scale ingestion tests for HBASE-5754. 
The problem is that the .META. table updates are not atomic while splitting a 
region. In SplitTransaction, there is a time lap between the marking the parent 
offline, and adding of daughters to the META table. This can result in clients 
using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat) 
missing regions which are made just offline, but the daughters are not added 
yet.

This patch is the approach 2 mentioned in the issue comments, mainly during 
META scan, if we detect that the region is split, we block until the 
information for the child regions are available in META and manually feed those 
rows to the MetaScanner. Although approach 3 (using local region transactions) 
seems cleaner, they are not available under branch 0.92, which I think should 
also incorporate this fix. I'll provide ports once we are clear for trunk. 

Also this patch does not fix MetaReader (see 
https://issues.apache.org/jira/browse/HBASE-3475). 


This addresses bug HBASE-5986.
https://issues.apache.org/jira/browse/HBASE-5986


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512 
  src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 5d4be3f 
  src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5cac9af 
  src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4 
  src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java 0ad9b18 
  src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 7b4f4a2 
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
 a8091e6 

Diff: https://reviews.apache.org/r/5133/diff


Testing
---

added extensive tests under TestEndToEndSplitTranscation, and ran existing unit 
tests. 


Thanks,

enis



 Clients can see holes in the META table when regions are being split
 

 Key: HBASE-5986
 URL: https://issues.apache.org/jira/browse/HBASE-5986
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Attachments: HBASE-5986-test_v1.patch


 We found this issue when running large scale ingestion tests for HBASE-5754. 
 The problem is that the .META. table updates are not atomic while splitting a 
 region. In SplitTransaction, there is a time lap between the marking the 
 parent offline, and adding of daughters to the META table. This can result in 
 clients using MetaScanner, of HTable.getStartEndKeys (used by the 
 TableInputFormat) missing regions which are made just offline, but the 
 daughters are not added yet. 
 This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277494#comment-13277494
 ] 

jirapos...@reviews.apache.org commented on HBASE-5986:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5133/#review7948
---


v2 looks much better.


src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
https://reviews.apache.org/r/5133/#comment17287

We may need to mark this JIRA an incompatible change.



src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
https://reviews.apache.org/r/5133/#comment17283

'with' - 'by'



src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
https://reviews.apache.org/r/5133/#comment17284

This class can be private.



src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
https://reviews.apache.org/r/5133/#comment17285

The interval can be made shorter.



src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
https://reviews.apache.org/r/5133/#comment17286

Name this method verifyRegionsOfHTable ?


- Ted


On 2012-05-17 00:58:21, enis wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5133/
bq.  ---
bq.  
bq.  (Updated 2012-05-17 00:58:21)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  We found this issue when running large scale ingestion tests for 
HBASE-5754. The problem is that the .META. table updates are not atomic while 
splitting a region. In SplitTransaction, there is a time lap between the 
marking the parent offline, and adding of daughters to the META table. This can 
result in clients using MetaScanner, of HTable.getStartEndKeys (used by the 
TableInputFormat) missing regions which are made just offline, but the 
daughters are not added yet.
bq.  
bq.  This patch is the approach 2 mentioned in the issue comments, mainly 
during META scan, if we detect that the region is split, we block until the 
information for the child regions are available in META and manually feed those 
rows to the MetaScanner. Although approach 3 (using local region transactions) 
seems cleaner, they are not available under branch 0.92, which I think should 
also incorporate this fix. I'll provide ports once we are clear for trunk. 
bq.  
bq.  Also this patch does not fix MetaReader (see 
https://issues.apache.org/jira/browse/HBASE-3475). 
bq.  
bq.  
bq.  This addresses bug HBASE-5986.
bq.  https://issues.apache.org/jira/browse/HBASE-5986
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512 
bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 5d4be3f 
bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 
5cac9af 
bq.src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4 
bq.src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 0ad9b18 
bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 7b4f4a2 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
 a8091e6 
bq.  
bq.  Diff: https://reviews.apache.org/r/5133/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  added extensive tests under TestEndToEndSplitTranscation, and ran existing 
unit tests. 
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  enis
bq.  
bq.



 Clients can see holes in the META table when regions are being split
 

 Key: HBASE-5986
 URL: https://issues.apache.org/jira/browse/HBASE-5986
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Attachments: 5986-v2.txt, HBASE-5986-test_v1.patch


 We found this issue when running large scale ingestion tests for HBASE-5754. 
 The problem is that the .META. table updates are not atomic while splitting a 
 region. In SplitTransaction, there is a time lap between the marking the 
 parent offline, and adding of daughters to the META table. This can result in 
 clients using MetaScanner, of HTable.getStartEndKeys (used by the 
 TableInputFormat) missing regions which are made just offline, but the 
 daughters are not added yet. 
 This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was 

[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277589#comment-13277589
 ] 

jirapos...@reviews.apache.org commented on HBASE-5453:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/#review7951
---



src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
https://reviews.apache.org/r/5130/#comment17291

This is still here?  Or just reviewboard showing it for some reason?



src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
https://reviews.apache.org/r/5130/#comment17290

This is what you are using to ensure the sizes cannot be the same?  Very 
nice!

I cannot find a call-site for this, though, maybe I missed it?


- Gregory


On 2012-05-16 23:56:18, Michael Stack wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5130/
bq.  ---
bq.  
bq.  (Updated 2012-05-16 23:56:18)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
bq.New  class to hold clusterid in.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
bq.Make it so can do pb serialization.  Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
bq.Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
bq.Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
bq.Make it so can do pb serialization.  Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
bq.ClusterId under ZK got renamed as ZKClusterId
bq.  M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
bq.Hide the Reference#Range enums.  Don't let them out of this class.
bq.Make it so can do pb serialization.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
bq.Use new methods on Reference for getting top and bottom.
bq.  M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
bq.ClusterId under zk has been renamed ZKClusterId.
bq.Use new ClusterId class too.
bq.  M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
bq.Use new clusterid class.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
bq.Move the RegionInfo convertion up into HRegionInfo instead of here.
bq.Added generic toDelimitedByteArray helper.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
bq.Use HRegionInfo convertions instead.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
bq.Use HRegionInfo convertions instead.
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
bq.Use new utility writing out .regioninfo files.
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
bq.Formatting.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
bq.Range in Reference is no longer public.
bq.Range in Reference is no longer public.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
bq.ClusterId got renamed ZKClusterId
bq.  M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
bq.Use new serialization utlity in HTD.
bq.  M  b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
bq.Generic method for writing dot file content.
bq.  M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
bq.Reference#Range is not public any more
bq.  M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java
bq.Deprecated getHRegionInfo, etc.
bq.  D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java
bq.  A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java
bq.Rename
bq.  A b/src/main/protobuf/ClusterId.proto
bq.Added file for ClusterId only since its written to fs and to zk.
bq.  A b/src/main/protobuf/FS.proto
bq.Protos for fs files.
bq.  M b/src/main/protobuf/ZooKeeper.proto
bq.Moved ClusterId out to own proto file
bq.  M b/src/main/protobuf/hbase.proto
bq.Added TableSchema and ColumnFamilySchema
bq.  
bq.  
bq.  This addresses bug hbase-5453.
bq.  https://issues.apache.org/jira/browse/hbase-5453
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.

[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277593#comment-13277593
 ] 

jirapos...@reviews.apache.org commented on HBASE-5453:
--



bq.  On 2012-05-17 05:41:53, Gregory Chanan wrote:
bq.   src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java, line 275
bq.   https://reviews.apache.org/r/5130/diff/3/?file=109313#file109313line275
bq.  
bq.   This is still here?  Or just reviewboard showing it for some reason?

Flotsam.  An abandoned direction that I failed to clean up.  Thanks.


bq.  On 2012-05-17 05:41:53, Gregory Chanan wrote:
bq.   src/main/java/org/apache/hadoop/hbase/util/FSUtils.java, line 91
bq.   https://reviews.apache.org/r/5130/diff/3/?file=109334#file109334line91
bq.  
bq.   This is what you are using to ensure the sizes cannot be the same?  
Very nice!
bq.   
bq.   I cannot find a call-site for this, though, maybe I missed it?

Ditto. I was going to use this to write .regioninfo and .tableinfo but then I 
ran into your review.  Study this method which I'll purge in the next version.  
See how it puts serialized class at top, two '\n's, and then a toString on the 
class?  Thats how .regioninfo files and .tableinfo files are currently written. 
 My patch now changes it so these files only contained the serialized object... 
no '\n' and no toString.  I do this so its very unlikely a Writable length will 
class w/ the length of a pb'd content (See HRegion for where we write 
.regioninfo and in FSTableDescriptors for where we write .tableinfo only we 
don't need the 'trick' here since we have actually read the file and can see if 
its pb'd).

Thanks for the review G.


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/#review7951
---


On 2012-05-16 23:56:18, Michael Stack wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5130/
bq.  ---
bq.  
bq.  (Updated 2012-05-16 23:56:18)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
bq.New  class to hold clusterid in.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
bq.Make it so can do pb serialization.  Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
bq.Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
bq.Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
bq.Make it so can do pb serialization.  Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
bq.ClusterId under ZK got renamed as ZKClusterId
bq.  M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
bq.Hide the Reference#Range enums.  Don't let them out of this class.
bq.Make it so can do pb serialization.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
bq.Use new methods on Reference for getting top and bottom.
bq.  M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
bq.ClusterId under zk has been renamed ZKClusterId.
bq.Use new ClusterId class too.
bq.  M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
bq.Use new clusterid class.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
bq.Move the RegionInfo convertion up into HRegionInfo instead of here.
bq.Added generic toDelimitedByteArray helper.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
bq.Use HRegionInfo convertions instead.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
bq.Use HRegionInfo convertions instead.
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
bq.Use new utility writing out .regioninfo files.
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
bq.Formatting.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
bq.Range in Reference is no longer public.
bq.Range in Reference is no longer public.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
bq.ClusterId got renamed ZKClusterId
bq.  M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
bq.Use new 

[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13276276#comment-13276276
 ] 

jirapos...@reviews.apache.org commented on HBASE-5453:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/
---

Review request for hbase.


Summary
---

A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
  New  class to hold clusterid in.
M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
  Make it so can do pb serialization.  Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
  Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
  Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
  Make it so can do pb serialization.  Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
  ClusterId under ZK got renamed as ZKClusterId
M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
  Hide the Reference#Range enums.  Don't let them out of this class.
  Make it so can do pb serialization.
M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
  Use new methods on Reference for getting top and bottom.
M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
  ClusterId under zk has been renamed ZKClusterId.
  Use new ClusterId class too.
M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
  Use new clusterid class.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
  Move the RegionInfo convertion up into HRegionInfo instead of here.
  Added generic toDelimitedByteArray helper.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
  Use HRegionInfo convertions instead.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
  Use HRegionInfo convertions instead.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
  Use new utility writing out .regioninfo files.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
  Formatting.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
  Range in Reference is no longer public.
  Range in Reference is no longer public.
M 
b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
M 
b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
  ClusterId got renamed ZKClusterId
M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
  Use new serialization utlity in HTD.
M  b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
  Generic method for writing dot file content.
M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
  Reference#Range is not public any more
M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java
  Deprecated getHRegionInfo, etc.
D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java
A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java
  Rename
A b/src/main/protobuf/ClusterId.proto
  Added file for ClusterId only since its written to fs and to zk.
A b/src/main/protobuf/FS.proto
  Protos for fs files.
M b/src/main/protobuf/ZooKeeper.proto
  Moved ClusterId out to own proto file
M b/src/main/protobuf/hbase.proto
  Added TableSchema and ColumnFamilySchema


This addresses bug hbase-5453.
https://issues.apache.org/jira/browse/hbase-5453


Diffs
-

  src/main/java/org/apache/hadoop/hbase/ClusterId.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 5862f15 
  src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 8d83ff3 
  src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java af89e3e 
  src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5cac9af 
  src/main/java/org/apache/hadoop/hbase/io/Reference.java 6360059 
  src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 
9e4ada9 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java 947ec5f 
  src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 5052878 
  src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java ccc964e 
  src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java dabfbab 
  src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java 45cb6cf 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClusterIdProtos.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/FSProtos.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 
058c006 
  

[jira] [Commented] (HBASE-6004) Adding more logging to help debugging MR job

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13276287#comment-13276287
 ] 

jirapos...@reviews.apache.org commented on HBASE-6004:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5131/
---

Review request for hbase.


Summary
---

Added some logging for MR debugging in case scanner times out.  The logging is 
disabled by default.
It will be helpful to know how much time spent in the scanner and, how much in 
the mapper task.
In case scanner issue, it is helpful to know the region server id, last 
successful rows and so on.


This addresses bug HBASE-6004.
https://issues.apache.org/jira/browse/HBASE-6004


Diffs
-

  src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 46b1c56 
  src/main/java/org/apache/hadoop/hbase/mapred/TableRecordReaderImpl.java 
42569fb 
  src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java 
1c8a393 

Diff: https://reviews.apache.org/r/5131/diff


Testing
---


Thanks,

Jimmy



 Adding more logging to help debugging MR job
 

 Key: HBASE-6004
 URL: https://issues.apache.org/jira/browse/HBASE-6004
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.94.0, 0.96.0
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
 Fix For: 0.94.0, 0.96.0

 Attachments: hbase-6004.patch


 MR job sometime fails because scanner expired. In this case, it will be 
 helpful to know the last successful row, the ip of the region sever, and so 
 on.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13276320#comment-13276320
 ] 

jirapos...@reviews.apache.org commented on HBASE-5453:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/#review7916
---


Do we have a story for when we can remove the writable stuff?


src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
https://reviews.apache.org/r/5130/#comment17217

Looks like we are stepping on each other's toes a bit; I defined my own 
protos for this and HTableDescriptor (see the review request for HBASE-5445).  
Your definitions are pretty close though; I'll convert mine over.



src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
https://reviews.apache.org/r/5130/#comment17218

not be what ou want - not be what you want



src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
https://reviews.apache.org/r/5130/#comment17219

Did you give any thought to storing the actual data in a TableSchema rather 
than converting?  Perhaps it's not worth it because:
1) we still need to maintain the writable for now, so would require 
rewriting that part, which is a waste
2) might not perform well, because we have to call copyFrom a bunch of 
times.



src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
https://reviews.apache.org/r/5130/#comment17220

Nice.  I wrote up similar things for HTableDescriptor/HColumnDescriptor; I 
can just get rid of them and use your convert functions.



src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
https://reviews.apache.org/r/5130/#comment17226

Are we sure this assumption is valid (that if the file lengths are the 
same, the files are the same format)?

We can't check if there is a pb-prefix at the start of file or something? 



src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
https://reviews.apache.org/r/5130/#comment17223

Same here, is this assumption valid?



src/main/protobuf/FS.proto
https://reviews.apache.org/r/5130/#comment17221

extra whitespace



src/main/protobuf/hbase.proto
https://reviews.apache.org/r/5130/#comment17222

extra whitespace


- Gregory


On 2012-05-15 22:14:17, Michael Stack wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5130/
bq.  ---
bq.  
bq.  (Updated 2012-05-15 22:14:17)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
bq.New  class to hold clusterid in.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
bq.Make it so can do pb serialization.  Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
bq.Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
bq.Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
bq.Make it so can do pb serialization.  Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
bq.ClusterId under ZK got renamed as ZKClusterId
bq.  M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
bq.Hide the Reference#Range enums.  Don't let them out of this class.
bq.Make it so can do pb serialization.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
bq.Use new methods on Reference for getting top and bottom.
bq.  M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
bq.ClusterId under zk has been renamed ZKClusterId.
bq.Use new ClusterId class too.
bq.  M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
bq.Use new clusterid class.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
bq.Move the RegionInfo convertion up into HRegionInfo instead of here.
bq.Added generic toDelimitedByteArray helper.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
bq.Use HRegionInfo convertions instead.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
bq.Use HRegionInfo convertions instead.
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
bq.Use new utility writing out .regioninfo files.
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
bq.Formatting.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
bq.

[jira] [Commented] (HBASE-6004) Adding more logging to help debugging MR job

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13276324#comment-13276324
 ] 

jirapos...@reviews.apache.org commented on HBASE-6004:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5131/#review7918
---



src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java
https://reviews.apache.org/r/5131/#comment17227

We control logging via log4j.properties files. Here also new configuration 
in another file to switch some additional logging on and off. Would it make 
more sense to create a logger class e.g. ScannerCallable.ActivityLog that logs 
at TRACE level and update log4j.properties with 
log4j.logger.org.apache.hadoop.hbase.client.ScannerCallable.ActivityLog = TRACE 
(default is INFO, i.e. disabled). Just a thought. We did something like this 
for security audit logging.



src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java
https://reviews.apache.org/r/5131/#comment17224

Maybe the client should just log this?



src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java
https://reviews.apache.org/r/5131/#comment17225

The new message seems more helpful than the old.



src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
https://reviews.apache.org/r/5131/#comment17228

We have other configuration variables that start with 'hbase.mapreduce.*'


- Andrew


On 2012-05-15 22:24:40, Jimmy Xiang wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5131/
bq.  ---
bq.  
bq.  (Updated 2012-05-15 22:24:40)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Added some logging for MR debugging in case scanner times out.  The 
logging is disabled by default.
bq.  It will be helpful to know how much time spent in the scanner and, how 
much in the mapper task.
bq.  In case scanner issue, it is helpful to know the region server id, last 
successful rows and so on.
bq.  
bq.  
bq.  This addresses bug HBASE-6004.
bq.  https://issues.apache.org/jira/browse/HBASE-6004
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 
46b1c56 
bq.src/main/java/org/apache/hadoop/hbase/mapred/TableRecordReaderImpl.java 
42569fb 
bq.
src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java 
1c8a393 
bq.  
bq.  Diff: https://reviews.apache.org/r/5131/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Jimmy
bq.  
bq.



 Adding more logging to help debugging MR job
 

 Key: HBASE-6004
 URL: https://issues.apache.org/jira/browse/HBASE-6004
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.94.0, 0.96.0
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
 Fix For: 0.94.0, 0.96.0

 Attachments: hbase-6004.patch


 MR job sometime fails because scanner expired. In this case, it will be 
 helpful to know the last successful row, the ip of the region sever, and so 
 on.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13276330#comment-13276330
 ] 

jirapos...@reviews.apache.org commented on HBASE-5453:
--



bq.  On 2012-05-15 23:08:35, Gregory Chanan wrote:
bq.   Do we have a story for when we can remove the writable stuff?

No.  Smile.  Being optimistic.  Have to start somewhere.  No harm marking this 
stuff deprecated in  meantime as heads-up.

I think we won't be able to migrate over these old classes.  I'm thinking we'll 
have to do new ones and deprecate the old given they inherit from Writable.  
It'll be a bit of a pain though given some of these classes come out in our 
public API.

But we have to start somewhere.


bq.  On 2012-05-15 23:08:35, Gregory Chanan wrote:
bq.   src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java, line 1017
bq.   
https://reviews.apache.org/r/5130/diff/1/?file=109035#file109035line1017
bq.  
bq.   Looks like we are stepping on each other's toes a bit; I defined my 
own protos for this and HTableDescriptor (see the review request for 
HBASE-5445).  Your definitions are pretty close though; I'll convert mine over.

Well, Andrew beat us both to it over in his REST pb stuff.  We need to 
reconcile his w/ ours too


bq.  On 2012-05-15 23:08:35, Gregory Chanan wrote:
bq.   src/main/java/org/apache/hadoop/hbase/HRegionInfo.java, line 954
bq.   https://reviews.apache.org/r/5130/diff/1/?file=109036#file109036line954
bq.  
bq.   not be what ou want - not be what you want

Fixed


bq.  On 2012-05-15 23:08:35, Gregory Chanan wrote:
bq.   src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java, line 1240
bq.   
https://reviews.apache.org/r/5130/diff/1/?file=109037#file109037line1240
bq.  
bq.   Did you give any thought to storing the actual data in a TableSchema 
rather than converting?  Perhaps it's not worth it because:
bq.   1) we still need to maintain the writable for now, so would require 
rewriting that part, which is a waste
bq.   2) might not perform well, because we have to call copyFrom a bunch 
of times.

Yeah.  Thought about it but we want Writable still converting old 
serializations.  Mostly I reasoned that these classes are rarely, relatively 
serialized, and performance is not important when we do want to pass these 
classes serialized.  Yeah, it'd be a pain converting.


bq.  On 2012-05-15 23:08:35, Gregory Chanan wrote:
bq.   src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 748
bq.   https://reviews.apache.org/r/5130/diff/1/?file=109050#file109050line748
bq.  
bq.   Are we sure this assumption is valid (that if the file lengths are 
the same, the files are the same format)?
bq.   
bq.   We can't check if there is a pb-prefix at the start of file or 
something?

This is my way of figuring if I need to convert the file.  There is an off 
chance that file could be same length though serialized with Writables.  Good 
one.  Let me fix that.


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/#review7916
---


On 2012-05-15 22:14:17, Michael Stack wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5130/
bq.  ---
bq.  
bq.  (Updated 2012-05-15 22:14:17)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
bq.New  class to hold clusterid in.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
bq.Make it so can do pb serialization.  Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
bq.Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
bq.Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
bq.Make it so can do pb serialization.  Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
bq.ClusterId under ZK got renamed as ZKClusterId
bq.  M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
bq.Hide the Reference#Range enums.  Don't let them out of this class.
bq.Make it so can do pb serialization.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
bq.Use new methods on Reference for getting top and bottom.
bq.  M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
bq.ClusterId under zk has been renamed ZKClusterId.
bq.Use new ClusterId class too.
bq.  M 

[jira] [Commented] (HBASE-6004) Adding more logging to help debugging MR job

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13276341#comment-13276341
 ] 

jirapos...@reviews.apache.org commented on HBASE-6004:
--



bq.  On 2012-05-15 23:11:04, Andrew Purtell wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java, line 
164
bq.   https://reviews.apache.org/r/5131/diff/1/?file=109074#file109074line164
bq.  
bq.   Maybe the client should just log this?

Client sometimes just gets a RetriesExhaustedException.


bq.  On 2012-05-15 23:11:04, Andrew Purtell wrote:
bq.   
src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java, 
line 51
bq.   https://reviews.apache.org/r/5131/diff/1/?file=109076#file109076line51
bq.  
bq.   We have other configuration variables that start with 
'hbase.mapreduce.*'

Will change to mapreduce.


bq.  On 2012-05-15 23:11:04, Andrew Purtell wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java, line 
54
bq.   https://reviews.apache.org/r/5131/diff/1/?file=109074#file109074line54
bq.  
bq.   We control logging via log4j.properties files. Here also new 
configuration in another file to switch some additional logging on and off. 
Would it make more sense to create a logger class e.g. 
ScannerCallable.ActivityLog that logs at TRACE level and update 
log4j.properties with 
log4j.logger.org.apache.hadoop.hbase.client.ScannerCallable.ActivityLog = TRACE 
(default is INFO, i.e. disabled). Just a thought. We did something like this 
for security audit logging.

It is a good idea.  Because of other parameters introduced, it's better to put 
them in the same place for now.


- Jimmy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5131/#review7918
---


On 2012-05-15 22:24:40, Jimmy Xiang wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5131/
bq.  ---
bq.  
bq.  (Updated 2012-05-15 22:24:40)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Added some logging for MR debugging in case scanner times out.  The 
logging is disabled by default.
bq.  It will be helpful to know how much time spent in the scanner and, how 
much in the mapper task.
bq.  In case scanner issue, it is helpful to know the region server id, last 
successful rows and so on.
bq.  
bq.  
bq.  This addresses bug HBASE-6004.
bq.  https://issues.apache.org/jira/browse/HBASE-6004
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 
46b1c56 
bq.src/main/java/org/apache/hadoop/hbase/mapred/TableRecordReaderImpl.java 
42569fb 
bq.
src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java 
1c8a393 
bq.  
bq.  Diff: https://reviews.apache.org/r/5131/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Jimmy
bq.  
bq.



 Adding more logging to help debugging MR job
 

 Key: HBASE-6004
 URL: https://issues.apache.org/jira/browse/HBASE-6004
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.94.0, 0.96.0
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
 Fix For: 0.94.0, 0.96.0

 Attachments: hbase-6004.patch


 MR job sometime fails because scanner expired. In this case, it will be 
 helpful to know the last successful row, the ip of the region sever, and so 
 on.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13276385#comment-13276385
 ] 

jirapos...@reviews.apache.org commented on HBASE-5986:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5133/
---

Review request for hbase.


Summary
---

We found this issue when running large scale ingestion tests for HBASE-5754. 
The problem is that the .META. table updates are not atomic while splitting a 
region. In SplitTransaction, there is a time lap between the marking the parent 
offline, and adding of daughters to the META table. This can result in clients 
using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat) 
missing regions which are made just offline, but the daughters are not added 
yet.


This addresses bug HBASE-5986.
https://issues.apache.org/jira/browse/HBASE-5986


Diffs
-

  src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512 
  src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4 
  src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999 
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
 a8091e6 

Diff: https://reviews.apache.org/r/5133/diff


Testing
---

added extensive tests under TestEndToEndSplitTranscation, and ran existing unit 
tests. 


Thanks,

enis



 Clients can see holes in the META table when regions are being split
 

 Key: HBASE-5986
 URL: https://issues.apache.org/jira/browse/HBASE-5986
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Attachments: HBASE-5986-test_v1.patch


 We found this issue when running large scale ingestion tests for HBASE-5754. 
 The problem is that the .META. table updates are not atomic while splitting a 
 region. In SplitTransaction, there is a time lap between the marking the 
 parent offline, and adding of daughters to the META table. This can result in 
 clients using MetaScanner, of HTable.getStartEndKeys (used by the 
 TableInputFormat) missing regions which are made just offline, but the 
 daughters are not added yet. 
 This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13276386#comment-13276386
 ] 

jirapos...@reviews.apache.org commented on HBASE-5986:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5133/
---

(Updated 2012-05-16 01:53:09.861265)


Review request for hbase.


Summary (updated)
---

We found this issue when running large scale ingestion tests for HBASE-5754. 
The problem is that the .META. table updates are not atomic while splitting a 
region. In SplitTransaction, there is a time lap between the marking the parent 
offline, and adding of daughters to the META table. This can result in clients 
using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat) 
missing regions which are made just offline, but the daughters are not added 
yet.

This patch is the approach 2 mentioned in the issue comments, mainly during 
META scan, if we detect that the region is split, we block until the 
information for the child regions are available in META and manually feed those 
rows to the MetaScanner. Although approach 3 (using local region transactions) 
seems cleaner, they are not available under branch 0.92, which I think should 
also incorporate this fix. I'll provide ports once we are clear for trunk. 

Also this patch does not fix MetaReader (see 
https://issues.apache.org/jira/browse/HBASE-3475). 


This addresses bug HBASE-5986.
https://issues.apache.org/jira/browse/HBASE-5986


Diffs
-

  src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512 
  src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4 
  src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999 
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
 a8091e6 

Diff: https://reviews.apache.org/r/5133/diff


Testing
---

added extensive tests under TestEndToEndSplitTranscation, and ran existing unit 
tests. 


Thanks,

enis



 Clients can see holes in the META table when regions are being split
 

 Key: HBASE-5986
 URL: https://issues.apache.org/jira/browse/HBASE-5986
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Attachments: HBASE-5986-test_v1.patch


 We found this issue when running large scale ingestion tests for HBASE-5754. 
 The problem is that the .META. table updates are not atomic while splitting a 
 region. In SplitTransaction, there is a time lap between the marking the 
 parent offline, and adding of daughters to the META table. This can result in 
 clients using MetaScanner, of HTable.getStartEndKeys (used by the 
 TableInputFormat) missing regions which are made just offline, but the 
 daughters are not added yet. 
 This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13276399#comment-13276399
 ] 

jirapos...@reviews.apache.org commented on HBASE-5986:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5133/#review7925
---



src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java
https://reviews.apache.org/r/5133/#comment17245

Remove white space.



src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
https://reviews.apache.org/r/5133/#comment17246

Can we add a config param for blockingTimeout ?



src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
https://reviews.apache.org/r/5133/#comment17247

Do we have to instantiate HTable every time ?



src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
https://reviews.apache.org/r/5133/#comment17250

null is always returned by getRegionResultBlocking() in case of time out.
How do we deal with that ?




src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
https://reviews.apache.org/r/5133/#comment17249

We shouldn't be passing blockingTimeout here.
We need to consider the amount of time spent in the call @ line 388.



src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
https://reviews.apache.org/r/5133/#comment17248

Please restore interrupted state of the thread.


- Ted


On 2012-05-16 01:53:09, enis wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5133/
bq.  ---
bq.  
bq.  (Updated 2012-05-16 01:53:09)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  We found this issue when running large scale ingestion tests for 
HBASE-5754. The problem is that the .META. table updates are not atomic while 
splitting a region. In SplitTransaction, there is a time lap between the 
marking the parent offline, and adding of daughters to the META table. This can 
result in clients using MetaScanner, of HTable.getStartEndKeys (used by the 
TableInputFormat) missing regions which are made just offline, but the 
daughters are not added yet.
bq.  
bq.  This patch is the approach 2 mentioned in the issue comments, mainly 
during META scan, if we detect that the region is split, we block until the 
information for the child regions are available in META and manually feed those 
rows to the MetaScanner. Although approach 3 (using local region transactions) 
seems cleaner, they are not available under branch 0.92, which I think should 
also incorporate this fix. I'll provide ports once we are clear for trunk. 
bq.  
bq.  Also this patch does not fix MetaReader (see 
https://issues.apache.org/jira/browse/HBASE-3475). 
bq.  
bq.  
bq.  This addresses bug HBASE-5986.
bq.  https://issues.apache.org/jira/browse/HBASE-5986
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512 
bq.src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4 
bq.src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
 a8091e6 
bq.  
bq.  Diff: https://reviews.apache.org/r/5133/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  added extensive tests under TestEndToEndSplitTranscation, and ran existing 
unit tests. 
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  enis
bq.  
bq.



 Clients can see holes in the META table when regions are being split
 

 Key: HBASE-5986
 URL: https://issues.apache.org/jira/browse/HBASE-5986
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Attachments: HBASE-5986-test_v1.patch


 We found this issue when running large scale ingestion tests for HBASE-5754. 
 The problem is that the .META. table updates are not atomic while splitting a 
 region. In SplitTransaction, there is a time lap between the marking the 
 parent offline, and adding of daughters to the META table. This can result in 
 clients using MetaScanner, of HTable.getStartEndKeys (used by the 
 TableInputFormat) missing regions which are made just offline, but the 
 daughters are not added yet. 
 This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 

[jira] [Commented] (HBASE-6004) Adding more logging to help debugging MR job

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13276403#comment-13276403
 ] 

jirapos...@reviews.apache.org commented on HBASE-6004:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5131/
---

(Updated 2012-05-16 02:56:16.913480)


Review request for hbase.


Summary
---

Added some logging for MR debugging in case scanner times out.  The logging is 
disabled by default.
It will be helpful to know how much time spent in the scanner and, how much in 
the mapper task.
In case scanner issue, it is helpful to know the region server id, last 
successful rows and so on.


This addresses bug HBASE-6004.
https://issues.apache.org/jira/browse/HBASE-6004


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 46b1c56 
  src/main/java/org/apache/hadoop/hbase/mapred/TableRecordReaderImpl.java 
42569fb 
  src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java 
1c8a393 

Diff: https://reviews.apache.org/r/5131/diff


Testing
---


Thanks,

Jimmy



 Adding more logging to help debugging MR job
 

 Key: HBASE-6004
 URL: https://issues.apache.org/jira/browse/HBASE-6004
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.94.0, 0.96.0
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
 Fix For: 0.94.0, 0.96.0

 Attachments: hbase-6004.patch


 MR job sometime fails because scanner expired. In this case, it will be 
 helpful to know the last successful row, the ip of the region sever, and so 
 on.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13276477#comment-13276477
 ] 

jirapos...@reviews.apache.org commented on HBASE-5986:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5133/#review7928
---



src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
https://reviews.apache.org/r/5133/#comment17254

Blocking time out is one factor on which we will be waiting.  So this 
blockingtimeout should be in lieu with 'fileSplitTimeout'?


- ramkrishna


On 2012-05-16 01:53:09, enis wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5133/
bq.  ---
bq.  
bq.  (Updated 2012-05-16 01:53:09)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  We found this issue when running large scale ingestion tests for 
HBASE-5754. The problem is that the .META. table updates are not atomic while 
splitting a region. In SplitTransaction, there is a time lap between the 
marking the parent offline, and adding of daughters to the META table. This can 
result in clients using MetaScanner, of HTable.getStartEndKeys (used by the 
TableInputFormat) missing regions which are made just offline, but the 
daughters are not added yet.
bq.  
bq.  This patch is the approach 2 mentioned in the issue comments, mainly 
during META scan, if we detect that the region is split, we block until the 
information for the child regions are available in META and manually feed those 
rows to the MetaScanner. Although approach 3 (using local region transactions) 
seems cleaner, they are not available under branch 0.92, which I think should 
also incorporate this fix. I'll provide ports once we are clear for trunk. 
bq.  
bq.  Also this patch does not fix MetaReader (see 
https://issues.apache.org/jira/browse/HBASE-3475). 
bq.  
bq.  
bq.  This addresses bug HBASE-5986.
bq.  https://issues.apache.org/jira/browse/HBASE-5986
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512 
bq.src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4 
bq.src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
 a8091e6 
bq.  
bq.  Diff: https://reviews.apache.org/r/5133/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  added extensive tests under TestEndToEndSplitTranscation, and ran existing 
unit tests. 
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  enis
bq.  
bq.



 Clients can see holes in the META table when regions are being split
 

 Key: HBASE-5986
 URL: https://issues.apache.org/jira/browse/HBASE-5986
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Attachments: HBASE-5986-test_v1.patch


 We found this issue when running large scale ingestion tests for HBASE-5754. 
 The problem is that the .META. table updates are not atomic while splitting a 
 region. In SplitTransaction, there is a time lap between the marking the 
 parent offline, and adding of daughters to the META table. This can result in 
 clients using MetaScanner, of HTable.getStartEndKeys (used by the 
 TableInputFormat) missing regions which are made just offline, but the 
 daughters are not added yet. 
 This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272727#comment-13272727
 ] 

jirapos...@reviews.apache.org commented on HBASE-5547:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4633/
---

(Updated 2012-05-10 20:28:50.691971)


Review request for hbase, Michael Stack and Lars Hofhansl.


Changes
---

Refactoring as per stack's comments. Also, updated the testing to avoid 
flapping. 

I think this is pretty close to ready for commit.


Summary
---

Essentially, whenever an hfile would be deleted, it is instead moved to the 
archive directory. In this impl, the archive directory is on a per table basis, 
but defaults to '.archive'. Removing hfiles occurs in three places - 
compaction, merge and catalog janitor. The former and two latter are distinctly 
different code paths, but but did pull out some similarities. The latter two 
end up calling the same method, so there should be a reasonable amount of 
overlap.

Implementation wise: 
Updated the HMasterInterface to pass the calls onto the zookeeper.
Added a zk listener to handle updates from the master to the RS to backup.
Added a utility for removing files and finding archive directories
Added tests for the regionserver and catalogjanitor approaches.
Added creation of manager in regionserver.


This addresses bug HBASE-5547.
https://issues.apache.org/jira/browse/HBASE-5547


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/HConstants.java 2f432c4 
  src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableMonitor.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/backup/HFileDisposer.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/backup/TableHFileArchiveTracker.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 5d4be3f 
  src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java e751f65 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java 0ad9b18 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1f09be1 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java f7ac81a 
  src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 
6884d53 
  src/main/java/org/apache/hadoop/hbase/regionserver/Store.java bf1618e 
  src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 33bc1d0 
  src/main/resources/hbase-default.xml 42d1c4e 
  src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java 9fba339 
  
src/test/java/org/apache/hadoop/hbase/backup/SimpleHFileArchiveTableMonitor.java
 PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchivingCleanup.java 
PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/backup/TestRegionDisposer.java 
PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 69ccc65 
  src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 1020374 
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java
 PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java 
PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 
3f61cfb 

Diff: https://reviews.apache.org/r/4633/diff


Testing
---

Added two tests for the separate cases - archiving via the regionserver and for 
the catalog tracker. Former runs in a mini cluster and also touches the changes 
to HMasterInterface and zookeeper.


Thanks,

Jesse



 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Attachments: java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, 
 java_HBASE-5547_v6.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272848#comment-13272848
 ] 

jirapos...@reviews.apache.org commented on HBASE-5547:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4633/#review7794
---


Heading to a meeting.
Below is first part of review.


src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java
https://reviews.apache.org/r/4633/#comment17105

We know all is true when table is null, right ?

Seems we don't need this boolean.



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java
https://reviews.apache.org/r/4633/#comment17100

aren't - are



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java
https://reviews.apache.org/r/4633/#comment17101

Either remove 'down' or move it to the end of the sentence



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java
https://reviews.apache.org/r/4633/#comment17102

Please check return value.

I think we should remember the files which we are unable to delete and 
display them at the end.



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java
https://reviews.apache.org/r/4633/#comment17104

Would anyone specify start == end ?



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableMonitor.java
https://reviews.apache.org/r/4633/#comment17106

License, please.



src/main/java/org/apache/hadoop/hbase/backup/HFileDisposer.java
https://reviews.apache.org/r/4633/#comment17107

License.



src/main/java/org/apache/hadoop/hbase/backup/HFileDisposer.java
https://reviews.apache.org/r/4633/#comment17108

This condition deserves an exception, I think.



src/main/java/org/apache/hadoop/hbase/backup/HFileDisposer.java
https://reviews.apache.org/r/4633/#comment17109

Is it possible to provide more information in this exception ?
e.g. by using toArchive collection ?


- Ted


On 2012-05-10 20:28:50, Jesse Yates wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4633/
bq.  ---
bq.  
bq.  (Updated 2012-05-10 20:28:50)
bq.  
bq.  
bq.  Review request for hbase, Michael Stack and Lars Hofhansl.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Essentially, whenever an hfile would be deleted, it is instead moved to 
the archive directory. In this impl, the archive directory is on a per table 
basis, but defaults to '.archive'. Removing hfiles occurs in three places - 
compaction, merge and catalog janitor. The former and two latter are distinctly 
different code paths, but but did pull out some similarities. The latter two 
end up calling the same method, so there should be a reasonable amount of 
overlap.
bq.  
bq.  Implementation wise: 
bq.  Updated the HMasterInterface to pass the calls onto the zookeeper.
bq.  Added a zk listener to handle updates from the master to the RS to 
backup.
bq.  Added a utility for removing files and finding archive directories
bq.  Added tests for the regionserver and catalogjanitor approaches.
bq.  Added creation of manager in regionserver.
bq.  
bq.  
bq.  This addresses bug HBASE-5547.
bq.  https://issues.apache.org/jira/browse/HBASE-5547
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/HConstants.java 2f432c4 
bq.src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java 
PRE-CREATION 
bq.
src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableMonitor.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/backup/HFileDisposer.java 
PRE-CREATION 
bq.
src/main/java/org/apache/hadoop/hbase/backup/TableHFileArchiveTracker.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 5d4be3f 
bq.src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java e751f65 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 0ad9b18 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1f09be1 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
f7ac81a 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 
6884d53 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/Store.java bf1618e 
bq.src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java 
PRE-CREATION 
bq.

[jira] [Commented] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272885#comment-13272885
 ] 

jirapos...@reviews.apache.org commented on HBASE-5732:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4953/
---

(Updated 2012-05-10 23:19:45.031886)


Review request for Ted Yu, Michael Stack and Andrew Purtell.


Changes
---

Upon testing with with a mapreduce job (PerformanceEvaluation.java), I found a 
bug to do with delegation tokens. This patch fixes that. 


Summary
---

Reviewboard request for HBASE-5732


This addresses bug HBASE-5732.
https://issues.apache.org/jira/browse/HBASE-5732


Diffs (updated)
-

  http://svn.apache.org/repos/asf/hbase/trunk/pom.xml 1336441 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/AdminProtocol.java
 1336441 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ClientProtocol.java
 1336441 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java
 1336441 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java
 1336441 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
 1336441 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/RegionServerStatusProtocol.java
 1336441 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
 1336441 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java
 1336441 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java
 1336441 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlFilter.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControllerProtocol.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/Permission.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/TablePermission.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/UserPermission.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/ZKPermissionWatcher.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java
 PRE-CREATION 
  

[jira] [Commented] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13271642#comment-13271642
 ] 

jirapos...@reviews.apache.org commented on HBASE-5732:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4953/#review7738
---


Mostly +1. A couple of minor comments, one question on API annotation.


http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java
https://reviews.apache.org/r/4953/#comment17034

Do we have a testcase for relogin?

In our production with the older implementation of secure HBase RPC, we see 
swarms of GSS initiaition failure due to missing TGT for 5-10 seconds, and we 
speculate this is a race around relogin.



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
https://reviews.apache.org/r/4953/#comment17035

Debug logging should be wrapped in a conditional, no biggie.



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
https://reviews.apache.org/r/4953/#comment17033

This is probably just going to add noise.



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java
https://reviews.apache.org/r/4953/#comment17036

Maybe should be security.hbase.*, what do you think?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java
https://reviews.apache.org/r/4953/#comment17037

It might be better to say 'Kerberos principal does not have the expected 
format'



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java
https://reviews.apache.org/r/4953/#comment17038

Should this be Public+Evolving?



http://svn.apache.org/repos/asf/hbase/trunk/src/test/resources/hbase-site.xml
https://reviews.apache.org/r/4953/#comment17040

Why this change? Unrelated junk from other work?


- Andrew


On 2012-05-08 21:48:09, Devaraj Das wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4953/
bq.  ---
bq.  
bq.  (Updated 2012-05-08 21:48:09)
bq.  
bq.  
bq.  Review request for Ted Yu, Michael Stack and Andrew Purtell.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Reviewboard request for HBASE-5732
bq.  
bq.  
bq.  This addresses bug HBASE-5732.
bq.  https://issues.apache.org/jira/browse/HBASE-5732
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.http://svn.apache.org/repos/asf/hbase/trunk/pom.xml 1335370 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/AdminProtocol.java
 1335370 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ClientProtocol.java
 1335370 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java
 1335370 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java
 1335370 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
 1335370 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/RegionServerStatusProtocol.java
 1335370 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
 1335370 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java
 1335370 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java
 1335370 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlFilter.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java
 PRE-CREATION 
bq.

[jira] [Commented] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13271919#comment-13271919
 ] 

jirapos...@reviews.apache.org commented on HBASE-5732:
--



bq.  On 2012-05-09 18:02:12, Andrew Purtell wrote:
bq.   Mostly +1. A couple of minor comments, one question on API annotation.

Thanks, Andrew. 


bq.  On 2012-05-09 18:02:12, Andrew Purtell wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java,
 line 677
bq.   https://reviews.apache.org/r/4953/diff/4/?file=107883#file107883line677
bq.  
bq.   Do we have a testcase for relogin?
bq.   
bq.   In our production with the older implementation of secure HBase RPC, 
we see swarms of GSS initiaition failure due to missing TGT for 5-10 seconds, 
and we speculate this is a race around relogin.

It's not straightforward to write a testcase for the security APIs, 
unfortunately. I'd propose we open a jira for the relogin issue (over in 
hadoop, iirc, we've fixed some issues to do with relogin races in the recent 
past; we should put the fixes here)...


bq.  On 2012-05-09 18:02:12, Andrew Purtell wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java,
 line 1328
bq.   
https://reviews.apache.org/r/4953/diff/4/?file=107884#file107884line1328
bq.  
bq.   Debug logging should be wrapped in a conditional, no biggie.

Done.


bq.  On 2012-05-09 18:02:12, Andrew Purtell wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java,
 line 1937
bq.   
https://reviews.apache.org/r/4953/diff/4/?file=107884#file107884line1937
bq.  
bq.   This is probably just going to add noise.

Removed.


bq.  On 2012-05-09 18:02:12, Andrew Purtell wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java,
 line 35
bq.   https://reviews.apache.org/r/4953/diff/4/?file=107889#file107889line35
bq.  
bq.   Maybe should be security.hbase.*, what do you think?

This is how it is in the current trunk.  But yeah, makes sense to have the key 
as security.hbase.*.


bq.  On 2012-05-09 18:02:12, Andrew Purtell wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java,
 line 94
bq.   https://reviews.apache.org/r/4953/diff/4/?file=107890#file107890line94
bq.  
bq.   It might be better to say 'Kerberos principal does not have the 
expected format'

Done.


bq.  On 2012-05-09 18:02:12, Andrew Purtell wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java,
 line 53
bq.   https://reviews.apache.org/r/4953/diff/4/?file=107892#file107892line53
bq.  
bq.   Should this be Public+Evolving?

Done. Although I was wondering whether it makes sense to have it as 
Private+Evolving. But if it is currently used by apps outside the core of 
hbase, it makes sense to have it as Public+Evolving..


bq.  On 2012-05-09 18:02:12, Andrew Purtell wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/test/resources/hbase-site.xml, 
line 126
bq.   https://reviews.apache.org/r/4953/diff/4/?file=107918#file107918line126
bq.  
bq.   Why this change? Unrelated junk from other work?

In the hbase-site.xml of security/src/test/resources, this block of 
configuration was there. Now that there is a single profile, I thought I should 
not lose this and added this in the src/test/resources/hbase-site.xml..


- Devaraj


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4953/#review7738
---


On 2012-05-09 23:03:45, Devaraj Das wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4953/
bq.  ---
bq.  
bq.  (Updated 2012-05-09 23:03:45)
bq.  
bq.  
bq.  Review request for Ted Yu, Michael Stack and Andrew Purtell.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Reviewboard request for HBASE-5732
bq.  
bq.  
bq.  This addresses bug HBASE-5732.
bq.  https://issues.apache.org/jira/browse/HBASE-5732
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.http://svn.apache.org/repos/asf/hbase/trunk/conf/hbase-policy.xml 
1335370 
bq.http://svn.apache.org/repos/asf/hbase/trunk/pom.xml 1335370 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/AdminProtocol.java
 1335370 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ClientProtocol.java
 1335370 
bq.

[jira] [Commented] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13270295#comment-13270295
 ] 

jirapos...@reviews.apache.org commented on HBASE-5732:
--



bq.  On 2012-05-01 23:02:34, Michael Stack wrote:
bq.   Here's a quick review of this fat patch.  Good stuff.

Yeah the patch is fat but most of it is refactoring and moving classes around.


bq.  On 2012-05-01 23:02:34, Michael Stack wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java,
 line 67
bq.   https://reviews.apache.org/r/4953/diff/1/?file=105837#file105837line67
bq.  
bq.   Should this security stuff be moved down here into ipc package?  Is 
it only place where security is referenced?

This is not the only class.. there are a bunch of classes present there..


bq.  On 2012-05-01 23:02:34, Michael Stack wrote:
bq.   http://svn.apache.org/repos/asf/hbase/trunk/pom.xml, line 1677
bq.   
https://reviews.apache.org/r/4953/diff/1/?file=105835#file105835line1677
bq.  
bq.   Now the underlying hadoop must support all the security apis?
bq.   
bq.   If not present, will we compile?

Yes, this will necessitate hadoop-1.0++..


bq.  On 2012-05-01 23:02:34, Michael Stack wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java,
 line 81
bq.   https://reviews.apache.org/r/4953/diff/1/?file=105837#file105837line81
bq.  
bq.   So, if underlying hadoop does not have these classes, we'll fail the 
build?

yes


bq.  On 2012-05-01 23:02:34, Michael Stack wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java,
 line 283
bq.   https://reviews.apache.org/r/4953/diff/1/?file=105837#file105837line283
bq.  
bq.   Should this be a fail?

This was what was there in the original code (SecureClient.java). I left it as 
it was..


bq.  On 2012-05-01 23:02:34, Michael Stack wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java,
 line 1495
bq.   
https://reviews.apache.org/r/4953/diff/1/?file=105838#file105838line1495
bq.  
bq.   Can you not give byte array to pb to parse?  Use builder and 
mergeFrom?  Not important.

Yeah this is not making any additional copy.


bq.  On 2012-05-01 23:02:34, Michael Stack wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java,
 line 67
bq.   https://reviews.apache.org/r/4953/diff/1/?file=105840#file105840line67
bq.  
bq.   Has to be public?

No.. I reverted this change.


bq.  On 2012-05-01 23:02:34, Michael Stack wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java,
 line 252
bq.   https://reviews.apache.org/r/4953/diff/1/?file=105840#file105840line252
bq.  
bq.   This is a pity removing the static?

Reverted.


bq.  On 2012-05-01 23:02:34, Michael Stack wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java,
 line 343
bq.   https://reviews.apache.org/r/4953/diff/1/?file=105840#file105840line343
bq.  
bq.   Where do these get shut down?

These are daemon threads (most of them are the RPC threads that we currently 
have). They will get shut down with the process.


bq.  On 2012-05-01 23:02:34, Michael Stack wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/UserPermission.java,
 line 138
bq.   https://reviews.apache.org/r/4953/diff/1/?file=105854#file105854line138
bq.  
bq.   This stuff is copied over from the /security dir in hbase?  Are 
there corresponding deletes?  What about some tests?

The patch has the deleted files marked as such. An example is:
Index: 
security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java
 (deleted)

I haven't added any tests as part of this patch since this is mostly 
refactoring. Existing tests cover the scenarios. Also, I am manually testing 
the Kerberos parts.


- Devaraj


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4953/#review7455
---


On 2012-05-01 20:27:30, Devaraj Das wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4953/
bq.  ---
bq.  
bq.  (Updated 2012-05-01 20:27:30)
bq.  
bq.  
bq.  Review request for Ted Yu, Michael Stack and Andrew Purtell.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Reviewboard request for HBASE-5732
bq.  
bq.  
bq.  

[jira] [Commented] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13270296#comment-13270296
 ] 

jirapos...@reviews.apache.org commented on HBASE-5732:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4953/
---

(Updated 2012-05-08 07:45:11.526354)


Review request for Ted Yu, Michael Stack and Andrew Purtell.


Changes
---

Updated patch. Tested manually with Kerberos.


Summary
---

Reviewboard request for HBASE-5732


This addresses bug HBASE-5732.
https://issues.apache.org/jira/browse/HBASE-5732


Diffs (updated)
-

  http://svn.apache.org/repos/asf/hbase/trunk/pom.xml 1335359 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/AdminProtocol.java
 1335359 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ClientProtocol.java
 1335359 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java
 1335359 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java
 1335359 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
 1335359 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/RegionServerStatusProtocol.java
 1335359 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
 1335359 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java
 1335359 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java
 1335359 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlFilter.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControllerProtocol.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/Permission.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/TablePermission.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/UserPermission.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/ZKPermissionWatcher.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java
 PRE-CREATION 
  http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPC.proto 
1335359 
  

[jira] [Commented] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13270565#comment-13270565
 ] 

jirapos...@reviews.apache.org commented on HBASE-5732:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4953/#review7684
---


It looks like testing turned up some interesting issues going by the diff 
between this version and the previous.  It works now?

Good on you DD.


http://svn.apache.org/repos/asf/hbase/trunk/pom.xml
https://reviews.apache.org/r/4953/#comment16912

Is this you or just difference between your patch and trunk?  i.e. did you 
pull this in?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
https://reviews.apache.org/r/4953/#comment16914

Should this be null so we fail w/ NPE if we try send a saslResponse w/o 
having first done a setSaslTokenResponse?

rb shows added white space.



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
https://reviews.apache.org/r/4953/#comment16915

Is this a bug fix?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
https://reviews.apache.org/r/4953/#comment16916

When do we do this?  When would we forego pb encoding?  When its done 
already?

Is this fix something that came of your manual testing?  If so, I heart 
testing!



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
https://reviews.apache.org/r/4953/#comment16917

Is this going to be annoying?   Happens on each rpc?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
https://reviews.apache.org/r/4953/#comment16918

ditto?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
https://reviews.apache.org/r/4953/#comment16919

Good.  Nice.



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
https://reviews.apache.org/r/4953/#comment16920

Also good.  Nice.



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
https://reviews.apache.org/r/4953/#comment16921

Woah.  Where'd this come from?  What was this doing in here?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
https://reviews.apache.org/r/4953/#comment16922

Ditto on the What!?  What was this doing in here (smile).



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/AdminProtocol.java
https://reviews.apache.org/r/4953/#comment16913

Ours is different from hadoops?


- Michael


On 2012-05-08 07:45:11, Devaraj Das wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4953/
bq.  ---
bq.  
bq.  (Updated 2012-05-08 07:45:11)
bq.  
bq.  
bq.  Review request for Ted Yu, Michael Stack and Andrew Purtell.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Reviewboard request for HBASE-5732
bq.  
bq.  
bq.  This addresses bug HBASE-5732.
bq.  https://issues.apache.org/jira/browse/HBASE-5732
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.http://svn.apache.org/repos/asf/hbase/trunk/pom.xml 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/AdminProtocol.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ClientProtocol.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/RegionServerStatusProtocol.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java
 PRE-CREATION 
bq.

[jira] [Commented] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13270718#comment-13270718
 ] 

jirapos...@reviews.apache.org commented on HBASE-5732:
--



bq.  On 2012-05-08 15:51:46, Michael Stack wrote:
bq.   It looks like testing turned up some interesting issues going by the 
diff between this version and the previous.  It works now?
bq.   
bq.   Good on you DD.

Yes, my testing (1 master, 1 regionserver, 1 client all authenticating on 
Kerberos with each other) didn't show up any issues. I exercised some of the 
shell commands from the client (create, list, put, get).


bq.  On 2012-05-08 15:51:46, Michael Stack wrote:
bq.   http://svn.apache.org/repos/asf/hbase/trunk/pom.xml, line 560
bq.   
https://reviews.apache.org/r/4953/diff/1-2/?file=105835#file105835line560
bq.  
bq.   Is this you or just difference between your patch and trunk?  i.e. 
did you pull this in?

Not sure. Went and looked at the last patch and it seemed to contain exactly 
what i intend to have in.


bq.  On 2012-05-08 15:51:46, Michael Stack wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java,
 line 1253
bq.   
https://reviews.apache.org/r/4953/diff/1-2/?file=105838#file105838line1253
bq.  
bq.   Is this a bug fix?

Yes .. in testing I realized that the whole block of code need to be 
conditional.. (and this is the case currently in SecureServer.java from where 
this is taken)


bq.  On 2012-05-08 15:51:46, Michael Stack wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java,
 line 1341
bq.   
https://reviews.apache.org/r/4953/diff/1-2/?file=105838#file105838line1341
bq.  
bq.   When do we do this?  When would we forego pb encoding?  When its 
done already?
bq.   
bq.   Is this fix something that came of your manual testing?  If so, I 
heart testing!

The sasl handshake is not part of the PB messaging. Yes, I discovered that I 
forgot to take care of this until I tested with security :-)


bq.  On 2012-05-08 15:51:46, Michael Stack wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java,
 line 1376
bq.   
https://reviews.apache.org/r/4953/diff/1-2/?file=105838#file105838line1376
bq.  
bq.   Is this going to be annoying?   Happens on each rpc?

Will revert (was for my debugging)


bq.  On 2012-05-08 15:51:46, Michael Stack wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java,
 line 1396
bq.   
https://reviews.apache.org/r/4953/diff/1-2/?file=105838#file105838line1396
bq.  
bq.   ditto?

Will revert (was for my debugging)


bq.  On 2012-05-08 15:51:46, Michael Stack wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java,
 line 259
bq.   
https://reviews.apache.org/r/4953/diff/1-2/?file=105840#file105840line259
bq.  
bq.   Woah.  Where'd this come from?  What was this doing in here?

This was an unused field.. So I removed it..


bq.  On 2012-05-08 15:51:46, Michael Stack wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java,
 line 298
bq.   
https://reviews.apache.org/r/4953/diff/1-2/?file=105840#file105840line298
bq.  
bq.   Ditto on the What!?  What was this doing in here (smile).

Will revert the What!? (was for my debugging) :-)


bq.  On 2012-05-08 15:51:46, Michael Stack wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/AdminProtocol.java,
 line 25
bq.   https://reviews.apache.org/r/4953/diff/2/?file=107703#file107703line25
bq.  
bq.   Ours is different from hadoops?

I don't know what the problem was .. maybe some obscure classpath issue .. but 
changing it to use hbase's class solved the problem.


bq.  On 2012-05-08 15:51:46, Michael Stack wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java,
 line 1149
bq.   
https://reviews.apache.org/r/4953/diff/1-2/?file=105838#file105838line1149
bq.  
bq.   Should this be null so we fail w/ NPE if we try send a saslResponse 
w/o having first done a setSaslTokenResponse?
bq.   
bq.   rb shows added white space.

Will remove the white-spaces that are introduced by this patch.

I am not sure I follow your question on the NPE.. But I'll do some cleanup on 
this anyway... 


- Devaraj


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4953/#review7684
---


On 2012-05-08 07:45:11, Devaraj Das wrote:
bq.  
bq.  

[jira] [Commented] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13270722#comment-13270722
 ] 

jirapos...@reviews.apache.org commented on HBASE-5732:
--



bq.  On 2012-05-02 23:02:22, Michael Stack wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java,
 line 1
bq.   https://reviews.apache.org/r/4953/diff/1/?file=105842#file105842line1
bq.  
bq.   This exception should be at top level in hbase?

Did you address this in your subsequent patch?


bq.  On 2012-05-02 23:02:22, Michael Stack wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java,
 line 19
bq.   https://reviews.apache.org/r/4953/diff/1/?file=105846#file105846line19
bq.  
bq.   This class should go up to the top level of hbase and not be hidden 
down here in security now it is used by both secure and insecure hbase?

Ditto


bq.  On 2012-05-02 23:02:22, Michael Stack wrote:
bq.   http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPC.proto, 
line 50
bq.   https://reviews.apache.org/r/4953/diff/1/?file=105864#file105864line50
bq.  
bq.   So, if no user, its insecure hbase?  Good.
bq.   
bq.   I don't see you regenerating pb stuff after making these changes in 
this proto file.

What about above?


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4953/#review7488
---


On 2012-05-08 07:45:11, Devaraj Das wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4953/
bq.  ---
bq.  
bq.  (Updated 2012-05-08 07:45:11)
bq.  
bq.  
bq.  Review request for Ted Yu, Michael Stack and Andrew Purtell.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Reviewboard request for HBASE-5732
bq.  
bq.  
bq.  This addresses bug HBASE-5732.
bq.  https://issues.apache.org/jira/browse/HBASE-5732
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.http://svn.apache.org/repos/asf/hbase/trunk/pom.xml 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/AdminProtocol.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ClientProtocol.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/RegionServerStatusProtocol.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlFilter.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControllerProtocol.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/Permission.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java
 PRE-CREATION 
bq.

[jira] [Commented] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13270724#comment-13270724
 ] 

jirapos...@reviews.apache.org commented on HBASE-5732:
--



bq.  On 2012-05-08 15:51:46, Michael Stack wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java,
 line 1149
bq.   
https://reviews.apache.org/r/4953/diff/1-2/?file=105838#file105838line1149
bq.  
bq.   Should this be null so we fail w/ NPE if we try send a saslResponse 
w/o having first done a setSaslTokenResponse?
bq.   
bq.   rb shows added white space.
bq.  
bq.  Devaraj Das wrote:
bq.  Will remove the white-spaces that are introduced by this patch.
bq.  
bq.  I am not sure I follow your question on the NPE.. But I'll do some 
cleanup on this anyway...

Above you assign saslResponse a null buffer.  Later there is a method to set 
the sasl response buffer.  I was suggesting you not assign a  null buffer so we 
fail fast w/ a NPE in those places where we forget to do a set of the sasl 
response.


bq.  On 2012-05-08 15:51:46, Michael Stack wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java,
 line 259
bq.   
https://reviews.apache.org/r/4953/diff/1-2/?file=105840#file105840line259
bq.  
bq.   Woah.  Where'd this come from?  What was this doing in here?
bq.  
bq.  Devaraj Das wrote:
bq.  This was an unused field.. So I removed it..

Good.  Thanks.


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4953/#review7684
---


On 2012-05-08 07:45:11, Devaraj Das wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4953/
bq.  ---
bq.  
bq.  (Updated 2012-05-08 07:45:11)
bq.  
bq.  
bq.  Review request for Ted Yu, Michael Stack and Andrew Purtell.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Reviewboard request for HBASE-5732
bq.  
bq.  
bq.  This addresses bug HBASE-5732.
bq.  https://issues.apache.org/jira/browse/HBASE-5732
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.http://svn.apache.org/repos/asf/hbase/trunk/pom.xml 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/AdminProtocol.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ClientProtocol.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/RegionServerStatusProtocol.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlFilter.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControllerProtocol.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/Permission.java
 PRE-CREATION 
bq.

[jira] [Commented] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13270791#comment-13270791
 ] 

jirapos...@reviews.apache.org commented on HBASE-5732:
--



bq.  On 2012-05-02 23:02:22, Michael Stack wrote:
bq.  

I am not sure what step I missed the last time (when I answered your questions) 
that reviewboard didn't publish the responses.. Trying again.


bq.  On 2012-05-02 23:02:22, Michael Stack wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java,
 line 1
bq.   https://reviews.apache.org/r/4953/diff/1/?file=105842#file105842line1
bq.  
bq.   This exception should be at top level in hbase?
bq.  
bq.  Michael Stack wrote:
bq.  Did you address this in your subsequent patch?

I left it where it was originally. I think its fine as is..


bq.  On 2012-05-02 23:02:22, Michael Stack wrote:
bq.   http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPC.proto, 
line 50
bq.   https://reviews.apache.org/r/4953/diff/1/?file=105864#file105864line50
bq.  
bq.   So, if no user, its insecure hbase?  Good.
bq.   
bq.   I don't see you regenerating pb stuff after making these changes in 
this proto file.
bq.  
bq.  Michael Stack wrote:
bq.  What about above?

On the first question, its to do with proxy users. Basically, 'proxy-user' 
allows someone like the Oozie server to access HBase on behalf of some other 
effective user. The HBase server could still keep track of who is the the real 
user and who is the effective user. Documented here - 
http://hadoop.apache.org/common/docs/current/Secure_Impersonation.html ). Again 
this was already there in the original code. I made the realuser optional since 
its not always going to be there.

There is actually - RPCProtos.java


bq.  On 2012-05-02 23:02:22, Michael Stack wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/test/resources/hbase-site.xml, 
line 129
bq.   https://reviews.apache.org/r/4953/diff/1/?file=105872#file105872line129
bq.  
bq.   What is this?  Mistake?

I merged in the stuff from hbase-site.xml from the security/src/test/resources 
into the src/test/resources one since the security one would go away (yeah you 
won't know about it unless you do a manual diff of the two hbase-site.xml 
files).


bq.  On 2012-05-02 23:02:22, Michael Stack wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java,
 line 19
bq.   https://reviews.apache.org/r/4953/diff/1/?file=105846#file105846line19
bq.  
bq.   This class should go up to the top level of hbase and not be hidden 
down here in security now it is used by both secure and insecure hbase?
bq.  
bq.  Michael Stack wrote:
bq.  Ditto

I'd like to leave it as is since the class aims to shim the security related 
aspects of 'User' (Other than that it would save lots of lines in the patch if 
the package name is kept intact).


- Devaraj


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4953/#review7488
---


On 2012-05-08 07:45:11, Devaraj Das wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4953/
bq.  ---
bq.  
bq.  (Updated 2012-05-08 07:45:11)
bq.  
bq.  
bq.  Review request for Ted Yu, Michael Stack and Andrew Purtell.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Reviewboard request for HBASE-5732
bq.  
bq.  
bq.  This addresses bug HBASE-5732.
bq.  https://issues.apache.org/jira/browse/HBASE-5732
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.http://svn.apache.org/repos/asf/hbase/trunk/pom.xml 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/AdminProtocol.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ClientProtocol.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/RegionServerStatusProtocol.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
 1335359 
bq.

[jira] [Commented] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13270793#comment-13270793
 ] 

jirapos...@reviews.apache.org commented on HBASE-5732:
--



bq.  On 2012-05-02 23:02:22, Michael Stack wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java,
 line 19
bq.   https://reviews.apache.org/r/4953/diff/1/?file=105846#file105846line19
bq.  
bq.   This class should go up to the top level of hbase and not be hidden 
down here in security now it is used by both secure and insecure hbase?
bq.  
bq.  Michael Stack wrote:
bq.  Ditto
bq.  
bq.  Devaraj Das wrote:
bq.  I'd like to leave it as is since the class aims to shim the security 
related aspects of 'User' (Other than that it would save lots of lines in the 
patch if the package name is kept intact).

np


bq.  On 2012-05-02 23:02:22, Michael Stack wrote:
bq.   http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPC.proto, 
line 50
bq.   https://reviews.apache.org/r/4953/diff/1/?file=105864#file105864line50
bq.  
bq.   So, if no user, its insecure hbase?  Good.
bq.   
bq.   I don't see you regenerating pb stuff after making these changes in 
this proto file.
bq.  
bq.  Michael Stack wrote:
bq.  What about above?
bq.  
bq.  Devaraj Das wrote:
bq.  On the first question, its to do with proxy users. Basically, 
'proxy-user' allows someone like the Oozie server to access HBase on behalf of 
some other effective user. The HBase server could still keep track of who is 
the the real user and who is the effective user. Documented here - 
http://hadoop.apache.org/common/docs/current/Secure_Impersonation.html ). Again 
this was already there in the original code. I made the realuser optional since 
its not always going to be there.
bq.  
bq.  There is actually - RPCProtos.java

I see 'RPCProtos.java' but IIRC, there is no corresponding changes in 
regenerated classes to match the changes in RPCProtos.java?  Did I miss 
something?


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4953/#review7488
---


On 2012-05-08 07:45:11, Devaraj Das wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4953/
bq.  ---
bq.  
bq.  (Updated 2012-05-08 07:45:11)
bq.  
bq.  
bq.  Review request for Ted Yu, Michael Stack and Andrew Purtell.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Reviewboard request for HBASE-5732
bq.  
bq.  
bq.  This addresses bug HBASE-5732.
bq.  https://issues.apache.org/jira/browse/HBASE-5732
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.http://svn.apache.org/repos/asf/hbase/trunk/pom.xml 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/AdminProtocol.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ClientProtocol.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/RegionServerStatusProtocol.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlFilter.java
 PRE-CREATION 
bq.

[jira] [Commented] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13270796#comment-13270796
 ] 

jirapos...@reviews.apache.org commented on HBASE-5732:
--



bq.  On 2012-05-02 23:02:22, Michael Stack wrote:
bq.   http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPC.proto, 
line 50
bq.   https://reviews.apache.org/r/4953/diff/1/?file=105864#file105864line50
bq.  
bq.   So, if no user, its insecure hbase?  Good.
bq.   
bq.   I don't see you regenerating pb stuff after making these changes in 
this proto file.
bq.  
bq.  Michael Stack wrote:
bq.  What about above?
bq.  
bq.  Devaraj Das wrote:
bq.  On the first question, its to do with proxy users. Basically, 
'proxy-user' allows someone like the Oozie server to access HBase on behalf of 
some other effective user. The HBase server could still keep track of who is 
the the real user and who is the effective user. Documented here - 
http://hadoop.apache.org/common/docs/current/Secure_Impersonation.html ). Again 
this was already there in the original code. I made the realuser optional since 
its not always going to be there.
bq.  
bq.  There is actually - RPCProtos.java
bq.  
bq.  Michael Stack wrote:
bq.  I see 'RPCProtos.java' but IIRC, there is no corresponding changes in 
regenerated classes to match the changes in RPCProtos.java?  Did I miss 
something?

The RPCProtos.java is the regenerated class.. The proto definition is in 
RPC.proto.. Both the files are there in the patch.


- Devaraj


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4953/#review7488
---


On 2012-05-08 07:45:11, Devaraj Das wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4953/
bq.  ---
bq.  
bq.  (Updated 2012-05-08 07:45:11)
bq.  
bq.  
bq.  Review request for Ted Yu, Michael Stack and Andrew Purtell.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Reviewboard request for HBASE-5732
bq.  
bq.  
bq.  This addresses bug HBASE-5732.
bq.  https://issues.apache.org/jira/browse/HBASE-5732
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.http://svn.apache.org/repos/asf/hbase/trunk/pom.xml 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/AdminProtocol.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ClientProtocol.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/RegionServerStatusProtocol.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlFilter.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControllerProtocol.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/Permission.java
 PRE-CREATION 
bq.

[jira] [Commented] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13270829#comment-13270829
 ] 

jirapos...@reviews.apache.org commented on HBASE-5732:
--



bq.  On 2012-05-02 23:02:22, Michael Stack wrote:
bq.   http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPC.proto, 
line 50
bq.   https://reviews.apache.org/r/4953/diff/1/?file=105864#file105864line50
bq.  
bq.   So, if no user, its insecure hbase?  Good.
bq.   
bq.   I don't see you regenerating pb stuff after making these changes in 
this proto file.
bq.  
bq.  Michael Stack wrote:
bq.  What about above?
bq.  
bq.  Devaraj Das wrote:
bq.  On the first question, its to do with proxy users. Basically, 
'proxy-user' allows someone like the Oozie server to access HBase on behalf of 
some other effective user. The HBase server could still keep track of who is 
the the real user and who is the effective user. Documented here - 
http://hadoop.apache.org/common/docs/current/Secure_Impersonation.html ). Again 
this was already there in the original code. I made the realuser optional since 
its not always going to be there.
bq.  
bq.  There is actually - RPCProtos.java
bq.  
bq.  Michael Stack wrote:
bq.  I see 'RPCProtos.java' but IIRC, there is no corresponding changes in 
regenerated classes to match the changes in RPCProtos.java?  Did I miss 
something?
bq.  
bq.  Devaraj Das wrote:
bq.  The RPCProtos.java is the regenerated class.. The proto definition is 
in RPC.proto.. Both the files are there in the patch.

I see.  My oversight.  Thanks.

So, you are going to make a new version of this patch to commit?  Good on you 
DD.


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4953/#review7488
---


On 2012-05-08 07:45:11, Devaraj Das wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4953/
bq.  ---
bq.  
bq.  (Updated 2012-05-08 07:45:11)
bq.  
bq.  
bq.  Review request for Ted Yu, Michael Stack and Andrew Purtell.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Reviewboard request for HBASE-5732
bq.  
bq.  
bq.  This addresses bug HBASE-5732.
bq.  https://issues.apache.org/jira/browse/HBASE-5732
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.http://svn.apache.org/repos/asf/hbase/trunk/pom.xml 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/AdminProtocol.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ClientProtocol.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/RegionServerStatusProtocol.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java
 1335359 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlFilter.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControllerProtocol.java
 

[jira] [Commented] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13270850#comment-13270850
 ] 

jirapos...@reviews.apache.org commented on HBASE-5732:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4953/
---

(Updated 2012-05-08 21:33:06.528218)


Review request for Ted Yu, Michael Stack and Andrew Purtell.


Changes
---

This is the updated patch with the last few comments incorporated.


Summary
---

Reviewboard request for HBASE-5732


This addresses bug HBASE-5732.
https://issues.apache.org/jira/browse/HBASE-5732


Diffs (updated)
-

  http://svn.apache.org/repos/asf/hbase/trunk/pom.xml 1335370 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/AdminProtocol.java
 1335370 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ClientProtocol.java
 1335370 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java
 1335370 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java
 1335370 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
 1335370 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/RegionServerStatusProtocol.java
 1335370 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
 1335370 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java
 1335370 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java
 1335370 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlFilter.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControllerProtocol.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/Permission.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/TablePermission.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/UserPermission.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/ZKPermissionWatcher.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java
 PRE-CREATION 
  http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPC.proto 

[jira] [Commented] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13270865#comment-13270865
 ] 

jirapos...@reviews.apache.org commented on HBASE-5732:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4953/
---

(Updated 2012-05-08 21:48:09.490984)


Review request for Ted Yu, Michael Stack and Andrew Purtell.


Changes
---

Sigh.. had forgotten to remove the white spaces.. This patch addresses that. 
Please note that I haven't removed white spaces from the files that got moved 
around. I only covered the stuff I added in the current files.


Summary
---

Reviewboard request for HBASE-5732


This addresses bug HBASE-5732.
https://issues.apache.org/jira/browse/HBASE-5732


Diffs (updated)
-

  http://svn.apache.org/repos/asf/hbase/trunk/pom.xml 1335370 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/AdminProtocol.java
 1335370 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ClientProtocol.java
 1335370 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java
 1335370 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java
 1335370 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
 1335370 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/RegionServerStatusProtocol.java
 1335370 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
 1335370 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java
 1335370 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java
 1335370 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlFilter.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControllerProtocol.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/Permission.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/TablePermission.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/UserPermission.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/ZKPermissionWatcher.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java
 PRE-CREATION 
  

[jira] [Commented] (HBASE-5935) Add Region-level PB-based calls to HMasterInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13270906#comment-13270906
 ] 

jirapos...@reviews.apache.org commented on HBASE-5935:
--



bq.  On 2012-05-08 04:55:21, Michael Stack wrote:
bq.   Looks good.  Some small potatoes below.  On this, I've put the new 
generated calls in HMasterInterface rather than a new class, because it is 
easier to have one class for now, until I convert everything.  This will only 
be temporary, but let me know how you want to handle.  Bump the version 
number? ... when will HMasterInterface go away?  What else all is to be done?  
Sure on bump the version number.  Pretty soon they'll be ignored so no harm 
using up a few.  Good stuff G.

Cool, I'll bump the version number for now.

I believe I've answered all your questions below.  Let me know if I missed 
anything.

Thanks for the review.


bq.  On 2012-05-08 04:55:21, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/master/HMaster.java, line 1213
bq.   
https://reviews.apache.org/r/5060/diff/2/?file=107651#file107651line1213
bq.  
bq.   FYI, spaces between if and open paren... and usually around 
operators.  Just FYI.

Thanks, will do.


bq.  On 2012-05-08 04:55:21, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java, line 1394
bq.   
https://reviews.apache.org/r/5060/diff/2/?file=107649#file107649line1394
bq.  
bq.   Is it right catching this here?  You seem to handle it at a higher 
level up in the move invocation?   Won't this clause catch all the possible SEs?
bq.   
bq.   Or I'm suffering myopia  and there is more going on than just this 
call invocation?

Good point, I can catch at a higher level.


bq.  On 2012-05-08 04:55:21, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/master/HMaster.java, line 1219
bq.   
https://reviews.apache.org/r/5060/diff/2/?file=107651#file107651line1219
bq.  
bq.   This needs paren (though its in the original)

Will do.


bq.  On 2012-05-08 04:55:21, Michael Stack wrote:
bq.   src/main/protobuf/Master.proto, line 71
bq.   https://reviews.apache.org/r/5060/diff/2/?file=107654#file107654line71
bq.  
bq.   Great.  Is this all that is in the master interface?

No, the rest of the functions in HMasterInterface need to be converted.  This 
is the region-level stuff, I still have column-level, table-level and 
miscellaneous stuff to do.  I've filed a JIRA for the column-level stuff; I'll 
file JIRAs for the others once I figure out how I'm going to break down the 
work.


bq.  On 2012-05-08 04:55:21, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/master/HMaster.java, line 1253
bq.   
https://reviews.apache.org/r/5060/diff/2/?file=107651#file107651line1253
bq.  
bq.   All IOEs now need to get converted to a SE?  I added an 
HBaseException recently.  Its a checked exception.  Should we have an 
HBaseServiceExceptoin?  SE is pb?

ServiceException is coming from pb, that's right.  So when we either get rid of 
HMasterInterface or repurpose it, the interface will define that we throw 
ServiceExceptions.


bq.  On 2012-05-08 04:55:21, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java, line 285
bq.   https://reviews.apache.org/r/5060/diff/2/?file=107650#file107650line285
bq.  
bq.   It does seem a little messy having to hoist all these pb classes up 
in here AND into HMasterInterface.
bq.   
bq.   HMasterInterface is going to go away?

I'm not sure what you mean by having to hoist all these pb classes up in here 
AND into HMasterInterface. -- this is HMasterInterface?

Also, I don't think PB will be nice and generated usable javadoc for you, so 
I'm not sure how to provide javadoc comments without redeclaring the functions? 
 I admit to being completely ignorant about this area.

Once everything in HMasterInterface is converted to use PB, we can either 
declare a new class for the representation (similar to 
RegionServerStatusProtocol) or just re-purpose HMasterInterface for that.  What 
is your preference?


- Gregory


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5060/#review7667
---


On 2012-05-08 01:34:20, Gregory Chanan wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5060/
bq.  ---
bq.  
bq.  (Updated 2012-05-08 01:34:20)
bq.  
bq.  
bq.  Review request for hbase, Michael Stack and Jimmy Xiang.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Converted all the region-level calls in HMasterInterface to use Protobufs 
(move, assign, unassign).

[jira] [Commented] (HBASE-5935) Add Region-level PB-based calls to HMasterInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13270911#comment-13270911
 ] 

jirapos...@reviews.apache.org commented on HBASE-5935:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5060/
---

(Updated 2012-05-08 22:49:04.533210)


Review request for hbase, Michael Stack and Jimmy Xiang.


Changes
---

Updated for Jimmy's and stack's comments.


Summary
---

Converted all the region-level calls in HMasterInterface to use Protobufs 
(move, assign, unassign).

I've put the new generated calls in HMasterInterface rather than a new class, 
because it is easier to have one class for now, until I convert everything.  
This will only be temporary, but let me know how you want to handle.  Bump the 
version number?


This addresses bug HBASE-5935.
https://issues.apache.org/jira/browse/HBASE-5935


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 5d4be3f 
  src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java ce81547 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java ccc7119 
  src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java a5e03e1 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/MasterProtos.java 
PRE-CREATION 
  src/main/protobuf/Master.proto PRE-CREATION 
  src/main/protobuf/RegionServerStatus.proto 9d7728f 
  src/test/java/org/apache/hadoop/hbase/TestDrainingServer.java e5de603 
  src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java 
83297cc 

Diff: https://reviews.apache.org/r/5060/diff


Testing
---


Thanks,

Gregory



 Add Region-level PB-based calls to HMasterInterface
 ---

 Key: HBASE-5935
 URL: https://issues.apache.org/jira/browse/HBASE-5935
 Project: HBase
  Issue Type: Task
  Components: ipc, master, migration
Reporter: Gregory Chanan
Assignee: Gregory Chanan
 Fix For: 0.96.0


 This should be a subtask of HBASE-5445, but since that is a subtask, I can't 
 also make this a subtask (apparently).
 This is for converting the region-level calls, i.e.:
 moveRegion
 assignRegion
 unassignRegion

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5935) Add Region-level PB-based calls to HMasterInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13270938#comment-13270938
 ] 

jirapos...@reviews.apache.org commented on HBASE-5935:
--



bq.  On 2012-05-08 04:55:21, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java, line 285
bq.   https://reviews.apache.org/r/5060/diff/2/?file=107650#file107650line285
bq.  
bq.   It does seem a little messy having to hoist all these pb classes up 
in here AND into HMasterInterface.
bq.   
bq.   HMasterInterface is going to go away?
bq.  
bq.  Gregory Chanan wrote:
bq.  I'm not sure what you mean by having to hoist all these pb classes up 
in here AND into HMasterInterface. -- this is HMasterInterface?
bq.  
bq.  Also, I don't think PB will be nice and generated usable javadoc for 
you, so I'm not sure how to provide javadoc comments without redeclaring the 
functions?  I admit to being completely ignorant about this area.
bq.  
bq.  Once everything in HMasterInterface is converted to use PB, we can 
either declare a new class for the representation (similar to 
RegionServerStatusProtocol) or just re-purpose HMasterInterface for that.  What 
is your preference?

I was just saying that we have generated pb pollution in both the Interface and 
in the Implementation, but its unavoidable really.

You can't have pb do javadoc (is my understanding) so don't worry about it (Not 
sure what I suggested but it doesn't seem important now)

Lets do what Jimmy did, make a new class and kill the old.


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5060/#review7667
---


On 2012-05-08 22:49:04, Gregory Chanan wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5060/
bq.  ---
bq.  
bq.  (Updated 2012-05-08 22:49:04)
bq.  
bq.  
bq.  Review request for hbase, Michael Stack and Jimmy Xiang.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Converted all the region-level calls in HMasterInterface to use Protobufs 
(move, assign, unassign).
bq.  
bq.  I've put the new generated calls in HMasterInterface rather than a new 
class, because it is easier to have one class for now, until I convert 
everything.  This will only be temporary, but let me know how you want to 
handle.  Bump the version number?
bq.  
bq.  
bq.  This addresses bug HBASE-5935.
bq.  https://issues.apache.org/jira/browse/HBASE-5935
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 5d4be3f 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java ce81547 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java ccc7119 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java 
a5e03e1 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/MasterProtos.java 
PRE-CREATION 
bq.src/main/protobuf/Master.proto PRE-CREATION 
bq.src/main/protobuf/RegionServerStatus.proto 9d7728f 
bq.src/test/java/org/apache/hadoop/hbase/TestDrainingServer.java e5de603 
bq.
src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java 
83297cc 
bq.  
bq.  Diff: https://reviews.apache.org/r/5060/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Gregory
bq.  
bq.



 Add Region-level PB-based calls to HMasterInterface
 ---

 Key: HBASE-5935
 URL: https://issues.apache.org/jira/browse/HBASE-5935
 Project: HBase
  Issue Type: Task
  Components: ipc, master, migration
Reporter: Gregory Chanan
Assignee: Gregory Chanan
 Fix For: 0.96.0

 Attachments: HBASE-5935-v2.patch


 This should be a subtask of HBASE-5445, but since that is a subtask, I can't 
 also make this a subtask (apparently).
 This is for converting the region-level calls, i.e.:
 moveRegion
 assignRegion
 unassignRegion

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5935) Add Region-level PB-based calls to HMasterInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13270942#comment-13270942
 ] 

jirapos...@reviews.apache.org commented on HBASE-5935:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5060/#review7709
---

Ship it!


+1  Put the patch up in JIRA so we can run it by hadoopqa. Good stuff G.

- Michael


On 2012-05-08 22:49:04, Gregory Chanan wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5060/
bq.  ---
bq.  
bq.  (Updated 2012-05-08 22:49:04)
bq.  
bq.  
bq.  Review request for hbase, Michael Stack and Jimmy Xiang.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Converted all the region-level calls in HMasterInterface to use Protobufs 
(move, assign, unassign).
bq.  
bq.  I've put the new generated calls in HMasterInterface rather than a new 
class, because it is easier to have one class for now, until I convert 
everything.  This will only be temporary, but let me know how you want to 
handle.  Bump the version number?
bq.  
bq.  
bq.  This addresses bug HBASE-5935.
bq.  https://issues.apache.org/jira/browse/HBASE-5935
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 5d4be3f 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java ce81547 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java ccc7119 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java 
a5e03e1 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/MasterProtos.java 
PRE-CREATION 
bq.src/main/protobuf/Master.proto PRE-CREATION 
bq.src/main/protobuf/RegionServerStatus.proto 9d7728f 
bq.src/test/java/org/apache/hadoop/hbase/TestDrainingServer.java e5de603 
bq.
src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java 
83297cc 
bq.  
bq.  Diff: https://reviews.apache.org/r/5060/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Gregory
bq.  
bq.



 Add Region-level PB-based calls to HMasterInterface
 ---

 Key: HBASE-5935
 URL: https://issues.apache.org/jira/browse/HBASE-5935
 Project: HBase
  Issue Type: Task
  Components: ipc, master, migration
Reporter: Gregory Chanan
Assignee: Gregory Chanan
 Fix For: 0.96.0

 Attachments: HBASE-5935-v2.patch


 This should be a subtask of HBASE-5445, but since that is a subtask, I can't 
 also make this a subtask (apparently).
 This is for converting the region-level calls, i.e.:
 moveRegion
 assignRegion
 unassignRegion

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5935) Add Region-level PB-based calls to HMasterInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13270065#comment-13270065
 ] 

jirapos...@reviews.apache.org commented on HBASE-5935:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5060/
---

Review request for hbase, Michael Stack and Jimmy Xiang.


Summary
---

Converted all the region-level calls in HMasterInterface to use Protobufs 
(move, assign, unassign).

I've put the new generated calls in HMasterInterface rather than a new class, 
because it is easier to have one class for now, until I convert everything.  
This will only be temporary, but let me know how you want to handle.  Bump the 
version number?


This addresses bug HBASE-5935.
https://issues.apache.org/jira/browse/HBASE-5935


Diffs
-

  src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java e3912c2 
  src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java ce81547 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java ccc7119 
  src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java d1e0993 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/MasterProtos.java 
PRE-CREATION 
  src/main/protobuf/Master.proto PRE-CREATION 
  src/main/protobuf/RegionServerStatus.proto 9d7728f 
  src/test/java/org/apache/hadoop/hbase/TestDrainingServer.java a1992c3 
  src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java 
bdec3ee 

Diff: https://reviews.apache.org/r/5060/diff


Testing
---


Thanks,

Gregory



 Add Region-level PB-based calls to HMasterInterface
 ---

 Key: HBASE-5935
 URL: https://issues.apache.org/jira/browse/HBASE-5935
 Project: HBase
  Issue Type: Task
  Components: ipc, master, migration
Reporter: Gregory Chanan
Assignee: Gregory Chanan
 Fix For: 0.96.0


 This should be a subtask of HBASE-5445, but since that is a subtask, I can't 
 also make this a subtask (apparently).
 This is for converting the region-level calls, i.e.:
 moveRegion
 assignRegion
 unassignRegion

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5935) Add Region-level PB-based calls to HMasterInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13270126#comment-13270126
 ] 

jirapos...@reviews.apache.org commented on HBASE-5935:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5060/
---

(Updated 2012-05-08 01:34:20.873632)


Review request for hbase, Michael Stack and Jimmy Xiang.


Changes
---

Update against newest trunk and added some additional error checking.


Summary
---

Converted all the region-level calls in HMasterInterface to use Protobufs 
(move, assign, unassign).

I've put the new generated calls in HMasterInterface rather than a new class, 
because it is easier to have one class for now, until I convert everything.  
This will only be temporary, but let me know how you want to handle.  Bump the 
version number?


This addresses bug HBASE-5935.
https://issues.apache.org/jira/browse/HBASE-5935


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java ce81547 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java ccc7119 
  src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 5d4be3f 
  src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java a5e03e1 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/MasterProtos.java 
PRE-CREATION 
  src/main/protobuf/Master.proto PRE-CREATION 
  src/main/protobuf/RegionServerStatus.proto 9d7728f 
  src/test/java/org/apache/hadoop/hbase/TestDrainingServer.java e5de603 
  src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java 
83297cc 

Diff: https://reviews.apache.org/r/5060/diff


Testing
---


Thanks,

Gregory



 Add Region-level PB-based calls to HMasterInterface
 ---

 Key: HBASE-5935
 URL: https://issues.apache.org/jira/browse/HBASE-5935
 Project: HBase
  Issue Type: Task
  Components: ipc, master, migration
Reporter: Gregory Chanan
Assignee: Gregory Chanan
 Fix For: 0.96.0


 This should be a subtask of HBASE-5445, but since that is a subtask, I can't 
 also make this a subtask (apparently).
 This is for converting the region-level calls, i.e.:
 moveRegion
 assignRegion
 unassignRegion

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5935) Add Region-level PB-based calls to HMasterInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13270199#comment-13270199
 ] 

jirapos...@reviews.apache.org commented on HBASE-5935:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5060/#review7667
---


Looks good.  Some small potatoes below.  On this, I've put the new generated 
calls in HMasterInterface rather than a new class, because it is easier to have 
one class for now, until I convert everything.  This will only be temporary, 
but let me know how you want to handle.  Bump the version number? ... when 
will HMasterInterface go away?  What else all is to be done?  Sure on bump the 
version number.  Pretty soon they'll be ignored so no harm using up a few.  
Good stuff G.


src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
https://reviews.apache.org/r/5060/#comment16879

Is it right catching this here?  You seem to handle it at a higher level up 
in the move invocation?   Won't this clause catch all the possible SEs?

Or I'm suffering myopia  and there is more going on than just this call 
invocation?



src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java
https://reviews.apache.org/r/5060/#comment16885

OK.  Not even deprecated.  Just removed.  Like Jimmy's remove of 
HRegionInterface and we can do this because these are internal APIs



src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java
https://reviews.apache.org/r/5060/#comment16880

It does seem a little messy having to hoist all these pb classes up in here 
AND into HMasterInterface.

HMasterInterface is going to go away?



src/main/java/org/apache/hadoop/hbase/master/HMaster.java
https://reviews.apache.org/r/5060/#comment16883

This seems like a big improvement, having an actual return.



src/main/java/org/apache/hadoop/hbase/master/HMaster.java
https://reviews.apache.org/r/5060/#comment16881

FYI, spaces between if and open paren... and usually around operators.  
Just FYI.



src/main/java/org/apache/hadoop/hbase/master/HMaster.java
https://reviews.apache.org/r/5060/#comment16882

This needs paren (though its in the original)



src/main/java/org/apache/hadoop/hbase/master/HMaster.java
https://reviews.apache.org/r/5060/#comment16884

All IOEs now need to get converted to a SE?  I added an HBaseException 
recently.  Its a checked exception.  Should we have an HBaseServiceExceptoin?  
SE is pb?



src/main/protobuf/Master.proto
https://reviews.apache.org/r/5060/#comment16886

Great.  Is this all that is in the master interface?



src/main/protobuf/RegionServerStatus.proto
https://reviews.apache.org/r/5060/#comment16887

Good


- Michael


On 2012-05-08 01:34:20, Gregory Chanan wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5060/
bq.  ---
bq.  
bq.  (Updated 2012-05-08 01:34:20)
bq.  
bq.  
bq.  Review request for hbase, Michael Stack and Jimmy Xiang.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Converted all the region-level calls in HMasterInterface to use Protobufs 
(move, assign, unassign).
bq.  
bq.  I've put the new generated calls in HMasterInterface rather than a new 
class, because it is easier to have one class for now, until I convert 
everything.  This will only be temporary, but let me know how you want to 
handle.  Bump the version number?
bq.  
bq.  
bq.  This addresses bug HBASE-5935.
bq.  https://issues.apache.org/jira/browse/HBASE-5935
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java ce81547 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java ccc7119 
bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 5d4be3f 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java 
a5e03e1 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/MasterProtos.java 
PRE-CREATION 
bq.src/main/protobuf/Master.proto PRE-CREATION 
bq.src/main/protobuf/RegionServerStatus.proto 9d7728f 
bq.src/test/java/org/apache/hadoop/hbase/TestDrainingServer.java e5de603 
bq.
src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java 
83297cc 
bq.  
bq.  Diff: https://reviews.apache.org/r/5060/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Gregory
bq.  
bq.



 Add Region-level PB-based calls to HMasterInterface
 ---

 Key: HBASE-5935
 URL: https://issues.apache.org/jira/browse/HBASE-5935
 

[jira] [Commented] (HBASE-5889) Remove HRegionInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13268526#comment-13268526
 ] 

jirapos...@reviews.apache.org commented on HBASE-5889:
--



bq.  On 2012-05-04 01:08:06, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java, line 
1295
bq.   
https://reviews.apache.org/r/4993/diff/1/?file=106388#file106388line1295
bq.  
bq.   HRegionServer util might be better place for this yes.  Or could 
they be put into a new class, RegionServerUtil in the regionserver package?  
Maybe they don't have to be public methods if done this way?
bq.   
bq.   Either sounds good to me boss.

It is good to have a new class RegionServerUtil.  Will do.


- Jimmy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4993/#review7539
---


On 2012-05-03 17:27:50, Jimmy Xiang wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4993/
bq.  ---
bq.  
bq.  (Updated 2012-05-03 17:27:50)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Removed HRegionInterface, and cleaned up the HRegionServer, moved pb code 
from RegionServer back to HRegionServer.
bq.  
bq.  The goal is to avoid two copies of region server code to maintain, and 
make it possible to avoid data type conversion in the sever side.
bq.  
bq.  Fixed some unit tests.  Now all region server unit tests test the new pb 
functions.
bq.  
bq.  Enhanced getServerInfo so that it returns the webui port too.
bq.  
bq.  
bq.  This addresses bug HBASE-5889.
bq.  https://issues.apache.org/jira/browse/HBASE-5889
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.conf/hbase-policy.xml e45f23c 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java
 0c4b4cb 
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
87f04f4 
bq.src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 
bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java e3912c2 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java fc9176d 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 757f98e 
bq.src/main/java/org/apache/hadoop/hbase/mapreduce/TableOutputFormat.java 
cd9b528 
bq.src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 212ee3e 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java 
d1e0993 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java 
81603af 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java 
fbf0127 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java 
db1333b 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 
ae2094d 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java 
8b45f03 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java 
827fb23 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
96ac8bd 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java 
4cb070e 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java 
c2c89ea 
bq.src/main/protobuf/Admin.proto 2ad6fb0 
bq.src/main/protobuf/RPC.proto 105fb3f 
bq.src/main/resources/hbase-default.xml f54b345 
bq.src/main/resources/hbase-webapps/master/table.jsp ca7310c 
bq.src/test/java/org/apache/hadoop/hbase/TestDrainingServer.java a1992c3 
bq.src/test/java/org/apache/hadoop/hbase/TestGlobalMemStoreSize.java 
ad77e0a 
bq.src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java 5574b7f 
bq.src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java 
3dfc94e 
bq.
src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java 
42092b7 
bq.src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java c270e28 
bq.src/test/java/org/apache/hadoop/hbase/client/TestMultiParallel.java 
c36272f 
bq.
src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java 
bdec3ee 
bq.
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
 7dbba66 
bq.
src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java
 3acb988 
bq.
src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java 
eb546a5 
bq.

[jira] [Commented] (HBASE-5889) Remove HRegionInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13268600#comment-13268600
 ] 

jirapos...@reviews.apache.org commented on HBASE-5889:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4993/
---

(Updated 2012-05-04 18:16:37.788666)


Review request for hbase.


Changes
---

Addressed Stack's comments.


Summary
---

Removed HRegionInterface, and cleaned up the HRegionServer, moved pb code from 
RegionServer back to HRegionServer.

The goal is to avoid two copies of region server code to maintain, and make it 
possible to avoid data type conversion in the sever side.

Fixed some unit tests.  Now all region server unit tests test the new pb 
functions.

Enhanced getServerInfo so that it returns the webui port too.


This addresses bug HBASE-5889.
https://issues.apache.org/jira/browse/HBASE-5889


Diffs (updated)
-

  conf/hbase-policy.xml e45f23c 
  
security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java
 fda40cc 
  src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
87f04f4 
  src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 
  src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java 8a383e4 
  src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java e3912c2 
  src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 11d8bf9 
  src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4 
  src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java 578b2b2 
  src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java fc9176d 
  src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 757f98e 
  src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 
9e4ada9 
  src/main/java/org/apache/hadoop/hbase/mapreduce/TableOutputFormat.java 
cd9b528 
  src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd 
  src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 6ba8ab0 
  src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 212ee3e 
  src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java d1e0993 
  src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java 81603af 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java 
fbf0127 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java 
db1333b 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 
ae2094d 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java 
8b45f03 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java 
827fb23 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8c8381b 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java 
4cb070e 
  src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java c2c89ea 
  src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerUtil.java 
PRE-CREATION 
  
src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
 5050df0 
  src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 7b4f4a2 
  src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java 9c3c9ef 
  src/main/protobuf/Admin.proto 2ad6fb0 
  src/main/protobuf/RPC.proto 105fb3f 
  src/main/resources/hbase-default.xml f54b345 
  src/test/java/org/apache/hadoop/hbase/TestDrainingServer.java a1992c3 
  src/test/java/org/apache/hadoop/hbase/TestGlobalMemStoreSize.java ad77e0a 
  src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java 5574b7f 
  src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java 3dfc94e 
  src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java 
42092b7 
  src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java c270e28 
  src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java 0079b13 
  src/test/java/org/apache/hadoop/hbase/client/TestMultiParallel.java c36272f 
  src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java 
bdec3ee 
  
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
 7dbba66 
  
src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java
 3acb988 
  src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java 
eb546a5 
  src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java ceba5cd 
  
src/test/java/org/apache/hadoop/hbase/master/TestMasterRestartAfterDisablingTable.java
 ec08b17 
  src/test/java/org/apache/hadoop/hbase/master/TestRollingRestart.java 30c6cf1 
  src/test/java/org/apache/hadoop/hbase/master/TestZKBasedOpenCloseRegion.java 
8c3f67e 

[jira] [Commented] (HBASE-5889) Remove HRegionInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13268785#comment-13268785
 ] 

jirapos...@reviews.apache.org commented on HBASE-5889:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4993/
---

(Updated 2012-05-04 22:54:46.398933)


Review request for hbase.


Summary
---

Removed HRegionInterface, and cleaned up the HRegionServer, moved pb code from 
RegionServer back to HRegionServer.

The goal is to avoid two copies of region server code to maintain, and make it 
possible to avoid data type conversion in the sever side.

Fixed some unit tests.  Now all region server unit tests test the new pb 
functions.

Enhanced getServerInfo so that it returns the webui port too.


This addresses bug HBASE-5889.
https://issues.apache.org/jira/browse/HBASE-5889


Diffs (updated)
-

  conf/hbase-policy.xml e45f23c 
  
security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java
 fda40cc 
  src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
87f04f4 
  src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 
  src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java e3912c2 
  src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java fc9176d 
  src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 757f98e 
  src/main/java/org/apache/hadoop/hbase/mapreduce/TableOutputFormat.java 
cd9b528 
  src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd 
  src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 6ba8ab0 
  src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 212ee3e 
  src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java d1e0993 
  src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java 81603af 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java 
fbf0127 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java 
db1333b 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 
ae2094d 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java 
8b45f03 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java 
827fb23 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8c8381b 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java 
4cb070e 
  src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java c2c89ea 
  src/main/protobuf/Admin.proto 2ad6fb0 
  src/main/protobuf/RPC.proto 105fb3f 
  src/main/resources/hbase-default.xml f54b345 
  src/test/java/org/apache/hadoop/hbase/TestDrainingServer.java a1992c3 
  src/test/java/org/apache/hadoop/hbase/TestGlobalMemStoreSize.java ad77e0a 
  src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java 5574b7f 
  src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java 3dfc94e 
  src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java 
42092b7 
  src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java c270e28 
  src/test/java/org/apache/hadoop/hbase/client/TestMultiParallel.java c36272f 
  src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java 
bdec3ee 
  
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
 7dbba66 
  
src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java
 3acb988 
  src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java 
eb546a5 
  src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java ceba5cd 
  
src/test/java/org/apache/hadoop/hbase/master/TestMasterRestartAfterDisablingTable.java
 ec08b17 
  src/test/java/org/apache/hadoop/hbase/master/TestRollingRestart.java 30c6cf1 
  src/test/java/org/apache/hadoop/hbase/master/TestZKBasedOpenCloseRegion.java 
8c3f67e 
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
 7bfe4cd 
  src/test/java/org/apache/hadoop/hbase/regionserver/TestRSStatusServlet.java 
ffce7e8 
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java 
aa5ca37 
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
 65fa948 
  src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java fc4a0a7 

Diff: https://reviews.apache.org/r/4993/diff


Testing
---

All regular and security profile tests are green before I rebased to the latest 
today.


Thanks,

Jimmy



 Remove HRegionInterface
 ---

 Key: HBASE-5889
 URL: https://issues.apache.org/jira/browse/HBASE-5889
 Project: HBase
  Issue Type: 

[jira] [Commented] (HBASE-5889) Remove HRegionInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13267613#comment-13267613
 ] 

jirapos...@reviews.apache.org commented on HBASE-5889:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4993/
---

Review request for hbase.


Summary
---

Removed HRegionInterface, and cleaned up the HRegionServer, moved pb code from 
RegionServer back to HRegionServer.

The goal is to avoid two copies of region server code to maintain, and make it 
possible to avoid data type conversion in the sever side.

Fixed some unit tests.  Now all region server unit tests test the new pb 
functions.

Enhanced getServerInfo so that it returns the webui port too.


This addresses bug HBASE-5889.
https://issues.apache.org/jira/browse/HBASE-5889


Diffs
-

  conf/hbase-policy.xml e45f23c 
  
security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java
 0c4b4cb 
  src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
87f04f4 
  src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 
  src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java e3912c2 
  src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java fc9176d 
  src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 757f98e 
  src/main/java/org/apache/hadoop/hbase/mapreduce/TableOutputFormat.java 
cd9b528 
  src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd 
  src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 212ee3e 
  src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java d1e0993 
  src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java 81603af 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java 
fbf0127 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java 
db1333b 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 
ae2094d 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java 
8b45f03 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java 
827fb23 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 96ac8bd 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java 
4cb070e 
  src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java c2c89ea 
  src/main/protobuf/Admin.proto 2ad6fb0 
  src/main/protobuf/RPC.proto 105fb3f 
  src/main/resources/hbase-default.xml f54b345 
  src/main/resources/hbase-webapps/master/table.jsp ca7310c 
  src/test/java/org/apache/hadoop/hbase/TestDrainingServer.java a1992c3 
  src/test/java/org/apache/hadoop/hbase/TestGlobalMemStoreSize.java ad77e0a 
  src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java 5574b7f 
  src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java 3dfc94e 
  src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java 
42092b7 
  src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java c270e28 
  src/test/java/org/apache/hadoop/hbase/client/TestMultiParallel.java c36272f 
  src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java 
bdec3ee 
  
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
 7dbba66 
  
src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java
 3acb988 
  src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java 
eb546a5 
  src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java ceba5cd 
  
src/test/java/org/apache/hadoop/hbase/master/TestMasterRestartAfterDisablingTable.java
 ec08b17 
  src/test/java/org/apache/hadoop/hbase/master/TestRollingRestart.java 30c6cf1 
  src/test/java/org/apache/hadoop/hbase/master/TestZKBasedOpenCloseRegion.java 
8c3f67e 
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
 7bfe4cd 
  src/test/java/org/apache/hadoop/hbase/regionserver/TestRSStatusServlet.java 
ffce7e8 
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java 
aa5ca37 
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
 65fa948 
  src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java fc4a0a7 

Diff: https://reviews.apache.org/r/4993/diff


Testing
---

All regular and security profile tests are green before I rebased to the latest 
today.


Thanks,

Jimmy



 Remove HRegionInterface
 ---

 Key: HBASE-5889
 URL: https://issues.apache.org/jira/browse/HBASE-5889
 Project: HBase
  Issue Type: Improvement
  Components: client, ipc, regionserver
   

[jira] [Commented] (HBASE-5676) Note that dfs.support.append does not have to be enabled post 1.x

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13267808#comment-13267808
 ] 

jirapos...@reviews.apache.org commented on HBASE-5676:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4999/
---

Review request for hbase.


Summary
---

Change append checks to sync and hflush checks, in keeping with the spirit of 
the original code.

Should only be applied to 0.96, as only 0.96 will support Hadoop 1.x+ according 
to HBASE-5536.


This addresses bug HBASE-5676.
https://issues.apache.org/jira/browse/HBASE-5676


Diffs
-

  src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon 
6bc2cf2 
  src/main/java/org/apache/hadoop/hbase/master/MasterStatusServlet.java a0aaaf4 
  src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java 302a3b1 
  src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 3d35d3e 
  src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollAbort.java 
3ea0a3b 
  src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 
cb7efc3 
  src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java 339a120 

Diff: https://reviews.apache.org/r/4999/diff


Testing
---

Passed local -PrunAllTests.  Passed test-patch.sh:


+1 overall.  

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 9 new or modified tests.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 findbugs.  The patch does not introduce any new Findbugs (version ) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.


Thanks,

David



 Note that dfs.support.append does not have to be enabled post 1.x
 -

 Key: HBASE-5676
 URL: https://issues.apache.org/jira/browse/HBASE-5676
 Project: HBase
  Issue Type: Task
Reporter: Eli Collins
Assignee: David S. Wang

 In Hadoop 1.x (HADOOP-8230) we are going to enable durable sync by default, 
 and remove the dfs.support.append option. What this means for you:
 - HBase will work out of the box on Hadoop 1.x, no need to tell people to 
 re-configure
 - You no longer have to enable append (which can result in data loss) to 
 enable HBase support

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5676) Note that dfs.support.append does not have to be enabled post 1.x

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13267843#comment-13267843
 ] 

jirapos...@reviews.apache.org commented on HBASE-5676:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4999/#review7524
---


Are you sure this is right? If you're using HBase against Hadoop 1.0, for 
example, the syncFs() call exists, but sync isn't enabled by default, so this 
would give you an incorrect result, I'd think

- Todd


On 2012-05-03 21:22:19, David Wang wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4999/
bq.  ---
bq.  
bq.  (Updated 2012-05-03 21:22:19)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Change append checks to sync and hflush checks, in keeping with the spirit 
of the original code.
bq.  
bq.  Should only be applied to 0.96, as only 0.96 will support Hadoop 1.x+ 
according to HBASE-5536.
bq.  
bq.  
bq.  This addresses bug HBASE-5676.
bq.  https://issues.apache.org/jira/browse/HBASE-5676
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon 
6bc2cf2 
bq.src/main/java/org/apache/hadoop/hbase/master/MasterStatusServlet.java 
a0aaaf4 
bq.src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java 302a3b1 
bq.src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 3d35d3e 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollAbort.java 
3ea0a3b 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 
cb7efc3 
bq.src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java 339a120 
bq.  
bq.  Diff: https://reviews.apache.org/r/4999/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Passed local -PrunAllTests.  Passed test-patch.sh:
bq.  
bq.  
bq.  +1 overall.  
bq.  
bq.  +1 @author.  The patch does not contain any @author tags.
bq.  
bq.  +1 tests included.  The patch appears to include 9 new or modified 
tests.
bq.  
bq.  +1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.
bq.  
bq.  +1 javadoc.  The javadoc tool did not generate any warning messages.
bq.  
bq.  +1 javac.  The applied patch does not increase the total number of 
javac compiler warnings.
bq.  
bq.  +1 findbugs.  The patch does not introduce any new Findbugs (version ) 
warnings.
bq.  
bq.  +1 release audit.  The applied patch does not increase the total 
number of release audit warnings.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  David
bq.  
bq.



 Note that dfs.support.append does not have to be enabled post 1.x
 -

 Key: HBASE-5676
 URL: https://issues.apache.org/jira/browse/HBASE-5676
 Project: HBase
  Issue Type: Task
Affects Versions: 0.96.0
Reporter: Eli Collins
Assignee: David S. Wang
 Attachments: HBASE-5676.patch


 In Hadoop 1.x (HADOOP-8230) we are going to enable durable sync by default, 
 and remove the dfs.support.append option. What this means for you:
 - HBase will work out of the box on Hadoop 1.x, no need to tell people to 
 re-configure
 - You no longer have to enable append (which can result in data loss) to 
 enable HBase support

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5444) Add PB-based calls to HMasterRegionInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13267856#comment-13267856
 ] 

jirapos...@reviews.apache.org commented on HBASE-5444:
--



bq.  On 2012-05-02 23:43:07, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/master/HMaster.java, line 80
bq.   https://reviews.apache.org/r/4463/diff/5/?file=106063#file106063line80
bq.  
bq.   We need this?
bq.  
bq.  Gregory Chanan wrote:
bq.  I use it to convert the PB serverName that is passed into 
HMaster.regionServerReport into a ServerName that the ServerManager 
understands.  Instead, we could have a ServerName static function that takes a 
PB ServerName and returns a ServerName.  We already have a bunch of these 
parse* functions already, e.g.
bq.  
bq.  public static ServerName parseVersionedServerName(final byte [] 
versionedBytes)
bq.  
bq.  Michael Stack wrote:
bq.  Recently in trunk, we added a ServerName.parseFrom that should be able 
to make sense of any set of bytes parsed it whether pbs or old style versioned 
bytes.  Could use this.
bq.  
bq.  Gregory Chanan wrote:
bq.  I don't think this applies.  From reading ServerName.parseFrom it 
looks like it requires the PBMagicPrefix, which this case doesn't have nor 
need.  Am I missing something?

If no pb magic, it then goes on to try and parse the bytes otherwise.  See 
other side of the pb check starting at line #332.


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4463/#review7492
---


On 2012-05-02 23:19:20, Gregory Chanan wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4463/
bq.  ---
bq.  
bq.  (Updated 2012-05-02 23:19:20)
bq.  
bq.  
bq.  Review request for hbase and Michael Stack.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Adds PB-based calls replacing HMasterRegionInterface.
bq.  
bq.  There are some temporary hacks, e.g. converting PB-based ServerLoad to 
existing HServerLoad so I didn't need to convert ClusterStatus (which brings in 
a lot of other changes).  That will be cleaned up in HBASE-5445.
bq.  
bq.  
bq.  This addresses bug HBASE-5444.
bq.  https://issues.apache.org/jira/browse/HBASE-5444
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon 
69434f7 
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
3c7c091 
bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 5d7f07b 
bq.src/main/java/org/apache/hadoop/hbase/ServerLoad.java PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 0db2760 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 973c7cb 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 
fd97830 
bq.src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java bb6ab3b 
bq.
src/main/java/org/apache/hadoop/hbase/ipc/RegionServerStatusProtocol.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 
f56127d 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 81e9023 
bq.src/main/java/org/apache/hadoop/hbase/master/MXBean.java 7f44dc2 
bq.src/main/java/org/apache/hadoop/hbase/master/MXBeanImpl.java 45b8fe7 
bq.src/main/java/org/apache/hadoop/hbase/master/MasterDumpServlet.java 
be63838 
bq.src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 80271b1 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 994cb76 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 
efcf74d 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java
 PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
ebffad6 
bq.src/main/protobuf/RegionServerStatus.proto PRE-CREATION 
bq.src/main/protobuf/hbase.proto 12e6053 
bq.src/main/resources/hbase-webapps/master/table.jsp 3ef1190 
bq.src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 72554cb 
bq.src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java 
d039be3 
bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 
36046f8 
bq.src/test/java/org/apache/hadoop/hbase/master/TestMXBean.java bd5fa90 
bq.src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java 
f8029ba 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java
 e99d251 
bq.  
bq.  Diff: https://reviews.apache.org/r/4463/diff
bq.  
bq.  
bq.  

[jira] [Commented] (HBASE-5889) Remove HRegionInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13267917#comment-13267917
 ] 

jirapos...@reviews.apache.org commented on HBASE-5889:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4993/#review7528
---


Excellent Jimmy.  High-level, too much has been moved to protobufutils IMO.  
Below I highlight where we have gone too far.  What you think?  Otherwise, 
these is not much of substance to my comments below.  I'd be up for committing 
this patch before it rots and addressing issues raised in a new jira if thats 
what you'd prefer.


security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java
https://reviews.apache.org/r/4993/#comment16658

Anyone working on removal of HMasterInterface?

Devaraj will need to pick up these changes if your patch goes in before his.



src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon
https://reviews.apache.org/r/4993/#comment16659

Is this right after Gregory's commit last night?  He hides pb Serverload 
inside a ServerLoad.  Perhaps have to go to pb RegionLoad still even after his 
commit?



src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon
https://reviews.apache.org/r/4993/#comment16660

ok



src/main/java/org/apache/hadoop/hbase/HConstants.java
https://reviews.apache.org/r/4993/#comment16662

These are just not used? 

We used have a mechanism where you could subclass regionserver and then 
configure it so your subclass was started out on the cluster.  You think that 
still possible?



src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
https://reviews.apache.org/r/4993/#comment16663

Hurray!



src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
https://reviews.apache.org/r/4993/#comment16664

Is it going too far adding methods like this into protobufutils?  These 
methods seem pretty core hbase facility, too core to be out in a util method?   
You have them here because there is a bunch of pb'ing going on?

I know I suggested that we move some of the transforms out here... but 
maybe this is taking it a bit far?

What you reckon?



src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
https://reviews.apache.org/r/4993/#comment16665

I think this kinda builder is a the right thing to have over in this util 
class.



src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
https://reviews.apache.org/r/4993/#comment1

Like I said to Gregory last night, its kinda hard hiding this pb stuff when 
you are in the class that is slinging them.



src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
https://reviews.apache.org/r/4993/#comment16667

Is this new or moved code?



src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java
https://reviews.apache.org/r/4993/#comment16668

This was no longer a good idea?



src/main/protobuf/Admin.proto
https://reviews.apache.org/r/4993/#comment16669

good



src/main/resources/hbase-default.xml
https://reviews.apache.org/r/4993/#comment16670

good



src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java
https://reviews.apache.org/r/4993/#comment16671

It is unexpected going to ProtobufUtils to get online regions.



src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java
https://reviews.apache.org/r/4993/#comment16672

ditto


- Michael


On 2012-05-03 17:27:50, Jimmy Xiang wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4993/
bq.  ---
bq.  
bq.  (Updated 2012-05-03 17:27:50)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Removed HRegionInterface, and cleaned up the HRegionServer, moved pb code 
from RegionServer back to HRegionServer.
bq.  
bq.  The goal is to avoid two copies of region server code to maintain, and 
make it possible to avoid data type conversion in the sever side.
bq.  
bq.  Fixed some unit tests.  Now all region server unit tests test the new pb 
functions.
bq.  
bq.  Enhanced getServerInfo so that it returns the webui port too.
bq.  
bq.  
bq.  This addresses bug HBASE-5889.
bq.  https://issues.apache.org/jira/browse/HBASE-5889
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.conf/hbase-policy.xml e45f23c 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java
 0c4b4cb 
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
87f04f4 
bq.

[jira] [Commented] (HBASE-5889) Remove HRegionInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13267928#comment-13267928
 ] 

jirapos...@reviews.apache.org commented on HBASE-5889:
--



bq.  On 2012-05-03 22:57:57, Michael Stack wrote:
bq.   
security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java,
 line 38
bq.   https://reviews.apache.org/r/4993/diff/1/?file=106380#file106380line38
bq.  
bq.   Anyone working on removal of HMasterInterface?
bq.   
bq.   Devaraj will need to pick up these changes if your patch goes in 
before his.

I'm working on the HMasterInterface.  See HBASE-5445.


bq.  On 2012-05-03 22:57:57, Michael Stack wrote:
bq.   
src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon, 
line 41
bq.   https://reviews.apache.org/r/4993/diff/1/?file=106381#file106381line41
bq.  
bq.   Is this right after Gregory's commit last night?  He hides pb 
Serverload inside a ServerLoad.  Perhaps have to go to pb RegionLoad still even 
after his commit?

There isn't currently a RegionLoad that hides pb RegionLoad in the way you've 
described with ServerLoad.  I actually wrote one up, but didn't include it 
because there is an existing RegionLoad that is needed until the 
HMasterInterface is done.  I felt it made more sense to wait rather than spend 
a bunch of time renaming stuff.

If this patch is committed as is, let's file a JIRA and assign it to me?


- Gregory


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4993/#review7528
---


On 2012-05-03 17:27:50, Jimmy Xiang wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4993/
bq.  ---
bq.  
bq.  (Updated 2012-05-03 17:27:50)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Removed HRegionInterface, and cleaned up the HRegionServer, moved pb code 
from RegionServer back to HRegionServer.
bq.  
bq.  The goal is to avoid two copies of region server code to maintain, and 
make it possible to avoid data type conversion in the sever side.
bq.  
bq.  Fixed some unit tests.  Now all region server unit tests test the new pb 
functions.
bq.  
bq.  Enhanced getServerInfo so that it returns the webui port too.
bq.  
bq.  
bq.  This addresses bug HBASE-5889.
bq.  https://issues.apache.org/jira/browse/HBASE-5889
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.conf/hbase-policy.xml e45f23c 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java
 0c4b4cb 
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
87f04f4 
bq.src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 
bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java e3912c2 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java fc9176d 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 757f98e 
bq.src/main/java/org/apache/hadoop/hbase/mapreduce/TableOutputFormat.java 
cd9b528 
bq.src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 212ee3e 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java 
d1e0993 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java 
81603af 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java 
fbf0127 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java 
db1333b 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 
ae2094d 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java 
8b45f03 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java 
827fb23 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
96ac8bd 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java 
4cb070e 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java 
c2c89ea 
bq.src/main/protobuf/Admin.proto 2ad6fb0 
bq.src/main/protobuf/RPC.proto 105fb3f 
bq.src/main/resources/hbase-default.xml f54b345 
bq.src/main/resources/hbase-webapps/master/table.jsp ca7310c 
bq.src/test/java/org/apache/hadoop/hbase/TestDrainingServer.java a1992c3 
bq.src/test/java/org/apache/hadoop/hbase/TestGlobalMemStoreSize.java 
ad77e0a 
bq.src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java 5574b7f 
bq.src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java 

[jira] [Commented] (HBASE-5889) Remove HRegionInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13267931#comment-13267931
 ] 

jirapos...@reviews.apache.org commented on HBASE-5889:
--



bq.  On 2012-05-03 22:57:57, Michael Stack wrote:
bq.   
src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon, 
line 41
bq.   https://reviews.apache.org/r/4993/diff/1/?file=106381#file106381line41
bq.  
bq.   Is this right after Gregory's commit last night?  He hides pb 
Serverload inside a ServerLoad.  Perhaps have to go to pb RegionLoad still even 
after his commit?
bq.  
bq.  Gregory Chanan wrote:
bq.  There isn't currently a RegionLoad that hides pb RegionLoad in the way 
you've described with ServerLoad.  I actually wrote one up, but didn't include 
it because there is an existing RegionLoad that is needed until the 
HMasterInterface is done.  I felt it made more sense to wait rather than spend 
a bunch of time renaming stuff.
bq.  
bq.  If this patch is committed as is, let's file a JIRA and assign it to 
me?

Grand!


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4993/#review7528
---


On 2012-05-03 17:27:50, Jimmy Xiang wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4993/
bq.  ---
bq.  
bq.  (Updated 2012-05-03 17:27:50)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Removed HRegionInterface, and cleaned up the HRegionServer, moved pb code 
from RegionServer back to HRegionServer.
bq.  
bq.  The goal is to avoid two copies of region server code to maintain, and 
make it possible to avoid data type conversion in the sever side.
bq.  
bq.  Fixed some unit tests.  Now all region server unit tests test the new pb 
functions.
bq.  
bq.  Enhanced getServerInfo so that it returns the webui port too.
bq.  
bq.  
bq.  This addresses bug HBASE-5889.
bq.  https://issues.apache.org/jira/browse/HBASE-5889
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.conf/hbase-policy.xml e45f23c 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java
 0c4b4cb 
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
87f04f4 
bq.src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 
bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java e3912c2 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java fc9176d 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 757f98e 
bq.src/main/java/org/apache/hadoop/hbase/mapreduce/TableOutputFormat.java 
cd9b528 
bq.src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 212ee3e 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java 
d1e0993 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java 
81603af 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java 
fbf0127 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java 
db1333b 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 
ae2094d 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java 
8b45f03 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java 
827fb23 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
96ac8bd 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java 
4cb070e 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java 
c2c89ea 
bq.src/main/protobuf/Admin.proto 2ad6fb0 
bq.src/main/protobuf/RPC.proto 105fb3f 
bq.src/main/resources/hbase-default.xml f54b345 
bq.src/main/resources/hbase-webapps/master/table.jsp ca7310c 
bq.src/test/java/org/apache/hadoop/hbase/TestDrainingServer.java a1992c3 
bq.src/test/java/org/apache/hadoop/hbase/TestGlobalMemStoreSize.java 
ad77e0a 
bq.src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java 5574b7f 
bq.src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java 
3dfc94e 
bq.
src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java 
42092b7 
bq.src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java c270e28 
bq.src/test/java/org/apache/hadoop/hbase/client/TestMultiParallel.java 
c36272f 
bq.
src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java 
bdec3ee 
bq.

[jira] [Commented] (HBASE-5889) Remove HRegionInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13267955#comment-13267955
 ] 

jirapos...@reviews.apache.org commented on HBASE-5889:
--



bq.  On 2012-05-03 22:57:57, Michael Stack wrote:
bq.   
src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon, 
line 41
bq.   https://reviews.apache.org/r/4993/diff/1/?file=106381#file106381line41
bq.  
bq.   Is this right after Gregory's commit last night?  He hides pb 
Serverload inside a ServerLoad.  Perhaps have to go to pb RegionLoad still even 
after his commit?
bq.  
bq.  Gregory Chanan wrote:
bq.  There isn't currently a RegionLoad that hides pb RegionLoad in the way 
you've described with ServerLoad.  I actually wrote one up, but didn't include 
it because there is an existing RegionLoad that is needed until the 
HMasterInterface is done.  I felt it made more sense to wait rather than spend 
a bunch of time renaming stuff.
bq.  
bq.  If this patch is committed as is, let's file a JIRA and assign it to 
me?
bq.  
bq.  Michael Stack wrote:
bq.  Grand!

I rebased to trunk and got Gregory's patch.


bq.  On 2012-05-03 22:57:57, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/HConstants.java, line 195
bq.   https://reviews.apache.org/r/4993/diff/1/?file=106382#file106382line195
bq.  
bq.   These are just not used? 
bq.   
bq.   We used have a mechanism where you could subclass regionserver and 
then configure it so your subclass was started out on the cluster.  You think 
that still possible?

It is still possible. Now, if we want to subclass regionserver, we need to use 
HConstants.REGION_SERVER_IMPL.  The HRegionInterface is not there any more.


bq.  On 2012-05-03 22:57:57, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java, 
line 185
bq.   https://reviews.apache.org/r/4993/diff/1/?file=106390#file106390line185
bq.  
bq.   I think this kinda builder is a the right thing to have over in this 
util class.

You are right.  I am concerned with the size of the util class. So I'd like to 
have several special util classes.


bq.  On 2012-05-03 22:57:57, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java, 
line 248
bq.   https://reviews.apache.org/r/4993/diff/1/?file=106396#file106396line248
bq.  
bq.   Is this new or moved code?

It is moved from RegionServer, which is originally fro HRegionServer.  Sorry 
for the confusion.


bq.  On 2012-05-03 22:57:57, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java, 
line 1657
bq.   
https://reviews.apache.org/r/4993/diff/1/?file=106398#file106398line1657
bq.  
bq.   This was no longer a good idea?

It is hard to maintain two copies of implementation, especially one is not unit 
tested any more.


bq.  On 2012-05-03 22:57:57, Michael Stack wrote:
bq.   
src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java, line 
688
bq.   https://reviews.apache.org/r/4993/diff/1/?file=106410#file106410line688
bq.  
bq.   It is unexpected going to ProtobufUtils to get online regions.

This method is used in many places.  Should I put the util in HRegionServer as 
a static helper method?


bq.  On 2012-05-03 22:57:57, Michael Stack wrote:
bq.   src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java, 
line 315
bq.   https://reviews.apache.org/r/4993/diff/1/?file=106414#file106414line315
bq.  
bq.   ditto

How about put it in HRegionServer as a static helper util?


bq.  On 2012-05-03 22:57:57, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java, line 
1295
bq.   
https://reviews.apache.org/r/4993/diff/1/?file=106388#file106388line1295
bq.  
bq.   Is it going too far adding methods like this into protobufutils?  
These methods seem pretty core hbase facility, too core to be out in a util 
method?   You have them here because there is a bunch of pb'ing going on?
bq.   
bq.   I know I suggested that we move some of the transforms out here... 
but maybe this is taking it a bit far?
bq.   
bq.   What you reckon?

openRegion is used in many test classes. That's why I have this in the util.  
Should I put in HRegionServer as a static helper util?


- Jimmy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4993/#review7528
---


On 2012-05-03 17:27:50, Jimmy Xiang wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4993/
bq.  ---
bq.  
bq.  (Updated 2012-05-03 

[jira] [Commented] (HBASE-5889) Remove HRegionInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13268001#comment-13268001
 ] 

jirapos...@reviews.apache.org commented on HBASE-5889:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4993/#review7539
---



src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
https://reviews.apache.org/r/4993/#comment16698

HRegionServer util might be better place for this yes.  Or could they be 
put into a new class, RegionServerUtil in the regionserver package?  Maybe they 
don't have to be public methods if done this way?

Either sounds good to me boss.


- Michael


On 2012-05-03 17:27:50, Jimmy Xiang wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4993/
bq.  ---
bq.  
bq.  (Updated 2012-05-03 17:27:50)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Removed HRegionInterface, and cleaned up the HRegionServer, moved pb code 
from RegionServer back to HRegionServer.
bq.  
bq.  The goal is to avoid two copies of region server code to maintain, and 
make it possible to avoid data type conversion in the sever side.
bq.  
bq.  Fixed some unit tests.  Now all region server unit tests test the new pb 
functions.
bq.  
bq.  Enhanced getServerInfo so that it returns the webui port too.
bq.  
bq.  
bq.  This addresses bug HBASE-5889.
bq.  https://issues.apache.org/jira/browse/HBASE-5889
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.conf/hbase-policy.xml e45f23c 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java
 0c4b4cb 
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
87f04f4 
bq.src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 
bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java e3912c2 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java fc9176d 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 757f98e 
bq.src/main/java/org/apache/hadoop/hbase/mapreduce/TableOutputFormat.java 
cd9b528 
bq.src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 212ee3e 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java 
d1e0993 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java 
81603af 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java 
fbf0127 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java 
db1333b 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 
ae2094d 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java 
8b45f03 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java 
827fb23 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
96ac8bd 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java 
4cb070e 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java 
c2c89ea 
bq.src/main/protobuf/Admin.proto 2ad6fb0 
bq.src/main/protobuf/RPC.proto 105fb3f 
bq.src/main/resources/hbase-default.xml f54b345 
bq.src/main/resources/hbase-webapps/master/table.jsp ca7310c 
bq.src/test/java/org/apache/hadoop/hbase/TestDrainingServer.java a1992c3 
bq.src/test/java/org/apache/hadoop/hbase/TestGlobalMemStoreSize.java 
ad77e0a 
bq.src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java 5574b7f 
bq.src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java 
3dfc94e 
bq.
src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java 
42092b7 
bq.src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java c270e28 
bq.src/test/java/org/apache/hadoop/hbase/client/TestMultiParallel.java 
c36272f 
bq.
src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java 
bdec3ee 
bq.
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
 7dbba66 
bq.
src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java
 3acb988 
bq.
src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java 
eb546a5 
bq.src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java 
ceba5cd 
bq.
src/test/java/org/apache/hadoop/hbase/master/TestMasterRestartAfterDisablingTable.java
 ec08b17 
bq.

[jira] [Commented] (HBASE-5889) Remove HRegionInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13268002#comment-13268002
 ] 

jirapos...@reviews.apache.org commented on HBASE-5889:
--



bq.  On 2012-05-03 22:57:57, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java, 
line 1657
bq.   
https://reviews.apache.org/r/4993/diff/1/?file=106398#file106398line1657
bq.  
bq.   This was no longer a good idea?
bq.  
bq.  Jimmy Xiang wrote:
bq.  It is hard to maintain two copies of implementation, especially one is 
not unit tested any more.

Ok.


bq.  On 2012-05-03 22:57:57, Michael Stack wrote:
bq.   
src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java, line 
688
bq.   https://reviews.apache.org/r/4993/diff/1/?file=106410#file106410line688
bq.  
bq.   It is unexpected going to ProtobufUtils to get online regions.
bq.  
bq.  Jimmy Xiang wrote:
bq.  This method is used in many places.  Should I put the util in 
HRegionServer as a static helper method?

See above.

In fact isn't onlineRegions a class of its own?  If so, could be a static 
method in there?


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4993/#review7528
---


On 2012-05-03 17:27:50, Jimmy Xiang wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4993/
bq.  ---
bq.  
bq.  (Updated 2012-05-03 17:27:50)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Removed HRegionInterface, and cleaned up the HRegionServer, moved pb code 
from RegionServer back to HRegionServer.
bq.  
bq.  The goal is to avoid two copies of region server code to maintain, and 
make it possible to avoid data type conversion in the sever side.
bq.  
bq.  Fixed some unit tests.  Now all region server unit tests test the new pb 
functions.
bq.  
bq.  Enhanced getServerInfo so that it returns the webui port too.
bq.  
bq.  
bq.  This addresses bug HBASE-5889.
bq.  https://issues.apache.org/jira/browse/HBASE-5889
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.conf/hbase-policy.xml e45f23c 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java
 0c4b4cb 
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
87f04f4 
bq.src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 
bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java e3912c2 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java fc9176d 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 757f98e 
bq.src/main/java/org/apache/hadoop/hbase/mapreduce/TableOutputFormat.java 
cd9b528 
bq.src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 212ee3e 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java 
d1e0993 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java 
81603af 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java 
fbf0127 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java 
db1333b 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 
ae2094d 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java 
8b45f03 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java 
827fb23 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
96ac8bd 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java 
4cb070e 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java 
c2c89ea 
bq.src/main/protobuf/Admin.proto 2ad6fb0 
bq.src/main/protobuf/RPC.proto 105fb3f 
bq.src/main/resources/hbase-default.xml f54b345 
bq.src/main/resources/hbase-webapps/master/table.jsp ca7310c 
bq.src/test/java/org/apache/hadoop/hbase/TestDrainingServer.java a1992c3 
bq.src/test/java/org/apache/hadoop/hbase/TestGlobalMemStoreSize.java 
ad77e0a 
bq.src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java 5574b7f 
bq.src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java 
3dfc94e 
bq.
src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java 
42092b7 
bq.src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java c270e28 
bq.src/test/java/org/apache/hadoop/hbase/client/TestMultiParallel.java 
c36272f 
bq.

[jira] [Commented] (HBASE-5625) Avoid byte buffer allocations when reading a value from a Result object

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266435#comment-13266435
 ] 

jirapos...@reviews.apache.org commented on HBASE-5625:
--



bq.  On 2012-04-28 23:39:41, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/KeyValue.java, line 181
bq.   https://reviews.apache.org/r/4607/diff/3/?file=104237#file104237line181
bq.  
bq.   So you have plans to use these elsewhere in the codebase?  If so, 
its ok that they are public.

Replaced manual calculations of infrastructure sizes with calls to the new 
methods in 'KeyValue'.


bq.  On 2012-04-28 23:39:41, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/KeyValue.java, line 542
bq.   https://reviews.apache.org/r/4607/diff/3/?file=104237#file104237line542
bq.  
bq.   This could be 'as its backing data buffer'?  If so, I can address 
that on commit

Fixed.


bq.  On 2012-04-28 23:39:41, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/Result.java, line 256
bq.   https://reviews.apache.org/r/4607/diff/3/?file=104238#file104238line256
bq.  
bq.   Here you are trying to make a smart guess on a buffer size that will 
serve for multiple invocations on binarySearch?  Your hope is that you'll not 
have to reallocate the buffer the next time you come through here because the 
buffer should have enough space in it to hold the next random KV that comes 
through here?

Exactly. The power of 2 implementation could have grown too rapidly, so I 
modified it to pad to a size equal to the smallest multiple of a configurable 
constant.


- Tudor


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4607/#review7365
---


On 2012-04-25 16:01:29, Tudor Scurtu wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4607/
bq.  ---
bq.  
bq.  (Updated 2012-04-25 16:01:29)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  When calling Result.getValue(), an extra dummy KeyValue and its associated 
underlying byte array are allocated, as well as a persistent buffer that will 
contain the returned value.
bq.  
bq.  These can be avoided by reusing a static array for the dummy object and by 
passing a ByteBuffer object as a value destination buffer to the read method.
bq.  
bq.  
bq.  This addresses bug HBASE-5625.
bq.  https://issues.apache.org/jira/browse/HBASE-5625
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/KeyValue.java 9ae9e02 
bq.src/main/java/org/apache/hadoop/hbase/client/Result.java df0b3ef 
bq.src/test/java/org/apache/hadoop/hbase/TestKeyValue.java 786d2df 
bq.src/test/java/org/apache/hadoop/hbase/client/TestResult.java f9e29c2 
bq.  
bq.  Diff: https://reviews.apache.org/r/4607/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Added value check to TestResult#testBasic and TestResult.testMultiVersion.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Tudor
bq.  
bq.



 Avoid byte buffer allocations when reading a value from a Result object
 ---

 Key: HBASE-5625
 URL: https://issues.apache.org/jira/browse/HBASE-5625
 Project: HBase
  Issue Type: Improvement
  Components: client
Affects Versions: 0.92.1
Reporter: Tudor Scurtu
Assignee: Tudor Scurtu
  Labels: patch
 Fix For: 0.96.0

 Attachments: 5625.txt, 5625v2.txt, 5625v3.txt, 5625v4.txt, 
 5625v5.txt, 5625v6.txt, 5625v7.txt, 5625v8.txt


 When calling Result.getValue(), an extra dummy KeyValue and its associated 
 underlying byte array are allocated, as well as a persistent buffer that will 
 contain the returned value.
 These can be avoided by reusing a static array for the dummy object and by 
 passing a ByteBuffer object as a value destination buffer to the read method.
 The current functionality is maintained, and we have added a separate method 
 call stack that employs the described changes. I will provide more details 
 with the patch.
 Running tests with a profiler, the reduction of read time seems to be of up 
 to 40%.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5625) Avoid byte buffer allocations when reading a value from a Result object

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266436#comment-13266436
 ] 

jirapos...@reviews.apache.org commented on HBASE-5625:
--



bq.  On 2012-04-02 17:34:38, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/KeyValue.java, line 616
bq.   https://reviews.apache.org/r/4607/diff/1/?file=97954#file97954line616
bq.  
bq.   How do I know the buffer is big enough?
bq.  
bq.  Tudor Scurtu wrote:
bq.  Added exception comment for when there is insufficient space remaining 
in the buffer. Is that what you meant?
bq.  
bq.  Michael Stack wrote:
bq.  I am not understanding how the allocation works.   It seems arbitrary 
unrelated to the actual result size that comes over from the server.  Is that 
so?  If so, it seems unfriendly throwing an exception when allocated size and 
what is returned from the server do not match.
bq.  
bq.  Tudor Scurtu wrote:
bq.  Added check with reallocation in 'Result.binarySearch()'. For this I 
had to add two methods in 'KeyValue' that calculate the number of bytes that 
are taken up in a 'KeyValue' object's underlying buffer 
('getKeyValueDataStructureSize()' and 'getKeyDataStructureSize()'). Is this ok, 
and if so, how about replacing all manual calculations of these values in the 
project with calls to the new methods?
bq.  
bq.  Michael Stack wrote:
bq.  I think I am beginning to understand what you are at (pardon me, I am 
a little slow).  You want to speed up finding KVs in big Results and part of 
the way in which you do this is reuse of a buffer you keep private in Result.  
The buffer will not match a specific KV usually it'll be too big and if it 
is too small, you'll allocate a buffer big enough, a new one.
bq.  
bq.  What locations would you put getKeyValueDataStructureSize into place?  
For example?

Replaced manual calculations of infrastructure sizes with calls to the new 
methods in 'KeyValue'.


- Tudor


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4607/#review6623
---


On 2012-04-25 16:01:29, Tudor Scurtu wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4607/
bq.  ---
bq.  
bq.  (Updated 2012-04-25 16:01:29)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  When calling Result.getValue(), an extra dummy KeyValue and its associated 
underlying byte array are allocated, as well as a persistent buffer that will 
contain the returned value.
bq.  
bq.  These can be avoided by reusing a static array for the dummy object and by 
passing a ByteBuffer object as a value destination buffer to the read method.
bq.  
bq.  
bq.  This addresses bug HBASE-5625.
bq.  https://issues.apache.org/jira/browse/HBASE-5625
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/KeyValue.java 9ae9e02 
bq.src/main/java/org/apache/hadoop/hbase/client/Result.java df0b3ef 
bq.src/test/java/org/apache/hadoop/hbase/TestKeyValue.java 786d2df 
bq.src/test/java/org/apache/hadoop/hbase/client/TestResult.java f9e29c2 
bq.  
bq.  Diff: https://reviews.apache.org/r/4607/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Added value check to TestResult#testBasic and TestResult.testMultiVersion.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Tudor
bq.  
bq.



 Avoid byte buffer allocations when reading a value from a Result object
 ---

 Key: HBASE-5625
 URL: https://issues.apache.org/jira/browse/HBASE-5625
 Project: HBase
  Issue Type: Improvement
  Components: client
Affects Versions: 0.92.1
Reporter: Tudor Scurtu
Assignee: Tudor Scurtu
  Labels: patch
 Fix For: 0.96.0

 Attachments: 5625.txt, 5625v2.txt, 5625v3.txt, 5625v4.txt, 
 5625v5.txt, 5625v6.txt, 5625v7.txt, 5625v8.txt


 When calling Result.getValue(), an extra dummy KeyValue and its associated 
 underlying byte array are allocated, as well as a persistent buffer that will 
 contain the returned value.
 These can be avoided by reusing a static array for the dummy object and by 
 passing a ByteBuffer object as a value destination buffer to the read method.
 The current functionality is maintained, and we have added a separate method 
 call stack that employs the described changes. I will provide more details 
 with the patch.
 Running tests with a profiler, the reduction of read time seems to be of up 
 to 40%.

--
This message is 

[jira] [Commented] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266994#comment-13266994
 ] 

jirapos...@reviews.apache.org commented on HBASE-5732:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4953/#review7488
---



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java
https://reviews.apache.org/r/4953/#comment16579

This exception should be at top level in hbase?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java
https://reviews.apache.org/r/4953/#comment16578

This class should go up to the top level of hbase and not be hidden down 
here in security now it is used by both secure and insecure hbase?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPC.proto
https://reviews.apache.org/r/4953/#comment16580

I missed this page of edits.

There is no corresponding remove of the /security directory.  Should it be 
included here?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPC.proto
https://reviews.apache.org/r/4953/#comment16581

So, if no user, its insecure hbase?  Good.

I don't see you regenerating pb stuff after making these changes in this 
proto file.



http://svn.apache.org/repos/asf/hbase/trunk/src/test/resources/hbase-site.xml
https://reviews.apache.org/r/4953/#comment16582

What is this?  Mistake?


- Michael


On 2012-05-01 20:27:30, Devaraj Das wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4953/
bq.  ---
bq.  
bq.  (Updated 2012-05-01 20:27:30)
bq.  
bq.  
bq.  Review request for Ted Yu, Michael Stack and Andrew Purtell.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Reviewboard request for HBASE-5732
bq.  
bq.  
bq.  This addresses bug HBASE-5732.
bq.  https://issues.apache.org/jira/browse/HBASE-5732
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java
 1332383 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java
 1332383 
bq.http://svn.apache.org/repos/asf/hbase/trunk/pom.xml 1332383 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
 1332383 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/Status.java
 1332383 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
 1332383 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java
 1332383 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java
 1332383 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlFilter.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControllerProtocol.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/Permission.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/TablePermission.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/UserPermission.java
 PRE-CREATION 

[jira] [Commented] (HBASE-5444) Add PB-based calls to HMasterRegionInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13267011#comment-13267011
 ] 

jirapos...@reviews.apache.org commented on HBASE-5444:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4463/
---

(Updated 2012-05-02 23:19:20.036448)


Review request for hbase and Michael Stack.


Changes
---

Make changes based on Ted's and Stack's comments.

- Fixed spacing
- Moved RegionServerStatusProtocol from master to ipc (not o.a.h.h -- hopefully 
that's okay)
- Created a ServerLoad object that encapsulated the PB ServerLoad object.


Summary
---

Adds PB-based calls replacing HMasterRegionInterface.

There are some temporary hacks, e.g. converting PB-based ServerLoad to existing 
HServerLoad so I didn't need to convert ClusterStatus (which brings in a lot of 
other changes).  That will be cleaned up in HBASE-5445.


This addresses bug HBASE-5444.
https://issues.apache.org/jira/browse/HBASE-5444


Diffs (updated)
-

  src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon 
69434f7 
  src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
3c7c091 
  src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 5d7f07b 
  src/main/java/org/apache/hadoop/hbase/ServerLoad.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 0db2760 
  src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 973c7cb 
  src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java fd97830 
  src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java bb6ab3b 
  src/main/java/org/apache/hadoop/hbase/ipc/RegionServerStatusProtocol.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java f56127d 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java 81e9023 
  src/main/java/org/apache/hadoop/hbase/master/MXBean.java 7f44dc2 
  src/main/java/org/apache/hadoop/hbase/master/MXBeanImpl.java 45b8fe7 
  src/main/java/org/apache/hadoop/hbase/master/MasterDumpServlet.java be63838 
  src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 80271b1 
  src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 994cb76 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 
efcf74d 
  
src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java
 PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java ebffad6 
  src/main/protobuf/RegionServerStatus.proto PRE-CREATION 
  src/main/protobuf/hbase.proto 12e6053 
  src/main/resources/hbase-webapps/master/table.jsp 3ef1190 
  src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 72554cb 
  src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java 
d039be3 
  src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 
36046f8 
  src/test/java/org/apache/hadoop/hbase/master/TestMXBean.java bd5fa90 
  src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java f8029ba 
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java
 e99d251 

Diff: https://reviews.apache.org/r/4463/diff


Testing
---

Ran jenkins job, all unit tests passed.


Thanks,

Gregory



 Add PB-based calls to HMasterRegionInterface
 

 Key: HBASE-5444
 URL: https://issues.apache.org/jira/browse/HBASE-5444
 Project: HBase
  Issue Type: Sub-task
  Components: ipc, master, migration, regionserver
Reporter: Todd Lipcon
Assignee: Gregory Chanan
 Attachments: HBASE-5444-v6-trunk.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5444) Add PB-based calls to HMasterRegionInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13267038#comment-13267038
 ] 

jirapos...@reviews.apache.org commented on HBASE-5444:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4463/#review7492
---


I'm +1 on this patch.  I think this much cleaner than previous versions.  I 
wanted more, of course, where users of the protocol would somehow be untouched 
or polluted by pbs but I realize that is asking for to much.  Good stuff 
Gregory.


src/main/java/org/apache/hadoop/hbase/ClusterStatus.java
https://reviews.apache.org/r/4463/#comment16588

We need this import?  Its for cp.  Thats ok I'd say One day we can hide 
that too..



src/main/java/org/apache/hadoop/hbase/master/HMaster.java
https://reviews.apache.org/r/4463/#comment16590

We can move the ipc protocol stuff to top level later... I was thinking 
that these classes shared by master and regionservers could be at o.a.h.h... 
but can do that later if it makes sense.  Lets get this pb stuff in first.





src/main/java/org/apache/hadoop/hbase/master/HMaster.java
https://reviews.apache.org/r/4463/#comment16589

We need this?



src/main/java/org/apache/hadoop/hbase/master/HMaster.java
https://reviews.apache.org/r/4463/#comment16591

Yeah, I suppose you can't hide these from the class that is implementing 
the protocol... 


- Michael


On 2012-05-02 23:19:20, Gregory Chanan wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4463/
bq.  ---
bq.  
bq.  (Updated 2012-05-02 23:19:20)
bq.  
bq.  
bq.  Review request for hbase and Michael Stack.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Adds PB-based calls replacing HMasterRegionInterface.
bq.  
bq.  There are some temporary hacks, e.g. converting PB-based ServerLoad to 
existing HServerLoad so I didn't need to convert ClusterStatus (which brings in 
a lot of other changes).  That will be cleaned up in HBASE-5445.
bq.  
bq.  
bq.  This addresses bug HBASE-5444.
bq.  https://issues.apache.org/jira/browse/HBASE-5444
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon 
69434f7 
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
3c7c091 
bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 5d7f07b 
bq.src/main/java/org/apache/hadoop/hbase/ServerLoad.java PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 0db2760 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 973c7cb 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 
fd97830 
bq.src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java bb6ab3b 
bq.
src/main/java/org/apache/hadoop/hbase/ipc/RegionServerStatusProtocol.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 
f56127d 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 81e9023 
bq.src/main/java/org/apache/hadoop/hbase/master/MXBean.java 7f44dc2 
bq.src/main/java/org/apache/hadoop/hbase/master/MXBeanImpl.java 45b8fe7 
bq.src/main/java/org/apache/hadoop/hbase/master/MasterDumpServlet.java 
be63838 
bq.src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 80271b1 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 994cb76 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 
efcf74d 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java
 PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
ebffad6 
bq.src/main/protobuf/RegionServerStatus.proto PRE-CREATION 
bq.src/main/protobuf/hbase.proto 12e6053 
bq.src/main/resources/hbase-webapps/master/table.jsp 3ef1190 
bq.src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 72554cb 
bq.src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java 
d039be3 
bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 
36046f8 
bq.src/test/java/org/apache/hadoop/hbase/master/TestMXBean.java bd5fa90 
bq.src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java 
f8029ba 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java
 e99d251 
bq.  
bq.  Diff: https://reviews.apache.org/r/4463/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Ran jenkins job, all unit tests passed.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Gregory
bq.  

[jira] [Commented] (HBASE-5444) Add PB-based calls to HMasterRegionInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13267053#comment-13267053
 ] 

jirapos...@reviews.apache.org commented on HBASE-5444:
--



bq.  On 2012-05-02 23:43:07, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/master/HMaster.java, line 79
bq.   https://reviews.apache.org/r/4463/diff/5/?file=106063#file106063line79
bq.  
bq.   We can move the ipc protocol stuff to top level later... I was 
thinking that these classes shared by master and regionservers could be at 
o.a.h.h... but can do that later if it makes sense.  Lets get this pb stuff in 
first.
bq.   
bq.  

Sounds good.


bq.  On 2012-05-02 23:43:07, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/master/HMaster.java, line 80
bq.   https://reviews.apache.org/r/4463/diff/5/?file=106063#file106063line80
bq.  
bq.   We need this?

I use it to convert the PB serverName that is passed into 
HMaster.regionServerReport into a ServerName that the ServerManager 
understands.  Instead, we could have a ServerName static function that takes a 
PB ServerName and returns a ServerName.  We already have a bunch of these 
parse* functions already, e.g.

public static ServerName parseVersionedServerName(final byte [] versionedBytes)


bq.  On 2012-05-02 23:43:07, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/ClusterStatus.java, line 36
bq.   https://reviews.apache.org/r/4463/diff/5/?file=106055#file106055line36
bq.  
bq.   We need this import?  Its for cp.  Thats ok I'd say One day we 
can hide that too..

I think we can get rid of this once I pb-ify the HMasterInterface.  I'll put it 
on my list to check out.  There's probably a few such cases.


- Gregory


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4463/#review7492
---


On 2012-05-02 23:19:20, Gregory Chanan wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4463/
bq.  ---
bq.  
bq.  (Updated 2012-05-02 23:19:20)
bq.  
bq.  
bq.  Review request for hbase and Michael Stack.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Adds PB-based calls replacing HMasterRegionInterface.
bq.  
bq.  There are some temporary hacks, e.g. converting PB-based ServerLoad to 
existing HServerLoad so I didn't need to convert ClusterStatus (which brings in 
a lot of other changes).  That will be cleaned up in HBASE-5445.
bq.  
bq.  
bq.  This addresses bug HBASE-5444.
bq.  https://issues.apache.org/jira/browse/HBASE-5444
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon 
69434f7 
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
3c7c091 
bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 5d7f07b 
bq.src/main/java/org/apache/hadoop/hbase/ServerLoad.java PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 0db2760 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 973c7cb 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 
fd97830 
bq.src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java bb6ab3b 
bq.
src/main/java/org/apache/hadoop/hbase/ipc/RegionServerStatusProtocol.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 
f56127d 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 81e9023 
bq.src/main/java/org/apache/hadoop/hbase/master/MXBean.java 7f44dc2 
bq.src/main/java/org/apache/hadoop/hbase/master/MXBeanImpl.java 45b8fe7 
bq.src/main/java/org/apache/hadoop/hbase/master/MasterDumpServlet.java 
be63838 
bq.src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 80271b1 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 994cb76 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 
efcf74d 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java
 PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
ebffad6 
bq.src/main/protobuf/RegionServerStatus.proto PRE-CREATION 
bq.src/main/protobuf/hbase.proto 12e6053 
bq.src/main/resources/hbase-webapps/master/table.jsp 3ef1190 
bq.src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 72554cb 
bq.src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java 
d039be3 
bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 
36046f8 
bq.

[jira] [Commented] (HBASE-5444) Add PB-based calls to HMasterRegionInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13267155#comment-13267155
 ] 

jirapos...@reviews.apache.org commented on HBASE-5444:
--



bq.  On 2012-05-02 23:43:07, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/master/HMaster.java, line 80
bq.   https://reviews.apache.org/r/4463/diff/5/?file=106063#file106063line80
bq.  
bq.   We need this?
bq.  
bq.  Gregory Chanan wrote:
bq.  I use it to convert the PB serverName that is passed into 
HMaster.regionServerReport into a ServerName that the ServerManager 
understands.  Instead, we could have a ServerName static function that takes a 
PB ServerName and returns a ServerName.  We already have a bunch of these 
parse* functions already, e.g.
bq.  
bq.  public static ServerName parseVersionedServerName(final byte [] 
versionedBytes)

Recently in trunk, we added a ServerName.parseFrom that should be able to make 
sense of any set of bytes parsed it whether pbs or old style versioned bytes.  
Could use this.


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4463/#review7492
---


On 2012-05-02 23:19:20, Gregory Chanan wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4463/
bq.  ---
bq.  
bq.  (Updated 2012-05-02 23:19:20)
bq.  
bq.  
bq.  Review request for hbase and Michael Stack.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Adds PB-based calls replacing HMasterRegionInterface.
bq.  
bq.  There are some temporary hacks, e.g. converting PB-based ServerLoad to 
existing HServerLoad so I didn't need to convert ClusterStatus (which brings in 
a lot of other changes).  That will be cleaned up in HBASE-5445.
bq.  
bq.  
bq.  This addresses bug HBASE-5444.
bq.  https://issues.apache.org/jira/browse/HBASE-5444
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon 
69434f7 
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
3c7c091 
bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 5d7f07b 
bq.src/main/java/org/apache/hadoop/hbase/ServerLoad.java PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 0db2760 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 973c7cb 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 
fd97830 
bq.src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java bb6ab3b 
bq.
src/main/java/org/apache/hadoop/hbase/ipc/RegionServerStatusProtocol.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 
f56127d 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 81e9023 
bq.src/main/java/org/apache/hadoop/hbase/master/MXBean.java 7f44dc2 
bq.src/main/java/org/apache/hadoop/hbase/master/MXBeanImpl.java 45b8fe7 
bq.src/main/java/org/apache/hadoop/hbase/master/MasterDumpServlet.java 
be63838 
bq.src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 80271b1 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 994cb76 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 
efcf74d 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java
 PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
ebffad6 
bq.src/main/protobuf/RegionServerStatus.proto PRE-CREATION 
bq.src/main/protobuf/hbase.proto 12e6053 
bq.src/main/resources/hbase-webapps/master/table.jsp 3ef1190 
bq.src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 72554cb 
bq.src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java 
d039be3 
bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 
36046f8 
bq.src/test/java/org/apache/hadoop/hbase/master/TestMXBean.java bd5fa90 
bq.src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java 
f8029ba 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java
 e99d251 
bq.  
bq.  Diff: https://reviews.apache.org/r/4463/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Ran jenkins job, all unit tests passed.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Gregory
bq.  
bq.



 Add PB-based calls to HMasterRegionInterface
 

 Key: HBASE-5444
 URL: https://issues.apache.org/jira/browse/HBASE-5444
 Project: HBase
  

[jira] [Commented] (HBASE-2214) Do HBASE-1996 -- setting size to return in scan rather than count of rows -- properly

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13265702#comment-13265702
 ] 

jirapos...@reviews.apache.org commented on HBASE-2214:
--



bq.  On 2012-04-30 20:03:34, Michael Stack wrote:
bq.   Where are we checking the size of the result made so far?  I don't see 
it in the below.  I'd expect it inside in the RegionScanner.  Any chance of a 
test?  Otherwise, patch looks great.
bq.  
bq.  ferdy wrote:
bq.  Please see the testing method in the Testing Done field above. (Not 
sure where to add a test in the project).
bq.  
bq.  Thanks for the feedback.

(One more thing, it works because of the previous work done in HBASE-1996. But 
it's arguably not the best way to implement it.)


- ferdy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4726/#review7383
---


On 2012-05-01 07:50:07, ferdy wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4726/
bq.  ---
bq.  
bq.  (Updated 2012-05-01 07:50:07)
bq.  
bq.  
bq.  Review request for hbase and Ted Yu.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  HBASE-2214 per scan max buffersize.
bq.  
bq.  
bq.  This addresses bug HBASE-2214.
bq.  https://issues.apache.org/jira/browse/HBASE-2214
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq./src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java 1330680 
bq./src/main/java/org/apache/hadoop/hbase/client/Scan.java 1330680 
bq./src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 
1330680 
bq./src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java 
1330680 
bq.
/src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java 
1330680 
bq.
/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java 
1330680 
bq.
/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 
1330680 
bq./src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java 
1330680 
bq.
/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java 
1330680 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1332544 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
1332544 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/RegionScanner.java 
1332544 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java 
1332544 
bq./src/main/protobuf/Client.proto 1330680 
bq.
/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
 1332544 
bq.  
bq.  Diff: https://reviews.apache.org/r/4726/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  It works when running this test:
bq.  
bq.  
bq.  new HBaseTestingUtility(conf).startMiniCluster();
bq.   
bq.  HBaseAdmin admin = new HBaseAdmin(conf);
bq.  if (!admin.tableExists(test)) {
bq.HTableDescriptor tableDesc = new HTableDescriptor(test);
bq.tableDesc.addFamily(new HColumnDescriptor(fam));
bq.admin.createTable(tableDesc);
bq.  }
bq.  
bq.  
bq.  HTable table = new HTable(conf, test);
bq.  Put put; 
bq.  
bq.  put = new Put(Bytes.toBytes(row1));
bq.  
put.add(Bytes.toBytes(fam),Bytes.toBytes(qual1),Bytes.toBytes(val1));
bq.  table.put(put);
bq.  
bq.  put = new Put(Bytes.toBytes(row2));
bq.  
put.add(Bytes.toBytes(fam),Bytes.toBytes(qual2),Bytes.toBytes(val2));
bq.  table.put(put);
bq.  
bq.  put = new Put(Bytes.toBytes(row3));
bq.  
put.add(Bytes.toBytes(fam),Bytes.toBytes(qual3),Bytes.toBytes(val3));
bq.  table.put(put);
bq.  
bq.  table.flushCommits();
bq.  //put a logging statement to ClientScanner#next() to see the effect.
bq.  {
bq.System.out.println(returns all rows at once because of the 
caching);
bq.Scan scan = new Scan();
bq.scan.setCaching(100);
bq.ResultScanner scanner = table.getScanner(scan);
bq.scanner.next(100);
bq.  }
bq.  {
bq.System.out.println(returns one row at a time because of the 
maxResultSize);
bq.Scan scan = new Scan();
bq.scan.setCaching(100);
bq.scan.setMaxResultSize(1);
bq.ResultScanner scanner = table.getScanner(scan);
bq.scanner.next(100);
bq.  }
bq.  
bq.  
bq.  See output:
bq.  
bq.  returns all rows at once because of the caching
bq.  2012-04-25 22:18:47,494 DEBUG [main] client.ClientScanner(94): Creating 
scanner over test starting at key ''
bq.  2012-04-25 22:18:47,494 DEBUG [main] client.ClientScanner(206): Advancing 

[jira] [Commented] (HBASE-5444) Add PB-based calls to HMasterRegionInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266009#comment-13266009
 ] 

jirapos...@reviews.apache.org commented on HBASE-5444:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4463/
---

(Updated 2012-05-01 19:53:51.307399)


Review request for hbase and Michael Stack.


Changes
---

Update against newest trunk and followed Ted's suggestion about breaking out 
totalRequestsCount computation into own function.


Summary
---

Adds PB-based calls replacing HMasterRegionInterface.

There are some temporary hacks, e.g. converting PB-based ServerLoad to existing 
HServerLoad so I didn't need to convert ClusterStatus (which brings in a lot of 
other changes).  That will be cleaned up in HBASE-5445.


This addresses bug HBASE-5444.
https://issues.apache.org/jira/browse/HBASE-5444


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 
  src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 0db2760 
  src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 973c7cb 
  src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java fd97830 
  src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java bb6ab3b 
  src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java f56127d 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java 81e9023 
  src/main/java/org/apache/hadoop/hbase/master/MXBean.java 7f44dc2 
  src/main/java/org/apache/hadoop/hbase/master/MXBeanImpl.java 45b8fe7 
  src/main/java/org/apache/hadoop/hbase/master/MasterDumpServlet.java be63838 
  src/main/java/org/apache/hadoop/hbase/master/RegionServerStatusProtocol.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 80271b1 
  src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 994cb76 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 
efcf74d 
  src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 5d7f07b 
  src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon 
69434f7 
  src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
3c7c091 
  
src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java
 PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java ebffad6 
  src/main/protobuf/RegionServerStatus.proto PRE-CREATION 
  src/main/protobuf/hbase.proto 12e6053 
  src/main/resources/hbase-webapps/master/table.jsp 3ef1190 
  src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 72554cb 
  src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java 
d039be3 
  src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 
36046f8 
  src/test/java/org/apache/hadoop/hbase/master/TestMXBean.java bd5fa90 
  src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java f8029ba 
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java
 e99d251 

Diff: https://reviews.apache.org/r/4463/diff


Testing
---

Ran jenkins job, all unit tests passed.


Thanks,

Gregory



 Add PB-based calls to HMasterRegionInterface
 

 Key: HBASE-5444
 URL: https://issues.apache.org/jira/browse/HBASE-5444
 Project: HBase
  Issue Type: Sub-task
  Components: ipc, master, migration, regionserver
Reporter: Todd Lipcon
Assignee: Gregory Chanan
 Attachments: HBASE-5444-v6-trunk.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5444) Add PB-based calls to HMasterRegionInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266028#comment-13266028
 ] 

jirapos...@reviews.apache.org commented on HBASE-5444:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4463/#review7441
---



src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
https://reviews.apache.org/r/4463/#comment16369

Insert a space between if and (.



src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
https://reviews.apache.org/r/4463/#comment16372

You can return list.toArray() directly, similar to line 1179.


- Ted


On 2012-05-01 19:53:51, Gregory Chanan wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4463/
bq.  ---
bq.  
bq.  (Updated 2012-05-01 19:53:51)
bq.  
bq.  
bq.  Review request for hbase and Michael Stack.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Adds PB-based calls replacing HMasterRegionInterface.
bq.  
bq.  There are some temporary hacks, e.g. converting PB-based ServerLoad to 
existing HServerLoad so I didn't need to convert ClusterStatus (which brings in 
a lot of other changes).  That will be cleaned up in HBASE-5445.
bq.  
bq.  
bq.  This addresses bug HBASE-5444.
bq.  https://issues.apache.org/jira/browse/HBASE-5444
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 0db2760 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 973c7cb 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 
fd97830 
bq.src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java bb6ab3b 
bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 
f56127d 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 81e9023 
bq.src/main/java/org/apache/hadoop/hbase/master/MXBean.java 7f44dc2 
bq.src/main/java/org/apache/hadoop/hbase/master/MXBeanImpl.java 45b8fe7 
bq.src/main/java/org/apache/hadoop/hbase/master/MasterDumpServlet.java 
be63838 
bq.
src/main/java/org/apache/hadoop/hbase/master/RegionServerStatusProtocol.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 80271b1 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 994cb76 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 
efcf74d 
bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 5d7f07b 
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon 
69434f7 
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
3c7c091 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java
 PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
ebffad6 
bq.src/main/protobuf/RegionServerStatus.proto PRE-CREATION 
bq.src/main/protobuf/hbase.proto 12e6053 
bq.src/main/resources/hbase-webapps/master/table.jsp 3ef1190 
bq.src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 72554cb 
bq.src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java 
d039be3 
bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 
36046f8 
bq.src/test/java/org/apache/hadoop/hbase/master/TestMXBean.java bd5fa90 
bq.src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java 
f8029ba 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java
 e99d251 
bq.  
bq.  Diff: https://reviews.apache.org/r/4463/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Ran jenkins job, all unit tests passed.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Gregory
bq.  
bq.



 Add PB-based calls to HMasterRegionInterface
 

 Key: HBASE-5444
 URL: https://issues.apache.org/jira/browse/HBASE-5444
 Project: HBase
  Issue Type: Sub-task
  Components: ipc, master, migration, regionserver
Reporter: Todd Lipcon
Assignee: Gregory Chanan
 Attachments: HBASE-5444-v6-trunk.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5444) Add PB-based calls to HMasterRegionInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266033#comment-13266033
 ] 

jirapos...@reviews.apache.org commented on HBASE-5444:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4463/#review7442
---


A few comments below the thrust of which are about encapsulating pb if possible 
rather than have it spread around in classes.See what you think.  It would 
not be hard to get me commit this as is.


src/main/java/org/apache/hadoop/hbase/HConstants.java
https://reviews.apache.org/r/4463/#comment16370

Were you going to move this down to where its used G?   Does it need to be 
up here?



src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java
https://reviews.apache.org/r/4463/#comment16371

Hurray!



src/main/java/org/apache/hadoop/hbase/master/MXBean.java
https://reviews.apache.org/r/4463/#comment16373

All of these classes are importing generated pb classes.  Would it be 
better to have a high-level ServerLoad class that hid inside it the pb stuff 
instead?  Less pb generated class pollution.



src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
https://reviews.apache.org/r/4463/#comment16374

Yeah, its kinda ugly having this package reach over into pb package.



src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
https://reviews.apache.org/r/4463/#comment16375

We should not be reaching over into the master package.  Put this protocol 
class at the top level since shared by master and regionserver?


- Michael


On 2012-05-01 19:53:51, Gregory Chanan wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4463/
bq.  ---
bq.  
bq.  (Updated 2012-05-01 19:53:51)
bq.  
bq.  
bq.  Review request for hbase and Michael Stack.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Adds PB-based calls replacing HMasterRegionInterface.
bq.  
bq.  There are some temporary hacks, e.g. converting PB-based ServerLoad to 
existing HServerLoad so I didn't need to convert ClusterStatus (which brings in 
a lot of other changes).  That will be cleaned up in HBASE-5445.
bq.  
bq.  
bq.  This addresses bug HBASE-5444.
bq.  https://issues.apache.org/jira/browse/HBASE-5444
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 0db2760 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 973c7cb 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 
fd97830 
bq.src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java bb6ab3b 
bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 
f56127d 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 81e9023 
bq.src/main/java/org/apache/hadoop/hbase/master/MXBean.java 7f44dc2 
bq.src/main/java/org/apache/hadoop/hbase/master/MXBeanImpl.java 45b8fe7 
bq.src/main/java/org/apache/hadoop/hbase/master/MasterDumpServlet.java 
be63838 
bq.
src/main/java/org/apache/hadoop/hbase/master/RegionServerStatusProtocol.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 80271b1 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 994cb76 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 
efcf74d 
bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 5d7f07b 
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon 
69434f7 
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
3c7c091 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java
 PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
ebffad6 
bq.src/main/protobuf/RegionServerStatus.proto PRE-CREATION 
bq.src/main/protobuf/hbase.proto 12e6053 
bq.src/main/resources/hbase-webapps/master/table.jsp 3ef1190 
bq.src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 72554cb 
bq.src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java 
d039be3 
bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 
36046f8 
bq.src/test/java/org/apache/hadoop/hbase/master/TestMXBean.java bd5fa90 
bq.src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java 
f8029ba 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java
 e99d251 
bq.  
bq.  Diff: 

[jira] [Commented] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266042#comment-13266042
 ] 

jirapos...@reviews.apache.org commented on HBASE-5732:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4953/
---

Review request for Ted Yu, Michael Stack and Andrew Purtell.


Summary
---

Reviewboard request for HBASE-5732


This addresses bug HBASE-5732.
https://issues.apache.org/jira/browse/HBASE-5732


Diffs
-

  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java
 1332383 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java
 1332383 
  http://svn.apache.org/repos/asf/hbase/trunk/pom.xml 1332383 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
 1332383 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/Status.java
 1332383 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
 1332383 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java
 1332383 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java
 1332383 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlFilter.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControllerProtocol.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/Permission.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/TablePermission.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/UserPermission.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/ZKPermissionWatcher.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java
 PRE-CREATION 
  http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPC.proto 
1332383 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/security/access/SecureTestUtil.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessControlFilter.java
 PRE-CREATION 
  

[jira] [Commented] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266068#comment-13266068
 ] 

jirapos...@reviews.apache.org commented on HBASE-5869:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4926/
---

(Updated 2012-05-01 20:42:36.337375)


Review request for hbase and Jimmy Xiang.


Changes
---

Same as original w/ a few fixes for tests that failed:

1. In distributed log tests, was failing to pick up the recovered.edits file 
because string passed included state of the split log task when what was wanted 
was servername only


Summary
---

Convert two zk users to pb: distributed log splitting and regions in transition.

Refactored distributed log splitting so we only serialize/deserialize in one 
location.
Less changes needed to do same for regions in transition.

Moves serialization/deserialization out of the ZKAssign, ZKSplit and into
the classes themselves so can encapsulate how serialization is done into one 
place
(try to make the ZK* classes just deal in bytes -- about 90% done).

Moved classes used by various packages up to top level to minimize imports
that are across package (zookeeper into protobuf and/or into regionserver and/or
master packages, etc).

A src/main/java/org/apache/hadoop/hbase/DeserializationException.java
  New generic deserialization exception.
A src/main/java/org/apache/hadoop/hbase/zookeeper/EmptyWatcher.java
D  src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java
  Moved under zookeeper package.
A src/main/java/org/apache/hadoop/hbase/HBaseException.java
  New base hbase exception as suggested by hbase-5796.  New 
DeserializationException
  inherits from this.
A src/main/java/org/apache/hadoop/hbase/RegionTransition.java
  State of a region in transition.  Top-level because used by a
  few top-level packages.  Encapsulates pb serialization/deserialization.
M src/main/java/org/apache/hadoop/hbase/ServerName.java
  Add method to deserialize a ServeName, etc.  Encapsulates pb'ing.
M src/main/java/org/apache/hadoop/hbase/SplitLogCounters.java
  Counters used by distributed log splitting.
A SplitLogTask
   Class that encapsulates log splitting state.  Also encapsulates pb'ing.
M src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java
  Implement code for state.  Added functions to go from code to state and vice
  versa.  Used serializing.
M src/main/java/org/apache/hadoop/hbase/executor/ExecutorService.java
  Remove unused imports.
D src/main/java/org/apache/hadoop/hbase/executor/RegionTransitionData.java
  Removed.  Replaced by RegionTransition moved to package top-level.
M src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java
M src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
  Use new DeserializationException. Move to using new RegionTransition
  from RegionTransitionData class.  Pass deserialized class rather than
  byte array.  Remove duplicated code.
M src/main/java/org/apache/hadoop/hbase/master/HMaster.java
  Use new ServerName parse method rather than ZKUtil one.
M src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
M src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java
  Redo to use new SplitLogTask and SplitLogCounter classes.
M src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
  expectPBMagicPrefix added
M src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
  Use new RegionTransition in place of RegionTransitionData.
M src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
  Define moved from ZKSplitLog to SplitLogManager.
M src/main/java/org/apache/hadoop/hbase/zookeeper/MasterAddressTracker.java
M src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java
  Changed method name from getZNodeData to toByteArray to match how we've
  named it elsewhere. Use new DeserializationException
M src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java
  Use new RegionTransion class
M src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java
  Moved stuff that was in here up into SplitLogManager where better
  belongs.  Also moved serialization/deserialization up into the
  class itself: SplitLogTask.  Moved counters out to SplitLogCounter class.
M src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
  Moved deserialization of ServerName out of here and up into ServerName.
M src/main/protobuf/ZooKeeper.proto
  Add two new classes, RegionTransition and SplitLogTask.


This addresses bug HBASE-5869.
https://issues.apache.org/jira/browse/HBASE-5869


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/DeserializationException.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java 

[jira] [Commented] (HBASE-5444) Add PB-based calls to HMasterRegionInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266168#comment-13266168
 ] 

jirapos...@reviews.apache.org commented on HBASE-5444:
--



bq.  On 2012-05-01 20:16:06, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java, line 
1128
bq.   
https://reviews.apache.org/r/4463/diff/4/?file=105814#file105814line1128
bq.  
bq.   Insert a space between if and (.

Will do.


bq.  On 2012-05-01 20:16:06, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java, line 
1145
bq.   
https://reviews.apache.org/r/4463/diff/4/?file=105814#file105814line1145
bq.  
bq.   You can return list.toArray() directly, similar to line 1179.

I still need to convert from Coprocessor to String (via getName) if I call 
toArray().  Am I missing something?


- Gregory


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4463/#review7441
---


On 2012-05-01 19:53:51, Gregory Chanan wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4463/
bq.  ---
bq.  
bq.  (Updated 2012-05-01 19:53:51)
bq.  
bq.  
bq.  Review request for hbase and Michael Stack.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Adds PB-based calls replacing HMasterRegionInterface.
bq.  
bq.  There are some temporary hacks, e.g. converting PB-based ServerLoad to 
existing HServerLoad so I didn't need to convert ClusterStatus (which brings in 
a lot of other changes).  That will be cleaned up in HBASE-5445.
bq.  
bq.  
bq.  This addresses bug HBASE-5444.
bq.  https://issues.apache.org/jira/browse/HBASE-5444
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 0db2760 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 973c7cb 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 
fd97830 
bq.src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java bb6ab3b 
bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 
f56127d 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 81e9023 
bq.src/main/java/org/apache/hadoop/hbase/master/MXBean.java 7f44dc2 
bq.src/main/java/org/apache/hadoop/hbase/master/MXBeanImpl.java 45b8fe7 
bq.src/main/java/org/apache/hadoop/hbase/master/MasterDumpServlet.java 
be63838 
bq.
src/main/java/org/apache/hadoop/hbase/master/RegionServerStatusProtocol.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 80271b1 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 994cb76 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 
efcf74d 
bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 5d7f07b 
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon 
69434f7 
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
3c7c091 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java
 PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
ebffad6 
bq.src/main/protobuf/RegionServerStatus.proto PRE-CREATION 
bq.src/main/protobuf/hbase.proto 12e6053 
bq.src/main/resources/hbase-webapps/master/table.jsp 3ef1190 
bq.src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 72554cb 
bq.src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java 
d039be3 
bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 
36046f8 
bq.src/test/java/org/apache/hadoop/hbase/master/TestMXBean.java bd5fa90 
bq.src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java 
f8029ba 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java
 e99d251 
bq.  
bq.  Diff: https://reviews.apache.org/r/4463/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Ran jenkins job, all unit tests passed.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Gregory
bq.  
bq.



 Add PB-based calls to HMasterRegionInterface
 

 Key: HBASE-5444
 URL: https://issues.apache.org/jira/browse/HBASE-5444
 Project: HBase
  Issue Type: Sub-task
  Components: ipc, master, migration, regionserver
Reporter: Todd Lipcon
Assignee: Gregory Chanan
 Attachments: HBASE-5444-v6-trunk.patch

[jira] [Commented] (HBASE-5444) Add PB-based calls to HMasterRegionInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266167#comment-13266167
 ] 

jirapos...@reviews.apache.org commented on HBASE-5444:
--



bq.  On 2012-05-01 20:20:09, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/HConstants.java, line 679
bq.   https://reviews.apache.org/r/4463/diff/4/?file=105802#file105802line679
bq.  
bq.   Were you going to move this down to where its used G?   Does it need 
to be up here?

I can just get rid of this and use null everywhere.  That's what it looks like 
Jimmy did.


bq.  On 2012-05-01 20:20:09, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java, 
line 173
bq.   https://reviews.apache.org/r/4463/diff/4/?file=105817#file105817line173
bq.  
bq.   We should not be reaching over into the master package.  Put this 
protocol class at the top level since shared by master and regionserver?

Will do.


bq.  On 2012-05-01 20:20:09, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/master/MXBean.java, line 23
bq.   https://reviews.apache.org/r/4463/diff/4/?file=105809#file105809line23
bq.  
bq.   All of these classes are importing generated pb classes.  Would it 
be better to have a high-level ServerLoad class that hid inside it the pb stuff 
instead?  Less pb generated class pollution.

I think that's a good idea.  I was considering doing that, but wasn't sure if I 
should do it for all the generated pb types.  Since ServerLoad is sprinkled 
everywhere, it seems like that is at least a good place to start.


- Gregory


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4463/#review7442
---


On 2012-05-01 19:53:51, Gregory Chanan wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4463/
bq.  ---
bq.  
bq.  (Updated 2012-05-01 19:53:51)
bq.  
bq.  
bq.  Review request for hbase and Michael Stack.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Adds PB-based calls replacing HMasterRegionInterface.
bq.  
bq.  There are some temporary hacks, e.g. converting PB-based ServerLoad to 
existing HServerLoad so I didn't need to convert ClusterStatus (which brings in 
a lot of other changes).  That will be cleaned up in HBASE-5445.
bq.  
bq.  
bq.  This addresses bug HBASE-5444.
bq.  https://issues.apache.org/jira/browse/HBASE-5444
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 0db2760 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 973c7cb 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 
fd97830 
bq.src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java bb6ab3b 
bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 
f56127d 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 81e9023 
bq.src/main/java/org/apache/hadoop/hbase/master/MXBean.java 7f44dc2 
bq.src/main/java/org/apache/hadoop/hbase/master/MXBeanImpl.java 45b8fe7 
bq.src/main/java/org/apache/hadoop/hbase/master/MasterDumpServlet.java 
be63838 
bq.
src/main/java/org/apache/hadoop/hbase/master/RegionServerStatusProtocol.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 80271b1 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 994cb76 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 
efcf74d 
bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 5d7f07b 
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon 
69434f7 
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
3c7c091 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java
 PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
ebffad6 
bq.src/main/protobuf/RegionServerStatus.proto PRE-CREATION 
bq.src/main/protobuf/hbase.proto 12e6053 
bq.src/main/resources/hbase-webapps/master/table.jsp 3ef1190 
bq.src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 72554cb 
bq.src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java 
d039be3 
bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 
36046f8 
bq.src/test/java/org/apache/hadoop/hbase/master/TestMXBean.java bd5fa90 
bq.src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java 
f8029ba 
bq.

[jira] [Commented] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266179#comment-13266179
 ] 

jirapos...@reviews.apache.org commented on HBASE-5869:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4926/#review7446
---



src/main/java/org/apache/hadoop/hbase/SplitLogCounters.java
https://reviews.apache.org/r/4926/#comment16378

Usually this line is not the first in a file.



src/main/java/org/apache/hadoop/hbase/SplitLogCounters.java
https://reviews.apache.org/r/4926/#comment16379

'log' seems redundant here.



src/main/java/org/apache/hadoop/hbase/SplitLogCounters.java
https://reviews.apache.org/r/4926/#comment16377

What if an AtomicInteger counter is added in the future ?



src/main/java/org/apache/hadoop/hbase/SplitLogTask.java
https://reviews.apache.org/r/4926/#comment16381

'date' - 'data'



src/main/java/org/apache/hadoop/hbase/SplitLogTask.java
https://reviews.apache.org/r/4926/#comment16382

'An' - 'A'



src/main/java/org/apache/hadoop/hbase/SplitLogTask.java
https://reviews.apache.org/r/4926/#comment16383

What would the first 64 bytes of data represent ?
Do we know that data.length = 64 ?



src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
https://reviews.apache.org/r/4926/#comment16408

Should e1 be included in the log ?
're-' before 'resubmit' is not necessary.


- Ted


On 2012-05-01 20:42:36, Michael Stack wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4926/
bq.  ---
bq.  
bq.  (Updated 2012-05-01 20:42:36)
bq.  
bq.  
bq.  Review request for hbase and Jimmy Xiang.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Convert two zk users to pb: distributed log splitting and regions in 
transition.
bq.  
bq.  Refactored distributed log splitting so we only serialize/deserialize in 
one location.
bq.  Less changes needed to do same for regions in transition.
bq.  
bq.  Moves serialization/deserialization out of the ZKAssign, ZKSplit and into
bq.  the classes themselves so can encapsulate how serialization is done into 
one place
bq.  (try to make the ZK* classes just deal in bytes -- about 90% done).
bq.  
bq.  Moved classes used by various packages up to top level to minimize imports
bq.  that are across package (zookeeper into protobuf and/or into regionserver 
and/or
bq.  master packages, etc).
bq.  
bq.  A src/main/java/org/apache/hadoop/hbase/DeserializationException.java
bq.New generic deserialization exception.
bq.  A src/main/java/org/apache/hadoop/hbase/zookeeper/EmptyWatcher.java
bq.  D  src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java
bq.Moved under zookeeper package.
bq.  A src/main/java/org/apache/hadoop/hbase/HBaseException.java
bq.New base hbase exception as suggested by hbase-5796.  New 
DeserializationException
bq.inherits from this.
bq.  A src/main/java/org/apache/hadoop/hbase/RegionTransition.java
bq.State of a region in transition.  Top-level because used by a
bq.few top-level packages.  Encapsulates pb serialization/deserialization.
bq.  M src/main/java/org/apache/hadoop/hbase/ServerName.java
bq.Add method to deserialize a ServeName, etc.  Encapsulates pb'ing.
bq.  M src/main/java/org/apache/hadoop/hbase/SplitLogCounters.java
bq.Counters used by distributed log splitting.
bq.  A SplitLogTask
bq. Class that encapsulates log splitting state.  Also encapsulates pb'ing.
bq.  M src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java
bq.Implement code for state.  Added functions to go from code to state and 
vice
bq.versa.  Used serializing.
bq.  M src/main/java/org/apache/hadoop/hbase/executor/ExecutorService.java
bq.Remove unused imports.
bq.  D src/main/java/org/apache/hadoop/hbase/executor/RegionTransitionData.java
bq.Removed.  Replaced by RegionTransition moved to package top-level.
bq.  M src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java
bq.  M src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
bq.Use new DeserializationException. Move to using new RegionTransition
bq.from RegionTransitionData class.  Pass deserialized class rather than
bq.byte array.  Remove duplicated code.
bq.  M src/main/java/org/apache/hadoop/hbase/master/HMaster.java
bq.Use new ServerName parse method rather than ZKUtil one.
bq.  M src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
bq.  M src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java
bq.Redo to use new SplitLogTask and SplitLogCounter classes.
bq.  M 

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266190#comment-13266190
 ] 

jirapos...@reviews.apache.org commented on HBASE-5547:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4633/#review7448
---


Great work.  High-level, do we have to have backup/archiving touch all parts of 
the code base?  Can we not hide it behind more general Interfaces at least when 
we get down into Store and Region classes


src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java
https://reviews.apache.org/r/4633/#comment16389

Tables or hfiles?



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java
https://reviews.apache.org/r/4633/#comment16384

Should this interface be at top level?  Its used out of the master and 
regionserver packages

The implementations could be stay under backup long as we only use the 
Interface?



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java
https://reviews.apache.org/r/4633/#comment16390

hfiles or tables?

Looks like this class would select hfiles by selected tables



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java
https://reviews.apache.org/r/4633/#comment16387

Tracker in hbase usually means tracking znodes up in zookeeper.  This 
tracks something else.  Could confuse.  Use different action?  Manager



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java
https://reviews.apache.org/r/4633/#comment16385

Good.  I like that you are taking a 'Server' here rather than a Master or 
Regionserver or even a ServerName.

Does this have to be public?  Seems like its utility used by the later 
HFileArchiveTracker.



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java
https://reviews.apache.org/r/4633/#comment16386

Do we have to have a set?  Does this Set change during live of this tracker?

Should javadoc say that previous members of Set are cleared out?



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java
https://reviews.apache.org/r/4633/#comment16388

good



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment16392

How does this tracker relate to the containing class?  They are different 
types of trackers?  Could confuse?



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment16391

Why pass it in?  Why not just create one and keep it running internally? 

The class HFileArchiveTableTracker is a little confusing 



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment16395

Want to mention in class comment that user needs to call start, etc., to 
start up the tracking?



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment16399

I think it kinda bad that zkw does this... you'd think each tracker would 
look after its own stuff (this is not you.. you are folllowing the established 
pattern as you should -- I'm just moaning)



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment16400

Is this what ZKUtil.getNodeName does?



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment16401

nit style.  if you did if (!path.startsWith...) return; ... you could save 
an indent of the whole method.  Next time.



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment16403

These znodes have nothing in them... just the tablename of the znode is 
used?  Nothing to pb then?



src/main/java/org/apache/hadoop/hbase/backup/ServerHFileTableArchiveTracker.java
https://reviews.apache.org/r/4633/#comment16405

This is under each table znode?  Why we inherit from 
HFileArchiveTableTracker?



src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
https://reviews.apache.org/r/4633/#comment16407

Is this doing backup? Or making it so the hfiles are not deleted on this 
table?



src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
https://reviews.apache.org/r/4633/#comment16409

ditto... this just throws a switch on the don't delete operation?  
Something else does the backup?



src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java
https://reviews.apache.org/r/4633/#comment16410

table hfiles?



src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java

[jira] [Commented] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266195#comment-13266195
 ] 

jirapos...@reviews.apache.org commented on HBASE-5869:
--



bq.  On 2012-05-01 21:59:01, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/SplitLogCounters.java, line 99
bq.   https://reviews.apache.org/r/4926/diff/3/?file=105878#file105878line99
bq.  
bq.   What if an AtomicInteger counter is added in the future ?

Open new JIRA.  This is just a move of existing code.


bq.  On 2012-05-01 21:59:01, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/SplitLogTask.java, line 155
bq.   https://reviews.apache.org/r/4926/diff/3/?file=105879#file105879line155
bq.  
bq.   What would the first 64 bytes of data represent ?
bq.   Do we know that data.length = 64 ?

Let me fix toStringBinary so it deals w/ case where data is  64 bytes.

Regards what this represents, it could be anything.  Just saving our logs from 
being filled w/ binary.


bq.  On 2012-05-01 21:59:01, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java, line 
567
bq.   https://reviews.apache.org/r/4926/diff/3/?file=105887#file105887line567
bq.  
bq.   Should e1 be included in the log ?
bq.   're-' before 'resubmit' is not necessary.

Will add it.

re- makes sense because this is retry inside exception handling.


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4926/#review7446
---


On 2012-05-01 20:42:36, Michael Stack wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4926/
bq.  ---
bq.  
bq.  (Updated 2012-05-01 20:42:36)
bq.  
bq.  
bq.  Review request for hbase and Jimmy Xiang.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Convert two zk users to pb: distributed log splitting and regions in 
transition.
bq.  
bq.  Refactored distributed log splitting so we only serialize/deserialize in 
one location.
bq.  Less changes needed to do same for regions in transition.
bq.  
bq.  Moves serialization/deserialization out of the ZKAssign, ZKSplit and into
bq.  the classes themselves so can encapsulate how serialization is done into 
one place
bq.  (try to make the ZK* classes just deal in bytes -- about 90% done).
bq.  
bq.  Moved classes used by various packages up to top level to minimize imports
bq.  that are across package (zookeeper into protobuf and/or into regionserver 
and/or
bq.  master packages, etc).
bq.  
bq.  A src/main/java/org/apache/hadoop/hbase/DeserializationException.java
bq.New generic deserialization exception.
bq.  A src/main/java/org/apache/hadoop/hbase/zookeeper/EmptyWatcher.java
bq.  D  src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java
bq.Moved under zookeeper package.
bq.  A src/main/java/org/apache/hadoop/hbase/HBaseException.java
bq.New base hbase exception as suggested by hbase-5796.  New 
DeserializationException
bq.inherits from this.
bq.  A src/main/java/org/apache/hadoop/hbase/RegionTransition.java
bq.State of a region in transition.  Top-level because used by a
bq.few top-level packages.  Encapsulates pb serialization/deserialization.
bq.  M src/main/java/org/apache/hadoop/hbase/ServerName.java
bq.Add method to deserialize a ServeName, etc.  Encapsulates pb'ing.
bq.  M src/main/java/org/apache/hadoop/hbase/SplitLogCounters.java
bq.Counters used by distributed log splitting.
bq.  A SplitLogTask
bq. Class that encapsulates log splitting state.  Also encapsulates pb'ing.
bq.  M src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java
bq.Implement code for state.  Added functions to go from code to state and 
vice
bq.versa.  Used serializing.
bq.  M src/main/java/org/apache/hadoop/hbase/executor/ExecutorService.java
bq.Remove unused imports.
bq.  D src/main/java/org/apache/hadoop/hbase/executor/RegionTransitionData.java
bq.Removed.  Replaced by RegionTransition moved to package top-level.
bq.  M src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java
bq.  M src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
bq.Use new DeserializationException. Move to using new RegionTransition
bq.from RegionTransitionData class.  Pass deserialized class rather than
bq.byte array.  Remove duplicated code.
bq.  M src/main/java/org/apache/hadoop/hbase/master/HMaster.java
bq.Use new ServerName parse method rather than ZKUtil one.
bq.  M src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
bq.  M src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java
bq.Redo to 

[jira] [Commented] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266198#comment-13266198
 ] 

jirapos...@reviews.apache.org commented on HBASE-5869:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4926/#review7452
---



src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
https://reviews.apache.org/r/4926/#comment16434

The port numbers don't match.


- Ted


On 2012-05-01 20:42:36, Michael Stack wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4926/
bq.  ---
bq.  
bq.  (Updated 2012-05-01 20:42:36)
bq.  
bq.  
bq.  Review request for hbase and Jimmy Xiang.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Convert two zk users to pb: distributed log splitting and regions in 
transition.
bq.  
bq.  Refactored distributed log splitting so we only serialize/deserialize in 
one location.
bq.  Less changes needed to do same for regions in transition.
bq.  
bq.  Moves serialization/deserialization out of the ZKAssign, ZKSplit and into
bq.  the classes themselves so can encapsulate how serialization is done into 
one place
bq.  (try to make the ZK* classes just deal in bytes -- about 90% done).
bq.  
bq.  Moved classes used by various packages up to top level to minimize imports
bq.  that are across package (zookeeper into protobuf and/or into regionserver 
and/or
bq.  master packages, etc).
bq.  
bq.  A src/main/java/org/apache/hadoop/hbase/DeserializationException.java
bq.New generic deserialization exception.
bq.  A src/main/java/org/apache/hadoop/hbase/zookeeper/EmptyWatcher.java
bq.  D  src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java
bq.Moved under zookeeper package.
bq.  A src/main/java/org/apache/hadoop/hbase/HBaseException.java
bq.New base hbase exception as suggested by hbase-5796.  New 
DeserializationException
bq.inherits from this.
bq.  A src/main/java/org/apache/hadoop/hbase/RegionTransition.java
bq.State of a region in transition.  Top-level because used by a
bq.few top-level packages.  Encapsulates pb serialization/deserialization.
bq.  M src/main/java/org/apache/hadoop/hbase/ServerName.java
bq.Add method to deserialize a ServeName, etc.  Encapsulates pb'ing.
bq.  M src/main/java/org/apache/hadoop/hbase/SplitLogCounters.java
bq.Counters used by distributed log splitting.
bq.  A SplitLogTask
bq. Class that encapsulates log splitting state.  Also encapsulates pb'ing.
bq.  M src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java
bq.Implement code for state.  Added functions to go from code to state and 
vice
bq.versa.  Used serializing.
bq.  M src/main/java/org/apache/hadoop/hbase/executor/ExecutorService.java
bq.Remove unused imports.
bq.  D src/main/java/org/apache/hadoop/hbase/executor/RegionTransitionData.java
bq.Removed.  Replaced by RegionTransition moved to package top-level.
bq.  M src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java
bq.  M src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
bq.Use new DeserializationException. Move to using new RegionTransition
bq.from RegionTransitionData class.  Pass deserialized class rather than
bq.byte array.  Remove duplicated code.
bq.  M src/main/java/org/apache/hadoop/hbase/master/HMaster.java
bq.Use new ServerName parse method rather than ZKUtil one.
bq.  M src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
bq.  M src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java
bq.Redo to use new SplitLogTask and SplitLogCounter classes.
bq.  M src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
bq.expectPBMagicPrefix added
bq.  M src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
bq.Use new RegionTransition in place of RegionTransitionData.
bq.  M src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
bq.Define moved from ZKSplitLog to SplitLogManager.
bq.  M src/main/java/org/apache/hadoop/hbase/zookeeper/MasterAddressTracker.java
bq.  M src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java
bq.Changed method name from getZNodeData to toByteArray to match how we've
bq.named it elsewhere. Use new DeserializationException
bq.  M src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java
bq.Use new RegionTransion class
bq.  M src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java
bq.Moved stuff that was in here up into SplitLogManager where better
bq.belongs.  Also moved serialization/deserialization up into the

[jira] [Commented] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266222#comment-13266222
 ] 

jirapos...@reviews.apache.org commented on HBASE-5732:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4953/#review7455
---


Here's a quick review of this fat patch.  Good stuff.


http://svn.apache.org/repos/asf/hbase/trunk/pom.xml
https://reviews.apache.org/r/4953/#comment16446

Now the underlying hadoop must support all the security apis?

If not present, will we compile?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java
https://reviews.apache.org/r/4953/#comment16448

Should this security stuff be moved down here into ipc package?  Is it only 
place where security is referenced?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java
https://reviews.apache.org/r/4953/#comment16452

So, if underlying hadoop does not have these classes, we'll fail the build?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java
https://reviews.apache.org/r/4953/#comment16453

Should this be a fail?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
https://reviews.apache.org/r/4953/#comment16457

ditto comment from above



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
https://reviews.apache.org/r/4953/#comment16458

Can you not give byte array to pb to parse?  Use builder and mergeFrom?  
Not important. 



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
https://reviews.apache.org/r/4953/#comment16459

Has to be public?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
https://reviews.apache.org/r/4953/#comment16460

This is a pity removing the static?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
https://reviews.apache.org/r/4953/#comment16461

Removing this static is good though.



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
https://reviews.apache.org/r/4953/#comment16462

Where do these get shut down?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/UserPermission.java
https://reviews.apache.org/r/4953/#comment16463

This stuff is copied over from the /security dir in hbase?  Are there 
corresponding deletes?  What about some tests?


- Michael


On 2012-05-01 20:27:30, Devaraj Das wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4953/
bq.  ---
bq.  
bq.  (Updated 2012-05-01 20:27:30)
bq.  
bq.  
bq.  Review request for Ted Yu, Michael Stack and Andrew Purtell.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Reviewboard request for HBASE-5732
bq.  
bq.  
bq.  This addresses bug HBASE-5732.
bq.  https://issues.apache.org/jira/browse/HBASE-5732
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java
 1332383 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java
 1332383 
bq.http://svn.apache.org/repos/asf/hbase/trunk/pom.xml 1332383 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
 1332383 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/Status.java
 1332383 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
 1332383 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java
 1332383 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java
 PRE-CREATION 
bq.

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266307#comment-13266307
 ] 

jirapos...@reviews.apache.org commented on HBASE-5547:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4633/#review7458
---


Thanks for the review. Going to do a bit of investigation as to more 
commonality/interface hiding, but unfortunately we handle hfiles in both the RS 
and Master (and need to actuate from the client), so we need to touch a lot. 


src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java
https://reviews.apache.org/r/4633/#comment16469

hfiles, tables are another patch :)



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java
https://reviews.apache.org/r/4633/#comment16468

top-level as in o.a.h.hbase? Seems like a bit of clutter and this is just a 
specialization on that packaging and IMO doesn't preclude either master or rs 
packages from usage.

But I can move it if there are strong feelings.



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java
https://reviews.apache.org/r/4633/#comment16470

your right, fubbing my commenting a bit.



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java
https://reviews.apache.org/r/4633/#comment16471

But its does tracking! And it doesn't actually actuate any changes on the 
rest of the cluster so Manager seems bad too. How about Monitor?



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java
https://reviews.apache.org/r/4633/#comment16472

Started thinking about your later comment about a factory in HRS.java and 
think I'm going with the static factory method there. This still needs to be 
public for testing, but in general we will use the the 
ServerArchiveTableTracker (and I'll add docs to that effect as well).



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java
https://reviews.apache.org/r/4633/#comment16473

It won't be used except for at startup, but seemed bad practice to 
prematurely read ZK before we get the start call on the parent. At the very 
least we can say that we are removing the previous tables though.



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment16474

Should be fixed with the naming change above.



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment16475

Yeah, its definitely a bit odd. Going with a factory here and only having 
the pass in option for testing exposure. See above (tabletracker) and below 
(hrs) for more info on the factory stuff.



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment16476

done.



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment16477

Yeah, its pretty gross. I've been thinking about the ways we can fix 
this/overhaul the zookeeper infrastructure.



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment16478

This is actually a little nicer in that it doesn't preclude you from having 
a table named the same thing as the archive directory.

Otherwise, you are checking the node vs. the node name of 
watcher.archiveHFileZNode, which could be the same. 



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment16479

Done. Seemed to be the prevailing style, but this goes back to the fact 
that the watcher stuff if crufty.



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment16480

correct! Though anything in them would naturally be pb-ed :)



src/main/java/org/apache/hadoop/hbase/backup/ServerHFileTableArchiveTracker.java
https://reviews.apache.org/r/4633/#comment16481

Inherit from tabletracker so we know when we should archive. However, seems 
like a bit cleaner to just let the subclasses just override registerTable() for 
testing.



src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
https://reviews.apache.org/r/4633/#comment16482

what do you mean by backup? Its moving them to another directory, rather 
than being deleted. Sounds like backup to me :)



src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
https://reviews.apache.org/r/4633/#comment16483

just throws a switch. should note its async, so you could have some extra 
backup files, but ZK is fast, so it shouldn' be that many.



src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java

[jira] [Commented] (HBASE-5712) Parallelize load of .regioninfo files in diagnostic/repair portion of hbck.

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264691#comment-13264691
 ] 

jirapos...@reviews.apache.org commented on HBASE-5712:
--



bq.  On 2012-04-27 23:27:20, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 204
bq.   https://reviews.apache.org/r/4883/diff/1/?file=104442#file104442line204
bq.  
bq.   This'll work but why not ConcurrentSkipListMap?

Sure, changed.


bq.  On 2012-04-27 23:27:20, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 418
bq.   https://reviews.apache.org/r/4883/diff/1/?file=104442#file104442line418
bq.  
bq.   +1 on suggested change

done.


bq.  On 2012-04-27 23:27:20, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 2835
bq.   
https://reviews.apache.org/r/4883/diff/1/?file=104442#file104442line2835
bq.  
bq.   Is this flag needed?  Why not just check thread is alive?  I see we 
can return with an error.  What happens if the return on 2816 happens?  Will 
the wait at #643 above be for ever?

This is not a thread but actually fed to an executor (thread pool) at line 637. 
 If the return happens on 2816, this is in a finally which will always mark the 
workitem as done. 

There are two other instances of this pattern that were originally in this code 
before I got to it -- I'd have used Futures (and have filed a follow on issue 
for it) but it works.


- jmhsieh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4883/#review7337
---


On 2012-04-26 01:42:01, jmhsieh wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4883/
bq.  ---
bq.  
bq.  (Updated 2012-04-26 01:42:01)
bq.  
bq.  
bq.  Review request for hbase, Ted Yu and Jimmy Xiang.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  * Parallelized load of .regioninfo files
bq.  * changed TreeMap to SortedMap in method signatures
bq.  * renamed a test's name.
bq.  
bq.  
bq.  This addresses bug HBASE-5712.
bq.  https://issues.apache.org/jira/browse/HBASE-5712
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 66156c2 
bq.src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java 6b64f10 
bq.  
bq.  Diff: https://reviews.apache.org/r/4883/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Ran patch 10x on trunk, passes.  Ran 1x on 0.92 and 0.94.
bq.  
bq.  Ther 0.90 version that is nearly identical except for ignoring changes 
near lines HBaseFsck lines 671-680.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  jmhsieh
bq.  
bq.



 Parallelize load of .regioninfo files in diagnostic/repair portion of hbck.
 ---

 Key: HBASE-5712
 URL: https://issues.apache.org/jira/browse/HBASE-5712
 Project: HBase
  Issue Type: Improvement
  Components: hbck
Affects Versions: 0.90.7, 0.92.2, 0.94.0, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0

 Attachments: hbase-5712-90-v2.patch, hbase-5712-90.patch, 
 hbase-5712-v2.patch, hbase-5712.patch


 On heavily loaded hdfs's some dfs nodes may not respond quickly and backs off 
 for 60s before attempting to read data from another datanode.  Portions of 
 the information gathered from hdfs (.regioninfo files) are loaded serially.  
 With HBase with clusters with 100's, or 1000's, or 1's regions 
 encountering these 60s delay blocks progress and can be very painful.  
 There is already some parallelization of portions of the hdfs information 
 load operations and the goal here is move the reading of .regioninfos into 
 the parallelized sections..

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13265032#comment-13265032
 ] 

jirapos...@reviews.apache.org commented on HBASE-5869:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4926/#review7379
---


Looks good to me.

- Jimmy


On 2012-04-28 23:42:52, Michael Stack wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4926/
bq.  ---
bq.  
bq.  (Updated 2012-04-28 23:42:52)
bq.  
bq.  
bq.  Review request for hbase and Jimmy Xiang.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Convert two zk users to pb: distributed log splitting and regions in 
transition.
bq.  
bq.  Refactored distributed log splitting so we only serialize/deserialize in 
one location.
bq.  Less changes needed to do same for regions in transition.
bq.  
bq.  Moves serialization/deserialization out of the ZKAssign, ZKSplit and into
bq.  the classes themselves so can encapsulate how serialization is done into 
one place
bq.  (try to make the ZK* classes just deal in bytes -- about 90% done).
bq.  
bq.  Moved classes used by various packages up to top level to minimize imports
bq.  that are across package (zookeeper into protobuf and/or into regionserver 
and/or
bq.  master packages, etc).
bq.  
bq.  A src/main/java/org/apache/hadoop/hbase/DeserializationException.java
bq.New generic deserialization exception.
bq.  A src/main/java/org/apache/hadoop/hbase/zookeeper/EmptyWatcher.java
bq.  D  src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java
bq.Moved under zookeeper package.
bq.  A src/main/java/org/apache/hadoop/hbase/HBaseException.java
bq.New base hbase exception as suggested by hbase-5796.  New 
DeserializationException
bq.inherits from this.
bq.  A src/main/java/org/apache/hadoop/hbase/RegionTransition.java
bq.State of a region in transition.  Top-level because used by a
bq.few top-level packages.  Encapsulates pb serialization/deserialization.
bq.  M src/main/java/org/apache/hadoop/hbase/ServerName.java
bq.Add method to deserialize a ServeName, etc.  Encapsulates pb'ing.
bq.  M src/main/java/org/apache/hadoop/hbase/SplitLogCounters.java
bq.Counters used by distributed log splitting.
bq.  A SplitLogTask
bq. Class that encapsulates log splitting state.  Also encapsulates pb'ing.
bq.  M src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java
bq.Implement code for state.  Added functions to go from code to state and 
vice
bq.versa.  Used serializing.
bq.  M src/main/java/org/apache/hadoop/hbase/executor/ExecutorService.java
bq.Remove unused imports.
bq.  D src/main/java/org/apache/hadoop/hbase/executor/RegionTransitionData.java
bq.Removed.  Replaced by RegionTransition moved to package top-level.
bq.  M src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java
bq.  M src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
bq.Use new DeserializationException. Move to using new RegionTransition
bq.from RegionTransitionData class.  Pass deserialized class rather than
bq.byte array.  Remove duplicated code.
bq.  M src/main/java/org/apache/hadoop/hbase/master/HMaster.java
bq.Use new ServerName parse method rather than ZKUtil one.
bq.  M src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
bq.  M src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java
bq.Redo to use new SplitLogTask and SplitLogCounter classes.
bq.  M src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
bq.expectPBMagicPrefix added
bq.  M src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
bq.Use new RegionTransition in place of RegionTransitionData.
bq.  M src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
bq.Define moved from ZKSplitLog to SplitLogManager.
bq.  M src/main/java/org/apache/hadoop/hbase/zookeeper/MasterAddressTracker.java
bq.  M src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java
bq.Changed method name from getZNodeData to toByteArray to match how we've
bq.named it elsewhere. Use new DeserializationException
bq.  M src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java
bq.Use new RegionTransion class
bq.  M src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java
bq.Moved stuff that was in here up into SplitLogManager where better
bq.belongs.  Also moved serialization/deserialization up into the
bq.class itself: SplitLogTask.  Moved counters out to SplitLogCounter class.
bq.  M 

[jira] [Commented] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13265031#comment-13265031
 ] 

jirapos...@reviews.apache.org commented on HBASE-5869:
--



bq.  On 2012-04-28 22:14:23, Jimmy Xiang wrote:
bq.   src/main/protobuf/ZooKeeper.proto, line 82
bq.   https://reviews.apache.org/r/4926/diff/1/?file=105372#file105372line82
bq.  
bq.   A task is a path, this is more like a task state, isn't it?
bq.  
bq.  Michael Stack wrote:
bq.  I can change this np.
bq.  
bq.  Currently I have the pb class named same as the class that wraps it.  
Should I change this?  Add a pb prefix or something?   Problem w/ that is that 
no other of the pb classes have the pb prefix.  They are in the generated 
package which is probably sufficient to distingush them?  My hope is to make it 
so the pbs do not leak outside of the class that serializes to them; e.g. this 
SplitLogTask class.
bq.  
bq.  Jimmy Xiang wrote:
bq.  I got your point. I prefer to have the pb class named the same as the 
wrapper class, if there is one.  Should we create a separate task state wrapper 
class if needed?
bq.  
bq.  Michael Stack wrote:
bq.  I just tried changing the name of this class from SplitLogTask to 
SplitLogTaskState and it don't seem right since you can do a 'getState' call on 
this class -- the class has State AND the origin of the task.  I'm going to 
leave the name as is.
bq.  
bq.  Ok on keeping names the same.  It should be fine if we can keep the pb 
stuff bottled up under the pb package or internal only to the class that uses 
the pb (except where pb comes out on server..)
bq.  
bq.  Thanks  Jimmy

Ok, that's fine with me.


bq.  On 2012-04-28 22:14:23, Jimmy Xiang wrote:
bq.   src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java, 
line 182
bq.   https://reviews.apache.org/r/4926/diff/1/?file=105357#file105357line182
bq.  
bq.   Should we abort? Under what scenario the parsing can fail, other 
than a conflict data format?
bq.  
bq.  Michael Stack wrote:
bq.  I thought I was just redoing what was there previous.  We could abort 
but maybe next time through the deserialization works because its been updated 
by another?  Or, we spew this error all over the logs and drive someone crazy?  
 Will look at it again.
bq.  
bq.  Michael Stack wrote:
bq.  Yeah, I'll leave this as is after looking at it.  Hopefully will be 
good on next go around.

Ok


- Jimmy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4926/#review7360
---


On 2012-04-28 23:42:52, Michael Stack wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4926/
bq.  ---
bq.  
bq.  (Updated 2012-04-28 23:42:52)
bq.  
bq.  
bq.  Review request for hbase and Jimmy Xiang.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Convert two zk users to pb: distributed log splitting and regions in 
transition.
bq.  
bq.  Refactored distributed log splitting so we only serialize/deserialize in 
one location.
bq.  Less changes needed to do same for regions in transition.
bq.  
bq.  Moves serialization/deserialization out of the ZKAssign, ZKSplit and into
bq.  the classes themselves so can encapsulate how serialization is done into 
one place
bq.  (try to make the ZK* classes just deal in bytes -- about 90% done).
bq.  
bq.  Moved classes used by various packages up to top level to minimize imports
bq.  that are across package (zookeeper into protobuf and/or into regionserver 
and/or
bq.  master packages, etc).
bq.  
bq.  A src/main/java/org/apache/hadoop/hbase/DeserializationException.java
bq.New generic deserialization exception.
bq.  A src/main/java/org/apache/hadoop/hbase/zookeeper/EmptyWatcher.java
bq.  D  src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java
bq.Moved under zookeeper package.
bq.  A src/main/java/org/apache/hadoop/hbase/HBaseException.java
bq.New base hbase exception as suggested by hbase-5796.  New 
DeserializationException
bq.inherits from this.
bq.  A src/main/java/org/apache/hadoop/hbase/RegionTransition.java
bq.State of a region in transition.  Top-level because used by a
bq.few top-level packages.  Encapsulates pb serialization/deserialization.
bq.  M src/main/java/org/apache/hadoop/hbase/ServerName.java
bq.Add method to deserialize a ServeName, etc.  Encapsulates pb'ing.
bq.  M src/main/java/org/apache/hadoop/hbase/SplitLogCounters.java
bq.Counters used by distributed log splitting.
bq.  A SplitLogTask
bq. Class that encapsulates log splitting state.  Also encapsulates pb'ing.
bq.  M 

[jira] [Commented] (HBASE-5785) Adding unit tests for protbuf utils introduced for HRegionInterface pb conversion

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13265069#comment-13265069
 ] 

jirapos...@reviews.apache.org commented on HBASE-5785:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4936/
---

Review request for hbase and Michael Stack.


Summary
---

I added some tests for that conversion methods.  For those helper utilities, 
they are tested in other tests implicitly.  We can add more later on if needed.


This addresses bug HBASE-5785.
https://issues.apache.org/jira/browse/HBASE-5785


Diffs
-

  src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 994cb76 
  src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java 9b594aa 
  src/test/java/org/apache/hadoop/hbase/protobuf/TestProtobufUtil.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/4936/diff


Testing
---

The new tests are green.


Thanks,

Jimmy



 Adding unit tests for protbuf utils introduced for HRegionInterface pb 
 conversion
 -

 Key: HBASE-5785
 URL: https://issues.apache.org/jira/browse/HBASE-5785
 Project: HBase
  Issue Type: Sub-task
  Components: ipc, master, migration, regionserver
Affects Versions: 0.96.0
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Critical
  Labels: noob
 Fix For: 0.96.0

 Attachments: hbase-5785.patch


 We need to add some unit tests for the probuf utilities to catch issues 
 earlier.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2214) Do HBASE-1996 -- setting size to return in scan rather than count of rows -- properly

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13265164#comment-13265164
 ] 

jirapos...@reviews.apache.org commented on HBASE-2214:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4726/#review7383
---


Where are we checking the size of the result made so far?  I don't see it in 
the below.  I'd expect it inside in the RegionScanner.  Any chance of a test?  
Otherwise, patch looks great.


/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java
https://reviews.apache.org/r/4726/#comment16293

Is this going to be annoying?  If a high-traffic server, won't this get 
logged once per request?  Perhaps thousands a second?



/src/main/java/org/apache/hadoop/hbase/client/Scan.java
https://reviews.apache.org/r/4726/#comment16294

Is this needed?  Is this set on Scan creation?  When would it change after 
Scan construction?  Or, are we using builder pattern here and its set after 
construction but before use?



/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
https://reviews.apache.org/r/4726/#comment16295

oh, I see how its used now.  ignore above comment.


- Michael


On 2012-04-26 08:18:40, ferdy wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4726/
bq.  ---
bq.  
bq.  (Updated 2012-04-26 08:18:40)
bq.  
bq.  
bq.  Review request for hbase and Ted Yu.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  HBASE-2214 per scan max buffersize.
bq.  
bq.  
bq.  This addresses bug HBASE-2214.
bq.  https://issues.apache.org/jira/browse/HBASE-2214
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq./src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java 1330680 
bq./src/main/java/org/apache/hadoop/hbase/client/Scan.java 1330680 
bq./src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 
1330680 
bq./src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java 
1330680 
bq.
/src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java 
1330680 
bq.
/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java 
1330680 
bq.
/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 
1330680 
bq./src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java 
1330680 
bq.
/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java 
1330680 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1330680 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
1330680 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/RegionScanner.java 
1330680 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java 
1330680 
bq./src/main/protobuf/Client.proto 1330680 
bq.
/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
 1330680 
bq.  
bq.  Diff: https://reviews.apache.org/r/4726/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  It works when running this test:
bq.  
bq.  
bq.  new HBaseTestingUtility(conf).startMiniCluster();
bq.   
bq.  HBaseAdmin admin = new HBaseAdmin(conf);
bq.  if (!admin.tableExists(test)) {
bq.HTableDescriptor tableDesc = new HTableDescriptor(test);
bq.tableDesc.addFamily(new HColumnDescriptor(fam));
bq.admin.createTable(tableDesc);
bq.  }
bq.  
bq.  
bq.  HTable table = new HTable(conf, test);
bq.  Put put; 
bq.  
bq.  put = new Put(Bytes.toBytes(row1));
bq.  
put.add(Bytes.toBytes(fam),Bytes.toBytes(qual1),Bytes.toBytes(val1));
bq.  table.put(put);
bq.  
bq.  put = new Put(Bytes.toBytes(row2));
bq.  
put.add(Bytes.toBytes(fam),Bytes.toBytes(qual2),Bytes.toBytes(val2));
bq.  table.put(put);
bq.  
bq.  put = new Put(Bytes.toBytes(row3));
bq.  
put.add(Bytes.toBytes(fam),Bytes.toBytes(qual3),Bytes.toBytes(val3));
bq.  table.put(put);
bq.  
bq.  table.flushCommits();
bq.  {
bq.System.out.println(returns all rows at once because of the 
caching);
bq.Scan scan = new Scan();
bq.scan.setCaching(100);
bq.ResultScanner scanner = table.getScanner(scan);
bq.scanner.next(100);
bq.  }
bq.  {
bq.System.out.println(returns one row at a time because of the 
maxResultSize);
bq.Scan scan = new Scan();
bq.scan.setCaching(100);
bq.scan.setMaxResultSize(1);
bq.ResultScanner scanner = table.getScanner(scan);
bq.scanner.next(100);
bq.  }
bq.  
bq.  
bq.  See 

[jira] [Commented] (HBASE-5444) Add PB-based calls to HMasterRegionInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13265585#comment-13265585
 ] 

jirapos...@reviews.apache.org commented on HBASE-5444:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4463/
---

(Updated 2012-05-01 01:29:14.986766)


Review request for hbase and Michael Stack.


Summary
---

Adds PB-based calls replacing HMasterRegionInterface.

There are some temporary hacks, e.g. converting PB-based ServerLoad to existing 
HServerLoad so I didn't need to convert ClusterStatus (which brings in a lot of 
other changes).  That will be cleaned up in HBASE-5445.


This addresses bug HBASE-5444.
https://issues.apache.org/jira/browse/HBASE-5444


Diffs
-

  src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon 
69434f7 
  src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
3c7c091 
  src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 5d7f07b 
  src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 
  src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 0db2760 
  src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 973c7cb 
  src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java fd97830 
  src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java bb6ab3b 
  src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java f56127d 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java 81e9023 
  src/main/java/org/apache/hadoop/hbase/master/MXBean.java 7f44dc2 
  src/main/java/org/apache/hadoop/hbase/master/MXBeanImpl.java 45b8fe7 
  src/main/java/org/apache/hadoop/hbase/master/MasterDumpServlet.java be63838 
  src/main/java/org/apache/hadoop/hbase/master/RegionServerStatusProtocol.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 80271b1 
  src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 994cb76 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 
efcf74d 
  
src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java
 PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java ebffad6 
  src/main/protobuf/RegionServerStatus.proto PRE-CREATION 
  src/main/protobuf/hbase.proto 12e6053 
  src/main/resources/hbase-webapps/master/table.jsp 3ef1190 
  src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 72554cb 
  src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java 
d039be3 
  src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 
36046f8 
  src/test/java/org/apache/hadoop/hbase/master/TestMXBean.java bd5fa90 
  src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java f8029ba 
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java
 e99d251 

Diff: https://reviews.apache.org/r/4463/diff


Testing
---

Ran jenkins job, all unit tests passed.


Thanks,

Gregory



 Add PB-based calls to HMasterRegionInterface
 

 Key: HBASE-5444
 URL: https://issues.apache.org/jira/browse/HBASE-5444
 Project: HBase
  Issue Type: Sub-task
  Components: ipc, master, migration, regionserver
Reporter: Todd Lipcon
Assignee: Gregory Chanan
 Attachments: HBASE-5444-v6-trunk.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13265609#comment-13265609
 ] 

jirapos...@reviews.apache.org commented on HBASE-5547:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4633/#review7410
---



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment16338

agreed. I'll try to find something better.



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment16339

zk add/remove conditions where you need to keep re-adding watches. figure 
we do the check for the table at the same time.



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment16340

removed the creation of the tracker in favor of just passing in one - its 
really light weight and this should remove some of the issues around it; in 
short, yes.



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment16341

switching this mechanism to allow more fine grained listening by table 
trackers.


- Jesse


On 2012-04-26 00:24:36, Jesse Yates wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4633/
bq.  ---
bq.  
bq.  (Updated 2012-04-26 00:24:36)
bq.  
bq.  
bq.  Review request for hbase, Michael Stack and Lars Hofhansl.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Essentially, whenever an hfile would be deleted, it is instead moved to 
the archive directory. In this impl, the archive directory is on a per table 
basis, but defaults to '.archive'. Removing hfiles occurs in three places - 
compaction, merge and catalog janitor. The former and two latter are distinctly 
different code paths, but but did pull out some similarities. The latter two 
end up calling the same method, so there should be a reasonable amount of 
overlap.
bq.  
bq.  Implementation wise: 
bq.  Updated the HMasterInterface to pass the calls onto the zookeeper.
bq.  Added a zk listener to handle updates from the master to the RS to 
backup.
bq.  Added a utility for removing files and finding archive directories
bq.  Added tests for the regionserver and catalogjanitor approaches.
bq.  Added creation of manager in regionserver.
bq.  
bq.  
bq.  This addresses bug HBASE-5547.
bq.  https://issues.apache.org/jira/browse/HBASE-5547
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 
bq.src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java ee16e72 
bq.src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java d47b83a 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7858846 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
61a5988 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerHFileTableArchiveTracker.java
 PRE-CREATION 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 
6884d53 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 
ea12da4 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/Store.java bf1618e 
bq.src/main/java/org/apache/hadoop/hbase/util/HFileArchiveCleanup.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 
4fc105f 
bq.src/main/resources/hbase-default.xml f54b345 
bq.src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 
a59e152 
bq.src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 
cedf31e 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java
 PRE-CREATION 
bq.src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java 
PRE-CREATION 
bq.src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 
7d02759 
bq.
src/test/java/org/apache/hadoop/hbase/util/TestHFileArchivingCleanup.java 
PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/4633/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13265613#comment-13265613
 ] 

jirapos...@reviews.apache.org commented on HBASE-5547:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4633/
---

(Updated 2012-05-01 02:42:34.215505)


Review request for hbase, Michael Stack and Lars Hofhansl.


Changes
---

 - TableTracker is now an abstract class that the catalogjanitor and rs are
both using to register for tracking a table.
 - Adding monitoring in admin to check for cj to join archiving (before was 
missed) and including test
 - Slight cleanup from Ted's comments.


Summary
---

Essentially, whenever an hfile would be deleted, it is instead moved to the 
archive directory. In this impl, the archive directory is on a per table basis, 
but defaults to '.archive'. Removing hfiles occurs in three places - 
compaction, merge and catalog janitor. The former and two latter are distinctly 
different code paths, but but did pull out some similarities. The latter two 
end up calling the same method, so there should be a reasonable amount of 
overlap.

Implementation wise: 
Updated the HMasterInterface to pass the calls onto the zookeeper.
Added a zk listener to handle updates from the master to the RS to backup.
Added a utility for removing files and finding archive directories
Added tests for the regionserver and catalogjanitor approaches.
Added creation of manager in regionserver.


This addresses bug HBASE-5547.
https://issues.apache.org/jira/browse/HBASE-5547


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 
  src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java 
PRE-CREATION 
  
src/main/java/org/apache/hadoop/hbase/backup/ServerHFileTableArchiveTracker.java
 PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java ee16e72 
  src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java d47b83a 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7858846 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 61a5988 
  src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 
6884d53 
  src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 
ea12da4 
  src/main/java/org/apache/hadoop/hbase/regionserver/Store.java bf1618e 
  src/main/java/org/apache/hadoop/hbase/util/HFileArchiveCleanup.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f 
  src/main/resources/hbase-default.xml f54b345 
  src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java a59e152 
  src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java cedf31e 
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java
 PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java 
PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 
7d02759 
  src/test/java/org/apache/hadoop/hbase/util/TestHFileArchivingCleanup.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/4633/diff


Testing
---

Added two tests for the separate cases - archiving via the regionserver and for 
the catalog tracker. Former runs in a mini cluster and also touches the changes 
to HMasterInterface and zookeeper.


Thanks,

Jesse



 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates

 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is 

[jira] [Commented] (HBASE-5444) Add PB-based calls to HMasterRegionInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13265615#comment-13265615
 ] 

jirapos...@reviews.apache.org commented on HBASE-5444:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4463/#review7411
---



src/main/resources/hbase-webapps/master/table.jsp
https://reviews.apache.org/r/4463/#comment16343

Can this computation be moved somewhere so that it can be reused ?


- Ted


On 2012-05-01 01:29:14, Gregory Chanan wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4463/
bq.  ---
bq.  
bq.  (Updated 2012-05-01 01:29:14)
bq.  
bq.  
bq.  Review request for hbase and Michael Stack.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Adds PB-based calls replacing HMasterRegionInterface.
bq.  
bq.  There are some temporary hacks, e.g. converting PB-based ServerLoad to 
existing HServerLoad so I didn't need to convert ClusterStatus (which brings in 
a lot of other changes).  That will be cleaned up in HBASE-5445.
bq.  
bq.  
bq.  This addresses bug HBASE-5444.
bq.  https://issues.apache.org/jira/browse/HBASE-5444
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon 
69434f7 
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
3c7c091 
bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 5d7f07b 
bq.src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 0db2760 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 973c7cb 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 
fd97830 
bq.src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java bb6ab3b 
bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 
f56127d 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 81e9023 
bq.src/main/java/org/apache/hadoop/hbase/master/MXBean.java 7f44dc2 
bq.src/main/java/org/apache/hadoop/hbase/master/MXBeanImpl.java 45b8fe7 
bq.src/main/java/org/apache/hadoop/hbase/master/MasterDumpServlet.java 
be63838 
bq.
src/main/java/org/apache/hadoop/hbase/master/RegionServerStatusProtocol.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 80271b1 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 994cb76 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 
efcf74d 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java
 PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
ebffad6 
bq.src/main/protobuf/RegionServerStatus.proto PRE-CREATION 
bq.src/main/protobuf/hbase.proto 12e6053 
bq.src/main/resources/hbase-webapps/master/table.jsp 3ef1190 
bq.src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 72554cb 
bq.src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java 
d039be3 
bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 
36046f8 
bq.src/test/java/org/apache/hadoop/hbase/master/TestMXBean.java bd5fa90 
bq.src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java 
f8029ba 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java
 e99d251 
bq.  
bq.  Diff: https://reviews.apache.org/r/4463/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Ran jenkins job, all unit tests passed.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Gregory
bq.  
bq.



 Add PB-based calls to HMasterRegionInterface
 

 Key: HBASE-5444
 URL: https://issues.apache.org/jira/browse/HBASE-5444
 Project: HBase
  Issue Type: Sub-task
  Components: ipc, master, migration, regionserver
Reporter: Todd Lipcon
Assignee: Gregory Chanan
 Attachments: HBASE-5444-v6-trunk.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264372#comment-13264372
 ] 

jirapos...@reviews.apache.org commented on HBASE-5869:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4926/
---

Review request for hbase and Jimmy Xiang.


Summary
---

Convert two zk users to pb: distributed log splitting and regions in transition.

Refactored distributed log splitting so we only serialize/deserialize in one 
location.
Less changes needed to do same for regions in transition.

Moves serialization/deserialization out of the ZKAssign, ZKSplit and into
the classes themselves so can encapsulate how serialization is done into one 
place
(try to make the ZK* classes just deal in bytes -- about 90% done).

Moved classes used by various packages up to top level to minimize imports
that are across package (zookeeper into protobuf and/or into regionserver and/or
master packages, etc).

A src/main/java/org/apache/hadoop/hbase/DeserializationException.java
  New generic deserialization exception.
A src/main/java/org/apache/hadoop/hbase/zookeeper/EmptyWatcher.java
D  src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java
  Moved under zookeeper package.
A src/main/java/org/apache/hadoop/hbase/HBaseException.java
  New base hbase exception as suggested by hbase-5796.  New 
DeserializationException
  inherits from this.
A src/main/java/org/apache/hadoop/hbase/RegionTransition.java
  State of a region in transition.  Top-level because used by a
  few top-level packages.  Encapsulates pb serialization/deserialization.
M src/main/java/org/apache/hadoop/hbase/ServerName.java
  Add method to deserialize a ServeName, etc.  Encapsulates pb'ing.
M src/main/java/org/apache/hadoop/hbase/SplitLogCounters.java
  Counters used by distributed log splitting.
A SplitLogTask
   Class that encapsulates log splitting state.  Also encapsulates pb'ing.
M src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java
  Implement code for state.  Added functions to go from code to state and vice
  versa.  Used serializing.
M src/main/java/org/apache/hadoop/hbase/executor/ExecutorService.java
  Remove unused imports.
D src/main/java/org/apache/hadoop/hbase/executor/RegionTransitionData.java
  Removed.  Replaced by RegionTransition moved to package top-level.
M src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java
M src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
  Use new DeserializationException. Move to using new RegionTransition
  from RegionTransitionData class.  Pass deserialized class rather than
  byte array.  Remove duplicated code.
M src/main/java/org/apache/hadoop/hbase/master/HMaster.java
  Use new ServerName parse method rather than ZKUtil one.
M src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
M src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java
  Redo to use new SplitLogTask and SplitLogCounter classes.
M src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
  expectPBMagicPrefix added
M src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
  Use new RegionTransition in place of RegionTransitionData.
M src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
  Define moved from ZKSplitLog to SplitLogManager.
M src/main/java/org/apache/hadoop/hbase/zookeeper/MasterAddressTracker.java
M src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java
  Changed method name from getZNodeData to toByteArray to match how we've
  named it elsewhere. Use new DeserializationException
M src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java
  Use new RegionTransion class
M src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java
  Moved stuff that was in here up into SplitLogManager where better
  belongs.  Also moved serialization/deserialization up into the
  class itself: SplitLogTask.  Moved counters out to SplitLogCounter class.
M src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
  Moved deserialization of ServerName out of here and up into ServerName.
M src/main/protobuf/ZooKeeper.proto
  Add two new classes, RegionTransition and SplitLogTask.


This addresses bug HBASE-5869.
https://issues.apache.org/jira/browse/HBASE-5869


Diffs
-

  src/main/java/org/apache/hadoop/hbase/DeserializationException.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java 9881ec2 
  src/main/java/org/apache/hadoop/hbase/HBaseException.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/RegionTransition.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/ServerName.java 8fdb624 
  src/main/java/org/apache/hadoop/hbase/SplitLogCounters.java PRE-CREATION