[
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276887#comment-13276887
]
[email protected] commented on HBASE-5453:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/
-----------------------------------------------------------
(Updated 2012-05-16 17:02:35.527708)
Review request for hbase.
Changes
-------
Gregory. I addressed your comments.
I've changed the format of .regioninfo and .tableinfo. Now instead of
serialized Writable followed by toString of the serialized object, instead its
just the serialized pb. This removes our having a human readable
.regioninfo/.tablinfo file but my guess no one relied on this anyways.
Having just serialized content in the file means a check of file length should
be enough figuring whether the file properly serialized. If ever a chance that
a Writable + its toString + two '\n' characters was equal to a serialized pb,
I'd think this a pathological state. If this state is not cleared up
'naturally' by splits or a schema change, then lets deal if it happens.
I only need this length-checking in one place on region open. I want to avoid
reading the .regioninfo file on region open. The alternative means more load on
NN and DNs at region open time which could be problematic at big-bang cluster
start (Thinking 500 nodes w/ 80k regions, an actual known case -- this is the
case I have in mind when I'm trying to avoid more load on NN/DNs).
Otherwise, Gregory's comments led to me to check and I was missing convertion
of fs files to pb in all cases (I was just reading the clusterid and
hbase.version files, not converting if still Writable). This should be fixed
now.
There are some failing tests still but running by hadoopqa to see what it says
anyways. Also putting up on rb to get feedback if problem w/ this approach.
Summary
-------
A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
New class to hold clusterid in.
M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
Make it so can do pb serialization. Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
Make it so can do pb serialization. Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
ClusterId under ZK got renamed as ZKClusterId
M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
Hide the Reference#Range enums. Don't let them out of this class.
Make it so can do pb serialization.
M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
Use new methods on Reference for getting top and bottom.
M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
ClusterId under zk has been renamed ZKClusterId.
Use new ClusterId class too.
M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
Use new clusterid class.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
Move the RegionInfo convertion up into HRegionInfo instead of here.
Added generic toDelimitedByteArray helper.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
Use HRegionInfo convertions instead.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
Use HRegionInfo convertions instead.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Use new utility writing out .regioninfo files.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
Formatting.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
Range in Reference is no longer public.
Range in Reference is no longer public.
M
b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
M
b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
ClusterId got renamed ZKClusterId
M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
Use new serialization utlity in HTD.
M b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
Generic method for writing dot file content.
M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
Reference#Range is not public any more
M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java
Deprecated getHRegionInfo, etc.
D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java
A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java
Rename
A b/src/main/protobuf/ClusterId.proto
Added file for ClusterId only since its written to fs and to zk.
A b/src/main/protobuf/FS.proto
Protos for fs files.
M b/src/main/protobuf/ZooKeeper.proto
Moved ClusterId out to own proto file
M b/src/main/protobuf/hbase.proto
Added TableSchema and ColumnFamilySchema
This addresses bug hbase-5453.
https://issues.apache.org/jira/browse/hbase-5453
Diffs (updated)
-----
src/main/java/org/apache/hadoop/hbase/ClusterId.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 5862f15
src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 8d83ff3
src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java af89e3e
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5cac9af
src/main/java/org/apache/hadoop/hbase/io/Reference.java 6360059
src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
9e4ada9
src/main/java/org/apache/hadoop/hbase/master/HMaster.java 947ec5f
src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 5052878
src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java ccc964e
src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java dabfbab
src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java 45cb6cf
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClusterIdProtos.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/protobuf/generated/FSProtos.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java
058c006
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java
20c7738
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 9f16fee
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 6dc0517
src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
6a9f2fe
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 5e1e16d
src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
5050df0
src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
049ed8d
src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java efb2b84
src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 3d35d3e
src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 7b4f4a2
src/main/java/org/apache/hadoop/hbase/util/Writables.java 3d20723
src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java f804810
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java PRE-CREATION
src/main/protobuf/ClusterId.proto PRE-CREATION
src/main/protobuf/FS.proto PRE-CREATION
src/main/protobuf/ZooKeeper.proto b72cb28
src/main/protobuf/hbase.proto 30a4c3f
src/test/java/org/apache/hadoop/hbase/TestHColumnDescriptor.java e7fa8b2
src/test/java/org/apache/hadoop/hbase/TestHTableDescriptor.java f7c0cca
src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java 83d8408
src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 69ccc65
src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 1020374
src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java
6dfba41
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 988d0bf
src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java 339a120
Diff: https://reviews.apache.org/r/5130/diff
Testing
-------
Thanks,
Michael
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Attachments: 5453.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
