[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13283833#comment-13283833 ] Hudson commented on HBASE-5453: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #16 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/16/]) HBASE-6063 Replication related failures on trunk after HBASE-5453 (Gregory Channan) (Revision 1342095) Result = FAILURE larsh : Files : * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ClusterId.java Switch on-disk formats (reference files, HFile meta fields, etc) to PB -- Key: HBASE-5453 URL: https://issues.apache.org/jira/browse/HBASE-5453 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Todd Lipcon Assignee: stack Fix For: 0.96.0 Attachments: 5453.txt, 5453v10.txt, 5453v11.txt, 5453v11.txt, 5453v12.txt, 5453v13.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13282118#comment-13282118 ] Hudson commented on HBASE-5453: --- Integrated in HBase-TRUNK #2919 (See [https://builds.apache.org/job/HBase-TRUNK/2919/]) HBASE-6063 Replication related failures on trunk after HBASE-5453 (Gregory Channan) (Revision 1342095) Result = FAILURE larsh : Files : * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ClusterId.java Switch on-disk formats (reference files, HFile meta fields, etc) to PB -- Key: HBASE-5453 URL: https://issues.apache.org/jira/browse/HBASE-5453 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Todd Lipcon Assignee: stack Fix For: 0.96.0 Attachments: 5453.txt, 5453v10.txt, 5453v11.txt, 5453v11.txt, 5453v12.txt, 5453v13.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13279053#comment-13279053 ] Hadoop QA commented on HBASE-5453: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12528107/5453v12.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 30 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 32 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildOverlap org.apache.hadoop.hbase.replication.TestReplication org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster org.apache.hadoop.hbase.replication.TestMultiSlaveReplication org.apache.hadoop.hbase.coprocessor.TestClassLoading org.apache.hadoop.hbase.replication.TestMasterReplication Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1934//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1934//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1934//console This message is automatically generated. Switch on-disk formats (reference files, HFile meta fields, etc) to PB -- Key: HBASE-5453 URL: https://issues.apache.org/jira/browse/HBASE-5453 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Todd Lipcon Assignee: stack Fix For: 0.96.0 Attachments: 5453.txt, 5453v10.txt, 5453v11.txt, 5453v11.txt, 5453v12.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13279116#comment-13279116 ] Hadoop QA commented on HBASE-5453: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12528129/5453v13.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 33 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 33 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestColumnSeeking Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1937//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1937//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1937//console This message is automatically generated. Switch on-disk formats (reference files, HFile meta fields, etc) to PB -- Key: HBASE-5453 URL: https://issues.apache.org/jira/browse/HBASE-5453 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Todd Lipcon Assignee: stack Fix For: 0.96.0 Attachments: 5453.txt, 5453v10.txt, 5453v11.txt, 5453v11.txt, 5453v12.txt, 5453v13.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13279134#comment-13279134 ] stack commented on HBASE-5453: -- The failed test passes locally. Will commit soon unless objection. Switch on-disk formats (reference files, HFile meta fields, etc) to PB -- Key: HBASE-5453 URL: https://issues.apache.org/jira/browse/HBASE-5453 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Todd Lipcon Assignee: stack Fix For: 0.96.0 Attachments: 5453.txt, 5453v10.txt, 5453v11.txt, 5453v11.txt, 5453v12.txt, 5453v13.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB
[
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13279137#comment-13279137
]
Zhihong Yu commented on HBASE-5453:
---
{code}
+ * @return This instance serialized with pb with pb magic prefix
{code}
Remove redundant 'with pb '.
{code}
+ * @return Convert this instance to a the pb column family type
{code}
Remove 'a '
Switch on-disk formats (reference files, HFile meta fields, etc) to PB
--
Key: HBASE-5453
URL: https://issues.apache.org/jira/browse/HBASE-5453
Project: HBase
Issue Type: Sub-task
Components: ipc, master, migration, regionserver
Reporter: Todd Lipcon
Assignee: stack
Fix For: 0.96.0
Attachments: 5453.txt, 5453v10.txt, 5453v11.txt, 5453v11.txt,
5453v12.txt, 5453v13.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277596#comment-13277596 ] jirapos...@reviews.apache.org commented on HBASE-5453: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5130/ --- (Updated 2012-05-17 05:58:11.354772) Review request for hbase. Changes --- Address Gregory's two review comments. Summary --- A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java New class to hold clusterid in. M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java Make it so can do pb serialization. Deprecated Writable serialization. M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java Make it so methods in here follow the pattern in HCD an HTD pb 'ing. Deprecated Writable serialization. M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java Make it so can do pb serialization. Deprecated Writable serialization. M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java ClusterId under ZK got renamed as ZKClusterId M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java Hide the Reference#Range enums. Don't let them out of this class. Make it so can do pb serialization. M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java Use new methods on Reference for getting top and bottom. M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java ClusterId under zk has been renamed ZKClusterId. Use new ClusterId class too. M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java Use new clusterid class. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java Move the RegionInfo convertion up into HRegionInfo instead of here. Added generic toDelimitedByteArray helper. M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java Use HRegionInfo convertions instead. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java Use HRegionInfo convertions instead. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java Use new utility writing out .regioninfo files. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java Formatting. M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java Range in Reference is no longer public. Range in Reference is no longer public. M b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java M b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java ClusterId got renamed ZKClusterId M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java Use new serialization utlity in HTD. M b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java Generic method for writing dot file content. M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java Reference#Range is not public any more M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java Deprecated getHRegionInfo, etc. D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java Rename A b/src/main/protobuf/ClusterId.proto Added file for ClusterId only since its written to fs and to zk. A b/src/main/protobuf/FS.proto Protos for fs files. M b/src/main/protobuf/ZooKeeper.proto Moved ClusterId out to own proto file M b/src/main/protobuf/hbase.proto Added TableSchema and ColumnFamilySchema This addresses bug hbase-5453. https://issues.apache.org/jira/browse/hbase-5453 Diffs (updated) - src/main/java/org/apache/hadoop/hbase/ClusterId.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 5862f15 src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 8d83ff3 src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java af89e3e src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5cac9af src/main/java/org/apache/hadoop/hbase/io/Reference.java 6360059 src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 9e4ada9 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 947ec5f src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 5052878 src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java ccc964e src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java dabfbab src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java 45cb6cf src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClusterIdProtos.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/protobuf/generated/FSProtos.java PRE-CREATION
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277620#comment-13277620 ] Hadoop QA commented on HBASE-5453: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12527785/5453v11.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 27 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 32 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestMultipleTimestamps org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildOverlap org.apache.hadoop.hbase.replication.TestReplication org.apache.hadoop.hbase.client.TestTimestampsFilter org.apache.hadoop.hbase.replication.TestMultiSlaveReplication org.apache.hadoop.hbase.client.TestFromClientSide org.apache.hadoop.hbase.mapreduce.TestImportExport org.apache.hadoop.hbase.replication.TestMasterReplication Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1906//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1906//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1906//console This message is automatically generated. Switch on-disk formats (reference files, HFile meta fields, etc) to PB -- Key: HBASE-5453 URL: https://issues.apache.org/jira/browse/HBASE-5453 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Todd Lipcon Assignee: stack Fix For: 0.96.0 Attachments: 5453.txt, 5453v10.txt, 5453v11.txt, 5453v11.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13278263#comment-13278263 ] jirapos...@reviews.apache.org commented on HBASE-5453: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5130/#review7963 --- Ship it! src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java https://reviews.apache.org/r/5130/#comment17303 Your logic sounds correct and I agree we should avoid having to read the file each time. It would be cool to just see an example of the old writable vs new pb size to ensure our intuition is correct. Worst case (let's say they happen to be the same) we could write out the new pb stuff concatenated with the old way of writing and then they are guaranteed to be different sizes. Hopefully we don't have to do something like that, though. - Gregory On 2012-05-17 05:58:11, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/5130/ bq. --- bq. bq. (Updated 2012-05-17 05:58:11) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java bq.New class to hold clusterid in. bq. M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java bq.Make it so can do pb serialization. Deprecated Writable serialization. bq. M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java bq.Make it so methods in here follow the pattern in HCD an HTD pb 'ing. bq.Deprecated Writable serialization. bq. M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java bq.Make it so can do pb serialization. Deprecated Writable serialization. bq. M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java bq.ClusterId under ZK got renamed as ZKClusterId bq. M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java bq.Hide the Reference#Range enums. Don't let them out of this class. bq.Make it so can do pb serialization. bq. M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java bq.Use new methods on Reference for getting top and bottom. bq. M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java bq.ClusterId under zk has been renamed ZKClusterId. bq.Use new ClusterId class too. bq. M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java bq.Use new clusterid class. bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java bq.Move the RegionInfo convertion up into HRegionInfo instead of here. bq.Added generic toDelimitedByteArray helper. bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java bq.Use HRegionInfo convertions instead. bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java bq.Use HRegionInfo convertions instead. bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java bq.Use new utility writing out .regioninfo files. bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java bq.Formatting. bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java bq.Range in Reference is no longer public. bq.Range in Reference is no longer public. bq. M b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java bq. M b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java bq.ClusterId got renamed ZKClusterId bq. M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java bq.Use new serialization utlity in HTD. bq. M b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java bq.Generic method for writing dot file content. bq. M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java bq.Reference#Range is not public any more bq. M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java bq.Deprecated getHRegionInfo, etc. bq. D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java bq. A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java bq.Rename bq. A b/src/main/protobuf/ClusterId.proto bq.Added file for ClusterId only since its written to fs and to zk. bq. A b/src/main/protobuf/FS.proto bq.Protos for fs files. bq. M b/src/main/protobuf/ZooKeeper.proto bq.Moved ClusterId out to own proto file bq. M b/src/main/protobuf/hbase.proto bq.Added TableSchema and ColumnFamilySchema bq. bq. bq. This
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13276887#comment-13276887 ] jirapos...@reviews.apache.org commented on HBASE-5453: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5130/ --- (Updated 2012-05-16 17:02:35.527708) Review request for hbase. Changes --- Gregory. I addressed your comments. I've changed the format of .regioninfo and .tableinfo. Now instead of serialized Writable followed by toString of the serialized object, instead its just the serialized pb. This removes our having a human readable .regioninfo/.tablinfo file but my guess no one relied on this anyways. Having just serialized content in the file means a check of file length should be enough figuring whether the file properly serialized. If ever a chance that a Writable + its toString + two '\n' characters was equal to a serialized pb, I'd think this a pathological state. If this state is not cleared up 'naturally' by splits or a schema change, then lets deal if it happens. I only need this length-checking in one place on region open. I want to avoid reading the .regioninfo file on region open. The alternative means more load on NN and DNs at region open time which could be problematic at big-bang cluster start (Thinking 500 nodes w/ 80k regions, an actual known case -- this is the case I have in mind when I'm trying to avoid more load on NN/DNs). Otherwise, Gregory's comments led to me to check and I was missing convertion of fs files to pb in all cases (I was just reading the clusterid and hbase.version files, not converting if still Writable). This should be fixed now. There are some failing tests still but running by hadoopqa to see what it says anyways. Also putting up on rb to get feedback if problem w/ this approach. Summary --- A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java New class to hold clusterid in. M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java Make it so can do pb serialization. Deprecated Writable serialization. M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java Make it so methods in here follow the pattern in HCD an HTD pb 'ing. Deprecated Writable serialization. M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java Make it so can do pb serialization. Deprecated Writable serialization. M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java ClusterId under ZK got renamed as ZKClusterId M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java Hide the Reference#Range enums. Don't let them out of this class. Make it so can do pb serialization. M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java Use new methods on Reference for getting top and bottom. M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java ClusterId under zk has been renamed ZKClusterId. Use new ClusterId class too. M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java Use new clusterid class. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java Move the RegionInfo convertion up into HRegionInfo instead of here. Added generic toDelimitedByteArray helper. M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java Use HRegionInfo convertions instead. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java Use HRegionInfo convertions instead. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java Use new utility writing out .regioninfo files. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java Formatting. M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java Range in Reference is no longer public. Range in Reference is no longer public. M b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java M b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java ClusterId got renamed ZKClusterId M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java Use new serialization utlity in HTD. M b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java Generic method for writing dot file content. M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java Reference#Range is not public any more M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java Deprecated getHRegionInfo, etc. D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java Rename A b/src/main/protobuf/ClusterId.proto Added file for ClusterId only since
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277200#comment-13277200 ] jirapos...@reviews.apache.org commented on HBASE-5453: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5130/#review7944 --- In a previous comment you said (about the HTableDescriptor/HColumnDesriptor pb stuff): Well, Andrew beat us both to it over in his REST pb stuff. We need to reconcile his w/ ours too Where is his stuff? I couldn't find it. Should we create a JIRA about reconciling? It would be nice to have something, however imperfect, up in trunk to work against, then we could fix up later. src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java https://reviews.apache.org/r/5130/#comment17274 This isn't ever read? src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java https://reviews.apache.org/r/5130/#comment17273 This constructor sets pbMade to false, but it should be true in this case, right? - Gregory On 2012-05-16 17:02:35, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/5130/ bq. --- bq. bq. (Updated 2012-05-16 17:02:35) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java bq.New class to hold clusterid in. bq. M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java bq.Make it so can do pb serialization. Deprecated Writable serialization. bq. M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java bq.Make it so methods in here follow the pattern in HCD an HTD pb 'ing. bq.Deprecated Writable serialization. bq. M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java bq.Make it so can do pb serialization. Deprecated Writable serialization. bq. M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java bq.ClusterId under ZK got renamed as ZKClusterId bq. M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java bq.Hide the Reference#Range enums. Don't let them out of this class. bq.Make it so can do pb serialization. bq. M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java bq.Use new methods on Reference for getting top and bottom. bq. M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java bq.ClusterId under zk has been renamed ZKClusterId. bq.Use new ClusterId class too. bq. M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java bq.Use new clusterid class. bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java bq.Move the RegionInfo convertion up into HRegionInfo instead of here. bq.Added generic toDelimitedByteArray helper. bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java bq.Use HRegionInfo convertions instead. bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java bq.Use HRegionInfo convertions instead. bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java bq.Use new utility writing out .regioninfo files. bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java bq.Formatting. bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java bq.Range in Reference is no longer public. bq.Range in Reference is no longer public. bq. M b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java bq. M b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java bq.ClusterId got renamed ZKClusterId bq. M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java bq.Use new serialization utlity in HTD. bq. M b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java bq.Generic method for writing dot file content. bq. M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java bq.Reference#Range is not public any more bq. M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java bq.Deprecated getHRegionInfo, etc. bq. D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java bq. A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java bq.Rename bq. A b/src/main/protobuf/ClusterId.proto bq.Added file for ClusterId only since its written to fs and to zk. bq. A b/src/main/protobuf/FS.proto bq.Protos for fs files. bq. M b/src/main/protobuf/ZooKeeper.proto bq.Moved ClusterId
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277273#comment-13277273 ] jirapos...@reviews.apache.org commented on HBASE-5453: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5130/ --- (Updated 2012-05-16 23:56:18.496225) Review request for hbase. Changes --- v10... the last thing I added to JIRA Summary --- A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java New class to hold clusterid in. M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java Make it so can do pb serialization. Deprecated Writable serialization. M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java Make it so methods in here follow the pattern in HCD an HTD pb 'ing. Deprecated Writable serialization. M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java Make it so can do pb serialization. Deprecated Writable serialization. M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java ClusterId under ZK got renamed as ZKClusterId M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java Hide the Reference#Range enums. Don't let them out of this class. Make it so can do pb serialization. M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java Use new methods on Reference for getting top and bottom. M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java ClusterId under zk has been renamed ZKClusterId. Use new ClusterId class too. M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java Use new clusterid class. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java Move the RegionInfo convertion up into HRegionInfo instead of here. Added generic toDelimitedByteArray helper. M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java Use HRegionInfo convertions instead. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java Use HRegionInfo convertions instead. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java Use new utility writing out .regioninfo files. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java Formatting. M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java Range in Reference is no longer public. Range in Reference is no longer public. M b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java M b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java ClusterId got renamed ZKClusterId M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java Use new serialization utlity in HTD. M b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java Generic method for writing dot file content. M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java Reference#Range is not public any more M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java Deprecated getHRegionInfo, etc. D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java Rename A b/src/main/protobuf/ClusterId.proto Added file for ClusterId only since its written to fs and to zk. A b/src/main/protobuf/FS.proto Protos for fs files. M b/src/main/protobuf/ZooKeeper.proto Moved ClusterId out to own proto file M b/src/main/protobuf/hbase.proto Added TableSchema and ColumnFamilySchema This addresses bug hbase-5453. https://issues.apache.org/jira/browse/hbase-5453 Diffs (updated) - src/main/java/org/apache/hadoop/hbase/ClusterId.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 5862f15 src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 8d83ff3 src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java af89e3e src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5cac9af src/main/java/org/apache/hadoop/hbase/io/Reference.java 6360059 src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 9e4ada9 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 947ec5f src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 5052878 src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java ccc964e src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java dabfbab src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java 45cb6cf src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClusterIdProtos.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/protobuf/generated/FSProtos.java PRE-CREATION
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277272#comment-13277272 ] jirapos...@reviews.apache.org commented on HBASE-5453: -- bq. On 2012-05-16 22:12:10, Gregory Chanan wrote: bq. In a previous comment you said (about the HTableDescriptor/HColumnDesriptor pb stuff): bq. Well, Andrew beat us both to it over in his REST pb stuff. We need to reconcile his w/ ours too bq. bq. Where is his stuff? I couldn't find it. Should we create a JIRA about reconciling? It would be nice to have something, however imperfect, up in trunk to work against, then we could fix up later. Its under src/main/resources/org I just tripped over it myself yesterday. I made HBASE-6026 to do the reconcile (after HBASE-6000 goes in). Are you down w/ the change in .regioninfo Gregory as means of 'ensuring' we don't have a Writable and pb serialization end up as same size? Thanks for review. bq. On 2012-05-16 22:12:10, Gregory Chanan wrote: bq. src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java, line 276 bq. https://reviews.apache.org/r/5130/diff/1-2/?file=109037#file109037line276 bq. bq. This isn't ever read? This is gone now. I used another technique figuring if object serialized -- read file fully into byte array and test for the pb prefix -- rather than this hacky setting attribute on class. - Michael --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5130/#review7944 --- On 2012-05-16 17:02:35, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/5130/ bq. --- bq. bq. (Updated 2012-05-16 17:02:35) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java bq.New class to hold clusterid in. bq. M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java bq.Make it so can do pb serialization. Deprecated Writable serialization. bq. M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java bq.Make it so methods in here follow the pattern in HCD an HTD pb 'ing. bq.Deprecated Writable serialization. bq. M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java bq.Make it so can do pb serialization. Deprecated Writable serialization. bq. M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java bq.ClusterId under ZK got renamed as ZKClusterId bq. M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java bq.Hide the Reference#Range enums. Don't let them out of this class. bq.Make it so can do pb serialization. bq. M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java bq.Use new methods on Reference for getting top and bottom. bq. M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java bq.ClusterId under zk has been renamed ZKClusterId. bq.Use new ClusterId class too. bq. M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java bq.Use new clusterid class. bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java bq.Move the RegionInfo convertion up into HRegionInfo instead of here. bq.Added generic toDelimitedByteArray helper. bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java bq.Use HRegionInfo convertions instead. bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java bq.Use HRegionInfo convertions instead. bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java bq.Use new utility writing out .regioninfo files. bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java bq.Formatting. bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java bq.Range in Reference is no longer public. bq.Range in Reference is no longer public. bq. M b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java bq. M b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java bq.ClusterId got renamed ZKClusterId bq. M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java bq.Use new serialization utlity in HTD. bq. M b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java bq.Generic method for writing dot file content. bq. M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java bq.Reference#Range is not public any more bq. M
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277589#comment-13277589 ] jirapos...@reviews.apache.org commented on HBASE-5453: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5130/#review7951 --- src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java https://reviews.apache.org/r/5130/#comment17291 This is still here? Or just reviewboard showing it for some reason? src/main/java/org/apache/hadoop/hbase/util/FSUtils.java https://reviews.apache.org/r/5130/#comment17290 This is what you are using to ensure the sizes cannot be the same? Very nice! I cannot find a call-site for this, though, maybe I missed it? - Gregory On 2012-05-16 23:56:18, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/5130/ bq. --- bq. bq. (Updated 2012-05-16 23:56:18) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java bq.New class to hold clusterid in. bq. M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java bq.Make it so can do pb serialization. Deprecated Writable serialization. bq. M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java bq.Make it so methods in here follow the pattern in HCD an HTD pb 'ing. bq.Deprecated Writable serialization. bq. M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java bq.Make it so can do pb serialization. Deprecated Writable serialization. bq. M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java bq.ClusterId under ZK got renamed as ZKClusterId bq. M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java bq.Hide the Reference#Range enums. Don't let them out of this class. bq.Make it so can do pb serialization. bq. M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java bq.Use new methods on Reference for getting top and bottom. bq. M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java bq.ClusterId under zk has been renamed ZKClusterId. bq.Use new ClusterId class too. bq. M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java bq.Use new clusterid class. bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java bq.Move the RegionInfo convertion up into HRegionInfo instead of here. bq.Added generic toDelimitedByteArray helper. bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java bq.Use HRegionInfo convertions instead. bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java bq.Use HRegionInfo convertions instead. bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java bq.Use new utility writing out .regioninfo files. bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java bq.Formatting. bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java bq.Range in Reference is no longer public. bq.Range in Reference is no longer public. bq. M b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java bq. M b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java bq.ClusterId got renamed ZKClusterId bq. M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java bq.Use new serialization utlity in HTD. bq. M b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java bq.Generic method for writing dot file content. bq. M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java bq.Reference#Range is not public any more bq. M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java bq.Deprecated getHRegionInfo, etc. bq. D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java bq. A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java bq.Rename bq. A b/src/main/protobuf/ClusterId.proto bq.Added file for ClusterId only since its written to fs and to zk. bq. A b/src/main/protobuf/FS.proto bq.Protos for fs files. bq. M b/src/main/protobuf/ZooKeeper.proto bq.Moved ClusterId out to own proto file bq. M b/src/main/protobuf/hbase.proto bq.Added TableSchema and ColumnFamilySchema bq. bq. bq. This addresses bug hbase-5453. bq. https://issues.apache.org/jira/browse/hbase-5453 bq. bq. bq. Diffs bq. - bq. bq.
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277593#comment-13277593 ] jirapos...@reviews.apache.org commented on HBASE-5453: -- bq. On 2012-05-17 05:41:53, Gregory Chanan wrote: bq. src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java, line 275 bq. https://reviews.apache.org/r/5130/diff/3/?file=109313#file109313line275 bq. bq. This is still here? Or just reviewboard showing it for some reason? Flotsam. An abandoned direction that I failed to clean up. Thanks. bq. On 2012-05-17 05:41:53, Gregory Chanan wrote: bq. src/main/java/org/apache/hadoop/hbase/util/FSUtils.java, line 91 bq. https://reviews.apache.org/r/5130/diff/3/?file=109334#file109334line91 bq. bq. This is what you are using to ensure the sizes cannot be the same? Very nice! bq. bq. I cannot find a call-site for this, though, maybe I missed it? Ditto. I was going to use this to write .regioninfo and .tableinfo but then I ran into your review. Study this method which I'll purge in the next version. See how it puts serialized class at top, two '\n's, and then a toString on the class? Thats how .regioninfo files and .tableinfo files are currently written. My patch now changes it so these files only contained the serialized object... no '\n' and no toString. I do this so its very unlikely a Writable length will class w/ the length of a pb'd content (See HRegion for where we write .regioninfo and in FSTableDescriptors for where we write .tableinfo only we don't need the 'trick' here since we have actually read the file and can see if its pb'd). Thanks for the review G. - Michael --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5130/#review7951 --- On 2012-05-16 23:56:18, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/5130/ bq. --- bq. bq. (Updated 2012-05-16 23:56:18) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java bq.New class to hold clusterid in. bq. M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java bq.Make it so can do pb serialization. Deprecated Writable serialization. bq. M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java bq.Make it so methods in here follow the pattern in HCD an HTD pb 'ing. bq.Deprecated Writable serialization. bq. M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java bq.Make it so can do pb serialization. Deprecated Writable serialization. bq. M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java bq.ClusterId under ZK got renamed as ZKClusterId bq. M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java bq.Hide the Reference#Range enums. Don't let them out of this class. bq.Make it so can do pb serialization. bq. M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java bq.Use new methods on Reference for getting top and bottom. bq. M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java bq.ClusterId under zk has been renamed ZKClusterId. bq.Use new ClusterId class too. bq. M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java bq.Use new clusterid class. bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java bq.Move the RegionInfo convertion up into HRegionInfo instead of here. bq.Added generic toDelimitedByteArray helper. bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java bq.Use HRegionInfo convertions instead. bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java bq.Use HRegionInfo convertions instead. bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java bq.Use new utility writing out .regioninfo files. bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java bq.Formatting. bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java bq.Range in Reference is no longer public. bq.Range in Reference is no longer public. bq. M b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java bq. M b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java bq.ClusterId got renamed ZKClusterId bq. M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java bq.Use new
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13276269#comment-13276269 ] Hadoop QA commented on HBASE-5453: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12527528/5453v3.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 24 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1879//console This message is automatically generated. Switch on-disk formats (reference files, HFile meta fields, etc) to PB -- Key: HBASE-5453 URL: https://issues.apache.org/jira/browse/HBASE-5453 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Todd Lipcon Assignee: stack Attachments: 5453.txt, 5453v2.txt, 5453v3.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13276276#comment-13276276 ] jirapos...@reviews.apache.org commented on HBASE-5453: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5130/ --- Review request for hbase. Summary --- A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java New class to hold clusterid in. M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java Make it so can do pb serialization. Deprecated Writable serialization. M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java Make it so methods in here follow the pattern in HCD an HTD pb 'ing. Deprecated Writable serialization. M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java Make it so can do pb serialization. Deprecated Writable serialization. M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java ClusterId under ZK got renamed as ZKClusterId M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java Hide the Reference#Range enums. Don't let them out of this class. Make it so can do pb serialization. M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java Use new methods on Reference for getting top and bottom. M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java ClusterId under zk has been renamed ZKClusterId. Use new ClusterId class too. M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java Use new clusterid class. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java Move the RegionInfo convertion up into HRegionInfo instead of here. Added generic toDelimitedByteArray helper. M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java Use HRegionInfo convertions instead. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java Use HRegionInfo convertions instead. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java Use new utility writing out .regioninfo files. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java Formatting. M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java Range in Reference is no longer public. Range in Reference is no longer public. M b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java M b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java ClusterId got renamed ZKClusterId M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java Use new serialization utlity in HTD. M b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java Generic method for writing dot file content. M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java Reference#Range is not public any more M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java Deprecated getHRegionInfo, etc. D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java Rename A b/src/main/protobuf/ClusterId.proto Added file for ClusterId only since its written to fs and to zk. A b/src/main/protobuf/FS.proto Protos for fs files. M b/src/main/protobuf/ZooKeeper.proto Moved ClusterId out to own proto file M b/src/main/protobuf/hbase.proto Added TableSchema and ColumnFamilySchema This addresses bug hbase-5453. https://issues.apache.org/jira/browse/hbase-5453 Diffs - src/main/java/org/apache/hadoop/hbase/ClusterId.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 5862f15 src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 8d83ff3 src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java af89e3e src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5cac9af src/main/java/org/apache/hadoop/hbase/io/Reference.java 6360059 src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 9e4ada9 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 947ec5f src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 5052878 src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java ccc964e src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java dabfbab src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java 45cb6cf src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClusterIdProtos.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/protobuf/generated/FSProtos.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 058c006
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13276320#comment-13276320 ] jirapos...@reviews.apache.org commented on HBASE-5453: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5130/#review7916 --- Do we have a story for when we can remove the writable stuff? src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java https://reviews.apache.org/r/5130/#comment17217 Looks like we are stepping on each other's toes a bit; I defined my own protos for this and HTableDescriptor (see the review request for HBASE-5445). Your definitions are pretty close though; I'll convert mine over. src/main/java/org/apache/hadoop/hbase/HRegionInfo.java https://reviews.apache.org/r/5130/#comment17218 not be what ou want - not be what you want src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java https://reviews.apache.org/r/5130/#comment17219 Did you give any thought to storing the actual data in a TableSchema rather than converting? Perhaps it's not worth it because: 1) we still need to maintain the writable for now, so would require rewriting that part, which is a waste 2) might not perform well, because we have to call copyFrom a bunch of times. src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java https://reviews.apache.org/r/5130/#comment17220 Nice. I wrote up similar things for HTableDescriptor/HColumnDescriptor; I can just get rid of them and use your convert functions. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java https://reviews.apache.org/r/5130/#comment17226 Are we sure this assumption is valid (that if the file lengths are the same, the files are the same format)? We can't check if there is a pb-prefix at the start of file or something? src/main/java/org/apache/hadoop/hbase/util/FSUtils.java https://reviews.apache.org/r/5130/#comment17223 Same here, is this assumption valid? src/main/protobuf/FS.proto https://reviews.apache.org/r/5130/#comment17221 extra whitespace src/main/protobuf/hbase.proto https://reviews.apache.org/r/5130/#comment17222 extra whitespace - Gregory On 2012-05-15 22:14:17, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/5130/ bq. --- bq. bq. (Updated 2012-05-15 22:14:17) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java bq.New class to hold clusterid in. bq. M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java bq.Make it so can do pb serialization. Deprecated Writable serialization. bq. M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java bq.Make it so methods in here follow the pattern in HCD an HTD pb 'ing. bq.Deprecated Writable serialization. bq. M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java bq.Make it so can do pb serialization. Deprecated Writable serialization. bq. M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java bq.ClusterId under ZK got renamed as ZKClusterId bq. M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java bq.Hide the Reference#Range enums. Don't let them out of this class. bq.Make it so can do pb serialization. bq. M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java bq.Use new methods on Reference for getting top and bottom. bq. M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java bq.ClusterId under zk has been renamed ZKClusterId. bq.Use new ClusterId class too. bq. M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java bq.Use new clusterid class. bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java bq.Move the RegionInfo convertion up into HRegionInfo instead of here. bq.Added generic toDelimitedByteArray helper. bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java bq.Use HRegionInfo convertions instead. bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java bq.Use HRegionInfo convertions instead. bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java bq.Use new utility writing out .regioninfo files. bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java bq.Formatting. bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java bq.
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13276330#comment-13276330 ] jirapos...@reviews.apache.org commented on HBASE-5453: -- bq. On 2012-05-15 23:08:35, Gregory Chanan wrote: bq. Do we have a story for when we can remove the writable stuff? No. Smile. Being optimistic. Have to start somewhere. No harm marking this stuff deprecated in meantime as heads-up. I think we won't be able to migrate over these old classes. I'm thinking we'll have to do new ones and deprecate the old given they inherit from Writable. It'll be a bit of a pain though given some of these classes come out in our public API. But we have to start somewhere. bq. On 2012-05-15 23:08:35, Gregory Chanan wrote: bq. src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java, line 1017 bq. https://reviews.apache.org/r/5130/diff/1/?file=109035#file109035line1017 bq. bq. Looks like we are stepping on each other's toes a bit; I defined my own protos for this and HTableDescriptor (see the review request for HBASE-5445). Your definitions are pretty close though; I'll convert mine over. Well, Andrew beat us both to it over in his REST pb stuff. We need to reconcile his w/ ours too bq. On 2012-05-15 23:08:35, Gregory Chanan wrote: bq. src/main/java/org/apache/hadoop/hbase/HRegionInfo.java, line 954 bq. https://reviews.apache.org/r/5130/diff/1/?file=109036#file109036line954 bq. bq. not be what ou want - not be what you want Fixed bq. On 2012-05-15 23:08:35, Gregory Chanan wrote: bq. src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java, line 1240 bq. https://reviews.apache.org/r/5130/diff/1/?file=109037#file109037line1240 bq. bq. Did you give any thought to storing the actual data in a TableSchema rather than converting? Perhaps it's not worth it because: bq. 1) we still need to maintain the writable for now, so would require rewriting that part, which is a waste bq. 2) might not perform well, because we have to call copyFrom a bunch of times. Yeah. Thought about it but we want Writable still converting old serializations. Mostly I reasoned that these classes are rarely, relatively serialized, and performance is not important when we do want to pass these classes serialized. Yeah, it'd be a pain converting. bq. On 2012-05-15 23:08:35, Gregory Chanan wrote: bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 748 bq. https://reviews.apache.org/r/5130/diff/1/?file=109050#file109050line748 bq. bq. Are we sure this assumption is valid (that if the file lengths are the same, the files are the same format)? bq. bq. We can't check if there is a pb-prefix at the start of file or something? This is my way of figuring if I need to convert the file. There is an off chance that file could be same length though serialized with Writables. Good one. Let me fix that. - Michael --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5130/#review7916 --- On 2012-05-15 22:14:17, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/5130/ bq. --- bq. bq. (Updated 2012-05-15 22:14:17) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java bq.New class to hold clusterid in. bq. M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java bq.Make it so can do pb serialization. Deprecated Writable serialization. bq. M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java bq.Make it so methods in here follow the pattern in HCD an HTD pb 'ing. bq.Deprecated Writable serialization. bq. M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java bq.Make it so can do pb serialization. Deprecated Writable serialization. bq. M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java bq.ClusterId under ZK got renamed as ZKClusterId bq. M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java bq.Hide the Reference#Range enums. Don't let them out of this class. bq.Make it so can do pb serialization. bq. M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java bq.Use new methods on Reference for getting top and bottom. bq. M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java bq.ClusterId under zk has been renamed ZKClusterId. bq.Use new ClusterId class too. bq. M
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13274159#comment-13274159 ] Jimmy Xiang commented on HBASE-5453: v2 looks good to me. Switch on-disk formats (reference files, HFile meta fields, etc) to PB -- Key: HBASE-5453 URL: https://issues.apache.org/jira/browse/HBASE-5453 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Todd Lipcon Assignee: stack Attachments: 5453.txt, 5453v2.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272503#comment-13272503 ] Jimmy Xiang commented on HBASE-5453: Good stuff. Are the proto file and the generated java file in a different patch? Switch on-disk formats (reference files, HFile meta fields, etc) to PB -- Key: HBASE-5453 URL: https://issues.apache.org/jira/browse/HBASE-5453 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Todd Lipcon Assignee: stack Attachments: 5453.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272539#comment-13272539 ] Jimmy Xiang commented on HBASE-5453: That would be great, I think. Thanks. Switch on-disk formats (reference files, HFile meta fields, etc) to PB -- Key: HBASE-5453 URL: https://issues.apache.org/jira/browse/HBASE-5453 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Todd Lipcon Assignee: stack Attachments: 5453.txt, 5453v2.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272549#comment-13272549 ] Hadoop QA commented on HBASE-5453: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526378/5453v2.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestColumnSeeking Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1834//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1834//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1834//console This message is automatically generated. Switch on-disk formats (reference files, HFile meta fields, etc) to PB -- Key: HBASE-5453 URL: https://issues.apache.org/jira/browse/HBASE-5453 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Todd Lipcon Assignee: stack Attachments: 5453.txt, 5453v2.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272718#comment-13272718 ] Todd Lipcon commented on HBASE-5453: Agree it would be great if HFile used protobuf where possible. But I don't think it needs to block 0.96, since we already have decent compatibility paths here, and the writables used are already fairly generic (mostly maps if I remember correctly). Switch on-disk formats (reference files, HFile meta fields, etc) to PB -- Key: HBASE-5453 URL: https://issues.apache.org/jira/browse/HBASE-5453 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Todd Lipcon Assignee: stack Attachments: 5453.txt, 5453v2.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB
[
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13271855#comment-13271855
]
stack commented on HBASE-5453:
--
Files to move over:
{code}
/${HBASE_ROOTDIR}/hbase.version
/${HBASE_ROOTDIR}/hbase.id
/${HBASE_ROOTDIR}/TABLENAME/.tableinfo
/${HBASE_ROOTDIR}/TABLENAME/REGIONNAME/.regioninfo
/${HBASE_ROOTDIR}/TABLENAME/REGIONNAME/COLUMN_FAMILY/reference
{code}
Let me see too where else we write; e.g. into hfiles. Is it necessary
bringing these over? They are versioned already w/ readers than can do v1, v2,
etc.
Let me see if I can do it in a portable way.
Switch on-disk formats (reference files, HFile meta fields, etc) to PB
--
Key: HBASE-5453
URL: https://issues.apache.org/jira/browse/HBASE-5453
Project: HBase
Issue Type: Sub-task
Components: ipc, master, migration, regionserver
Reporter: Todd Lipcon
Assignee: stack
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13271859#comment-13271859 ] Elliott Clark commented on HBASE-5453: -- Are you talking about KeyValue as well ? Switch on-disk formats (reference files, HFile meta fields, etc) to PB -- Key: HBASE-5453 URL: https://issues.apache.org/jira/browse/HBASE-5453 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Todd Lipcon Assignee: stack -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13271864#comment-13271864 ] stack commented on HBASE-5453: -- no to kv Switch on-disk formats (reference files, HFile meta fields, etc) to PB -- Key: HBASE-5453 URL: https://issues.apache.org/jira/browse/HBASE-5453 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Todd Lipcon Assignee: stack -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
