[jira] [Commented] (HBASE-8737) [replication] Change replication RPC to use cell blocks
[ https://issues.apache.org/jira/browse/HBASE-8737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13698235#comment-13698235 ] Chris Trezzo commented on HBASE-8737: - +1 Looks good to me! [replication] Change replication RPC to use cell blocks --- Key: HBASE-8737 URL: https://issues.apache.org/jira/browse/HBASE-8737 Project: HBase Issue Type: Improvement Components: Replication Reporter: Chris Trezzo Assignee: stack Priority: Critical Fix For: 0.98.0, 0.95.2 Attachments: 0001-HBASE-8737-replication-Change-replication-RPC-to-use.patch, 8737.txt, 8737v3.txt, 8737v4.txt, 8737v5.txt, 8737v6.txt Currently, the replication rpc that ships edits simply dumps the byte value of WAL edit key/value pairs into a protobuf message. Modify the replication rpc mechanism to use cell blocks so it can leverage encoding and compression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8737) [replication] Change replication RPC to use cell blocks
[ https://issues.apache.org/jira/browse/HBASE-8737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13698282#comment-13698282 ] Jesse Yates commented on HBASE-8737: Good on ya stack - glad to see this going in any chance for something like it for 0.94, or should I look to doing that myself ?:) [replication] Change replication RPC to use cell blocks --- Key: HBASE-8737 URL: https://issues.apache.org/jira/browse/HBASE-8737 Project: HBase Issue Type: Improvement Components: Replication Reporter: Chris Trezzo Assignee: stack Priority: Critical Fix For: 0.98.0, 0.95.2 Attachments: 0001-HBASE-8737-replication-Change-replication-RPC-to-use.patch, 8737.txt, 8737v3.txt, 8737v4.txt, 8737v5.txt, 8737v6.txt Currently, the replication rpc that ships edits simply dumps the byte value of WAL edit key/value pairs into a protobuf message. Modify the replication rpc mechanism to use cell blocks so it can leverage encoding and compression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8737) [replication] Change replication RPC to use cell blocks
[ https://issues.apache.org/jira/browse/HBASE-8737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13698294#comment-13698294 ] stack commented on HBASE-8737: -- It don't make sense in 0.94, right? Its pb dependent and requires our new cellblocky stuff -- and a jailer! [replication] Change replication RPC to use cell blocks --- Key: HBASE-8737 URL: https://issues.apache.org/jira/browse/HBASE-8737 Project: HBase Issue Type: Improvement Components: Replication Reporter: Chris Trezzo Assignee: stack Priority: Critical Fix For: 0.98.0, 0.95.2 Attachments: 0001-HBASE-8737-replication-Change-replication-RPC-to-use.patch, 8737.txt, 8737v3.txt, 8737v4.txt, 8737v5.txt, 8737v6.txt Currently, the replication rpc that ships edits simply dumps the byte value of WAL edit key/value pairs into a protobuf message. Modify the replication rpc mechanism to use cell blocks so it can leverage encoding and compression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8737) [replication] Change replication RPC to use cell blocks
[ https://issues.apache.org/jira/browse/HBASE-8737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13698299#comment-13698299 ] Hudson commented on HBASE-8737: --- Integrated in hbase-0.95 #281 (See [https://builds.apache.org/job/hbase-0.95/281/]) HBASE-8737 [replication] Change replication RPC to use cell blocks (Revision 1499119) Result = FAILURE stack : Files : * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Put.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java * /hbase/branches/0.95/hbase-client/src/test/java/org/apache/hadoop/hbase/ipc/TestIPCUtil.java * /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java * /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java * /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/io/HeapSize.java * /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/io/SizedCellScanner.java * /hbase/branches/0.95/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java * /hbase/branches/0.95/hbase-protocol/src/main/protobuf/Admin.proto * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ReplicationProtbufUtil.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReplicationSinkService.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReplicationSourceService.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/protobuf/TestReplicationProtobuf.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSmallTests.java [replication] Change replication RPC to use cell blocks --- Key: HBASE-8737 URL: https://issues.apache.org/jira/browse/HBASE-8737 Project: HBase Issue Type: Improvement Components: Replication Reporter: Chris Trezzo Assignee: stack Priority: Critical Fix For: 0.98.0, 0.95.2 Attachments: 0001-HBASE-8737-replication-Change-replication-RPC-to-use.patch, 8737.txt, 8737v3.txt, 8737v4.txt, 8737v5.txt, 8737v6.txt Currently, the replication rpc that ships edits simply dumps the byte value of WAL edit key/value pairs into a protobuf message. Modify the replication rpc mechanism to use cell blocks so it can leverage encoding and compression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8737) [replication] Change replication RPC to use cell blocks
[ https://issues.apache.org/jira/browse/HBASE-8737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13698324#comment-13698324 ] Jesse Yates commented on HBASE-8737: [~saint@gmail.com] yeah, but I want the cell/WALEdit encoding stuff in replication. Makes it a little more difficult to manage custom encoding because you have to have the codc on both sides, but doable [replication] Change replication RPC to use cell blocks --- Key: HBASE-8737 URL: https://issues.apache.org/jira/browse/HBASE-8737 Project: HBase Issue Type: Improvement Components: Replication Reporter: Chris Trezzo Assignee: stack Priority: Critical Fix For: 0.98.0, 0.95.2 Attachments: 0001-HBASE-8737-replication-Change-replication-RPC-to-use.patch, 8737.txt, 8737v3.txt, 8737v4.txt, 8737v5.txt, 8737v6.txt Currently, the replication rpc that ships edits simply dumps the byte value of WAL edit key/value pairs into a protobuf message. Modify the replication rpc mechanism to use cell blocks so it can leverage encoding and compression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8737) [replication] Change replication RPC to use cell blocks
[ https://issues.apache.org/jira/browse/HBASE-8737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13698382#comment-13698382 ] Hudson commented on HBASE-8737: --- Integrated in hbase-0.95-on-hadoop2 #157 (See [https://builds.apache.org/job/hbase-0.95-on-hadoop2/157/]) HBASE-8737 [replication] Change replication RPC to use cell blocks (Revision 1499119) Result = FAILURE stack : Files : * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Put.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java * /hbase/branches/0.95/hbase-client/src/test/java/org/apache/hadoop/hbase/ipc/TestIPCUtil.java * /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java * /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java * /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/io/HeapSize.java * /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/io/SizedCellScanner.java * /hbase/branches/0.95/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java * /hbase/branches/0.95/hbase-protocol/src/main/protobuf/Admin.proto * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ReplicationProtbufUtil.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReplicationSinkService.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReplicationSourceService.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/protobuf/TestReplicationProtobuf.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSmallTests.java [replication] Change replication RPC to use cell blocks --- Key: HBASE-8737 URL: https://issues.apache.org/jira/browse/HBASE-8737 Project: HBase Issue Type: Improvement Components: Replication Reporter: Chris Trezzo Assignee: stack Priority: Critical Fix For: 0.98.0, 0.95.2 Attachments: 0001-HBASE-8737-replication-Change-replication-RPC-to-use.patch, 8737.txt, 8737v3.txt, 8737v4.txt, 8737v5.txt, 8737v6.txt Currently, the replication rpc that ships edits simply dumps the byte value of WAL edit key/value pairs into a protobuf message. Modify the replication rpc mechanism to use cell blocks so it can leverage encoding and compression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8737) [replication] Change replication RPC to use cell blocks
[ https://issues.apache.org/jira/browse/HBASE-8737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13698477#comment-13698477 ] Hudson commented on HBASE-8737: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #593 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/593/]) HBASE-8737 [replication] Change replication RPC to use cell blocks (Revision 1499118) Result = FAILURE stack : Files : * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Put.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java * /hbase/trunk/hbase-client/src/test/java/org/apache/hadoop/hbase/ipc/TestIPCUtil.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/HeapSize.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/SizedCellScanner.java * /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java * /hbase/trunk/hbase-protocol/src/main/protobuf/Admin.proto * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ReplicationProtbufUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReplicationSinkService.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReplicationSourceService.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/protobuf/TestReplicationProtobuf.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSmallTests.java [replication] Change replication RPC to use cell blocks --- Key: HBASE-8737 URL: https://issues.apache.org/jira/browse/HBASE-8737 Project: HBase Issue Type: Improvement Components: Replication Reporter: Chris Trezzo Assignee: stack Priority: Critical Fix For: 0.98.0, 0.95.2 Attachments: 0001-HBASE-8737-replication-Change-replication-RPC-to-use.patch, 8737.txt, 8737v3.txt, 8737v4.txt, 8737v5.txt, 8737v6.txt Currently, the replication rpc that ships edits simply dumps the byte value of WAL edit key/value pairs into a protobuf message. Modify the replication rpc mechanism to use cell blocks so it can leverage encoding and compression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8737) [replication] Change replication RPC to use cell blocks
[ https://issues.apache.org/jira/browse/HBASE-8737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13698530#comment-13698530 ] Hudson commented on HBASE-8737: --- Integrated in HBase-TRUNK #4209 (See [https://builds.apache.org/job/HBase-TRUNK/4209/]) HBASE-8737 [replication] Change replication RPC to use cell blocks (Revision 1499118) Result = FAILURE stack : Files : * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Put.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java * /hbase/trunk/hbase-client/src/test/java/org/apache/hadoop/hbase/ipc/TestIPCUtil.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/HeapSize.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/SizedCellScanner.java * /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java * /hbase/trunk/hbase-protocol/src/main/protobuf/Admin.proto * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ReplicationProtbufUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReplicationSinkService.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReplicationSourceService.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/protobuf/TestReplicationProtobuf.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSmallTests.java [replication] Change replication RPC to use cell blocks --- Key: HBASE-8737 URL: https://issues.apache.org/jira/browse/HBASE-8737 Project: HBase Issue Type: Improvement Components: Replication Reporter: Chris Trezzo Assignee: stack Priority: Critical Fix For: 0.98.0, 0.95.2 Attachments: 0001-HBASE-8737-replication-Change-replication-RPC-to-use.patch, 8737.txt, 8737v3.txt, 8737v4.txt, 8737v5.txt, 8737v6.txt Currently, the replication rpc that ships edits simply dumps the byte value of WAL edit key/value pairs into a protobuf message. Modify the replication rpc mechanism to use cell blocks so it can leverage encoding and compression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8737) [replication] Change replication RPC to use cell blocks
[ https://issues.apache.org/jira/browse/HBASE-8737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13698551#comment-13698551 ] stack commented on HBASE-8737: -- bq. Stack yeah, but I want the cell/WALEdit encoding stuff in replication. Makes it a little more difficult to manage custom encoding because you have to have the codc on both sides, but doable Sketch it in new issue Jesse? [replication] Change replication RPC to use cell blocks --- Key: HBASE-8737 URL: https://issues.apache.org/jira/browse/HBASE-8737 Project: HBase Issue Type: Improvement Components: Replication Reporter: Chris Trezzo Assignee: stack Priority: Critical Fix For: 0.98.0, 0.95.2 Attachments: 0001-HBASE-8737-replication-Change-replication-RPC-to-use.patch, 8737.txt, 8737v3.txt, 8737v4.txt, 8737v5.txt, 8737v6.txt Currently, the replication rpc that ships edits simply dumps the byte value of WAL edit key/value pairs into a protobuf message. Modify the replication rpc mechanism to use cell blocks so it can leverage encoding and compression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8737) [replication] Change replication RPC to use cell blocks
[ https://issues.apache.org/jira/browse/HBASE-8737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13697253#comment-13697253 ] stack commented on HBASE-8737: -- Updated the diff on rb. [replication] Change replication RPC to use cell blocks --- Key: HBASE-8737 URL: https://issues.apache.org/jira/browse/HBASE-8737 Project: HBase Issue Type: Improvement Components: Replication Reporter: Chris Trezzo Assignee: stack Priority: Critical Fix For: 0.98.0, 0.95.2 Attachments: 0001-HBASE-8737-replication-Change-replication-RPC-to-use.patch, 8737.txt, 8737v3.txt, 8737v4.txt, 8737v5.txt Currently, the replication rpc that ships edits simply dumps the byte value of WAL edit key/value pairs into a protobuf message. Modify the replication rpc mechanism to use cell blocks so it can leverage encoding and compression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8737) [replication] Change replication RPC to use cell blocks
[ https://issues.apache.org/jira/browse/HBASE-8737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13697258#comment-13697258 ] Jesse Yates commented on HBASE-8737: I'm a little slow today... any chance we can figure out likely data sizes based on whether or not we are getting a SizedCellScanner? So, if its sized, then we should use that (agreeing with you above), but when do we not get that and can we make any assumptions about the size of the data? A 2x improvement is a pretty big jump... it might be worth doing (1) about instrumenting this a little bit so people can see if they should switch to using the counting and then (2) making the counting optional? I may be trying to generalize too much though :) [replication] Change replication RPC to use cell blocks --- Key: HBASE-8737 URL: https://issues.apache.org/jira/browse/HBASE-8737 Project: HBase Issue Type: Improvement Components: Replication Reporter: Chris Trezzo Assignee: stack Priority: Critical Fix For: 0.98.0, 0.95.2 Attachments: 0001-HBASE-8737-replication-Change-replication-RPC-to-use.patch, 8737.txt, 8737v3.txt, 8737v4.txt, 8737v5.txt Currently, the replication rpc that ships edits simply dumps the byte value of WAL edit key/value pairs into a protobuf message. Modify the replication rpc mechanism to use cell blocks so it can leverage encoding and compression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8737) [replication] Change replication RPC to use cell blocks
[ https://issues.apache.org/jira/browse/HBASE-8737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13697299#comment-13697299 ] stack commented on HBASE-8737: -- bq. ...but when do we not get that and can we make any assumptions about the size of the data? Could look at first few and come up w/ some heuristics... bq. I may be trying to generalize too much though Maybe (smile). Or rather, for another issue. This one is about getting replication up on cellblocks. As is, we'll spew if lots of resizing. Current option would be to up the default buffer size... or work on some heuristic for later I'd say? Any chance of a +1? Good on you Jesse. [replication] Change replication RPC to use cell blocks --- Key: HBASE-8737 URL: https://issues.apache.org/jira/browse/HBASE-8737 Project: HBase Issue Type: Improvement Components: Replication Reporter: Chris Trezzo Assignee: stack Priority: Critical Fix For: 0.98.0, 0.95.2 Attachments: 0001-HBASE-8737-replication-Change-replication-RPC-to-use.patch, 8737.txt, 8737v3.txt, 8737v4.txt, 8737v5.txt Currently, the replication rpc that ships edits simply dumps the byte value of WAL edit key/value pairs into a protobuf message. Modify the replication rpc mechanism to use cell blocks so it can leverage encoding and compression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8737) [replication] Change replication RPC to use cell blocks
[ https://issues.apache.org/jira/browse/HBASE-8737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13696012#comment-13696012 ] Chris Trezzo commented on HBASE-8737: - Thanks [~saint@gmail.com]! Just made a few comments in addition to the ones already made. [replication] Change replication RPC to use cell blocks --- Key: HBASE-8737 URL: https://issues.apache.org/jira/browse/HBASE-8737 Project: HBase Issue Type: Improvement Components: Replication Reporter: Chris Trezzo Assignee: stack Priority: Critical Fix For: 0.98.0, 0.95.2 Attachments: 0001-HBASE-8737-replication-Change-replication-RPC-to-use.patch, 8737.txt, 8737v3.txt, 8737v4.txt Currently, the replication rpc that ships edits simply dumps the byte value of WAL edit key/value pairs into a protobuf message. Modify the replication rpc mechanism to use cell blocks so it can leverage encoding and compression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8737) [replication] Change replication RPC to use cell blocks
[
https://issues.apache.org/jira/browse/HBASE-8737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13694542#comment-13694542
]
Hadoop QA commented on HBASE-8737:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12589843/8737v3.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplicationQueueFailoverCompressed
org.apache.hadoop.hbase.replication.TestReplicationSmallTests
org.apache.hadoop.hbase.replication.TestReplicationDisableInactivePeer
org.apache.hadoop.hbase.replication.TestReplicationQueueFailover
org.apache.hadoop.hbase.replication.TestMasterReplication
org.apache.hadoop.hbase.replication.TestMultiSlaveReplication
org.apache.hadoop.hbase.security.access.TestAccessController
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/6154//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6154//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6154//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6154//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6154//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6154//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6154//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6154//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6154//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6154//console
This message is automatically generated.
[replication] Change replication RPC to use cell blocks
---
Key: HBASE-8737
URL: https://issues.apache.org/jira/browse/HBASE-8737
Project: HBase
Issue Type: Improvement
Components: Replication
Reporter: Chris Trezzo
Assignee: stack
Priority: Critical
Fix For: 0.98.0, 0.95.2
Attachments:
0001-HBASE-8737-replication-Change-replication-RPC-to-use.patch, 8737.txt,
8737v3.txt
Currently, the replication rpc that ships edits simply dumps the byte value
of WAL edit key/value pairs into a protobuf message.
Modify the replication rpc mechanism to use cell blocks so it can leverage
encoding and compression.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8737) [replication] Change replication RPC to use cell blocks
[
https://issues.apache.org/jira/browse/HBASE-8737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13694917#comment-13694917
]
Hadoop QA commented on HBASE-8737:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12589929/8737v4.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.security.access.TestAccessController
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/6156//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6156//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6156//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6156//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6156//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6156//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6156//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6156//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6156//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6156//console
This message is automatically generated.
[replication] Change replication RPC to use cell blocks
---
Key: HBASE-8737
URL: https://issues.apache.org/jira/browse/HBASE-8737
Project: HBase
Issue Type: Improvement
Components: Replication
Reporter: Chris Trezzo
Assignee: stack
Priority: Critical
Fix For: 0.98.0, 0.95.2
Attachments:
0001-HBASE-8737-replication-Change-replication-RPC-to-use.patch, 8737.txt,
8737v3.txt, 8737v4.txt
Currently, the replication rpc that ships edits simply dumps the byte value
of WAL edit key/value pairs into a protobuf message.
Modify the replication rpc mechanism to use cell blocks so it can leverage
encoding and compression.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8737) [replication] Change replication RPC to use cell blocks
[ https://issues.apache.org/jira/browse/HBASE-8737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13694927#comment-13694927 ] stack commented on HBASE-8737: -- TestAccessController should be fixed now (a an issue apart from this patch). Any chance of a review of this critical issue? Thanks. [replication] Change replication RPC to use cell blocks --- Key: HBASE-8737 URL: https://issues.apache.org/jira/browse/HBASE-8737 Project: HBase Issue Type: Improvement Components: Replication Reporter: Chris Trezzo Assignee: stack Priority: Critical Fix For: 0.98.0, 0.95.2 Attachments: 0001-HBASE-8737-replication-Change-replication-RPC-to-use.patch, 8737.txt, 8737v3.txt, 8737v4.txt Currently, the replication rpc that ships edits simply dumps the byte value of WAL edit key/value pairs into a protobuf message. Modify the replication rpc mechanism to use cell blocks so it can leverage encoding and compression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8737) [replication] Change replication RPC to use cell blocks
[ https://issues.apache.org/jira/browse/HBASE-8737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13695085#comment-13695085 ] Jesse Yates commented on HBASE-8737: Mind throwing it up on RB? This fella is a bit big... [replication] Change replication RPC to use cell blocks --- Key: HBASE-8737 URL: https://issues.apache.org/jira/browse/HBASE-8737 Project: HBase Issue Type: Improvement Components: Replication Reporter: Chris Trezzo Assignee: stack Priority: Critical Fix For: 0.98.0, 0.95.2 Attachments: 0001-HBASE-8737-replication-Change-replication-RPC-to-use.patch, 8737.txt, 8737v3.txt, 8737v4.txt Currently, the replication rpc that ships edits simply dumps the byte value of WAL edit key/value pairs into a protobuf message. Modify the replication rpc mechanism to use cell blocks so it can leverage encoding and compression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8737) [replication] Change replication RPC to use cell blocks
[ https://issues.apache.org/jira/browse/HBASE-8737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13695101#comment-13695101 ] stack commented on HBASE-8737: -- [~jesse_yates] Pardon me. Here you go: https://reviews.apache.org/r/12142/ [replication] Change replication RPC to use cell blocks --- Key: HBASE-8737 URL: https://issues.apache.org/jira/browse/HBASE-8737 Project: HBase Issue Type: Improvement Components: Replication Reporter: Chris Trezzo Assignee: stack Priority: Critical Fix For: 0.98.0, 0.95.2 Attachments: 0001-HBASE-8737-replication-Change-replication-RPC-to-use.patch, 8737.txt, 8737v3.txt, 8737v4.txt Currently, the replication rpc that ships edits simply dumps the byte value of WAL edit key/value pairs into a protobuf message. Modify the replication rpc mechanism to use cell blocks so it can leverage encoding and compression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8737) [replication] Change replication RPC to use cell blocks
[
https://issues.apache.org/jira/browse/HBASE-8737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13694118#comment-13694118
]
Hadoop QA commented on HBASE-8737:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12589774/0001-HBASE-8737-replication-Change-replication-RPC-to-use.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestCheckTestClasses
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/6147//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6147//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6147//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6147//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6147//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6147//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6147//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6147//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6147//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6147//console
This message is automatically generated.
[replication] Change replication RPC to use cell blocks
---
Key: HBASE-8737
URL: https://issues.apache.org/jira/browse/HBASE-8737
Project: HBase
Issue Type: Improvement
Components: Replication
Reporter: Chris Trezzo
Assignee: stack
Priority: Critical
Fix For: 0.98.0, 0.95.2
Attachments:
0001-HBASE-8737-replication-Change-replication-RPC-to-use.patch, 8737.txt
Currently, the replication rpc that ships edits simply dumps the byte value
of WAL edit key/value pairs into a protobuf message.
Modify the replication rpc mechanism to use cell blocks so it can leverage
encoding and compression.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
