Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15660 )
Change subject: wire_protocol: change columnar serialization of varlen data to match Arrow ...................................................................... wire_protocol: change columnar serialization of varlen data to match Arrow This changes the format of variable-length columns serialized on the wire to match Apache Arrow instead of our internal column format. The Arrow format consists of an array of n+1 offsets for n rows, such that the data for cell 'n' spans offset offsets[n]...offsets[n+1]. The obvious advantage here is that clients can zero-copy into Arrow structures since the format is compatible. The less obvious advantage is that we are going from 16 bytes (sizeof(Slice)) to 4 bytes (sizeof(uint32_t offset)) for each serialized string, so this should be a savings even for non-arrow users of the API. This patch also adds some more sanity checking of the wire format in the client API so that it's not the responsibility of the caller to guard against malicious servers. Change-Id: Iadf728744feb83f5980e62bea4fd7634a1a52467 Reviewed-on: http://gerrit.cloudera.org:8080/15660 Tested-by: Kudu Jenkins Reviewed-by: Andrew Wong <aw...@cloudera.com> --- M src/kudu/client/client-test.cc M src/kudu/client/columnar_scan_batch.cc M src/kudu/client/columnar_scan_batch.h M src/kudu/client/scanner-internal.cc M src/kudu/client/scanner-internal.h M src/kudu/common/columnar_serialization.cc M src/kudu/common/columnar_serialization.h M src/kudu/common/rowblock.h M src/kudu/common/wire_protocol-test.cc M src/kudu/common/wire_protocol.proto M src/kudu/tserver/tablet_server-test.cc M src/kudu/tserver/tablet_service.cc 12 files changed, 256 insertions(+), 121 deletions(-) Approvals: Kudu Jenkins: Verified Andrew Wong: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/15660 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iadf728744feb83f5980e62bea4fd7634a1a52467 Gerrit-Change-Number: 15660 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon <t...@apache.org> Gerrit-Reviewer: Alexey Serbin <aser...@cloudera.com> Gerrit-Reviewer: Andrew Wong <andrew.w...@cloudera.com> Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com> Gerrit-Reviewer: Grant Henke <granthe...@apache.org> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Todd Lipcon <t...@apache.org>
