Hi, I am trying to serialize messages to and from arrays using zlib compression.
Attached is a trivial example of what I'm trying to do.
The code asserts at line 64: the messages read from the buffer is empty.
My problem is: there is no indication whatsoever from protobuf that
something went wrong, either during encoding or decoding. I can only detect
something went wrong by comparing the decoded result against the original,
which is a bit of a pain.
There are variations around that theme. If I add these two lines after
t.set_a():
t.set_b( 42 );
t.set_c( 42 );
Then serialization still passes but deserialization fails. It doesn't tell
my why, though. Nothing is ever sent to the log handler.
If I increase the size of the buffer prior to encoding, then everything
goes through fine and the assertion passes. Evidently, zlib has issues with
small buffers and sometimes the compressed result ends up being bigger than
the input. That's something I can understand, but it leaves me with a bunch
of questions:
- Why am I not getting any error on serialization when trying to serialize
into a buffer that's too small?
- Why am I not getting any error on deserialization when trying to parse
from a buffer that was too small?
- What should the size of my buffer be to guarantee that serialization
will pass? Even adding 1KB to it, I still have problems on some production
cases.
Thanks,
A.
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
#include "test.pb.h"
#include <google/protobuf/io/zero_copy_stream.h>
#include <google/protobuf/io/zero_copy_stream_impl_lite.h>
#include <google/protobuf/io/gzip_stream.h>
#include <vector>
#include <stdexcept>
#include <iostream>
typedef std::vector<uint8_t> Blob;
void myLogHandler( google::protobuf::LogLevel level, const char* fileName, int line, const std::string& message )
{
std::cerr << level << "[" << fileName << ":" << line << " " << message << std::endl;
}
void compressedBlobToMessage( const Blob& b, google::protobuf::Message& proto )
{
const int s = int( b.size() );
google::protobuf::io::ArrayInputStream bStream( b.data(), s );
google::protobuf::io::GzipInputStream zipStream( &bStream, google::protobuf::io::GzipInputStream::ZLIB );
if( !proto.ParseFromZeroCopyStream( &zipStream ) ) throw std::runtime_error( "Failed to parse protobuf from compressed array" );
}
void messageToCompressedBlob( const google::protobuf::Message& message, Blob& b )
{
int s = message.ByteSize();
b.resize( s );
{
google::protobuf::io::ArrayOutputStream bStream( const_cast<unsigned char*>( b.data() ), s );
google::protobuf::io::GzipOutputStream::Options opt;
opt.format = google::protobuf::io::GzipOutputStream::ZLIB;
google::protobuf::io::GzipOutputStream zipStream( &bStream, opt );
if( !message.SerializeToZeroCopyStream( &zipStream ) ) throw std::runtime_error( "Failed to serialize protobuf to array" );
zipStream.Flush();
zipStream.Close();
assert( zipStream.ByteCount() == message.ByteSize() );
assert( bStream.ByteCount() <= s );
s = bStream.ByteCount();
}
b.resize( s );
}
int main( int argc, char** argv )
{
google::protobuf::SetLogHandler( myLogHandler );
Blob buffer;
Test t;
t.set_a( 42 );
std::cout << "t=" << t.DebugString() << std::endl;
messageToCompressedBlob( t, buffer );
Test t2;
compressedBlobToMessage( buffer, t2 );
std::cout << "t2=" << t2.DebugString() << std::endl;
assert( t.a() == t2.a() );
return 0;
}
test.proto
Description: Binary data
