Hi Feng,

I was using SerializeToOstream and ParseFromCodedStream.  I had to use the
SetTotalBytesLimit, which is not required by ParseFromArray.  Does this
mean that I can use a byte field with a greater size than INT_MAX with
ParseFromArray?

Using SerializeToArray and ParseToArray, the performance has improved:
serializing is at 700ms, and deserializing went down to 561ms.  It is the
same order of magnitude, which is a lot better.

I tried with a ~2GB byte array to quickly estimate the scaling.
Fortunately it looks to be linear!  I wonder if the assignment step
(set_payload) can also be made closer to memcpy.

   - resize vector: 1050 ms
   - memcy: 560 ms
   - assigning the protobuf: ~2500 ms
   - seralizing the protobuf: ~1500 ms
   - deserializing the protobuf: ~1800 ms


It would be great to have some c++11 move semantics for this in a future
version of the library.  I think this is better than the Aliasing option
that you mention because that would require careful management of the
lifetime of the memory being aliased.

Thanks!
Mohamed

On Thu, Jun 30, 2016 at 2:57 PM, Feng Xiao <xiaof...@google.com> wrote:

>
>
> On Thu, Jun 30, 2016 at 9:00 AM, Mohamed Koubaa <mohamed.kou...@ansys.com>
> wrote:
>
>> Hello,
>>
>> I'm using the official proto3 cpp project.
>>
>> My organization is interested in using protocol buffers to exchange
>> messages between services.  We solve physics simulation problems and deal
>> with a mix of structured metadata and large amounts of numerical data (on
>> the order of 1-10GB).
>>
>> I ran some quick tests to investigate the feasibility of doing this with
>> protobuf.
>>
>> message ByteContainer {
>>   string name = 1;
>>   bytes payload = 2;
>>   string other_data = 3;
>> }
>>
>> What I found was surprising.  Here are the relative serialization speeds
>> of a bytes payload of 800 million bytes:
>>
>>    - resizing a std::vector<uint8_t> to 800,000,000: 416 ms
>>    - memcpy of an initialized char* (named buffer) of the same size into
>>    that vector: 190ms
>>    - byte_container.set_file(buffer, length): 1004ms
>>    - serializing the protobuf: 2000ms
>>    - deserializing the protobuf: 1800ms
>>
>> How did you serialize and deserialize the protobuf message? There are
> different APIs for different input/output types. For your case, I think the
> ParseFromArray() and SerializeToArray() should have comparable performance
> to memcpy.
>
>
>> I understand that protobufs are not intended for messages of this scale
>> (the documentation warns to use messages under 1MB), and that protobufs
>> must use some custom memory allocation that is optimized in a different
>> direction.
>>
>> I think that for byte messages, it is reasonable to expect performance on
>> the same order of magnitude of memcpy.  This is the case with Avro
>> (although we really really don't like the avro cpp API).
>>
>> Is this possible to fix in the proto library?  If not for the general
>> 'bytes' object, what if we add a tag like:
>>
>> bytes payload = 2; [huge]
>>
>
> There is actually already a ctype option:
> https://github.com/google/protobuf/blob/master/src/google/protobuf/descriptor.proto#L457
>
> It allows you to declare the actual C++ type used for a string/bytes
> field. For example:
>
> bytes payload = 2 [ctype = STRING_PIECE];  // StringPiece is basically
> pair<char*, size_t>.
>
> And internally we have a ParseFromArrayWithAliasing() method that will
> just make the StringPiece field point to the input buffer without copying
> anything.
>
> The other ctype = CORD uses a Cord class that will share memory when you
> copy a Cord object and do COPY-ON-WRITE.
>
> Unfortunately we haven't gotten time to include these ctype support in
> opensource protobuf but they are on our list (probably after 3.0.0 is
> released).
>
>
>>
>>
>> Thanks!
>>
>> Mohamed Koubaa
>> Software Developer
>> ANSYS Inc
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Reply via email to