Ok. Thanks Adam for clarifying.
Regards,
David
On Fri, Jun 18, 2021 at 6:42 PM Adam Cozzette <acozze...@google.com> wrote:
> Actually protobuf messages must be strictly less than 2 GiB in size. It
> looks like you're using protobuf-c and perhaps that implementation will
> allow somewhat larger sizes, but the implementations maintained by Google
> enforce a 2 GiB limit. There is no real workaround except to break up the
> data into smaller pieces.
>
> On Fri, Jun 11, 2021 at 8:05 AM David Yat Sin <jd.yat...@gmail.com> wrote:
>
>> Hi,
>> I am trying to store large buffers that can potentially grow to 32 GB,
>> but when using the bytes type, I see the unpack fails when the size is just
>> below 4GB.
>>
>> Is that a known limitation? Would you have a recommendation on how to
>> pack buffers > 32 GB.
>>
>> proto file:
>> -------------------------------
>> syntax = "proto2";
>>
>> message msg_test {
>> required bytes buf = 1;
>> }
>> -------------------------------
>>
>> C code:
>> -------------------------------
>> //#define PROTOBUF_BYTES_SIZE ((size_t)(0xFFFFFFFF)) //Fails
>> #define PROTOBUF_BYTES_SIZE ((size_t)(0xFFFFFFFF-5)) //Succeeds
>>
>>
>> int main(int argc, char** argv)
>> {
>> int ret = 0;
>> MsgTest *msg, *msg2;
>> size_t len;
>> uint8_t *buf;
>>
>> msg = malloc(sizeof(*msg));
>> if (!msg) {
>> printf("Failed to allocate proto structure\n");
>> return -ENOMEM;
>> }
>>
>> msg_test__init(msg);
>> msg->buf.data = malloc(PROTOBUF_BYTES_SIZE);
>> msg->buf.len = PROTOBUF_BYTES_SIZE;
>>
>> len = msg_test__get_packed_size(msg);
>>
>> buf = malloc(len);
>> if (!buf) {
>> printf("Failed to allocate memory (len:0x%lx)\n", len);
>> return -ENOMEM;
>> }
>>
>> msg_test__pack(msg, buf);
>>
>> //msg2 = msg_test__unpack(&my_allocator, len, buf);
>> msg2 = msg_test__unpack(NULL, len, buf);
>> if (msg2) {
>> printf("Unpack successful\n");
>> msg_test__free_unpacked(msg2, NULL);
>> } else {
>> printf("Unpack failed\n");
>> }
>> return ret;
>> }
>> -------------------------------
>>
>>
>> Thanks,
>> David
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/protobuf/1a5543f3-9dc7-4046-af60-954595859967n%40googlegroups.com
>> <https://groups.google.com/d/msgid/protobuf/1a5543f3-9dc7-4046-af60-954595859967n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/protobuf/CAMPHVD-1w-cNdxmpwc-%3Daz%2BQ6RosHVi0h87T3zhKofT6qD7CvA%40mail.gmail.com.
