Actually protobuf messages must be strictly less than 2 GiB in size. It looks like you're using protobuf-c and perhaps that implementation will allow somewhat larger sizes, but the implementations maintained by Google enforce a 2 GiB limit. There is no real workaround except to break up the data into smaller pieces.
On Fri, Jun 11, 2021 at 8:05 AM David Yat Sin <[email protected]> wrote: > Hi, > I am trying to store large buffers that can potentially grow to 32 GB, but > when using the bytes type, I see the unpack fails when the size is just > below 4GB. > > Is that a known limitation? Would you have a recommendation on how to pack > buffers > 32 GB. > > proto file: > ------------------------------- > syntax = "proto2"; > > message msg_test { > required bytes buf = 1; > } > ------------------------------- > > C code: > ------------------------------- > //#define PROTOBUF_BYTES_SIZE ((size_t)(0xFFFFFFFF)) //Fails > #define PROTOBUF_BYTES_SIZE ((size_t)(0xFFFFFFFF-5)) //Succeeds > > > int main(int argc, char** argv) > { > int ret = 0; > MsgTest *msg, *msg2; > size_t len; > uint8_t *buf; > > msg = malloc(sizeof(*msg)); > if (!msg) { > printf("Failed to allocate proto structure\n"); > return -ENOMEM; > } > > msg_test__init(msg); > msg->buf.data = malloc(PROTOBUF_BYTES_SIZE); > msg->buf.len = PROTOBUF_BYTES_SIZE; > > len = msg_test__get_packed_size(msg); > > buf = malloc(len); > if (!buf) { > printf("Failed to allocate memory (len:0x%lx)\n", len); > return -ENOMEM; > } > > msg_test__pack(msg, buf); > > //msg2 = msg_test__unpack(&my_allocator, len, buf); > msg2 = msg_test__unpack(NULL, len, buf); > if (msg2) { > printf("Unpack successful\n"); > msg_test__free_unpacked(msg2, NULL); > } else { > printf("Unpack failed\n"); > } > return ret; > } > ------------------------------- > > > Thanks, > David > > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/protobuf/1a5543f3-9dc7-4046-af60-954595859967n%40googlegroups.com > <https://groups.google.com/d/msgid/protobuf/1a5543f3-9dc7-4046-af60-954595859967n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/CADqAXr7ytQ3SSjDgdaXHRWweNqSbR85tJbEPXnTFB_RBAX2AyA%40mail.gmail.com.
