Ok. Thanks Adam for clarifying. Regards, David
On Fri, Jun 18, 2021 at 6:42 PM Adam Cozzette <[email protected]> wrote: > Actually protobuf messages must be strictly less than 2 GiB in size. It > looks like you're using protobuf-c and perhaps that implementation will > allow somewhat larger sizes, but the implementations maintained by Google > enforce a 2 GiB limit. There is no real workaround except to break up the > data into smaller pieces. > > On Fri, Jun 11, 2021 at 8:05 AM David Yat Sin <[email protected]> wrote: > >> Hi, >> I am trying to store large buffers that can potentially grow to 32 GB, >> but when using the bytes type, I see the unpack fails when the size is just >> below 4GB. >> >> Is that a known limitation? Would you have a recommendation on how to >> pack buffers > 32 GB. >> >> proto file: >> ------------------------------- >> syntax = "proto2"; >> >> message msg_test { >> required bytes buf = 1; >> } >> ------------------------------- >> >> C code: >> ------------------------------- >> //#define PROTOBUF_BYTES_SIZE ((size_t)(0xFFFFFFFF)) //Fails >> #define PROTOBUF_BYTES_SIZE ((size_t)(0xFFFFFFFF-5)) //Succeeds >> >> >> int main(int argc, char** argv) >> { >> int ret = 0; >> MsgTest *msg, *msg2; >> size_t len; >> uint8_t *buf; >> >> msg = malloc(sizeof(*msg)); >> if (!msg) { >> printf("Failed to allocate proto structure\n"); >> return -ENOMEM; >> } >> >> msg_test__init(msg); >> msg->buf.data = malloc(PROTOBUF_BYTES_SIZE); >> msg->buf.len = PROTOBUF_BYTES_SIZE; >> >> len = msg_test__get_packed_size(msg); >> >> buf = malloc(len); >> if (!buf) { >> printf("Failed to allocate memory (len:0x%lx)\n", len); >> return -ENOMEM; >> } >> >> msg_test__pack(msg, buf); >> >> //msg2 = msg_test__unpack(&my_allocator, len, buf); >> msg2 = msg_test__unpack(NULL, len, buf); >> if (msg2) { >> printf("Unpack successful\n"); >> msg_test__free_unpacked(msg2, NULL); >> } else { >> printf("Unpack failed\n"); >> } >> return ret; >> } >> ------------------------------- >> >> >> Thanks, >> David >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Protocol Buffers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/protobuf/1a5543f3-9dc7-4046-af60-954595859967n%40googlegroups.com >> <https://groups.google.com/d/msgid/protobuf/1a5543f3-9dc7-4046-af60-954595859967n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/CAMPHVD-1w-cNdxmpwc-%3Daz%2BQ6RosHVi0h87T3zhKofT6qD7CvA%40mail.gmail.com.
