A few more thoughts: - Random data doesn't tend to compress well - try to measure the benefit of compression for your preferred message layout using data that is typical for your application. - It's always possible to add helper functions/classes to make it easier to deal with inconvenient message layouts.
-Nadav On Mon, May 13, 2019 at 2:25 PM 'Adam Cozzette' via Protocol Buffers < [email protected]> wrote: > I have not looked into size savings from compression, but your > uncompressed sizes sound right assuming about 3 bytes per int64 (those use > a variable-size integer so the size depends on the value being stored). I > think there's a tradeoff here between ease of use and serialized size, but > if it's important for your use case to keep serialized size small then this > technique sounds like one which is worth considering for sure. > > *From: *'Boaz Yaniv' via Protocol Buffers <[email protected]> > *Date: *Sun, May 12, 2019 at 6:41 AM > *To: *Protocol Buffers > > Hi, >> I read recently protocol-buffers encoding >> <https://developers.google.com/protocol-buffers/docs/encoding#packed> >> and notice a way to save space >> It is better to hold a repeated value than a repeated message of a value >> protobuf is saving data on each message (header/type/length), so saving a >> repeated message of two int64 will cost more than saving 2 repeated int64 >> (int64 as an example). >> >> I Used protobuf-java version: 3.4.0 >> Made a test to check it, with and without compression (LZ4) see results >> bellow (this is a similar case we have in production) >> >> message Head1 { >> repeated Data d1 = 1; >> } >> >> message Data { >> int64 v1 = 1; >> int64 v2 = 2; >> } >> message Head2 { >> repeated int64 v1 = 1; >> repeated int64 v2 = 2; >> } >> >> *With 400 messages of Head1 and Head2 (same random values in each >> message):* >> Message 'Head1' Uncompressed data size is: 3985 bytes >> Message 'Head1' compressed data size is: *3697* bytes >> >> Message 'Head2' Uncompressed data size is: 2391 bytes >> Message 'Head2' compressed data size is: *2402* bytes --> 35% less >> >> *Questions:* >> The problem is I am losing schema ordering on app side and I will have >> to keep lists (in Head2) syncd all the time >> >> Is this correct or I am missing something? >> >> By adding a new flag to the proto it can save lots of data in the encoded >> proto (in case its relevant) >> >> I tested also with writing to cassandra and the save is huge +40%!!!! >> >> Thoughts? >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Protocol Buffers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/protobuf. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/protobuf/69b44003-5821-4678-9ba7-18c1a7a05ee5%40googlegroups.com >> <https://groups.google.com/d/msgid/protobuf/69b44003-5821-4678-9ba7-18c1a7a05ee5%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/protobuf. > To view this discussion on the web visit > https://groups.google.com/d/msgid/protobuf/CADqAXr4tkcoOb2NYYkgBwEVD3MofC43Cq_hWr%3D8EvCJCTB-z1g%40mail.gmail.com > <https://groups.google.com/d/msgid/protobuf/CADqAXr4tkcoOb2NYYkgBwEVD3MofC43Cq_hWr%3D8EvCJCTB-z1g%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- -Nadav -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/protobuf. To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/CANZcNErTUCZjL9zaPybvUqO-9Qh%2BnuMkdKU2SknJMDGpkzhT9g%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
