[protobuf] Re: Serialization of primitive types
Hi Jason, that's what I was looking for. Thanks for the info! Gabor On Jun 20, 6:22 pm, Jason Hsueh wrote: > I didn't really go through the whole thread, so I might have missed > something, but it's not clear to me how you plan to distinguish different > values in the data stream. But protobuf's serialization primitives are in > CodedInputStream/CodedOutputStream. > > > > > > > > On Mon, Jun 20, 2011 at 8:49 AM, gabor.dicso wrote: > > Hello, thanks for all the responses. Sorry if my goals were not > > perfectly clear. To put it simple, what I am looking for is > > essentially something like "byte[] intBytes = > > ProtoBuf.intToBytes(100);" / "int i = > > ProtoBuf.readInt(bytesOfSerializedObject);" so that the values are > > serialized and deserialized platform-independently. Such functionality > > is obviously present in the ProtoBuf codebase. It would be great if it > > was made public so I could use it in my code. To save CPU time and > > memory, I want to avoid wrapping primitive types or any similar > > workarounds, it's going to be a very frequent operation. > > > The reason why I need all this is that I must map the information > > stored in the fields of my data classes to a list of special data > > classes from third-party code. It's not a simple field-by-field > > mapping, e.g. two fields may need to be mapped to the same third-party > > class serialized after each other. This mapping is where I require > > cross-platform binary serialization. All these operations must be made > > by generated code based on annotations in my data classes. What at > > first may seem like over-complicating things is actually my effort to > > make things as simple and effective as possible. :) > > > As for now, I finally decided to go with Thrift's binary > > serialization. But if it ever becomes possible to use ProtoBuf's > > serialization functionality in a similar way, I'd definitely like to > > hear about it. :) > > > Gabor > > > On Jun 16, 8:29 pm, Christopher Smith wrote: > > > I think Gabor wants to avoid the overhead of implementing all that > > > additional bookkeeping as it'd slow down development. Something that > > > would effectively generate a protobuf descriptor so that it'd stay > > > consistent with changes in the Java code. > > > > I would suggest looking at the protostuff project: > > > >http://code.google.com/p/protostuff/ > > > > I think it has all that is needed to achieve the goals Gabor is looking > > for. > > > > --Chris > > > > 2011/6/16 Miguel Muñoz : > > > > > I agree with Marc. When things get complicated, it's a good idea to > > > > separate your tasks. It seems like your java class, which generates > > > > some of the data based on other data, is one issue, and your > > > > serialization is a separate issue. (I know it would be nice to just > > > > make that class serializable, but that may be where you make things > > > > complicated.) > > > > > When I want to serialize my classes with protobufs, I create a > > > > separate protobuf object to just handle serialization. Then I create a > > > > utility class that transfers data between my protobuf object and my > > > > java class. Then it's easy to add a constructor to my java class that > > > > takes a protobuf object and defers the work to the utility class. > > > > > When I transfer data using protobufs, I don't convert to the protobuf > > > > format until the last possible moment before sending, and I > > > > immediately convert to the java class on receiving data. That lets me > > > > put my protobuf objects behind a facade, so I don't need to know the > > > > serialization details. > > > > > -- Miguel Muñoz > > > > > On Jun 15, 7:07 am, "gabor.dicso" wrote: > > > >> Hi all, > > > > >> I would like to be able to serialize primitive types platform- > > > >> independently. I have hand-written Java data classes and I want to > > > >> serialize their primitive fields using a cross-platform framework. > > > >> These classes can not be generated, they must be written by hand, > > > >> additional code is generated based upon them. Also, serializing the > > > >> object as a whole isn't an option either, because the fields sometimes > > > >> have to be processed before serializing their values. I have to > > > >> serialize the fields separately. It must be made cross-platform > > > >> because the values will be stored in a database and they may be read > > > >> from other platforms. Creating wrapper PB-objects for each primitive > > > >> type is an overhead I must avoid because the operation will be done > > > >> very frequently and with large amounts of data. > > > > >> I found that Protocol Buffers addresses cross-platform serialization > > > >> of objects, but I could not figure out how to use it as a > > > >> serialization framework for primitive types (without having > > > >> created .proto descriptors). Is it possible to use PB as a cross- > > > >> platform serializer-deserializer framework for primitive types? > > > >> Thanks, >
Re: [protobuf] Re: Serialization of primitive types
I didn't really go through the whole thread, so I might have missed something, but it's not clear to me how you plan to distinguish different values in the data stream. But protobuf's serialization primitives are in CodedInputStream/CodedOutputStream. On Mon, Jun 20, 2011 at 8:49 AM, gabor.dicso wrote: > Hello, thanks for all the responses. Sorry if my goals were not > perfectly clear. To put it simple, what I am looking for is > essentially something like "byte[] intBytes = > ProtoBuf.intToBytes(100);" / "int i = > ProtoBuf.readInt(bytesOfSerializedObject);" so that the values are > serialized and deserialized platform-independently. Such functionality > is obviously present in the ProtoBuf codebase. It would be great if it > was made public so I could use it in my code. To save CPU time and > memory, I want to avoid wrapping primitive types or any similar > workarounds, it's going to be a very frequent operation. > > The reason why I need all this is that I must map the information > stored in the fields of my data classes to a list of special data > classes from third-party code. It's not a simple field-by-field > mapping, e.g. two fields may need to be mapped to the same third-party > class serialized after each other. This mapping is where I require > cross-platform binary serialization. All these operations must be made > by generated code based on annotations in my data classes. What at > first may seem like over-complicating things is actually my effort to > make things as simple and effective as possible. :) > > As for now, I finally decided to go with Thrift's binary > serialization. But if it ever becomes possible to use ProtoBuf's > serialization functionality in a similar way, I'd definitely like to > hear about it. :) > > Gabor > > On Jun 16, 8:29 pm, Christopher Smith wrote: > > I think Gabor wants to avoid the overhead of implementing all that > > additional bookkeeping as it'd slow down development. Something that > > would effectively generate a protobuf descriptor so that it'd stay > > consistent with changes in the Java code. > > > > I would suggest looking at the protostuff project: > > > > http://code.google.com/p/protostuff/ > > > > I think it has all that is needed to achieve the goals Gabor is looking > for. > > > > --Chris > > > > 2011/6/16 Miguel Muñoz : > > > > > > > > > > > > > > > > > > > > > I agree with Marc. When things get complicated, it's a good idea to > > > separate your tasks. It seems like your java class, which generates > > > some of the data based on other data, is one issue, and your > > > serialization is a separate issue. (I know it would be nice to just > > > make that class serializable, but that may be where you make things > > > complicated.) > > > > > When I want to serialize my classes with protobufs, I create a > > > separate protobuf object to just handle serialization. Then I create a > > > utility class that transfers data between my protobuf object and my > > > java class. Then it's easy to add a constructor to my java class that > > > takes a protobuf object and defers the work to the utility class. > > > > > When I transfer data using protobufs, I don't convert to the protobuf > > > format until the last possible moment before sending, and I > > > immediately convert to the java class on receiving data. That lets me > > > put my protobuf objects behind a facade, so I don't need to know the > > > serialization details. > > > > > -- Miguel Muñoz > > > > > On Jun 15, 7:07 am, "gabor.dicso" wrote: > > >> Hi all, > > > > >> I would like to be able to serialize primitive types platform- > > >> independently. I have hand-written Java data classes and I want to > > >> serialize their primitive fields using a cross-platform framework. > > >> These classes can not be generated, they must be written by hand, > > >> additional code is generated based upon them. Also, serializing the > > >> object as a whole isn't an option either, because the fields sometimes > > >> have to be processed before serializing their values. I have to > > >> serialize the fields separately. It must be made cross-platform > > >> because the values will be stored in a database and they may be read > > >> from other platforms. Creating wrapper PB-objects for each primitive > > >> type is an overhead I must avoid because the operation will be done > > >> very frequently and with large amounts of data. > > > > >> I found that Protocol Buffers addresses cross-platform serialization > > >> of objects, but I could not figure out how to use it as a > > >> serialization framework for primitive types (without having > > >> created .proto descriptors). Is it possible to use PB as a cross- > > >> platform serializer-deserializer framework for primitive types? > > >> Thanks, > > > > >> Gabor Dicso > > > > > -- > > > You received this message because you are subscribed to the Google > Groups "Protocol Buffers" group. > > > To post to this group, send email to protobuf@googlegroups.com. > > >
[protobuf] Re: Serialization of primitive types
Hello, thanks for all the responses. Sorry if my goals were not perfectly clear. To put it simple, what I am looking for is essentially something like "byte[] intBytes = ProtoBuf.intToBytes(100);" / "int i = ProtoBuf.readInt(bytesOfSerializedObject);" so that the values are serialized and deserialized platform-independently. Such functionality is obviously present in the ProtoBuf codebase. It would be great if it was made public so I could use it in my code. To save CPU time and memory, I want to avoid wrapping primitive types or any similar workarounds, it's going to be a very frequent operation. The reason why I need all this is that I must map the information stored in the fields of my data classes to a list of special data classes from third-party code. It's not a simple field-by-field mapping, e.g. two fields may need to be mapped to the same third-party class serialized after each other. This mapping is where I require cross-platform binary serialization. All these operations must be made by generated code based on annotations in my data classes. What at first may seem like over-complicating things is actually my effort to make things as simple and effective as possible. :) As for now, I finally decided to go with Thrift's binary serialization. But if it ever becomes possible to use ProtoBuf's serialization functionality in a similar way, I'd definitely like to hear about it. :) Gabor On Jun 16, 8:29 pm, Christopher Smith wrote: > I think Gabor wants to avoid the overhead of implementing all that > additional bookkeeping as it'd slow down development. Something that > would effectively generate a protobuf descriptor so that it'd stay > consistent with changes in the Java code. > > I would suggest looking at the protostuff project: > > http://code.google.com/p/protostuff/ > > I think it has all that is needed to achieve the goals Gabor is looking for. > > --Chris > > 2011/6/16 Miguel Muñoz : > > > > > > > > > > > I agree with Marc. When things get complicated, it's a good idea to > > separate your tasks. It seems like your java class, which generates > > some of the data based on other data, is one issue, and your > > serialization is a separate issue. (I know it would be nice to just > > make that class serializable, but that may be where you make things > > complicated.) > > > When I want to serialize my classes with protobufs, I create a > > separate protobuf object to just handle serialization. Then I create a > > utility class that transfers data between my protobuf object and my > > java class. Then it's easy to add a constructor to my java class that > > takes a protobuf object and defers the work to the utility class. > > > When I transfer data using protobufs, I don't convert to the protobuf > > format until the last possible moment before sending, and I > > immediately convert to the java class on receiving data. That lets me > > put my protobuf objects behind a facade, so I don't need to know the > > serialization details. > > > -- Miguel Muñoz > > > On Jun 15, 7:07 am, "gabor.dicso" wrote: > >> Hi all, > > >> I would like to be able to serialize primitive types platform- > >> independently. I have hand-written Java data classes and I want to > >> serialize their primitive fields using a cross-platform framework. > >> These classes can not be generated, they must be written by hand, > >> additional code is generated based upon them. Also, serializing the > >> object as a whole isn't an option either, because the fields sometimes > >> have to be processed before serializing their values. I have to > >> serialize the fields separately. It must be made cross-platform > >> because the values will be stored in a database and they may be read > >> from other platforms. Creating wrapper PB-objects for each primitive > >> type is an overhead I must avoid because the operation will be done > >> very frequently and with large amounts of data. > > >> I found that Protocol Buffers addresses cross-platform serialization > >> of objects, but I could not figure out how to use it as a > >> serialization framework for primitive types (without having > >> created .proto descriptors). Is it possible to use PB as a cross- > >> platform serializer-deserializer framework for primitive types? > >> Thanks, > > >> Gabor Dicso > > > -- > > You received this message because you are subscribed to the Google Groups > > "Protocol Buffers" group. > > To post to this group, send email to protobuf@googlegroups.com. > > To unsubscribe from this group, send email to > > protobuf+unsubscr...@googlegroups.com. > > For more options, visit this group > > athttp://groups.google.com/group/protobuf?hl=en. > > -- > Chris -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups
Re: [protobuf] Re: Serialization of primitive types
I think Gabor wants to avoid the overhead of implementing all that additional bookkeeping as it'd slow down development. Something that would effectively generate a protobuf descriptor so that it'd stay consistent with changes in the Java code. I would suggest looking at the protostuff project: http://code.google.com/p/protostuff/ I think it has all that is needed to achieve the goals Gabor is looking for. --Chris 2011/6/16 Miguel Muñoz : > I agree with Marc. When things get complicated, it's a good idea to > separate your tasks. It seems like your java class, which generates > some of the data based on other data, is one issue, and your > serialization is a separate issue. (I know it would be nice to just > make that class serializable, but that may be where you make things > complicated.) > > When I want to serialize my classes with protobufs, I create a > separate protobuf object to just handle serialization. Then I create a > utility class that transfers data between my protobuf object and my > java class. Then it's easy to add a constructor to my java class that > takes a protobuf object and defers the work to the utility class. > > When I transfer data using protobufs, I don't convert to the protobuf > format until the last possible moment before sending, and I > immediately convert to the java class on receiving data. That lets me > put my protobuf objects behind a facade, so I don't need to know the > serialization details. > > -- Miguel Muñoz > > > On Jun 15, 7:07 am, "gabor.dicso" wrote: >> Hi all, >> >> I would like to be able to serialize primitive types platform- >> independently. I have hand-written Java data classes and I want to >> serialize their primitive fields using a cross-platform framework. >> These classes can not be generated, they must be written by hand, >> additional code is generated based upon them. Also, serializing the >> object as a whole isn't an option either, because the fields sometimes >> have to be processed before serializing their values. I have to >> serialize the fields separately. It must be made cross-platform >> because the values will be stored in a database and they may be read >> from other platforms. Creating wrapper PB-objects for each primitive >> type is an overhead I must avoid because the operation will be done >> very frequently and with large amounts of data. >> >> I found that Protocol Buffers addresses cross-platform serialization >> of objects, but I could not figure out how to use it as a >> serialization framework for primitive types (without having >> created .proto descriptors). Is it possible to use PB as a cross- >> platform serializer-deserializer framework for primitive types? >> Thanks, >> >> Gabor Dicso > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To post to this group, send email to protobuf@googlegroups.com. > To unsubscribe from this group, send email to > protobuf+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/protobuf?hl=en. > > -- Chris -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: Serialization of primitive types
I agree with Marc. When things get complicated, it's a good idea to separate your tasks. It seems like your java class, which generates some of the data based on other data, is one issue, and your serialization is a separate issue. (I know it would be nice to just make that class serializable, but that may be where you make things complicated.) When I want to serialize my classes with protobufs, I create a separate protobuf object to just handle serialization. Then I create a utility class that transfers data between my protobuf object and my java class. Then it's easy to add a constructor to my java class that takes a protobuf object and defers the work to the utility class. When I transfer data using protobufs, I don't convert to the protobuf format until the last possible moment before sending, and I immediately convert to the java class on receiving data. That lets me put my protobuf objects behind a facade, so I don't need to know the serialization details. -- Miguel Muñoz On Jun 15, 7:07 am, "gabor.dicso" wrote: > Hi all, > > I would like to be able to serialize primitive types platform- > independently. I have hand-written Java data classes and I want to > serialize their primitive fields using a cross-platform framework. > These classes can not be generated, they must be written by hand, > additional code is generated based upon them. Also, serializing the > object as a whole isn't an option either, because the fields sometimes > have to be processed before serializing their values. I have to > serialize the fields separately. It must be made cross-platform > because the values will be stored in a database and they may be read > from other platforms. Creating wrapper PB-objects for each primitive > type is an overhead I must avoid because the operation will be done > very frequently and with large amounts of data. > > I found that Protocol Buffers addresses cross-platform serialization > of objects, but I could not figure out how to use it as a > serialization framework for primitive types (without having > created .proto descriptors). Is it possible to use PB as a cross- > platform serializer-deserializer framework for primitive types? > Thanks, > > Gabor Dicso -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
