Re: Serializing Large Collections: SerializePartial ParsePartial
I think the idea is to break up very large data sets into smaller packets so they can be 'streamed'. When I think of something like seismic data, stream based event handling makes the most sense. Can the data points be processed individually somehow, or do you need access to all of them (in memory) at once? -bmadigan On Oct 22, 6:54 pm, [EMAIL PROTECTED] wrote: > OK, that makes sense. Thanks for the quick reply. > > I work at a seismic earthquake data center. We're looking at using > protocol buffers as a means of internally moving around processed > chunks of data. Seems to work pretty well, as long as the chunks > aren't too large > (which is a problem one way or another). But working with ~5 million > data points > doesn't seem to be any problem. > > "some other container format." Not exactly sure what that would > look like. > > Thanks again. > -B > > On Oct 22, 3:26 pm, "Kenton Varda" <[EMAIL PROTECTED]> wrote: > > > The "Partial" serialize and parse routines actually do something completely > > unrelated: they allow the message to be missing required fields. So, that > > doesn't help you. > > I'm afraid protocol buffers are not designed for storing very large > > collections in a single message. Instead, you should be thinking about > > representing each item in a collection using a protocol message, but then > > using some other container format. Using protocol buffers here makes the > > container format simpler because it only needs to deal with raw strings > > rather than having to worry about structured data. > > > On Wed, Oct 22, 2008 at 10:19 AM, <[EMAIL PROTECTED]> wrote: > > > > Is there a general strategy/methodology for dealing with very large > > > collections so that they do not need to be > > > completely held in memory before serializing and de-serializing? I > > > see, for example, SerializePartialToOstream() > > > and ParsePartialFromIstream() but no documentation of how to use it. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: null values should be treated as no value
I agree! Fail fast or don't fail. I wonder if there is a reason for
not failing in the builder.
Maybe fail fast behavior should be a compiler option.
On Nov 13, 9:39 am, bivas <[EMAIL PROTECTED]> wrote:
> Hi,
> I'm using the Java output feature of protobuf for my app.
> I found that protobuf doesn't handle null values as well as i would
> expect it.
>
> for example:
>
> message MyMessage {
>required int32 id = 1;
>optional string code = 2;
>
> }
>
> in java:
>
> MyMessage message = MyMessage.newBuilder().setId(3).setCode(null).build
> ();
> message.writeTo(System.out); // <--- this will result a
> NullPointerException when trying to write the code field
>
> the reason is that in the created java source the setCode() method
> looks like this:
>
> public Builder setCode(java.lang.String value) {
> result.hasCode = true; // the field flag indicate a value
> while there isn't!
> result.code_ = value;
> return this;
>
> }
>
> method should look like:
>
> // for optional fields
> public Builder setCode(java.lang.String value) {
> result.hasCode = (value != null) ;
> result.code_ = value;
> return this;
>
> }
>
> // for required fields
> public Builder setCode(java.lang.String value) {
> assert value != null;
> result.hasCode = true;
> result.code_ = value;
> return this;
>
> }
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
proto file naming and outer class collisions
I think I'm missing somthing in the style guidelines. If I have a proto file called "trip_summary.proto", which contains a message "TripSummary", protoc throws the following error: -protoc-single: [exec] --java_out: trip_summary.proto: Cannot generate Java output because the file's outer class name, "TripSummary", matches the name of one of the types declared inside it. Please either rename the type or use the java_outer_classname option to specify a different outer class name for the .proto file So, renaming the file to 'trip_summary_protos.proto', protoc generates 2 Java files, one called TripSummaryProtos.java, containing a static TripSummaryProtos, which contains an inner TripSummary class. Setting the option 'java_multiple_files=true', I get 2 java files, with the TripSummary class referencing all of the static members of TripSummaryProtos. This doesn't cause any problems with using TripSummary class (that I know of), but why is the container class necessary? Should I just define all of my messages in one file and ignore the outer class in my code? -bmadigan --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: proto file naming and outer class collisions
I guess it's a good idea to treat the .proto file as a package in java terms. On Nov 13, 12:04 pm, bmadigan <[EMAIL PROTECTED]> wrote: > I think I'm missing somthing in the style guidelines. If I have a > proto file called "trip_summary.proto", which contains a message > "TripSummary", protoc throws the following error: > > -protoc-single: > [exec] --java_out: trip_summary.proto: Cannot generate Java > output because the file's outer class name, "TripSummary", matches the > name of one of the types declared inside it. Please either rename the > type or use the java_outer_classname option to specify a different > outer class name for the .proto file > > So, renaming the file to 'trip_summary_protos.proto', protoc generates > 2 Java files, one called TripSummaryProtos.java, containing a static > TripSummaryProtos, which contains an inner TripSummary class. > Setting the option 'java_multiple_files=true', I get 2 java files, > with the TripSummary class referencing all of the static members of > TripSummaryProtos. This doesn't cause any problems with using > TripSummary class (that I know of), but why is the container class > necessary? > Should I just define all of my messages in one file and ignore the > outer class in my code? > -bmadigan --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: proto file naming and outer class collisions
Thanks. No need to change anything. If it aint broke, don't fix it! Once I got the extensions mechanism, it all made sense. -b On Nov 13, 1:41 pm, Kenton Varda <[EMAIL PROTECTED]> wrote: > BTW, you didn't have to rename your .proto file. You could have just used > the java_outer_classname to make it produce a different class name. > > On Thu, Nov 13, 2008 at 11:40 AM, Kenton Varda <[EMAIL PROTECTED]> wrote: > > You're using the java_multiple_files option, I take it? > > The outer class is still necessary to store the file's descriptor and any > > top-level extension definitions. I suppose the descriptor could be moved > > into the first class defined in the file but it will take some work and > > seems like a low priority. I don't think it ever makes sense to put the > > extensions inside some other class but not all protos have top-level > > extensions. > > > On Thu, Nov 13, 2008 at 10:04 AM, bmadigan <[EMAIL PROTECTED]> wrote: > > >> I think I'm missing somthing in the style guidelines. If I have a > >> proto file called "trip_summary.proto", which contains a message > >> "TripSummary", protoc throws the following error: > > >> -protoc-single: > >> [exec] --java_out: trip_summary.proto: Cannot generate Java > >> output because the file's outer class name, "TripSummary", matches the > >> name of one of the types declared inside it. Please either rename the > >> type or use the java_outer_classname option to specify a different > >> outer class name for the .proto file > > >> So, renaming the file to 'trip_summary_protos.proto', protoc generates > >> 2 Java files, one called TripSummaryProtos.java, containing a static > >> TripSummaryProtos, which contains an inner TripSummary class. > >> Setting the option 'java_multiple_files=true', I get 2 java files, > >> with the TripSummary class referencing all of the static members of > >> TripSummaryProtos. This doesn't cause any problems with using > >> TripSummary class (that I know of), but why is the container class > >> necessary? > >> Should I just define all of my messages in one file and ignore the > >> outer class in my code? > >> -bmadigan --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
