[protobuf] Initial compiling with Visual Studio 2010
Hello, When I try to compile the solution found in vsprojects under Visual Studio 2010, I get multiple errors, the first of which is: Visual Studio 2010\Projects\protobuf-2.3.0\gtest\include\gtest/ internal/gtest-tuple.h(745): error C3855: 'std::tr1::tuple_element': template parameter '_Idx' is incompatible with the declaration Following errors include, but aren't limited to: 2ClCompile: 2 gtest-death-test.cc Visual Studio 2010\Projects\protobuf-2.3.0\gtest\include\gtest/ internal/gtest-tuple.h(757): error C2039: 'type' : is not a member of 'std::tr1::tuple_element' Visual Studio 2010\Projects\protobuf-2.3.0\gtest\include\gtest/ internal/gtest-tuple.h(742) : see declaration of 'std::tr1::tuple_element' Visual Studio 2010\Projects\protobuf-2.3.0\gtest\include\gtest/ internal/gtest-tuple.h(757): error C2146: syntax error : missing ',' before identifier 'type' While some files seen to be compiling without errors, other files don't compile (such as the one mentioned above). Are there any settings should have set within VS 2010 that I missed? Regards, Karthik -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: Protocol buffers and large data sets
Hi, I've been struggling to figure out just exactly how to do the many smaller messages approach. I've implemented this strategy, which is working except for a byte limit problem: http://groups.google.com/group/protobuf/browse_thread/thread/038cc4ad000b4265/95981da7e07ce197?hide_quotes=no I also raised the byte limit using SetTotalBytesLimit to maxint. I use a python program to read my data form disk and package it up into messages that are roughly 110 bytes each. Then I pipe it to a C++ program that reads messages and crunches. But, I still have a problem because the total number of bytes of all my smaller messages is greater than maxint and the C++ fails to read when it hits the limit. I like the protobuf approach to passing data, I just need to remove that limit. What can I do? Thanks, Terri On May 17, 7:00 pm, Jason Hsueh jas...@google.com wrote: There is a default byte size limit of 64MB when parsing protocol buffers - if a message is larger than that, it will fail to parse. This can be configured if you really need to parse larger messages, but it is generally not recommended. Additionally, ByteSize() returns a 32-bit integer, so there's an implicit limit on the size of data that can be serialized. You can certainly use protocol buffers in large data sets, but it's not recommended to have your entire data set be represented by a single message. Instead, see if you can break it up into smaller messages. On Mon, May 17, 2010 at 1:05 PM, sanikumbh saniku...@gmail.com wrote: I wanted to get some opinion on large data sets and protocol buffers. Protocol Buffer project page by google says that for data 1 megabytes, one should consider something different but they don’t mention what would happen if one crosses this limit. Are there any known failure modes when it comes to the large data sets? What are your observations, recommendations from your experience on this front? -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group athttp://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: Protocol buffers and large data sets
My guess is that you're using a single CodedInputStream to read all your
input, repeatedly calling message.ParseFromCodedStream(). Instead, create a
new CodedInputStream for each message. If you construct it on the stack,
there is no significant overhead to doing this:
while (true) {
CodedInputStream stream(input);
// read one message, or break if at EOF
}
On Mon, May 24, 2010 at 12:21 PM, Terri terri.k...@gmail.com wrote:
Hi,
I've been struggling to figure out just exactly how to do the many
smaller messages approach. I've implemented this strategy, which is
working except for a byte limit problem:
http://groups.google.com/group/protobuf/browse_thread/thread/038cc4ad000b4265/95981da7e07ce197?hide_quotes=no
I also raised the byte limit using SetTotalBytesLimit to maxint.
I use a python program to read my data form disk and package it up
into messages that are roughly 110 bytes each. Then I pipe it to a C++
program that reads messages and crunches. But, I still have a problem
because the total number of bytes of all my smaller messages is
greater than maxint and the C++ fails to read when it hits the limit.
I like the protobuf approach to passing data, I just need to remove
that limit.
What can I do?
Thanks,
Terri
On May 17, 7:00 pm, Jason Hsueh jas...@google.com wrote:
There is a default byte size limit of 64MB when parsing protocol buffers
-
if a message is larger than that, it will fail to parse. This can be
configured if you really need to parse larger messages, but it is
generally
not recommended. Additionally, ByteSize() returns a 32-bit integer, so
there's an implicit limit on the size of data that can be serialized.
You can certainly use protocol buffers in large data sets, but it's not
recommended to have your entire data set be represented by a single
message.
Instead, see if you can break it up into smaller messages.
On Mon, May 17, 2010 at 1:05 PM, sanikumbh saniku...@gmail.com wrote:
I wanted to get some opinion on large data sets and protocol buffers.
Protocol Buffer project page by google says that for data 1
megabytes, one should consider something different but they don’t
mention what would happen if one crosses this limit. Are there any
known failure modes when it comes to the large data sets?
What are your observations, recommendations from your experience on
this front?
--
You received this message because you are subscribed to the Google
Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
protobuf%2bunsubscr...@googlegroups.comprotobuf%252bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group athttp://
groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
[protobuf] Issue 190 in protobuf: [C++] Change TextFormat::Printer::TextGenerator class from 'private' to 'protected'
Status: New Owner: Labels: Type-Defect Priority-Medium New issue 190 by marthaler: [C++] Change TextFormat::Printer::TextGenerator class from 'private' to 'protected' http://code.google.com/p/protobuf/issues/detail?id=190 What steps will reproduce the problem? 1. Derive the TextFormat::Printer class to create additional string output formats using the Print(...), PrintField(...), PrintFieldName(...), PrintFieldValue(...) functions. 2. The TextFormat::Printer::TextGenerator class is private, so the above code will not work unless you also create another internal class to replicate the same exact functionality. 3. What is the expected output? What do you see instead? It would be nice to move the TextGenerator class to protected so that C++ child classes that derive from TextFormat::Printer can use the default TextGenerator class. Please use labels and text to provide additional information. Definitely a feature enhancement request, but it would make string output in C++ much easier (as it seems to be in Python). -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: Issue 190 in protobuf: [C++] Change TextFormat::Printer::TextGenerator class from 'private' to 'protected'
Updates: Status: WorkingAsIntended Comment #1 on issue 190 by ken...@google.com: [C++] Change TextFormat::Printer::TextGenerator class from 'private' to 'protected' http://code.google.com/p/protobuf/issues/detail?id=190 TextFormat::Printer is not intended to be subclassed. We do not want to expose private implementation details like TextGenerator because those details could change in the future (for example, I'm considering replacing it with protobuf::io::Printer). -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: Issue 190 in protobuf: [C++] Change TextFormat::Printer::TextGenerator class from 'private' to 'protected'
Comment #2 on issue 190 by marthaler: [C++] Change TextFormat::Printer::TextGenerator class from 'private' to 'protected' http://code.google.com/p/protobuf/issues/detail?id=190 I understand. I just thought you could look at it since creating different output formats in C++ is much more convoluted than in Python or Java. Python uses the cStringIO package and java uses the Appendable interface, but (as I understand it) there doesn't seem to be something comparable in C++, hence the need for the TextGenerator class interfacing to the ZeroCopyOutputStream class methods. Seeing as there is a simple workaround (by creating another private TextGenerator class to interface with the ZeroCopyOutputStream), it does not seem to be high-priority. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: Issue 190 in protobuf: [C++] Change TextFormat::Printer::TextGenerator class from 'private' to 'protected'
Comment #3 on issue 190 by ken...@google.com: [C++] Change TextFormat::Printer::TextGenerator class from 'private' to 'protected' http://code.google.com/p/protobuf/issues/detail?id=190 Did you look at google::protobuf::io::Printer? -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: Issue 190 in protobuf: [C++] Change TextFormat::Printer::TextGenerator class from 'private' to 'protected'
Comment #4 on issue 190 by marthaler: [C++] Change TextFormat::Printer::TextGenerator class from 'private' to 'protected' http://code.google.com/p/protobuf/issues/detail?id=190 I took a look at google::protobuf::io::Printer, in fact, I initially thought it was related to the google::protobuf::TextFormat::Printer class, and it lead to much confusion. However, since google::protobuf::io::Printer does not deal with the formating of google::protobuf::Message objects and (as I currently understand it) deals with text replacement of map objects, I think it's not what I want (or at least I couldn't figure out how the variable_delimeter was related to the indentation level). I specifically followed the google::protobuf::TextFormat::PrintToString(google::protobuf::Message, std::string) function, since the DebugString() class of methods boil down to using that method to format an output string. That PrintToString method, of course, is just a wrapper to the message reflection and the PrintField and PrintUnknownFields which uses an instance of the TextGenerator class. The initial request was to make the TextGenerator class protected, so that I could then write a class that extends from the TextFormat::Printer class and use the generator.Print(const char* text), Indent() and Outdent() methods and just worry about the TextFormat::Printer::PrintField(const Message message, const Reflection* reflection, const FieldDescriptor* field, TextGenerator generator) function and corresponding PrintFieldName, PrintFieldValue functions. These three functions in the C++ code determine the string representation of the protocol buffer message. I see what you're saying with using the protofuf::io::Printer to replace the TextGenerator class since they seem to share very similar ZeroCopyOutputStream interface code, but until then C++ developers that need to create a method to change the string output format of a protobuf will be using a copy of the internal TextGenerator class. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
