Re: [protobuf] Re: Licensing, again
I see. There are no pending patents either. I'm not sure why the FAQ was worded ambiguously like that. We'll get it fixed, though we'll probably have to run it by legal, so it might take a bit. On Wed, Jun 22, 2011 at 12:51 AM, John Haine j.l.hain...@googlemail.comwrote: Kenton, many thanks for the prompt reply. Apologies for being pedantic, but after all this is a legal question! The FAQ states that Google have no _issued_ patents. My understanding (I'm UK based) is that in US law an issued patent is one that is granted. So in principle there could be pending patent applications that cover PB which are still under examination and not yet issued. Under the Apache licence this wouldn't matter since the patent clause makes no distinction, but absent a patent licensing clause in the BSD, taken with the FAQ statement, there is some ambiguity here. I wouldn't suggest that you move to a dual-licence regime but perhaps the FAQ could be amended if indeed it is the case that there are no patents, pending or issued, that cover PB? Thanks, and apologies again for being pedantic! John. On Jun 21, 6:21 pm, Kenton Varda ken...@google.com wrote: I'm not sure I understand the question. As the FAQ says, we hold no patents on this code. What is your concern, exactly? If we dual-licensed the code under BSD and Apache, would that help? On Tue, Jun 21, 2011 at 2:32 AM, John Haine j.l.hain...@googlemail.com wrote: First, apologies if this question has already been answered! Anyway, here goes... Originally, PB was licensed under Apache 2.0. Later, the license was changed to BSD to make it more compatible with GPL2. On the FAQ there is the following statement: Does Google have any patents on Protocol Buffers? Google currently has no issued patents on Protocol Buffers, and we are happy to address any concerns around Protocol Buffers and patents that people may have. Now the advantage of the Apache license is that it includes a specific statement about the licensing of patents instantiated in the code, which is not present in the BSD license. What is Google's policy on this now the license has changed please? John. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Licensing, again
I'm not sure I understand the question. As the FAQ says, we hold no patents on this code. What is your concern, exactly? If we dual-licensed the code under BSD and Apache, would that help? On Tue, Jun 21, 2011 at 2:32 AM, John Haine j.l.hain...@googlemail.comwrote: First, apologies if this question has already been answered! Anyway, here goes... Originally, PB was licensed under Apache 2.0. Later, the license was changed to BSD to make it more compatible with GPL2. On the FAQ there is the following statement: Does Google have any patents on Protocol Buffers? Google currently has no issued patents on Protocol Buffers, and we are happy to address any concerns around Protocol Buffers and patents that people may have. Now the advantage of the Apache license is that it includes a specific statement about the licensing of patents instantiated in the code, which is not present in the BSD license. What is Google's policy on this now the license has changed please? John. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] undefined reference to GzipInputStream
GzipInputStream is only compiled into the library if zlib is available, since it depends on zlib. On Unix you should be able to fix this by installing libz.so, on MSVC you may need to modify the project files a bit. On Wed, May 18, 2011 at 12:18 AM, Shin for.shin1...@gmail.com wrote: I'm learning how to use the gzip stream. The error msg always bother me. It saying: undefined reference to `google::protobuf::io::GzipInputStream::GzipInputStream(google::protobuf::io::ZeroCopyInputStream*, google::protobuf::io::GzipInputStream::Format, int) It seems only found the header file but no dynamic lib. Does it mean I didn't install the lib correctly? I have installed the newest zlib version and re-installed the protobuf, but the problem is still existing. Thanks for ur help. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: New protobuf feature proposal: Generated classes for streaming / visitors
On Sat, Apr 2, 2011 at 3:53 PM, Jeffrey Damick jeffreydam...@gmail.comwrote: This may be a naive question, but wouldn't the format in text_format be a prime example another protocol? It seems that if you are able to reuse the vistor generate the text format, then it would be easily extendable by others for json or the latest encoding of the week.. I look forward to seeing it pushed into the tree. TextFormat is already implemented purely in terms of public interfaces -- namely, the reflection interface. Thus it is already possible to write, say, a JSON encoder/decoder for protobufs, and indeed several people have done this. The current visitor proposal (which I haven't had time to work on in awhile, but will get back to eventually...) does not provide any new way to implement TextFormat, because all visitor classes are type-specific. In other words, to implement TextFormat via visitors you would need to write an implementation for every single type, rather than one implementation that covers all types. This could perhaps be solved by inventing some sort of generic visitor adapter, but I haven't done any such thing in my patch, since reflection already solves most of the same problems. thanks -jeff On Feb 8, 2:34 pm, Kenton Varda ken...@google.com wrote: On Tue, Feb 8, 2011 at 5:47 AM, Evan Jones ev...@mit.edu wrote: The Visitor class has two standard implementations: Writer and Filler. MyStream::Writer writes the visited fields to a CodedOutputStream, using the same wire format as would be used to encode MyStream as one big message. Imagine I wanted a different protocol. Eg. I want something that checksums each message, or maybe compresses them, etc. Will I need to subclass MessageType::Visitor for each stream that I want to encode? Or will I need to change the code generator? To do these things generically, we'd need to introduce some sort of equivalent of Reflection for streams. This certainly seems like it could be a useful addition to the family, but I wanted to get the basic functionality out there first and then see if this is needed. Note that I expect people will generally only stream their top-level message. Although the proposal allows for streaming sub-messages as well, I expect that people will normally want to parse them into message objects which are handled whole. So, you only have to manually implement the top-level stream, and then you can invoke some reflective algorithm from there. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Linker Errors and stdcall vs cdecl calling conventions (VC++ 2008)
We never actually chose to use cdecl; it was simply the default used by MSVC. This whole idea that the programmer should have to choose between calling conventions is a bit strange and inconvenient. The LIBPROTOBUF_EXPORT annotations are annoying enough; I hope we don't have to litter our code with calling convention annotations as well. Is there any way we can modify the existing LIBPROTOBUF_EXPORT macro to specify calling conventions too? Your work-around of compiling the DLL with different settings should work fine. I can't imagine any reason why stdcall would cause problems. Out of curiosity, what is the binary size difference between the cdecl version and the stdcall version? The MSDN docs claim stdcall produces smaller binaries. On Tue, Mar 15, 2011 at 5:05 AM, Kostya Y kosty...@gmail.com wrote: Hello It seems like the calling convention used in libprotobuf's vc project is cdecl while the Microsoft's preferred way (or the default one) is stdcall. Is there a reason why cdecl was chosen? I am getting linker errors as the functions are not strictly defined as cdelc so the linker assumes that it should use default project settings which in my case is stdcall. As a result I get the following errors: error LNK2001: unresolved external symbol void __stdcall google::protobuf::internal::VerifyVersion(int,int,char const *) (? VerifyVersion@internal@protobuf@google@@YGXHHPBD@Z) As you can see it expects it to be __stdcall (projects settings are set to default naming convention stdcall) while it is exported as cdecl. Any suggestions how to work around it? Thanks - Kostya -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] compound message allows colon
It's a feature. This has always been allowed. No one really knows why.
On Fri, Feb 18, 2011 at 9:53 AM, naderp paul.na...@gmail.com wrote:
Hi,
Given the following .proto definition:
message person {
required string name = 1;
required int32 id = 2;
}
I noticed the following message syntax is valid:
person: {
name: fred
id: 3
}
i.e., the generated parser will quite happily accept a colon after the
compound message name (person: in this case).
Is this a bug or a feature?
Cheers
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Suspected bug in java 2.4.0a Sub-builders
The first issue is the intended behavior. It matches C++, where calling
mutable_inner() instantiates the inner message. There needs to be a way to
mark a sub-message present without actually setting any of its friends.
The second issue looks genuine, although it seems to me that it would cause
no behavioral differences, only some wasted instructions. Is this correct?
2.4.0a is actually a final release. In fact, it is the successor to
2.4.0, which had a build problem. If there are significant bugs we can do a
2.4.0b or 2.4.1 to fix them.
On Tue, Feb 22, 2011 at 6:54 AM, jamesm james.mul...@gmail.com wrote:
I have found two things that look to me like bugs in sub-builder
handling; one is in the generated code; the second is in the base
class GeneratedMessage.
Consider the following:
message Inner {
optional int32 somefield = 1;
}
message Outer{
optional Inner inner = 1;
}
Generated code for sub builder is roughtly as follows:
public Inner.Builder getInnerBuilder() {
bitField0_ |= 0x0004; //wrong - we've not changed it yet
onChanged(); //wrong - we've not changed it yset
return getInnerFieldBuilder().getBuilder();
}
The net effect is that calling:
outer.getInnerBuilder() marks outer as dirty; which in my view is
incorrect
The second issue is in GeneratedMessage.Builder:
protected final void onChanged() {
if (isClean builderParent != null) {
builderParent.markDirty();
// Don't keep dispatching invalidations until build is called
again.
isClean = false;
}
}
This again seems to be incorrect; it should be as follows:
protected final void onChanged() {
if (isClean builderParent != null) {
builderParent.markDirty();
}
// Don't keep dispatching invalidations until build is called
again.
isClean = false;
}
I can see how these two bugs together would pass a basic Junit; but
they showed up in some more extensive tests that are part of the
system that I'm building.
With these changes made we would have the desired effect that only
when a modification is made to the sub builder does that builder (and
it's parent) get marked as dirty.
Please can you confirm that the above are bugs and whether they can be
fixed before 2.4.0 final?
Thanks
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] RpcChannel and RpcController Implementation
For AppEngine you'll need to accept all incoming requests over HTTP. You probably don't want to use the RPC interfaces -- just use the regular servlet interface and use encoded protobufs in the request and response bodies. On Wed, Feb 16, 2011 at 10:39 PM, Amit amit.s...@gmail.com wrote: Hi All, I am very new to Protocol Buffer. I went through the documentation and found that we have to provide our own implementation of RpcChannel and RpcController interface. I'm not sure that what would I write in the implemented methods. I'm using JAVA in my application. Basically my requirement is to expose the API from Google App Engine and consumes it in stand-alone java application. It would be great if you could provide me an example for that. Any help will be appreciated. Thanks, Amit -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: Why does ARMv5 thinks their are 2 implementations for these 4 functions in repeated_field.CC?
A lot of C++ compilers do not implement the (ridiculously complicated) standard correctly, especially when it comes to complex templates. If you can provide a patch which fixes your problem, we could consider adopting it, but otherwise the only advice I can give is: use GCC, Clang, or MSVC. On Fri, Feb 11, 2011 at 2:35 AM, RainBow drvir...@gmail.com wrote: I have no clue what QT v4.6 is doing internally. I took all the PB files and put them in a .mmp and compiled it through Carbide v2.7 IDE for armv5 (which uses exactly same version of armcc as posted above). Surprisingly, it compiled very smoothly at the very first attempt. QT is gaytown...fer sure !!! -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: .proto file with enumeration only
Unix and Windows EOLs should work, but old-Mac EOLs (which were just '\r's
with no '\n') probably will not work. I thought that Mac stopped using that
style with OSX, though, so I'm not sure why you'd be seeing this problem 10
years later.
On Wed, Feb 16, 2011 at 11:16 AM, dan jiggersp...@gmail.com wrote:
okay... that does not appear to be the whole story. it also has
something to do with unix-style end-of-line vs. windows and/or mac
style end-of-line. behaves correctly with unix-style. comment on the
first line fails on whatever style i had it set to before. also fails
if you have a line commented out in the middle of the enumeration like
so:
enum foo {
bar = 1;
// baz = 2;
}
On Feb 16, 1:53 pm, dan jiggersp...@gmail.com wrote:
solved it. apparently you can't have a comment as the first line of
your file.
On Feb 16, 11:40 am, dan jiggersp...@gmail.com wrote:
i have a .proto with nothing but an enumeration in it. it compiles,
but the output .h and .cc file are basically empty stubs. i made a
test file with
enum foo {
bar = 1;
}
and that compiles just fine... the file i'm having problems with is
the essentially the same but has about 125 values.
any ideas?
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Typed Array Support
Well, first we'd need a Javascript protobuf implementation. There are a couple third-party ones, but Google doesn't provide one at this time. The usual way to deal with Javascript is to encode your message in JSON, e.g. using the code provided by http://code.google.com/p/protobuf-java-format/. On Wed, Feb 16, 2011 at 3:38 AM, jd unicom...@gmail.com wrote: Are there any plans for protocol buffers to have support for the javascript typed arrays that just rolled out with WebGL in Chrome? I would like to add some communication to my WebGL demo at http://www.pcprogramming.com but don't want to hand write all the message classes in javascript and c++. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Error when build universal binary on Mac OS Tiger 10.4 sdk
From the error message, it seems like GCC doesn't like being passed /Volumes/Data/XCode/SDKs/MacOSX10.4u.sdk in CXXFLAGS. Are you sure this is needed at all? If so, maybe this should be passed as LIBS or LDFLAGS instead? On Tue, Feb 22, 2011 at 3:50 AM, pick_tim nguyenquycu...@gmail.com wrote: I tried compile protobuf on Mac OS Tiger, My command : ./configure CXXFLAGS=/Volumes/Data/XCode/SDKs/MacOSX10.4u.sdk -arch i386 -arch ppc -mmacosx-version-min=10.4 --disable-dependency- tracking. - output : checking build system type... i386-apple-darwin10.6.0 checking host system type... i386-apple-darwin10.6.0 checking target system type... i386-apple-darwin10.6.0 checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... ./install-sh -c -d checking for gawk... no checking for mawk... no checking for nawk... no checking for awk... awk checking whether make sets $(MAKE)... yes checking for gcc... gcc checking whether the C compiler works... yes checking for C compiler default output file name... a.out checking for suffix of executables... checking whether we are cross compiling... no checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking for style of include used by make... GNU checking dependency style of gcc... none checking for g++... g++ checking whether we are using the GNU C++ compiler... no checking whether g++ accepts -g... yes checking dependency style of g++... none checking how to run the C++ preprocessor... g++ -E checking for grep that handles long lines and -e... /usr/bin/grep checking for egrep... /usr/bin/grep -E checking for ANSI C header files... no checking for sys/types.h... no checking for sys/stat.h... no checking for stdlib.h... no checking for string.h... no checking for memory.h... no checking for strings.h... no checking for inttypes.h... no checking for stdint.h... no checking for unistd.h... no checking minix/config.h usability... no checking minix/config.h presence... no checking for minix/config.h... no checking whether it is safe to define __EXTENSIONS__... no checking C++ compiler flags.. use user-supplied: /Volumes/Data/ XCode/SDKs/MacOSX10.4u.sdk -arch i386 -arch ppc -mmacosx-version- min=10.4 checking whether __SUNPRO_CC is declared... no checking for a sed that does not truncate output... /usr/bin/sed checking for fgrep... /usr/bin/grep -F checking for ld used by gcc... /usr/libexec/gcc/i686-apple- darwin10/4.2.1/ld checking if the linker (/usr/libexec/gcc/i686-apple-darwin10/4.2.1/ld) is GNU ld... no checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm checking the name lister (/usr/bin/nm) interface... BSD nm checking whether ln -s works... yes checking the maximum length of command line arguments... 196608 checking whether the shell understands some XSI constructs... yes checking whether the shell understands +=... yes checking for /usr/libexec/gcc/i686-apple-darwin10/4.2.1/ld option to reload object files... -r checking for objdump... no checking how to recognize dependent libraries... pass_all checking for ar... ar checking for strip... strip checking for ranlib... ranlib checking command to parse /usr/bin/nm output from gcc object... failed checking for dsymutil... dsymutil checking for nmedit... nmedit checking for lipo... lipo checking for otool... otool checking for otool64... no checking for -single_module linker flag... yes checking for -exported_symbols_list linker flag... no checking for dlfcn.h... no checking whether we are using the GNU C++ compiler... (cached) no checking whether g++ accepts -g... (cached) yes checking dependency style of g++... (cached) none checking how to run the C++ preprocessor... g++ -E checking for objdir... .libs checking if gcc supports -fno-rtti -fno-exceptions... no checking for gcc option to produce PIC... -fno-common -DPIC checking if gcc PIC flag -fno-common -DPIC works... yes checking if gcc static flag -static works... no checking if gcc supports -c -o file.o... yes checking if gcc supports -c -o file.o... (cached) yes checking whether the gcc linker (/usr/libexec/gcc/i686-apple- darwin10/4.2.1/ld) supports shared libraries... yes checking dynamic linker characteristics... darwin10.6.0 dyld checking how to hardcode library paths into programs... immediate checking whether stripping libraries is possible... yes checking if libtool supports shared libraries... yes checking whether to build shared libraries... yes checking whether to build static libraries... yes checking whether the g++ linker (/usr/libexec/gcc/i686-apple- darwin10/4.2.1/ld) supports shared libraries... no libtool.m4: error: problem compiling CXX test program checking for g++ option to produce PIC... -DPIC
Re: [protobuf] Number of Bits Written to Memory
Protobufs provide their own encoding. You can't control exactly how the bytes are written, unless you write your own encoder, which would defeat the purpose. On Thu, Feb 3, 2011 at 8:58 AM, Tanya turtle...@gmail.com wrote: Hi, we are looking at replacing a XML/JAXB project with something similar to protocol buffers, especially since it's multi-platform, but there are a couple of concerns we have seen that would prevent us from switching. I'd like to verify that what we think is actually true before passing on what seems to be a very cool solution Google has come up with. My main question revolves around how the bytes are stored. In our current solution, we store by bit, so some items have very strange amounts of bits being stored (i.e. 5 bits, 14 bits). We noticed that protocol buffers seem to only store by byte or word (very even amounts of bits) and have a unique way of encoding. Basically, we need some way of specifying how the bytes are encoded/ written to memory, and we need to specify the number of bits written each time. Is this possible to do in protocol buffers? And if so, how would it be accomplished? Thanks. Any help is greatly appreciated. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Extending the compiler and programatically importing options
Nope, there's no way to do this, sorry.
On Tue, Feb 15, 2011 at 4:49 AM, Arkaitz Jimenez arkai...@gmail.com wrote:
Hi,
I've my own compiler extension that adds some code over the standard
code generated by cppgenerator.
Now, I want to add a custom option, I know I can do that by adding the
custom option extending with a proto file like extend
google.protobuf.MethodOptions{ optional bool foo=5;}.
If I do it that way any .proto file that uses it will need access to
the file that defines the extension and import it.
I was wondering if there is a way of embedding it somehow in my
compiler so that the file is not needed.
My app is basically a codegenerator based on cppgenerator and
google::protobuf::compiler::CommandLineInterface cli where I register
it and then run cli.Run.
I guess somehow it should be possible to add the custom descriptor to
the generator at runtime without the need of the proto file.
Thanks
Arkaitz
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Feature request: ability to suppress logging of message fields
You can always write custom code that formats messages any way you want for your logs. The TextFormat class is based purely on public interfaces, so you can copy it and modify it. You can define a custom option for flagging fields that you want removed from the logs and use that custom option in your modified TextFormat. I don't think we want to implement this feature directly in the base implementation. There are too many little feature requests like this, and if we implemented them all the library would get quite bloated. On Thu, Feb 17, 2011 at 4:10 PM, Robert H robert.huff...@gmail.com wrote: If you use protobuf for confidential data, enabling debugt logging exposes the data. It would be nice to have a field option that directs the TextFormat class to not log a field. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Creating a FileDescriptorSet of all .proto files used in a certain executable
As you note, you can't iterate over all compiled-in protos. However, if you can collect a set of root protos that you care about, it's easy to traverse down the tree to collect all their dependencies. In general, trying to enumerate everything that is linked in tends to lead to trouble. The linker may drop things that it thinks are unused. Technically the C++ standard does not even require initializers to be run until the first time their translation unit is accessed, although most compilers run all initializers at startup. It can also be confusing if linking in a new library causes behavioral changes in unrelated parts of your program. I actually wish that we had never introduced the singleton generated_pool in the C++ library since it creates so much confusion -- the Java protobuf library has no central registry, and this avoids a lot of problems. On Thu, Feb 17, 2011 at 9:42 AM, Oren Shemesh shemesh.o...@gmail.comwrote: Hello, My executable is composed of multiple .proto files, some of them importing others. I would like the executable to communicate with a generic program (Which is compiled once and does not change), using protobuf messages. So, I need the generic program to get the descriptors of messages used in the communication at run-time, from my executable. I understand that to do this, I need the executable to create a FileDescriptorSet containing all the files needed to describe a certain message, serialize this set, pass it to my generic program, which will then create a descriptor pool and feed it with the deserialized FileDescriptorSet. Now that I have this populated pool, the generic program can find any descriptor by it's name, as communicated from the executable. My question is: Is there a method to automatically create the file descriptor set of all files used inside a given application, in proper topological order (i.e. imported files appearing before files depending on them) ? I know that protoc can create such a set, but this set reflects the set of files used in a single protoc run, and my application has multiple files. I thought about iterating all descriptors found in the generated_pool, gathering the set of files, sorting them and creating this set, but there is no way to iterate all descriptors in the pool. It seems to me that in order to gather all file descriptor protos used in my executable, I need to manually define a single .proto file importing all other files, but I do not like this idea because it means taht every time I add a new .proto file to my application, this 'master proto' file needs to be updated. (Yes, the build system could do it automatically). Is there a better way ? Thanks, Oren. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] New protobuf feature proposal: Generated classes for streaming / visitors
On Tue, Feb 8, 2011 at 11:23 AM, Evan Jones ev...@mit.edu wrote:
Sorry, just an example of why you might want a different protocol. If I've
streamed 10e9 messages to disk, I don't want this stream to break if there
is some weird corruption in the middle, so I want some protocol that can
resume from corruption.
Ah, yes. This isn't an appropriate protocol for enormous files. It's
more targeted at network protocols.
Although, you might be able to build a decent seekable file protocol on top
of it, by choosing a random string to use as a sync point, then writing that
string every now and then...
message FileStream {
repeated string sync_point = 1;
repeated Foo foo = 2;
repeated Bar bar = 3;
...
}
When writing, after every few messages, write a copy of sync_point. Then,
you can seek to an arbitrary position in the file by looking for a nearby
copy of the sync point byte sequence, and starting to parse immediately
after that. The sync point just needs to be an 128-bit (or so)
cryptographically random sequence, chosen differently for each file, so that
there's no chance that the bytes will appear in the file by accident.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] header update in building protobuf 2.4.0a on cygwin
Re-adding the list so that someone can consider these problems. On Mon, Feb 7, 2011 at 5:10 AM, Jim Olson jimo...@gmail.com wrote: I think it was gcc/g++ 3.3, yes it's a couple years old. Other problems I discovered : protoc has to be installed parallel to the java/src dir otherwise the maven pom.xml can't find it. I saw postings about the problems this caused but no solutions. Perhaps you can put this in the README or INSTALL. And cygwin expects ; rather than : as the CLASSPATH path separator which I think caused problems in the generated add_person_java and list_person_java shell scripts in the examples dir. It took me a while to figure this out because I don't usually use cygwin for java work. Thanks Jim Olson On Sun, Feb 6, 2011 at 3:33 PM, Kenton Varda ken...@google.com wrote: What Cygwin version? What GCC version? We do test releases on Cygwin so I'm guessing you're using an older version. The error looks like a problem that old GCCs had (though we may want to work around it). On Sat, Feb 5, 2011 at 3:59 PM, jimo555 jimo...@gmail.com wrote: Just wanted to pass along a simple fix I had to make to get the protobuf package to build on cygwin. I had to make const': ./google/protobuf/descriptor.h:1152: error: `google::protobuf::internal::Mutex*google::protobuf::DescriptorPool::mutex_' is private this member field public. It all built ok after doing this. (I also had to move the dir to C: as it had a link error due to being under C:/Documents and Settings.) Here is the error message excerpt I got: libtool: compile: g++ -DHAVE_CONFIG_H -I. -I.. -Wall -Wwrite-strings - Woverloaded-virtual -Wno-sign-compare -O2 -g -DND EBUG -MT structurally_valid.lo -MD -MP -MF .deps/ structurally_valid.Tpo -c google/protobuf/stubs/structurally_valid.cc -DDLL_EXPORT -DPIC -o .libs/structurally_valid.o libtool: compile: g++ -DHAVE_CONFIG_H -I. -I.. -Wall -Wwrite-strings - Woverloaded-virtual -Wno-sign-compare -O2 -g -DND EBUG -MT structurally_valid.lo -MD -MP -MF .deps/ structurally_valid.Tpo -c google/protobuf/stubs/structurally_valid.cc - o structurally_valid.o /dev/null 21 mv -f .deps/structurally_valid.Tpo .deps/structurally_valid.Plo /bin/sh ../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -Wall -Wwrite-strings -Woverloaded-virtu al -Wno-sign-compare -O2 -g -DNDEBUG -MT descriptor.lo -MD -MP - MF .deps/descriptor.Tpo -c -o descriptor.lo `test -f 'go ogle/protobuf/descriptor.cc' || echo './'`google/protobuf/ descriptor.cc libtool: compile: g++ -DHAVE_CONFIG_H -I. -I.. -Wall -Wwrite-strings - Woverloaded-virtual -Wno-sign-compare -O2 -g -DND EBUG -MT descriptor.lo -MD -MP -MF .deps/descriptor.Tpo -c google/ protobuf/descriptor.cc -DDLL_EXPORT -DPIC -o .libs/de scriptor.o google/protobuf/descriptor.cc: In member function `virtual const google::protobuf::FieldDescriptor* google::protobuf::De scriptorBuilder::OptionInterpreter::AggregateOptionFinder::FindExtension(google::protobuf::Message*, const std::string) const': ./google/protobuf/descriptor.h:1152: error: `google::protobuf::internal::Mutex*google::protobuf::DescriptorPool::mutex_' is private google/protobuf/descriptor.cc:4341: error: within this context ./google/protobuf/descriptor.h:1152: error: `google::protobuf::internal::Mutex*google::protobuf::DescriptorPool::mutex_' is private google/protobuf/descriptor.cc:4342: error: within this context make[2]: *** [descriptor.lo] Error 1 make[2]: Leaving directory `/cygdrive/c/Documents and Settings/Jim/My Documents/java/protobuf-2.4.0a/src' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/cygdrive/c/Documents and Settings/Jim/My Documents/java/protobuf-2.4.0a' make: *** [all] Error 2 -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Circular package import problem
With protoc (the standard compiler), this should produce a simple error. Since Ken said he got StackOverFlow exception, it sounds like he's using some other compiler (not written in C++). On Mon, Feb 7, 2011 at 10:26 AM, Pherl Liu liuj...@google.com wrote: You probably need to break up your proto files to eliminate the circular dependency.. (i.e. have multiple .proto files for a package, and only import necessary .proto files). On Mon, Feb 7, 2011 at 5:42 PM, Ken ke.lane@gmail.com wrote: Hi, I have two proto files: A.proto and B.proto. If I import package B in A and import A in B, I got StackOverFlow exception during compiling. Does anyone know any sulotion for it? Thanks, Ken -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] header update in building protobuf 2.4.0a on cygwin
What Cygwin version? What GCC version? We do test releases on Cygwin so I'm guessing you're using an older version. The error looks like a problem that old GCCs had (though we may want to work around it). On Sat, Feb 5, 2011 at 3:59 PM, jimo555 jimo...@gmail.com wrote: Just wanted to pass along a simple fix I had to make to get the protobuf package to build on cygwin. I had to make const': ./google/protobuf/descriptor.h:1152: error: `google::protobuf::internal::Mutex*google::protobuf::DescriptorPool::mutex_' is private this member field public. It all built ok after doing this. (I also had to move the dir to C: as it had a link error due to being under C:/Documents and Settings.) Here is the error message excerpt I got: libtool: compile: g++ -DHAVE_CONFIG_H -I. -I.. -Wall -Wwrite-strings - Woverloaded-virtual -Wno-sign-compare -O2 -g -DND EBUG -MT structurally_valid.lo -MD -MP -MF .deps/ structurally_valid.Tpo -c google/protobuf/stubs/structurally_valid.cc -DDLL_EXPORT -DPIC -o .libs/structurally_valid.o libtool: compile: g++ -DHAVE_CONFIG_H -I. -I.. -Wall -Wwrite-strings - Woverloaded-virtual -Wno-sign-compare -O2 -g -DND EBUG -MT structurally_valid.lo -MD -MP -MF .deps/ structurally_valid.Tpo -c google/protobuf/stubs/structurally_valid.cc - o structurally_valid.o /dev/null 21 mv -f .deps/structurally_valid.Tpo .deps/structurally_valid.Plo /bin/sh ../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -Wall -Wwrite-strings -Woverloaded-virtu al -Wno-sign-compare -O2 -g -DNDEBUG -MT descriptor.lo -MD -MP - MF .deps/descriptor.Tpo -c -o descriptor.lo `test -f 'go ogle/protobuf/descriptor.cc' || echo './'`google/protobuf/ descriptor.cc libtool: compile: g++ -DHAVE_CONFIG_H -I. -I.. -Wall -Wwrite-strings - Woverloaded-virtual -Wno-sign-compare -O2 -g -DND EBUG -MT descriptor.lo -MD -MP -MF .deps/descriptor.Tpo -c google/ protobuf/descriptor.cc -DDLL_EXPORT -DPIC -o .libs/de scriptor.o google/protobuf/descriptor.cc: In member function `virtual const google::protobuf::FieldDescriptor* google::protobuf::De scriptorBuilder::OptionInterpreter::AggregateOptionFinder::FindExtension(google::protobuf::Message*, const std::string) const': ./google/protobuf/descriptor.h:1152: error: `google::protobuf::internal::Mutex*google::protobuf::DescriptorPool::mutex_' is private google/protobuf/descriptor.cc:4341: error: within this context ./google/protobuf/descriptor.h:1152: error: `google::protobuf::internal::Mutex*google::protobuf::DescriptorPool::mutex_' is private google/protobuf/descriptor.cc:4342: error: within this context make[2]: *** [descriptor.lo] Error 1 make[2]: Leaving directory `/cygdrive/c/Documents and Settings/Jim/My Documents/java/protobuf-2.4.0a/src' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/cygdrive/c/Documents and Settings/Jim/My Documents/java/protobuf-2.4.0a' make: *** [all] Error 2 -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: New protobuf feature proposal: Generated classes for streaming / visitors
Unfortunately, the Google C++ Style Guide prescribes inconsistency. Only
simple inline methods can use lowercase-with-underscores naming; everything
else is supposed to use capitalized camelcase.
On Sat, Feb 5, 2011 at 8:40 AM, Frank Chu f...@google.com wrote:
Can the naming be
visit_bar()
visit_baz()
then? It's good to have some consistency.
Frank
On Sat, Feb 5, 2011 at 08:06, Kenton Varda ken...@google.com wrote:
On Wed, Feb 2, 2011 at 10:13 AM, Henner Zeller
henner.zel...@googlemail.com wrote:
I guess the naming is confusing in the example. The Visit is per
field-name; but since the typed is named the same as the field in this
example, it is confusing.
Yes, sorry. Better example:
message MyStream {
option generate_visitors = true;
repeated Foo bar = 1;
repeated Foo baz = 2;
}
creates:
class MyStream::Visitor {
public:
virtual ~Visitor();
virtual void VisitBar(const Foo value);
virtual void VisitBaz(const Foo value);
};
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: New protobuf feature proposal: Generated classes for streaming / visitors
On Wed, Feb 2, 2011 at 10:13 AM, Henner Zeller henner.zel...@googlemail.com
wrote:
I guess the naming is confusing in the example. The Visit is per
field-name; but since the typed is named the same as the field in this
example, it is confusing.
Yes, sorry. Better example:
message MyStream {
option generate_visitors = true;
repeated Foo bar = 1;
repeated Foo baz = 2;
}
creates:
class MyStream::Visitor {
public:
virtual ~Visitor();
virtual void VisitBar(const Foo value);
virtual void VisitBaz(const Foo value);
};
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] 2.4.0a released.
On Wed, Feb 2, 2011 at 9:59 PM, David Yu david.yu@gmail.com wrote: Where the jars at? (Or do we have to build the jars from source?) Have to build from source. (Though we're going to try to get the Maven repo updated. Long story.) I'm trying to benchmark 2.4.0a with its java perf improvements. The lazy decoding support is cool (like the activemq impl of protobuf). Note that lazy decoding only applies to strings -- they are decoded from UTF-8 bytes to String objects on first access. Sub-messages are still eagerly decoded, although we've been thinking about adding support for lazy sub-messages. (It's not always as great as it seems -- lazy decoding adds a lot of bookkeeping overhead, especially if you want to be thread-safe. We've actually discovered that the lazy UTF-8 decoding is a loss in some use cases; it will be tuned further for the next release.) -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] New protobuf feature proposal: Generated classes for streaming / visitors
Hello open source protobuf users,
*Background*
Probably the biggest deficiency in the open source protocol buffers
libraries today is a lack of built-in support for handling streams of
messages. True, it's not too hard for users to support it manually, by
prefixing each message with its size as described here:
http://code.google.com/apis/protocolbuffers/docs/techniques.html#streaming
However, this is awkward, and typically requires users to reach into the
low-level CodedInputStream/CodedOutputStream classes and do a lot of work
manually. Furthermore, many users want to handle streams
of heterogeneous message types. We tell them to wrap their messages in an
outer type using the union pattern:
http://code.google.com/apis/protocolbuffers/docs/techniques.html#union
But this is kind of ugly and has unnecessary overhead.
These problems never really came up in our internal usage, because inside
Google we have an RPC system and other utility code which builds on top of
protocol buffers and provides appropriate abstraction. While we'd like to
open source this code, a lot of it is large, somewhat messy, and highly
interdependent with unrelated parts of our environment, and no one has had
the time to rewrite it all cleanly (as we did with protocol buffers itself).
*Proposed solution: Generated Visitors*
I've been wanting to fix this for some time now, but didn't really have a
good idea how. CodedInputStream is annoyingly low-level, but I couldn't
think of much better an interface for reading a stream of messages off the
wire.
A couple weeks ago, though, I realized that I had been failing to consider
how new kinds of code generation could help this problem. I was trying to
think of solutions that would go into the protobuf base library, not
solutions that were generated by the protocol compiler.
So then it became pretty clear: A protobuf message definition can also be
interpreted as a definition for a streaming protocol. Each field in the
message is a kind of item in the stream.
// A stream of Foo and Bar messages, and also strings.
message MyStream {
option generate_visitors = true; // enables generation of streaming
classes
repeated Foo foo = 1;
repeated Bar bar = 2;
repeated string baz = 3;
}
All we need to do is generate code appropriate for treating MyStream as a
stream, rather than one big message.
My approach is to generate two interfaces, each with two provided
implementations. The interfaces are Visitor and Guide.
MyStream::Visitor looks like this:
class MyStream::Visitor {
public:
virtual ~Visitor();
virtual void VisitFoo(const Foo foo);
virtual void VisitBar(const Bar bar);
virtual void VisitBaz(const std::string baz);
};
The Visitor class has two standard implementations: Writer and Filler.
MyStream::Writer writes the visited fields to a CodedOutputStream, using
the same wire format as would be used to encode MyStream as one big message.
MyStream::Filler fills in a MyStream message object with the visited
values.
Meanwhile, Guides are objects that drive Visitors.
class MyStream::Guide {
public:
virtual ~Guide();
// Call the methods of the visitor on the Guide's data.
virtual void Accept(MyStream::Visitor* visitor) = 0;
// Just fill in a message object directly rather than use a visitor.
virtual void Fill(MyStream* message) = 0;
};
The two standard implementations of Guide are Reader and Walker.
MyStream::Reader reads items from a CodedInputStream and passes them to the
visitor. MyStream::Walker walks over a MyStream message object and passes
all the fields to the visitor.
To handle a stream of messages, simply attach a Reader to your own Visitor
implementation. Your visitor's methods will then be called as each item is
parsed, kind of like SAX XML parsing, but type-safe.
*Nonblocking I/O*
The Reader type declared above is based on blocking I/O, but many users
would prefer a non-blocking approach. I'm less sure how to handle this, but
my thought was that we could provide a utility class like:
class NonblockingHelper {
public:
template typename MessageType
NonblockingHelper(typename MessageType::Visitor* visitor);
// Push data into the buffer. If the data completes any fields,
// they will be passed to the underlying visitor. Any left-over data
// is remembered for the next call.
void PushData(void* data, int size);
};
With this, you can use whatever non-blocking I/O mechanism you want, and
just have to push the data into the NonblockingHelper, which will take care
of calling the Visitor as necessary.
*C++ implementation*
I've written up a patch implementing this for C++ (not yet including the
nonblocking part):
http://codereview.appspot.com/4077052
*Feedback*
What do you think?
I know I'm excited to use this in some of my own side projects (which is why
I spent my weekend working on it), but before adding this to the official
implementation we
Re: [protobuf] Repeated Fields Encoding
The encoding is documented in detail here:
http://code.google.com/apis/protocolbuffers/docs/encoding.html
http://code.google.com/apis/protocolbuffers/docs/encoding.htmlThe short
answer is, yes, repeated fields are literally encoded as repeated individual
values, unless you use packed encoding.
On Tue, Feb 1, 2011 at 3:01 AM, Timothy Parez timothypa...@gmail.comwrote:
Hello,
Considering the following proto file:
message FileDescriptor
{
required string Filename = 1;
optional int64 Size = 2 [default = 0];
}
message FileList
{
repeated FileDescriptor Files = 1;
}
If you create something like this:
(and I'm duplicating the data because it made it easier to spot in a
hex editor)
files.Files.Add(new FileDescriptor() { Filename = AAA, Size =
100 });
files.Files.Add(new FileDescriptor() { Filename = AAA, Size =
100 });
files.Files.Add(new FileDescriptor() { Filename = AAA, Size =
100 });
files.Files.Add(new FileDescriptor() { Filename = AAA, Size =
100 });
and then serialize it using the Protobuf.Serializer I expected it to
generate something like
Tag for the FileList - Id 1, WireType 2 = 0x0A
Length of the payload (all the bytes for all the files that follow)
But instead I found everything is simply repeated.
0A 0B 0A 07 41 41 41 41 41 41 41 10 64
0A 0B 0A 07 41 41 41 41 41 41 41 10 64
0A 0B 0A 07 41 41 41 41 41 41 41 10 64
0A 0B 0A 07 41 41 41 41 41 41 41 10 64
I'm wondering, is this an implementation detail (and allowed by the
protobol buffer specifications)
or a requirement of the google protocol buffer specifications ?
It does seem to add quite a bit of overhead, imagine the FileList has
other properties,
they would be repeated for every instance of FileDescriptor ?
Or am I missing something ?
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Services not compiling
The *_generic_services options now default to false. You must explicitly
set them to true, e.g.:
option cc_generic_services = true;
option java_generic_services = true;
option py_generic_services = true;
We decided to make these off-by-default because we want to encourage RPC
implementations to provide their own code generator plugins that generate
code appropriate for that implementation, rather than relying on protoc's
least-common-denominator generic code.
The documentation has not been updated for 2.4.0 yet.
On Tue, Feb 1, 2011 at 12:46 PM, jdoliner jdoli...@gmail.com wrote:
Hi,
I'm new to protocol buffers so apologies if this is something stupid.
(I searched the forum and didn't find anything). I'm having a problem
getting services to compile. I have the following population.proto
file:
package population;
message addrinfo {
required fixed32 ip = 1;
required uint32 port = 2;
required uint32 id = 3;
}
message Join_initial {
required addrinfo addr = 1;
}
message Join_propose {
required addrinfo addr = 1;
}
message Join_respond {
required bool agree = 1;
}
message Join_mk_official {
required addrinfo addr = 1;
}
message Join_ack_official {
}
message Join_welcome {
repeated addrinfo addrs= 1;
}
service JoinService {
rpc Join (Join_initial) returns (Join_welcome);
rpc Propose (Join_propose) returns (Join_respond);
rpc Make_official (Join_mk_official) returns (Join_ack_official);
}
And when I compile it with:
protoc --cpp_out=. population.proto
I get population.pb.{cc|h} but nowhere in either one is there mention
of the JoinService class. Am I doing something wrong?
My libprotoc version is 2.4.0.
Thanks.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: New protobuf feature proposal: Generated classes for streaming / visitors
On Tue, Feb 1, 2011 at 3:17 PM, Jason Hsueh jas...@google.com wrote: Conceptually this sounds great, the big question to me is whether this should be implemented as an option in the compiler or as a separate plugin. I haven't taken a thorough look at the patch, but I'd guess it adds a decent amount to the core code generator. I have a preference for the plugin approach, but of course I'm primarily an internal protobuf user, so I'm willing to be convinced otherwise :-) Would using a plugin, possibly even shipped with the standard implementation, make this feature too inconvenient to use? Or is there enough demand for this that it warrants implementing as an option? First of all, note that this feature is off by default. You have to turn it on with the generate_visitors message-level option. The only new code added to the base library is a couple templates in WireFormatLite, which are of course never instantiated if you don't generate visitor code. There are a few reasons I prefer to make this part of the base code generator: - If you look at the patch, you'll see that the code generation for the two Guide classes actually shares a lot with the code generation for MergeFromCodedStream and SerializeWithCachedSizes. To make this a plugin, either we'd have to expose parts of the C++ code generator internals publicly (eww) or we'd have to reproduce a lot of code (also eww). - The Reader and Writer classes directly use WireFormatLite, which is a private interface. - It seems clear that this feature is widely desired by open source users. We're not talking about a niche use case here. Regarding the proposed interfaces: I can imagine some applications where the const refs passed to the visitor methods may be too restrictive - the user may instead want to take ownership of the object. e.g., suppose the stream is a series of requests, and each of the visitor handlers needs to start some asynchronous work. It would be good to hear if users have use cases that don't quite fit into this model (or at least if the existing use cases will work). Interesting point. In the Reader case, it's creating new objects, so in theory it ought to be able to hand off ownership to the Visitor it calls. But, the Walker is walking an existing object and thus clearly cannot give up ownership. It seems clear that some use cases need const references, which means that the only way we could support ownership passing is by adding another parallel set of methods. I suppose they could have default implementations that delegate to the const reference versions, in which case only people who wanted to optimize for them would need to override them. But I'd like to see that this is really desired first -- it's easy enough to add later. Also note that my code currently doesn't reuse message objects, but improving it to do so would be straightforward. A Reader could allocate one object of each sub-message type for reuse. But, it seems like that wouldn't play well with ownership-passing. On Tue, Feb 1, 2011 at 10:45 AM, Kenton Varda ken...@google.com wrote: Hello open source protobuf users, *Background* Probably the biggest deficiency in the open source protocol buffers libraries today is a lack of built-in support for handling streams of messages. True, it's not too hard for users to support it manually, by prefixing each message with its size as described here: http://code.google.com/apis/protocolbuffers/docs/techniques.html#streaming However, this is awkward, and typically requires users to reach into the low-level CodedInputStream/CodedOutputStream classes and do a lot of work manually. Furthermore, many users want to handle streams of heterogeneous message types. We tell them to wrap their messages in an outer type using the union pattern: http://code.google.com/apis/protocolbuffers/docs/techniques.html#union But this is kind of ugly and has unnecessary overhead. These problems never really came up in our internal usage, because inside Google we have an RPC system and other utility code which builds on top of protocol buffers and provides appropriate abstraction. While we'd like to open source this code, a lot of it is large, somewhat messy, and highly interdependent with unrelated parts of our environment, and no one has had the time to rewrite it all cleanly (as we did with protocol buffers itself). *Proposed solution: Generated Visitors* I've been wanting to fix this for some time now, but didn't really have a good idea how. CodedInputStream is annoyingly low-level, but I couldn't think of much better an interface for reading a stream of messages off the wire. A couple weeks ago, though, I realized that I had been failing to consider how new kinds of code generation could help this problem. I was trying to think of solutions that would go into the protobuf base library, not solutions that were generated by the protocol compiler. So then it became
Re: [protobuf] Re: inheritance.. well sort of... and FieldDescriptors
You can't use the same field descriptors for the four classes. But, note
that one thing you *can* do is define a base type that just contains the
shared fields, and then parse any of the other types as this type in order
to access the common fields. Since the field numbers match, they are
compatible.
On Sat, Jan 29, 2011 at 1:32 PM, koert koertkuip...@gmail.com wrote:
I thought about these options and settled on multiple classes that all
share a few fields. So indeed duplicate every shared field.
On Jan 28, 2:28 pm, TJ Rothwell tj.rothw...@gmail.com wrote:
Is there a best practice for this use case?
Here are some options.
// Duplicate every field (sounds like you're doing this)
message FooRequest {
required string prompt = 1;
required int64 timeout = 2;
required Foo foo = 3;
}
message BarRequest {
required string prompt = 1;
required int64 timeout = 2;
required Bar bar = 3;
}
// Client code--something like this anyway
Foo foo = Foo.newBuilder().build();
FooRequest fooRequest =
FooRequest.newBuilder().setPrompt().setTimeout(0).
setFoo(foo).build();
Bar bar = Bar.newBuilder().build();
BarRequest barRequest =
BarRequest.newBuilder().setPrompt().setTimeout(0).
setBar(bar).build();
// Share a base message and extend it
message BaseRequest {
required string prompt = 1;
required int64 timeout = 2;
extensions 1000 to max; // reserved for extensions}
message BazRequest {
extends BaseRequest {
optional BazRequest baz = 1000;
}
required Baz baz = 1;
}
// Client Code
Baz baz = Baz.newBuilder().build();
BazRequest bazRequest = BazRequest.newBuilder().setBaz(baz).build();
BaseRequest request =
BaseRequest.newBuilder().setPrompt().setTimeout(0).setExtension(BazRequest.baz,
bazRequest).build();
// so given a BaseRequest, how do we know if it's is a Baz or something
else? -- lots of hasExtension() calls? that doesn't sound good and you
have
to keep track of an ExtensionRegistry if you're using a generic
Reader/Writer.
// baz is a base? not so much
// baz has a base? sounds right
// this approach doesn't seem right
// Composition
message CommonRequest {
required string prompt = 1;
required int64 timeout = 2;}
message QuxRequest {
required CommonRequest common = 1;
required Qux qux = 2;
}
// Client Code
Qux baz = Qux.newBuilder().build();
CommonRequest.Builder crb =
CommonRequest.newBuilder().setPrompt().setTimeout(0);
QuxRequest bazRequest
= QuxRequest.newBuilder().setCommon(crb).setQux(qux).build();
// the challenge here would be determining how the common message is
defined. If you have dozens of requests that match on some fields here,
some
there... it may get complicated.
YMMV,
-- TJ
On Fri, Jan 28, 2011 at 12:59 PM, koert koertkuip...@gmail.com wrote:
i have several proto message definitions that all share the first 4
fields. its as if they are all subclasses of one protobuf message
format.
in java can i create the FieldDescriptors for these 4 fields once and
use them for the getters and setters of all these message classes? it
would save me a lot in terms of logic and maybe also lead to somewhat
bettter performance.
best koert
--
You received this message because you are subscribed to the Google
Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
protobuf%2bunsubscr...@googlegroups.comprotobuf%252bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Building Protobuffers under linux
I think if -lprotobuf wasn't found, the linker would complain about that first, rather than complain about undefined references -- unless you have some other libprotobuf in your library path which is being taken instead. Maybe an older version? On Wed, Jan 19, 2011 at 10:29 AM, Jalitt jal...@gmail.com wrote: I am trying to build google protocol buffers in linux, and then link it into a build. for configure options, I am passing in --prefix=SOME_LOCAL_DIRECTORY make, make check, and make install all work just fine, and the library files are chilling in SOME_LOCAL_DIRECTORY. Elsewhere, I have a crazy folder heirarchy //one_folder/another/ and_another/CompileDirectory/ I copy the built protocol buffers library files from SOME_LOCAL_DIRECTORY and move them to /CompileDirectory/libs /CompileDirectory/includes has the include files In the makefile in CompileDirectory, I specific /CompileDirectory/libs for linking, and -lprotobuf Everything builds just fine, but during linking, I get a billion undefined reference errors. Clearly it is not linking properly with the library file. I went into the libs directory and nm libprotobuf.a and I see the symbols are properly published. Anyone have any ideas why its not properly linking libprotobuf.a? Is it a ./configfure option I am missing? -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Problem with accents in python while unpacking a message
When using non-ASCII characters, make sure you are using the unicode type, not str, to avoid any ambiguity in character encoding. Also, I vaguely remember a bug fix related to this that is in version 2.4.0, so you might try that. On Thu, Jan 20, 2011 at 10:39 AM, Louhike louh...@gmail.com wrote: Hi, I'm using Google Protobuf with python on a project. My problem is I get an error while my program tries to build an instance with the function google.protobuf.text_format.merge() if the message contains accents (“utf-8 can't decode the byte \xe9” with the character 'é' for example). I need to keep the accents but I don’t find a solution to do it. It may seem simple but I’m in an early learning phase of programming and I'm often stuck on little things like that. Any help would be useful. Thanks, Louhike -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: protobuf not handling special characters between Java server and C++ client
On Tue, Jan 25, 2011 at 8:57 PM, Hitesh Jethwani hjethwa...@gmail.comwrote: if on the stream writer, I add something like: writer.write(new String(msg.getBytes(), UTF8).getBytes()) instead of simply writer.write(msg.getBytes()), I see the characters as expected on the C++ client. However this I believe messes up with the protobuf headers, so on C++ I receive only a partial file upto the entry that contains one such character. Still not sure on the above though. The reason this appears to work is because String.getBytes() encodes in ISO-8859-1 encoding by default. This encoding represents each character as exactly one byte, and can only represent character codes U+ through U+00FF. Since you are decoding the bytes as UTF-8 and then encoding them as ISO-8859-1, and since the character 'É' happens to be within the ISO-8859-1 range, you effectively decoded this character into a single byte. On the C++ side, the protobuf library does not verify that the parsed bytes are actually valid UTF-8 (except in debug mode); it just passes them through. So the string you see there includes the 'É' character as one byte. However, you end up getting a parser error because the length of the string (in bytes) ends up being different from the length given in the encoded message. The length was originally computed with 'É' represented as two bytes, but now it is only one byte, so the length is wrong. In general, decoding arbitrary bytes (like a protobuf) as if they were UTF-8 will lose information, so converting bytes - UTF-8 - bytes will corrupt the bytes. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Zig Zag Encoding (vs two's complement encoding)
The reason ZigZag encoding evolved as it did is because it can be applied as a separate layer on top of varint encoding. Originally, ZigZag was not officially part of the protobuf spec, but just something some people did manually to encode negative numbers efficiently. Eventually we made it official. Your encoding would work equally well, but cannot really be implemented independently of varint itself, therefore could not have evolved into existence in this way. On Fri, Jan 14, 2011 at 12:07 AM, David Srbecky dsrbe...@gmail.com wrote: 126 is 0111 1110 in binary. We truncate it form the left to groups of seven: 000 110 and then encode it as usual (least significant group first): 1 110 0 000. It takes two bytes to encode, but so does the zig-zag encoding. In fact, the zig-zag encoding and the two's complement encoding always take the same number of bytes to encode a given number. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] best way to create message from prototype + data
Yep, that's the best thing to do right now.
We're thinking of introducing a separate Parser interface, which would
allow you to do:
message = prototype.getParserForType().parse(data);
On Tue, Jan 18, 2011 at 7:34 AM, koert koertkuip...@gmail.com wrote:
A very common situation for me is when i get handed a Message
prototype plus a byte[] data, and then i am supposed to create the
Message object from it.
Since my prototype is just a Message i cannot use parseFrom(), and i
have to resort to:
try {
message = prototype.newBuilderForType().mergeFrom(data).build();
} catch (InvalidProtocolBufferException e2) {
// do something
} catch (UninitializedMessageException e3) {
// do something
}
Is this the right way to go about?
Documents seem to suggest to first do builder =
newBuilderForType().mergeFrom(), then check if it builder is
initialized, and then do build(). However this is more verbose and i
do not see the benefit.
Best Koert
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: 2.4.0 release candidate #1
I tested the release on MSVC 2005 and FreeBSD. Both worked fine. I also tested this on my home machine running stock Ubuntu 10.10, and I got the same error we've been seeing on our corp workstations. Thus it seems this problem is not an issue with our corp setup as I had initially thought. Instead, it's either an Ubuntu thing or a Python 2.6 thing... $ python setup.py test --- This script requires setuptools version 0.6c9 to run (even to display help). I will attempt to download it for you (from http://pypi.python.org/packages/2.6/s/setuptools/), but you may need to enable firewall access for this script first. I will start the download in 15 seconds. (Note: if this machine does not have network access, please obtain the file http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c9-py2.6.egg and place it in this directory before rerunning this script.) --- Downloading http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c9-py2.6.egg Generating google/protobuf/descriptor_pb2.py... Generating google/protobuf/compiler/plugin_pb2.py... /usr/lib/python2.6/distutils/dist.py:266: UserWarning: Unknown distribution option: 'namespace_packages' warnings.warn(msg) /usr/lib/python2.6/distutils/dist.py:266: UserWarning: Unknown distribution option: 'test_suite' warnings.warn(msg) usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: setup.py --help [cmd1 cmd2 ...] or: setup.py --help-commands or: setup.py cmd --help error: invalid command 'test' -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: Problem with java generation when using imports
I get:
// optional .KmObject kmobject = 1;
public static final int KMOBJECT_FIELD_NUMBER = 1;
private boolean hasKmobject;
private KmObjectProtos.KmObject kmobject_ =
KmObjectProtos.KmObject.getDefaultInstance();
public boolean hasKmobject() { return hasKmobject; }
public KmObjectProtos.KmObject getKmobject() { return kmobject_; }
On Wed, Jan 12, 2011 at 3:23 AM, Meghana meghana.vishwan...@gmail.comwrote:
Thanks for the reply Kenton.
Here's the example -
proto files :
option java_outer_classname = KmObjectProtos;
message KmObject {
optional string id = 1;
optional string eTag = 2;
optional string eTagContents = 3;
optional string cDate = 4;
optional string uDate = 5;
optional bool sysInternal = 6;
}
option java_outer_classname = ArtifactProtos;
import KmObject.proto;
message Artifact {
optional KmObject kmobject = 1;
optional string artifactType = 2;
optional string contentHash = 3;
}
Generated Java -
// optional .KmObject kmobject = 1;
public static final int KMOBJECT_FIELD_NUMBER = 1;
private boolean hasKmobject;
private KmObject.KmObject kmobject_;
public boolean hasKmobject() { return hasKmobject; }
public KmObject.KmObject getKmobject() { return kmobject_; }
I was expecting it to be KmObjectProtos.KmObject.
On Jan 12, 2:26 am, Kenton Varda ken...@google.com wrote:
Can you provide a small, self-contained example?
On Mon, Jan 10, 2011 at 7:13 AM, Meghana meghana.vishwan...@gmail.com
wrote:
Hi,
I am using imports in my proto files. So, B has an import of A. When I
generate the java code for B, all references to A are specified as A.A
i.e. A(dot)A. It does not seem to be picking up the
java_outer_classname from A.
Any idea what's going wrong?
Thanks.
--
You received this message because you are subscribed to the Google
Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
protobuf%2bunsubscr...@googlegroups.comprotobuf%252bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Using the --proto_path option
On Tue, Jan 11, 2011 at 1:30 PM, Collin blackfr...@gmail.com wrote: I am trying to understand how --proto_path works. Let's say in my .proto file have the following import lines: import descriptor.proto; import google/protobuf/descriptor.proto; Delete the first of these two imports, and then use --proto_path=/usr/local/include. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: Truncated varint decoding a message just built
I see, I think the encoding problem was fixed in 2.3.0. The type checker is still wrong, though. On Thu, Jan 6, 2011 at 3:17 PM, BigBaaadBob bigbaaad...@gmail.com wrote: 2.2.0a-0.1ubuntu1 On Dec 28 2010, 2:55 pm, Kenton Varda ken...@google.com wrote: Strange. It looks like there is a bug in the type checker for booleans -- it will accept any integer. However, this doesn't explain the encoding error. Looking at the code, I can't see how this could happen; any non-zero integer value should result in 1 being written to the wire. The problem you describe sounds like an obscure problem that happens in C++, but I can't see why it would happen in Python. What version of protobufs are you using? On Thu, Dec 23, 2010 at 10:10 PM, BigBaaadBob bigbaaad...@gmail.com wrote: Turns out it is simple: put a big number into a bool. The encoding side doesn't complain, but the decoder does. A bug on my part, but suboptimal behavior for the library. On Dec 22, 6:09 pm, Kenton Varda ken...@google.com wrote: Can you provide a small, self-contained example program demonstrating this problem? On Wed, Dec 22, 2010 at 4:52 PM, BigBaaadBob bigbaaad...@gmail.com wrote: I have Python code doing something like this: m.whatever = somevalue; etc. s = m.SerializeToString() m.Clear() m.ParseFromString(s) And the m.ParseFromString throws google.protobuf.message.DecodeError: Truncated varint. (And this is the only error thrown! If I print m I don't get any errors.) It seems like protobuf should be able to read something it successfully creates, no? Suggestions? -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com protobuf%2bunsubscr...@googlegroups.c om protobuf%2bunsubscr...@googlegroups.c om . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com protobuf%2bunsubscr...@googlegroups.c om . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Problem with java generation when using imports
Can you provide a small, self-contained example? On Mon, Jan 10, 2011 at 7:13 AM, Meghana meghana.vishwan...@gmail.comwrote: Hi, I am using imports in my proto files. So, B has an import of A. When I generate the java code for B, all references to A are specified as A.A i.e. A(dot)A. It does not seem to be picking up the java_outer_classname from A. Any idea what's going wrong? Thanks. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] LIBPROTOBUF_EXPORT Missing Headers
OK. What is the problem? On Mon, Jan 10, 2011 at 11:46 AM, Collin blackfr...@gmail.com wrote: The following headers have LIBPROTOBUF_EXPORT in them but are installed with `make install.` google/protobuf/stubs/strutil.h google/protobuf/stubs/substitute.h -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Using a ByteBuffer instead of a ByteString?
On Mon, Jan 10, 2011 at 9:45 PM, Nicolae Mihalache xproma...@gmail.comwrote: Hello, I recently started to use GPB, great software! :) But I have noticed in java that it is impossible to create a message containing a bytes fields without copying some buffers around. For example if I have a encoded message of 1MB with a few regular fields and one big bytes field, decoding the message will make a copy of the entire buffer instead of keeping a reference to it. We are actually looking at fixing this by allowing ByteStrings to share buffers. Even worse when encoding: if I read some data from file, does not seem possible to put it directly into a ByteString so I have to make first a byte[], then copy it into the ByteString and when encoding, it makes yet another byte[]. ByteString provides multiple methods of construction. One is to copy from a byte array. Another is to use an OutputStream that writes into a ByteString. In future versions, we are looking at making it possible to concatenate ByteStrings without a copy. But yes, if you start with a byte[], and you want a ByteString with the same content, you are going to need to make a copy, because ByteString has to guarantee immutability. So my question: is it possible to make an exception from the immutability for the bytes fields and use java.nio.ByteBuffers instead of ByteStrings? No, sorry, making any exception to immutability would end up unraveling the whole library. You can go from ByteString to ByteBuffer without a copy (by calling asReadOnlyByteBuffer()), but you can't go the other way, because there is no way to know given a ByteBuffer pointer whether or not someone might be able to modify it in the future. Storing ByteBuffer in message objects directly has additional problems. ByteBuffer is a stateful class -- it maintains a pointer to the current read location, for example. So a protocol message object with ByteBuffers inside it would be thread-hostile no matter how you look at it. This just leads to too many problems... -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: 2.4.0 release candidate #1
On Wed, Jan 5, 2011 at 10:33 PM, Scratch scratch@gmail.com wrote: Thanks! That's right what I needed. And one more question - will sub-builders be introduced for lite package? Sorry, but probably not. The whole point of the lite package is to be as small as possible, and the sub-builders stuff creates a significant amount of code bloat. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Linker issues: undefined reference to `a_message__get_packed_size'
The author of protobuf-c might not read this list. You might want to e-mail
him directly.
But I would suggest looking at the contents of tut.pb-c.c to see whether it
actually defines a_message__pack. Maybe the problem will be obvious.
On Tue, Dec 28, 2010 at 3:08 PM, Suhail Doshi digitalwarf...@gmail.comwrote:
I am using the protobuf-c extension out there.
Getting the following errors:
gcc tut.c tut.pb-c.c -o tut -lprotobuf-c -lprotobuf -lpthread
/tmp/ccBHwTBl.o: In function `main':
tut.c:(.text+0xab): undefined reference to
`a_message__get_packed_size'
tut.c:(.text+0xcc): undefined reference to `a_message__pack'
collect2: ld returned 1 exit status
tut.c
#include tut.pb-
c.h
#include
stdio.h
#include
stdlib.h
int main(int argc, char **argv)
{
AMessage msg =
AMESSAGE__INIT;
void
*buf;
unsigned
len;
if (argc 2 || argc 3)
{
fprintf(stderr, usage: pack a [b]
\n);
return
1;
}
msg.a =
atoi(argv[1]);
if (argc == 3)
{
msg.has_b =
1;
msg.b =
atoi(argv[2]);
}
len =
a_message__get_packed_size(msg);
buf =
malloc(len);
a_message__pack(msg,
buf);
fwrite(buf, len, 1,
stdout);
return
0;
}
tut-pb-c.h
--
/* Generated by the protocol buffer compiler. DO NOT EDIT! */
#ifndef PROTOBUF_C_tut_2eproto__INCLUDED
#define PROTOBUF_C_tut_2eproto__INCLUDED
#include google/protobuf-c/protobuf-c.h
PROTOBUF_C_BEGIN_DECLS
typedef struct _AMessage AMessage;
/* --- enums --- */
/* --- messages --- */
struct _AMessage
{
ProtobufCMessage base;
int32_t a;
protobuf_c_boolean has_b;
int32_t b;
};
#define AMESSAGE__INIT \
{ PROTOBUF_C_MESSAGE_INIT (amessage__descriptor) \
, 0, 0,0 }
/* AMessage methods */
void amessage__init
(AMessage *message);
size_t amessage__get_packed_size
(const AMessage *message);
size_t amessage__pack
(const AMessage *message,
uint8_t *out);
size_t amessage__pack_to_buffer
(const AMessage *message,
ProtobufCBuffer *buffer);
AMessage *
amessage__unpack
(ProtobufCAllocator *allocator,
size_t len,
const uint8_t *data);
void amessage__free_unpacked
(AMessage *message,
ProtobufCAllocator *allocator);
/* --- per-message closures --- */
typedef void (*AMessage_Closure)
(const AMessage *message,
void *closure_data);
/* --- services --- */
/* --- descriptors --- */
extern const ProtobufCMessageDescriptor amessage__descriptor;
PROTOBUF_C_END_DECLS
#endif /* PROTOBUF_tut_2eproto__INCLUDED */
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: protobuffer suport composite object in stack not in heap?
On Tue, Jan 4, 2011 at 5:25 PM, Igor Gatis igorga...@gmail.com wrote: A while ago, a colleague had a memory leak reusing a PB message which contained a repeated field. If I'm not mistaken the problem was that pb_message::Clear() calls vectorsomething::clear() and string::clear() which does not really release the memory allocated. I can't really tell for sure actually. @Kenton, does that make any sense? If yes, is there a way to avoid it? As Evan says, this is by design. The memory is not leaked -- it will be reused when the message object is reused, and deleted when the message object is deleted. The actual problem that your colleague probably observed is that if you happen to parse one message which is much larger than usual, the object will allocate a bunch of memory for that one large message, and then will keep it around even after parsing smaller messages. So your memory usage is determined by the largest message you parse, rather than by the average. You can also run into problems if you have a message type whose instances vary widely in shape. E.g. if type Foo has optional fields of type Bar and Baz, and you parse one instance of Foo that contains a Bar, then reuse the Foo to parse a message containing a Baz, then the Foo has allocated both Bar and Baz and will hold on to them. Thus the Foo is actually using more memory than was needed for either of the two messages it parsed. In practice these problems can manifest as memory usage that monotonically increases over the life of the process, although the rate of increase slows over time. A way to avoid this problem is to call SpaceUsed() to find out how much memory the object is using at any particular time. Once it crosses some threshold, delete the object and create a new one. Another approach is to reuse each objects at most N times -- this saves most of the allocation costs while preventing memory usage from growing without bound. Of course, all of this applies *only* to C++. Java protobuf objects are not reusable (since they are immutable), and in Python memory is discarded on Clear(). -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: Transfer protocol buffer object to POJO java object
No, the camel-case name is not stored anywhere. You will need to construct
it yourself. If performance is a concern, just cache the results in a
MapFieldDescriptor, String.
On Wed, Dec 29, 2010 at 8:11 AM, Sean Nguyen sontran...@gmail.com wrote:
I am writing a utility base on your suggestion and I run into a
problem of getting a camel name out of a FieldDescriptor.
My proto message is like this:
message Person
{
optional string last_name = 1;
optional string first_name = 2;
}
When java object is generated it uses camel field name as lastName,
firstName. Is there a way to get the camel field name out? I don't
want to get the original field name name : last_name, first_name and
convert it again to camel field name every time that I want to do
getter and setter on java pojo (very inefficient).
Thanks,
On Dec 28, 5:16 pm, Kenton Varda ken...@google.com wrote:
I don't know of any existing tools for this. You could write code that
does
this via reflection (protobuf reflection on the protobuf object, and
basic
java reflection on the POJO). Or, you could write a protoc plugin which
generates the code you need, though that will be a lot more complicated.
On Tue, Dec 28, 2010 at 3:55 PM, Sean Nguyen sontran...@gmail.com
wrote:
Hi,
I have a protocol buffer object like:
message PBPerson
{
optional string lastName = 1;
optional string firstName = 2;
}
I also have another java object
class JavaPerson
{
private String lastName;
private String firstName;
// setter and getter
}
I want to convert from PBPerson to JavaPerson and vice versa and I
don't want to do it manually by writing getter and setter for each
fields because my object can have more than 10 fields. Is there a
utility from protocol buffer that helps me doing that. So the would
expect a utility class that does something like:
JavaPerson javaPerson = PBConverter.convert(PBPerson pbPerson);
PBPerson pbPerson = PBConverter.convert(JavaPerson javaPerson);
Thanks,
--
You received this message because you are subscribed to the Google
Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
protobuf%2bunsubscr...@googlegroups.comprotobuf%252bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: Truncated varint decoding a message just built
Strange. It looks like there is a bug in the type checker for booleans -- it will accept any integer. However, this doesn't explain the encoding error. Looking at the code, I can't see how this could happen; any non-zero integer value should result in 1 being written to the wire. The problem you describe sounds like an obscure problem that happens in C++, but I can't see why it would happen in Python. What version of protobufs are you using? On Thu, Dec 23, 2010 at 10:10 PM, BigBaaadBob bigbaaad...@gmail.com wrote: Turns out it is simple: put a big number into a bool. The encoding side doesn't complain, but the decoder does. A bug on my part, but suboptimal behavior for the library. On Dec 22, 6:09 pm, Kenton Varda ken...@google.com wrote: Can you provide a small, self-contained example program demonstrating this problem? On Wed, Dec 22, 2010 at 4:52 PM, BigBaaadBob bigbaaad...@gmail.com wrote: I have Python code doing something like this: m.whatever = somevalue; etc. s = m.SerializeToString() m.Clear() m.ParseFromString(s) And the m.ParseFromString throws google.protobuf.message.DecodeError: Truncated varint. (And this is the only error thrown! If I print m I don't get any errors.) It seems like protobuf should be able to read something it successfully creates, no? Suggestions? -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com protobuf%2bunsubscr...@googlegroups.c om . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Stronger introspection
You can use the type's descriptor to inspect the fields it contains and what
all their types are. MyType.DESCRIPTOR is the descriptor for MyType. Once
you've figured out what field you want to read/write, use getattr() /
setattr().
Note that the C++ and Java interfaces for this are somewhat cleaner and more
mature than the Python interface. But you should be able to do it in Python
too.
On Thu, Dec 23, 2010 at 10:33 PM, BigBaaadBob bigbaaad...@gmail.com wrote:
I have a couple of applications where I need to do some generic coding
with protobufs. In one case I have a gui that needs to display
selected contents out of a message, and in another case I need to
transcode from a different message format into a protobuf. It is
trivial to hand-write hardcoded stuff for both of these, but it
violates the DRY principle, and is error prone, etc.
What I want to do in both cases (the gui) is, by conventionally using
the same names for the gui or other message fields as the fields in
the protobuf messages, use introspection to do the field mapping.
So, for example, given a list of PyQT widgets, I would use the widget
names to go find a like-named field in the protobuf message. But
since the protobuf message is likely to be composite, and in fact it
might be a subfield I need, I have to do some serious introspection.
Also, I want to do this kind of at init time BEFORE I receive a
message.
For example, I might have a PyQT widget named tt_m1, and I want to get
the .temperatures.tt_m1.val field out of the following protobuf
message structure by setting up a mapping between the widget and the
protobuf field at init time. I'd like to do similar things in the
opposite direction: given a selected set of protobuf fields, transcode
from another message into protobuf contents.
Suggestions?
message Analog {
required double val = 1;
enum AnalogStatus {
SENSOR_NOMINAL = 0;
SENSOR_UNDER_LIMIT = 1;
SENSOR_OVER_LIMIT = 2;
SENSOR_FAULT = 3;
SENSOR_NO_DATA = 4;
}
required AnalogStatus status = 2;
optional double limit = 3;
}
message Temperatures {
optional Analog tt_m1 = 1;
optional Analog tt_m2 = 2;
optional Analog tt_m3 = 3;
optional Analog tt_m4 = 4;
}
message StatusUpdate {
...stuff...
optional Temperatures temperatures = 7;
...stuff...
}
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: Converting JSON Text Into PB Bytes Type
You should probably contact the authors of protobuf-java-format; I'm not
sure if they pay attention to this list.
Base64 is the best way to encode arbitrary (non-text) data as text.
However, it's really up to the JSON converter code you are using to decide
what format to use. As far as I know, JSON does not specify any particular
encoding for arbitrary bytes.
On Sat, Dec 25, 2010 at 2:40 AM, yalmasri y.alma...@gmail.com wrote:
Thanks for the reply.
I encoded Hello world in Base64 and I got this SGVsbG8gd29ybGQ=.
That doesn't look close to the format I sent in the question. Here's
the code snippet I used:
import org.apache.commons.codec.binary.Base64;
public class Codec {
public static void main(String[] args) {
try {
String clearText = Hello world;
// Base64
System.out.println(Encoded: +
Base64.encodeBase64String(clearText.getBytes()));
} catch (Exception e) {
e.printStackTrace();
}
}
}
I'm using this: http://code.google.com/p/protobuf-java-format/ for
JSON -- PB conversion
On Dec 23, 3:48 am, Kenton Varda ken...@google.com wrote:
You will probably need to base64-encode bytes fields before embedding
them
in JSON, since JSON is a text format.
Which JSON - protobuf converter are you using? Or did you write your
own?
On Tue, Dec 21, 2010 at 11:41 PM, yalmasri y.alma...@gmail.com wrote:
Hello,
One of our customers is still using legacy Java 1.4 and therefore
could not use PB. For that we created an intermediate service that
receives text messages in plain JSON format then convert them into
generated PB objects.
In one object we have a field named desc of type bytes which will
hold a textual description of the arriving message. When the customer
wants to send English text for desc the value in JSON will be plain
readable English, but if non-ASCII is to be used, how the value in
JSON will look like? It look to me something like:
\0069\0062\006J\0062\006J\000 \006\'\006D\0069\006E\006J\006D\006\f
\000 \006D\006B\006/\000 \006\'\006F\006*\006G\006I\000
But I really don't know how to get this out of non-ASCII characters.
Can anyone help?
--
You received this message because you are subscribed to the Google
Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
protobuf%2bunsubscr...@googlegroups.comprotobuf%252bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] how can we deal with multi-lingual string? (c++)
The dirty little secret of std::wstring is that it does not actually deal with non-ASCII characters on all platforms. On some platforms wchar_t is 8-bit just like char! You should avoid using wstring and wchar_t for this reason; define your own types that are exactly what you need. For protocol buffers, we take the convention of using regular 8-bit chars but always using UTF-8 encoding. On Sat, Dec 25, 2010 at 8:10 PM, alcohol alcoho...@gmail.com wrote: without std::wstring support, how can we deal with strings consists of Ascii, Chinese,Japanese, Korean characters? -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] C# client to Java based server
Protocol buffers itself has no built-in RPC implementation. You have to find an RPC implementation that supports whatever languages you are interested in, or write your own. It's not too hard to write a simple RPC implementation given protocol buffers as a base. Sending protobufs over HTTP is a popular solution that lets you leverage existing infrastructure. On Sun, Dec 26, 2010 at 11:55 PM, Tommy tommyhan...@gmail.com wrote: Hello, I originally was using JAX-WS to communicate between client and server. This works well when my client is C# and server is Java based service. I have been asked to speed up the performance using Protocol Buffers to do binary serialization instead of text-based serialization. Knowing that I can no longer use Soap and had to figure out another transport mechanism. Is RPC the best way? Can I use RPC when my client and server are using C# and Java respectively? I noticed the example of CXF-protobuf link that shows how you can make the connection between Java and Java using RPC, but nothing about if the Client is C#. thanks, Tommy -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Transfer protocol buffer object to POJO java object
I don't know of any existing tools for this. You could write code that does
this via reflection (protobuf reflection on the protobuf object, and basic
java reflection on the POJO). Or, you could write a protoc plugin which
generates the code you need, though that will be a lot more complicated.
On Tue, Dec 28, 2010 at 3:55 PM, Sean Nguyen sontran...@gmail.com wrote:
Hi,
I have a protocol buffer object like:
message PBPerson
{
optional string lastName = 1;
optional string firstName = 2;
}
I also have another java object
class JavaPerson
{
private String lastName;
private String firstName;
// setter and getter
}
I want to convert from PBPerson to JavaPerson and vice versa and I
don't want to do it manually by writing getter and setter for each
fields because my object can have more than 10 fields. Is there a
utility from protocol buffer that helps me doing that. So the would
expect a utility class that does something like:
JavaPerson javaPerson = PBConverter.convert(PBPerson pbPerson);
PBPerson pbPerson = PBConverter.convert(JavaPerson javaPerson);
Thanks,
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Sorting Protocol Buffers with Hadoop MapReduce
Hmm, you might want to consider parsing both messages into DynamicMessages (or into the generated message classes if you have them compiled in) and then using a protobuf-reflection-based algorithm (not to be confused with Java reflection; see Message#getField()). Your current code is very much coupled to the wire format, and relies on tags being in a particular order (which is not technically guaranteed). On the other hand, your code is probably faster than a reflection-based algorithm would be. On Thu, Dec 16, 2010 at 3:18 PM, Owen O'Malley omal...@apache.org wrote: All, I'm hooking in ProtoBuf (as well as Avro, and Thrift) into Hadoop MapReduce. In order for that to make sense, I need to be able to sort on the protobuf messages. Hadoop uses compare function over the bytes of two serialized objects. Obviously, I could just use a memcmp, but that will lead to a sort order that is hard to explain to users. This function should lead to the obvious sort order, which will be much easier to understand. The rough idea is that I iterate over the fields sorted to be in id order and compare them based on their type. If a message is missing a field that has a default value, the default value is used. The code is here: http://bit.ly/f8Scfo It compiles and works in simple testing. I need to do more testing, but I thought I'd see if anyone here would be willing to take a look at my code for correctness and soundness. -- Owen -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Why not allow null values in Java for scalar types?
There are problems with this:
1) Boxing and unboxing primitives is relatively expensive, compared to just
passing them as primitives. If performance matters to you at all (and for
many protobuf users, it does), you probably don't want this.
2) If you accept messages from untrusted sources, your resulting code will
be more prone to security problems. For example, say you write some code
where the field foo is declared optional but, in practice, is always set.
It's likely that you're going to end up with some code that *assumes* that
it is set, and so doesn't check for null. This code may pass all your tests
because people unfortunately don't usually test against invalid inputs. If
some malicious user then sends you a message missing that field, your code
is going to crash. The protobuf design avoids that by having the getters
return a default value if the field is not set, so if you forget to check,
it's not a huge deal.
On Fri, Dec 17, 2010 at 8:25 AM, James james.j.re...@gmail.com wrote:
I am new to using Protocol Buffers and was really surprised why a
compiler option is not available to control how scalar types are
mapped in Java when generating the message objects. I understand the
concept of having a null int in C++ is not possible, but isn't is
really backwards to have to call another method in order to check if a
value has been set in Java? Having a null reference in Java does
exactly this!
In reading the reference docs:
http://code.google.com/apis/protocolbuffers/docs/reference/java-generated.html
The compiler will generate the following accessor methods in both the
message class and its builder:
* bool hasFoo(): Returns true if the field is set.
* int getFoo(): Returns the current value of the field. If the
field is not set, returns the default value.
Why not give protoc another control flag that would map all primitive
data types to their corresponding objects? (i.e. This would just map
a int - Integer or double - Double or a long - Long) This surely
would not break the protocol, but would give a much cleaner and Java
like way of programming. I just feel like I am working in Java but
with a C++ like interface. Java now has auto-boxing so really if
you like working with int and doubles instead of the Object
counterparts, you would still be okay...
Wouldn't you rather code like a Java Programmer by writing:
if(getFoo() != null)
{
// do something with getFoo()
}
instead of:
if(hasFoo())
{
// do something with getFoo()
}
Think of how this works when you are not trying to pass along the data
to another method that already takes an Integer object and that method
is ok with the reference being null.
I rather write:
doSomethingWithFoo(getFoo());
instead of
doSomethingWithFoo(hasFoo() ? getFoo() : null);
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] ambiguous Builder errors when compiling a Maven project
I've never seen this error. It doesn't make much sense -- protobuf
generated code should not contain any imports. Are you modifying the
generated code in some way?
On Sun, Dec 19, 2010 at 11:47 PM, Xiao Ling bardm...@gmail.com wrote:
Hi All,
I have a generated protobuf java file which compiles alone under both
ECJ and Oracle jdk. When the file is compiled together with other Java
and Scala files in a Maven project, the compilation keeps outputing
some error messages as follows:
DocumentProtos.java:1092: error: reference to Builder is ambiguous;
it is both defined in object Document and imported subsequently by
import Mention._
public Builder newBuilderForType() { return newBuilder(); }
^
DocumentProtos.java:1096: error: reference to Builder is ambiguous;
it is both defined in object Document and imported subsequently by
import Mention._
public Builder toBuilder() { return newBuilder(this); }
Does anybody have any idea why so?
Thanks,
-XL
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Reading and writing multiple messages to a file
You can only have one CodedInputStream wrapping a ZeroCopyInputStream at a
time, otherwise they will confuse each other with buffering conflicts.
CodedInputStreams are cheap to construct, so you can just make one on the
stack in a block, like:
{
CodedInputStream coded_input(input_stream_p.get());
success = coded_input.ReadLittleEndian64(size);
}
On Tue, Dec 21, 2010 at 6:56 AM, rodrigo benenson
rodrigo.benen...@gmail.com wrote:
Hello all !
I am trying to save multiple Protocol Buffer messages into a file and then
reading them back.
All messages (except the first) have the same type.
My code now works. But I had to use a workaround instead. The code looks as
follows.
boost::uint64_t size;
const bool read_size_success =
input_coded_stream_p-ReadLittleEndian64(size);
bool read_data_success = false;
const bool use_zero_copy_stream = false;
if(use_zero_copy_stream)
{
read_data_success =
data.ParseFromBoundedZeroCopyStream(input_stream_p.get(),
static_castint(size));
}
else
{ // work around
std::string buffer;
input_coded_stream_p-ReadString(buffer, size);
read_data_success = data.ParseFromString(buffer);
}
if (read_size_success == false or read_data_success == false)
{
throw std::runtime_error(Failed to read a data message during
DataSequenceDataType::read);
}
The code above works fine when using use_zero_copy_stream == falsehowever
when I set
use_zero_copy_stream to true then the read fails (read_data_success ==
false).
Obviously input_coded_stream_p was constructed using input_stream_p.
Why is ParseFromBoundedZeroCopyStream not working as expected ? What is
the proper way of using it ?
Thanks for your answers.
Best regards,
rodrigob.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] ambiguous Builder errors when compiling a Maven project
Sorry, I don't know how the scala plugin works. It sounds like it has a
bug. Perhaps it generates its own code based on the Java code, and somehow
that code ends up broken given this particular input...
On Wed, Dec 22, 2010 at 4:39 PM, Xiao Ling bardm...@gmail.com wrote:
You're right. There is no imports in the code. The message was printed
out by maven. As said, the file compiles ok by its own.
Here is the situation:
I have both scala and java code in my project. As long as there is
some scala code calls the protobuf code (which is in java), the
maven-scala-plugin (that is reponsible for compiling both scala and
java code) gives the error messages.
Thanks,
--Xiao
On Wed, Dec 22, 2010 at 4:13 PM, Kenton Varda ken...@google.com wrote:
I've never seen this error. It doesn't make much sense -- protobuf
generated code should not contain any imports. Are you modifying the
generated code in some way?
On Sun, Dec 19, 2010 at 11:47 PM, Xiao Ling bardm...@gmail.com wrote:
Hi All,
I have a generated protobuf java file which compiles alone under both
ECJ and Oracle jdk. When the file is compiled together with other Java
and Scala files in a Maven project, the compilation keeps outputing
some error messages as follows:
DocumentProtos.java:1092: error: reference to Builder is ambiguous;
it is both defined in object Document and imported subsequently by
import Mention._
public Builder newBuilderForType() { return newBuilder(); }
^
DocumentProtos.java:1096: error: reference to Builder is ambiguous;
it is both defined in object Document and imported subsequently by
import Mention._
public Builder toBuilder() { return newBuilder(this); }
Does anybody have any idea why so?
Thanks,
-XL
--
You received this message because you are subscribed to the Google
Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Nullpointer Exception when Message.parseFrom(byte [])
Indeed, this was a bug that we fixed. On Wed, Dec 8, 2010 at 7:04 AM, jim horng jimho...@gmail.com wrote: Hi Kenton, here are the stacktrace, i found i'm using 2.1.0 java.lang.NullPointerException at com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:422) at com.google.protobuf.GeneratedMessage$Builder.parseUnknownField(GeneratedMessage.java:289) at com.trendmicro.mist.proto.MistMessage$MessageBlock$Builder.mergeFrom(MistMessage.java:621) at com.trendmicro.mist.proto.MistMessage$MessageBlock$Builder.mergeFrom(MistMessage.java:602) at com.trendmicro.mist.proto.MistMessage$MessageBlock$Builder.mergeFrom(MistMessage.java:504) at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:358) at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:351) at com.trendmicro.mist.Session.runProducer(Session.java:288) at com.trendmicro.mist.Session.run(Session.java:805) at java.lang.Thread.run(Thread.java:619) After I switch to 2.3.0, the exception become much more reasonable: Exception in thread main com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero). at com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:68) at com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:90) at test.Test$Person$Builder.mergeFrom(Test.java:279) at test.Test$Person$Builder.mergeFrom(Test.java:1) at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:244) at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:183) at com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:162) at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:661) at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:183) at com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:153) at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:654) at test.Test$Person.parseFrom(Test.java:119) at test.Test_pb_NPE.main(Test_pb_NPE.java:21) thanks On Wed, Dec 8, 2010 at 11:34 AM, Kenton Varda ken...@google.com wrote: The file you attached is not useful without source code. But what would be even better is if you could provide the stack trace for the NullPointerException. The parser should never throw NPE and I'm not aware of any bugs which cause it to throw NPE. BTW, it's very easy for bytes which are not actually a protocol buffer to parse correctly as a protocol buffer. The protobuf encoding is pretty dense. So I would suggest that you instead use some sort of tagging mechanism to distinguish protobuf data from other kinds of data that you need to process. On Mon, Dec 6, 2010 at 2:41 AM, jim horng jimho...@gmail.com wrote: Hi all, It occur to me that when I'm trying to parse any of the binary data files I listed below, using Message.parseFrom(byte []). it said nullpointer exception. To my best knowledge, I think when the raw bytes does not contain required field, protobuf API should return invalidprotobufexception when using .parseFrom(), or at least it will throw uninitializeprotobuf exception when .build(). so that client will know what happen and have change to handle them correctly. But if it return nullpointerexception, it will be hard to know what really happen, it can only be roughly assumed message parsing is failed. My goal is just to determine if the received binary files is an expected protobuf message or not. Please kindly advise or if this is work as design under some reason. Thanks a lot ! code snippet String filepath = np1.pb; byte[] file_byteary = getBytesFromFile(new File(filepath)); xxx.Connection.Builder conn_builder = xxx.Connection.newBuilder(); xxx.Connection MB_tmp = xxx.Connection.parseFrom(file_byteary); conn_builder.mergeFrom(MB_tmp); System.out.println(is init? + conn_builder.isInitialized()); assertTrue(conn_builder.build() != null); /code snippet * test data https://docs.google.com/leaf?id=0B9CkCymjWlZLNWMyMGNiNjEtMDJiZi00NDY0LTgzMWUtMjI1N2I0MWVhNjlhhl=en -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- Best Regards, Jim(洪懷謙) -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto
[protobuf] Allow protobuf-related job postings on this list?
Hi all, I received an e-mail from a recruiting company asking whether it would be OK to post a job listing on this mailing list. Apparently, they need someone experienced with protocol buffers. I was originally going to just say no, but on further thought I actually don't know what's appropriate. It is, after all, protobuf-related. I suppose it all depends on how you -- the list members -- feel. Should I allow it? (Obviously I would still reject non-protobuf-related solicitations.) Discuss. -Kenton -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] 2.4.0 and lazy UTF-8 conversions in Java
On Wed, Dec 1, 2010 at 3:33 AM, Evan Jones ev...@mit.edu wrote: The instanceof approach to switch between the two is a good idea. When I wrote my implementation, I was concerned about the thread-safeness issues, although I don't think I ever considered this particular version. However, I think this can be made thread-safe, even without volatile (although I only understand the JMM enough to be dangerous). Well, Jeremy Manson literally wrote the book on the Java memory model, and he says it works. :) -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Preferred Citation?
There's a (slightly outdated) list of contributors here:
http://code.google.com/p/protobuf/source/browse/trunk/CONTRIBUTORS.txt
But I think it's probably best to list Google as the author. I did not
invent protocol buffers; I just wrote version 2 and open sourced it, and I
had help.
I don't really know how citations work, though.
On Fri, Dec 3, 2010 at 7:25 PM, Dan Homerick danhomer...@gmail.com wrote:
I'm writing up a master's project MS which has used protobufs extensively.
Is there a preferred article for crediting protobufs with? If not, I could
cite it with something like (Bibtex):
@MISC{protobuf,
title={Protocol Buffers},
author={Kenton Varda},
howpublished={\url{http://code.google.com/apis/protocolbuffers/}},
}
but I'm sure that the authors list is woefully incomplete. Is there a more
complete list of authors available?
Alternatively, I could just put the project's URL in a footnote.
- Dan
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: java parse with class known at runtime (and compiled proto)
Evan is correct. The best way to write code which deals with a generic protobuf type is to have it take a default instance as a parameter. From that you can do everything else. This is actually better than passing around Class objects, because it allows users to use DynamicMessages with your code. Using Class objects forces users to use only generated types. Also, Java reflection may be slow or even unavailable on some platforms. On Mon, Dec 6, 2010 at 7:51 AM, Evan Jones ev...@mit.edu wrote: On Dec 6, 2010, at 10:31 , Koert Kuipers wrote: But that doesn't make a parseFrom() in message interface invalid, does it? Indeed some other information outside the raw bytes will be needed to pick to right Message subclass. But that's fine. Oh, sorry, I misunderstood your question, so my answer is somewhat invalid. One could then: 1) pick the right subclass of Message based upon some information outside the raw bytes (in my case something stored in a protobuf wrapper around the raw bytes) 2) call subclass.parseFrom(bytes) now we have to jump through more hoops for step 2 (create instance of Message subclass, newBuilderForType, mergeFrom, isInitialized, build) The MessageLite.Builder interface has a mergeFrom method that does what you want. What you should do is something like: * Get a MessageLite instance for the message type you want to parse (eg. something like MyMessageType.getDefaultInstance(), or MessageLite.getDefaultInstanceForType()) * Hold on to that MessageLite instance in some sort of registry. (HashMapInteger, MessageLite?) * When you get a message, look at the protobuf wrapper to determine the type. * Look up the prototype MessageLite instance in your registry. * Call prototypeInstance.newBuilderForType().mergeFrom(bytes).build() This only creates a single instance of the message each time. The .build() method will automatically check that the message is initialized, so you don't need to call isInitialized (although you may want to catch the exception it could throw?). This Builder pattern is used so that the Message objects are immutable. This means they can be passed between threads without requiring any synchronization. See: http://code.google.com/apis/protocolbuffers/docs/javatutorial.html#builders Hope this helps, Evan -- Evan Jones http://evanjones.ca/ -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] File already exists in database on Mac OSX
I assume this is happening at startup? Is it possible that you are somehow linking two copies of the same .pb.cc into your binary? There are cases where the linker doesn't catch this, especially when dynamic linking is involved, but sometimes even with static linking. Speaking of which, are you linking dynamically or statically? I am not aware of any outstanding issues that would cause this, but C++ initialization behavior can be crazy complicated and can even vary between platforms. On Mon, Dec 6, 2010 at 8:05 AM, Alex Nixon alex.nixon...@gmail.com wrote: Hello, I'm receiving a File already exists in database error from the protocol buffers library at runtime on Mac OSX, and I'm unsure as to what's causing it or where to look to isolate the problem. I have the exact same codebase compiling on Windows and Linux and they both work fine - so I wondered if this is a protocol buffers issue? That said, I was unable to find any outstanding issues in the bug tracker which match my symptoms. All of my .proto files have different names and are in the same directory. The generated .pb.h and .pb.cc files on Mac OSX are identical to those generated on Linux. Any advice would be much appreciated. Thanks in advance, - Alex Nixon -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Any Python wrappers for the C++ implementation?
On Tue, Dec 7, 2010 at 9:19 PM, Yang Zhang yanghates...@gmail.com wrote: Also, note that if you explicitly compile C++ versions of your messages and link them into the process, they'll be even faster. (If you don't, the library falls back to DynamicMessage which is not as fast as generated code.) I'm trying to decipher that last hint, but having some trouble - what exactly do you mean / how do I do that? I'm just using protoc --py_out=... and PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp. I'm not completely sure what I mean, because I don't have much experience with Python C Extensions. Basically I'm saying you should additionally generate C++ code using protoc, the compile that into a C extension (even with no interface), and then load it into your Python process. Simply having the C++ code for your message types present will make them faster. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Any Python wrappers for the C++ implementation?
Protobuf 2.4.0 will include an implementation of the Python API that is backed by C++ objects. The interface is identical to the existing Python API, and you can wrap it around existing C++ objects or have it construct its own. This code is already is SVN. Unfortunately the team is someone backlogged and we haven't been able to make a lot of progress on an official release. But it should be a lot easier to get the SVN code working than to write your own. :) On Tue, Nov 30, 2010 at 11:22 PM, Yang Zhang yanghates...@gmail.com wrote: Has anyone written (a tool for generating) Python wrappers around the C++ generated code and is willing to share this? I'm looking to do the same, so this would save me a bit of research time. (It's fine if it's not a general tool and this is specific to some schema.) Thanks! -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] 2.4.0 and lazy UTF-8 conversions in Java
To answer your questions:
1) It is thread-safe.
2) No locking or other synchronization overhead is involved.
The string data is stored as an Object pointer which, at any particular
time, may point at either a String or a ByteString. When one form or the
other is requested, we convert the representation to that form if necessary,
then return it. In other words:
String getFoo() {
Object ref = foo;
if (ref instanceof String) {
return (String) ref;
} else {
String str = ((ByteString) ref).toStringUtf8();
foo = str;
return str;
}
}
This is thread-safe because the Java memory model guarantees that pointer
writes are atomic, even for non-volatile pointers (otherwise you'd have a
potential security hole!). If two threads happen to try to access the field
at the same time, the worst that can happen is that they will both decode
the string independently. Since they'll get the same result, it doesn't
matter which of the two pointers is stored long-term. (Although, clients
that expect getters to consistently return the same object, as opposed to
just equal objects, may be broken. But people should not be doing that
anyway.)
We checked with Jeremy Manson to make sure this implementation works.
Working at Google is awesome like that. :)
On Tue, Nov 30, 2010 at 12:58 PM, Blair Zajac bl...@orcaware.com wrote:
I saw this change coming up in 2.4.0:
Added lazy conversion of UTF-8 encoded strings to String objects to
improve performance.
Is the lazyness thread safe?
Without looking at the implementation, then if it isn't thread safe, I
would guess this isn't much overhead, but if it is thread safe and you know
you're going to use all the string fields, then does it hurt performance
instead?
Regards,
Blair
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] 2.4.0 and lazy UTF-8 conversions in Java
On Tue, Nov 30, 2010 at 1:21 PM, Evan Jones ev...@mit.edu wrote: * I'll guess that it will not be slower if you access all the strings. Currently, the parsing process copies the raw bytes from the input buffer into an individual byte array, then converts that to a String. This is, sadly, the most efficient thing you can do, since you need special code to create Strings. Therefore, doing lazy conversion isn't going to be slower. The objects already have both byte[] and String fields for each string due to an encoding improvement I contributed, so this should be nearly a pure win. BTW, we actually ended up reverting your change and replacing it with the new implementation. We found that having two references increased memory pressure too much. I thought I had mentioned this to you; sorry if I forgot. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Encoding/Decoding extension
Seems fine to me.
One point: Rather than use ListFields(), I would recommend adding a type ID
field to Message, e.g.:
message Message {
optional int32 type_id = 1;
extensions 5 to max;
}
Reflection is kind of annoying to use, and can be somewhat slow.
On Thu, Nov 25, 2010 at 5:15 AM, Benoit Laurent laurent...@gmail.comwrote:
Hi,
I've done some experiment using protocol buffers extension but I kind
of wondering if I am doing things the right way. For the encoding I've
seen some explanation on the multi message encoding, and what I am
doing seems ok. On the other hand I am kind of suspicious about my
decoding routine.
my .proto file is pretty straightforward
message Message {
extensions 5 to max;
}
message Dummy {
optional float x = 1;
optional float y = 2;
}
message Foo {
optional int v = 1;
}
extend Message {
optional Dummy dummy = 5;
}
extend Message {
optional Foo foo = 50001;
}
My encoding code look like :
int fd = open(argv[1], O_CREAT|O_RDONLY|O_WRONLY);
google::protobuf::io::ZeroCopyOutputStream* raw_output = new
google::protobuf::io::FileOutputStream(fd);
google::protobuf::io::CodedOutputStream* coded_output = new
google::protobuf::io::CodedOutputStream(raw_output);
ns_message::Message message;
ns_message::Dummy* dummy =
message.MutableExtension(ns_message::dummy);
dummy-set_x(5);
int size = message.ByteSize();
coded_output-WriteVarint32(size);
message.SerializeToCodedStream(coded_output);
message.ClearExtension(ns_message::dummy);
ns_message::Foo* zoom = message.MutableExtension(csi_message::zoom);
[encoding the same way]
For the decoding here is what I am doing:
int fd = open(argv[1], O_RDONLY);
google::protobuf::io::ZeroCopyInputStream* raw_input = new
google::protobuf::io::FileInputStream(fd);
google::protobuf::io::CodedInputStream* coded_input = new
google::protobuf::io::CodedInputStream(raw_input);
ns_message::Message message;
google::protobuf::uint32 size;
while ( coded_input-ReadVarint32(size) ) {
int previous = coded_input-PushLimit(size);
message.ParseFromCodedStream(coded_input);
coded_input-PopLimit(previous);
std::vectorconst google::protobuf::FieldDescriptor* fields;
const google::protobuf::Reflection* reflection =
message.GetReflection();
reflection-ListFields(message, fields);
for (int i = 0; i fields.size(); ++i) {
if (fields[i]-is_extension()) {
getExtensionMessage(message, fields[i]);
}
}
}
getExtensionMessage doesn't do anything fancy for the moment
if(field-name() == dummy) {
csi_message::* Dummy = dummy.MutableExtension(ns_message::dummy);
print(*dummy);
}
Thanks in advance for your comments.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: building libprotobuf-lite for iPhone
On Tue, Nov 23, 2010 at 3:47 AM, Marcus Better mar...@better.se wrote:
On 1 Nov, 16:39, Marcus Better mar...@better.se wrote:
I wonder if anyone has built libprotobuf-lite for the iOS, and if
there are any special pitfalls?
Yes I have, and of course I will share my findings with the world for
the benefit of mankind.
:)
I also have a follow-up question: the libprotobuf-lite.a static
library is about 500 kB which seems a little heavy. Is this normal?
I've tried to strip the symbol information but maybe that didn't
succeed.
Last I checked it was more like 100k on gcc/x86/linux, but on another
compiler/arch/platform I suppose anything could happen. Note that if you
don't use extensions, you can drop extension_set.cc from the library which
saves a considerable amount of binary size.
We're using iOS SDK 4.0 with the PLBlocks compiler. The script needs
some adjustments if you are not using PLBlocks.
This produces the ARM library:
-
export ARCH=arm-apple-darwin10
export ARCH_PREFIX=$ARCH-
export PLATFORM=iPhoneOS
export SDKVER=4.0
export DEVROOT=/Developer/Platforms/${PLATFORM}.platform/Developer
export SDKROOT=$DEVROOT/SDKs/${PLATFORM}$SDKVER.sdk
export PKG_CONFIG_PATH=$SDKROOT/usr/lib/pkgconfig:$DEVROOT/usr/lib/
pkgconfig
export AS=$DEVROOT/usr/bin/as
export ASCPP=$DEVROOT/usr/bin/as
export AR=$DEVROOT/usr/bin/ar
export RANLIB=$DEVROOT/usr/bin/ranlib
export CPP=$DEVROOT/usr/bin/cpp
export CXXCPP=$DEVROOT/usr/bin/cpp
#export CC=$DEVROOT/usr/bin/arm-apple-darwin10-llvm-gcc-4.2
PLBLOCKS=/Developer/Library/PLBlocks/Compilers/gcc42-iphoneos3.2
export CC=$PLBLOCKS/usr/bin/${ARCH_PREFIX}gcc-4.2.1
#export CXX=$DEVROOT/usr/bin/arm-apple-darwin10-llvm-g++-4.2
export CXX=$PLBLOCKS/usr/bin/${ARCH_PREFIX}g++-4.2.1
#export LD=$DEVROOT/usr/bin/ld
export LD=$PLBLOCKS/usr/bin/ld
export STRIP=$DEVROOT/usr/bin/strip
export CPPFLAGS=-pipe -no-cpp-precomp -I$SDKROOT/usr/lib/gcc/${ARCH}/
4.2.1/include -I$SDKROOT/usr/include -I$DEVROOT/usr/include
export CFLAGS=-fmessage-length=0 -pipe -fpascal-strings -no-cpp-
precomp -miphoneos-version-min=3.1.3 --sysroot='$SDKROOT' -isystem
$SDKROOT/usr/lib/gcc/${ARCH}/4.2.1/include -isystem $SDKROOT/usr/
include -isystem $DEVROOT/usr/include
export CXXFLAGS=$CFLAGS -I$SDKROOT/usr/include/c++/4.2.1/${ARCH}/v6
export LDFLAGS=--sysroot='$SDKROOT' -L$SDKROOT/usr/lib -L$DEVROOT/usr/
lib
./configure --host=${ARCH} --with-protoc=protoc --enable-static --
disable-shared
--
And the i386 version for the simulator:
---
export ARCH=i686-apple-darwin10
export ARCH_PREFIX=${ARCH}-
export PLATFORM=iPhoneSimulator
export SDKVER=4.0
export DEVROOT=/Developer/Platforms/${PLATFORM}.platform/Developer
export SDKROOT=$DEVROOT/SDKs/${PLATFORM}$SDKVER.sdk
export PKG_CONFIG_PATH=$SDKROOT/usr/lib/pkgconfig:$DEVROOT/usr/lib/
pkgconfig
export AS=$DEVROOT/usr/bin/as
export ASCPP=$DEVROOT/usr/bin/as
export AR=$DEVROOT/usr/bin/ar
export RANLIB=$DEVROOT/usr/bin/ranlib
export CPP=$DEVROOT/usr/bin/cpp
export CXXCPP=$DEVROOT/usr/bin/cpp
#export CC=$DEVROOT/usr/bin/arm-apple-darwin10-llvm-gcc-4.2
PLBLOCKS=/Developer/Library/PLBlocks/Compilers/gcc42-macosx10.6
export CC=$PLBLOCKS/usr/bin/${ARCH_PREFIX}gcc-4.2.1
#export CXX=$DEVROOT/usr/bin/arm-apple-darwin10-llvm-g++-4.2
export CXX=$PLBLOCKS/usr/bin/${ARCH_PREFIX}g++-4.2.1
#export LD=$DEVROOT/usr/bin/ld
export LD=$PLBLOCKS/usr/bin/ld
export STRIP=$DEVROOT/usr/bin/strip
export CPPFLAGS=-pipe -no-cpp-precomp -I$SDKROOT/usr/lib/gcc/${ARCH}/
4.2.1/include -I$SDKROOT/usr/include -I$DEVROOT/usr/include
export CFLAGS=-fmessage-length=0 -pipe -fpascal-strings -no-cpp-
precomp -miphoneos-version-min=3.1.3 --sysroot='$SDKROOT' -isystem
$SDKROOT/usr/lib/gcc/${ARCH}/4.2.1/include -isystem $SDKROOT/usr/
include -isystem $DEVROOT/usr/include
export CXXFLAGS=$CFLAGS -I$SDKROOT/usr/include/c++/4.2.1/${ARCH}/v6
export LDFLAGS=--sysroot='$SDKROOT' -L$SDKROOT/usr/lib -L$DEVROOT/usr/
lib
./configure --with-protoc=protoc --enable-static --disable-shared
--
Cheers,
Marcus
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] [PATCH] Display non-present fields with TextFormat.
Hi Brenden, Thanks for your interest in improving protocol buffers. I can see how this option would be useful to some people. Unfortunately, there are a lot of options that different people find useful, and if we implemented all of them, we'd end up with a very boated library that no one wants to use. Thus, depressingly, half of my job as the maintainer of protocol buffers is refusing feature requests and contributions. I even have to reject contributions from other Google employees on a daily basis. It is actually pretty easy to write a custom TextFormat implementation with whatever modifications you want. The TextFormat code itself does not depend on any internal interfaces. Also, you can reuse the code to format individual values by calling TextFormat::PrintFieldValue(). So, what I generally tell people who want to change TextFormat is to write your own class similar to TextFormat that implements your alternative encoding. I think that would be appropriate in this case. Sorry. On Tue, Nov 23, 2010 at 5:04 PM, Brenden Matthews bren...@diddyinc.comwrote: Greetings, One of the best features of protocol buffers is the notion of presence. Another great feature is the ability to do 'message.PrintDebugString()'. One problem, however, is that fields which are not present do not get displayed when printing the debug string. Proposed solution: add an optional parameter to display optional fields and their default values (if applicable) even when they aren't present. Attached is a patch which implements this. Please provide comments/suggestions, and let me know if this is the right place for this. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: python enum values option
I see, the problem is that you've created a copy of descriptor.proto and are
extending that rather than extending the original file.
To correctly extend descriptor.proto, you need to import it like:
import google/protobuf/descriptor.proto;
The path name is important, because it determines which module your
generated code imports and extends. You want it to import
google.protobuf.descriptor_pb2 (which is included in the protobuf library),
not your own local copy of descriptor_pb2.
Note that when you install protocol buffers, a copy of descriptor.proto is
placed in $PREFIX/include/google/protobuf (where $PREFIX is usually /usr or
/usr/local). So you should invoke protoc like:
protoc -I. -I/usr/include ooo.proto
This way ooo.proto will be able to import google/protobuf/descriptor.proto,
which protoc will find in /usr/include.
On Thu, Nov 18, 2010 at 4:32 AM, Vsevolod Zadubrovsky zadubrov...@gmail.com
wrote:
Kenton,
I didn't find how to attach file in the group discussion, so am replying to
you directly.
In attach you will find the source proto file, the generated *pb2.py files,
and the test program enum_test.py
I used: google protobuf 2.3.0 and proto compiler 2.3.0 win32 from here:
http://code.google.com/p/protobuf/downloads/list
The test output is:
E:\projects\gpbpython enum_test.py
serialized request☺►☻:
Traceback (most recent call last):
File enum_test.py, line 13, in module ext =
opts.Extensions[proto.verbose_enum_value_option]
File
C:\Python27\lib\site-packages\protobuf-2.3.0-py2.7.egg\google\protobuf\reflection.py,
line 1068, in __getitem__
_VerifyExtensionHandle(self._extended_message, extension_handle)
File
C:\Python27\lib\site-packages\protobuf-2.3.0-py2.7.egg\google\protobuf\reflection.py,
line 229, in _VerifyExtensionHandle
message.DESCRIPTOR.full_name))
KeyError: 'Extension verbose_enum_value_option extends message type
google.protobuf.EnumValueOptions, but this message is of type
google.protobuf.EnumValueOptions.'
Thnk you very much for support
2010/11/17 Kenton Varda ken...@google.com
That is strange. Can you provide a minimal but complete program that
reproduces this problem, so I can look at it?
On Wed, Nov 10, 2010 at 12:42 AM, Vsevolod Zadubrovsky
zadubrov...@gmail.com wrote:
It seems to me, it doesn't work at all.
I got this:
(Pdb) import ooo
(Pdb) vals_by_nums =
req.DESCRIPTOR.fields_by_name['error'].enum_type.values_by_number[req.error]
(Pdb) options = vals_by_nums.GetOptions()
(Pdb) options.Extensions[ooo.verbose_enum_value_option]
*** KeyError: 'Extension verbose_enum_value_option extends message
type google.protobuf.EnumValueOptions, but this message is of type
google.protobuf.EnumValueOptions.'
I've checked the source code, and it looks like raised from here
(google.protobuf.reflection, line 224):
if extension_handle.containing_type is not message.DESCRIPTOR:
raise KeyError('Extension %s extends message type %s, but this
'
'message is of type %s.' %
(extension_handle.full_name,
extension_handle.containing_type.full_name,
message.DESCRIPTOR.full_name))
Any ideas ? may be it is already solved in trunk version?
Thanks
On Nov 10, 1:50 am, Kenton Varda ken...@google.com wrote:
You need to use the descriptor for the enum type. Unfortunately this
interface isn't very well fleshed-out in Python. I think you'd have to
write something like:
Request.DESCRIPTOR.fields_by_name['error'].enum_type.values_by_number[message.error].options.Extensions[verbose_enum_value_option]
I haven't checked that that's exactly correct, but it gives you an
idea. We
should probably improve this.
On Tue, Nov 9, 2010 at 4:12 AM, Vsevolod Zadubrovsky
zadubrov...@gmail.comwrote:
Hi, I'm stuck with getting the enum value option in Python. The proto
file is:
import descriptor.proto;
extend google.protobuf.EnumValueOptions {
optional string verbose_enum_value_option = 50005;
}
enum ErrorType {
OK = 1 [(verbose_enum_value_option) = OK];
SOME_ERROR = 2 [(verbose_enum_value_option) = Some Error
verbose
message];
}
message Request {
required bool success = 1;
optional ErrorType error = 2;
}
When I receive the Request message, I can access the 'error' field,
and its type is int, that's actually ok. But how can I get the
verbose_enum_value_option of 'error' field value ?
Thanks
p.s. The goal is to keep error types and their verbose error messages
in one place, available for every service in our project, so the
error
messages would've been identical.
--
You received this message because you are subscribed to the Google
Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf
Re: [protobuf] Re: omitting tag numbers
On Tue, Nov 9, 2010 at 10:42 PM, Christopher Smith cbsm...@gmail.comwrote: This aspect could be mostly mitigated by integrating a metadata header in to files. For systems with this kind of an approach look at Avro Hessian. Problems with that: 1) Protobufs are routinely used to encode small messages of just a few bytes. Metadata would almost certainly be larger than the actual messages in such cases. 2) This metadata would add an extra layer of indirection into the parsing process which would probably make it much slower than it is today. 3) Interpreting the metadata itself to build that table would add additional time and memory overhead. Presumably this would have to involve looking up field names in hash maps -- expensive operations compared to the things the protobuf parser does today. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: python enum values option
That is strange. Can you provide a minimal but complete program that
reproduces this problem, so I can look at it?
On Wed, Nov 10, 2010 at 12:42 AM, Vsevolod Zadubrovsky
zadubrov...@gmail.com wrote:
It seems to me, it doesn't work at all.
I got this:
(Pdb) import ooo
(Pdb) vals_by_nums =
req.DESCRIPTOR.fields_by_name['error'].enum_type.values_by_number[req.error]
(Pdb) options = vals_by_nums.GetOptions()
(Pdb) options.Extensions[ooo.verbose_enum_value_option]
*** KeyError: 'Extension verbose_enum_value_option extends message
type google.protobuf.EnumValueOptions, but this message is of type
google.protobuf.EnumValueOptions.'
I've checked the source code, and it looks like raised from here
(google.protobuf.reflection, line 224):
if extension_handle.containing_type is not message.DESCRIPTOR:
raise KeyError('Extension %s extends message type %s, but this
'
'message is of type %s.' %
(extension_handle.full_name,
extension_handle.containing_type.full_name,
message.DESCRIPTOR.full_name))
Any ideas ? may be it is already solved in trunk version?
Thanks
On Nov 10, 1:50 am, Kenton Varda ken...@google.com wrote:
You need to use the descriptor for the enum type. Unfortunately this
interface isn't very well fleshed-out in Python. I think you'd have to
write something like:
Request.DESCRIPTOR.fields_by_name['error'].enum_type.values_by_number[message.error].options.Extensions[verbose_enum_value_option]
I haven't checked that that's exactly correct, but it gives you an idea.
We
should probably improve this.
On Tue, Nov 9, 2010 at 4:12 AM, Vsevolod Zadubrovsky
zadubrov...@gmail.comwrote:
Hi, I'm stuck with getting the enum value option in Python. The proto
file is:
import descriptor.proto;
extend google.protobuf.EnumValueOptions {
optional string verbose_enum_value_option = 50005;
}
enum ErrorType {
OK = 1 [(verbose_enum_value_option) = OK];
SOME_ERROR = 2 [(verbose_enum_value_option) = Some Error
verbose
message];
}
message Request {
required bool success = 1;
optional ErrorType error = 2;
}
When I receive the Request message, I can access the 'error' field,
and its type is int, that's actually ok. But how can I get the
verbose_enum_value_option of 'error' field value ?
Thanks
p.s. The goal is to keep error types and their verbose error messages
in one place, available for every service in our project, so the error
messages would've been identical.
--
You received this message because you are subscribed to the Google
Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
protobuf%2bunsubscr...@googlegroups.comprotobuf%252bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: Swap() on Windows
On Wed, Nov 10, 2010 at 6:51 PM, Anand Ganesh anantharamangan...@gmail.comwrote: Hi Kenton, all, When you have a message which contains a string member variable, the variable is initialized to a default-value. So you have ::std::string *name_; static const ::std::string _default_name_; name_ = _default_name; Seems like since _default_name is declared 'const' the microsoft compiler is optimizing for cache localization and putting _default_name_ inside each individual object (like it would for a member variable). Hence _default_name_ has different addresses in two different objects. Functionally this is not a problem because there's no data-sharing (as in, one object modifying the static variable to a new value and others reading the new value from it) because it is declared const. This doesn't make sense. _default_name_ is only defined once, in the .pb.cc file. The compiler should only be compiling that file once, and therefore should only be producing one instance of the object. Other translation units which #include your .pb.h cannot possibly have their own definition of _default_name_ because they don't know what it should be initialized to. So the only way this is possible is if you are actually linking together two separate instances of the same .pb.o file. Perhaps you have statically linked your .pb.o into two different DLLs, and you are then trying to pass objects between them? This will not work -- they need to share a single instance of the object file. This problem is not unique to protocol buffers; it is generally unsafe to link two copies of the same class into one process and expect them to be compatible. You will either have to link the .pb.o file into a DLL that can be shared by the other two, or (probably safer) you could serialize the message before passing it across the DLL boundary and then parse it again on the other side. See also the warnings about DLLs in vsprojects/readme.txt. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] The generated java code crashes if I try to integrate it into an Eclipse Rich Client application
The generated code contains a big string literal which contains metadata about the types. The metadata is itself a protocol buffer, and is decoded at startup. It looks like it is somehow being corrupted in your environment. I don't know how that would happen. I recently wrote an Eclipse plugin that used protocol buffers and did not see any problems. That said, you can try adding this to your .proto file: option optimize_for = LITE_RUNTIME; This turns off a number of features, including self-description. Thus, the generated code will no longer contain an embedded descriptor at all. On Thu, Nov 11, 2010 at 11:00 AM, Alsmom2005 gundanu...@gmail.com wrote: Hi, I have problems integrating Java GPB generated code into an Eclipse Rich Client Platform Application. The same code works fine in a standalone java project but it crashes in the RCP environment at the first 'usage' of an enum value defined in GPB code: Exception in thread Thread-3 java.lang.ExceptionInInitializerError at com.google.protobuf.DescriptorProtos $FileDescriptorProto.clinit(DescriptorProtos.java:1687) at com.google.protobuf.Descriptors $FileDescriptor.internalBuildGeneratedFileFrom(Descriptors.java:338) at NGCMOSMsg.NGCMOSMessages.clinit(NGCMOSMessages.java:11255) at NGCMOSMsg.NGCMOSMessages$MOSEndpoints.clinit(NGCMOSMessages.java: 353) at communication.PostOffice.init(PostOffice.java:31) at communication.PostOffice.getPostOffice(PostOffice.java:106) at ca.mda.moscs.common.Utilities$1.run(Utilities.java:130) Caused by: java.lang.IllegalArgumentException: Failed to parse protocol buffer descriptor for generated code. at com.google.protobuf.Descriptors $FileDescriptor.internalBuildGeneratedFileFrom(Descriptors.java:341) at com.google.protobuf.DescriptorProtos.clinit(DescriptorProtos.java: 12656) ... 7 more Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message tag had invalid wire type. at com.google.protobuf.InvalidProtocolBufferException.invalidWireType(InvalidProtocolBufferException.java: 86) at com.google.protobuf.UnknownFieldSet $Builder.mergeFieldFrom(UnknownFieldSet.java:600) at com.google.protobuf.GeneratedMessage $Builder.parseUnknownField(GeneratedMessage.java:320) at com.google.protobuf.DescriptorProtos$DescriptorProto $Builder.mergeFrom(DescriptorProtos.java:2742) at com.google.protobuf.DescriptorProtos$DescriptorProto $Builder.mergeFrom(DescriptorProtos.java:1) at com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java: 285) at com.google.protobuf.DescriptorProtos$FileDescriptorProto $Builder.mergeFrom(DescriptorProtos.java:1143) at com.google.protobuf.DescriptorProtos$FileDescriptorProto $Builder.mergeFrom(DescriptorProtos.java:1) at com.google.protobuf.AbstractMessage $Builder.mergeFrom(AbstractMessage.java:288) at com.google.protobuf.AbstractMessage $Builder.mergeFrom(AbstractMessage.java:1) at com.google.protobuf.AbstractMessageLite $Builder.mergeFrom(AbstractMessageLite.java:184) at com.google.protobuf.AbstractMessage $Builder.mergeFrom(AbstractMessage.java:792) at com.google.protobuf.AbstractMessage $Builder.mergeFrom(AbstractMessage.java:1) at com.google.protobuf.AbstractMessageLite $Builder.mergeFrom(AbstractMessageLite.java:174) at com.google.protobuf.AbstractMessage $Builder.mergeFrom(AbstractMessage.java:785) at com.google.protobuf.DescriptorProtos $FileDescriptorProto.parseFrom(DescriptorProtos.java:820) at com.google.protobuf.Descriptors $FileDescriptor.internalBuildGeneratedFileFrom(Descriptors.java:338) ... 8 more java.lang.NullPointerException at com.instantiations.designer.core.xml.model.description.ComponentPresentationHelper $ComponentPresentationCache.getCacheFile(ComponentPresentationHelper.java: 428) at com.instantiations.designer.core.xml.model.description.ComponentPresentationHelper $ComponentPresentationCache.init(ComponentPresentationHelper.java: 322) at com.instantiations.designer.core.xml.model.description.ComponentPresentationHelper.getCache(ComponentPresentationHelper.java: 299) at com.instantiations.designer.core.xml.model.description.ComponentPresentationHelper.access $0(ComponentPresentationHelper.java:295) at com.instantiations.designer.core.xml.model.description.ComponentPresentationHelper $1.run(ComponentPresentationHelper.java:504) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55) java.lang.NullPointerException at com.instantiations.designer.core.xml.model.description.ComponentPresentationHelper $ComponentPresentationCache.getCacheFile(ComponentPresentationHelper.java: 428) at com.instantiations.designer.core.xml.model.description.ComponentPresentationHelper
Re: [protobuf] Build error on HPUX11.11 with aCC(HP ANSI C++ B3910B A.03.37)
It's possible that aCC does not correctly implement the C++ standard. Templates are really complicated and few compilers get them completely right. Unfortunately all I can say is if you get it working, send a patch. We do not use this compiler at Google so I can't test it myself. On Fri, Nov 12, 2010 at 1:38 AM, Xray renxi...@gmail.com wrote: Hi, Everybody I'm trying to build protobuffer on HPUX with aCC, but can't made it successfully. First of all, I run ./configure CXXFLAGS=-ext -AA -D__osf__, then make. aCC reports that it can't recorgnize a line in google/protobuf/ stubs/common.h: aCC -DHAVE_CONFIG_H -I. -I..-D_REENTRANT -ext -AA -D__osf__ -c - o main.o `test -f 'google/protobuf/compiler/main.cc' || echo './'`google/protobuf/compiler/main.cc Error 20: ./google/protobuf/stubs/common.h, line 314 # 'type name' expected before ''. template bool ^ So I try to modify common.h by changing it from template bool to template bool b Then I make again, and it passed through, but a while later, many other error information appears: libtool: compile: aCC -DHAVE_CONFIG_H -I. -I.. -D_REENTRANT -ext -AA - D__osf__ -c +Maked google/protobuf/stubs/hash.cc -o hash.o /dev/null 21 source='google/protobuf/extension_set.cc' object='extension_set.lo' libtool=yes \ DEPDIR=.deps depmode=hp2 /bin/sh ../depcomp \ /bin/sh ../libtool --tag=CXX --mode=compile aCC - DHAVE_CONFIG_H -I. -I..-D_REENTRANT -ext -AA -D__osf__ -c -o extension_set.lo `test -f 'google/protobuf/extension_set.cc' || echo './'`google/protobuf/extension_set.cc libtool: compile: aCC -DHAVE_CONFIG_H -I. -I.. -D_REENTRANT -ext -AA - D__osf__ -c +Maked google/protobuf/extension_set.cc +Z -DPIC -o .libs/ extension_set.o Error 173: ./google/protobuf/repeated_field.h, line 800 # Redefined symbol 'TypeHandler'; previously defined at [./google/protobuf/repeated_field.h, line 796]. class RepeatedPtrFieldstring::TypeHandler ^^^ Error 219: ./google/protobuf/repeated_field.h, line 891 # Cannot return value where return type of function is void. return RepeatedPtrFieldBase::AddClearedTypeHandler(value); Error 219: ./google/protobuf/repeated_field.h, line 901 # Cannot return value where return type of function is void. return RepeatedPtrFieldBase::Reserve(new_size); ^^^ Warning 921: ./google/protobuf/wire_format_lite.h, line 89 # A trailing comma is not allowed in an enumeration. WIRETYPE_VARINT = 0, ^^ .. It was really confusing, was it beacuse of some options of aCC I didn't set correctly? Pls kindly help, thanks! -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] extending enums
There is no explicit support for this. You can, of course, just use an
integer instead.
On Sat, Nov 13, 2010 at 8:10 AM, sabretoothedhamster
sabretoothedhams...@gmail.com wrote:
Hello everyone!
Is there any way to extend not only a message, but an enum also? E.g.
having the example from
http://code.google.com/apis/protocolbuffers/docs/techniques.html
I would like to keep OneMessage description separated into two files.
Core part with Foo and Bar descriptions in one .proto file (like in
the second pre section), and the second .proto file with Baz
description. It is clear how to organize this on a message level, but
I would like also to have enum Type { FOO = 1; BAR = 2; } in the first
file, and enum Type { BAZ = 3; } in the second. Is it possible at all?
Any workarounds?
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Visual Studio 2010
I test every release on MSVC (2005 and 2008; 2010 wasn't around yet when 2.3.0 was released). I did not see these problems. What version are you using? You could try downloading the latest version of gtest from googletest.googlecode.com and replacing the gtest directory with it. Note that as of the last release, the VS projects for gtest needed a couple small tweaks to the linker settings -- see the autogen.sh in the top-level protobuf dir which (among other things) applies these tweaks. On Sun, Nov 14, 2010 at 9:06 AM, GDotBookGenius gdotthe...@gmail.comwrote: I am trying to built the VS solution under the protobuf-2.3.0/ vsprojects folder, but it comes up with errors. 2549 of them to be exact. It seems to onl y be the test projects that fail to build (gtest, gtest_main, test_plugin, tests, lite_test). The problem seems to be with the template classes defined in gtest-tuple.h. The warning at the top of this file says it was generated by a script. Which script was it? How can I fix this? -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Parse from a iterator range possible?
I think your best bet is to construct an std::string from the iterator range, and then parse from that. STL-style iterators are awkward because they achieve polymorphism via templates. In order for the protobuf parser to be able to use arbitrary iterators directly, the whole parser would have to be an enormous template. This would be impractical for a lot of reasons. On Mon, Nov 15, 2010 at 1:30 PM, idleman evoo...@gmail.com wrote: Hello everyone, I am just wondering if it should be possible to parse from a iterator range? Say something like this: template typename FowardIterator bool ParseFromRange(ForwardIterator, ForwordIterator); Is it possible? Could it easily be implemented? Thanks in advance! -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: protocol buffer message that accepts a variety of different arguments
On Tue, Nov 16, 2010 at 1:28 PM, George george.georg...@hotmail.com wrote: Thank you for your advice. I’m implementing a component model that has a different type-system than protocol buffer. For the connectors that are IPC or inter technology I would like to base the serialization on protocol buffers. There is always an option based on the IDL definition I to generate specific protocol buffer messages that represent the exact argument list per method, but this looks over kill for me. That's the way protocol buffers are intended to be used. I.e. you would have unique input and output types for every method. Trying to have a generic type defeats a lot of the purpose of protocol buffers -- you lose the type safety, some of the efficiency, the forward- and backwards-compatibility, etc. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: Why to reinvent the wheel ?
On Tue, Nov 9, 2010 at 6:15 AM, Kalki70 kalki...@gmail.com wrote: Well, Google, with all their resources, could have, instead of creating something like ASN.1, but different, put some effort developing some apis, like those from protobuf, but for ASN.1. They could have supported maybe a subset of full ASN.1, but they would still be using an standard, and it would be easier to communicate with existing systems that support ASN.1 As Chris astutely points out, the complexity of ASN.1 makes it much harder to produce a high-quality implementation. Google's resource are not unlimited. Engineering time spend implementing obscure primitive types or other unnecessary features is time NOT spent improving the speed and robustness of the implementation. Oh, come on, you are not being serious. You can say many of those things. What do you mean, for instance : faster ?? ASN.1 has no speed. The speed comes from the ASN.1 compiler. I challenge you, then, to show me an ASN.1 implementation that is faster than our C++ protobuf implementation. Given the 10+-year head start, there should be one, right? More robust ?? I see there are, like with any development, bugs that are being fixed. Better to stick with somethign that has been used for over 20 years, if you think about More robust: There are bugs, but not critical ones. Remember that essentially *all* of Google's server-to-server communications use protobufs. Easier to understand, well, you saw my example, and it is very easy to understand. Sorry, but no. The syntax is not intuitive to C++ or Java developers the way protobuf's syntax is. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] fails to parse from string
It sounds like you probably have extra bytes at the end of mystr which are not part of the protobuf. The parser parses everything before those bytes just fine, but then chokes when it gets to the bytes it doesn't recognize. Please make sure you only pass in the exact bytes which came out of the serializer at the other end, no more, no less. On Tue, Nov 9, 2010 at 1:11 PM, Brad Lira snmp.apa...@gmail.com wrote: I have a c++ client/server application that sends messages using protocol buffers, however, at the server side, when i call address_book.ParseFromString(mystr) it returns false, but it actually gets the message correctly from client side, so i am not sure why it thinks that parsing has failed. any ideas? thanks, brad -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] python enum values option
You need to use the descriptor for the enum type. Unfortunately this
interface isn't very well fleshed-out in Python. I think you'd have to
write something like:
Request.DESCRIPTOR.fields_by_name['error'].enum_type.values_by_number[message.error].options.Extensions[verbose_enum_value_option]
I haven't checked that that's exactly correct, but it gives you an idea. We
should probably improve this.
On Tue, Nov 9, 2010 at 4:12 AM, Vsevolod Zadubrovsky
zadubrov...@gmail.comwrote:
Hi, I'm stuck with getting the enum value option in Python. The proto
file is:
import descriptor.proto;
extend google.protobuf.EnumValueOptions {
optional string verbose_enum_value_option = 50005;
}
enum ErrorType {
OK = 1 [(verbose_enum_value_option) = OK];
SOME_ERROR = 2 [(verbose_enum_value_option) = Some Error verbose
message];
}
message Request {
required bool success = 1;
optional ErrorType error = 2;
}
When I receive the Request message, I can access the 'error' field,
and its type is int, that's actually ok. But how can I get the
verbose_enum_value_option of 'error' field value ?
Thanks
p.s. The goal is to keep error types and their verbose error messages
in one place, available for every service in our project, so the error
messages would've been identical.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] 'python setup.py test' failed
I haven't seen this problem, but my wild guess is that for some reason your environment is using distutils instead of setuptools. They are similar, but distutils lacks some features, such as the test command. I'm not sure why this would happen, though. On Tue, Nov 9, 2010 at 1:05 AM, cy ora...@gmail.com wrote: As told in protobuf-2.3.0/python/README.TXT, I run '$python setup.py test'. However failed as bellow: /usr/lib/python2.6/distutils/dist.py:266: UserWarning: Unknown distribution option: 'namespace_packages' warnings.warn(msg) /usr/lib/python2.6/distutils/dist.py:266: UserWarning: Unknown distribution option: 'test_suite' warnings.warn(msg) usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: setup.py --help [cmd1 cmd2 ...] or: setup.py --help-commands or: setup.py cmd --help error: invalid command 'test' What's the problem? -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] protoc.exe for 2.2.0a
It's the same as 2.2.0. The only change in 2.2.0a affects the runtime library on Linux. On Fri, Nov 5, 2010 at 12:29 PM, Sherry sherr...@gmail.com wrote: Is there a downloadable windows binary for the protoc compiler for 2.2.0a? -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] optional imports for protos
Technically, there is no such file google/protobuf/csharp_options.proto. The canonical directory google/protobuf should contain only files from the official Google protobuf implementation. But it seems the C# implementation you're using has chosen to put their own file under this path. I guess that's not a huge deal -- particularly because you're probably using Jon Skeet's implementation and he does technically work for Google -- but I would have asked him to use a different path. Anyway, including the C# options in the Java build should not be a problem. It just requires that you generate Java classes for csharp_options.proto and include those in your application. It's mildly annoying, and we've considered doing something about it, but for the moment this is the only option, as we have no concept of optional imports or compile-time-only imports. On Mon, Nov 8, 2010 at 10:51 AM, jebrick jebr...@gmail.com wrote: I am running into an issue with generating protobuf code for c# and java from the same protos. The protos have an import for the dotnet side (import google/protobuf/csharp_options.proto;) which causes the runtime java code to fail when it tries to generate the Descriptors. Are their any build options to exclude that import for the java_out calls vs a dotnet build? -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] link issue with proto in different project
There is a command-line option to add an export macro to all definitions.
Use something like:
protoc --cpp_out=dllexport_decl=MY_EXPORT_MACRO:outdir
However, be very careful when passing protobuf objects across a DLL
boundary. The code on both sides must be dynamically linked against the
same version of the C runtime, otherwise you run into all kinds of binary
compatibility and heap issues. See vsprojects/readme.txt for more
information.
On Fri, Nov 5, 2010 at 1:15 AM, yang yu kojiyij...@gmail.com wrote:
I'm working on windows, some of my projects are dlls.
i need to add __declspec dllexport/dllimport symbol with the function or
variables to use the project as dll.
but when i use protoc generate the source for the proto file, of course
they don't have the dllexport/dllimport symbol.
the problem is after build dll A(have a proto), when i build B link the
project A, i got some link error like this:
error LNK2019: unresolved external symbol void
PB:protobuf_AddDesc_Point2df_2eproto(void)..
because the function void PB:protobuf_AddDesc_Point2df_2eproto(void) in
dll A didn't export,
so the project B link faild.
anything help would be wonderful.
---the situation like
this---
in DLL-A, have a proto like this:
-point.proto-
package PB;
message Point2df
{
required float x = 1;
required float y = 2;
}
in DLL-B, have a proto like this;
-path.proto-
package PB;
import point.proto;
message Path
{
repeated Point2df points = 1;
}
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Why to reinvent the wheel ?
My understanding of ASN.1 is that it has no affordance for forwards- and
backwards-compatibility, which is critical in distributed systems where the
components are constantly changing.
On Mon, Nov 8, 2010 at 7:34 PM, Kalki70 kalki...@gmail.com wrote:
Hello,
I just discovered this developers tool, and I can't understand why it
was invented. Why didn't Google use ASN.1, which is standard and it is
used for this, to make a language, platform independent description of
data to be enconded later as XML, or different binary formats, that
can be faster and more efficient?
All this is like reinventing ASN.1
For instance, the example shown on the web page :
message Person {
required string name = 1;
required int32 id = 2;
optional string email = 3;
enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}
message PhoneNumber {
required string number = 1;
optional PhoneType type = 2 [default = HOME];
}
In ASN.1, which is a standard used for over 20years, the same could be
written similar to this (I haven't used it in a while, maybe I made
some mistakes ):
PhoneType ::= ENUMERATED { MOBILE, HOME, WORK }
PhoneNumber ::= SEQUENCE
{
number [1] IA5String ,
phone [2] PhoneType DEFAULT HOME
}
Person ::= SEQUENCE
{
name [1] IA5String ,
id [2] INTEGER,
email [3] OCTET STRING OPTIONAL
phone [4] SET OF PhoneNumber
}
Best regards,
Luis
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Why to reinvent the wheel ?
OK, I looked into this again (something I do once every few years when
someone points it out).
ASN.1 *by default* has no extensibility, but you can use tags, as I see you
have done in your example. This should not be an option. Everything should
be extensible by default, because people are very bad at predicting whether
they will need to extend something later.
The bigger problem with ASN.1, though, is that it is way over-complicated.
It has way too many primitive types. It has options that are not needed.
The encoding, even though it is binary, is much larger than protocol
buffers'. The definition syntax looks nothing like modern programming
languages. And worse of all, it's very hard to find good ASN.1
documentation on the web.
It is also hard to draw a fair comparison without identifying a particular
implementation of ASN.1 to compare against. Most implementations I've seen
are rudimentary at best. They might generate some basic code, but they
don't offer things like descriptors and reflection.
So yeah. Basically, Protocol Buffers is a simpler, cleaner, smaller,
faster, more robust, and easier-to-understand ASN.1.
On Mon, Nov 8, 2010 at 9:42 PM, Kenton Varda ken...@google.com wrote:
My understanding of ASN.1 is that it has no affordance for forwards- and
backwards-compatibility, which is critical in distributed systems where the
components are constantly changing.
On Mon, Nov 8, 2010 at 7:34 PM, Kalki70 kalki...@gmail.com wrote:
Hello,
I just discovered this developers tool, and I can't understand why it
was invented. Why didn't Google use ASN.1, which is standard and it is
used for this, to make a language, platform independent description of
data to be enconded later as XML, or different binary formats, that
can be faster and more efficient?
All this is like reinventing ASN.1
For instance, the example shown on the web page :
message Person {
required string name = 1;
required int32 id = 2;
optional string email = 3;
enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}
message PhoneNumber {
required string number = 1;
optional PhoneType type = 2 [default = HOME];
}
In ASN.1, which is a standard used for over 20years, the same could be
written similar to this (I haven't used it in a while, maybe I made
some mistakes ):
PhoneType ::= ENUMERATED { MOBILE, HOME, WORK }
PhoneNumber ::= SEQUENCE
{
number [1] IA5String ,
phone [2] PhoneType DEFAULT HOME
}
Person ::= SEQUENCE
{
name [1] IA5String ,
id [2] INTEGER,
email [3] OCTET STRING OPTIONAL
phone [4] SET OF PhoneNumber
}
Best regards,
Luis
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: 2.4 release?
To be clear, there is no target release date. However, we just pushed this code out to SVN earlier today. It's still experimental and may or may not be working, but feel free to give it a try. On Tue, Nov 2, 2010 at 10:02 AM, Zachary Turner divisorthe...@gmail.comwrote: Just wanted to follow up on this and see if this is still on target to be released within a month or so, and still include a beta of the python C++ work. Thanks, Zach On Sep 14, 11:28 am, Kenton Varda ken...@google.com wrote: Petar mostly finished this work, but then left the company. We're trying to tie up the loose ends so that we can get it out, but everyone is pretty busy. We'll probably do 2.4.0 within a month or two as we have a new team member who needs to learn the release process, and it should at least include a beta of this work. On Tue, Sep 14, 2010 at 10:36 AM, Zachary Turner divisorthe...@gmail.comwrote: Reading through old threads, I found one that said that the next release would probably include Petar's python wrapper for native C++ protobufs, greatly increasing speed of protubufs when calling from Python. Another post mentioned that this was expected to be done sometime this summer. Now that summer has come and gone, does anyone have an update about when this release might be ready? We're considering re-inventing a python / C++ wrapper just because the performance is such a blocker on what we're working on at the moment, but given how long it's been in the works, it seems like it's such a black hole of work that we're going to embark on, whereas the magic hammer solution might only just around the corner. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com protobuf%2bunsubscr...@googlegroups.c om . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Fantastic
Thanks! You're right, we usually only hear when things don't work. It's nice to hear a success story from time to time. :) On Tue, Nov 2, 2010 at 3:57 AM, Doodle totheleftpa...@gmail.com wrote: I figure not all the emails you get should be problems. This thing is great took 3 hours to make it do what it I wanted it to(serialization on Java) and I bet I do not even have a clue yet what it can do. Hell now I am not even committed to staying in one language on the back end now I guess thats quite the added bonus for a serialization library !...@*! -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: Message missing required fields exception when parsing a message that has required fields defaulted
On Wed, Oct 27, 2010 at 8:55 AM, locky true.n...@gmail.com wrote: The producers of the message are actually C# and not C++ as I first mentioned (I only deal with the Java implementation). The scenario however, is the same. It is possible that the C# implementation (which is not written by Google) accidentally interpreted default values differently. You should ask the author. @Kenton, I thought default values were never sent over the wire. According to the 'Updating A Message Type' section of the language guide ...Changing a default value is generally OK, as long as you remember that default values are never sent over the wire Are you saying this is not the case? What the documentation means is that the value you write in the .proto file is never used directly when serializing a message. However, if you explicitly set a field to a value that happens to be equal to the default value, it *is* sent. In this case we don't think of the value as being the default value. Seems like the wording of that text needs to be improved to avoid ambiguity. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Migration from 2.0.3 to 2.3.0 problem on Mac OS –arch ppc
No ideas here. Let us know if you find a solution. A google search for ___floatundidf shows other people seeing this error in other contexts. On Wed, Oct 27, 2010 at 12:33 PM, Irene Georgieva irened...@gmail.comwrote: Hi, I would like to migrate from 2.0.3 to 2.3.0. Everything looks ok except that I can’t link my existing projects on Mac OS with gcc 4.0 – arch ppc (only, i386 works fine). I receive: Undefined symbols: ___floatundidf, referenced from: google::protobuf::DescriptorBuilder::OptionInterpreter::SetOptionValue(google::protobuf::FieldDescriptor const*, google::protobuf::UnknownFieldSet*)in libprotobuf.a(descriptor.o) google::protobuf::TextFormat::Parser::ParserImpl::ConsumeDouble(double*)in libprotobuf.a(text_format.o) ___floatundisf, referenced from: google::protobuf::DescriptorBuilder::OptionInterpreter::SetOptionValue(google::protobuf::FieldDescriptor const*, google::protobuf::UnknownFieldSet*)in libprotobuf.a(descriptor.o) ld: symbol(s) not found It looks like this code in SetOptionValue isn’t changed from 2.0.3. What is then? Any ideas? Thanks, -i -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Newbie questions
On Tue, Nov 2, 2010 at 5:06 AM, Ofir Ben-Ezra ofir.bene...@gmail.comwrote: Hi, I'm rather new to protocol buffers and to protobuf-net. I have 2 questions: 1. I'm trying to find out how can I generate XML file from a given .proto file and vice versa? I'm not aware of any tool for this. 2. I know this is done automatically in VS2008 plug in, but how can I generate C# classes from a given .proto file? Google does not provide a C# implementation, but there are a couple third-party ones. You would have to follow the documentation for one of those projects. Thanks, Ofir -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Compiling with Rtools
I'm not sure if the Rprotobuf people read this list. You might look to see
if they have their own mailing list or contact address.
On Wed, Oct 20, 2010 at 1:39 PM, Travis bacch...@gmail.com wrote:
Hello all,
I'm trying to load/save protobuf format data directly in R, so writing
a R external function in C++. As visual studio doesn't support this
usage, I have to compile the code with Rtools (which is essentially
mingw and cygwin). However, when I try to compile it, I receive an
error because of including the protobuf header.
C:\tR CMD SHLIB test.cpp
g++ -IC:/PROGRA~1/R/R-211~1.1/include -O2 -Wall -c test.cpp
-o test.o
In file included from c:\rtools\mingw\bin\../lib/gcc/mingw32/4.2.1-
sjlj/include/c++/bits/locale_facets.h:1536,
from c:\rtools\mingw\bin\../lib/gcc/mingw32/4.2.1-
sjlj/include/c++/bits/basic_ios.h:44,
from c:\rtools\mingw\bin\../lib/gcc/mingw32/4.2.1-
sjlj/include/c++/ios:50,
from c:\rtools\mingw\bin\../lib/gcc/mingw32/4.2.1-
sjlj/include/c++/ostream:45,
from c:\rtools\mingw\bin\../lib/gcc/mingw32/4.2.1-
sjlj/include/c++/iterator:70,
from C:/PROGRA~1/R/R-211~1.1/include/google/protobuf/
repeated_field.h:50,
from Person.pb.h:23,
from test.cpp:5:
c:\rtools\mingw\bin\../lib/gcc/mingw32/4.2.1-sjlj/include/c++/bits/
codecvt.h:219:45: error: macro length passed 4 arguments, but takes
just 1
test.cpp:108:4: warning: no newline at end of file
In file included from c:\rtools\mingw\bin\../lib/gcc/mingw32/4.2.1-
sjlj/include/c++/bits/locale_facets.h:1536,
from c:\rtools\mingw\bin\../lib/gcc/mingw32/4.2.1-
sjlj/include/c++/bits/basic_ios.h:44,
from c:\rtools\mingw\bin\../lib/gcc/mingw32/4.2.1-
sjlj/include/c++/ios:50,
from c:\rtools\mingw\bin\../lib/gcc/mingw32/4.2.1-
sjlj/include/c++/ostream:45,
from c:\rtools\mingw\bin\../lib/gcc/mingw32/4.2.1-
sjlj/include/c++/iterator:70,
from C:/PROGRA~1/R/R-211~1.1/include/google/protobuf/
repeated_field.h:50,
from Person.pb.h:23,
from test.cpp:5:
c:\rtools\mingw\bin\../lib/gcc/mingw32/4.2.1-sjlj/include/c++/bits/
codecvt.h:218: error: expected ';' before 'const'
c:\rtools\mingw\bin\../lib/gcc/mingw32/4.2.1-sjlj/include/c++/bits/
codecvt.h:222: error: expected `;' before 'int'
In file included from Person.pb.h:24,
from test.cpp:5:
C:/PROGRA~1/R/R-211~1.1/include/google/protobuf/extension_set.h:450:
error: expected identifier before '(' token
C:/PROGRA~1/R/R-211~1.1/include/google/protobuf/extension_set.h:450:
error: expected `)' before ',' token
test.cpp: In function 'void cpp_test(int*, double*, double*, int*,
double*, double*)':
make: *** [test.o] Error 1
The code in codecvt.h:219:45 is :
int
length(state_type __state, const extern_type* __from,
const extern_type* __end, size_t __max) const
{ return this-do_length(__state, __from, __end, __max); }
The code in include/google/protobuf/extension_set.h:450 is:
int GetSize() const;
void Free();
int SpaceUsedExcludingSelf() const;
I'm using windows xp sp3, protobuf 2.3.0, g++ (GCC) 4.2.1-sjlj
(mingw32-2), R version 2.11.1.
I suspect the reason that protobuf is just not compatible with g++
version 4.2.1 or the g++ in Rtools is somewhat downgraded version of
full g++, thus does not match with protobuf.
Does anyone have same issues / know how to fix this? Thank you!
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] New Ruby Protocol Buffers library
I'd like to add this to the third-party wiki, but I'm not sure which link to use. Can you suggest (or create) a general-purpose landing page? On Thu, Oct 21, 2010 at 10:02 AM, Brian Palmer br...@codekitchen.netwrote: Mozy has just open sourced their implementation of Protocol Buffers for Ruby. The implementation has been in use internally at Mozy for over a year. This implementation has put a lot of focus on serialization/deserialization performance, and completeness. The ruby protobuf compiler calls out to protoc to do the heavy lifting, so it's using the same parser as the official compiler. It looks like now there's a plugin system for protoc itself, so if we were writing this project today we probably would've just made the compile-to-ruby functionality a plugin. Enjoy! The repo: http://github.com/mozy/ruby-protocol-buffers The gem: https://rubygems.org/gems/ruby-protocol-buffers The docs: http://rubydoc.info/gems/ruby-protocol-buffers/0.8.4/frames -- Brian Palmer -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Extensions Management
First of all, I suggest expanding the extension range:
extensions 1 to max;
This gives you 2^29 numbers to work with (the remaining three bits are
reserved by the wire format to identify types).
Now you can easily assign a large range of extension numbers to each office,
or even to each developer, and they can use them as they see fit.
Or if you just chose numbers randomly, you'd have to have around 10k
extensions before you'd need to start worrying about collisions.
On Mon, Oct 25, 2010 at 12:03 PM, maninder batth
batth.manin...@gmail.comwrote:
We have a geographically distributed team and instant communications
are not possible. Given this constraint in mind, i am wondering how to
manage extensions. To clarify my question, consider a message like
message Request {
extensions 1 to 100;
}
Different developers would be working on extending Request. For
example developer A uses the above request in a Client.proto file,
such as
extend Request {
optional int64 clientId = 1;
}
Developer B, uses the above request in a Location.proto file such as
extend Request {
optional int64 locationId = 1;
}
Question is how can developer A and B, extend the message Request,
without worrying about extension slots being already occupied by some
other message?
As shown above, both developers reserved the slot 1, with different Id
types.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Message missing required fields exception when parsing a message that has required fields defaulted
On Fri, Oct 22, 2010 at 9:12 AM, locky true.n...@gmail.com wrote: From what I can see the issue is as follows: As the stringProperty value is the same as the default (an empty string) it is not sent over the wire. No, this is not correct. Whether or not a value is sent on the wire depends only on whether or not you explicitly called the setter. It doesn't matter if you set it to its default value. If you called set_property(), the field will be sent, even if you set it to the empty string. If you did not call set_property(), the field will not be sent. You can call has_property() to determine whether or not it was set. I strongly suspect that your code contains some other bug, and the bytes being received in Java do not actually match the bytes sent from C++. The most common problem we see is that people try to store the bytes in a C-style NUL-terminated string, like: std::string str = message.SerializeAsString(); char* cstr = str.c_str(); // DOES NOT WORK send(cstr); // DOES NOT WORK The problem is that encoded protobufs contain arbitrary bytes, some of which may be zero. A C-style char* string is considered to end at the first zero byte, but the encoded protobuf may extend beyond that. You need to use code like this instead: std::string str = message.SerializeAsString(); send(str.data(), str.size()); -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] protocol buffer within a protocol buffer from C++ to Java
The others answered your main question, but there is another problem:
On Mon, Oct 25, 2010 at 6:45 PM, Paul mjpabl...@gmail.com wrote:
char* str_buf;
str_buf = new char[meas_rec.ByteSize()];
string str = str_buf;
Here you are allocating a char array, and then *copying* it into a
std::string. This does nothing useful, and it may actually crash depending
on the random contents of the allocated array.
These three lines should be replaced with just:
string str;
You do not have to resize the string ahead of time, as this is done
automatically by SerializeToString().
meas_rec.SerializeToString(str);
m.set_meas_rec_str(str);
Incidentally, another, more efficient way to write this would be:
meas_req.SerializeToString(m.mutable_meas_rec_str());
Also, you probably know this, but just in case: You can just do:
message Measurement {
required string id = 1;
optional MeasRec meas_rec = 2;
}
message MeasRec {
optional int id = 1;
}
This way you don't have to manually encode the sub-message.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: How to import protocol buffer definitions from another Python package?
Right, --proto_path should always point at the root of your source tree, and all imports should be fully-qualified. Probably the documentation for import in the language guide should include a bit more text on this, as it comes up a lot. If you file a bug I'll update it when I get time. Thanks for offering to help, BTW. Unfortunately our documentation is not in a repository that can easily be edited from the outside. This was a mistake I made when setting up the project, but oh well... On Tue, Oct 26, 2010 at 1:10 PM, Sancho sanc...@gmail.com wrote: Nevermind! I fixed it... The solution was to use import sift/sift_descriptors.proto in filtered_descriptors.proto, and then point protoc to --proto_path=../ instead of --proto_path=../sift. Then, protoc generates python code that does the import as import sift.sift_descriptors_pb2. This isn't explained well in the documentation. How can I help write something? On Oct 26, 1:03 pm, Sancho sanc...@gmail.com wrote: Sorry, transcription error... I meant import sift_descriptors_pb2 where I wrote import sift_descriptors_pb2.py both times. On Oct 26, 12:58 pm, Sancho sanc...@gmail.com wrote: I have my directory structure like this: root/ sift/ __init__.py sift_descriptors.proto sift_descriptors_pb2.py project/ __init__.py filtered_descriptors.proto filtered_descriptors_pb2.py filtered_descriptors_test.py The root directory is in my $PYTHONPATH. I build root/sift/sift_descriptors_pb2.py using protoc --python_out=./ sift_descriptors.proto I build root/project/filtered_descriptors_pb2.py using /cs/public/lib/ pkg/protobuf/bin/protoc --proto_path=../sift --proto_path=./ -- python_out=./ filtered_descriptors.proto In filtered_descriptors.proto, I use import sift_descriptors.proto The problem is that in filtered_descriptors_pb2.py, there's a statement that just does this bare import: import sift_descriptors_pb2.py, without reference via the module name as would be needed: from sift import sift_descriptors_pb2.py. What am I doing wrong? -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Message vs MessageLite
Instead of maintaining two identical files, I suggest writing a build rule that just constructs the lite version from the regular version. You could use a make rule like: my_lite.proto: my.proto cp my.proto my_lite.proto echo option optimize_for = LITE_RUNTIME; my_lite.proto This way the files cannot get out-of-sync. See also comment 11 in this thread: http://code.google.com/p/protobuf/issues/detail?id=187#c11 There I provide a protoc plugin which converts inputs to lite mode. It's a more complicated approach but less hacky. On Tue, Oct 26, 2010 at 2:56 PM, Alsmom2005 gundanu...@gmail.com wrote: Hi all, Is it ok if the serialization is made using libprotobuf library and the deserialization (on the other end) is made using code built with libprotobuf-lite library ? That meaning 2 .proto files (the only difference bw those two file is that one contains 'option optimize_for = LITE_RUNTIME') . Thank you in advance! -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] buffer sizes when sending messages from c++ to java
On Tue, Oct 19, 2010 at 6:28 PM, Paul mjpabl...@gmail.com wrote: int numBytesForDelim = sizeof(int); You seem to be assuming that the delimiter is a simple fixed-width int. coded_output-WriteVarint32(snap1.ByteSize()); But you are actually writing a varint32, which can be anywhere between 1 and 5 bytes depending on the value. Use CodedOutputStream::Varint32Size() to compute the number of bytes needed to encode a particular value. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
