Re: [protobuf] Any Python wrappers for the C++ implementation?
Is the advice in this thread, particularly with respect to generating C++ message implementations, still valid for modern versions of the Python protobuf runtime? Brief spelunking through the Python codebase didn't yield a clear mechanism for how messages are automagically discovered. On Tuesday, December 7, 2010 at 10:49:43 PM UTC-8, Yang Zhang wrote: > > On Tue, Dec 7, 2010 at 9:40 PM, Kenton Varda> wrote: > > On Tue, Dec 7, 2010 at 9:19 PM, Yang Zhang > wrote: > >> > >> > Also, note that if you explicitly compile C++ versions of your > >> > messages and link them into the process, they'll be even faster. (If > >> > you > >> > don't, the library falls back to DynamicMessage which is not as fast > as > >> > generated code.) > >> > >> I'm trying to decipher that last hint, but having some trouble - what > >> exactly do you mean / how do I do that? I'm just using protoc > >> --py_out=... and PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp. > > > > I'm not completely sure what I mean, because I don't have much experience > > with Python C Extensions. Basically I'm saying you should additionally > > generate C++ code using protoc, the compile that into a C extension (even > > with no interface), and then load it into your Python process. Simply > > having the C++ code for your message types present will make them faster. > > Ah, my understanding now is that: > > - Python code ordinarily (without > PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp) uses pure Python > (generated code) to parse/serialize messages. > > - Python code *with* PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp) uses > generic C++ code that dynamically parses/serializes messages (via > DynamicMessage/reflection), as opposed to using any pre-generated C++ > code. > > - Python code with PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp actually > also *searches for the symbols for any pre-generated C++ code in the > current process*, and uses them if available instead of > DynamicMessage...? (This is via some global DescriptorPool magic?) > > Sounds like pretty weird behavior, but indeed, now I get even faster > processing. The following run shows ~68x and ~13x speedups vs. ~15x > and ~8x (my original speedup calculations were ~15x and ~8x, not ~12x > and ~7x...not sure how I got those, I probably was going off a > different set of measurements): > > $ PYTHONPATH=build/lib.linux-x86_64-2.6/:$PYTHONPATH > PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp python sandbox/pbbench.py > out.ini > noop: 1.6188621521e-07 > ser: 6.39575719833e-06 > parse: 4.55250144005e-05 > msg size: 10730 > > This was simple to do. I added a C extension to my setup.py: > > <<< > setup( > ... > ext_modules=[Extension('podpb', > sources=['cpp/podpb.c','cpp/main.pb.cc'], libraries=['protobuf'])], > ... > ) > >>> > > Generate the second source file with `protoc --cpp_out=cpp`, and > create the first one to set up an empty Python module: > > <<< > #include > > static PyMethodDef PodMethods[] = { > {NULL, NULL, 0, NULL}/* Sentinel */ > }; > > PyMODINIT_FUNC > initpodpb(void) > { > PyObject *m; > > m = Py_InitModule("podpb", PodMethods); > if (m == NULL) > return; > } > >>> > > Now `python setup.py build` should build everything. Just import the > module (podpb in our case) and you're good. > > Awesome tip, thanks Kenton. I foresee additions to the documentation > in protobuf's near future :) > -- > Yang Zhang > http://yz.mit.edu/ > > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Any Python wrappers for the C++ implementation?
On Tue, Dec 7, 2010 at 7:08 PM, Kenton Varda ken...@google.com wrote: Cool. Serialization and parsing themselves should actually be improved even more than that, but having other Python code around it waters down the numbers. :) The times are from a minimal microbenchmark using Python's timeit module: nruns = 1000 nwarmups = 100 es = ... # the protobufs def ser(): return [e.SerializeToString() for e in es] def parse(ses): for se in ses: pb.Email().ParseFromString(se) t = timeit.Timer(lambda:None) t.timeit(nwarmups) print 'noop:', t.timeit(nruns) / nruns t = timeit.Timer(ser) t.timeit(nwarmups) print 'ser:', t.timeit(nruns) / nruns / len(es) ses = ser() t = timeit.Timer(lambda: parse(ses)) t.timeit(nwarmups) print 'parse:', t.timeit(nruns) / nruns / len(es) print 'msg size:', sum(len(se) for se in ses) / len(ses) Also, note that if you explicitly compile C++ versions of your messages and link them into the process, they'll be even faster. (If you don't, the library falls back to DynamicMessage which is not as fast as generated code.) I'm trying to decipher that last hint, but having some trouble - what exactly do you mean / how do I do that? I'm just using protoc --py_out=... and PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp. As for when 2.4.0 might be released, it's hard to say. There's a lot of work to do, and we have a new person doing this release so he has to learn the process. Also, holidays are coming up. So, I'd guess it will be ready sometime in January. Thanks for the estimate; even a ballpark without commitment is useful. -- Yang Zhang http://yz.mit.edu/ -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Any Python wrappers for the C++ implementation?
On Tue, Dec 7, 2010 at 9:19 PM, Yang Zhang yanghates...@gmail.com wrote: Also, note that if you explicitly compile C++ versions of your messages and link them into the process, they'll be even faster. (If you don't, the library falls back to DynamicMessage which is not as fast as generated code.) I'm trying to decipher that last hint, but having some trouble - what exactly do you mean / how do I do that? I'm just using protoc --py_out=... and PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp. I'm not completely sure what I mean, because I don't have much experience with Python C Extensions. Basically I'm saying you should additionally generate C++ code using protoc, the compile that into a C extension (even with no interface), and then load it into your Python process. Simply having the C++ code for your message types present will make them faster. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Any Python wrappers for the C++ implementation?
Protobuf 2.4.0 will include an implementation of the Python API that is backed by C++ objects. The interface is identical to the existing Python API, and you can wrap it around existing C++ objects or have it construct its own. This code is already is SVN. Unfortunately the team is someone backlogged and we haven't been able to make a lot of progress on an official release. But it should be a lot easier to get the SVN code working than to write your own. :) On Tue, Nov 30, 2010 at 11:22 PM, Yang Zhang yanghates...@gmail.com wrote: Has anyone written (a tool for generating) Python wrappers around the C++ generated code and is willing to share this? I'm looking to do the same, so this would save me a bit of research time. (It's fine if it's not a general tool and this is specific to some schema.) Thanks! -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Any Python wrappers for the C++ implementation?
Thanks Kenton, we'll take a look. Out of curiosity, any ETA on 2.4.0? On Wed, Dec 1, 2010 at 12:04 PM, Kenton Varda ken...@google.com wrote: Protobuf 2.4.0 will include an implementation of the Python API that is backed by C++ objects. The interface is identical to the existing Python API, and you can wrap it around existing C++ objects or have it construct its own. This code is already is SVN. Unfortunately the team is someone backlogged and we haven't been able to make a lot of progress on an official release. But it should be a lot easier to get the SVN code working than to write your own. :) On Tue, Nov 30, 2010 at 11:22 PM, Yang Zhang yanghates...@gmail.com wrote: Has anyone written (a tool for generating) Python wrappers around the C++ generated code and is willing to share this? I'm looking to do the same, so this would save me a bit of research time. (It's fine if it's not a general tool and this is specific to some schema.) Thanks! -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.