[protobuf] Re: Python3 Protobufs
On Thursday, August 22, 2013 2:36:07 PM UTC-7, Charles Law wrote: I'd be interested in getting this done. I do want it merged in at some point this will only make it easier. Since I did some of the translation at work I'm talking to legal about getting the contributor license agreement signed. Free time is very rare though. It might (not for sure, but just might) make sense to take the latest svn branch and merge my code into that. It took a couple iterations to get it right so there are some commits that don't add anything in the end. The hardest thing I think was finding all the places strings should be bytes. I might have the time to do this in the next month or 2. Thanks, Charles It would be great to have Python3 support. A couple other projects I am involved with have recently switch to a single code base (to avoid the need to do the 2to3 dance), and they have reported very positively about their experience. If you are interested, here is an excellent summary of how the SymPy project handles things (they dropped support for Python = 2.5 though, not sure how possible that is for protbuf). http://ondrejcertik.blogspot.com/2013/08/how-to-support-both-python-2-and-3.html I believe the numpy, scipy, ipython, and matplotlib folks are also doing this, or are very close to having it done. One of the SymPy maintainers worked for Continuum over the summer and posted about this here as well: http://asmeurersympy.wordpress.com/2013/08/22/python-3-single-codebase-vs-2to3/ http://asmeurersympy.wordpress.com/2013/08/09/using-python-3-as-my-default-python/ Anyway, if you want me to review any patches, ping me and I'll try to do so. Luke -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out.
[protobuf] Re: Protocol Buffers on an embedded system
On Thursday, August 22, 2013 6:27:08 AM UTC-7, Trepi Dacious wrote: 1) I'm guessing the STM32F4 MCU? In that case yes, I've used protobuf in an identical environment (chibios, etc.). You guessed it. Yeah, Olimex STM32-H407, ChibiOS/RT, fatfs, and nanopb. It was *extremely* helpful to be able to log intermediate data as optional fields without breaking backwards compatibility. 2) Yes, I've used nanopb extensively, and it works very well. I've only run into one issue with inter-operation with the Java version of protobuf (and also protoc) - nanopb perfectly reasonably doesn't check that the enum values you use are actually in the enum, whereas some other implementations do. The result of this is not terrible - if you encode with nanopb and decode with such an implementation, the decoder will just ignore the invalid enum values, possibly adding them as unknown fields. In addition you might want to validate your own enums on the MCU before encoding and after decoding, if the results of an invalid value are bad... In general it's really no harder to use nanopb on the firmware and GPB on the PC than to use the same on both ends, since they both follow the protobuf spec., although it's a little different for me since I'm using Java on the PC so I'm automatically not using the same libraries. I've also tried with a ruby protobuf library, which was also happy talking to nanopb. Yeah, I agree. I've been looking at the nanopb+stm32+fatfs generated data on my PC using the C++ Google protobuf library. Works very well. I originally implemented things in Python, but it is silly slow compare to C++. I could just be doing something stupid but I saw 100x speedups by using the C++ API directly. As a general protobuf point, I've found it very helpful to stick with the standard delimited form for multiple messages, where the data has a varint encoded length for each message in the data. The Google Java protobuf library supports reading this directly, and encoding it with nanopb is also pretty easy. I will probably do this next time. What I ended up doing was writing 2-byte (LSB, then MSB) message size delimiters prior to each message. In my case, it probably wasn't worth the extra work and I should have gone with the varint apprach. The nice thing about my approach is that I only had to encode the data once and it gets encoded directly into the memory buffer, so there is no double encoding or copying of the message after it is encoded. On the flip side, I had to make my buffers a bit bigger and handle the case where the message overflows, and copy the overflowing bytes into the next buffer. You can check it out here in case you are interested: https://github.com/hazelnusse/robot.bicycle/blob/master/firmware/src/sample_buffer.cpp#L42 Such optimizing was probably not necessary for my application, but if you were trying to push the hardware harder, it might be. I actually found this post looking for anyone with some code to wrap a fatfs file as a nanopb stream, I'll continue my search! Let me know if you find something like that, it would be really cool to have. Luke -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out.
[protobuf] Re: Python3 Protobufs
Charles, Thanks for taking the time to do this! Is it possible to make your github repo be based off of the latest svn checkout of GPB? I have used the instructions here [0] to do this for other projects where I wanted to use git but the official code was managed with svn (as in this case). The nice thing about this is then hopefully your work could be merged into GPB down the road since it would be based off the most recent commit and the merge conflicts would be eliminated. It looks like your initial svn checkout of the GPB repos was a year ago so all the python3 tests you've got are based on version 2.4.1. I've already set up git svn clone of the official google protobuf svn repo here: https://github.com/hazelnusse/protobuf-git-svn If you were to clone this and see if you could apply all your patches to the latest svn checkout, then hopefully the maintainers of GPB would be more willing to get the ball rolling on Python3. Let me know if I can help at all, I am using GPB in a project where everything else is Python 3 but I have to also make sure Python 2 is available and it would be nice if GBP supported both. Luke [0] -- http://stackoverflow.com/questions/465042/is-it-possible-to-have-a-subversion-repository-as-a-git-submodule -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[protobuf] Re: Is it possible to statically link protobuf without using pthread?
I'm also trying to do the same thing, specifically for an embedded ARM Cortex-M4 processor. I am using the GCC ARM Embedded [0] toolchain. Did you have any luck with this? Dale [0] -- https://launchpad.net/gcc-arm-embedded On Tuesday, January 15, 2013 4:00:05 AM UTC-8, hash code wrote: I can link protobuf statically with pthread: g++ -g -Wall example.pb.cc example.cc -o example -static -lprotobuf -lpthread However I am going to use protobuf on the real time OS where pthread library is not available. Is it possible to configure protobuf that way? -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[protobuf] Protocol Buffers on an embedded system
I am collecting data from sensors with an ARM Cortex-M4 running and RTOS, and logging the data in binary format to an micro SD card, which is then periodically transferred to a PC for analysis. I am using the FatFS [0] FAT32 filesystem library along with the ChibiOS RTOS [1]. After a few revisions of my data format, I realized that this problem is directly solved by Google Protocol Buffers and that I should be using something like it to manage the serialization/deserialization of my data. However, it seems there are some challenges to get my firmware to build. The two issues I'm facing are: 1) pthreads isn't available for my target. 2) Target device is an ARM Cortex-M4 with the armv7m-e instruction set. 3) ostream does not work out of the box in the embedded environment (though I might be able to serialize to a string) Here are my questions: 1) Has anybody successfully used protocol buffers in an identical or very similar environment? If so, can you point me to your code so I can see how you made it work, or offer any advice on what is needed? 2) I am tempted to use another project called nanopb [2] because it seems like it might be easier to get working on my embedded system. Has anybody here used it who can report on how well it worked when inter-operating with GPB? nanopb should be compatible with GPB but I just want to make sure the process is actually smooth sailing. All my embedded code is in C++ so I would prefer to use GPB over nanopb if possible, but if it is too much to make GPB work on the embedded system then I guess I would have to use both (nanopb on firmware and GPB on PC that decodes data from the SD card). Thank you for any insights or thoughts. Dale [0] -- http://elm-chan.org/fsw/ff/00index_e.html [1] -- http://chibios.org/dokuwiki/doku.php [2] -- http://koti.kapsi.fi/jpa/nanopb/ -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
