Re: Cross-compile of Google Protocol Buffers fails
On Wed, May 20, 2009 at 12:15 PM, reid.mad...@tektronix.com wrote: Kenton, Okay, let's say I build and install GPB on the current machine before building a cross-compiled version for deployment. When the cross-compiled build runs which 'protoc' does it use in its testing? Well, given the option you provide, it uses the version installed locally and not the cross-compiled version I just built (which for obvious reasons it cannot use). So, 6 months from now I build a new cross-compiled version of GPB on that machine. Which version of protoc does it use in it's tests? Same answer as before, it uses the version it built 6-months ago. This testing has absolutely zero worth because it is not testing anything I build using the cross-compiler. As I said before, the tests won't compile in this case, because it will detect that the version of protoc is incompatible. You will have to update the local protoc first, and then the tests are testing the right thing. Only when GPB is built natively can tests be performed to validate the results of the build. When building with a cross-compiler the protoc, and associated libraries, generated by the build can never be tested on the local machine. To hack up the GPB build process to run tests using a locally installed GPB (which may be years old), instead of the cross-compiled results, is a completely invalid exercise. It doesn't test anything that the build produces and leads to a false sense of success. In the GNU build paradigm, when a cross-compiler is being used, no attempt is ever made to run tests. It is a good model to follow. No, sorry, but any model which involves skipping tests when there's a reasonable chance they may not pass is not a good model to follow, and would never fly at Google. IMO, you should provide a --cross-compile option so the build knows when to not run tests… Or, better yet, use the auto-tools to detect cross-compilation environments and auto-disable testing… Reid *From:* Kenton Varda [mailto:ken...@google.com] *Sent:* Friday, May 15, 2009 9:23 PM *To:* Madsen, Reid *Cc:* anthony.p...@pri.com.au; protobuf@googlegroups.com *Subject:* Re: Cross-compile of Google Protocol Buffers fails On Fri, May 15, 2009 at 5:06 PM, reid.mad...@tektronix.com wrote: Kenton, The flaw in your assumption is that I actually use the current machine to build my product. I don't. I'm cross-compiling on platform A (redhat) for deployment on platform B (mips). Other than the cross-compilation, I never, ever, need to run 'protoc' on platform A (redhat). The cross-compiled protoc compiler is never used until after it has been deployed on platform B (mips). Clear? OK, but I think this situation is unusual, and I don't think that the extra step of compiling a local copy of protoc is that difficult. You don't even have to install it. Just do: tar jxvf protobuf-2.1.0.tar.bz2 cd protobuf-2.1.0 mkdir host target cd host ../configure make cd ../target ../configure --with-protoc=../host/src/protoc [YOUR FLAGS HERE] make Now, with your fix, I must build GPB on platform A (redhat) just so I can cross-compile GPB for platform B (mips). That is an entirely unnecessary dependency. Since the tests can never be run using the cross-compiled protoc, there is absolutely no need to generate the files used by those tests. This is false. The tests are testing that the protocol buffers *runtime* works on the target machine. Protoc is used to *compile* the tests, just like you'd use protoc to compile your product that uses protocol buffers. It's very important that the *output* of protoc does, in fact, compile and execute properly for the target, even if you do not run protoc itself on the target. Furthermore, if there was a protoc installed locally on platform A (redhat), there is no guarantee that is is compatible with the version I am cross-compiling for platform B (mips) -- which would again require me to build a local version… Protoc produces identical output on all platforms. The only way the host protoc could be incompatible with the target is if they were built from different versions of the protocol buffers package. In this case, you will get a C++ compiler error when compiling the output, as the output contains preprocessor directives to verify version compatibility. In the GNU software world, when a cross-compile is being performed, no tests are performed, and no attempt is made to generate files used by the tests. It requires a leap of faith… That is the model you should be following. That is a terrible model. Tests are important and skipping tests is a very bad idea. I have not tested protocol buffers on MIPS since I do not have access to any MIPS machines. It's very possible that the protobuf library contains bugs on MIPS. It is therefore very important that you run the tests on MIPS to verify
Re: Cross-compile of Google Protocol Buffers fails
On Thu, May 14, 2009 at 4:08 PM, reid.mad...@tektronix.com wrote: There's only one problem with this. It assumes that you have a locally compiled and installed the protoc compiler. Thus, before building a cross-compiled version, one must also build a native version. This is an unnecessary dependency that should be avoided. You have to install the compiler locally anyway in order to compile any code that uses protocol buffers, right? So why is this a problem? Instead, you should provide a mechanism to disable the generation of all tests files. Unfortunately autotool makes this hard. It's mechanism for dynamically-generated sources is very weak and you cannot tell it only generate these if building tests. But also, you really should build the tests and run them on your target platform, to make sure there are no subtle problems with protocol buffers on that platform. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
RE: Cross-compile of Google Protocol Buffers fails
Kenton, The flaw in your assumption is that I actually use the current machine to build my product. I don't. I'm cross-compiling on platform A (redhat) for deployment on platform B (mips). Other than the cross-compilation, I never, ever, need to run 'protoc' on platform A (redhat). The cross-compiled protoc compiler is never used until after it has been deployed on platform B (mips). Clear? Now, with your fix, I must build GPB on platform A (redhat) just so I can cross-compile GPB for platform B (mips). That is an entirely unnecessary dependency. Since the tests can never be run using the cross-compiled protoc, there is absolutely no need to generate the files used by those tests. Furthermore, if there was a protoc installed locally on platform A (redhat), there is no guarantee that is is compatible with the version I am cross-compiling for platform B (mips) -- which would again require me to build a local version... In the GNU software world, when a cross-compile is being performed, no tests are performed, and no attempt is made to generate files used by the tests. It requires a leap of faith... That is the model you should be following. The autotools support this, but is extremely obtuse. Luckily for me I am only a consumer of GNU components, and not an author of any... As a supplier of third party tools you should work to resolve this issue. Until there is a better solution, I will continue to rip out the test parts from the GPB makefiles so that my cross-compiles don't croak. Regards, Reid Madsen From: Kenton Varda [mailto:ken...@google.com] Sent: Friday, May 15, 2009 6:23 PM To: Madsen, Reid Cc: anthony.p...@pri.com.au; protobuf@googlegroups.com Subject: Re: Cross-compile of Google Protocol Buffers fails On Thu, May 14, 2009 at 4:08 PM, reid.mad...@tektronix.commailto:reid.mad...@tektronix.com wrote: There's only one problem with this. It assumes that you have a locally compiled and installed the protoc compiler. Thus, before building a cross-compiled version, one must also build a native version. This is an unnecessary dependency that should be avoided. You have to install the compiler locally anyway in order to compile any code that uses protocol buffers, right? So why is this a problem? Instead, you should provide a mechanism to disable the generation of all tests files. Unfortunately autotool makes this hard. It's mechanism for dynamically-generated sources is very weak and you cannot tell it only generate these if building tests. But also, you really should build the tests and run them on your target platform, to make sure there are no subtle problems with protocol buffers on that platform. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Cross-compile of Google Protocol Buffers fails
Hi Reid, I guess Kenton would be happy about a patch as well, seems you already know where to look. That is the nice thing about OpenSource: you can contribute and have the case you care about fixed as opposed to closed source where you hope that it is on the someones' agenda. Get involved. -h On Fri, May 15, 2009 at 5:06 PM, reid.mad...@tektronix.com wrote: Kenton, The flaw in your assumption is that I actually use the current machine to build my product. I don't. I'm cross-compiling on platform A (redhat) for deployment on platform B (mips). Other than the cross-compilation, I never, ever, need to run 'protoc' on platform A (redhat). The cross-compiled protoc compiler is never used until after it has been deployed on platform B (mips). Clear? Now, with your fix, I must build GPB on platform A (redhat) just so I can cross-compile GPB for platform B (mips). That is an entirely unnecessary dependency. Since the tests can never be run using the cross-compiled protoc, there is absolutely no need to generate the files used by those tests. Furthermore, if there was a protoc installed locally on platform A (redhat), there is no guarantee that is is compatible with the version I am cross-compiling for platform B (mips) -- which would again require me to build a local version… In the GNU software world, when a cross-compile is being performed, no tests are performed, and no attempt is made to generate files used by the tests. It requires a leap of faith… That is the model you should be following. The autotools support this, but is extremely obtuse. Luckily for me I am only a consumer of GNU components, and not an author of any… As a supplier of third party tools you should work to resolve this issue. Until there is a better solution, I will continue to rip out the test parts from the GPB makefiles so that my cross-compiles don't croak. Regards, Reid Madsen From: Kenton Varda [mailto:ken...@google.com] Sent: Friday, May 15, 2009 6:23 PM To: Madsen, Reid Cc: anthony.p...@pri.com.au; protobuf@googlegroups.com Subject: Re: Cross-compile of Google Protocol Buffers fails On Thu, May 14, 2009 at 4:08 PM, reid.mad...@tektronix.com wrote: There's only one problem with this. It assumes that you have a locally compiled and installed the protoc compiler. Thus, before building a cross-compiled version, one must also build a native version. This is an unnecessary dependency that should be avoided. You have to install the compiler locally anyway in order to compile any code that uses protocol buffers, right? So why is this a problem? Instead, you should provide a mechanism to disable the generation of all tests files. Unfortunately autotool makes this hard. It's mechanism for dynamically-generated sources is very weak and you cannot tell it only generate these if building tests. But also, you really should build the tests and run them on your target platform, to make sure there are no subtle problems with protocol buffers on that platform. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Cross-compile of Google Protocol Buffers fails
On Fri, May 15, 2009 at 5:06 PM, reid.mad...@tektronix.com wrote: Kenton, The flaw in your assumption is that I actually use the current machine to build my product. I don't. I'm cross-compiling on platform A (redhat) for deployment on platform B (mips). Other than the cross-compilation, I never, ever, need to run 'protoc' on platform A (redhat). The cross-compiled protoc compiler is never used until after it has been deployed on platform B (mips). Clear? OK, but I think this situation is unusual, and I don't think that the extra step of compiling a local copy of protoc is that difficult. You don't even have to install it. Just do: tar jxvf protobuf-2.1.0.tar.bz2 cd protobuf-2.1.0 mkdir host target cd host ../configure make cd ../target ../configure --with-protoc=../host/src/protoc [YOUR FLAGS HERE] make Now, with your fix, I must build GPB on platform A (redhat) just so I can cross-compile GPB for platform B (mips). That is an entirely unnecessary dependency. Since the tests can never be run using the cross-compiled protoc, there is absolutely no need to generate the files used by those tests. This is false. The tests are testing that the protocol buffers *runtime* works on the target machine. Protoc is used to *compile* the tests, just like you'd use protoc to compile your product that uses protocol buffers. It's very important that the *output* of protoc does, in fact, compile and execute properly for the target, even if you do not run protoc itself on the target. Furthermore, if there was a protoc installed locally on platform A (redhat), there is no guarantee that is is compatible with the version I am cross-compiling for platform B (mips) -- which would again require me to build a local version… Protoc produces identical output on all platforms. The only way the host protoc could be incompatible with the target is if they were built from different versions of the protocol buffers package. In this case, you will get a C++ compiler error when compiling the output, as the output contains preprocessor directives to verify version compatibility. In the GNU software world, when a cross-compile is being performed, no tests are performed, and no attempt is made to generate files used by the tests. It requires a leap of faith… That is the model you should be following. That is a terrible model. Tests are important and skipping tests is a very bad idea. I have not tested protocol buffers on MIPS since I do not have access to any MIPS machines. It's very possible that the protobuf library contains bugs on MIPS. It is therefore very important that you run the tests on MIPS to verify that this is not the case. As a supplier of third party tools you should work to resolve this issue. Until there is a better solution, I will continue to rip out the test parts from the GPB makefiles so that my cross-compiles don't croak. If you were paying Google or me for this tool, then what you say might make sense. In the open source world, though, things are different: if Protocol Buffers does not meet your needs, you should submit a patch to resolve the issue. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
RE: Cross-compile of Google Protocol Buffers fails
There's only one problem with this. It assumes that you have a locally compiled and installed the protoc compiler. Thus, before building a cross-compiled version, one must also build a native version. This is an unnecessary dependency that should be avoided. Instead, you should provide a mechanism to disable the generation of all tests files. Reid From: Kenton Varda [mailto:ken...@google.com] Sent: Thursday, May 14, 2009 3:16 PM To: Madsen, Reid; anthonyp Cc: protobuf@googlegroups.com Subject: Re: Cross-compile of Google Protocol Buffers fails FYI: This is fixed in the 2.1.0 release. Use the --with-protoc option to configure to tell it to use a pre-compiled protoc executable during the build process, instead of using the one it just built. This should allow cross-compiling. On Mon, Feb 2, 2009 at 5:54 PM, Kenton Varda ken...@google.commailto:ken...@google.com wrote: Tests are not run as part of the build. I think the actual problem is that the makefile is attempting to invoke protoc to generate some .pb.* files which are then compiled into the tests (though I think automake builds them even if you aren't compiling tests because it is too dumb to know). You should be able to prevent this by making sure those files are already there before you run make, e.g. by copying the .pb.* files over from a directory where you have run a normal build. I suppose we should find some trick to make this easier... On Mon, Feb 2, 2009 at 12:37 PM, reid.mad...@tektronix.commailto:reid.mad...@tektronix.com wrote: The build procedures for GPB don't work when cross-compiling. The crux of the matter is that the build attempts to run tests using cross-compiled binaries. This of course fails because the cross-compiled binaries are incompatible with the current host's architecture. To support cross-compilation, tests should never be run as a part of the build, but should be deferred to a make check, or make tests step. ___ Reid Madsen | Principal Engineer | Tektronix, Inc. | 469-330-4018 | reid.mad...@tek.commailto:reid.mad...@tek.commailto:reid.mad...@tek.commailto:reid.mad...@tek.com --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Cross-compile of Google Protocol Buffers fails
It's on my todo list. On Tue, Feb 24, 2009 at 3:28 PM, anthonyp anthony.p...@pri.com.au wrote: I experienced the same problem when cross-compiling. As Kenton suggested you can copy the files prior to building but it does not prevent the makefile file from attempting to generate the files again, therefore failing the build process. So not only do you need to copy the generated files over, but you also need to modify the src/ makefile.am file to disable the process of generating *.pb.* files. After modifying the makefile.am you can then use automake to generate the makefile.in file. I was able to successfully cross-compile the google buffers source code to an ARM platform, but I would suggest that this should be automated by google in future releases. On Feb 3, 11:54 am, Kenton Varda ken...@google.com wrote: Tests are not run as part of the build. I think the actual problem is that the makefile is attempting to invoke protoc to generate some .pb.* files which are then compiled into the tests (though I think automake builds them even if you aren't compiling tests because it is too dumb to know). You should be able to prevent this by making sure those files are already there before you run make, e.g. by copying the .pb.* files over from a directory where you have run a normal build. I suppose we should find some trick to make this easier... On Mon, Feb 2, 2009 at 12:37 PM, reid.mad...@tektronix.com wrote: The build procedures for GPB don't work when cross-compiling. The crux of the matter is that the build attempts to run tests using cross-compiled binaries. This of course fails because the cross-compiled binaries are incompatible with the current host's architecture. To support cross-compilation, tests should never be run as a part of the build, but should be deferred to a make check, or make tests step. ___ Reid Madsen | Principal Engineer | Tektronix, Inc. | 469-330-4018 | reid.mad...@tek.commailto:reid.mad...@tek.com --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
