Re: Difficulties building Swig-generated C++ code for proton-jni binding (due to nested typedef union)
On 21 March 2013 12:39, Keith W keith.w...@gmail.com wrote: Hi Cliff The winsock.h change resolved the remaining C++ compilation errors. Will this be part of your patch? The next error I see is still from a java step within binding/java/CMakeLists.txt - a problem with determining a list of java source files generated by swig. Do you see the same? This is code that Phil and I contributed so we'll take a look - unless you have already started. I see the issue. The colon separating windows drive is being treated as if it were a path separator and this is causing the scipt to bomb out. I'll raise a Jira and fix the CMakeLists.txt. X:\src\proton\build\ALL_BUILD.vcxproj (default target) (1) - X:\src\proton\build\proton-c\bindings\java\proton-jni.vcxproj (default target ) (7) - X:\src\proton\build\proton-c\bindings\java\proton-jni-sourcefile-list.vcxproj (default target) (8) - (CustomBuild target) - C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppCommon.targets (151,5): error MSB6006: cmd.exe exited with code 1. [X:\src\proton\build\prot on-c\bindings\java\proton-jni-sourcefile-list.vcxproj] 0 Warning(s) 1 Error(s) Time Elapsed 00:00:48.50 regards, Keith On 20 March 2013 16:57, Cliff Jansen cliffjan...@gmail.com wrote: Hi Keith, Sorry, the compile errors you see on Windows should go away with the following in java.i: 26a27,29 #if defined(_WIN32) ! defined(__CYGWIN__) #include winsock2.h #endif I am able to generate protocol.h and encodings.h on my Windows systems (a range of XP - Windows 8, VS2008 - VS2012, 32/64 bit). I am using fairly up to date versions of python, cmake and swig. So I am not sure why it is failing for you. Note that Windows support is still a work in progress. For 0.4 I only have completed building proton-c on Windows and the limited ctest infrastructure (i.e. syntax errors and CMakeLists.txt). Some proton-test tests fail and investigation is pending. I wonder if it is possible to move your workaround into its own .i file which would be located alongside codec.h? java.i would then include it using a %import. This approach would also facilitate sharing with the other bindings (surely they would need the same solution??). Let me think about that or perhaps another way to address your (sensible) concern and get back to you later today or tomorrow. Cliff On Wed, Mar 20, 2013 at 11:33 AM, Keith W keith.w...@gmail.com wrote: Hello Cliff I too share your dislike of my suggested change to code.h:) I do prefer your alternative despite the duplication. However I worry that the copy - deep within the java bindings tree would be overlooked when changes are made to codec.h and the two would diverge, leading to obscure defects. I wonder if it is possible to move your workaround into its own .i file which would be located alongside codec.h? java.i would then include it using a %import. This approach would also facilitate sharing with the other bindings (surely they would need the same solution??). I had just got this working yesterday on Linux and was trying a Windows build before posting a review. It fails in cmake scripts working with windows paths (and maybe other problems). I see an issue on Windows (Windows 7/MS Visual Studio 2010/Python 2.7.3) with the generation of protocol.h and encodings.h. It behaves as if the cmd-shell is ignoring the redirection (). I see the generated file flash by in cmd window and no .h is written. I'm working around by executing: python X:/src/proton/proton-c/env.py PYTHONPATH=X:/src/proton/proton-c python X:/src/proton/proton-c/src/protocol.h.py X:/src/proton/buildwin/proton-c/protocol.h python X:/src/proton/proton-c/env.py PYTHONPATH=X:/src/proton/proton-c python X:/src/proton/proton-c/src/codec/encodings.h.py X:/src/proton/buildwin/proton-c/encodings.h Do you see this? My windows box behaves oddly at the best of times. On Windows I still see a number of C++ compilation errors from javaJAVA_wrap.cxx. Do you see these? I haven't started to investigate yet. 11X:\src\proton\proton-c\include\proton/driver_extras.h(40): error C2146: syntax error : missing ';' before identifier 'pn_socket_t' 11X:\src\proton\proton-c\include\proton/driver_extras.h(40): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int 11X:\src\proton\proton-c\include\proton/driver_extras.h(40): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int 11X:\src\proton\proton-c\include\proton/driver_extras.h(53): error C2061: syntax error : identifier 'pn_socket_t' 11X:\src\proton\proton-c\include\proton/driver_extras.h(63): error C2061: syntax error : identifier 'pn_socket_t' 11javaJAVA_wrap.cxx(1062): warning C4018: '' : signed/unsigned mismatch 11javaJAVA_wrap.cxx(1089): warning C4018: '' : signed/unsigned mismatch 11javaJAVA_wrap.cxx(7754): error C2146: syntax error :
Re: Difficulties building Swig-generated C++ code for proton-jni binding (due to nested typedef union)
Hello Cliff I too share your dislike of my suggested change to code.h:) I do prefer your alternative despite the duplication. However I worry that the copy - deep within the java bindings tree would be overlooked when changes are made to codec.h and the two would diverge, leading to obscure defects. I wonder if it is possible to move your workaround into its own .i file which would be located alongside codec.h? java.i would then include it using a %import. This approach would also facilitate sharing with the other bindings (surely they would need the same solution??). I had just got this working yesterday on Linux and was trying a Windows build before posting a review. It fails in cmake scripts working with windows paths (and maybe other problems). I see an issue on Windows (Windows 7/MS Visual Studio 2010/Python 2.7.3) with the generation of protocol.h and encodings.h. It behaves as if the cmd-shell is ignoring the redirection (). I see the generated file flash by in cmd window and no .h is written. I'm working around by executing: python X:/src/proton/proton-c/env.py PYTHONPATH=X:/src/proton/proton-c python X:/src/proton/proton-c/src/protocol.h.py X:/src/proton/buildwin/proton-c/protocol.h python X:/src/proton/proton-c/env.py PYTHONPATH=X:/src/proton/proton-c python X:/src/proton/proton-c/src/codec/encodings.h.py X:/src/proton/buildwin/proton-c/encodings.h Do you see this? My windows box behaves oddly at the best of times. On Windows I still see a number of C++ compilation errors from javaJAVA_wrap.cxx. Do you see these? I haven't started to investigate yet. 11X:\src\proton\proton-c\include\proton/driver_extras.h(40): error C2146: syntax error : missing ';' before identifier 'pn_socket_t' 11X:\src\proton\proton-c\include\proton/driver_extras.h(40): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int 11X:\src\proton\proton-c\include\proton/driver_extras.h(40): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int 11X:\src\proton\proton-c\include\proton/driver_extras.h(53): error C2061: syntax error : identifier 'pn_socket_t' 11X:\src\proton\proton-c\include\proton/driver_extras.h(63): error C2061: syntax error : identifier 'pn_socket_t' 11javaJAVA_wrap.cxx(1062): warning C4018: '' : signed/unsigned mismatch 11javaJAVA_wrap.cxx(1089): warning C4018: '' : signed/unsigned mismatch 11javaJAVA_wrap.cxx(7754): error C2146: syntax error : missing ';' before identifier 'arg2' 11javaJAVA_wrap.cxx(7754): error C2065: 'arg2' : undeclared identifier 11javaJAVA_wrap.cxx(7761): error C2065: 'arg2' : undeclared identifier 11javaJAVA_wrap.cxx(7761): error C2146: syntax error : missing ';' before identifier 'jarg2' 11javaJAVA_wrap.cxx(7763): error C2065: 'arg2' : undeclared identifier 11javaJAVA_wrap.cxx(7772): error C2146: syntax error : missing ';' before identifier 'arg2' 11javaJAVA_wrap.cxx(7772): error C2065: 'arg2' : undeclared identifier 11javaJAVA_wrap.cxx(7779): error C2065: 'arg2' : undeclared identifier 11javaJAVA_wrap.cxx(7779): error C2146: syntax error : missing ';' before identifier 'jarg2' 11javaJAVA_wrap.cxx(7781): error C2065: 'arg2' : undeclared identifier Let me know if you wish to tackle the remaining windows build issues. I can't offer to take these on firstly as my focus is Engine at present, and C/C++ it is no longer my area of expertise (12 years of rust). I will pitch in and help with any questions you may have with the JNI aspects if I can. I am also in the process of getting this build set up on Apache Jenkins. Kind regards, Keith On 20 March 2013 03:49, Cliff Jansen cliffjan...@gmail.com wrote: Hi Keith, Very nice work there, thanks. I've been looking at this too and had an alternate solution to the nested union problem (https://issues.apache.org/jira/secure/attachment/12574482/pn254-cj-0.patch). It is based on the observation that there is nothing wrong with the definition in codec.h that swig complains about to begin with. That, plus it irks me to change a public interface file just to satisfy the one C99 unfriendly compiler in the world. So I looked for a solution that was specific to swig. Sadly, swig handled this just fine in the 1.X series, precisely to handle cases such as here. However that apparently exacerbated problems elsewhere and, as far as I can tell, their new fancy %nestedworkaround fix in swig 2.X can't work in our case and leaves us high and dry. The patch I provided just restores the swig 1.X behaviour. The downside is the (twice) repeated re-definition of the union (bleh, my swig knowledge could find no better solution). Your patch is cleaner in that respect. If my patch seems distasteful in its foibles, I would look to Rafi for his views on changing codec.h. Let me know what you think. I had just got this working yesterday on Linux and was trying a Windows build before posting a review. It fails in cmake scripts working with
Re: Difficulties building Swig-generated C++ code for proton-jni binding (due to nested typedef union)
Hi Keith, Sorry, the compile errors you see on Windows should go away with the following in java.i: 26a27,29 #if defined(_WIN32) ! defined(__CYGWIN__) #include winsock2.h #endif I am able to generate protocol.h and encodings.h on my Windows systems (a range of XP - Windows 8, VS2008 - VS2012, 32/64 bit). I am using fairly up to date versions of python, cmake and swig. So I am not sure why it is failing for you. Note that Windows support is still a work in progress. For 0.4 I only have completed building proton-c on Windows and the limited ctest infrastructure (i.e. syntax errors and CMakeLists.txt). Some proton-test tests fail and investigation is pending. I wonder if it is possible to move your workaround into its own .i file which would be located alongside codec.h? java.i would then include it using a %import. This approach would also facilitate sharing with the other bindings (surely they would need the same solution??). Let me think about that or perhaps another way to address your (sensible) concern and get back to you later today or tomorrow. Cliff On Wed, Mar 20, 2013 at 11:33 AM, Keith W keith.w...@gmail.com wrote: Hello Cliff I too share your dislike of my suggested change to code.h:) I do prefer your alternative despite the duplication. However I worry that the copy - deep within the java bindings tree would be overlooked when changes are made to codec.h and the two would diverge, leading to obscure defects. I wonder if it is possible to move your workaround into its own .i file which would be located alongside codec.h? java.i would then include it using a %import. This approach would also facilitate sharing with the other bindings (surely they would need the same solution??). I had just got this working yesterday on Linux and was trying a Windows build before posting a review. It fails in cmake scripts working with windows paths (and maybe other problems). I see an issue on Windows (Windows 7/MS Visual Studio 2010/Python 2.7.3) with the generation of protocol.h and encodings.h. It behaves as if the cmd-shell is ignoring the redirection (). I see the generated file flash by in cmd window and no .h is written. I'm working around by executing: python X:/src/proton/proton-c/env.py PYTHONPATH=X:/src/proton/proton-c python X:/src/proton/proton-c/src/protocol.h.py X:/src/proton/buildwin/proton-c/protocol.h python X:/src/proton/proton-c/env.py PYTHONPATH=X:/src/proton/proton-c python X:/src/proton/proton-c/src/codec/encodings.h.py X:/src/proton/buildwin/proton-c/encodings.h Do you see this? My windows box behaves oddly at the best of times. On Windows I still see a number of C++ compilation errors from javaJAVA_wrap.cxx. Do you see these? I haven't started to investigate yet. 11X:\src\proton\proton-c\include\proton/driver_extras.h(40): error C2146: syntax error : missing ';' before identifier 'pn_socket_t' 11X:\src\proton\proton-c\include\proton/driver_extras.h(40): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int 11X:\src\proton\proton-c\include\proton/driver_extras.h(40): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int 11X:\src\proton\proton-c\include\proton/driver_extras.h(53): error C2061: syntax error : identifier 'pn_socket_t' 11X:\src\proton\proton-c\include\proton/driver_extras.h(63): error C2061: syntax error : identifier 'pn_socket_t' 11javaJAVA_wrap.cxx(1062): warning C4018: '' : signed/unsigned mismatch 11javaJAVA_wrap.cxx(1089): warning C4018: '' : signed/unsigned mismatch 11javaJAVA_wrap.cxx(7754): error C2146: syntax error : missing ';' before identifier 'arg2' 11javaJAVA_wrap.cxx(7754): error C2065: 'arg2' : undeclared identifier 11javaJAVA_wrap.cxx(7761): error C2065: 'arg2' : undeclared identifier 11javaJAVA_wrap.cxx(7761): error C2146: syntax error : missing ';' before identifier 'jarg2' 11javaJAVA_wrap.cxx(7763): error C2065: 'arg2' : undeclared identifier 11javaJAVA_wrap.cxx(7772): error C2146: syntax error : missing ';' before identifier 'arg2' 11javaJAVA_wrap.cxx(7772): error C2065: 'arg2' : undeclared identifier 11javaJAVA_wrap.cxx(7779): error C2065: 'arg2' : undeclared identifier 11javaJAVA_wrap.cxx(7779): error C2146: syntax error : missing ';' before identifier 'jarg2' 11javaJAVA_wrap.cxx(7781): error C2065: 'arg2' : undeclared identifier Let me know if you wish to tackle the remaining windows build issues. I can't offer to take these on firstly as my focus is Engine at present, and C/C++ it is no longer my area of expertise (12 years of rust). I will pitch in and help with any questions you may have with the JNI aspects if I can. I am also in the process of getting this build set up on Apache Jenkins. Kind regards, Keith
Re: Difficulties building Swig-generated C++ code for proton-jni binding (due to nested typedef union)
On 1 March 2013 16:15, Cliff Jansen cliffjan...@gmail.com wrote: I did all that and verified the c++ swig call was used and g++ on javaJAVA_wrap.cxx. I also needed the fix in PROTON-249 to progress on Windows. Let me know if you have made further progress. Otherwise, I will keep digging. Cliff Hi Cliff I made a commit under PROTON-254 that a) changes bindings/java/CMakeLists.txt to use CPLUSPLUS ON directive in the case where BUILD_WITH_CXX has been enabled b) resolves the compilation errors that resulted from compiling the swig generated C++ code. That leaves the problem with Nested union not currently supported' reported by Swig. You'll remember that Swig doesn't support nested types when operating in C++ mode, and so ignores the nested union within pn_atom_t. This means swig fails to generate the pn_atom_t_u java class and so our hand-rolled java binding code (java) fails to compile. I've attached a patch to PROTON-254 that works around the issue by simply moving the union to the top level. This seems to avoid the swig limitation seemingly without breaking anything else. Do you see a problem with this approach? Can you think of an neater alternative? The patch allows the JNI tests to pass under C++ on Ubuntu and MacOSX. I'm yet to try on Windows. Kind regards, Keith.
Re: Difficulties building Swig-generated C++ code for proton-jni binding (due to nested typedef union)
Hi Keith, Very nice work there, thanks. I've been looking at this too and had an alternate solution to the nested union problem (https://issues.apache.org/jira/secure/attachment/12574482/pn254-cj-0.patch). It is based on the observation that there is nothing wrong with the definition in codec.h that swig complains about to begin with. That, plus it irks me to change a public interface file just to satisfy the one C99 unfriendly compiler in the world. So I looked for a solution that was specific to swig. Sadly, swig handled this just fine in the 1.X series, precisely to handle cases such as here. However that apparently exacerbated problems elsewhere and, as far as I can tell, their new fancy %nestedworkaround fix in swig 2.X can't work in our case and leaves us high and dry. The patch I provided just restores the swig 1.X behaviour. The downside is the (twice) repeated re-definition of the union (bleh, my swig knowledge could find no better solution). Your patch is cleaner in that respect. If my patch seems distasteful in its foibles, I would look to Rafi for his views on changing codec.h. Let me know what you think. I had just got this working yesterday on Linux and was trying a Windows build before posting a review. It fails in cmake scripts working with windows paths (and maybe other problems). But it does compile the swig part in Visual Studio. Let me know if you wish to tackle the remaining windows build issues. Cliff On Tue, Mar 19, 2013 at 7:50 PM, Keith W keith.w...@gmail.com wrote: On 1 March 2013 16:15, Cliff Jansen cliffjan...@gmail.com wrote: I did all that and verified the c++ swig call was used and g++ on javaJAVA_wrap.cxx. I also needed the fix in PROTON-249 to progress on Windows. Let me know if you have made further progress. Otherwise, I will keep digging. Cliff Hi Cliff I made a commit under PROTON-254 that a) changes bindings/java/CMakeLists.txt to use CPLUSPLUS ON directive in the case where BUILD_WITH_CXX has been enabled b) resolves the compilation errors that resulted from compiling the swig generated C++ code. That leaves the problem with Nested union not currently supported' reported by Swig. You'll remember that Swig doesn't support nested types when operating in C++ mode, and so ignores the nested union within pn_atom_t. This means swig fails to generate the pn_atom_t_u java class and so our hand-rolled java binding code (java) fails to compile. I've attached a patch to PROTON-254 that works around the issue by simply moving the union to the top level. This seems to avoid the swig limitation seemingly without breaking anything else. Do you see a problem with this approach? Can you think of an neater alternative? The patch allows the JNI tests to pass under C++ on Ubuntu and MacOSX. I'm yet to try on Windows. Kind regards, Keith.
Re: Difficulties building Swig-generated C++ code for proton-jni binding (due to nested typedef union)
I am trying to catch up with you to reproduce your problem. So far I can build on linux with cmake -DBUILD_WITH_CXX=ON. That works, so we know the problem is unrelated to swig itself or the switch to C++. It is either some problem with Visual Studio or a lack of strictness with g++. The warning you are seeing is from the Visual C++ compiler, and is just a warning. It is indeed a high profile suspect, but perhaps a red herring also. Right now I am trying to untangle the exact steps cmake is feeding into the vcxproj file so that I can compile the swig generated file by hand and zero in on the problem. But I don't claim to be in a better position to solve this than you. I doubt I will progress this much before the end of my day here. If you make further progress, let me know so that we can minimise duplicated effort. Cliff On Thu, Feb 28, 2013 at 11:41 AM, Phil Harvey p...@philharveyonline.com wrote: I have been working with Keith on PROTON-249 (Build fails on Win8 / VS 2012 with path error [1]). When building Proton from MS Visual Studio, we understand that a C++ (rather than C) compiler is used. We therefore tried doing a C++ build on Linux as a first step (i.e.running cmake with -DBUILD_WITH_CXX=ON), and ran into a number of problems - see PROTON-254 [2]. Most of the problems relate to the stricter rules in C++ around casting etc, and are easy to fix. However, the fact that Swig doesn't support nested unions in C++ typedef's means that it doesn't generate Java class pn_atom_t_u, which is our hand-written Java code depends on. We're interested in opinions about the best way forward, particularly from anyone who faced similar problems when building the other language bindings using C++. Thanks, Phil [1] https://issues.apache.org/jira/browse/PROTON-249 [2] https://issues.apache.org/jira/browse/PROTON-254
Re: Difficulties building Swig-generated C++ code for proton-jni binding (due to nested typedef union)
Hi Cliff, Thanks for looking into this. Simply setting the BUILD_WITH_CXX flag with a clean checkout doesn't expose the problem because it doesn't affect the language that Swig uses. You'd need to also: - Edit bindings/java/CMakeLists.txt to tell Swig to honour the BUILD_WITH_CXX flag and use C++ rather than C when generating the proton-jni code, like so: if (BUILD_WITH_CXX) SET_SOURCE_FILES_PROPERTIES(java.i PROPERTIES CPLUSPLUS ON) endif (BUILD_WITH_CXX) - Fix (or comment out) the offending lines in the generated code that violate C++'s stricter rules around casting void*'s etc. I'm not expecting you to actually do this ... unless you're feeling *really* keen :-) This allows the build to Swig-generate and compile the proton-jni C++ and Java code. However, the Java code that uses it fails to compile because Swig didn't generate the pn_atom_t_u Java class, due to its lack of support for nested unions in typedefs. It's all rather convoluted, but hopefully that clarifies the problem somewhat. Phil Not sure if that clears On 1 March 2013 08:47, Cliff Jansen cliffjan...@gmail.com wrote: I am trying to catch up with you to reproduce your problem. So far I can build on linux with cmake -DBUILD_WITH_CXX=ON. That works, so we know the problem is unrelated to swig itself or the switch to C++. It is either some problem with Visual Studio or a lack of strictness with g++. The warning you are seeing is from the Visual C++ compiler, and is just a warning. It is indeed a high profile suspect, but perhaps a red herring also. Right now I am trying to untangle the exact steps cmake is feeding into the vcxproj file so that I can compile the swig generated file by hand and zero in on the problem. But I don't claim to be in a better position to solve this than you. I doubt I will progress this much before the end of my day here. If you make further progress, let me know so that we can minimise duplicated effort. Cliff On Thu, Feb 28, 2013 at 11:41 AM, Phil Harvey p...@philharveyonline.com wrote: I have been working with Keith on PROTON-249 (Build fails on Win8 / VS 2012 with path error [1]). When building Proton from MS Visual Studio, we understand that a C++ (rather than C) compiler is used. We therefore tried doing a C++ build on Linux as a first step (i.e.running cmake with -DBUILD_WITH_CXX=ON), and ran into a number of problems - see PROTON-254 [2]. Most of the problems relate to the stricter rules in C++ around casting etc, and are easy to fix. However, the fact that Swig doesn't support nested unions in C++ typedef's means that it doesn't generate Java class pn_atom_t_u, which is our hand-written Java code depends on. We're interested in opinions about the best way forward, particularly from anyone who faced similar problems when building the other language bindings using C++. Thanks, Phil [1] https://issues.apache.org/jira/browse/PROTON-249 [2] https://issues.apache.org/jira/browse/PROTON-254
Difficulties building Swig-generated C++ code for proton-jni binding (due to nested typedef union)
I have been working with Keith on PROTON-249 (Build fails on Win8 / VS 2012 with path error [1]). When building Proton from MS Visual Studio, we understand that a C++ (rather than C) compiler is used. We therefore tried doing a C++ build on Linux as a first step (i.e.running cmake with -DBUILD_WITH_CXX=ON), and ran into a number of problems - see PROTON-254 [2]. Most of the problems relate to the stricter rules in C++ around casting etc, and are easy to fix. However, the fact that Swig doesn't support nested unions in C++ typedef's means that it doesn't generate Java class pn_atom_t_u, which is our hand-written Java code depends on. We're interested in opinions about the best way forward, particularly from anyone who faced similar problems when building the other language bindings using C++. Thanks, Phil [1] https://issues.apache.org/jira/browse/PROTON-249 [2] https://issues.apache.org/jira/browse/PROTON-254
