proton-j/proton-c integration primer
This question has been raised a couple times on IRC now, so I figured I should write up a quick summary. If you're newish to proton development, you might be wondering why there are two apparently independent projects (proton-c and proton-j) that share the same git repo and release cycle. These projects aren't actually quite as independent as they look. There is a common test suite that runs against both implementations to help keep them in sync with each other. This is the python test suite that lives underneath the top level tests/python directory. These tests have been integrated into the maven build via jython so that the whole java build looks like a normal maven build and java developers don't need to deal with cmake or getting jython themselves or anything that's outside of the normal java experience. Likewise, the cmake build will detect if java is unavailable and opt out of building the java code, so it's easy for C developers to pretend that Java doesn't exist. It also does this for all the different bindings, e.g. if perl or ruby isn't installed it will not attempt to build them. As a core proton developer, it's a good idea to make a point of having all the optional stuff enabled. This will ensure that you are testing any of your changes against the widest range of code possible. It's also good to be aware of exactly how the common test suite integrates with both proton-c and proton-j. The python binding actually consists of a hand written wrapper layer that encapsulates the raw swig generated API. The raw swig API is actually much lower level and is available in the 'cproton' module. When you run the python test suite in jython, the swig generated cproton doesn't actually exist since it is a C module. Instead, you get the cproton.py that resides in the java source tree (proton-j/src/main/resources/cproton.py). This cproton.py and its dependent files serve as a shim that adapts between the Java API and the C API. These files can actually be quite handy as a reference for how to translate a given idiom from proton-j into proton-c or vice versa. You might notice some parts of the shim are stubbed out, e.g. like this: def pn_flowcontroller(window): raise Skipped() Throwing a Skipped exception will signal to the test runner that whatever code is being called is not available, and the test will be reported as skipped rather than flagged as an error. I've put this content up on the proton web site here: http://qpid.apache.org/proton/proton-j-proton-c.html --Rafael
Re: [java] Message codec improvements
Overall I think this is a good direction. I made a bunch of more detailed comments on the patch. --Rafael On Mon, Feb 2, 2015 at 1:42 PM, Rajith Muditha Attapattu rajit...@gmail.com wrote: Rafi, further to our discussion I have posted a patch to illustrate the approach we plan to take. This should enable me to make progress until you get a chance to make further changes on the Decoder side. Regards, Rajith On Thu, Jan 29, 2015 at 3:59 PM, Rajith Muditha Attapattu rajit...@gmail.com wrote: More questions. For all the maps we return should we restrict them to String, Object or should it be Object, Object ? Technically one could use a Number (int, long) etc as a key.. Any opinion here? ;) On Thu, Jan 29, 2015 at 12:48 PM, Rafael Schloming r...@alum.mit.edu wrote: On Thu, Jan 29, 2015 at 12:28 PM, Rajith Muditha Attapattu rajit...@gmail.com wrote: Rafi could u pls answer the two questions I had in the code? 1. Your uint method only takes an int .. shouldn't it take a long? Bcos it could contain a value larger than a java int? To be honest I don't quite remember for sure, but I think it will do the two's complement and put it on the wire as a proper unsigned value. In other words I think it's just using the int type as a convenient/efficient way to pass around 4 bytes. 2. What should I use for boolean? there is no getBoolean .. or an equivalent method I think you might need to add this. I probably just omitted it. --Rafael
Re: puzzling issue with javascript bindings
FYI, thanks to some help from Fraser, this issue should now be resolved. --Rafael On Sat, Jan 31, 2015 at 1:24 PM, Rafael Schloming r...@alum.mit.edu wrote: On Sat, Jan 31, 2015 at 12:27 PM, Fraser Adams fraser.ad...@blueyonder.co.uk wrote: Hopefully you got the build I mailed you, but as an update I've upgraded my system to the latest emscripten incoming: emcc (Emscripten GCC-like replacement + linker emulating GNU ld ) 1.29.8 clang version 3.5.0 And did a Debug build on a clean system as before and I'm still not seeing any issues like the one you described and my recv.js seems pretty happy however many times I call send.js I'm pretty stumped!! What system are you running? Is there anything quirky? Do you have any exotic compilers or an unusual path? I did get your build and it worked fine for me. I emailed you a copy of my build just in case you are curious. My versions are: emcc (Emscripten GCC-like replacement) 1.29.0 (commit fdf24b478e1b26c0362a1627aca49ef82fd53f6a) clang version 3.4 My system is just stock fedora 20 as far as I'm aware. I installed emscripten by just following the directions in the README. --Rafael Frase On 31/01/15 15:37, Rafael Schloming wrote: Any chance you could send me a copy of your proton.js so I can try on my system? --Rafael On Sat, Jan 31, 2015 at 10:32 AM, Fraser Adams fraser.ad...@blueyonder.co.uk wrote: Hi again Rafi, As I'm on a roll today. So I've just done: cd build make clean rm CMakeCache.txt cmake -DCMAKE_BUILD_TYPE=Debug .. (which gives the message: JavaScript build type is Debug) make and when I did ./recv.js and in another window ./send.js I'm not seeing any issue: ./recv.js pipe() returning an error as we do not support them Address: amqp://0.0.0.0 Subject: UK.WEATHER Content: Hello World! Address: amqp://0.0.0.0 Subject: UK.WEATHER Content: Hello World! Address: amqp://0.0.0.0 Subject: UK.WEATHER Content: Hello World! Address: amqp://0.0.0.0 Subject: UK.WEATHER Content: Hello World! Address: amqp://0.0.0.0 Subject: UK.WEATHER Content: Hello World! Address: amqp://0.0.0.0 Subject: UK.WEATHER Content: Hello World! Address: amqp://0.0.0.0 Subject: UK.WEATHER Content: Hello World! Address: amqp://0.0.0.0 Subject: UK.WEATHER Content: Hello World! Looking in node_modules/qpid-proton-messenger/lib/proton-messenger.js I've definitely got the unminified/Debug version in play (which is also borne out by the pipe() returning an error as we do not support them message from emscripten's stub pipe call). So I'm not seeing what you are seeing on my system. I'm running Linux Mint 17.1 Rebecca emcc (Emscripten GCC-like replacement + linker emulating GNU ld ) 1.28.2 clang version 3.4 node v0.10.33 I doubt the changes I've just committed would make any difference, so there's definitely something weird. I'll update my emscripten/fastcomp versions and see if that introduces this quirk. On the plus side, at least it wasn't my imagination and I really haven't seen this behaviour before :-D If it *is* an emscripten issue it'd be good to have a minimal reproducer to attach to an issue report. Frase On 31/01/15 12:17, Rafael Schloming wrote: Hi, I recently uncovered a puzzling issue with the Javascript bindings. If I fire up the simple recv.js example and then run send.js, it works fine the first time, but the second time around I get the error below. This apparently has been happening for a while (possibly a few months), however I haven't noticed because it only happens with debug builds, and only then the second time around. With a regular build everything seems to work fine. I googled around a bit for this particular error and there are a bunch of threads talking about how casting function pointers doesn't work with emscripten and you need to avoid that, but I don't believe we actually do that ever, and I see nothing to indicate that this sort of error would go away with a non Debug build. The stack trace (which you only get after rebuilding with ASSERTIONS=2) seems pretty straightforward. It is inside pn_class_new which delegates to a function pointer held in the pn_class_t struct that is passed in. I believe that function pointer lookup is what is failing, but when I put printfs in the C code to confirm this, the problem goes away. At this point I'm kind of left scratching my head. The only thing I can think of short of a compiler bug is that somehow the pn_class_t struct is becoming corrupted, but I would expect valgrind to show up any sort of memory issues like that, unless somehow it is only being triggered from the javascript binding. Anyways, I figured I should give people a heads up. The workaround is easy enough, just build with non Debug build and everything seems to work fine. Beyond that, any insight would be greatly appreciated. == [rhs@localhost proton]$
Non-reactive engine examples?
Are there any example apps for Python that don't use the reactive APIs? -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpBlkS2f_1Sx.pgp Description: PGP signature
[jira] [Commented] (PROTON-812) LinkException needs an attribute that indicates the reason for the exception.
[ https://issues.apache.org/jira/browse/PROTON-812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301393#comment-14301393 ] ASF subversion and git services commented on PROTON-812: Commit 213e463c0589bdf08dff10c46bf2ce844c8d in qpid-proton's branch refs/heads/master from [~gsim] [ https://git-wip-us.apache.org/repos/asf?p=qpid-proton.git;h=213e463 ] PROTON-812: added exceptions to convey link detach and connection close conditions to users of synchronous utilities LinkException needs an attribute that indicates the reason for the exception. - Key: PROTON-812 URL: https://issues.apache.org/jira/browse/PROTON-812 Project: Qpid Proton Issue Type: Improvement Affects Versions: 0.8 Reporter: Jeff Ortel Assignee: Gordon Sim LinkException needs an attribute that indicates the reason for the exception so that the exception can be appropriately delt with. For example: A link exception caused when a node does not exist would be delt with differently than a link exception caused by an issue in the link chain involving dispatch router. Having constants for the error codes would be desireable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PROTON-811) [PATCH] proton-j: no way to implement idle timeout of a connection
[ https://issues.apache.org/jira/browse/PROTON-811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301326#comment-14301326 ] Rafael H. Schloming commented on PROTON-811: Hi Adrian, Thanks for reworking this. I tried applying this and ran into a few more issues. Minor nitpick, I noticed after applying the patch that git diff shows up extra whitespace in the files. We try not to have extraneous whitespace. Your editor probably has a setting that will clean that sort of thing up for you automatically. It might be worth looking into. Beyond that, I had some trouble compiling. I think maybe your java version is newer, because my compiler barfed on the diamond operator () used in the tests. I fixed that easily enough, but then there were a bunch of missing imports. I'm not sure how this worked for you unless maybe you're compiling against an older/alternate version of the code? Once I added the missing imports, I noticed a few failures in the jython test suite. FYI, there are a number of tests written in python that get run against both the proton-c and proton-j implementations. These tests help us keep things consistent between the two and give us better/shared coverage. There are a set of shims in proton-j/src/main/resources that adapt between the Java and C APIs. There's actually already test coverage for idle timeouts in the python test suite, those tests currently just skip for the Java impl because those parts of the shim raise the Skipped() exception. I attempted to enable those pieces of the shim to give some better coverage, however this showed up a difference in the APIs. The C API stores the idle timeout on the transport object, whereas you have it in the Connection object. If we could resolve this difference then it would be pretty straightforward to enable the portion of the shim that deals with idle timeouts and get added test coverage from the python test suite. [PATCH] proton-j: no way to implement idle timeout of a connection -- Key: PROTON-811 URL: https://issues.apache.org/jira/browse/PROTON-811 Project: Qpid Proton Issue Type: Bug Components: proton-j Affects Versions: 0.8 Reporter: Adrian Preston Attachments: 0001-idle_timeout.patch, 0001-proton-j-updates-for-idle-timeout-mk3.patch, 0001-proton-j-updates-for-idle-timeout.patch Proton-J does not provide access to idle timeout values and there appears to be no way to send a empty frame (as per section 2.4.5 of the AMQP 1.0 standard) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (PROTON-812) LinkException needs an attribute that indicates the reason for the exception.
[ https://issues.apache.org/jira/browse/PROTON-812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gordon Sim resolved PROTON-812. --- Resolution: Fixed Fix Version/s: 0.9 There is now a LinkDetached subclass of LinkException, which has the 'condition' the peer set on the detach as a property. LinkException needs an attribute that indicates the reason for the exception. - Key: PROTON-812 URL: https://issues.apache.org/jira/browse/PROTON-812 Project: Qpid Proton Issue Type: Improvement Affects Versions: 0.8 Reporter: Jeff Ortel Assignee: Gordon Sim Fix For: 0.9 LinkException needs an attribute that indicates the reason for the exception so that the exception can be appropriately delt with. For example: A link exception caused when a node does not exist would be delt with differently than a link exception caused by an issue in the link chain involving dispatch router. Having constants for the error codes would be desireable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [java] Message codec improvements
Rafi, further to our discussion I have posted a patch to illustrate the approach we plan to take. This should enable me to make progress until you get a chance to make further changes on the Decoder side. Regards, Rajith On Thu, Jan 29, 2015 at 3:59 PM, Rajith Muditha Attapattu rajit...@gmail.com wrote: More questions. For all the maps we return should we restrict them to String, Object or should it be Object, Object ? Technically one could use a Number (int, long) etc as a key.. Any opinion here? ;) On Thu, Jan 29, 2015 at 12:48 PM, Rafael Schloming r...@alum.mit.edu wrote: On Thu, Jan 29, 2015 at 12:28 PM, Rajith Muditha Attapattu rajit...@gmail.com wrote: Rafi could u pls answer the two questions I had in the code? 1. Your uint method only takes an int .. shouldn't it take a long? Bcos it could contain a value larger than a java int? To be honest I don't quite remember for sure, but I think it will do the two's complement and put it on the wire as a proper unsigned value. In other words I think it's just using the int type as a convenient/efficient way to pass around 4 bytes. 2. What should I use for boolean? there is no getBoolean .. or an equivalent method I think you might need to add this. I probably just omitted it. --Rafael
Re: puzzling issue with javascript bindings
That's very generous of Rafael - he spotted the wood when I was still staring at the trees :-) I was quite impressed. I think I can at least claim the dubious honour of being one of the very few people who has managed to introduce an uninitialised pointer error into JavaScript code! I don't know whether to be embarrassed or proud :-D Frase On 02/02/15 12:43, Rafael Schloming wrote: FYI, thanks to some help from Fraser, this issue should now be resolved. --Rafael On Sat, Jan 31, 2015 at 1:24 PM, Rafael Schloming r...@alum.mit.edu wrote: On Sat, Jan 31, 2015 at 12:27 PM, Fraser Adams fraser.ad...@blueyonder.co.uk wrote: Hopefully you got the build I mailed you, but as an update I've upgraded my system to the latest emscripten incoming: emcc (Emscripten GCC-like replacement + linker emulating GNU ld ) 1.29.8 clang version 3.5.0 And did a Debug build on a clean system as before and I'm still not seeing any issues like the one you described and my recv.js seems pretty happy however many times I call send.js I'm pretty stumped!! What system are you running? Is there anything quirky? Do you have any exotic compilers or an unusual path? I did get your build and it worked fine for me. I emailed you a copy of my build just in case you are curious. My versions are: emcc (Emscripten GCC-like replacement) 1.29.0 (commit fdf24b478e1b26c0362a1627aca49ef82fd53f6a) clang version 3.4 My system is just stock fedora 20 as far as I'm aware. I installed emscripten by just following the directions in the README. --Rafael Frase On 31/01/15 15:37, Rafael Schloming wrote: Any chance you could send me a copy of your proton.js so I can try on my system? --Rafael On Sat, Jan 31, 2015 at 10:32 AM, Fraser Adams fraser.ad...@blueyonder.co.uk wrote: Hi again Rafi, As I'm on a roll today. So I've just done: cd build make clean rm CMakeCache.txt cmake -DCMAKE_BUILD_TYPE=Debug .. (which gives the message: JavaScript build type is Debug) make and when I did ./recv.js and in another window ./send.js I'm not seeing any issue: ./recv.js pipe() returning an error as we do not support them Address: amqp://0.0.0.0 Subject: UK.WEATHER Content: Hello World! Address: amqp://0.0.0.0 Subject: UK.WEATHER Content: Hello World! Address: amqp://0.0.0.0 Subject: UK.WEATHER Content: Hello World! Address: amqp://0.0.0.0 Subject: UK.WEATHER Content: Hello World! Address: amqp://0.0.0.0 Subject: UK.WEATHER Content: Hello World! Address: amqp://0.0.0.0 Subject: UK.WEATHER Content: Hello World! Address: amqp://0.0.0.0 Subject: UK.WEATHER Content: Hello World! Address: amqp://0.0.0.0 Subject: UK.WEATHER Content: Hello World! Looking in node_modules/qpid-proton-messenger/lib/proton-messenger.js I've definitely got the unminified/Debug version in play (which is also borne out by the pipe() returning an error as we do not support them message from emscripten's stub pipe call). So I'm not seeing what you are seeing on my system. I'm running Linux Mint 17.1 Rebecca emcc (Emscripten GCC-like replacement + linker emulating GNU ld ) 1.28.2 clang version 3.4 node v0.10.33 I doubt the changes I've just committed would make any difference, so there's definitely something weird. I'll update my emscripten/fastcomp versions and see if that introduces this quirk. On the plus side, at least it wasn't my imagination and I really haven't seen this behaviour before :-D If it *is* an emscripten issue it'd be good to have a minimal reproducer to attach to an issue report. Frase On 31/01/15 12:17, Rafael Schloming wrote: Hi, I recently uncovered a puzzling issue with the Javascript bindings. If I fire up the simple recv.js example and then run send.js, it works fine the first time, but the second time around I get the error below. This apparently has been happening for a while (possibly a few months), however I haven't noticed because it only happens with debug builds, and only then the second time around. With a regular build everything seems to work fine. I googled around a bit for this particular error and there are a bunch of threads talking about how casting function pointers doesn't work with emscripten and you need to avoid that, but I don't believe we actually do that ever, and I see nothing to indicate that this sort of error would go away with a non Debug build. The stack trace (which you only get after rebuilding with ASSERTIONS=2) seems pretty straightforward. It is inside pn_class_new which delegates to a function pointer held in the pn_class_t struct that is passed in. I believe that function pointer lookup is what is failing, but when I put printfs in the C code to confirm this, the problem goes away. At this point I'm kind of left scratching my head. The only thing I can think of short of a compiler bug is that somehow the pn_class_t struct is becoming corrupted, but I would expect valgrind to show up any sort of memory issues like that, unless somehow it is only being triggered from the
[jira] [Commented] (PROTON-812) LinkException needs an attribute that indicates the reason for the exception.
[ https://issues.apache.org/jira/browse/PROTON-812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301530#comment-14301530 ] Gordon Sim commented on PROTON-812: --- When a peer detaches a link, it can supply a condition that is a string code that provides some context or reason for the detach. The correct behaviour on receiving an attach is always to echo an attach back, but then to follow that immediately with a detach. So, in general, in order to distinguish between two cases of the peer detaching a link there would need to be distinct conditions associated with the two cases. This change allows you to determine the condition. There is one special case, where the node the link is to be attached to doesn't exist and is not going to be created, the appropriate response is to send back an attach with no source or target (depending on direction of link) at all. That is in effect a precursor to an immediate detach with not-found as the condition. I don't know how dispatch router will handle case 2 above. For case 1, against qpidd, you will get a LinkException on create_receiver or create_sender, due to the source.target respectively being null on the attach sent back by the broker. LinkException needs an attribute that indicates the reason for the exception. - Key: PROTON-812 URL: https://issues.apache.org/jira/browse/PROTON-812 Project: Qpid Proton Issue Type: Improvement Affects Versions: 0.8 Reporter: Jeff Ortel Assignee: Gordon Sim Fix For: 0.9 LinkException needs an attribute that indicates the reason for the exception so that the exception can be appropriately delt with. For example: A link exception caused when a node does not exist would be delt with differently than a link exception caused by an issue in the link chain involving dispatch router. Having constants for the error codes would be desireable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PROTON-812) LinkException needs an attribute that indicates the reason for the exception.
[ https://issues.apache.org/jira/browse/PROTON-812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301493#comment-14301493 ] Jeff Ortel commented on PROTON-812: --- This looks great but not sure how it will disambiguate causes for create_receiver() to fail. Between: 1. node does not exist on the broker. 2. intermediate dispatch router not connected/running so the link chain could not be created. Can you explain? LinkException needs an attribute that indicates the reason for the exception. - Key: PROTON-812 URL: https://issues.apache.org/jira/browse/PROTON-812 Project: Qpid Proton Issue Type: Improvement Affects Versions: 0.8 Reporter: Jeff Ortel Assignee: Gordon Sim Fix For: 0.9 LinkException needs an attribute that indicates the reason for the exception so that the exception can be appropriately delt with. For example: A link exception caused when a node does not exist would be delt with differently than a link exception caused by an issue in the link chain involving dispatch router. Having constants for the error codes would be desireable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
