Re: 0.5 RC2
On Wed, Aug 14, 2013 at 04:04:33PM -0400, Rafael Schloming wrote: > There's been a bunch of fixes now from RC1 testing, so I figured it's about > time to put out an RC2. You can find it here, please check it out: Build a package for Fedora, installed and tested it and all looks good. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpSQ_10bqf_p.pgp Description: PGP signature
Re: [VOTE]: Release Proton 0.5 RC3 as 0.5 final
On Thu, Aug 22, 2013 at 12:26:54PM -0400, Rafael Schloming wrote: > Hi Everyone, > > After a bunch more tests/fixes I think we're ready to go for a vote now. > I've posted 0.5 RC3 in the usual places: > > Source is here: > > http://people.apache.org/~rhs/qpid-proton-0.5rc3/ > > Java binaries are here: > > https://repository.apache.org/content/repositories/orgapacheqpid-109/ > > I've attached the svn change log for everything since RC2. Please > peruse/test and register your vote: > [X] Yes, release 0.5 RC3 as 0.5 final [ ] No, 0.5 RC3 has the following issues... -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpmvriR5vY3d.pgp Description: PGP signature
Re: question about proton error philosophy
On Mon, Sep 16, 2013 at 12:33:29PM -0400, Hiram Chirino wrote: > Do other APIs reset the errno? I could have sworn they didn't. "Successful system calls and library functions never reset errno to 0, so this variable may have a nonzero value as a consequence of an error from a previous call. Furthermore, SUSv3 permits a successful function call to set errno to a nonzero value (although few functions do this). Therefore, when checking for an error, we should always first check if the function return value indicates an error, and only then examine errno to determine the cause of the error." - The Linux Programming Interface, pg 49. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpEFRS5MxPIW.pgp Description: PGP signature
Non-standard install directories for dynamic languages
I have a JIRA in my queue [1] that relates to a few things regarding installation directories. I spoke with Gordon briefly this morning regarding this issue, which (correct me if I'm wrong) boils down to how the code is installed when a non-standard location is targeted. IOW, if you configure your environment with: cmake -DCMAKE_INSTALL_PREFIX:PATH=/some/path . the dynamic languages (Python, Ruby, Perl) don't use this path. That is because the CMake environment asks each language for the absolute path to where their extensions are to be installed and does not prefix that. This last part is due to a few things, not the least of which is that CMAKE_INSTALL_PREFIX is never empty (it's default is "/usr/local") unless explicitly blanked on the commandline. So we shouldn't always prepend it to the dynamic languages. Possible solutions to this problem are: 1) provide a command line switch when setting up the build environment that toggles prepending CMAKE_INSTALL_PREFIX to the language binding paths, or 2) allow each language binding path to be specified on the command line, overriding what the languages themselves return, or 3) do nothing special. [1] http://issues.apache.org/jira/browse/PROTON-342 -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpeQw1Uju3zr.pgp Description: PGP signature
Re: Simple guide to use it
On Wed, Oct 16, 2013 at 04:39:34PM +0800, Azmi abdul rahman wrote: > Hi, > > i have build the protoon 0.5. and i have created send.exe and recv.exe. > > but how do i link it to python ? > > from the sample codes, i can see that i need to import proton. but there's > no setup.py for me to install in the 1st place. > > Pls pls help me. I'll be happy to help you. Currently the way to install the Proton Python bindings is to use the CMake build environment and do "make install". That will get the proper site package directory from your installed Python to know where to put the libraries. There are existing Python examples for send and recv that are in the examples/messenger/py directory. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpvlNlSZtD5x.pgp Description: PGP signature
Re: please review: Ruby Messenger Doc
On Tue, Nov 19, 2013 at 09:12:53AM -0500, Michael Goulish wrote: > I wanted to show you this in lovely HTML, but all my attempts thus far > (outside of the usual Ruby framework) have created only travesties of > the proper format: diseased and horrible things, lurching through the > stygian depths of my browser like ... like... Can you ping me in IRC? I'd like to see what issues you're having with generating the HTML documentation and see if we can't solve this. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpq6LFFBM31c.pgp Description: PGP signature
Dynamic language install and CMAKE_INSTALL_PREFIX
I have a proposed changeset [1] to meet this request (they were pushed, but after a discussion this morning I've reverted the changes) and want to get feedback on it before changing the default behavior for Proton. Currently no language binding honors the CMAKE_INSTALL_PREFIX path. So, if your current Ruby keeps its vendor extensions in /usr/share/ruby/vendor_ruby, then if you set CMAKE_INSTALL_PREFIX to /var/tmp/foo, the bindings won't isntall there. With the changes I'm proposing, the build environment asks the language for 1) the install prefix for directories and then 2) the directory for vendor extensions. It then does a string substition, replacing the install prefix in the directory with the CMAKE_INSTALL_PREFIX value provided. For power users, there's a way to override this by specifying, for each language, a different install prefix only for that language. So, for example, you could pass in RUBY_INSTALL_PREFIX to have the build environment use that instead of CMAKE_INSTALL_PREFIX. I'd like to get more input from others who use the dynamic languages as to whether this approach finds a good middle ground for what everybody expects when installing and using Proton. [1] http://reviews.apache.org/r/16004/ -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpWaM6ynA2BM.pgp Description: PGP signature
Re: Dynamic language install and CMAKE_INSTALL_PREFIX
On Thu, Dec 05, 2013 at 01:57:22PM -0500, Rafael Schloming wrote:
> So overall I'd say this change should have some kind of switch to control
> whether the QUERIED_LOCATION is used directly, and I'd argue that for the
> CUSTOM_LOCATION we should just pass through directly what the user supplies
> and not attempt to merge it with the queried value. As for the
> CONSTRUCTED_LOCATION, it's worth noting that we don't necessarily need to
> compute that either, we could just pick an arbitrary location, e.g.
> ${CMAKE_INSTALL_PREFIX}/lib64/proton-bindings or some such thing.
I find the simplicity in this scenario to be very attractive. It also
avoids situations like what I saw with the PHP ini directory location,
to use project-defined directories for defaults.
What I've seen, for each of the language bindings, is a need to know:
1. the directory to install platform-independent modules,
2. the directory to install platform-specific modules, and
3. the directory to install configuration (PHP only)
So perhaps ${LANG}_LIBDIR, ${LANG}_ARCHDIR and ${LANG}_CONFDIR? In an
RPM specfile we could define each one using the provide language's macro
and it would be a fairly easy integration point. And if you don't define
them then they work as you suggest above.
> Wherever we end up, we should also probably abstract the behaviour into a
> macro so that the behaviour is easier to keep consistent between bindings,
> and so that new bindings pick up the same behaviour automatically.
+1
--
Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
Delivering value year after year.
Red Hat ranks #1 in value among software vendors.
http://www.redhat.com/promo/vendor/
pgpmrakI63iJX.pgp
Description: PGP signature
Re: Dynamic language install and CMAKE_INSTALL_PREFIX
On Thu, Dec 05, 2013 at 02:41:53PM -0500, Rafael Schloming wrote:
> On Thu, Dec 5, 2013 at 2:29 PM, Darryl L. Pierce wrote:
> > On Thu, Dec 05, 2013 at 01:57:22PM -0500, Rafael Schloming wrote:
> >
> > > So overall I'd say this change should have some kind of switch to control
> > > whether the QUERIED_LOCATION is used directly, and I'd argue that for the
> > > CUSTOM_LOCATION we should just pass through directly what the user
> > supplies
> > > and not attempt to merge it with the queried value. As for the
> > > CONSTRUCTED_LOCATION, it's worth noting that we don't necessarily need to
> > > compute that either, we could just pick an arbitrary location, e.g.
> > > ${CMAKE_INSTALL_PREFIX}/lib64/proton-bindings or some such thing.
> >
> > I find the simplicity in this scenario to be very attractive. It also
> > avoids situations like what I saw with the PHP ini directory location,
> > to use project-defined directories for defaults.
> >
> > What I've seen, for each of the language bindings, is a need to know:
> >
> > 1. the directory to install platform-independent modules,
> > 2. the directory to install platform-specific modules, and
> > 3. the directory to install configuration (PHP only)
> >
> > So perhaps ${LANG}_LIBDIR, ${LANG}_ARCHDIR and ${LANG}_CONFDIR? In an
> > RPM specfile we could define each one using the provide language's macro
> > and it would be a fairly easy integration point. And if you don't define
> > them then they work as you suggest above.
>
> Sounds good to me. We could also use one for docs. The python bindings have
> documentation, and hopefully the other bindings will eventually need
> somewhere for docs as well.
For anything outside of the inline documentation, I would think we'd use
omething like $CMAKE_INSTALL_PREFIX/share/doc/proton/$LANG for an install
location it's not defined currently).
--
Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
Delivering value year after year.
Red Hat ranks #1 in value among software vendors.
http://www.redhat.com/promo/vendor/
pgpi6PNcijqoJ.pgp
Description: PGP signature
Re: improving cross language maintainability
On Thu, Dec 19, 2013 at 08:49:59AM -0500, Rafael Schloming wrote: > I would love to hear thoughts and/or alternative ideas on how to improve > things. I would like to start addressing this early in the 0.7 development > cycle. In a similar way, I'm trying to keep our Ruby and Perl bindings in parity as best I can with what's going on in the C and Python code. Can we use JIRA to create umbrella tasks for when new features are added, with subtasks that are binding specific? Or if there's a change to the C code that would require a change in the bindings, have the C code be the top JIRA and the language bindings be subtasks to that? That way I wouldn't need to look through commits to see what's changed in C to know what should be added to the other languages. Also, could we add a component for each of the language bindings? It doesn't feel right to add a JIRA for something in Ruby that's at the Ruby level but have its component be "proton-c". -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgp5KU9SF4aFs.pgp Description: PGP signature
Re: improving cross language maintainability
On Thu, Dec 19, 2013 at 04:07:38PM -0500, Rafael Schloming wrote: > On Thu, Dec 19, 2013 at 9:01 AM, Darryl L. Pierce wrote: > > On Thu, Dec 19, 2013 at 08:49:59AM -0500, Rafael Schloming wrote: > > > > > I would love to hear thoughts and/or alternative ideas on how to improve > > > things. I would like to start addressing this early in the 0.7 > > development > > > cycle. > > > > In a similar way, I'm trying to keep our Ruby and Perl bindings in > > parity as best I can with what's going on in the C and Python code. Can > > we use JIRA to create umbrella tasks for when new features are added, > > with subtasks that are binding specific? Or if there's a change to the C > > code that would require a change in the bindings, have the C code be the > > top JIRA and the language bindings be subtasks to that? That way I > > wouldn't need to look through commits to see what's changed in C to know > > what should be added to the other languages. > > > > Also, could we add a component for each of the language bindings? It > > doesn't feel right to add a JIRA for something in Ruby that's at the > > Ruby level but have its component be "proton-c". > > > > It certainly makes sense to make it as easy as possible to track what is > going on, and I can see how that would help keep bindings up to date where > there is interest and resources to do so. However we do this though, I > don't want to just brainlessly duplicate each C jira across every binding > (not that you're necessarily suggesting this). > > The problem I have with that approach is that there isn't equivalent > interest/resources associated with each binding, so e.g. if we were to make > every JIRA a full umbrella that depends on sub tasks for each binding we > would continually accumulate php jiras that never end up getting closed off > because we don't keep php as up to date as the other bindings, and this in > turn would cause the umbrella JIRAs to never get closed off. Jira is really > a task oriented tool, and I think JIRAs should really only be created when > there is intention/interest to actually complete the task they represent, > otherwise they usually end up being noise/clutter that will eventually be > irrelevant and out of date. I'd suggest that perhaps a more document > oriented description of those features for which we are trying to keep > parity would possibly be helpful. Definitely. Maybe a wiki page that lists the set of feeatures being done in C can be used as a backlog for the other language bindingss. Not necessarily a full blown, prioritized scrum backlog. But minimally a central list of what's done in C/Python that can be used as a guide? > All that said, I'm certainly sure we can improve our usage of JIRA, and > I've gone ahead and added ruby-binding, python-binding, perl-binding, and > php-binding components as you suggest. Thank you! :D -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpdNTqveVHxm.pgp Description: PGP signature
Re: [VOTE]: Release Proton 0.6 RC3 as 0.6 final
On Thu, Jan 02, 2014 at 02:17:12PM -0500, Rafael Schloming wrote: > Hi Everyone, > > It looks like there haven't been any major issues reported so far with 0.6 > RC3, so I guess it's about time to call for a formal vote. > > Source is here: > > http://people.apache.org/~rhs/qpid-proton-0.6rc3/ > > Java binaries are here: > > https://repository.apache.org/content/repositories/orgapacheqpid-003/ > > Please peruse/test and register your vote: > > [X] Yes, release 0.6 RC3 as 0.6 final > [ ] No, 0.6 RC3 has the following issues... -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpisXgmzf_Lr.pgp Description: PGP signature
Getting -9 when setting blocking to 0 in Perl
Since, in Perl, we don't have a true/false value, I try to turn off blocking in qpid::perl::Messenger with: my $msgr = qpid::perl::Messenger->new(); $msgr->set_blocking(0); // I just added this as a passthrough to pn_messenger_set_blocking If I pass in 0, "0" or undef to signify it's non-blocking, I get a -9 error. If I pass in 1 or some other value, things work correctly. Any ideas? -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpv7G0STaMKY.pgp Description: PGP signature
Re: Getting -9 when setting blocking to 0 in Perl
On Thu, Jan 09, 2014 at 10:19:36AM -0500, Rafael Schloming wrote:
> Well, proton/error.h defines -9 to be PN_INPROGRESS, however given that
> this is the source code of pn_messenger_set_blocking:
>
> int pn_messenger_set_blocking(pn_messenger_t *messenger, bool blocking)
> {
> messenger->blocking = blocking;
> return 0;
> }
>
> I don't see how it's possible for it to return anything other than 0.
Sorry, I meant that I'm getting a -9 error condition on calling
pn_messenger_receive when blocking is set to 0.
I added to perl.i a conversion for the bool type so that it Perl->C
would use true/false and for C->Perl would use 1/undef. With that done
the Perl bindings stopped getting a -9 on calling receive.
I'll post it on Review Board once I get it working (I'm writing a Perl
version of {send,recv}_async to test this new feature).
--
Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
Delivering value year after year.
Red Hat ranks #1 in value among software vendors.
http://www.redhat.com/promo/vendor/
pgpq9KkU0RufP.pgp
Description: PGP signature
Re: Getting -9 when setting blocking to 0 in Perl
On Thu, Jan 09, 2014 at 03:19:52PM -0500, Rafael Schloming wrote: > In the case of the C API getting a PN_INPROGRESS error (-9) from recv is > expected behaviour if you're in non blocking mode. That just means that > there is blocking work that was deferred. Okay, that explains why it didn't start working until bool was properly translated to/from Perl. Now, another issue that's cropped up with the Perl work is that, if pn_messenger_work is called with -1 then Perl never gets a message unless I explicitly kill the sender (I'm using the Python send_async script to interact with the recv_async.pl Perl script). If I set work to timeout then it times out but only gets the late (the "die") message. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgp9nE6FSg7lC.pgp Description: PGP signature
Re: Getting -9 when setting blocking to 0 in Perl
On Thu, Jan 09, 2014 at 04:06:38PM -0500, Darryl L. Pierce wrote: > On Thu, Jan 09, 2014 at 03:19:52PM -0500, Rafael Schloming wrote: > > In the case of the C API getting a PN_INPROGRESS error (-9) from recv is > > expected behaviour if you're in non blocking mode. That just means that > > there is blocking work that was deferred. > > Okay, that explains why it didn't start working until bool was properly > translated to/from Perl. > > Now, another issue that's cropped up with the Perl work is that, if > pn_messenger_work is called with -1 then Perl never gets a message > unless I explicitly kill the sender (I'm using the Python send_async > script to interact with the recv_async.pl Perl script). If I set work to > timeout then it times out but only gets the late (the "die") message. This morning Ted asked whether Perl uses pthreads or not. The Perl binary links against libpthread.so.0 on Linux, so I assume it's using them. I've tried spawning the actual asynchronous process on its own separate thread and still no joy: the call to work(-1) never returns even when messages are prending. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpv9bWQfR1eG.pgp Description: PGP signature
Re: Qpid Proton messenger on 32 bit linux
On Mon, Jan 27, 2014 at 12:43:49PM -0800, serega wrote: > Does the messenger work on 32 bit linux? I build the 32 bit messenger by > setting the variable > export CFLAGS=-m32 It most definitely works on 32-bit systems when built on such; i.e., I have 32-bit test systems where I've built/run the code and it's worked fine. What I haven't done is build 32-bit on a 64-bit platform and then run on 64-bit as well. Can you provide more details? For example, capture the output of: LD_DEBUG=all [your executable cmdline] and pastebin that somewhere so we can see what's going on with your environment? -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgp6XlbNhVEFO.pgp Description: PGP signature
Re: error when building proton
On Wed, Feb 05, 2014 at 12:08:48PM -0500, Justin Ross wrote: > Watch out. On Fedora at least, there are two uuid packages, > uuid[-devel] and and libuuid[-devel]. Perhaps the same confusing > situation exists on Ubuntu. On Debian the proper package is uuid-dev. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpxfKciddcTw.pgp Description: PGP signature
Re: [jira] [Commented] (PROTON-524) [proton-c] C++ build fails on Centos6
On Wed, Feb 26, 2014 at 08:15:20PM +, ASF subversion and git services (JIRA) wrote: > > [ > https://issues.apache.org/jira/browse/PROTON-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13913442#comment-13913442 > ] > > ASF subversion and git services commented on PROTON-524: > > > Commit 1572226 from [~kgiusti] in branch 'proton/trunk' > [ https://svn.apache.org/r1572226 ] > > PROTON-524: use old-style function pointers for compatibility with older SWIG > tools Hrm, what's different between CentOS6 and RHEL6 in this regard? I ask since Proton builds on EPEL6 as is. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgphr7VvtHEe2.pgp Description: PGP signature
Selectable readable/writable APIs
The names are very confusing to me. When I first looked at them they sounded like boolean methods regaring the readable/writable state of the underlying descriptor. But from what I've gathered they actually are methods to _perform_ reading and writing, respectively. Can we change those names to something more like a verb or an action phrase than an adjective? Maybe "receive_bytes" and "send_bytes" or "read" and "flush"? -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpRRN4cZch0X.pgp Description: PGP signature
Can only access ~24 selectables before blowing up...
In working with passive mode and selectables, I've hit a strange reproducible problem. I have a simple add-on to Messenger that lets me have it push incoming messages to an EventMachine channel [1]. If I send messages to the echo example app (it's not echoing yet), it will consistently receive around 24 messages before dying. I've tried putting delays in between the sends. I've sent the messages in batches (tried 100, tried sending in units of 10 and 5). Each time it blows up around the 24th message. The underlying error the comes up is always the same: ---8<[snip]--- *** I received: This is message 22 ??? read_array=[9, 7] write_array=[9, 7] +++ fd=9 : rarray=[] +++ fd=7 : rarray=[#] +++ fd=9 : warray=[] +++ fd=7 : warray=[#] ### capacity=16384 pending=0 ??? read_array=[9, 7, 12] write_array=[9, 7, 12] +++ fd=9 : rarray=[] +++ fd=7 : rarray=[#] +++ fd=12 : rarray=[#, #] +++ fd=9 : warray=[] +++ fd=7 : warray=[#] +++ fd=12 : warray=[#, #] ### capacity=16384 pending=0 ??? read_array=[9, 7, 12] write_array=[9, 7, 12] +++ fd=9 : rarray=[] +++ fd=7 : rarray=[#] +++ fd=12 : rarray=[#, #] +++ fd=9 : warray=[] +++ fd=7 : warray=[#] +++ fd=12 : warray=[#, #] recv: Bad file descriptor [0x7f88b80700c0]:ERROR[-2] AMQP header mismatch: '' (connection aborted) CONNECTION ERROR connection aborted (remote) !!! deleting fd=12 ??? read_array=[9, 7] write_array=[9, 7] +++ fd=9 : rarray=[] /home/mcpierce/Programming/eventful-qpid-proton/lib/eventful-qpid-proton/event_machine.rb:80:in `initialize': Bad file descriptor (Errno::EBADF) from /home/mcpierce/Programming/eventful-qpid-proton/lib/eventful-qpid-proton/event_machine.rb:80:in `new' from /home/mcpierce/Programming/eventful-qpid-proton/lib/eventful-qpid-proton/event_machine.rb:80:in `block (3 levels) in start_event_monitor' from /home/mcpierce/Programming/eventful-qpid-proton/lib/eventful-qpid-proton/event_machine.rb:78:in `each' from /home/mcpierce/Programming/eventful-qpid-proton/lib/eventful-qpid-proton/event_machine.rb:78:in `block (2 levels) in start_event_monitor' from /home/mcpierce/Programming/eventful-qpid-proton/lib/eventful-qpid-proton/event_machine.rb:39:in `loop' from /home/mcpierce/Programming/eventful-qpid-proton/lib/eventful-qpid-proton/event_machine.rb:39:in `block in start_event_monitor' ruby-mri: /home/mcpierce/Programming/Proton/proton-c/src/object/object.c:99: pn_free: Assertion `pn_refcount(object) == 1' failed. Aborted ---8<[snip]--- At the point where the error occurs, Ruby is attempting to wrap the fileno returned by the Selectable in an IO object. In this case it is almost always the number 9 that's coming up. Anybody see anything related to this before? [1] https://github.com/mcpierce/eventful-qpid-proton -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpBhKnix2SGL.pgp Description: PGP signature
Re: Can only access ~24 selectables before blowing up...
On Thu, Apr 10, 2014 at 06:41:50AM -0400, Rafael Schloming wrote: > Based on the assertion it could be a bug in the ruby binding around > selectables, however I don't see any of that code on trunk or in the git > repo you pointed to. Are you using a patched version of proton? Correct, it's on a branch in my github repo [1]. I'll dig into that and see if anything's up there. [1] http://github.com/mcpierce/Proton/tree/PROTON-531-enable-passive-mode-in-ruby -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgp3C62sJg1aa.pgp Description: PGP signature
Messenger.receive(1) causing sluggish behavior
Using the Ruby Chatty example in my examples repo [1] I see painful slow runtime performance. To run the example, first launch the Guru process: $ ruby guru.rb Then, in another terminal, start a single Student instance: $ ruby student What you'll see is the student starts off with a 1 or 2 messages, but then immediately gets sluggish. If you start another instance of Student things degrade very quickly. If, however, in student.rb we change: $msgr.receive(1) to: $msgr.receive(-1) then there is no apparently sluggishness until several thousand messages have been sent, and multiple instances of Student are able to run without slowing down. [1] https://github.com/mcpierce/qpid-ruby-examples/tree/Rafi-slow-server-performance -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgp3TJm4l284j.pgp Description: PGP signature
Re: Messenger.receive(1) causing sluggish behavior
On Thu, May 01, 2014 at 09:29:56AM -0400, Rafael Schloming wrote: > (and many others) you don't really even need more than one address. If you > modify your example to just use "~/replies", it will work properly > regardless of what value you pass to recv. I did this, set the Messenger back to recv(1) and the speed increased. Now it starts around 2300 msg/sec and drops off from there. Though I would wager part of that drop off in speed isn't due to communications issues but because the student does random guesses and avoids resending previous guesses. When I comment out the checks for duplicate guesses it stays at around 2300 msg/sec. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpl_WVzMpPWP.pgp Description: PGP signature
Re: Messenger.receive(1) causing sluggish behavior
On Thu, May 01, 2014 at 04:02:51PM -0400, Darryl L. Pierce wrote: > On Thu, May 01, 2014 at 09:29:56AM -0400, Rafael Schloming wrote: > > > (and many others) you don't really even need more than one address. If you > > modify your example to just use "~/replies", it will work properly > > regardless of what value you pass to recv. > > I did this, set the Messenger back to recv(1) and the speed increased. > Now it starts around 2300 msg/sec and drops off from there. Though I > would wager part of that drop off in speed isn't due to communications > issues but because the student does random guesses and avoids resending > previous guesses. When I comment out the checks for duplicate guesses it > stays at around 2300 msg/sec. Another interesting point: memory usage has dropped off significantly, which I guess is expected given that there aren't a bunch of links being used now. Before I would see guru get to about 20% of memory used while running, this after throwing many different instances of student at it. Now it sits consistently at 0.1% memory usage. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpqFKfCTOl5W.pgp Description: PGP signature
Re: map values in message body
On Thu, May 15, 2014 at 01:04:13PM -0400, Ernest Allen wrote:
> Does proton messenger support sending complex maps as the message body? That
> is, maps whose values are other maps or lists and not just strings?
>
> When I send a message and the body is encoded as qpid::proton::MAP, the map
> values appear to have been converted to strings.
>
> For example (in perl)
>
> In the sender
> $msg->set_body({"foo" => 2, "bar" => [3]}, qpid::proton::MAP);
>
> In the receiver when I dump the body, I'm getting
> $VAR1 = \{
> 'bar' => 'ARRAY(0xa2aad0)',
> 'foo' => '2'
> };
>
> It appears that the array has been converted to the string 'ARRAY(0x...)'.
> And the 2 is now a string '2'.
> I don't think this is just the way the receiver is dumping the message body
> as I'm not able to convert the results to the expected type or variable.
>
> When I send a perl HASH instead of an array as one of the values, it is
> coming through as the string'HASH(0x)'.
This should now be fixed on trunk. I've updated the recv.pl example to
display the full body of received messages to verify that all values are
being passed.
Now, that being said, there's a caveat with Perl. Since there's no way
for Perl to determine what any scalar value is (beyond the class name
for blessed objects) everything has to be inserted into the data object
as a string. So, for example, if you put the following into the body of
a message in Perl:
$msg->set_body({"fibonacci" => [1, 1, 2, 3, 5, 8, 13]});
then the receive will get:
{"fibonacci" => ["1", "1", "2", "3", "5", "8", "13"]}
I'll continue to look for some way to improve this, but for now at least
the structure of complex hashes and arrays are preserved in Perl.
--
Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
Delivering value year after year.
Red Hat ranks #1 in value among software vendors.
http://www.redhat.com/promo/vendor/
pgpalvNfYs_43.pgp
Description: PGP signature
qpid-proton-java RPM dependencies
Here is what the original packager had put in as dependencies: BuildRequires: maven-local BuildRequires: mvn(junit:junit) BuildRequires: mvn(org.apache.maven.doxia:doxia-module-markdown) BuildRequires: mvn(org.apache.maven.plugins:maven-site-plugin) BuildRequires: mvn(org.bouncycastle:bcpkix-jdk15on) BuildRequires: mvn(org.apache.geronimo.specs:geronimo-jms_1.1_spec) BuildRequires: mvn(org.apache.geronimo.specs:specs) BuildRequires: mvn(org.fusesource.hawtbuf:hawtbuf) # TODO hawtdispatch-version>=1.18 BuildRequires: mvn(org.fusesource.hawtdispatch:hawtdispatch-transport) BuildRequires: mvn(org.mockito:mockito-core) -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpcHgR2pLp6x.pgp Description: PGP signature
Launchpad PPA : qpid-proton
https://launchpad.net/~mcpierce/+archive/qpid-proton Currently it has the qpid-proton 0.7-1.1 package in it. I'm going to package up Dispatch 0.2 for it next. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpBVnKjOPKra.pgp Description: PGP signature
Re: `make dist` for proton-c
On Thu, Aug 28, 2014 at 03:32:06PM +0100, chris snow wrote: > The Apache Stratos [1] project is currently investigating using Proton > to connect to an ActiveMQ AMQP 1.0 broker using Python. > > The Stratos project is a PaaS that manages VM instances. Each VM > instance may have different Linux operating systems (e.g. Debian or > Redhat deriavatives). The VM instances are provisioned at runtime > using puppet to install the necessary software such as client software > for communicating instance status back to the Stratos AMQP 1.0 server. > To minimise the provisioning time, it would be ideal if we could > install binaries such as proton rather than having to compile it from > source. > > Are there any plans to provide distribution binaries for proton-c? If > not, do you recommend an approach we can follow to build the binaries > once on a development machine so those binaries can be copied onto the > Stratos puppet server (e.g. make dist)? Hi, Chris. Can you give a little more detail on what specifically you are looking for? Currently we have binary packages for Fedora, RHEL (via EPEL), Ubuntu and Debian Linux distros supporting 32- and 64-bit platforms. For Fedora and RHEL these are a part of the standard repositores. For Debian and Ubuntu we have packages that are in testing [1] and also in our PPA [2], respectively. [1] http://mentors.debian.net/package/qpid-proton [2] https://launchpad.net/~qpid/+archive/ubuntu/released -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpByZjnzgss8.pgp Description: PGP signature
Re: `make dist` for proton-c
On Thu, Aug 28, 2014 at 07:11:12PM +0100, chris snow wrote: > Hi Darryl, > > Great stuff - will check those links out. Thank you. > > Are those links referenced from the proton web page? AFAIK they are not, though it's probably something that could be made available. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpWrdfUEBAYh.pgp Description: PGP signature
Re: [VOTE]: migrate the proton repo to use git
On Thu, Oct 30, 2014 at 06:59:54AM -0400, Rafael Schloming wrote: > Hi Everyone, > > I'm planning on updating the release script for 0.9 to automate the last > few details of the release process and to do proper branching. Given that > the release script must make use of the canonical repo to work properly, it > occurs to me that if we're going to switch over to git then now would be a > good time so that I only need to update the release script once. I'd > therefore like to call for a formal vote to approve the switch. > > [ X ] Yes, migrate the proton repo over to git. > [ ] No, keep it in svn. A little late to the vote. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpY2KOJuSJIn.pgp Description: PGP signature
Re: There really is a problem with new qpid-proton repo, the old trunk and new master diverge in 2012
On Thu, Nov 06, 2014 at 03:30:22PM -0500, Andrew Stitcher wrote: > On Thu, 2014-11-06 at 15:03 -0500, Andrew Stitcher wrote: > > On Thu, 2014-11-06 at 14:54 -0500, Andrew Stitcher wrote: > > > On Thu, 2014-11-06 at 11:41 -0500, Rafael Schloming wrote: > > > > Hi Everyone, > > > > > > > > The git migration is complete. The svn repo is now read only. You can > > > > find > > > > instructions for accessing the git repo here: > > > > > > > > https://git-wip-us.apache.org/ > > > > > > There appears to be a problem already with the migration! > > > > Ah, no actual problem here! The dev branch is now called > > "master" (unless we want to change it back for nostalgia's sake!) > > To re contradict myself! There really is a problem here. > > The migration has produced a repo that is different (has a different > branching structure) from the previous mirrors. This appears to be due > to differences in committer naming between the previous mirror repo and > the new repo. The first name difference was in a 2012 commit: > 31a106a324f006cb4c817c655b4b30bef4a65465 > > But there appear to be at least 2 committers: Darryl Pierce and Cliff > Jansen that have names that are different in the 2 branches. > > Since this will surely cause pain for the automated github mirroring and > existing repos which many people have we need to sort this out soonest. Is this due to my having followed Apache's steps and updating my commit email address to be "mcpie...@apache.org"? -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpI8NqSNxwpY.pgp Description: PGP signature
Re: There really is a problem with new qpid-proton repo, the old trunk and new master diverge in 2012
On Mon, Nov 10, 2014 at 08:08:12AM -0500, Darryl L. Pierce wrote: > On Thu, Nov 06, 2014 at 03:30:22PM -0500, Andrew Stitcher wrote: > > On Thu, 2014-11-06 at 15:03 -0500, Andrew Stitcher wrote: > > > On Thu, 2014-11-06 at 14:54 -0500, Andrew Stitcher wrote: > > > > On Thu, 2014-11-06 at 11:41 -0500, Rafael Schloming wrote: > > > > > Hi Everyone, > > > > > > > > > > The git migration is complete. The svn repo is now read only. You can > > > > > find > > > > > instructions for accessing the git repo here: > > > > > > > > > > https://git-wip-us.apache.org/ > > > > > > > > There appears to be a problem already with the migration! > > > > > > Ah, no actual problem here! The dev branch is now called > > > "master" (unless we want to change it back for nostalgia's sake!) > > > > To re contradict myself! There really is a problem here. > > > > The migration has produced a repo that is different (has a different > > branching structure) from the previous mirrors. This appears to be due > > to differences in committer naming between the previous mirror repo and > > the new repo. The first name difference was in a 2012 commit: > > 31a106a324f006cb4c817c655b4b30bef4a65465 > > > > But there appear to be at least 2 committers: Darryl Pierce and Cliff > > Jansen that have names that are different in the 2 branches. > > > > Since this will surely cause pain for the automated github mirroring and > > existing repos which many people have we need to sort this out soonest. > > Is this due to my having followed Apache's steps and updating my commit > email address to be "mcpie...@apache.org"? Someone emailed me offlist about this and confirmed that was the case. So, for me, I've removed the entry from the config for Proton. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpNJz5tcUL1P.pgp Description: PGP signature
Ruby and blocking calls
There was a JIRA [1] opened recently concerning blocking calls in the Ruby bindings, and asking for a means to release the global interpreter lock for such calls. The first issue to deal with is to provide a solution that would work on both Ruby 1.8 and Ruby 1.9/2.0. In the latter case we have the rb_thread_blocking_region() [2] Ruby API that can be used to mark a method as blocking in order to release the GIL for a period of time. However, there is no equivalent method in Ruby 1.8. But what there is available in Ruby 1.8+ is the ability to wait on a file descriptor. But this ability is also already exposed in the Ruby bindings themselves using passive mode [3] to avoid the call to Messenger.receive blocking the interpreter. It strikes me that the solution should leverage the passive mode capabilities of the underlying C code. After all, that's it's raison d'etre, to make it possible in bindings to write non-blocking code code. Thoughts? Ideas? [1] http://issues.apache.org/jira/browse/PROTON-752 [2] http://tinyurl.com/naxtdfw [3] http://issues.apache.org/jira/browse/PROTON-531 -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpQIMKV969Iu.pgp Description: PGP signature
Re: Proton Python install shenanigans
On Sun, Nov 30, 2014 at 04:31:20PM +, Fraser Adams wrote: > I've been tearing my hair out a bit trying to figure why I can't see the > relevant Python modules. > > I build Proton on a clean Linux Mint 17 system (which is based on Ubuntu > 14.04) > > make all builds successfully and sudo make install installs the following: > > fadams@zappa /usr/local/lib/proton/bindings/python $ ls -al > total 2120 > drwxr-xr-x 2 root root4096 Nov 30 12:28 . > drwxr-xr-x 5 root root4096 Nov 30 12:20 .. > -rw-r--r-- 1 root root 89278 Nov 30 11:29 cproton.py > -rw-r--r-- 1 root root 103293 Nov 30 12:28 cproton.pyc > -rw-r--r-- 1 root root 103293 Nov 30 12:28 cproton.pyo > -rw-r--r-- 1 root root 1445471 Nov 30 12:20 _cproton.so > -rw-r--r-- 1 root root 109162 Nov 30 11:20 __init__.py > -rw-r--r-- 1 root root 151464 Nov 30 12:28 __init__.pyc > -rw-r--r-- 1 root root 151257 Nov 30 12:28 __init__.pyo > > > I set PYTHONPATH=/usr/local/lib/proton/bindings/python > > but when I try > ./msgr-recv.py > > I get > > Traceback (most recent call last): > File "./msgr-recv.py", line 23, in > from proton import * > ImportError: No module named proton > > > Now I'm thinking that make install hasn't pulled all of the necessary files > into /usr/local/lib/proton/bindings/python the reason I'm thinking that is > when I tried running > source config.sh > > I noticed that the PYTHONPATH it created looked like > > PYTHONPATH=/home/fadams/qpid/git/qpid-proton/tests/python:/home/fadams/qpid/git/qpid-proton/proton-c/bindings/python:/home/fadams/qpid/git/qpid-proton/build/proton-c/bindings/python > > and this *does* appear to work, the significant bit seems to be the > /home/fadams/qpid/git/qpid-proton/proton-c/bindings/python bit, because when > I tried: > > PYTHONPATH=/home/fadams/qpid/git/qpid-proton/proton-c/bindings/python:/usr/local/lib/proton/bindings/python > > that worked too. > > So I'm thinking that the directory proton and its contents in > qpid-proton/proton-c/bindings/python need to be copied to > > /usr/local/lib/proton/bindings/python > > as part of the make install process too. > > Does that seem about right? The code was migrated to a directory, so should be installing to: /path/to/python/site-libs/proton ? -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpGzPDd5AjFD.pgp Description: PGP signature
Fixing the merge commit in the git repo?
Would it be possible for someone (Rafi?) to fix the merge commits in the Git repo? I'm working on some stuff and, when I tried to do a rebase I accidentally went a few commits further back and git wanted to then rebase 65 commits... On our individual ends, once it's done, we should be able to just do: $ git pull --rebase and have our individual repos fixed as a result. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpsO2YbjQg7u.pgp Description: PGP signature
Re: Fixing the merge commit in the git repo?
On Thu, Dec 11, 2014 at 02:35:05PM -0500, Rafael Schloming wrote: > Can you provide a bit more detail? I'm not the most expert git user, so I'm > not sure exactly what you're asking for, much less how to do it. In my case, my usual workflow is to do small changes and commit them locally. If a change goes with a commit a few back I'll do: $ git commit -a --fixup=[early commit's hash] Then, when I'm ready to commit I'll do: $ git rebase -i HEAD~[# of commits to process] --autosquash and git will put the commits in the right order so I can squash them down. Normally I just do "HEAD~25" rather than counting the commits since we usually had a nice, linear stream of commits in our old repo that was cloned from Subversion. But when I did it this time that the commits overlapped with this merge commit: ===[snip]=== commit e8029597b825e49ee5ab2b7c8bd5d042e57321bb Merge: db437cf a5cb27e Author: Rafael Schloming Date: Fri Nov 28 08:43:03 2014 -0500 Merge branch 'javascript' ===[snip]=== This made git them try to rebase all of those changes, which numbered up to about 60 or so commits that are out of sequence. So I had to abort that rebase and count the commits instead and do it again. When merging branches, etc. we can avoid this if, before merging branch A into master we go to branch A and rebase it on master. That way we avoid merge commits like this. To fix it, you would need to: 1. git rebase -i HEAD~[# of commits to the merge above] 2. when it pops up the list of commits, just save (you're not going to re-order them at that point) 3. check for any merge commits, dual adds or deletes with "git status" 4. fix any issues that you see: fix merges, etc. 5. git add (or del) files as needed 6. git rebase --continue 7. go to 3 as needed until all is fixed -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpGCgaCe_wBM.pgp Description: PGP signature
Re: Fixing the merge commit in the git repo?
On Thu, Dec 11, 2014 at 04:16:29PM -0500, Clebert Suconic wrote: > Rebasing and pushing is not a good option IMO. We have been using pull > requests from GitHub and pushing them through Apache. It's working very well > for us. > > Committing directly to Apachea may get you these issues. > > We can provide you guys more information on how we are doing on activemq6 if > you are interested. So if we don't want to go back and fix history (which is fine) can we take steps to avoid pushing merge commits in future? It's just an ugly and confusing situation to get into with them. For anybody who's not familiar with a good git workflow: 1. create a task branch : git checkout -b my-working-branch 2. create that voodoo that you do 3. when done, rebase your task branch on master : git rebase -i master 4. checkout master and merge : git checkout master; git merge my-working-branch 5. make sure master is up to date : git pull --rebase 6. push your changes 7. profit! This way we don't have those ugly diamonds in the gitg graph and issues like this in future. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpHx0DDTzxNY.pgp Description: PGP signature
Re: Fixing the merge commit in the git repo?
On Thu, Dec 11, 2014 at 04:16:29PM -0500, Clebert Suconic wrote: > Rebasing and pushing is not a good option IMO. We have been using pull > requests from GitHub and pushing them through Apache. It's working very well > for us. > > Committing directly to Apachea may get you these issues. > > We can provide you guys more information on how we are doing on activemq6 if > you are interested. Additionally, while working on a task branch, to resynch with master do a rebase: $ rebase -i master rather than merging master down onto your task branch. I saw a *lot* of that while examining the merge commits. Rebasing is by far one of the most awesome features of git. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpeYfzyU3P0t.pgp Description: PGP signature
Re: Fixing the merge commit in the git repo?
On Thu, Dec 11, 2014 at 07:19:13PM -0500, Clebert Suconic wrote: > Right but you can’t ever push -f on an apache branch. you can rebase as much > as you like .. and it’s awesome I agree… > But others would lose reference if you rebased and pushed -f.. that’s why > it’s forbidden at the apache git. > > And that’s why we are very careful on merging commits. I agree, as it can cause issues for someone who wasn't expecting history to have been fixed. Way back in the day, on a previous project, we had a commit hook that would reject merge commits, forcing the developer to fix their history before pushing anything. Does Apache support such hooks in projects? -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgp_XV0TUjtq5.pgp Description: PGP signature
Re: Fixing the merge commit in the git repo?
On Thu, Dec 11, 2014 at 06:47:06PM -0500, Clebert Suconic wrote: > We do a slightly different approach. If you guys are doing git I really think > you guys should consider it. > > 1. create a task branch : git checkout -b my-working-branch > > 2. create that voodoo that you do > > 3. when done, rebase your task branch on master : git rebase -i master > > 4. push to your github fork > 5. send a github Pull Request towards the github fork of the apache project > > 6. Someone else (yeah… someone who was not the original committer) should > merge it after review > > 7 That someone else will do: git merge --no-ff github/pr/PRNumber > ( We ellected to have merge committs with our committs, so we always have two > people to blame in case things go wrong) > > If you elect to not have the merge commit like we have, you can simply use > git pull.. and that should presever the Hashes from the original commit. > > 8. The apache bot will close the PR unless you rebased the commit (which you > are not supposed to) > > > > Why we do that? > > > I - to avoid merge errors like it just happened > II - it increases community > iii - we have a theory that everyone will do stupid things eventually... this > is an extra layer of protection :) > > > You could look at our README for more information: We are still getting > started with it and we have based this on how other apache projects are using > git and github: > > > https://github.com/apache/activemq-6/blob/master/README.md I like the idea of pull requests and explicit peer reviews for changes. But it's above my pay grade to do anything more than envy such a work flow. :D -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpVnaw8sVRxg.pgp Description: PGP signature
Re: Fixing the merge commit in the git repo?
On Fri, Dec 12, 2014 at 01:43:45PM +, Gordon Sim wrote: > On 12/12/2014 12:16 PM, Darryl L. Pierce wrote: > >I like the idea of pull requests and explicit peer reviews for changes. > >But it's above my pay grade to do anything more than envy such a work > >flow. :D > > Pay grade isn't relevant on an Apache project. I was being facetious. I had mentioned pull requests a while back and got the response that (at least then) it wasn't being considered. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpfW3_aoy4LV.pgp Description: PGP signature
Re: Fixing the merge commit in the git repo?
On Fri, Dec 12, 2014 at 09:48:05AM -0500, Clebert Suconic wrote: > In my experience there is no way to proper work with git without PRs and peer > review.. but that’s up to you guys. I feel like my comment, which was just a joke, might have been taken the wrong way. For that I apologize... -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpdG50SwZL1q.pgp Description: PGP signature
Re: Fixing the merge commit in the git repo?
On Fri, Dec 12, 2014 at 02:15:21PM -0500, Andrew Stitcher wrote: > On Thu, 2014-12-11 at 17:11 -0500, Darryl L. Pierce wrote: > > Additionally, while working on a task branch, to resynch with master do > > a rebase: > > > > $ rebase -i master > > > > rather than merging master down onto your task branch. I saw a *lot* of > > that while examining the merge commits. Rebasing is by far one of the > > most awesome features of git. > > I agree and disagree with this simplistic position. > > If (and only if) your task branch is yours and yours alone and no one > has ever relied on it then you can safely rebase it. Actually I find > that rebasing is a lot more useful to get my commits in a logical > sequence of smaller working commits (by using rebase -i). Sorry, I should have clarified that this is my assumption, that a task branch is a private branch that you are working on and not one you're sharing with others. > On the other hand merging topic branches into master is also perfectly > sensible to finish off work on a topic branch especially a long lived > one. > > A long lived topic branch will necessarily have some level of merging > from master to keep it up to date. > > On the other hand I agree that merging from master just before merging > to master is irritating and pointless. > > Andrew > > -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpXSHYo3efcH.pgp Description: PGP signature
Re: Fixing the merge commit in the git repo?
On Fri, Dec 12, 2014 at 02:37:53PM -0500, Rafael Schloming wrote: > So what was the right thing to do here? I have to admit I struggled a bit > with the git portion of getting the whole thing landed. The changes were > (mostly) isolated to a subdirectory, yet whatever git incantations I seemed > to come up with forced me to painstakingly hand merge lots of conflicts > over and over again. What I ended up doing was mostly stumbled into in an > attempt to avoid all that grunge. Yeah, that's because there were several spots where master was merged into the javascript branch. So all of the changes that lived in master were already present in javascript (as a part of those merges) but were not in the same deltas as they live on master. So git was seeing the same changes show up in multiple separate commits and needed someone else to decide which was the right one. That's why rebasing your task branch in git is the Right Way(tm) to prepare a task branch for merging up to master. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpQwi65tvZDY.pgp Description: PGP signature
Re: Welcome Dominic Evans as a Qpid committer
On Mon, Dec 15, 2014 at 11:03:44AM +, Robbie Gemmell wrote: > The Apache Qpid PMC have voted to grant commit rights to Dominic Evans in > recognition of his contributions to and involvement with Proton. > > Welcome, Dominic! Congrats, Dominic! -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpPbsdn3MojE.pgp Description: PGP signature
Re: c reactor / gordon's examples
On Wed, Jan 14, 2015 at 08:28:50AM -0500, Rafael Schloming wrote:
> import sys
>
> reactor = Reactor(Program())
> reactor.acceptor("0.0.0.0", "5672")
> reactor.connection(Client("localhost:5672", sys.argv[1:]))
> reactor.run()
I like the simplicity of how a simple app can be setup. The above can be
used inside of an EventMachine block in Ruby. Would we be able to pass a
file descriptor to the underlying C code? I'm wondering if we could
provide an AMQPServer similar to others provided [1].
[1] https://github.com/eventmachine/eventmachine/tree/master/lib/em/protocols
--
Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
Delivering value year after year.
Red Hat ranks #1 in value among software vendors.
http://www.redhat.com/promo/vendor/
pgpJtcVpn667n.pgp
Description: PGP signature
Ruby and the Engine APIs
I've been working on providing the low-level engine APIs in Ruby over the past few weeks, and from what I can see I think I'm near to completion regarding wrapping them. However, I'm not sure if I'm missing something since I don't see a way to use the code to create a connection. :D Specifically, what I want to do next is create examples of working with these low-level APIs, initially a simple send/receive example would be best with a receiver that listens for a connection and a sender that connects to it, all while being able to toggle tracing the communication, etc. I see nothing existing for other languages to use as a guide, so am a little stumped at the moment. The work I've done to date is here: https://github.com/mcpierce/Proton/tree/PROTON-799-Ruby-engine-apis -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpZ0NMIWLvW8.pgp Description: PGP signature
Re: removing the proton driver API
On Thu, Jan 22, 2015 at 10:50:22AM -0500, Rafael Schloming wrote: > I just noticed that dispatch seems to have it's own copy of driver.c now. I > think that means the driver API is now dead code as messenger, the new > reactor stuff, etc all use the newer selector API. > > Is anyone else using/aware of anyone using this code in anyway? I would > like to at a minimum deprecate it for 0.9, and preferably remove it > entirely if it is in fact currently unused. Are there other APIs, like listener or connector, that'll be removed as well? -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgps7d72Xxvta.pgp Description: PGP signature
Ruby memory management (was: Ruby and the Engine APIs)
On Thu, Jan 22, 2015 at 12:08:52PM -0500, Rafael Schloming wrote: > The most important thing to get worked out for this is the memory > management semantics between C and Ruby. From what I can tell from your > branch, it looks like you haven't done that yet. My initial readings into how Ruby handles object references initially lead me to believe that, with Swig, we're receiving some benefit already WRT object management. Since Ruby uses mark and sweep rather than reference counts, when a Ruby object (or an C object wrapped by Swig as a Ruby object) goes out of reference then it'll get garbage collected. But I won't lie: there might be some subtle detail I'm missing here. I played around with this last night after your suggestion and wrote some stuff on a side branch [1]. It's a simple test that uses the following types: * pn_rubyref_t - C type in Proton that's wrapped by Swig; holds a void * reference to any kind of object * Farkle - a pure Ruby class I then wrote a simple app that creates 1M instances of rb_rubyref_t and 1M instances of Farkle as assigns a Farkle to each rb_rubyref_t. It then puts them each into an array, which is size limited to 100k entries (to allow any GC to run while the app is going). I also added finalizer function to Farkle that just outputs some text when the object is being garbage collected. What I see as the app runs is the output of the Farkle instances being garbage collected while there are still instances being created. The app then ends by sleeping and then exiting. Before and after sleeping the app outputs the number of objects (via ObjectSpace) exist for each type. Running it multiple times I see all of the pn_rubyref_t and Farkle instances being cleaned up, no memory leaks. I then changed things to make a circular reference between Farkle and pn_rubyref_t. Re-ran the tests and still see the objects getting cleaned up. I also ran top to keep an eye on memory usage for the ruby-mri process. O_o (1) [J:0/1028] mcpierce@mcpierce-laptop:cmake (0.3-fedora) $ top -b | grep ruby-mri 19989 mcpierce 20 0 188964 23228 6380 S 53.3 0.3 0:00.55 ruby-mri 19989 mcpierce 20 0 202644 36892 6380 S 18.3 0.5 0:01.10 ruby-mri 19989 mcpierce 20 0 207492 41784 6380 R 19.9 0.5 0:01.70 ruby-mri 19989 mcpierce 20 0 227748 62108 6380 S 17.6 0.8 0:02.23 ruby-mri 19989 mcpierce 20 0 229796 64020 6380 R 22.3 0.8 0:02.90 ruby-mri 19989 mcpierce 20 0 233096 67188 6380 R 57.1 0.8 0:03.42 ruby-mri 19989 mcpierce 20 0 233096 67188 6380 S 0.7 0.8 0:03.44 ruby-mri 19989 mcpierce 20 0 233096 67188 6380 S 0.0 0.8 0:03.44 ruby-mri 19989 mcpierce 20 0 233096 67188 6380 S 0.0 0.8 0:03.44 ruby-mri 19989 mcpierce 20 0 233096 67188 6380 S 1.3 0.8 0:03.48 ruby-mri 19989 mcpierce 20 0 235340 69564 6380 R 48.5 0.9 0:04.94 ruby-mri 19989 mcpierce 20 0 235340 69564 6380 S 47.5 0.9 0:06.37 ruby-mri 19989 mcpierce 20 0 235340 69564 6380 R 50.5 0.9 0:07.89 ruby-mri 19989 mcpierce 20 0 235340 69564 6380 S 48.0 0.9 0:09.34 ruby-mri I only show those lines since, after that point, the virtual memory footprint didn't grow for the run os the apps run. [1] https://github.com/mcpierce/Proton/tree/c-to-ruby-reference-gc-check -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgp0PMt9ATWz_.pgp Description: PGP signature
Re: Ruby memory management (was: Ruby and the Engine APIs)
On Fri, Jan 23, 2015 at 11:02:59AM -0500, Rafael Schloming wrote: > For more info on how to integrate with Ruby's GC you can read this > article[1]. It's one of the few pieces of documentation I've found that > actually explain how to keep a reference from C to a Ruby object. > > [1] > http://clalance.blogspot.com/2013/11/writing-ruby-extensions-in-c-part-13.html This blog post shows how to manually do what Swig is doing for us: represent a C struct as something Ruby can touch, with hooks to release memory when GC runs on the Ruby wrapper. But, sadly, it's not showing what we need, which is how to assign a reference to a Ruby object to a C struct and then have Ruby not GC that Ruby object. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgptytdsND3ZS.pgp Description: PGP signature
Re: Ruby memory management (was: Ruby and the Engine APIs)
On Fri, Jan 23, 2015 at 01:49:34PM -0500, Rafael Schloming wrote: > It does talk about what swig does, but it also talks about the other > direction: > > "If the C structure references other ruby objects, then the mark function > pointer must also be provided and must properly mark the other objects with > rb_gc_mark()" What I meant is it's not showing how to do that. Swig is the one that's generating that Ruby struct wrapping, but I haven't found (yet) if they expose to users a way to tap into that adn do what we want. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpnb2qwpo8uP.pgp Description: PGP signature
Re: Ruby memory management (was: Ruby and the Engine APIs)
On Fri, Jan 23, 2015 at 03:25:33PM -0500, Rafael Schloming wrote: > On Fri, Jan 23, 2015 at 2:08 PM, Darryl L. Pierce > wrote: > > > On Fri, Jan 23, 2015 at 01:49:34PM -0500, Rafael Schloming wrote: > > > It does talk about what swig does, but it also talks about the other > > > direction: > > > > > > "If the C structure references other ruby objects, then the mark function > > > pointer must also be provided and must properly mark the other objects > > with > > > rb_gc_mark()" > > > > What I meant is it's not showing how to do that. Swig is the one that's > > generating that Ruby struct wrapping, but I haven't found (yet) if they > > expose to users a way to tap into that adn do what we want. > > > > If you can find a way for swig to let you control it that's great, but I > was thinking we could just wrap this one struct by hand. It shouldn't be a > whole lot of code for just that struct, and swig can still handle the rest > of them. +1 Though, I was hoping we could avoid having to manually do things... -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpX0na7kqfSD.pgp Description: PGP signature
Re: Ruby memory management (was: Ruby and the Engine APIs)
On Fri, Jan 23, 2015 at 03:46:34PM -0500, Darryl L. Pierce wrote: > +1 Though, I was hoping we could avoid having to manually do things... So I have a working POC that assigns a Ruby object to a C struct in such a way as to keep the Ruby object from being reaped. The solution (for now) stores the object in a hidden global array for such objects for as long as they're held by the C structure and, when C is deleted or the reference changed, the object is removed from the array and available for reaping. I submitted a question to the Swig users mailing list, but that seems to be pretty low traffic and effectively unmanned ATM. Only 15 posts there in the last month and none of them have followups. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpAejYR2VHPv.pgp Description: PGP signature
Re: Ruby memory management (was: Ruby and the Engine APIs)
On Wed, Jan 28, 2015 at 06:04:57AM -0500, Rafael Schloming wrote: > On the face of it this sounds like it could be quite brittle and probably > more complicated than just forgetting about swig for the one pn_rubyref_t > struct and wrapping it manually. Did you attempt the latter option at all? Yes, there is a POC of this on my branch as well. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgphTZBbOghP3.pgp Description: PGP signature
Re: Ruby memory management (was: Ruby and the Engine APIs)
On Wed, Jan 28, 2015 at 09:19:29AM -0500, Rafael Schloming wrote: > On Wed, Jan 28, 2015 at 9:06 AM, Darryl L. Pierce > wrote: > > > On Wed, Jan 28, 2015 at 06:04:57AM -0500, Rafael Schloming wrote: > > > On the face of it this sounds like it could be quite brittle and probably > > > more complicated than just forgetting about swig for the one pn_rubyref_t > > > struct and wrapping it manually. Did you attempt the latter option at > > all? > > > > Yes, there is a POC of this on my branch as well. > > > > Did it work? Can you send me a pointer to it? Yes, it works. It uses Data_Wrap_Struct to encapsulate the pn_rubyref_t type, rb_gc_mark to mark any Ruby object held by that type against reaping, and does appropriate alloc and free operations on instances of the struct. https://github.com/mcpierce/Proton/tree/c-to-ruby-reference-gc-check -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpRloAgXb9mC.pgp Description: PGP signature
Re: Ruby memory management (was: Ruby and the Engine APIs)
On Wed, Jan 28, 2015 at 12:06:44PM -0500, Rafael Schloming wrote: > Why did you reject it then? Reject it? I don't recall rejecting any option. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgp_t6ZoIcDDx.pgp Description: PGP signature
Re: Ruby memory management (was: Ruby and the Engine APIs)
On Wed, Jan 28, 2015 at 12:06:44PM -0500, Rafael Schloming wrote: > Why did you reject it then? Are you referring to this? "Though, I was hoping we could avoid having to manually do things..." What I meant was that I would like to keep the work within the confines of the Swig code. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgp41pGyVkchT.pgp Description: PGP signature
Re: Ruby memory management (was: Ruby and the Engine APIs)
On Wed, Jan 28, 2015 at 01:10:28PM -0500, Rafael Schloming wrote: > On Wed, Jan 28, 2015 at 1:05 PM, Darryl L. Pierce > wrote: > > > On Wed, Jan 28, 2015 at 12:06:44PM -0500, Rafael Schloming wrote: > > > Why did you reject it then? > > > > Reject it? I don't recall rejecting any option. > > I meant why did you post about the global array thing and not this. Is > there some reason you think one is preferred to the other? I described it since it was the one different from what we discussed early based on the blog posts. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpldLHb8zHLD.pgp Description: PGP signature
Questions regarding pn_record_* APIs
Can someone provide a little detail on the set of APIs? I'm looking to get a better understanding of the PYCTX value defined in the wrapper.py class and how it is uses in the APIs when passed to pn_record_def. I'm looking to emulate this behavior in the Ruby but am unclear as to what the PYCTX value represents. Any insight would be helpful. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpAyZ6OoOVeX.pgp Description: PGP signature
Ruby, Proton Engine and Records
So over the last week I've been working on a way to avoid leaking memory or causing segfaults when the underlying C library is giving a reference to a Ruby object. And after much coding and gnashing of teeth I foudn the easiest way to do this with manually wrapping the pn_record_t struct. That would allow for adding a mark function to mark any Ruby objects so that they could not be reaped during garbage collection. And this would work fine but for one problem: the instances of pn_record are created by transport, etc. as part of *their* initializations. So there is no way for us to manually wrap the pn_record_t type for Ruby use in Swig. So looking at things, I have an alternate method that's not brittle and which gives us the feature we're looking for. From the Swig code, we can wrap the call to pn_record_set, at which point we have access to the Ruby wrapper for the pn_record_t type. We can then add/modify instance variables on that Ruby wrapper. Specifically, we can add and then update one named pn_record_attrs (or any other arbitrary name) and use it to track any objects that're added or replaced for the record. And the bonus is that, when the pn_record_t is deleted, it's Ruby wrapper is deleted as well and _all_ instance variables (including our Ruby objects) will be reaped along with it. I'm going to do some more testing on this, but my initial findings support this memory management scheme as viable. I have a test application that creates 500k+ values, storing them in a single record instance. Iterating back over the records and recovering them after running garbage collect returns variables as expected. Commenting out the portion that manages the wrapper instance variable and the same tests fail with a segfault while accessing the records. The work can all be checked out here [1], with the changes to the ruby.i file as the most recent commit in the branch. I'd love to get some feedback on this work in general as well. [1] https://github.com/mcpierce/Proton/tree/PROTON-799-Ruby-engine-apis -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgp3CnIpetAP6.pgp Description: PGP signature
Re: Ruby memory management (was: Ruby and the Engine APIs)
On Wed, Jan 28, 2015 at 01:22:45PM -0500, Rafael Schloming wrote: > Also, have you been able to validate your testing strategy for either/both > of these POCs? Can you generate seg faults and/or valgrind warnings when > you intentionally comment out the line of code that keeps the reference > alive? The POC that uses manual wrapping of a C struct works correctly, preventing objects from being reaped without leaking memory. I validated this by creating exhaustive (1M+) instances of both pure Ruby and C structs that have been wrapped via the Data_Wrap_Struct, assigning the Ruby object to the C struct so that only C held a reference to it, then calling GC.start to reap objects and then checking that the expected number of the pure Ruby objects still existed, via ObjectSpace.each_object([class]).count. Accessing the C-help Ruby object and doing functions such as class_eval on it worked without segmentation faults. I then ensures that it wasn't a fluke by commenting out, in the function that marks the Ruby object in C to keep it from being reaped, and re-running the tests. The app *immediately* segfaults after trying to class_eval the first Ruby object after garbage collection. So this path is the right one to follow. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpjpMUrU8T1Y.pgp Description: PGP signature
Non-reactive engine examples?
Are there any example apps for Python that don't use the reactive APIs? -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpBlkS2f_1Sx.pgp Description: PGP signature
Re: Non-reactive engine examples?
On Mon, Feb 02, 2015 at 09:12:22AM -0500, Darryl L. Pierce wrote: > Are there any example apps for Python that don't use the reactive APIs? If no examples, perhaps a primer of how the logical components of the enger interact to help guide providing an example? -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpCxB5cYjxAI.pgp Description: PGP signature
Re: Ruby, Proton Engine and Records
On Fri, Feb 06, 2015 at 08:37:07AM -0500, Rafael Schloming wrote: > Is this the approach you referenced in your other email? (Same request here > for a pointer to the highlighted key bits of the approach.) No, this was a third approach to the problem, since we can't directly affect the object created by Swig via Data_Wrap_Struct. You can see the specific changes to ruby.i here [1]. http://github.com/mcpierce/Proton/commit/1e66f39535e71dc376d69abde7b5741929d33a3d -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgprYbkagDgS7.pgp Description: PGP signature
Ruby: Engine send experiencing problems
So I'm hitting an interesting problem now with the work I'm doing on the Ruby engine APIs. I have both engine_send.rb and engine_recv.rb [1] examples written and working well. Today I changed them to be single-threaded applications, and that's when things got interesting. With a single threaded sender and receiver, the receiver hand handle messages up to a point (from about 2400 to 4800 messages) before blowing up with a EPIPE error. In debugging this I added a small piece of output to show the state of the sender. And suddenly the problem disappeared! I comment out the this piece of debugging and the problem comes back. Talking with a cube neighbor, I asked for a suggestion from him. His idea was to add a small sleep (0.0001s) at the spot where the debug line was executed in the code. Problem goes away. Change that to "sleep 0". Problem goes away. Comment the line out, and the problem comes back. So it's a very interesting issue. Adding the sleep 0 still involves a kernel call, so there's still even the smallest of delays there. But it seems that, for some reason, the code seems to be overrunning the outgoing buffer (first theory) and causing the connection to close. If you look at the selectable.rb class [2] it reads as only pushing as many bytes into the transport as it says it can handle. *LAST MINUTE EDIT* When turning on debugging in the send app (--debug, which just outputs a single line displaying an event when it occurs) is enough to keep the problem from occurring. It feels like this problem might be a buffer having a problem, though whether it's on the send or the receive side I'm not sure yet. The Ruby code is based on the Python code for handling reading and writing, and it asks the underlying Transport how many bytes can be written to it before trying to do so. [1] https://github.com/mcpierce/Proton/tree/PROTON-799-Ruby-engine-apis/examples/ruby [2] https://github.com/mcpierce/Proton/blob/PROTON-799-Ruby-engine-apis/examples/ruby/lib/selectable.rb -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgprHNmRNGs10.pgp Description: PGP signature
pn_incref/pn_decref and Ruby
Do I need to use reference counting from within the Proton library even though I'm writing in a language that doesn't use them? I asked because I see calls to pn_incref/pn_decref in the Python bindings. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgp5O0WzpfHeA.pgp Description: PGP signature
Proton and Ruby object references...
Revisiting this topic given newly discovered issues: Back in January I had a solution that worked quite well with passing in Ruby objects to the C library. Ruby objects could be attached to Swig wrappers around structs so that their lifecycle's were kept; i.e., when C object X was garbaged collected, its wrapper would go away, and associated objects Y & Z were reaped since they no longer had a parent. However, it's now come up that the Swig wrappers themselves don't seem to live a terribly long time in some cases. Also, asking Swig for the same struct more than once results in a completely different Swig wrapper coming back; i.e., there's no one-to-one mapping between the Swig wrapper and the underlying struct. And, the big problem, the Ruby object passed to the C structs are still vulnerable to being reaped as a result of this. To prove this, I created two test APIs to store and retrieve a single pn_transport_t instance [1]. I create a transport, store it and then fetch it again and get two different objects that are are not equal: t = Cproton.pn_transport => t now holds a reference to an instance of pn_transport_t Cproton.pn_transport_set_mine t => that pn_transport_t struct is now pointed to in the C libraries u = Cproton.pn_transport_get_mine => u now holds a reference to the same instance of pn_transport_t t == u => false -- comparison fails Additionally, the pn_record_get API seems to have an issue with retrieving stored objects that may or may not be related to this issue. I've written a reproducer [2] that simply stores and retrieves attachments for a transport, creates a hash, forces garbage collection and then shows object statistics for the MyHash and wrapper for the pn_record_t type in Ruby space. Specifically, if lines 101 and 103-4 are commented out, the reproducer runs fine and creates a whole lot of new transports before going back for the original and getting its attachments. If lines 102 thru 104 are commented out the reproducer runs fine and retrieves the same attachments over and over without a problem. BUT, if 101-2 are commented out and the reproducer alternates between getting the stored one and a new one, it blows up on the first call to retrieve the stored reference after getting a single new transport. This doesn't seem to make sense since I can create 1k new transports and then get my old one back, but creating a single new one and then getting the old one back it blows up. And where it blows up is on displaying the retrieved attrs object. I know that this issue is related to garbage collection: I can add the attrs variable to a global array and the error never happens in the last scenario. But this doesn't seem to be a good solution since it would require a lot of extra code to track and remove objects when a C struct is deleted. So, in the end, I need a better way to store and retrieve Ruby objects in the library than is currently being done. It works well with the engine APIs since you're not creating and then re-creating objects, so there's not a lot of storing and fetching going on. But with the reactive APIs, it wants to store object state details and recreate them via the various class wrap methods. And this is failing for Ruby since Ruby's mark and sweep is deleting objects that C is holding. [1] pn_transport_set_mine, pn_transport_get_mine [2] http://github.com/mcpierce/Proton/blob/PROTON-799-Ruby-engine-apis/examples/ruby/wrapper_test.rb -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpCQP9zPoED5.pgp Description: PGP signature
Re: Proton and Ruby object references...
On Fri, Mar 06, 2015 at 04:41:34PM -0500, Darryl L. Pierce wrote: > Revisiting this topic given newly discovered issues: Thinking about this over the weekend, it feels like it would be a huge nightmare to try and put a shim in place to keep a Ruby object held by a C struct from being reaped. Especially given that Swig is recreating Ruby wrappers and losing state information on them. But I had an idea taken from a previous project that feels like it would solve this issue. Instead of having a Proton C struct hold a reference to a Ruby object and dealing with the GC issues, why not instead have that C struct hold a unique string? Within Ruby we can maintain a private (not visible to the application) hashtable that uses that string value as a key to point to a Ruby object that is the true value of interest. To assist this, we would add a function pointer to the record APIs (pn_record_remove) that gets called when a record is deleted. It passes in the void* value for each field in the record, if the function is assigned, and that function can work as a hook to remove the object from the global hash. Why do this instead of having C hold a Ruby object and stuffing the Ruby object into some sort of array? I think it would be a simpler, cleaner operation to have C not know if's holding onto Ruby. So, for example, if something happened to the hash or the object held in the hash, we wouldn't get a segfault to access the value from C: instead, we would have a key with no matching value. Something that would be easier to debug than what could appear to be a random segfault. Thoughts? -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpG_tP74PzQd.pgp Description: PGP signature
Re: VOTE: Release Proton 0.9-rc-1 as 0.9 final
On Mon, Mar 09, 2015 at 09:13:07AM -0400, Ken Giusti wrote: > Anyone else getting the following errors when building the docs? > > > ;; This buffer is for notes you don't want to save, and for Lisp evaluation. > ;; If you want to create a file, visit that file with C-x C-f, > ;; then enter the text in that file's own buffer. > > Generating example index... > finalizing index lists... > lookup cache used 974/65536 hits=10705 misses=982 > finished... > Built target docs-c > [ > Error: Could not find a file or object named proton/handlers.py > Error: Could not find a file or object named proton/reactor.py > Error: Could not find a file or object named proton/utils.py > Error: Could not find a file or object named proton/wrapper.py > [.. > + > | File > /home/kgiusti/work/proton/0.9rc1/qpid-proton-0.9-rc-1/proton-c/bindings/python/proton/__init__.py, > line 3778, in proton.Url > | Warning: Line 3781: Improper paragraph indentation. > | > > [] > Built target docs-py I'm seeing it on my end, yes. The build finishes, but the installed docs don't include the messaging package or anything outside of the proton module. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpMx07M9x1Wb.pgp Description: PGP signature
Re: [VOTE]: Proton 0.9-rc-3
On Tue, Mar 17, 2015 at 09:42:18AM +1300, Rafael Schloming wrote: > Hi Everyone, > > Here's a quick respin of 0.9-rc-3. The only changes from rc-2 are exactly > those two mentioned on the rc-2 vote thread. I've included them at the end > for reference. You can find the source artifacts in the usual location: > > https://people.apache.org/~rhs/qpid-proton-0.9-rc-3/ > > Java binaries are here: > > https://repository.apache.org/content/repositories/orgapacheqpid-1031 > > Please check it out and register your vote: > > [ X ] Yes, release Proton 0.9-rc-3 as 0.9 final > [ ] No, because ... Packaged for Fedora. Installed all packages. Tested across language bindings (Python, Perl, Ruby). Found a single non-blocking issue in Perl that I've fixed and pushed. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpWRFD4rhjeD.pgp Description: PGP signature
Strange memory issues with handlers
Seeing a strange issue while debugging a segfault issue with reactors
and Ruby. I'm seeing the char* value for a record showing up as the
dispatch address for either a handler->dispatch or else one of its
children.
= Records
In Ruby, to avoid objects being reaped by GC, they aren't given to the C
library. Instead, a char* is malloced and the string snprintf'd with the
hex address of the pn_record_t, the pn_handle_t and the value being
stored. That string is given to the pn_record_set to store, while a ruby
version of the string value is created and used as the key to stuff the
actual Ruby object into a hidden global hash.
Later, when pn_record_get is called, it retrieves that previous key
value and uses it to retrieve the object from the global hash.
Lastly, if registered, a cleanup function is called that allows for
deleting the malloc'd key, which also removes the entry from the hash
and allows the Ruby string and object to be GC'd.
= Handlers And The Memory Issue
So, like with the above records API, Ruby handlers can't be stored in
the C libraries directly. Instead, the code uses the struct:
typedef struct {
char *handler_key;
} pni_rbhandler_t;
The value for handler_key is, like with records, the hex form of the
Ruby object and the pni_rbhandler_t object created. And the handler is
stored in the global hash using that key value so it can be retrieved
later. And the pn_handler_finalize method removes the key and value from
the hash later.
= The Problem
In diagnosing the segfault I found that, when it happens, about 80% of
the time the segfault is caused by handler->dispatch having a bad value.
Specifically I saw values like this for the memory address held:
3536373137302d393231
Looking it, it looked like an ASCII string. So converting it to ASCII I
saw that it was, in this case, the string "567170-921". And when
something like that happened, there was a record stored previous that
had a key containing the string "129-071765"; i.e., the address pointed
to by handler->dispatch was overwritten by the string that had been
stored by a call to pn_record_set in a separate patch of code.
But the problem is that nothing outside of the library has direct access
to assign these values to fields, and I haven't quite tracked down where
the leak is in the code.
--
Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
Delivering value year after year.
Red Hat ranks #1 in value among software vendors.
http://www.redhat.com/promo/vendor/
pgpwv4XvpFkae.pgp
Description: PGP signature
Re: [jira] [Commented] (PROTON-344) Need Ruby version of msgr-send/msgr-recv tools
On Tue, Apr 28, 2015 at 07:51:06PM +, ASF subversion and git services (JIRA) wrote: > > [ > https://issues.apache.org/jira/browse/PROTON-344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14517884#comment-14517884 > ] > > ASF subversion and git services commented on PROTON-344: > > > Commit 5f06462f37fa10d2da36e7755fa813c5b57986bd in qpid-proton's branch > refs/heads/master from [~astitcher] > [ https://git-wip-us.apache.org/repos/asf?p=qpid-proton.git;h=5f06462 ] > > PROTON-344: Fix code not to trigger pedantic C++ warning I think some commits are being applied to the wrong JIRA. I see a few commits against PROTON-344 (which is requestion a Ruby send/receive tool (which exists) but they're all from Andrew Stitcher for SASL work. I think they might be for PROTON-334 instead of -344. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgp6pYERYOrBh.pgp Description: PGP signature
Introducing the Ruby Reactive APIs
I've been working on this codebase since the beginning of the year. The two branches [1, 2] in my git repo represent the low-level engine APIs and the higher-level reactive APIs, respectively. I'm still working through the set of example apps for the reactive APIs, but at this point I feel this is close enough that I want to start getting feedback from people. == Memory Concerns Of particular important is memory management: the Proton libraries use reference counting to manage object lifespans, while Ruby uses mark and sweep operations for garbage collection. So ensuring that pure Ruby objects aren't reaped when they've only known to the Proton libraries, in the case of event handlers specifically, has been a challenge and one that's sure to have some cases that need fixing. The first model explored was to attachment the Ruby wrapper objects to the Swig-generated wrappers for the underlying C structs in Proton. Which worked at first, but turned out to be not useful. The reason being that the Swig bindings were themselves being reaped when they went out of scope; i.e., Swig doesn't maintain them by providing a mark operation until disposal of the underlying C structs. So this path, while initially promising, was discarded. The current model uses a hash table that is attached to the Qpid::Proton module. When objects are stored for use by the C libraries, they are tucked away in this hash table with a unique key generated based on memory addresses. A copy of that key, as a char*, is given to Proton to use later when the object is being retrieved. To help with this, two additional callback APIs were added to the Proton libraries: pn_record_set_callback and pn_record_has_callback. These two functions work to help register a method to be called whenever a record is deleted to enable memory management. This way the above-mentioned key can be properly deleted, and the value stored in the hash table discarded. The reference counting aspect of the Proton libraries is a concern as well. The code currently increments and decrements references in the same places as the Python code, but there are likely more places where such reference accounting need to be added. [1] http://github.com/mcpierce/Proton/tree/PROTON-799-Ruby-engine-apis [2] http://github.com/mcpierce/Proton/tree/PROTON-781-reactive-ruby-apis -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpXevt2WqzQw.pgp Description: PGP signature
Re: Introducing the Ruby Reactive APIs
On Thu, May 07, 2015 at 09:57:49AM -0400, Rafael Schloming wrote: > On Thu, May 7, 2015 at 9:41 AM, Darryl L. Pierce wrote: > > To help with this, two additional callback APIs were added to the Proton > > libraries: pn_record_set_callback and pn_record_has_callback. These two > > functions work to help register a method to be called whenever a record > > is deleted to enable memory management. This way the above-mentioned key > > can be properly deleted, and the value stored in the hash table > > discarded. > > I would need to see the code in detail, but I suspect you don't need to add > a pn_record_set_callback/get_callback to achieve roughly the functionality. > I *think* you could simply define a pn_class_t that is a reference counted > holder of your key. You could then put your callback logic in the finalizer > for that class, and when proton's reference counting triggers the > finalizer, it will run the callback logic at the appropriate time. (edit) As I was writing up a description of the code I realized I have already done what you suggest above WRT the pni_rbhandler_t type. I could use the same logic to create a pni_rbrecord_t type and manage its lifecycle the same way the handler's lifecycles are managed, yeah? -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpFxlAV42Im8.pgp Description: PGP signature
Re: Introducing the Ruby Reactive APIs
On Thu, May 07, 2015 at 11:32:33AM -0400, Rafael Schloming wrote: > On Thu, May 7, 2015 at 10:40 AM, Darryl L. Pierce > wrote: > > > On Thu, May 07, 2015 at 09:57:49AM -0400, Rafael Schloming wrote: > > > On Thu, May 7, 2015 at 9:41 AM, Darryl L. Pierce > > wrote: > > > > > > To help with this, two additional callback APIs were added to the > > Proton > > > > libraries: pn_record_set_callback and pn_record_has_callback. These two > > > > functions work to help register a method to be called whenever a record > > > > is deleted to enable memory management. This way the above-mentioned > > key > > > > can be properly deleted, and the value stored in the hash table > > > > discarded. > > > > > > I would need to see the code in detail, but I suspect you don't need to > > add > > > a pn_record_set_callback/get_callback to achieve roughly the > > functionality. > > > I *think* you could simply define a pn_class_t that is a reference > > counted > > > holder of your key. You could then put your callback logic in the > > finalizer > > > for that class, and when proton's reference counting triggers the > > > finalizer, it will run the callback logic at the appropriate time. > > > > (edit) > > > > As I was writing up a description of the code I realized I have already > > done what you suggest above WRT the pni_rbhandler_t type. I could use > > the same logic to create a pni_rbrecord_t type and manage its lifecycle > > the same way the handler's lifecycles are managed, yeah? > > Yes, I believe so. Since records are created when a struct if initially created, I'm not sure how to go about attaching the key to its lifecycle since the dynamic language isn't explicitly creating the record. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpPrDtwTjlAX.pgp Description: PGP signature
When pushing to the git repo, please avoid merge commits...
A request for all, please avoid doing merge commits on your Git repo. When these hit the shared repo it can cause Git to try and "fix" [1] the commit when rebasing. The way I do commits with Git is to rebase my work branch on master before merging it into master. That way it's a true merge and not a merge commit. To do this: 1. Rebase your work branch: task-branch $ git rebase -i master 2. Fix any merge issues on the work branch until you get a clean set of commits. 3. Switch to master and then merge your branch: task-branch $ git checkout master master-branch $ git merge task-branch Thanks. [1] On my task branches I use autofix to merge changes back to the appropriate previous commit in my branch. When I'm ready to merge them I use "git rebase -i HEAD~[some # of commits] --autofix". Git then takes all of the autofix commits and puts them after the appropriate commit to merge together. If I a merge commit is accidentally included in there then git will try to rebase that merge commit and all hell can break loose. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpfWWtjMVeGY.pgp Description: PGP signature
Re: When pushing to the git repo, please avoid merge commits...
On Mon, May 11, 2015 at 09:23:27AM -0400, Darryl L. Pierce wrote: > A request for all, please avoid doing merge commits on your Git repo. > When these hit the shared repo it can cause Git to try and "fix" [1] the > commit when rebasing. > > The way I do commits with Git is to rebase my work branch on master > before merging it into master. That way it's a true merge and not a > merge commit. To do this: > > 1. Rebase your work branch: > > task-branch $ git rebase -i master > > 2. Fix any merge issues on the work branch until you get a clean set of >commits. > 3. Switch to master and then merge your branch: > > task-branch $ git checkout master > master-branch $ git merge task-branch > > Thanks. > > > [1] On my task branches I use autofix to merge changes back to the > appropriate previous commit in my branch. When I'm ready to merge them I > use "git rebase -i HEAD~[some # of commits] --autofix". Git then takes > all of the autofix commits and puts them after the appropriate commit to > merge together. > > If I a merge commit is accidentally included in there then git will try > to rebase that merge commit and all hell can break loose. Dug out this old post to my blog that has my steps for doing the commits on a task-based branch: http://mcpierce.blogspot.com/2012/08/git-fixup-and-autosquash.html I've added a new comment on this post with the steps for doing an autofix commit, but will write up a newer version of the post to include this inline. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgp_NkrC53d2E.pgp Description: PGP signature
Re: Introducing the Ruby Reactive APIs
On Thu, May 07, 2015 at 01:02:13PM -0400, Rafael Schloming wrote: > The way the python code does this is by checking whenever a C object is > returned to python code. If the record contains an attachment indicating > that the C object has previously been wrapped, it uses this to > construct/retrieve an appropriate wrapper object. If it doesn't have the > appropriate attachment then it uses the record API to define/set the > attachment to the appropriate value. I presume you could do something > similar with ruby. After we chatted the other day, I've tried the following approach, using the pn_transport_t type as my test bed since it has relatively fewer dependencies. However, the plumbing in Proton for objects isn't quite clear to me and my code's not quite working the way we had discussed and I'm not sure why. The goal is to have a type that will live for as long as one of the impls in Proton lives; i.e., when we create something like pn_transport_t, the attachment created for this would hold some type that will get finalized when the pn_transport_t type is finalized. And that type would be the hook to clean up the single instance of a Ruby class that wraps the underlying C type. I've created a new type in the ruby.i descriptor for Swig and named it pn_rbkey_t, with three files: void* registry (a pointer to an object held in Ruby), char* method (the name of a method to invoke on that object), and char* key_value (the argument to be passed to that method). The code defines pn_rbkey_initialize and pn_rbkey_finalize methods, as well as getter and setter methods for the three fields. But I've put debugging into the code and never see the pn_rbkey_finalize method being invoked. My registry_test app does the following: 1. create an instance of pn_transport_t: impl = Cproton.pn_transport 2. create a Ruby Transport object: transport = Transport.wrap(impl) a. puts a weak reference to the Transport into the hashtable b. creates a pn_rbkey_t object and sets it as the sole record for the pn_transport_t object c. calls Cproton.pn_incref on the pn_rbkey_t instance 3. remove the reference: transport = nil 4. call garbage collection: ObjectSpace.garbage_collect 5. get the object back: transport = Transport.wrap(impl) a. calls pn_transport_attachment and retrieves the record created in 2 b. should then be able to get the key_value from the pn_rbkey_t type c. should then get the object out of the hashtable to return It's at step 5 that the example app segfaults. The segfault happens when, from Ruby, there's a call to print the attachment retrieve in 5a. Swig isn't failing since it's returning a value that seems to have been cached. But when Swig tries to retrieve data from the pn_rbkey_t struct underneath of it, *THAT* seems to have been reaped by Proton and Swig then segfaults, thinking there was an object still under the covers. Any ideas or suggestions of where to look for what's going on? -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpmyZ1IC9zbA.pgp Description: PGP signature
Re: Introducing the Ruby Reactive APIs
On Tue, May 12, 2015 at 05:45:20AM -0400, Rafael Schloming wrote: > Can you post an isolated reproducer with just your definition of pn_rbkey_t > and a code version of the 5 steps that lead to the seg fault? On my PROTON-781-reactive-ruby-apis branch is an example named "$REPO/examples/ruby/registry_test.rb" which does it. It's very pared down, only creating 3 Transport instances and consistently produces the segfault. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpPzg3g1inst.pgp Description: PGP signature
Re: Introducing the Ruby Reactive APIs
On Tue, May 12, 2015 at 09:44:41AM -0400, Rafael Schloming wrote: > On Tue, May 12, 2015 at 8:34 AM, Darryl L. Pierce > wrote: > > > On Tue, May 12, 2015 at 05:45:20AM -0400, Rafael Schloming wrote: > > > Can you post an isolated reproducer with just your definition of > > pn_rbkey_t > > > and a code version of the 5 steps that lead to the seg fault? > > > > On my PROTON-781-reactive-ruby-apis branch is an example named > > "$REPO/examples/ruby/registry_test.rb" which does it. It's very pared > > down, only > > creating 3 Transport instances and consistently produces the segfault. > > > > That branch has about a 15 thousand line delta from master. That's a lot of > lines of code to hide a subtle memory bug. The pn_rbkey_t definition and > enough ruby code to use it in a proof of concept should only require a few > hundred line delta from master. I suggest producing just this delta for two > reasons. 1) Just the act of producing it will help narrow down where the > bug is, and 2) it gives me a much smaller delta to look at so I can be more > useful to you. > > (I did run the registry_test.rb that you pointed to through a debugger, > however its not obvious what the issue is upon inspection of the trace, and > while the ruby.i delta is pretty self contained, the details of how you are > using it from ruby are somewhere buried in the 15K diff.) I've pulled the pertinent pieces into a new branch: http://github.com/mcpierce/Proton/tree/rbkey-isolation The changes are all here: http://github.com/mcpierce/Proton/commit/2dc770992a7c02fe2054a4a77325a199e39b9c94 and it blows up exactly as I've been experiencing. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgppsp4k7wEmA.pgp Description: PGP signature
Re: Introducing the Ruby Reactive APIs
On Tue, May 12, 2015 at 11:12:40AM -0400, Rafael Schloming wrote: > I made a bunch of line comments. Excellent, thanks for that. I'll update the code and reply with my results. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgp4ODW9uN9H0.pgp Description: PGP signature
Re: When pushing to the git repo, please avoid merge commits...
On Tue, May 12, 2015 at 02:10:59PM -0400, Alan Conway wrote: > On Mon, 2015-05-11 at 09:36 -0400, Darryl L. Pierce wrote: > master-branch $ git merge --ff-only task-branch > > That will refuse to merge unless the merge is a "fast-forward", which is > the simple "copy my commits to the end of trunk" that we want. It is > just a safety check - if you do what Darryl says it has no effect, but > if you forgot to rebase properly git will complain instead of creating > an unintended merge commit. +1 I've worked on a project that flat out rejects any commits that aren't FF. And it was very simple and clear to do a git --blame to look up where changes came from. > > http://mcpierce.blogspot.com/2012/08/git-fixup-and-autosquash.html > > > autosquash! Lovely! Git is full of nice surprises. When I learned the core set of tricks I use for working (interactive rebase, autofix/autosquash, bisect, cherry picking) is when I finally felt I was working with something more than just a version control system, but a power tool. :D -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpPw4RchyTEz.pgp Description: PGP signature
Re: Introducing the Ruby Reactive APIs
On Tue, May 12, 2015 at 11:12:40AM -0400, Rafael Schloming wrote: > I made a bunch of line comments. Things appear to be working in the scaled back example except for one thing: when retrieving the record with pn_record_get the Swig wrapper returned is of type SWIG:TYPE_p_void rather than the expected wrapper for pn_rbkey_t. I think that Swig is confused by the type being returned by pn_record_get and can't distinguish what the appropriate wrapper should be. When the return value is then used for fetching the key, it fails with: TypeError: Expected argument 0 of type pn_rbkey_t *, but got SWIG::TYPE_p_void #http://www.redhat.com/promo/vendor/ pgpCohtKHXqb9.pgp Description: PGP signature
Re: Introducing the Ruby Reactive APIs
On Tue, May 12, 2015 at 04:10:54PM -0400, Darryl L. Pierce wrote: > On Tue, May 12, 2015 at 11:12:40AM -0400, Rafael Schloming wrote: > > I made a bunch of line comments. > > Things appear to be working in the scaled back example except for one > thing: when retrieving the record with pn_record_get the Swig wrapper > returned is of type SWIG:TYPE_p_void rather than the expected wrapper > for pn_rbkey_t. I think that Swig is confused by the type being returned > by pn_record_get and can't distinguish what the appropriate wrapper > should be. > > When the return value is then used for fetching the key, it fails with: > > TypeError: Expected argument 0 of type pn_rbkey_t *, but got > SWIG::TYPE_p_void #http://www.redhat.com/promo/vendor/ pgpNSkhYEFc0V.pgp Description: PGP signature
Re: Development workflow and release process [WAS: Concurrent Go API for proton is, erm, GO!]
On Tue, May 12, 2015 at 07:06:19PM -0400, Alan Conway wrote: > On Thu, 2015-05-07 at 15:42 -0400, Rafael Schloming wrote: > > > > 1. For developing new language bindings (and really for any development > > > > work that will involve enough new stuff to have a noisy commit history) > > > we > > > > use branches. This is already the case with the Ruby/C++/Python3 > > > bindings, > > > > as well as the SASL work. > > IMO this is a good policy. Go work is now on branch `go`. I left > bindings/go/README.md with a pointer to the new branch so people > following older discussions can find it. > > > I think I pretty much said go ahead on master, so apologies for singling > > you out. > "go" ahead. Hahaha. Just when I think we've done all those jokes, > there's always another one. I do like this. I would love to push my Reactive changes up to a ruby-reactive branch on the core repo and let people examine it there. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpfzdxQl8p7e.pgp Description: PGP signature
Ruby Engine APIs up for review
I've pushed these APIs up for public review on their own branch [1] in the Apache git repo. They're on a branch named ruby-engine-apis. Please take a look, tinker and please provide feedback. :D [1] http://git-wip-us.apache.org/repos/asf?p=qpid-proton.git;a=shortlog;h=refs/heads/ruby-engine-apis -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpIVhSmU6rV8.pgp Description: PGP signature
Direct receive not responding in Ruby...
Got an interesting bug I'm digging into currently. In Ruby I've written analogs to the direct_* and simple_ send/recv example apps. The simple versions all work well and can interact with Python as expected. However, the direct_recv.rb example isn't. When the container starts listening for incoming direct connections, I see the following frames come in when the simple_send.rb client (or Python analog) connects: <- @open(16) <- @begin(17) <- @attach(18) But what I don't see is the outgoing connection connections that occurr in the Python version: -> @open(16) -> @open(17) -> @open(18) So the client connects but the flow never begins between the two. I'm digging into the acceptor code but don't see where Ruby is behaving differently from Python. Any suggestions? -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgp4bxwIzS6qx.pgp Description: PGP signature
Re: Direct receive not responding in Ruby...
On Thu, May 21, 2015 at 04:37:09PM +0100, Gordon Sim wrote: > In python, it is the EndpointStateHandler, that is part of MessagingHandler > which is responsible for that. If ruby is doing something similar, I'd start > by checking if the equivalent handler is being called. As always you nailed it on the head. I'm in your debt, Gordon. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpyVksnSX8EE.pgp Description: PGP signature
Re: New language bindings - when are they ready? [was Re: Ruby Engine APIs up for review]
On Mon, May 25, 2015 at 12:46:32PM -0400, Alan Conway wrote:
> On Thu, 2015-05-14 at 16:28 -0400, Darryl L. Pierce wrote:
> > I've pushed these APIs up for public review on their own branch [1] in
> > the Apache git repo. They're on a branch named ruby-engine-apis.
> >
> > Please take a look, tinker and please provide feedback. :D
> >
> > [1]
> > http://git-wip-us.apache.org/repos/asf?p=qpid-proton.git;a=shortlog;h=refs/heads/ruby-engine-apis
>
> We have a bunch of new bindings on branches (Go, Ruby, C++) When is a
> binding ready for trunk?
>
> I would propose the following as a basic critiera:
>
> - Complete and tested for basic use.
> - API documentation in language-normal form (godoc, rdoc, doxgygen)
> - Core set of examples.
> - Automated testing of all examples.
> - Getting started README that explains how to run the examples.
>
> I would suggest the following core examples for all bindings:
>
> - send: connect to endpoint, send N messages.
> - receive: connect to endpoint, receive N messages.
> - broker: accept connections, in-memory queues created on demand.
> - request_response: synchronous request-response client.
> - echo_server: accept connections, echo message to reply-to address on
> same connection.
This sounds like a reasonable minimal set of examples to provide a POC
of the language.
Additionally, for each class of APIs (messaging, engine, reactive) we
should have matching examples as well: reactive_send, reactive_recv,
etc. This goes hand-in-hand with your later points WRT proper naming.
> This gives us a *small* set of examples that still has good coverage of
> key scenarios (brokered and client-server) and can be used *without* any
> external broker. This can be the common introduction to all our
> bindings.
>
> The python binding has all these, with different names. Rationale for
> proposing the above names:
>
> - The simplest send/receive examples should be called simply
> "send"/"receive". Python has "simple_send"/"simple_receive" which seems
> redundant.
> - "request_response" is more descriptive than
> "sync_client" ("sync_client" is a daft name, my bad)
> - "echo_server" describes what the server does, just "server" is to
> generic. The broker is also a server, and there may be other example
> servers.
>
> I can live with the python names if there's a consensus for that but now
> might be a good time to rationalize. I do think the python examples
> should be re-organized to start with these core examples. Right now the
> python README requires you to start by installing an unspecified
> "intermediary" which is going to cause confusion and frustration.
I was equally confused at first. When I see two examples with similar
names, like direct_send and direct_recv, my first thought was that the
two should be used together. So in addition to better names, the README
file(s) could groups together by topic to make it clearer how to use the
examples. Topics like "Simple Messaging", "Reactive Messaging", etc.
that notes the example app and how it works. I would find that easier to
follow and more intuitive than the current README.
> The very first README for every binding should show how to run
> send/receive examples against the example broker, and request_response
> against the example server. AFTER we have demonstrated that proton is
> useful *on its own* we can and should provide further examples of using
> it with other brokers, tornado, ruby/go/whatever specific frameworks and
> what have you.
>
> Some bindings (Go, possibly C++) provide additional API or toolkit to
> server multiple connections concurrently. They should also provide
> concurrent versions of the core examples. The go examples have send and
> receive take a list of addresses as arguments and send/receive on all
> addresses concurrently. I would suggest that as a model.
>
> Interested in feedback! I'll be working on Go and C++ in coming weeks so
> I'd like to co-ordinate with ruby and python efforts so we can present
> something coherent to our users.
--
Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
Delivering value year after year.
Red Hat ranks #1 in value among software vendors.
http://www.redhat.com/promo/vendor/
pgpsk7mgiqTLQ.pgp
Description: PGP signature
Pushing the Ruby Engine APIs to master...
It's been a while (14 days) since I pu up the pull request for the Ruby engine APIs [1]. If nobody has any objections, and since it won't affect anything outside of the ruby bindings, I plan to push this by COB tomorrow (Wednesday 03 June). [1] https://github.com/apache/qpid-proton/pull/33 -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpG10Go0KyYP.pgp Description: PGP signature
Re: Strange behaviour for pn_messenger_send on CentOS 6
On Thu, Jun 04, 2015 at 03:12:51PM +0100, Frank Quinn wrote: > I hit a strange issue today when setting up a qpid proton development > environment on a fresh CentOS 6 VM. I first found the issue in our > application, but when I went a little deeper, I realized I could recreate > the issue with the qpid proton send and recv example applications. All you > need to do is run ‘send’ on its own and the pn_messenger_send call hangs > indefinitely. If you start ‘recv’ first, it works fine, but ‘send’ on its > own hangs every time. > > This is contrary to its behaviour on my Fedora 21 laptop (latest yum > provisioned 0.8 version) where it always attempts once, logs a failure, > then exits (which is what I would expect). > > This effectively deadlocks our application. So far, I’ve tried compiling > qpid proton c myself (both 0.8 and 0.9.1), setting pn_messenger_send > timeout to 1 (it was previously -1), turning off iptables entirely and > disabling selinux and rebooting but no luck. Is this something you folks > have seen before? Hrm, this isn't something I've heard reported before. Does it do the same if you use the Python recv.py example as well? Also, can you do the following: $ PN_TRACE_FRM=1 ./recv [options] and share the output displayed? Also, is this solely with binaries you've built or are you installed RPMs from EPEL for Proton? -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpMIyo8RW3eT.pgp Description: PGP signature
Re: Proton-c Null Messages
On Mon, Jun 08, 2015 at 05:56:14AM -0700, logty wrote: > Hi all, > > I just had a quick question about proton-c returning null messages. I am > calling pn_messenger_get on my message and on my messenger and getting back > messages that are null. When I call: > > pn_bytes_t b = pn_data_get_bytes(pn_message_body(message)); > > calls to b.start and b.size create segfaults. Any ideas as to why this might > be happening? At the moment I am running a test with large messages (several > megabytes) being sent to the server. > > Thanks for the help! Are you *certain* you're getting messages back? What's the return value on the call to pn_messenger_get? -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpq1f55F5Ztp.pgp Description: PGP signature
Re: Proton-c Null Messages
On Mon, Jun 08, 2015 at 01:47:03PM -0700, logty wrote: > The strange thing is pn_messenger_get is returning 0, I have code to throw an > error if it isn't 0. Also, the large messages work fine on RabbitMQ but not > on Apache Apollo. What's the output when you run your client with: $ PN_TRACE_FRM=1 [your client cmdline] ? Do you see the message being received on the client side? -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpxSErwfJzge.pgp Description: PGP signature
Re: Strange behaviour for pn_messenger_send on CentOS 6
On Tue, Jun 09, 2015 at 10:41:51PM +0100, Frank Quinn wrote: > Just tried this and can recreate on a 64 bit laptop running CentOS 6.6 > natively too. The send application never exits if a receiver is not yet > ready. Can anyone else see this or am I going mad? Are these bits you've built or did you install them from RPM (not sure if you said before, so please remind me)? And are you running a Qpid or other broker? I ask because it seems you're getting a connection (no "connection refused" error) but something else is causing the failure, the SASL header mismatch. How is your SASL configuration setup on the broker to which you're connecting? -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpjlOinpZ7_p.pgp Description: PGP signature
Re: Strange behaviour for pn_messenger_send on CentOS 6
On Wed, Jun 10, 2015 at 02:17:23PM +0100, Frank Quinn wrote: > You can recreate this on a CentOS 6.6 box by installing qpid-proton-c-devel > using yum from EPEL, then compiling the example application that comes with > it in /usr/share/proton/examples/messenger/send.c. > > There is no broker with these example applications - it's point to point. > > There is a matching recv.c application there too. If you start recv first, > it works fine, but if you don't, the send application hangs which I believe > is new behaviour. > > Again, this does not happen on my Fedora laptop - only on CentOS 6. I'm able to recreate this on RHEL6 as well using the RPMs from EPEL. I see the connection aborted notice in the trace and then it fails to exit. It appears to be repeatedly calling pn_transport_closed(). The stack trace I see on EL6 is: (gdb) backtrace #0 pn_transport_closed (transport=0x804fa28) at /home/mcpierce/Programming/Proton/proton-c/src/transport/transport.c:2802 #1 0x00140ea8 in pni_connection_capacity (ctx=0x804f920) at /home/mcpierce/Programming/Proton/proton-c/src/messenger/messenger.c:166 #2 pni_connection_update (ctx=0x804f920) at /home/mcpierce/Programming/Proton/proton-c/src/messenger/messenger.c:196 #3 pni_conn_modified (ctx=0x804f920) at /home/mcpierce/Programming/Proton/proton-c/src/messenger/messenger.c:225 #4 0x00141071 in pn_messenger_process_transport (messenger=0x804cb40, event=0x805c520) at /home/mcpierce/Programming/Proton/proton-c/src/messenger/messenger.c:1201 #5 0x00141134 in pn_messenger_process_events (messenger=0x804cb40) at /home/mcpierce/Programming/Proton/proton-c/src/messenger/messenger.c:1252 #6 0x00141f83 in pni_connection_readable (sel=0x804f958) at /home/mcpierce/Programming/Proton/proton-c/src/messenger/messenger.c:261 #7 0x00143425 in pn_selectable_readable (selectable=0x804f958) at /home/mcpierce/Programming/Proton/proton-c/src/selectable.c:204 #8 0x00141483 in pn_messenger_process (messenger=0x804cb40) at /home/mcpierce/Programming/Proton/proton-c/src/messenger/messenger.c:1310 #9 0x001415c8 in pn_messenger_tsync (messenger=0x804cb40, predicate=0x13da40 , timeout=-1) at /home/mcpierce/Programming/Proton/proton-c/src/messenger/messenger.c:1379 #10 0x00141a97 in pn_messenger_sync (messenger=0x804cb40, predicate=0x13da40 ) at /home/mcpierce/Programming/Proton/proton-c/src/messenger/messenger.c:1410 #11 0x00141c8c in pn_messenger_send (messenger=0x804cb40, n=-1) at /home/mcpierce/Programming/Proton/proton-c/src/messenger/messenger.c:2119 #12 0x08048e59 in main (argc=-72537468, argv=0xb7ffe000) at /home/mcpierce/Programming/Proton/examples/c/messenger/send.c:102 The same backtrace on F22 is: (gdb) backtrace #0 pn_transport_closed (transport=transport@entry=0x60ac40) at /home/mcpierce/Programming/Proton/proton-c/src/transport/transport.c:2801 #1 0x77bbf0a8 in pni_connection_capacity (sel=0x60aaf0) at /home/mcpierce/Programming/Proton/proton-c/src/messenger/messenger.c:166 #2 pni_connection_update (sel=0x60aaf0) at /home/mcpierce/Programming/Proton/proton-c/src/messenger/messenger.c:196 #3 pni_conn_modified (ctx=0x60aa80) at /home/mcpierce/Programming/Proton/proton-c/src/messenger/messenger.c:225 #4 0x77bbf135 in pn_messenger_process_transport (messenger=messenger@entry=0x605970, event=event@entry=0x61a240) at /home/mcpierce/Programming/Proton/proton-c/src/messenger/messenger.c:1201 #5 0x77bbf27b in pn_messenger_process_events (messenger=messenger@entry=0x605970) at /home/mcpierce/Programming/Proton/proton-c/src/messenger/messenger.c:1252 #6 0x77bbf6d9 in pni_connection_readable (sel=0x60aaf0) at /home/mcpierce/Programming/Proton/proton-c/src/messenger/messenger.c:261 #7 0x77bbf7e8 in pn_messenger_process (messenger=messenger@entry=0x605970) at /home/mcpierce/Programming/Proton/proton-c/src/messenger/messenger.c:1310 #8 0x77bbf94f in pn_messenger_tsync (messenger=0x605970, predicate=0x77bbc420 , timeout=-1) at /home/mcpierce/Programming/Proton/proton-c/src/messenger/messenger.c:1379 #9 0x00401158 in main (argc=, argv=) at /home/mcpierce/Programming/Proton/examples/c/messenger/send.c:102 There's definitely a difference in the runtime behavior of Fedora vs. RHEL in this case. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpS6Fao6o8uN.pgp Description: PGP signature
Re: [GitHub] qpid-proton pull request: Proton 781 ruby reactor apis
On Mon, Jun 08, 2015 at 07:10:34PM +, mcpierce wrote: > GitHub user mcpierce opened a pull request: > > https://github.com/apache/qpid-proton/pull/36 > > Proton 781 ruby reactor apis > > The Ruby Reactor APIs plus example applications. Hi, all. If nobody objects, I'm going to merge this into trunk tomorrow (18 June) at COB EST unless anybody objects. This shouldn't have any impact on code outside of the Ruby bindings. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpju48VFZdNP.pgp Description: PGP signature
Inconsistent state value declarations...
In writing the swing contracts for the Proton APIs, I've come across a few inconsistencies with the way state and return values are declared. In sasl.h the pn_sasl_state_t and pn_sasl_outcome_t values are declared using enums. Similarly, in engine.h the pn_disposition_t type is declared using an enum. But pn_state_t values are defined using precompiler defines, same with pn_trace_t values and specific error codes. I know there are reasons to do things certain ways, but what I'd like to know is: can we move the enums to being defined values instead? The reason I ask is that I can then provide better support in the language binding contracts. Contracts can't handle enumeration values but can be told what values are allowed for the other types. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpqJs1aiFGpD.pgp Description: PGP signature
Re: Inconsistent state value declarations...
On Wed, Aug 08, 2012 at 10:53:47AM -0400, Rafael Schloming wrote: > The extra compiler checking you get with the enums is kind of nice from an > API perspective. What exactly is the problem with enum values in the > contracts? You can't declare a check against an enum value for an API. So we can't leverage swig's contract feature to ensure that Ruby, for example, doesn't try to pass a different value or type for such an API. To quote their documentation: "Regrettably, there is no automatic way to perform similar checks with enums values. Maybe in a future release." -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpfzTfHI8lGd.pgp Description: PGP signature
Re: [jira] [Commented] (PROTON-1) Provides %contract assurances in the swig layer for language bindings.
On Thu, Aug 16, 2012 at 01:08:38AM +1100, Rafael H. Schloming (JIRA) wrote: > Ok, going over this one with a less congested head, I've noticed an issue. > The %contract stuff is great for making exceptions occur, but at least for > things like checking return error codes it seems to swallow the actual reason > for the exception, e.g.: > > File "/home/rhs/bin/proton-test", line 331, in run > phase() > File "/home/rhs/proton/tests/proton_tests/messenger.py", line 93, in > testSendBogus > pn_messenger_put(self.client, msg) > RuntimeError: Contract violation: ensure: (result==0) > > > I think this is a problem given that things like, pn_messenger_send returns > an error code which indicates what has actually gone wrong, and there is a > big difference between say a PN_TIMEOUT (indicates that send has blocked for > longer than the timeout set for the messenger, not necessarily an error > per/se), and PN_STATE_ERR, (indicates the messenger is in some kind of > illegal state). Talking this over in IRC, Rafe and I agree that the contracts are likely a little over-zealous in handling error conditions. So I'm going to rework the patch to not raise exceptions on error response codes. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpqnqkiB0Hq5.pgp Description: PGP signature
