Re: improving cross language maintainability

2014-01-08 Thread Rafael Schloming
On Tue, Jan 7, 2014 at 3:24 PM, Rob Godfrey rob.j.godf...@gmail.com wrote:

 On 20 December 2013 19:49, Rafael Schloming r...@alum.mit.edu wrote:

  On Fri, Dec 20, 2013 at 11:43 AM, Fraser Adams 
  fraser.ad...@blueyonder.co.uk wrote:
 
   I've been following this thread with interest and I guess that Rob's
   comment
  
  
   
  
   However I think that would require us to be
   organised differently with a recognition that the Engine and Messenger
  are
   conceptually separate (with Messenger depending on the Engine API, and
   having no more authority over defining/testing the engine API than any
   other client
   
  
   Struck some resonance.
  
   I think perhaps the layering is OK if you're familiar with the code
 base
   and perhaps it's more about packaging than layering, but FWIW coming
  into
   proton quite cold it seems like messenger and engine are essentially
   combined. Now I know that's not the case now from the responses to my
   how does it all hang together question of a few weeks ago, but in
 terms
   of packaging that's still kind of the case.
  
   So what I mean by that is that qpid::messaging and proton messenger are
  as
   far as has been discussed with me peer APIs - both high level that
 can
   achieve similar things albeit with different semantics, whereas engine
  is a
   lower level API.
  
   Why is it in that case that the proton library is built that contains
  both
   messenger and engine? OK, so it's convenient, but as far as I'm aware
   neither qpid::messaging nor Ted's dispatch router actually use
 messenger
  at
   all, they both use engine? That would suggest to me that engine and
   messenger probably ought to be exposed as separate libraries? (and
   presumably there's a similar position in Java where the JMS
  implementation
   is engine not messenger based - though I've not looked at the jar
   packaging).
  
   I guess (perhaps related) it just might be better if messenger was in a
   separate source tree, which I'd agree might be a faff, but would
 clarify
   the hierarchical rather than peer relationship between messenger and
  engine.
  
   So I guess that my take is that Rafael is perfectly accurate when he
 says
   that I believe they are currently layered precisely the way you
   describe that certainly seems to be the case at the logical level, but
  at
   the physical level it's an awful lot less clear of the layering -
   certainly from the perspective of a novice trying to navigate the code
  base.
  
 
  There is certainly a pragmatic element in keeping messenger and engine
  bundled the way they are. From a development perspective, having
 messenger
  in the same code base makes testing significantly easier.


 It might make it easier for you (since you have a complete view of every
 piece of the code and design) - for myself I found that it made things
 *much* harder, as tests were running through multiple layers rather than
 being isolated to a single component.  When a test failed it was entirely
 unclear which layer the failure resided in or even why the outcome should
 be as the test expeted.  Personally I'd say the testing is a strong
 argument for separation (although it means more work upfront it makes it
 easier for everyone in the long run).


I'm not suggesting messenger tests are in any way a substitute for API
tests against the engine itself, rather that are other important categories
of testing, e.g. soak testing, that would require building something that
is pretty much equivalent to what messenger already does.




  Writing a simple
  command line program to fire off a few messages in order to reproduce
 some
  scenario is quite trivial with messenger, but would involve lots of
 boiler
  plate with just the engine directly. That's something I would like to
 work
  on improving about the engine API, but it is very slow and difficult to
  make changes given the current setup I described in my original post.



  Given
  that, realistically I think if we were to pull messenger into a separate
  code base, there would be multiple sources of duplicate effort, not only
  having to duplicate/produce another non trivial multi lingual build
 system,
  but we would also need to build up a test harness that duplicates a good
  chunk of what messenger already does.


 Every other project that wants to use the Proton engine will also be
 writing tests for their use of the API. And they'll have different use
 cases and different pattern that may not be something supported by
 Messenger.

 Sure, it's great that Messenger tests will implicitly test the Engine, but
 they are not a substitute for Engine tests.


As I said above, I don't believe we have been using them as a substitute
for engine tests, rather they are a tool for discovering more bugs. Just
having shallow test coverage of an API doesn't provide much confidence that
the API will work in real world scenarios, e.g. early engine tests verified
that sending/receiving 1, 2, or even 10 

Re: improving cross language maintainability

2014-01-07 Thread Rafael Schloming
So far there has been some good discussion on this thread. I can certainly
appreciate the frustration that has been expressed, specifically around
under defined APIs, and confusion regarding the various components of
proton. I believe there are steps we can take to improve both of those
situations. I've created additional components in JIRA for tracking changes
across bindings, and I'm currently working on some content describing
proton's overall architecture and how messenger and engine fit within it
and relate to each other.

That said, no amount of documentation or JIRA usage will help with the
issues I raised in my original post. In particular making even small code
changes requires touching too many different parts: (C code, Java impl,
Java API, JNI binding, C shim, Java shim, and python tests). As a first
step towards improving this I'd like to reiterate my proposal of removing
the JNI binding for the following reasons:

  - It is experiencing significant bit rot.
  - It is not currently used outside of testing.
  - It only provides a minimal amount of additional coverage (some 19 or so
tests as compared to the 266 tests already in the shared python test suite).
  - It is a significant maintenance burden as it covers the entire engine
API rather than just messenger.
  - It is unlikely to ever be of production value, because a JNI layer
designed for testing an existing C library has fundamentally different
requirements than a JNI layer designed to improve the performance of an
existing Java library.

--Rafael


Re: improving cross language maintainability

2014-01-07 Thread Rob Godfrey
On 20 December 2013 19:49, Rafael Schloming r...@alum.mit.edu wrote:

 On Fri, Dec 20, 2013 at 11:43 AM, Fraser Adams 
 fraser.ad...@blueyonder.co.uk wrote:

  I've been following this thread with interest and I guess that Rob's
  comment
 
 
  
 
  However I think that would require us to be
  organised differently with a recognition that the Engine and Messenger
 are
  conceptually separate (with Messenger depending on the Engine API, and
  having no more authority over defining/testing the engine API than any
  other client
  
 
  Struck some resonance.
 
  I think perhaps the layering is OK if you're familiar with the code base
  and perhaps it's more about packaging than layering, but FWIW coming
 into
  proton quite cold it seems like messenger and engine are essentially
  combined. Now I know that's not the case now from the responses to my
  how does it all hang together question of a few weeks ago, but in terms
  of packaging that's still kind of the case.
 
  So what I mean by that is that qpid::messaging and proton messenger are
 as
  far as has been discussed with me peer APIs - both high level that can
  achieve similar things albeit with different semantics, whereas engine
 is a
  lower level API.
 
  Why is it in that case that the proton library is built that contains
 both
  messenger and engine? OK, so it's convenient, but as far as I'm aware
  neither qpid::messaging nor Ted's dispatch router actually use messenger
 at
  all, they both use engine? That would suggest to me that engine and
  messenger probably ought to be exposed as separate libraries? (and
  presumably there's a similar position in Java where the JMS
 implementation
  is engine not messenger based - though I've not looked at the jar
  packaging).
 
  I guess (perhaps related) it just might be better if messenger was in a
  separate source tree, which I'd agree might be a faff, but would clarify
  the hierarchical rather than peer relationship between messenger and
 engine.
 
  So I guess that my take is that Rafael is perfectly accurate when he says
  that I believe they are currently layered precisely the way you
  describe that certainly seems to be the case at the logical level, but
 at
  the physical level it's an awful lot less clear of the layering -
  certainly from the perspective of a novice trying to navigate the code
 base.
 

 There is certainly a pragmatic element in keeping messenger and engine
 bundled the way they are. From a development perspective, having messenger
 in the same code base makes testing significantly easier.


It might make it easier for you (since you have a complete view of every
piece of the code and design) - for myself I found that it made things
*much* harder, as tests were running through multiple layers rather than
being isolated to a single component.  When a test failed it was entirely
unclear which layer the failure resided in or even why the outcome should
be as the test expeted.  Personally I'd say the testing is a strong
argument for separation (although it means more work upfront it makes it
easier for everyone in the long run).


 Writing a simple
 command line program to fire off a few messages in order to reproduce some
 scenario is quite trivial with messenger, but would involve lots of boiler
 plate with just the engine directly. That's something I would like to work
 on improving about the engine API, but it is very slow and difficult to
 make changes given the current setup I described in my original post.



 Given
 that, realistically I think if we were to pull messenger into a separate
 code base, there would be multiple sources of duplicate effort, not only
 having to duplicate/produce another non trivial multi lingual build system,
 but we would also need to build up a test harness that duplicates a good
 chunk of what messenger already does.


Every other project that wants to use the Proton engine will also be
writing tests for their use of the API. And they'll have different use
cases and different pattern that may not be something supported by
Messenger.

Sure, it's great that Messenger tests will implicitly test the Engine, but
they are not a substitute for Engine tests.


 Granted a chunk of that is
 boilerplate that should be somehow refactored into the engine API proper so
 that it is easier to use in general, and if this were to happen such a
 split would probably be more feasible, but that kind of refactoring is
 really prohibitively expensive given all the different shims and bindings
 that would be impacted.


And this points to an issue, with the focus on Messenger the Engine API is
not being enhanced... and it needs people and effort focussed on it.
Messenger should be a client of the Engine API, the Engine shouldn't be a
slave to Messenger.  Other users (whether in Qpid or not) should have equal
influence of the Engine API design, and equal ability to try to shape it.
Right now I find the Engine unattaractive to try to use within the Qpid
Java 

Re: improving cross language maintainability

2013-12-23 Thread Ted Ross


On 12/20/2013 01:49 PM, Rafael Schloming wrote:

On Fri, Dec 20, 2013 at 11:43 AM, Fraser Adams 
fraser.ad...@blueyonder.co.uk wrote:


I've been following this thread with interest and I guess that Rob's
comment




However I think that would require us to be
organised differently with a recognition that the Engine and Messenger are
conceptually separate (with Messenger depending on the Engine API, and
having no more authority over defining/testing the engine API than any
other client


Struck some resonance.

I think perhaps the layering is OK if you're familiar with the code base
and perhaps it's more about packaging than layering, but FWIW coming into
proton quite cold it seems like messenger and engine are essentially
combined. Now I know that's not the case now from the responses to my
how does it all hang together question of a few weeks ago, but in terms
of packaging that's still kind of the case.

So what I mean by that is that qpid::messaging and proton messenger are as
far as has been discussed with me peer APIs - both high level that can
achieve similar things albeit with different semantics, whereas engine is a
lower level API.

Why is it in that case that the proton library is built that contains both
messenger and engine? OK, so it's convenient, but as far as I'm aware
neither qpid::messaging nor Ted's dispatch router actually use messenger at
all, they both use engine? That would suggest to me that engine and
messenger probably ought to be exposed as separate libraries? (and
presumably there's a similar position in Java where the JMS implementation
is engine not messenger based - though I've not looked at the jar
packaging).

I guess (perhaps related) it just might be better if messenger was in a
separate source tree, which I'd agree might be a faff, but would clarify
the hierarchical rather than peer relationship between messenger and engine.

So I guess that my take is that Rafael is perfectly accurate when he says
that I believe they are currently layered precisely the way you
describe that certainly seems to be the case at the logical level, but at
the physical level it's an awful lot less clear of the layering -
certainly from the perspective of a novice trying to navigate the code base.


There is certainly a pragmatic element in keeping messenger and engine
bundled the way they are. From a development perspective, having messenger
in the same code base makes testing significantly easier. Writing a simple
command line program to fire off a few messages in order to reproduce some
scenario is quite trivial with messenger, but would involve lots of boiler
plate with just the engine directly. That's something I would like to work
on improving about the engine API, but it is very slow and difficult to
make changes given the current setup I described in my original post. Given
that, realistically I think if we were to pull messenger into a separate
code base, there would be multiple sources of duplicate effort, not only
having to duplicate/produce another non trivial multi lingual build system,
but we would also need to build up a test harness that duplicates a good
chunk of what messenger already does. Granted a chunk of that is
boilerplate that should be somehow refactored into the engine API proper so
that it is easier to use in general, and if this were to happen such a
split would probably be more feasible, but that kind of refactoring is
really prohibitively expensive given all the different shims and bindings
that would be impacted.

I'd also point out that from a user perspective I think there is some
synergy in bundling a higher level tool together with the engine. It is
very rare that you are going to want to embed the engine in something and
not want some convenient high level way to send that thing a message. Case
in point, Ted's dispatch router proper doesn't actually use messenger as
you say, however he ships it with several command line scripts that do use
messenger. Splitting them up would result in both him and his users having
to deal with an extra dependency.


This is true.  The system tests in the Dispatch test suite are also 
built on Messenger.  I think Messenger is sufficiently lightweight that 
there's no issue with it being bundled with Engine.  The separation I'm 
more interested in is Messenger+Engine and Driver.  I would like to see 
it be easier to use alternate drivers for Proton-based software.





I'd have to agree also with Rob about the horizontal scaling aspects of
lack of comments and documentation, particularly with respect to engine.
I'm currently trying to get a JavaScript_messenger_  working because that
seems tractable given the examples and documentation around messenger
(though as you know even there I've had a fair few complications) my
personal preference/familiarity is however around the JMS/qpid::messaging
API and I'd ultimately quite like to have a JavaScript binding for that,
but at this stage I wouldn't know where to begin and I'd 

Re: improving cross language maintainability

2013-12-20 Thread Rafael Schloming
On Fri, Dec 20, 2013 at 5:46 AM, Rob Godfrey rob.j.godf...@gmail.comwrote:

 Since I have pretty much given up on trying to work on proton, I'm not sure
 that my opinion counts for much anymore... however I suppose I'm a little
 curious how the JNI binding differs in extra work vs. PHP or Ruby or
 such. Similarly why the API distinction is hasty when again it should
 (modulo some language idiom changes) reflect the API that the C is
 presenting to Ruby/PHP/Python/etc.


That's a really good question. One obvious factor is that the JNI bindings
cover the full surface of the engine API, whereas the PHP and Ruby bindings
only cover the messenger API, the latter being much smaller. The other
factor is that we have a different workflow with how we maintain the
scripting language bindings. From a design perspective each of those
bindings are more tightly coupled to the C code since that is the only
implementation they use, but from a workflow perspective each one is
actually more independent than the Java binding. When we add a feature to
the C messenger implementation we don't try to add it simultaneously across
all the bindings. It generally goes into python because that is how we
test, and it goes into perl and ruby pretty quickly after that because
Darryl is good about keeping those up to date, but, e.g., in the case of
php, we haven't tried to keep the binding at full feature parity. Finally,
the JNI binding is a bit of a different animal from the others in that it
is not just performing a simple wrapping of the C API and directly exposing
that to Java. It is adapting between the C API underneath it and the Java
API above it.

As for the hastiness, perhaps that was a poor choice of words. The problem
with the Java API/impl split is that the API was produced from an existing
implementation after the fact in order to facilitate testing. As such it
isn't well suited to be implemented by two independent implementations,
particularly one that is JNI based.



 To me the fact that APIs aren't defined but that instead the only way of
 knowing what a particular call should do seems to be 1) look at the code in
 C and, if it's not clear, then 2) ask Rafi; is why I no longer feel it
 productive to try to contribute.  As such, it seems that this change is
 more about vertical scaling (making it easier for Rafi, and those sitting
 very near him, to work more productively), than horizontal scaling
 (allowing people who are not Rafi or with easy access to him to work on
 it).


Really there are two separate issues here. You're talking about conceptual
barriers to contribution, and I'm talking about mechanical barriers. Both
are barriers to contribution, and both prevent horizontal scaling. Fixing
some of the mechanical barriers to contribution does happen to also improve
vertical scaling, but as I said in my original post, I think the more
important factor is making it easier/possible for people to make complete
contributions.

FWIW, I agree we need lots of additional work around API definition and
documentation. Having more time for that is also one of the reasons I would
like to reduce some of the overhead involved in the current development
process.



 Overall I don't care very much about Messenger in Java (either pure Java or
 JNI), but I do care about the Engine.  When the Proton Engine was first
 discussed it was always said that the C code should be able to be called
 from within Java as an alternative to a pure Java impl. - has this changed?


I'm fine with that as a goal, however I can't say it's a priority for me.
More importantly, as I've said before, I don't believe the use of JNI in
production is at all compatible with the use of JNI for testing purposes.
The current JNI binding exposes each part of the engine API in very fine
detail. This what you want for testing, so you can exercise every part of
the C API from Java. This is probably exactly the opposite of what you'd
want for production use though. I expect you'd want to minimize the number
of times you cross the C/Java boundary and just tie into the engine API at
key points that are shifting a lot of bytes around.



 Going back to the horizontal vs. vertical scaling... If the project were
 structured differently I would be very happy to contribute to the engine
 side of things in Java.  However I think that would require us to be
 organised differently with a recognition that the Engine and Messenger are
 conceptually separate (with Messenger depending on the Engine API, and
 having no more authority over defining/testing the engine API than any
 other client - such as the C++ broker, ActiveMQ, or the upcoming JMS clent,
 etc.). As such I would rather see us fix the definition of the API (if you
 believe it to be hasty) rather than simply try to remove any notion of
 their being an API which is distinct from the implementation.


I'm not sure exactly what you're asking for here with respect to Messenger
and Engine. I believe they are currently 

Re: improving cross language maintainability

2013-12-20 Thread Darryl L. Pierce
On Thu, Dec 19, 2013 at 04:07:38PM -0500, Rafael Schloming wrote:
 On Thu, Dec 19, 2013 at 9:01 AM, Darryl L. Pierce dpie...@redhat.comwrote:
  On Thu, Dec 19, 2013 at 08:49:59AM -0500, Rafael Schloming wrote:
  snip
   I would love to hear thoughts and/or alternative ideas on how to improve
   things. I would like to start addressing this early in the 0.7
  development
   cycle.
 
  In a similar way, I'm trying to keep our Ruby and Perl bindings in
  parity as best I can with what's going on in the C and Python code. Can
  we use JIRA to create umbrella tasks for when new features are added,
  with subtasks that are binding specific? Or if there's a change to the C
  code that would require a change in the bindings, have the C code be the
  top JIRA and the language bindings be subtasks to that? That way I
  wouldn't need to look through commits to see what's changed in C to know
  what should be added to the other languages.
 
  Also, could we add a component for each of the language bindings? It
  doesn't feel right to add a JIRA for something in Ruby that's at the
  Ruby level but have its component be proton-c.
 
 
 It certainly makes sense to make it as easy as possible to track what is
 going on, and I can see how that would help keep bindings up to date where
 there is interest and resources to do so. However we do this though, I
 don't want to just brainlessly duplicate each C jira across every binding
 (not that you're necessarily suggesting this).
 
 The problem I have with that approach is that there isn't equivalent
 interest/resources associated with each binding, so e.g. if we were to make
 every JIRA a full umbrella that depends on sub tasks for each binding we
 would continually accumulate php jiras that never end up getting closed off
 because we don't keep php as up to date as the other bindings, and this in
 turn would cause the umbrella JIRAs to never get closed off. Jira is really
 a task oriented tool, and I think JIRAs should really only be created when
 there is intention/interest to actually complete the task they represent,
 otherwise they usually end up being noise/clutter that will eventually be
 irrelevant and out of date. I'd suggest that perhaps a more document
 oriented description of those features for which we are trying to keep
 parity would possibly be helpful.

Definitely. Maybe a wiki page that lists the set of feeatures being done
in C can be used as a backlog for the other language bindingss. Not
necessarily a full blown, prioritized scrum backlog. But minimally a
central list of what's done in C/Python that can be used as a guide?

 All that said, I'm certainly sure we can improve our usage of JIRA, and
 I've gone ahead and added ruby-binding, python-binding, perl-binding, and
 php-binding components as you suggest.

Thank you! :D

-- 
Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
Delivering value year after year.
Red Hat ranks #1 in value among software vendors.
http://www.redhat.com/promo/vendor/



pgpdNTqveVHxm.pgp
Description: PGP signature


Re: improving cross language maintainability

2013-12-20 Thread Fraser Adams

I've been following this thread with interest and I guess that Rob's comment



However I think that would require us to be
organised differently with a recognition that the Engine and Messenger are
conceptually separate (with Messenger depending on the Engine API, and
having no more authority over defining/testing the engine API than any
other client


Struck some resonance.

I think perhaps the layering is OK if you're familiar with the code base and perhaps it's more about 
packaging than layering, but FWIW coming into proton quite cold it seems like messenger and 
engine are essentially combined. Now I know that's not the case now from the responses to my how does 
it all hang together question of a few weeks ago, but in terms of packaging that's still kind of the case.

So what I mean by that is that qpid::messaging and proton messenger are as far as has 
been discussed with me peer APIs - both high level that can achieve similar 
things albeit with different semantics, whereas engine is a lower level API.

Why is it in that case that the proton library is built that contains both 
messenger and engine? OK, so it's convenient, but as far as I'm aware neither 
qpid::messaging nor Ted's dispatch router actually use messenger at all, they 
both use engine? That would suggest to me that engine and messenger probably 
ought to be exposed as separate libraries? (and presumably there's a similar 
position in Java where the JMS implementation is engine not messenger based - 
though I've not looked at the jar packaging).

I guess (perhaps related) it just might be better if messenger was in a 
separate source tree, which I'd agree might be a faff, but would clarify the 
hierarchical rather than peer relationship between messenger and engine.

So I guess that my take is that Rafael is perfectly accurate when he says that 
I believe they are currently layered precisely the way you
describe that certainly seems to be the case at the logical level, but at the 
physical level it's an awful lot less clear of the layering - certainly from the 
perspective of a novice trying to navigate the code base.

I'd have to agree also with Rob about the horizontal scaling aspects of lack of comments and 
documentation, particularly with respect to engine. I'm currently trying to get a 
JavaScript_messenger_  working because that seems tractable given the examples and documentation 
around messenger (though as you know even there I've had a fair few complications) my personal 
preference/familiarity is however around the JMS/qpid::messaging API and I'd ultimately quite like 
to have a JavaScript binding for that, but at this stage I wouldn't know where to begin 
and I'd probably end up reverse engineering the work Gordon has done with 
qpid::messaging and/or Rob and co. are doing with the new JMS implementation.

I'd be interested to hear from Gordon and Ted about how they went about 
building capability on top of the engine API.

I hope I'm not coming across as over-critical - email is a bit rubbish at 
conveying emotion :-) I guess that I just want to add my 2p from the 
perspective of someone coming in new with no previous exposure to the code 
base, which is hopefully a fairly objective position.
Best regards,
Frase


On 20/12/13 15:57, Rafael Schloming wrote:

On Fri, Dec 20, 2013 at 5:46 AM, Rob Godfrey rob.j.godf...@gmail.comwrote:


Since I have pretty much given up on trying to work on proton, I'm not sure
that my opinion counts for much anymore... however I suppose I'm a little
curious how the JNI binding differs in extra work vs. PHP or Ruby or
such. Similarly why the API distinction is hasty when again it should
(modulo some language idiom changes) reflect the API that the C is
presenting to Ruby/PHP/Python/etc.


That's a really good question. One obvious factor is that the JNI bindings
cover the full surface of the engine API, whereas the PHP and Ruby bindings
only cover the messenger API, the latter being much smaller. The other
factor is that we have a different workflow with how we maintain the
scripting language bindings. From a design perspective each of those
bindings are more tightly coupled to the C code since that is the only
implementation they use, but from a workflow perspective each one is
actually more independent than the Java binding. When we add a feature to
the C messenger implementation we don't try to add it simultaneously across
all the bindings. It generally goes into python because that is how we
test, and it goes into perl and ruby pretty quickly after that because
Darryl is good about keeping those up to date, but, e.g., in the case of
php, we haven't tried to keep the binding at full feature parity. Finally,
the JNI binding is a bit of a different animal from the others in that it
is not just performing a simple wrapping of the C API and directly exposing
that to Java. It is adapting between the C API underneath it and the Java
API above it.

As for the hastiness, perhaps that was a poor 

Re: improving cross language maintainability

2013-12-20 Thread Rafael Schloming
On Fri, Dec 20, 2013 at 11:43 AM, Fraser Adams 
fraser.ad...@blueyonder.co.uk wrote:

 I've been following this thread with interest and I guess that Rob's
 comment


 

 However I think that would require us to be
 organised differently with a recognition that the Engine and Messenger are
 conceptually separate (with Messenger depending on the Engine API, and
 having no more authority over defining/testing the engine API than any
 other client
 

 Struck some resonance.

 I think perhaps the layering is OK if you're familiar with the code base
 and perhaps it's more about packaging than layering, but FWIW coming into
 proton quite cold it seems like messenger and engine are essentially
 combined. Now I know that's not the case now from the responses to my
 how does it all hang together question of a few weeks ago, but in terms
 of packaging that's still kind of the case.

 So what I mean by that is that qpid::messaging and proton messenger are as
 far as has been discussed with me peer APIs - both high level that can
 achieve similar things albeit with different semantics, whereas engine is a
 lower level API.

 Why is it in that case that the proton library is built that contains both
 messenger and engine? OK, so it's convenient, but as far as I'm aware
 neither qpid::messaging nor Ted's dispatch router actually use messenger at
 all, they both use engine? That would suggest to me that engine and
 messenger probably ought to be exposed as separate libraries? (and
 presumably there's a similar position in Java where the JMS implementation
 is engine not messenger based - though I've not looked at the jar
 packaging).

 I guess (perhaps related) it just might be better if messenger was in a
 separate source tree, which I'd agree might be a faff, but would clarify
 the hierarchical rather than peer relationship between messenger and engine.

 So I guess that my take is that Rafael is perfectly accurate when he says
 that I believe they are currently layered precisely the way you
 describe that certainly seems to be the case at the logical level, but at
 the physical level it's an awful lot less clear of the layering -
 certainly from the perspective of a novice trying to navigate the code base.


There is certainly a pragmatic element in keeping messenger and engine
bundled the way they are. From a development perspective, having messenger
in the same code base makes testing significantly easier. Writing a simple
command line program to fire off a few messages in order to reproduce some
scenario is quite trivial with messenger, but would involve lots of boiler
plate with just the engine directly. That's something I would like to work
on improving about the engine API, but it is very slow and difficult to
make changes given the current setup I described in my original post. Given
that, realistically I think if we were to pull messenger into a separate
code base, there would be multiple sources of duplicate effort, not only
having to duplicate/produce another non trivial multi lingual build system,
but we would also need to build up a test harness that duplicates a good
chunk of what messenger already does. Granted a chunk of that is
boilerplate that should be somehow refactored into the engine API proper so
that it is easier to use in general, and if this were to happen such a
split would probably be more feasible, but that kind of refactoring is
really prohibitively expensive given all the different shims and bindings
that would be impacted.

I'd also point out that from a user perspective I think there is some
synergy in bundling a higher level tool together with the engine. It is
very rare that you are going to want to embed the engine in something and
not want some convenient high level way to send that thing a message. Case
in point, Ted's dispatch router proper doesn't actually use messenger as
you say, however he ships it with several command line scripts that do use
messenger. Splitting them up would result in both him and his users having
to deal with an extra dependency.


 I'd have to agree also with Rob about the horizontal scaling aspects of
 lack of comments and documentation, particularly with respect to engine.
 I'm currently trying to get a JavaScript_messenger_  working because that
 seems tractable given the examples and documentation around messenger
 (though as you know even there I've had a fair few complications) my
 personal preference/familiarity is however around the JMS/qpid::messaging
 API and I'd ultimately quite like to have a JavaScript binding for that,
 but at this stage I wouldn't know where to begin and I'd probably end up
 reverse engineering the work Gordon has done with qpid::messaging and/or
 Rob and co. are doing with the new JMS implementation.

 I'd be interested to hear from Gordon and Ted about how they went about
 building capability on top of the engine API.

 I hope I'm not coming across as over-critical - email is a bit rubbish at
 conveying emotion :-) I 

Re: improving cross language maintainability

2013-12-19 Thread Darryl L. Pierce
On Thu, Dec 19, 2013 at 08:49:59AM -0500, Rafael Schloming wrote:
snip 
 I would love to hear thoughts and/or alternative ideas on how to improve
 things. I would like to start addressing this early in the 0.7 development
 cycle.

In a similar way, I'm trying to keep our Ruby and Perl bindings in
parity as best I can with what's going on in the C and Python code. Can
we use JIRA to create umbrella tasks for when new features are added,
with subtasks that are binding specific? Or if there's a change to the C
code that would require a change in the bindings, have the C code be the
top JIRA and the language bindings be subtasks to that? That way I
wouldn't need to look through commits to see what's changed in C to know
what should be added to the other languages.

Also, could we add a component for each of the language bindings? It
doesn't feel right to add a JIRA for something in Ruby that's at the
Ruby level but have its component be proton-c.

-- 
Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
Delivering value year after year.
Red Hat ranks #1 in value among software vendors.
http://www.redhat.com/promo/vendor/



pgp5KU9SF4aFs.pgp
Description: PGP signature


Re: improving cross language maintainability

2013-12-19 Thread Rafael Schloming
On Thu, Dec 19, 2013 at 9:01 AM, Darryl L. Pierce dpie...@redhat.comwrote:

 On Thu, Dec 19, 2013 at 08:49:59AM -0500, Rafael Schloming wrote:
 snip
  I would love to hear thoughts and/or alternative ideas on how to improve
  things. I would like to start addressing this early in the 0.7
 development
  cycle.

 In a similar way, I'm trying to keep our Ruby and Perl bindings in
 parity as best I can with what's going on in the C and Python code. Can
 we use JIRA to create umbrella tasks for when new features are added,
 with subtasks that are binding specific? Or if there's a change to the C
 code that would require a change in the bindings, have the C code be the
 top JIRA and the language bindings be subtasks to that? That way I
 wouldn't need to look through commits to see what's changed in C to know
 what should be added to the other languages.

 Also, could we add a component for each of the language bindings? It
 doesn't feel right to add a JIRA for something in Ruby that's at the
 Ruby level but have its component be proton-c.


It certainly makes sense to make it as easy as possible to track what is
going on, and I can see how that would help keep bindings up to date where
there is interest and resources to do so. However we do this though, I
don't want to just brainlessly duplicate each C jira across every binding
(not that you're necessarily suggesting this).

The problem I have with that approach is that there isn't equivalent
interest/resources associated with each binding, so e.g. if we were to make
every JIRA a full umbrella that depends on sub tasks for each binding we
would continually accumulate php jiras that never end up getting closed off
because we don't keep php as up to date as the other bindings, and this in
turn would cause the umbrella JIRAs to never get closed off. Jira is really
a task oriented tool, and I think JIRAs should really only be created when
there is intention/interest to actually complete the task they represent,
otherwise they usually end up being noise/clutter that will eventually be
irrelevant and out of date. I'd suggest that perhaps a more document
oriented description of those features for which we are trying to keep
parity would possibly be helpful.

All that said, I'm certainly sure we can improve our usage of JIRA, and
I've gone ahead and added ruby-binding, python-binding, perl-binding, and
php-binding components as you suggest.

--Rafael