I worked with Keith on this proposal so I should state up front that I'm
not coming to this debate from a neutral standpoint.

Hopefully we can find a solution that is acceptable to everyone.  To this
end, we listed our understanding of the requirements on
https://issues.apache.org/jira/browse/PROTON-194.  I'm hoping that this
discussion will allow us to clarify our requirements, such that the best
technical solution naturally follows.

I've added some comments in-line below...

On 18 January 2013 19:29, Rafael Schloming <r...@alum.mit.edu> wrote:

> On Fri, Jan 18, 2013 at 11:17 AM, Keith W <keith.w...@gmail.com> wrote:
>
> > We are currently in the process of implementing the proton-jni binding
> > for the proton-c library that implements the Java Proton-API, allow
> > Java users to choose the C based proton stack if they wish. This work
> > is being performed on the jni-branch under PROTON-192 (for the JNI
> > work) and PROTON-194 (for the build system changes).
> >
> > Currently, Proton has two independent build systems: one for the
> > proton-c and its ruby/perl/python/php bindings (based on Cmake/Make),
> > and second a separate build system for proton-j (based on Maven).  As
> > proton-jni will cut across both technology areas, non trivial changes
> > are required to both build systems.
> >
> > The nub of the problem is the sharing of the Java Proton-API between
> > both proton-c and proton-j trees. Solutions based on svn-external and
> > a simple tree copy have been considered and discussed at length on
> > conference calls.  We have identified drawbacks in both solutions.
> >
>
> To be honest I don't think we've sufficiently explored the "copy" option.
> While its true there were a lot of hypothetical issues thrown around on the
> calls, many of them have quite reasonable solutions that may well be less
> work than the alternatives.
>
> In my experience, maintaining two copies of any code is usually a bad
thing.  However, I try to be open minded so I agree that it's worth
exploring this option.  I'd be interested to hear your opinion on (a) the
scenarios when it would be acceptable for these two copies to diverge and
(b) the mechanism you're envisaging for achieving convergence.  I imagine
there are both technical and "process" dimensions to making this work.

>
> > This email proposes another solution. The hope is that this proposal
> > can be developed on list into a solution that is acceptable to all.
> >
> > Proposal:
> >
> > Move the Java Proton-API to the top level so that it can be shared
> > simply and conveniently by both proton-j and proton-c.
> >
> > * Maven builds the proton-api JAR to a well known location
> > * Cmake/make builds proton-c and all bindings including java. As the
> > building of the java binding requires the Java Proton API, it is
> > optional and only takes place if proton-api has been previously
> > created by Maven (or found by other means).
> > * Maven builds of proton-j
> > * Maven runs the system tests against either proton-c or proton-j. The
> > system tests are currently written in Python but are being augmented
> > with new ones written in Java.
> >
> > Proposed Directory Structure:
> >
> > proton
> > |-- release.sh/bat                  # Builds, tests and packages
> > proton-c and proton-j
> > |-- pom.xml
> > |
> > |-- proton-api                      # Java Proton-API
> > |   |-- pom.xml                     # Will create proton-api.jar at a
> > well known location in tree
> > |       `-- main
> > |
> > |-- proton-c                        # Proton-C and Proton-C bindings
> > |   |-- CMakeLists.txt
> > |   `-- bindings
> > |       |-- CMakeLists.txt
> > |       `-- java
> > |           |-- CMakeLists.txt
> > |           `-- jni
> > |               `-- CMakeLists.txt  # Creates proton-jni.jar using
> > proton-api.jar from a well known
> > |                                   # location in tree or skip if jar
> > cannot be found
> > |
> > |-- proton-j                        # Proton-J
> > |   |-- pom.xml                     # Creates proton-j.jar using
> > proton-api.jar (found via Maven)
> > |   `-- src
> > |       `-- main
> > |
> > `-- tests                           # Python and Java based system
> > tests that test equally Proton-C and
> >     |                               # Proton-J.
> >     |-- pom.xml
> >     `-- src
> >         `-- test
> >
> > Use cases:
> >
> > usecase #1 - Proton-C Developer exclusively focused on Proton-C
> >
> > This developer may choose to check out the proton-c subtree.  The
> > build tool set remains unchanged from today i.e. cmake and make.  By
> > default, all bindings will be built expect for the java bindings (as
> > Cmake would fail to find the proton-api.jar).  For flexibility, we
> > would include option to have cmake search another directory allowing
> > proton-api.jar to be found in non-standard locations.
> >
> > usecase #2 - Proton-C Developer who wishes to run all system tests
> >
> > This developer must check out the complete proton tree.  The build
> > tool set now includes maven in order to build the proton-api and run
> > the complete system test suite.
> >
> > Typical commands used by this developer would be:
> > mvn -f proton-api/pom.xml install  # build and install proton-api to
> > the well known location
> > <build proton-c>
> > mvn test                           # run all the system tests against
> > proton-c.
> >
> > usecase #3 - Proton-J Developer
> >
> > This developer must check out the complete proton tree.   The build
> > tool set is maven.
> >
> > mvn test -Pproton-j                # build all and run the system
> > tests against proton-j (the -P switch to specifies the Maven profile
> > to use)
> >
> > usecase #4 - Release manager
> >
> > The release manager must check out the complete proton tree.  The tool
> > set will be maven, cmake, make in order to build and test all
> > artefacts. A release script could be added to make this a single step.
> >
> > release.sh/bat
> >
>
> This results in something that is quite awkward for the C build. For one
> thing I'm not sure an svn export of the proton-c directory would be
> considered releasable under this scheme as it would include the java
> binding, but not the source code necessary to build it, and apache policy
> requires releases to include full source code. Regardless it would no
> longer be a useful/sensible artifact to end-users since they couldn't
> actually build the java binding.
>

This doesn't seem like an insurmountable problem, though admittedly I am
less familiar with the proton-c release process than you.  A couple of
suggestions follow.

1. I had imagined that a user wanting to use proton-c's would simply treat
proton-api.jar as an optional build dependency (optional in the sense that
it's only required if you want to build the JNI binding).  I think it is
reasonable for them to retrieve this separately, e.g. by downloading it
from the Apache Maven repo, or building it if they wish.  Java developers
are unlikely to balk at installing libraries in their local Maven repo.

2. You seem to be implying that an svn export of the proton-c sub-folder
needs to be completely self-contained, i.e. it would necessarily contain
the proton-api code.  Is this really true?  I could be persuaded that a
proton-c *release* needs to be self-contained.  If this really is the case,
could we delay copying the proton-api code into the proton-c tree until
build time?  We wouldn't commit that duplicated code to SVN, but we would
include it in the released source tarball.

Of the two suggestions above I favour the first one, but its feasibility
depends on what our requirements really are.


> It also has the drawback of adding a maven dependency to building the java
> binding. The way this is done on the jni branch now where the java API code
> is built with cmake actually has some significant advantages since the java
> binding would still be buildable on platforms that don't have a
> sufficiently recent maven to build the Java bits (e.g. RHEL and Debian).
>

I believe it would be rare for a team to want to build the JNI bindings but
to be unable to access a reasonably recent version of Maven.  If proton-j
requires an unreasonably recent Maven version then that's a separate
problem that we need to address anyway.
Out of interest, what is the problem on RHEL and Debian?


> --Rafael
>

Phil

Reply via email to