Hi Everyone, I've been struggling with some of the cross language maintenance aspects of proton for a while, and I believe we need to take some steps to improve the situation. I'm one of a tiny number of people (two possibly) who regularly commit changes to both the Java and C codebase and attempt to keep the two in sync and at feature parity. Because of this, not everyone is necessarily aware of the process, but to summarize the issue, currently there are far too many moving parts and layers of indirection involved. This is not only a significant drag on my personal productivity, but perhaps more importantly it is a significant barrier to growing the number of proton contributors in general as all those moving parts and layers of indirection need to be understood before being able to make complete contributions.
To get an idea of what's going on here I think it helps to look at what's involved in a simple change. Recently I noticed an edge case around status updates in the messenger interface. The fix involved adding another value to messenger's status enum and making a trivial logic change to make use of that value under the appropriate circumstances. You can look at the full commit here[1] if you like, but it breaks down in the following way: Changes to C implementation (.h and .c file): 8 LOC Changes to the python test shim for C: 3 LOC Changes to the Java API: 2 LOC Changes to the pure Java implementation: 6 LOC Changes to the JNI binding: 4 LOC Changes to the python test shim for Java: 3 LOC Changes to the python test suite: 2 LOC Now obviously from a personal productivity perspective it is at a minimum annoying to have to touch so many different parts in order to make even a trivial change, but setting that aside for the moment, what is really sobering about this is that each one of the above parts involve a non trivial learning curve on their own, and while it's true that only a few lines of code are needed in each area, it is necessary to understand each piece before being able to write the correct few lines of code. This presents a pretty daunting hurdle for bringing new contributors to the codebase up to the level needed to make even a trivial change like the one above. The JNI binding and the python test shim for Java both exist to serve similar purposes, namely to facilitate running a common test suite against both implementations in order to ensure common behaviour. The python test shims allow the python test suite to run against both proton-c and proton-j (via jython), and the JNI binding allows pure Java tests to be run against proton-c as well as proton-j. Currently by my count there are about 19 such tests in Java. By comparison there are about 266 tests in the python test suite. Also, judging by commits, the python test suite is growing/actively maintained, and the java system tests (the subset of java tests that are run against both implementations) are neither. On top of this the JNI binding itself has suffered significantly from bit rot. As far as I know it is not made use of outside of our own test infrastructure, and currently about 50% of the tests run against it are skipped because it is only minimally maintained when necessary to get the tests to pass. Because of all this I'm proposing that we remove the JNI binding and roll back the Java API/impl split that was (hastily) done to facilitate the test infrastructure. This should significantly simplify the development process relative to what it is now, and while there is still more learning curve than desired with the python shims, I believe this will put us in a position to improve the shims, remove duplication, and minimize the overhead associated with them, ultimately allowing us to make the codebase more transparently communicate its design and hopefully lessen the learning curve for new contributions. I would love to hear thoughts and/or alternative ideas on how to improve things. I would like to start addressing this early in the 0.7 development cycle. [1] http://svn.apache.org/r1551950
