Thanks for your reply, Niels. Sorry to let this go so long without responding to your points. There was a holiday in the US and I was off for a bit.
For CI integration, we all agree that it needs to happen. I think splitting the implementations should happen before CI integration because it will be very different depending on whether or not we choose to split. One of the benefits of splitting is that we get easier CI and better testing because it is much easier to set up more tests. It's trivial to add another interpreter version to the environment in Travis, but not so much in Docker. I understand the concern about the effect splitting will have on the unmaintained implementations, but I think it is a mistake to avoid splitting because of it. Implementations like PHP are already unmaintained, and docker allows us to freeze the environment so we further avoid updating it. It isn't helpful to continue releasing the same code under a new version numbers, which is all we're currently doing. The downside of keeping all of the implementations together is that we are losing contributions. I haven't seen any changes in months on the JS implementation, but the outside repository has commits from 2 days ago. We aren't able to release fast enough to keep active contributors interested. That's a long-term problem that I think we need to address, and the best way I can think of is to make it so fast-moving implementations can improve without being held back by the unmaintained ones. I'm not suggesting we completely abandon the unmaintained implementations. I think we should set them up like the active implementations with CI so that if people want to contribute, they are easy to revive. I like your idea to have Java and the spec be co-located and to consider Java a reference implementation. Some of the others use Java's CLI for testing, so that would work well. rb On Thu, Nov 24, 2016 at 5:37 AM, Niels Basjes <ni...@basjes.nl> wrote: > Hi, > > Regarding the CI integration (like Travis) I think this is something we > should do anyway. So +1 on that one. > I would like to see it set up that > - any (github) merge request is built automatically. > - any commit to master is built automatically AND (if successful) pushed to > a maven repo as SNAPSHOT. > > About splitting up the project per language: I share the concern that some > languages will start to fall behind. > Also: > Assume have a separate "project" that manages the specification and the > test cases that all languages must be able to handle/read/write in order to > classify as "Follows Avro specification version X". With that we can > release a new version of the <insert language here> libraries independently > from the others. > > A slightly alternative route would be do make Java the 'reference > implementation' by combining Spec&Java. > That way the specification and java would be a single project and the > others would be separate. > The effect is then that with a new release of the Java the version of the > 'spec' would be updated too and we will have testing libraries available to > creating test cases. > Al in all ... not too different from what we now have. > > I think for now we should implement the CI stuff (preferable like I > indicated above). > My instinct tells me that with that in place any 'build breaking change' in > 'any language' will be caught very quickly and should not be a problem. > > Niels Basjes > > P.S. The CI build should do as many of the things currently in the build.sh > scripts. This includes things like running rat. > > > On Thu, Nov 24, 2016 at 12:10 AM, Ryan Blue <rb...@netflix.com.invalid> > wrote: > > > Simon brought up compatibility as well. This is an existing problem and > > right now we have some interop tests where some languages generate data > and > > then read all of the data from other languages. I don't think this > problem > > changes much if we move to separate releases. > > > > We can still run these tests, but could change them so that we update > just > > one version of a particular language to pull in an RC and use released > > versions of other implementations. That could make it easier to spot > > problems because only one language is changing at a time. > > > > Of course, just having these tests doesn't mean they are kept up to date. > > Ruby's snappy implementation was incompatible with the others until > > recently. I think that overall, splitting the projects won't result in a > > noticeable change in interoperability. > > > > rb > > > > On Mon, Nov 21, 2016 at 10:20 AM, Zoltan Ivanfi <z...@cloudera.com> wrote: > > > > > Hi, > > > > > > A question just came to my mind: How would inter-language compatibility > > > work after the split? How will we now which versions of the different > > > language bindings support the same features and allow bidirectional > > > exchange of data? > > > > > > Or is it an already existing problem? I don't know too much about the > > > language bindings you mentioned; if some of them have not been worked > on > > > recently, does this mean that they are lacking in support for some > > features > > > of the specification? If I use libraries from the same release of Avro > in > > > software components of different languages, are they not guaranteed to > be > > > compatible with each other? > > > > > > Thanks, > > > > > > Zoltan > > > > > > On Sat, Nov 19, 2016 at 10:03 PM Ryan Blue <rb...@netflix.com.invalid> > > > wrote: > > > > > > > This is in response to Tom's concerns: > > > > > > > > > When I did the 1.8.0 release I had to fix a few failing tests for a > > > > couple of languages, but it was mainly just doing library updates. > > > > > > > > I think there will be less trouble with builds if we have separate > > > > releases, but my main motivation is to avoid languages blocking one > > > another > > > > and make it possible to get more releases out. I think we could have > > had > > > > several JavaScript releases this year if JS could have been released > > > > independently, but the rest of the community isn't moving fast enough > > for > > > > that. And that's fine; we need to fix bugs in C before we put out a C > > > > release, but we don't want C to prevent other language contributors > > from > > > > moving forward. > > > > > > > > > I'm not at all convinced that splitting the project into > per-language > > > > pieces > > > > will help. It will be a lot of work and I think we'll just get lots > of > > > > orphaned languages that never get released. > > > > > > > > Splitting does two positive things: it allows releases to happen > > > > independently and it makes maintenance and contribution easier. > > > > > > > > Independent releases are a net positive, in my opinion. It is > important > > > to > > > > get contributions into releases so that contributors benefit from > their > > > > work. Otherwise, what is the incentive to contribute back? Releasing > > > > implementations independently allows us to get contributions back to > > the > > > > community faster and decreases the likelihood that someone loses > > interest > > > > because they don't get to use what they fixed. > > > > > > > > I think that a better release cadence for active languages outweighs > > the > > > > cost of orphaned languages. I agree that it's likely for, say, PHP to > > not > > > > be released again. But what is the value of releasing code that > hasn't > > > > changed? Splitting would no longer push us to periodically fix > > languages > > > > when they break, but this irregular maintenance isn't substantially > > > > different than having no releases. I think there's a strong case that > > it > > > is > > > > better for downstream consumers to see that the version number hasn't > > > > changed rather than having one identical release after another that > > > gives a > > > > false sense that the implementation is actively maintained. > > > > > > > > The second positive effect of splitting from above is to make > > maintenance > > > > and contribution easier. A good example is adding more environments > to > > CI > > > > tests. We aren't going to add 3 versions of Ruby to the Docker image, > > but > > > > it's simple to set up in a CI environment for typical Ruby projects. > > > > Language-specific repositories allow us to take better advantage of > > > > existing tools, while structuring each repository for people familiar > > > with > > > > that language, making it easier for the people most likely to > > contribute > > > to > > > > do so. I know it isn't horribly difficult today, but I think it's a > > step > > > in > > > > the right direction to make this easier. > > > > > > > > rb > > > > > > > > On Wed, Nov 16, 2016 at 8:41 PM, Sean Busbey <bus...@cloudera.com> > > > wrote: > > > > > > > > > On Tue, Nov 15, 2016 at 3:45 PM, Doug Cutting <cutt...@gmail.com> > > > wrote: > > > > > > On Mon, Nov 14, 2016 at 8:40 PM, Ryan Blue > > <rb...@netflix.com.invalid > > > > > > > > > wrote: > > > > > >> we don't have enough votes for a release > > > > > > > > > > > > I don't think that's true. You might not have gotten enough > votes > > > > > > within a few days. I too have had that problem for several > > releases. > > > > > > But when that happened I would send personal messages to a few > PMC > > > > > > members asking them if they had time to please review a release. > > > Some > > > > > > PMC members get busy with other things and fall behind on Avro > > > emails. > > > > > > A few reminders never failed to rouse enough reviews & votes. > > > > > > > > > > > > > > > > In HBase we've largely switched away from release votes that have a > > > > > fixed deadline due to the need to corral PMC votes. > > > > > > > > > > Instead, we now state the minimum voting period (almost always > 72hrs > > > > > per ASF policy) and set a courtesy notice of when the RM would like > > to > > > > > close the vote. Then we just keep pinging private@ or individuals > > > > > until we get enough votes for a result, or find something to start > > > > > getting -1s. > > > > > > > > > > We could try something like that for a bit to see if "flog the > bushes > > > > > for PMC participation" is enough to keep us on a steady stream of > > > > > releases. > > > > > > > > > > It would also probably help to make explicit to the community that > > > > > showing up to test and vote on RCs is "acting like a PMC" in the > eyes > > > > > of some number of current PMC members (it is for me, for example). > > > > > > > > > > -- > > > > > busbey > > > > > > > > > > > > > > > > > > > > > -- > > > > Ryan Blue > > > > Software Engineer > > > > Netflix > > > > > > > > > > > > > > > -- > > Ryan Blue > > Software Engineer > > Netflix > > > > > > -- > Best regards / Met vriendelijke groeten, > > Niels Basjes > -- Ryan Blue Software Engineer Netflix
