I'm not at all convinced that splitting the project into per-language pieces will help. It will be a lot of work and I think we'll just get lots of orphaned languages that never get released.
When I did the 1.8.0 release I had to fix a few failing tests for a couple of languages, but it was mainly just doing library updates. As others have pointed out, we can catch those kinds of problems earlier by having CI that uses the Docker image. I think that's the biggest thing to fix at this point (see AVRO-1705, AVRO-1887). Optimizing that build can come later. Cheers, Tom On Tue, Nov 15, 2016 at 1:10 PM, Zoltan Ivanfi <[email protected]> wrote: > Hi, > > I may have been ambiguous in my suggestion about the mailing lists. I did > not intend to suggest separating mailing lists for the different language > bindings. What I would like to separate from each other instead are real > discussions and automated notifications (JIRAs, pull request, etc.). > > Zoltan > > On Tue, Nov 15, 2016 at 5:40 AM Ryan Blue <[email protected]> > wrote: > > > Thanks for responding, everyone! > > > > First, on the subject of the Attic: I don't think Avro is headed to the > > Attic soon, but I think that the current level of engagement is too low > to > > refresh the active part of the community -- we don't have enough votes > for > > a release and it takes a long time for commits to get reviewed, and > longer > > to be released. Not being able to rebuild the community is the path to > the > > Attic, sooner or later. It's encouraging is that we are getting > > contributions, and that's what I want to build on by making it easier on > > both sides. I don't mean to be overly negative, I just think we have a > > problem to address. > > > > I agree that the build process is much improved with Docker, but I still > > don't think it's adequate. I just wouldn't expect someone to go through > the > > lengthy process to build our Docker image to contribute, and we know, > from > > commits that break the Docker build, that committers don't either. That > > signals to me that we need something lighter weight. A second reason is > > that we aren't testing everything that we probably should: Ruby, for > > example, blocked the last release because tests failed in a few versions > > that aren't part of the docker image. It's simple to set up those > > environments in Travis CI, but a big cost if we use a big, common docker > > image. > > > > I would drive this effort if we get agreement on it, but I think we > should > > make sure we agree on what it would look like. Suraj points out that we > > have two python implementations to maintain, and Zoltan suggests that we > > could separate lists and everything. I wouldn't go so far as to > completely > > separate the projects. I'd want to keep the same lists and function as a > > single community because many of us do work on multiple languages, but I > > would make it easy to spend time on one or two implementations without > the > > distraction of the rest. > > > > As a consequence, if some implementations are no longer used, we have a > > fairly natural way of moving on. Maybe there isn't another php release > > after splitting. I'd be fine with that if it improves the others. To a > > degree, I also like that we're forced to care about the other > > implementations, but right now I think it is too much of a weight: I > think > > we are missing out on people that would otherwise contribute and become > > part of the community because of the slow release schedule and difficult > > processes. > > > > rb > > > > On Mon, Nov 14, 2016 at 3:36 PM, Doug Cutting <[email protected]> wrote: > > > > > I disagree that Avro is heading towards the Attic. Its rate of > > > contribution has been slow but steady for years. That's the nature of > > > this project. It had a larger number of contributions in its first > > > few years, when new languages and substantial features were being > > > added, but since then, we see a relatively steady stream of bug fixes > > > and minor features. Here are issues fixed per year: > > > > > > 2009 213 > > > 2010 345 > > > 2011 240 > > > 2012 183 > > > 2013 130 > > > 2014 103 > > > 2015 71 > > > 2016 107 > > > > > > This year is actually an uptick from the past few. Projects go to the > > > Attic when there are zero commits and zero releases for years, with > > > fewer than 3 PMC members who'll respond to emails at all. We're > > > committing several contributions per week. It sometimes takes a > > > little prodding to get enough PMC members to vote on a release, but > > > they're out there and will eventually vote and help get a release out. > > > > > > Pre-commit tests would be great to have, facilitating development & > > > releases. +1 > > > > > > Breaking the project into multiple products, each released separately, > > > could be a significant task, especially when you include getting the > > > first release out for each. On one hand, it would make per-language > > > releases easier, but it might also let some build problems languish > > > undetected. Build issues are sometimes the result of other projects > > > and distributions changing and may only be detected when you re-build, > > > even if code hasn't changed. Lastly, getting PMC reviews & votes for > > > more-frequent, single-language releases might be harder than > > > less-frequent, multi-language releases. To be clear, I don't strongly > > > oppose splitting things up, but I don't think it will be easy & may > > > create some new problems as it resolves existing ones. > > > > > > I installed a new version of Linux on my laptop a few months ago and > > > was able to recreate Avro's docker image relatively quickly and > > > painlessly. I just now started it for the first time since, and it > > > took 10 minutes to update. Building all languages took another 5 > > > minutes. Each step required only a single command and proceeded > > > without failure. It's a little slow, but not prohibitive. Overall > > > the process is much better than before we had docker. Sure, it could > > > be improved, but it's not unworkable. I find other steps of releases > > > more tiresome. > > > > > > Would it be nice to be able to get a Java release out more quickly and > > > easily? For sure, but getting there might not be that simple. Are > > > you proposing to drive this effort, following it through to > > > completion, nursing it if it stumbles, perhaps reverting it if it > > > somehow fails? (Hadoop was once split & reunified.) > > > > > > To some degree, splitting assumes that each implementation has a > > > sufficient community to maintain itself independently. We have a > > > viable community combined, but I worry that breaking the project up > > > could fragment it into projects that are too tiny to make releases. > > > Right now we're all forced to care a bit about other implementations. > > > > > > Doug > > > > > > On Sun, Nov 13, 2016 at 12:21 PM, Ryan Blue <[email protected]> wrote: > > > > Hi everyone, > > > > > > > > We tried to release Avro 1.8.2 this week, but the release vote failed > > > > because only two PMC members voted on the candidate and we didn't > have > > > > enough binding votes to pass. There was a minor problem (in my > opinion) > > > > with the candidate that could have been the reason why there weren't > > more > > > > votes. If there's anyone out there that didn't vote because of this, > > > please > > > > say so. Otherwise, this appears to be related to declining engagement > > in > > > > the community and that's a major problem I want to discuss. > > > > > > > > Right now, we aren't getting contributions reviewed, committed, and > > > > released in time for new contributors to become part of the community > > and > > > > refresh the set of active committers and PMC members. If that > > continues, > > > > this community is heading for the Attic. I think we can build back an > > > > active committer base and I'd love to discuss how to do that on this > > > thread. > > > > > > > > To get us started, I have a couple of ideas. > > > > > > > > I think we need to make it easier to participate. I've brought up > these > > > > ideas before when we moved to git: I think we need to separate the > > > > implementations into their own repositories and set up CI tests for > > each > > > > one. > > > > > > > > Right now, a contributor has to wait for a review to get feedback > and a > > > > committer has to build a contribution and run tests. If we set up CI, > > > then > > > > the contributor gets automated feedback and committers can spend > their > > > time > > > > on substantive review, rather than making sure the patch builds and > > tests > > > > pass. Making it easier for contributors and committers will help > > increase > > > > participation. > > > > > > > > Similarly, the build and release process is too difficult. It took me > > > hours > > > > to get the docker image built so I could make a release candidate, > > > because > > > > of a failure rate of about 1/500 downloading and installing > packages. I > > > had > > > > to try ~20 times before one happily completed. While the docker image > > > helps > > > > a lot, the real problem we need to solve is how difficult it is to > > build > > > > all of Avro. The docker image helps, but no one really uses it until > > it's > > > > time to check a release. Instead, we all build and test how we are > used > > > to > > > > for a particular language implementation: maven for Java, Rake for > > ruby, > > > > etc. That's why the build.sh scripts get broken and we don't notice, > > and > > > > why the only problem with the latest RC was that it didn't pass C# > > tests > > > > outside of docker. > > > > > > > > The current build also makes implementation releases dependent on one > > > > another. Last release, C and ruby problems caused a multi-week delay, > > and > > > > this release we want to get the C# test environment fixed before the > > next > > > > candidate. All of this makes it take longer for contributions to make > > it > > > > out, which undermines the motivation for people to contribute. > > > > > > > > Separating the implementations will allow us to structure each > > repository > > > > how the contributors and committers for that language expect it to > be. > > We > > > > can also set up per-implementation CI easily through Travis CI. And > the > > > > biggest benefit is separating the releases, so that Python, for > > example, > > > > can release a bug fix without waiting months for unrelated changes in > > > Java. > > > > > > > > From my perspective, these two things are a good place to start. To > > > > everyone still reading, what do you think? > > > > > > > > rb > > > > > > > > > > > > -- > > > > Ryan Blue > > > > > > > > > > > -- > > Ryan Blue > > Software Engineer > > Netflix > > >
