Thanks for responding, everyone! First, on the subject of the Attic: I don't think Avro is headed to the Attic soon, but I think that the current level of engagement is too low to refresh the active part of the community -- we don't have enough votes for a release and it takes a long time for commits to get reviewed, and longer to be released. Not being able to rebuild the community is the path to the Attic, sooner or later. It's encouraging is that we are getting contributions, and that's what I want to build on by making it easier on both sides. I don't mean to be overly negative, I just think we have a problem to address.
I agree that the build process is much improved with Docker, but I still don't think it's adequate. I just wouldn't expect someone to go through the lengthy process to build our Docker image to contribute, and we know, from commits that break the Docker build, that committers don't either. That signals to me that we need something lighter weight. A second reason is that we aren't testing everything that we probably should: Ruby, for example, blocked the last release because tests failed in a few versions that aren't part of the docker image. It's simple to set up those environments in Travis CI, but a big cost if we use a big, common docker image. I would drive this effort if we get agreement on it, but I think we should make sure we agree on what it would look like. Suraj points out that we have two python implementations to maintain, and Zoltan suggests that we could separate lists and everything. I wouldn't go so far as to completely separate the projects. I'd want to keep the same lists and function as a single community because many of us do work on multiple languages, but I would make it easy to spend time on one or two implementations without the distraction of the rest. As a consequence, if some implementations are no longer used, we have a fairly natural way of moving on. Maybe there isn't another php release after splitting. I'd be fine with that if it improves the others. To a degree, I also like that we're forced to care about the other implementations, but right now I think it is too much of a weight: I think we are missing out on people that would otherwise contribute and become part of the community because of the slow release schedule and difficult processes. rb On Mon, Nov 14, 2016 at 3:36 PM, Doug Cutting <[email protected]> wrote: > I disagree that Avro is heading towards the Attic. Its rate of > contribution has been slow but steady for years. That's the nature of > this project. It had a larger number of contributions in its first > few years, when new languages and substantial features were being > added, but since then, we see a relatively steady stream of bug fixes > and minor features. Here are issues fixed per year: > > 2009 213 > 2010 345 > 2011 240 > 2012 183 > 2013 130 > 2014 103 > 2015 71 > 2016 107 > > This year is actually an uptick from the past few. Projects go to the > Attic when there are zero commits and zero releases for years, with > fewer than 3 PMC members who'll respond to emails at all. We're > committing several contributions per week. It sometimes takes a > little prodding to get enough PMC members to vote on a release, but > they're out there and will eventually vote and help get a release out. > > Pre-commit tests would be great to have, facilitating development & > releases. +1 > > Breaking the project into multiple products, each released separately, > could be a significant task, especially when you include getting the > first release out for each. On one hand, it would make per-language > releases easier, but it might also let some build problems languish > undetected. Build issues are sometimes the result of other projects > and distributions changing and may only be detected when you re-build, > even if code hasn't changed. Lastly, getting PMC reviews & votes for > more-frequent, single-language releases might be harder than > less-frequent, multi-language releases. To be clear, I don't strongly > oppose splitting things up, but I don't think it will be easy & may > create some new problems as it resolves existing ones. > > I installed a new version of Linux on my laptop a few months ago and > was able to recreate Avro's docker image relatively quickly and > painlessly. I just now started it for the first time since, and it > took 10 minutes to update. Building all languages took another 5 > minutes. Each step required only a single command and proceeded > without failure. It's a little slow, but not prohibitive. Overall > the process is much better than before we had docker. Sure, it could > be improved, but it's not unworkable. I find other steps of releases > more tiresome. > > Would it be nice to be able to get a Java release out more quickly and > easily? For sure, but getting there might not be that simple. Are > you proposing to drive this effort, following it through to > completion, nursing it if it stumbles, perhaps reverting it if it > somehow fails? (Hadoop was once split & reunified.) > > To some degree, splitting assumes that each implementation has a > sufficient community to maintain itself independently. We have a > viable community combined, but I worry that breaking the project up > could fragment it into projects that are too tiny to make releases. > Right now we're all forced to care a bit about other implementations. > > Doug > > On Sun, Nov 13, 2016 at 12:21 PM, Ryan Blue <[email protected]> wrote: > > Hi everyone, > > > > We tried to release Avro 1.8.2 this week, but the release vote failed > > because only two PMC members voted on the candidate and we didn't have > > enough binding votes to pass. There was a minor problem (in my opinion) > > with the candidate that could have been the reason why there weren't more > > votes. If there's anyone out there that didn't vote because of this, > please > > say so. Otherwise, this appears to be related to declining engagement in > > the community and that's a major problem I want to discuss. > > > > Right now, we aren't getting contributions reviewed, committed, and > > released in time for new contributors to become part of the community and > > refresh the set of active committers and PMC members. If that continues, > > this community is heading for the Attic. I think we can build back an > > active committer base and I'd love to discuss how to do that on this > thread. > > > > To get us started, I have a couple of ideas. > > > > I think we need to make it easier to participate. I've brought up these > > ideas before when we moved to git: I think we need to separate the > > implementations into their own repositories and set up CI tests for each > > one. > > > > Right now, a contributor has to wait for a review to get feedback and a > > committer has to build a contribution and run tests. If we set up CI, > then > > the contributor gets automated feedback and committers can spend their > time > > on substantive review, rather than making sure the patch builds and tests > > pass. Making it easier for contributors and committers will help increase > > participation. > > > > Similarly, the build and release process is too difficult. It took me > hours > > to get the docker image built so I could make a release candidate, > because > > of a failure rate of about 1/500 downloading and installing packages. I > had > > to try ~20 times before one happily completed. While the docker image > helps > > a lot, the real problem we need to solve is how difficult it is to build > > all of Avro. The docker image helps, but no one really uses it until it's > > time to check a release. Instead, we all build and test how we are used > to > > for a particular language implementation: maven for Java, Rake for ruby, > > etc. That's why the build.sh scripts get broken and we don't notice, and > > why the only problem with the latest RC was that it didn't pass C# tests > > outside of docker. > > > > The current build also makes implementation releases dependent on one > > another. Last release, C and ruby problems caused a multi-week delay, and > > this release we want to get the C# test environment fixed before the next > > candidate. All of this makes it take longer for contributions to make it > > out, which undermines the motivation for people to contribute. > > > > Separating the implementations will allow us to structure each repository > > how the contributors and committers for that language expect it to be. We > > can also set up per-implementation CI easily through Travis CI. And the > > biggest benefit is separating the releases, so that Python, for example, > > can release a bug fix without waiting months for unrelated changes in > Java. > > > > From my perspective, these two things are a good place to start. To > > everyone still reading, what do you think? > > > > rb > > > > > > -- > > Ryan Blue > -- Ryan Blue Software Engineer Netflix
