Hi, I may have been ambiguous in my suggestion about the mailing lists. I did not intend to suggest separating mailing lists for the different language bindings. What I would like to separate from each other instead are real discussions and automated notifications (JIRAs, pull request, etc.).
Zoltan On Tue, Nov 15, 2016 at 5:40 AM Ryan Blue <[email protected]> wrote: > Thanks for responding, everyone! > > First, on the subject of the Attic: I don't think Avro is headed to the > Attic soon, but I think that the current level of engagement is too low to > refresh the active part of the community -- we don't have enough votes for > a release and it takes a long time for commits to get reviewed, and longer > to be released. Not being able to rebuild the community is the path to the > Attic, sooner or later. It's encouraging is that we are getting > contributions, and that's what I want to build on by making it easier on > both sides. I don't mean to be overly negative, I just think we have a > problem to address. > > I agree that the build process is much improved with Docker, but I still > don't think it's adequate. I just wouldn't expect someone to go through the > lengthy process to build our Docker image to contribute, and we know, from > commits that break the Docker build, that committers don't either. That > signals to me that we need something lighter weight. A second reason is > that we aren't testing everything that we probably should: Ruby, for > example, blocked the last release because tests failed in a few versions > that aren't part of the docker image. It's simple to set up those > environments in Travis CI, but a big cost if we use a big, common docker > image. > > I would drive this effort if we get agreement on it, but I think we should > make sure we agree on what it would look like. Suraj points out that we > have two python implementations to maintain, and Zoltan suggests that we > could separate lists and everything. I wouldn't go so far as to completely > separate the projects. I'd want to keep the same lists and function as a > single community because many of us do work on multiple languages, but I > would make it easy to spend time on one or two implementations without the > distraction of the rest. > > As a consequence, if some implementations are no longer used, we have a > fairly natural way of moving on. Maybe there isn't another php release > after splitting. I'd be fine with that if it improves the others. To a > degree, I also like that we're forced to care about the other > implementations, but right now I think it is too much of a weight: I think > we are missing out on people that would otherwise contribute and become > part of the community because of the slow release schedule and difficult > processes. > > rb > > On Mon, Nov 14, 2016 at 3:36 PM, Doug Cutting <[email protected]> wrote: > > > I disagree that Avro is heading towards the Attic. Its rate of > > contribution has been slow but steady for years. That's the nature of > > this project. It had a larger number of contributions in its first > > few years, when new languages and substantial features were being > > added, but since then, we see a relatively steady stream of bug fixes > > and minor features. Here are issues fixed per year: > > > > 2009 213 > > 2010 345 > > 2011 240 > > 2012 183 > > 2013 130 > > 2014 103 > > 2015 71 > > 2016 107 > > > > This year is actually an uptick from the past few. Projects go to the > > Attic when there are zero commits and zero releases for years, with > > fewer than 3 PMC members who'll respond to emails at all. We're > > committing several contributions per week. It sometimes takes a > > little prodding to get enough PMC members to vote on a release, but > > they're out there and will eventually vote and help get a release out. > > > > Pre-commit tests would be great to have, facilitating development & > > releases. +1 > > > > Breaking the project into multiple products, each released separately, > > could be a significant task, especially when you include getting the > > first release out for each. On one hand, it would make per-language > > releases easier, but it might also let some build problems languish > > undetected. Build issues are sometimes the result of other projects > > and distributions changing and may only be detected when you re-build, > > even if code hasn't changed. Lastly, getting PMC reviews & votes for > > more-frequent, single-language releases might be harder than > > less-frequent, multi-language releases. To be clear, I don't strongly > > oppose splitting things up, but I don't think it will be easy & may > > create some new problems as it resolves existing ones. > > > > I installed a new version of Linux on my laptop a few months ago and > > was able to recreate Avro's docker image relatively quickly and > > painlessly. I just now started it for the first time since, and it > > took 10 minutes to update. Building all languages took another 5 > > minutes. Each step required only a single command and proceeded > > without failure. It's a little slow, but not prohibitive. Overall > > the process is much better than before we had docker. Sure, it could > > be improved, but it's not unworkable. I find other steps of releases > > more tiresome. > > > > Would it be nice to be able to get a Java release out more quickly and > > easily? For sure, but getting there might not be that simple. Are > > you proposing to drive this effort, following it through to > > completion, nursing it if it stumbles, perhaps reverting it if it > > somehow fails? (Hadoop was once split & reunified.) > > > > To some degree, splitting assumes that each implementation has a > > sufficient community to maintain itself independently. We have a > > viable community combined, but I worry that breaking the project up > > could fragment it into projects that are too tiny to make releases. > > Right now we're all forced to care a bit about other implementations. > > > > Doug > > > > On Sun, Nov 13, 2016 at 12:21 PM, Ryan Blue <[email protected]> wrote: > > > Hi everyone, > > > > > > We tried to release Avro 1.8.2 this week, but the release vote failed > > > because only two PMC members voted on the candidate and we didn't have > > > enough binding votes to pass. There was a minor problem (in my opinion) > > > with the candidate that could have been the reason why there weren't > more > > > votes. If there's anyone out there that didn't vote because of this, > > please > > > say so. Otherwise, this appears to be related to declining engagement > in > > > the community and that's a major problem I want to discuss. > > > > > > Right now, we aren't getting contributions reviewed, committed, and > > > released in time for new contributors to become part of the community > and > > > refresh the set of active committers and PMC members. If that > continues, > > > this community is heading for the Attic. I think we can build back an > > > active committer base and I'd love to discuss how to do that on this > > thread. > > > > > > To get us started, I have a couple of ideas. > > > > > > I think we need to make it easier to participate. I've brought up these > > > ideas before when we moved to git: I think we need to separate the > > > implementations into their own repositories and set up CI tests for > each > > > one. > > > > > > Right now, a contributor has to wait for a review to get feedback and a > > > committer has to build a contribution and run tests. If we set up CI, > > then > > > the contributor gets automated feedback and committers can spend their > > time > > > on substantive review, rather than making sure the patch builds and > tests > > > pass. Making it easier for contributors and committers will help > increase > > > participation. > > > > > > Similarly, the build and release process is too difficult. It took me > > hours > > > to get the docker image built so I could make a release candidate, > > because > > > of a failure rate of about 1/500 downloading and installing packages. I > > had > > > to try ~20 times before one happily completed. While the docker image > > helps > > > a lot, the real problem we need to solve is how difficult it is to > build > > > all of Avro. The docker image helps, but no one really uses it until > it's > > > time to check a release. Instead, we all build and test how we are used > > to > > > for a particular language implementation: maven for Java, Rake for > ruby, > > > etc. That's why the build.sh scripts get broken and we don't notice, > and > > > why the only problem with the latest RC was that it didn't pass C# > tests > > > outside of docker. > > > > > > The current build also makes implementation releases dependent on one > > > another. Last release, C and ruby problems caused a multi-week delay, > and > > > this release we want to get the C# test environment fixed before the > next > > > candidate. All of this makes it take longer for contributions to make > it > > > out, which undermines the motivation for people to contribute. > > > > > > Separating the implementations will allow us to structure each > repository > > > how the contributors and committers for that language expect it to be. > We > > > can also set up per-implementation CI easily through Travis CI. And the > > > biggest benefit is separating the releases, so that Python, for > example, > > > can release a bug fix without waiting months for unrelated changes in > > Java. > > > > > > From my perspective, these two things are a good place to start. To > > > everyone still reading, what do you think? > > > > > > rb > > > > > > > > > -- > > > Ryan Blue > > > > > > -- > Ryan Blue > Software Engineer > Netflix >
