Hi Bernd. On Feb 17, 2011, at 7:43 AM, Bernd Fondermann wrote: > > We have the very unfortunate situation here at Hadoop where Apache > Hadoop is not the primary and foremost place of Hadoop development. > Instead, code is developed internally at Yahoo and then contributed in > (smaller or larger) chunks to Hadoop.
This has been the situation in the past, but as you can see in the last month, this has changed. Yahoo! has publicly committed to move their development into the main code base, and you can see they have started doing this with the 20.100 branch, and their recent commits to trunk. Combine this with Nige taking on the 0.22 release branch, (and sheperding it into a stable release) and I think we have are addressing your concerns. They have also started bringing the discussions back on the list, see the recent discussion about Jobtracker-nextgen Arun has re-started in MAPREDUCE-279. I'm not saying it's perfect, but I think the major players understand there is an issue, and they are *ALL* moving in the right direction. > This is open source development upside down. > It is not ok for people to diff ASF svn against their internal code > and provide the diff as a patch without reviewing IP first for every > line of code changed. > For larger chunks I'd suggest to even go via the Incubator IP clearance > process. > Only then will we force committers to primarily work here in the open > and return to what I'd consider a healthy project. > > To be honest: Hadoop is in the process of falling apart. > Contrib Code gets moved out of Apache instead of being maintained here. > Discussions are seldom consense-driven. > Release branches stagnate. True. releases do take a long time. This is mainly due to it being extremely hard to test and verify that a release is stable. It's not enough to just run the thing on 4 machines, you need at least 50 to test some of the major problems. This requires some serious $ for someone to verify. > Downstream projects like HBase don't get proper support. > Production setups are made from 3rd party distributions. > Development is not happening here, but elsewhere behind corporate doors. > Discussion about future developments are started on corporate blogs ( > http://developer.yahoo.com/blogs/hadoop/posts/2011/02/mapreduce-nextgen/ > ) instead of on the proper mailing list. > Hurdles for committing are way too high. > On the bright side, new committers and PMC members are added, this is > an improvement. > > I'd suggest to move away from relying on large code dumps from > corporations, and move back to the ASF-proven "individual committer > commits on trunk"-model where more committers can get involved. > If that means not to support high end cluster sizes for some months, > well, so be it. > Average committers cannot run - e.g. test - on high > end cluster sizes. If that would mean they cannot participate, then > the open source project better concentrate on small and medium sized > cluster instead. Well.. that's one approach.. but there are several companies out there who rely on apache's hadoop to power their large clusters, so I'd hate to see hadoop become something that only runs well on 10-nodes.. as I don't think that will help anyone either. > > Bernd
