On 9 February 2015 at 21:18:52, Colin P. McCabe
(cmcc...@apache.org<mailto:cmcc...@apache.org>) wrote:
What happened with the Crucible experiment? Did we get a chance to
try that out? That would be a great way to speed up patch reviews,
and one that is well-integrated with JIRA.
I am -1 on Gerrit unless we can find a way to mirror the comments to
JIRA. I think splitting the discussion is a far worse thing that
letting a few minor patches languish for a while (even assuming that
gerrit would solve this, which seems unclear to me). The health of
the community is most important.
I think it is normal and healthy to post on hdfs-dev, email
developers, or hold a meeting to try to promote your patch and/or
idea. Some of the discussion here seems to be assuming that Hadoop is
a machine for turning patch available JIRAs into commits. It's not.
It's a community, and sometimes it is necessary to email people or
talk to them to get them to help with your JIRA.
I know your heart is in the right place, but the JIRA examples given
here are not that persuasive. Both of them are things that we would
not encounter on a real cluster (nobody uses Hadoop with ipv6, nobody
uses Hadoop without setting up DNS).
Got some bad news there. The real world is messy, and the way Hadoop tends to
fail right now leaves java stack traces that tend to leave people assuming its
Hadoop side.
Messy networks are extra commonplace amongst people learning to use Hadoop
themselves, future community members, and when you are bringing up VMs.
In production, well, talk to your colleagues in support and say "how often do
you field network-related problems?", followed by "do you think Hadoop could do
more to help here?"
But, if we find a specific set
of issues that the community has ignored (such as good error messages
in bad networking setups, configuration issues, etc.), then we could
create an umbrella JIRA and make a sustained effort to get it done.
Seems like a good strategy.
I've just created https://issues.apache.org/jira/browse/HADOOP-11571, "get S3a
production ready". It shipped in Hadoop 2.6; now it's out in the wild the bug
reports are starting to come back in. Mostly scale related; some failure
handling, some improvements to work behind proxies and with non-AWS endpoints.
1. To date all the s3a code has come from none committers; the original
codebase
2. Most of the ongoing dev from is Thomas Demoor at amplidata,
3. There's been some support via AWS (HADOOP-10714),
4. There's been a couple of patches from Ted Yu after hbase backups keeled
over from too many threads
One thing that is notable about the s3a (or any of the object store
filesystems) is that Jenkins does not run the tests. Anyone proposing to +1 a
patch based on a Jenkins run (see HADOOP-11488) is going to get a -1 from me;
it takes 30-60 minutes for a test run. You get a bill of about 50c/month for
participating this project
To date I've been the sole committer running the tests, reviewing the code and
with a vague idea of what's being going on. That's because (a) I care about
object stores after my experience with getting swift:// in, and (b) I'm not
recommending that anyone use it in production until its been field-tested more.
Who is going to assist me review and test these patches?
Perhaps we could also do things like batching findbugs fixes into
fewer JIRAs, as has been suggested before.
A detail. Findbugs is not the problem
