Hi, We've got the HADOOP-8545<https://issues.apache.org/jira/browse/HADOOP-8545>patch stable enough that I'd like it checked in and in the 2.1 beta so that we can see what problems occur in the field.
The patch adds a new module, hadoop-tools/hadoop-openstack, so it doesn't break anything in hadoop-common/hdfs/yarn, and it includes more method-by-method tests than any of the non-HDFS filesystem clients have, including basic scale tests and the corner-case problems that we're going to have to talk about in terms of FS API spec and tests. In github, there are some very basic scale tests running Pig jobs against a configurable amount of data, showing that we can run pig against GB of data. We've run these tests against rackspace US, rackspace US and hpcloud -the public Swift services -as well as private deployments of the latest version of OpenStack. This makes us confident that we've got authentication right, and as we get consistent outcomes from all clusters, we aren't relying some cluster-/provider- specific detail. So: please can people review it! The site documentation (did I mention we've got that?) lists how to connect to a swift service, and how to test against them. If anyone does want to run the test themselves, try the docs and then escalate to us if there are any problems. Steve Loughran, Dmitry Mezhenskiy (Mirantis) & David Dobbins (Rackspace)
