I've just merged in HADOOP-18073 and the S3A connector now uses the V2 AWS SDK.
This is a significant change at the source code level. Any applications using the internal extension/override points in the filesystem connector are likely to break. - Mostly that's code in cloudera -our problem. - my little cloudstore jar already looks for the new files, but you can't compile its S3 specific stuff (-Pextra) as getS3ClientForTesting() is gone, replaced by an accessor in S3AInternals interface, and the client is currently v2 sync. - HBoss test code fails; I've got a WiP fix for HBoss too (HBASE-28056) The aws-sdk-bundle JAR has been replaced by the shaded V2 equivalent, "bundle.jar", which is now exported by the hadoop-aws module. No, it's not the most useful of names. Expect your first build to take a while... The v2 sdk will coexist with the old one; but anything excluding the aws-sdk-jar from hadoop-aws/hadoop-cloud-storage poms is going to get the v2 one. There's a final stabilisation JIRA: HADOOP-18886. S3A: AWS SDK V2 Migration: stabilization Add any problems which surface there. As usual, testing of different deployment configurations is always essential. Finally, big credit to Ahmar Suhail for this major piece of work. The AWS v2 SDK changes every single classname *and every single method name* making it an absolute nightmare of a migration, a classic "what not to do in version 2.0" change. I think it's because of the move to machine generated classes off some schema, and a builder-only constructor mechanism. This is why it's such a big upgrade. Steve
