steveloughran commented on issue #1332: HADOOP-16445. Allow separate custom signing algorithms for S3 and DDB URL: https://github.com/apache/hadoop/pull/1332#issuecomment-531861052 we also use the credentials for talking to STS (session credentials) and I have an ambition to talk to Amazon SQS to subscribe to changes in an S3 bucket for spark streaming. As a result, I'm thinking about how to make that possible after your changes. I'm also slowly trying to stop `S3AUtils` getting any worse as the one stop "throw all our static static stuff in here. `S3AUtils` is a merge troublespot and has grown too big. I'm reluctant to move existing stuff, because it makes backporting so hard, but I'd like to make the amount of new stuff we had there nearly non-existent. In particular #970 is adding a new class `org.apache.hadoop.fs.s3a.impl.NetworkBinding` where networking stuff can go; I'm not sure the best place for authentication stuff. Maybe a class alongside that `org.apache.hadoop.fs.s3a.impl.AwsConfigurationFactory`. What do you think? Also, rather than a new method `createAwsConfForDdb` alongside the existing one, I'd rather than existing one was extended to take a string declaring what the configurations for, e.g : "s3", "ddb", "sts" ... I'm proposing a string over an enum to maintain binary compatibility for applications which add/use a new service. If the name is unknown, we could just warn and return the "default" configuration. So how about adding the new operations into some into some `AwsConfigurationFactory.createAwsConf(string purpose,...)` method, with `S3AUtils.createAwsConf()` tagged as deprecated and invoking the new method. I'm reluctant to cut it, as I suspect some people (me) have been using it elsewhere. With that, I'm now going to to some minor review of the patch at the line-by-line level. Those comments come secondary to what I've just suggested.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
