adnanhemani commented on PR #6544: URL: https://github.com/apache/hadoop/pull/6544#issuecomment-1985378218
Hi Steve, thanks for reviewing this as you finished up with the Hadoop release work, I really appreciate you taking the time to look at this. I'm re-running the integration and unit tests after I made the changes you suggested above - I will ping this thread when the code is pushed and ready for a re-review. Responding to your earlier questions: > I'm curious about whether it is possible to test this with an ITest everywhere -and what happens? would it be possible to write a test for this? While it should be technically possible to do so, I'm not sure there's an easy (or acceptable) way to do so. In order to test this functionality with an ITest, we would need to set up an Access Grants instance, locations, and access grants themselves prior to actually running the client-side code, which we are integrating. Creating and tearing down those resources would need a good deal of effort imo - and do not come for free cost-wise either. Going back to the note we've made explicitly in the documentation, if there is a problem with functionality, this would be a general problem with the S3 Access Grants plugin - and not with S3A, as our unit test is capturing all code used for enabling this plugin. I believe the ITests on the S3 Access Grants plugin should then be sufficient for covering the full testing coverage. > now, how does this integrate with the usual auth mechanism? the standard credentials passed in are used to auth with (something? what?) to get the session credentials. Good question - when using S3 Access Grants, the flow would be to use your standard credentials to authenticate yourself to the S3 Access Grants server. Which would then hand you back a new set of credentials once you are authorized - this set of credentials should have access to the S3 objects you are looking to access. The Access Grants on the S3 Access Grants server are set by your account/organization's data admin, who is in charge of determining what sets of permissions each user should be able to receive. (I'm not sure if I'm answering the right question here - please feel free to elaborate on the question if I didn't answer it properly). > How are complex ops like rename() coped with? By `rename`, I'm assuming you mean the mechanism where we copy the object to the new name and then deleting the original object? If so, CopyObject is not supported at this moment but the S3 Access Grants team is working on a new revision of the plugin where they are able to support this in more constrained situations - such as when the object source and destination fall under the same S3 Access Grants prefix. Similar situation for DeleteObjects calls. I don't believe there's a S3 API specifically for `rename` - please correct me if I'm wrong. If there are any other complex/not-so-popular S3 actions that the user attempts to do, the S3 Access Grants plugin will not attempt to do credential vending and instead always instruct the S3 Client to fall back to using the user's provided authentication credentials. > When do things fail and how? specifically, are new errors likely to be raised when S3 requests are made, and if so is their translation valid? So given that, functionally, the only thing we are changing is the credentials which the user is using to access the S3 data, there should not be any changes to the set of errors that can be returned from S3 while attempting to access the data itself. However, there could be an error thrown while the user is attempting to authorize against the S3 Access Grants server. In this case, I believe the only exception the S3 Access Grants plugin is allowed to throw is a `SdkServiceException`. But per my understanding, this is simply passed onto the S3 client itself, which is fully in control of how to handle this error - and will likely send this error back to the user. Exactly which exception the user would receive back, I'm not completely sure but per my testing, it is (correctly) not causing the user to retry the credential vending process. I can research which error specifically this may be. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
