steveloughran commented on pull request #2455: URL: https://github.com/apache/hadoop/pull/2455#issuecomment-735744062
> I wonder if the hadoop jenkins test can be set up to do the s3a test automatically like other tests. 1. we can't give it credentials for security reasons -even if we only issued short-lived session credentials, getting them would be as trivial as submitting a PR which printed them. Same for abfs 2, if someone isn't set up to run the tests, they aren't set up to deal with regressions or debug why their own patch doesn't work. 3. There's an extra benefit -because everyone's config is slightly different (network, endpoints, encryption, etc) we get better coverage of test configurations by having different people run the tests. It's not unusual for a patch to get merged in but which a few days later needs a followup as someone else finds a regression in their test setup. I would like more test runs, e.g the daily jenkins runs, to at least have credentials, but I've yet to come up with a good design for secure execution. It'd need something like * isolated AWS account (billed to who?) * two IAM roles: #1: with limited access to a single s3 bucket, #2: with the permission to call assumeRole on role #1 * something on build setup to call assumeRole at start of run and issue role credentials valid for a few hours max * the jenkins scripts would only get those role credentials That still leaves with the "what to do at the end of the run" problem. Maybe: revoke all sessions under a specific role through the relevant IAM API call -this might work if role#2 has the permissions and you only ever have one active session in role #1, because we'd have to revoke all sessions in that role. See: not easy. Put your error stack traces into the PR. A single test failure isn't enough to block a patch if we can identify a cause and say "this is independent". Given you are seeing things I'm not, that's something we need to understand. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
