On 14 Feb 2018, at 19:56, Tayyebi, Ameen <tayye...@amazon.com<mailto:tayye...@amazon.com>> wrote:
Newbie question: I want to add system/integration tests for the new functionality. There are a set of existing tests around Spark Catalog that I can leverage. Great. The provider I’m writing is backed by a web service though which is part of an AWS account. I can write the tests using a mocked client that somehow clones the behavior of the webservice, but I’ll get the most value if I actually run the tests against a real AWS Glue account. Benefits/cost of both mock: fast, fault injection, jenkins can run without credentials, but you need to maintain the mock. Failing a mock may just be a false alarm; success doesn't mean that much. live: needs credentials for runs, runs up bills, slow at distance or scale. Gives real answers about whether things work. Harder to inject faults. Way more complex test setup. Personally, I prefer live & try and set tests up to cost a few cents and run fast, even though that keeps them out of Jenkins-based test runs for submitted patches. After all: you do need the real test cases somewhere. How do you guys deal with external dependencies for system tests? That's credentais, For the hadoop core FS tests, all the tests which need (aws, azure) secrets are * tagged as integration tests and run in a different maven phase * only executed if the credentials are set in a non-SCM managed file, test/resources/auth-keys.xml, where its safest to actually pull them in from elsewhere via XInclude, so your secrets are never in the git-managed dirs * Are designed to run in parallel so you can do a full test run in < 10 minutes, even remotely * Apart from the scale tests which work with multiple MB and let you define that scale up to many GB and 1000s of files/directories if you are running in-infra. * Have really long timeouts, again, configurable for long-haul & scale tests you can look at the code there, and note the testing docs, which have a strict "declare the specific infra endpoint you ran against or nobody will even look at your code". That forces honesty in all submissions, even from ourselves https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/site/markdown/testing_azure.md The hadoop-aws module now has a fault injecting client built into the release JAR, something you can turn on from a hadoop config option. This is designed to let integration tests verify that the layers above are resilient to those failures being injected (retryable throttle exceptions, repeatedly observable listing inconsistencies, soon, broken connections on read() calls. Anything downstream can turn it on to see what breaks Spark 2.3 has a hadoop-cloud module which pulls in those FS clients and their transient modules, but not any tests. I keep my core set here https://github.com/hortonworks-spark/cloud-integration with the test suite trait and base class https://github.com/hortonworks-spark/cloud-integration/blob/master/cloud-examples/src/main/scala/com/hortonworks/spark/cloud/CloudSuite.scala These pick up a path to a config file which you keep out of the source tree, again for security reaosh mvn test -Dcloud.test.configuration.file=/Users/stevel/Projects/sparkwork/cloud-test-configs/s3a.xml ... I'd look at the POM there for how things propagate, and how that suite trait has a ctest() method which only registers a test case for execution of the conditions are met, where conditions include the credentials being provided https://github.com/hortonworks-spark/cloud-integration/blob/master/cloud-examples/src/main/scala/com/hortonworks/spark/cloud/CloudSuiteTrait.scala#L62 Is there an AWS account that is used for this purpose by any chance? Sadly no. Microsoft give apache committers Azure credits if you ask nicely, but not from your colleagues. Now, if you were able to change that policy, things would be nice Now I know how AWS STS works, with the AssumeRole API allowing you to create tokens which are only valid for a few minutes, I think it could actually be possible to have a jenkins setup where Jenkins creates some temporary AWS Credentials for the duration of a test run, revokes them after, for a role restricted to AWS resources, so that even patches from untrusted sources could be run with the credentials. You'd need to give jenkins the full keys though, and keep them locked down, somehow, + maybe refresh them nightly for bonus security (& restrict those to verify few rights too: assume role for that restricted role would be enough)
