Noah - thanks for your thoughts here! oatts looks like a really interesting project - it's awesome to see more push towards OpenAPI tooling. I'm about to install it and try it out on one of my specs locally. I'll let you know how it goes and report any issues (if any). Would love to contribute, so I'll keep an eye on the repo. I really like the idea of contract testing the spec against the backend early in development, we're trying to push that it should absolutely be done after development as well.
What you said about the problem/solution space is definitely in-line with what I've been working on with Assertible. if you can't assert against actual values in the expected responses, to > merely validate the schema of the response body (for JSON, at least) is > definitely a win (unsure if you Assertible is actually doing that just > after a quick glance). WRT populating auth values, that's a tough one. ... > Maybe you can expose a hook that they configure to pull some creds at > runtime, that are not stored and they can change whenever? The JSON Schema validation is a really good idea - we do already have a test assertion for validating an HTTP response against JSON Schema <https://assertible.com/docs/guide/assertions#json-schema>, enabling this for Swagger specs out of the box is doable, I'll take a look. On the auth problem again, that's right on the money! Assertible has a concept of setup steps <https://assertible.com/docs/guide/tests#setup-steps>, which are basically requests run *before* the actual test that can be used to populate variables. Auth and route parameters are the driving use-case there: make a request to fetch a token or similar, and use a {{variable}} in the header of the test. Route parameters work the same, but if a `default` value is specified for the parameter in the spec, we can actually fully populate this and not require any additional setup. Cases like this really make it easy to just import -> test. I like that oatts and Assertible are at separate ends of the testing spectrum, too -- the more tools we can get around this the better. Lets chat if you're interested in sharing ideas! I'm @no_d_here on Twitter > if thats easier than this or email. That sounds great -- I just watched oatts and followed you on Twitter. Looking forward to your updates. If you're able to try out Assertible, I'd love to get your feedback. It will probably be easier to chat over email (codyreich...@gmail.com) so we don't pull this thread too far off topic. - Cody Reichert On Monday, May 8, 2017 at 9:55:30 PM UTC-5, Noah Dietz wrote: > > Hey y'all! > > Great conversation! This is something I've been working in/on for a few > years now, here and there. I started by working on > github.com/apigee-127/swagger-test-templates , a unit test scaffolding > generator (the concept of which I think was mentioned a bit in conversation > here earlier?). It parses a swagger spec and generates unit tests for every > path/operation/status code defined, while exposing configurations for > supplying parameters and expected response bodies. The main goal is to > accelerate API development by enabling developers to create test code for > their APIs early and often, thus strengthening the contract between > implementation and specification. It was meant to be integrated with > swagger-node, but has gotten a decent amount of use as an individual > module. > > I've since been working on an unofficial rewrite built on > github.com/apigee-127/sway . The rewrite can be found at > github.com/noahdietz/oatts and I'd love feedback/contributors as it is in > the early stages. > > @Cody, I love the problems you're solving and bringing up. One thing I > have found is valuable is if you can't assert against actual values in the > expected responses, to merely validate the schema of the response body (for > JSON, at least) is definitely a win (unsure if you Assertible is actually > doing that just after a quick glance). WRT populating auth values, that's a > tough one. With the aforementioned swagger-test-templates, because it was > designed to generate test code locally, we added support for loading values > from a config file or the environment for such sensitive things, rather > than hard code them. This is tougher when all done in the cloud for them. > Maybe you can expose a hook that they configure to pull some creds at > runtime, that are not stored and they can change whenever? Lets chat if > you're interested in sharing ideas! I'm @no_d_here on Twitter if thats > easier than this or email. > > On Monday, May 8, 2017 at 4:14:57 PM UTC-7, Cody Reichert wrote: >> >> I've been really interested in this problem for a while now, and your >> assumption is correct that it's definitely possible. We've been working on >> solving this at Assertible <https://assertible.com>, and a couple of the >> harder parts we've hit are: >> >> Route parameters and other variables. Required "setup" for PUT/DELETE/etc >> requests. Setting up auth. In a lot of cases there just *has* to be some >> additional configuration. >> >> That said, we've built what I this is a very good starting point to >> automatically generate a test suite from a Swagger spec. >> >> We put together a post on Testing an API using Swagger >> <https://assertible.com/blog/testing-an-api-using-swagger>. >> >> Basically, when you import a Swagger spec, a test is generated for each >> unique endpoint/method combination (like *GET* /users and *POST* >> /users). Route parameters are automatically filled in as "*variables*" >> and auth is stubbed out from the "*securityDefinitions*" found in the >> swagger spec. The actual tests/validations are done by creating " >> *assertions*" that are created for the expected status code response. >> >> For example, importing the Twitter Swagger spec would give you something >> like this: >> >> >> <https://lh3.googleusercontent.com/-oIsCAXZYBAw/WRD6ZB5aRII/AAAAAAAAAU8/WwTJhE9fsuESsNPdWjhaz3Bl7UI5zq0kQCLcB/s1600/assertible-twitter-swagger-import.png> >> What are your (or anyone elses) thoughts on this approach? We've got a >> lot of plans/ideas for how we can dynamically generate even more tests and >> assertions from a Swagger spec. The idea is that, you simply have to import >> your spec once, and you're set up with automated tests, monitoring, failure >> alerts, etc, right out of the box. >> >> Let me know what you think - I'm really interested in others people's >> thoughts about testing from a spec! >> >> - Cody Reichert >> codyre...@gmail.com >> https://assertible.com >> >> On Friday, August 16, 2013 at 6:38:35 AM UTC-5, chrokh wrote: >>> >>> I'm new to Swagger but I believe I'm getting a clearer picture of the >>> eco-system, and it looks fantastic. >>> >>> However, I'm wondering whether there's a module that allows me to run >>> automated tests against my actual API to see whether it conforms with my >>> Swagger spec. *Without* server integration. >>> >>> Assuming you have a Swagger spec-file and a running API it sounds >>> reasonable to me that this should be possible? Basically, black-box >>> testing. Is there something like this? >>> >>> Thanks in advance! >>> >> -- You received this message because you are subscribed to the Google Groups "Swagger" group. To unsubscribe from this group and stop receiving emails from it, send an email to swagger-swaggersocket+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
