I don’t have sufficient information to tell you what I’d do in this
situation, but here are some general principles to help you think through
the tradeoffs here.
- Whether or not I’d create a user schema per example really depends on
how long that operation takes. If it’s a pretty quick operation (e.g.
milliseconds) I’d probably just make one per example; since these are
end-to-end tests they’re expected to be slow and doing something like a
before(:context) hook caries a high maintenance cost since it has so
many caveats compared to a typical before(:example) hook.
- I personally don’t follow the “one expectation per example” rule at
all. It was a useful corrective to poorly written tests that contained tons
of expectations and were hard to debug, but bears a high cost (much higher
than I’m willing to pay) due to all the time wasted repeating setup. In
fast, isolated unit tests, I do follow a principle I’d call “one behavior
per example” (where a single behavior can often be specified with a single
expectation, but not always). I care a lot about keeping a fast, snappy
test suite so in slower integration or acceptance tests I write much
courser-grained tests that may encapsulate a whole workflow.
- I tend to use :aggregate_failures for all my integration and
acceptance tests.
- If a user schema can safely be re-used in many examples, and creation
of the schema was slow enough to warrant sharing it among multiple
examples, I’d probably use a different approach than a before(:context)
hook. Instead, I’d probably do something like this:
# spec/spec_helper.rbmodule UserSchemaCache
def self.schema
@schema ||= SecureRandom.uuid.tap do |schema_name|
SchemaService.new.create_schema(schema_name)
end
endend
RSpec.configure do |config|
config.when_first_matching_example_defined(:needs_schema) do
UserSchemaCache.schema
endend
# some_spec.rb
RSpec.describe 'Authentication', :needs_schema do
# ...end
That allows you to use the same user schema among all your end-to-end
examples, incurring the cost of creating it only once. The
when_first_matching_example_defined hook is only invoked if there’s an
example matching :needs_schema, so there’s no cost when running your other
specs.
HTH,
Myron
On Tue, Aug 1, 2017 at 5:34 PM, Jon Gordon <[email protected]> wrote:
> Hello again Myron :)
>
> I'm guessing that's why I couldn't find many before(:all) examples online.
> I wasn't aware of aggregate_failures, that's pretty cool feature! I always
> tried to follow the 'single assertion per test' principle. However, the
> example above is small portion of a bigger Spec. In the same Spec file I
> will also try to delete an entity (which require me to create an entity
> first), get a list of entities (create at least 2 entities) and expect
> exception to get raised when I'm trying to create an entity that I already
> created. All of those require a working schema. I can't group all up them
> under a single aggregate_failures test. So sadly, I will have to use
> instance variable, or to wrap a set block like the example you shared above
> (unless you have another idea)
>
> But I'm wondering, let's say I could control every point in the process
> and design this whole spec from scratch, having the option to do whatever I
> want on each of the remote micro-services. What would be the proper way of
> addressing it? Doesn't feel like the tests above are something specials,
> more like a standard REST service testing. Here's the option I can think
> about:
>
> 1. Create the schema over and over each test with 'before' block like so :
>
> let(schema_name) { SecureRandom.uuid.to }
>
> before
> user_schema.create_schema(schema_name)
> end
>
> it 'my test' do
> puts schema_name
> end
>
> This will be slower as schema will getting created every-test, and I only
> really need a single one as the schema is just a dependency and no the main
> focus of the test. But - perhaps clearer to read and avoid sharing context
> with before(:all).
>
> 2. Pre-populate the remote test database with default user schema with
> default name. I can either create it directly on the database when I spin
> the environment or from spec_helper . I can even store the default values
> in YML file, create an object from it (config) and use it on test.
>
> payload = JSON.parse(File.read('spec/acceptence/fixtures/feature.json'))
> payload['schema_name'] = config(:default_schema_name)
>
> This however makes the test less clear because not all information is
> being exposed when you read just the spec file. I would like to do it the
> 'proper' way - as I can probably talk with the other teams in the future -
> makes those test better align with the standards.
> Thanks for the help again!
>
>
>
> On Tuesday, August 1, 2017 at 12:48:45 PM UTC+3, Myron Marston wrote:
>
>> before(:all) hooks require special care and we usually recommend you
>> avoid them. Most RSpec tooling assumes a per-example lifecycle for
>> resources (such as DB transactions, test doubles, and let memoization),
>> and before(:all) hooks operate outside that lifecycle. If you’re not
>> careful to manage the resources explicitly you’re likely to experience
>> problems from using :all hooks. For this reason we don’t provide
>> syntactic sugar for them (such as a let construct), but it’s pretty
>> trivial to create your own construct if you want:
>>
>> module BeforeAllConstructs
>> def set(name, &block)
>> attr_reader name
>> before(:context) { instance_variable_set("@#{name}", block.call }
>> endend
>> RSpec.configure do |config|
>> config.extend BeforeAllConstructsend
>>
>> With this in place, you could use set in place of let to have a
>> construct like let designed for :all/:context hooks.
>>
>> That said, in your case, I wouldn’t recommend you take that route. It
>> doesn’t seem like having two separate examples here provides you any clear
>> benefit, and the fact you are considering using a before(:all) hook
>> indicates you want to do some computation once, and then make multiple
>> assertions about that. Instead, I’d recommend you combine the two examples,
>> but then use :aggregate_failures so that you get a list of all failures
>> (and not just the first one). Here’s how you could do that:
>>
>> RSpec.describe 'Authentication' do
>> subject { AuthenticationService.new }
>> let(:user_schema) { SchemaService.new }
>>
>> context 'When creating a user that does not exists ' do
>> it 'creates a new user', :aggregate_failures do
>> schema_name = SecureRandom.uuid.to_s
>> user_schema.create_schema(schema_name)
>> payload =
>> JSON.parse(File.read('spec/acceptence/fixtures/feature.json'))
>> payload['schema_name'] = schema_name
>>
>> response = subject.create_user(user: 'my_user', payload: payload)
>>
>> expect(response.code).to eq 200
>>
>> response = subject.get_user_by_category(category: payload['category'])
>> remote_entity = JSON.parse(response.body)
>>
>> expect(payload.to_json).to eq(
>> remote_entity['list'][unique_value]
>> )
>> end
>> endend
>>
>> HTH,
>> Myron
>>
>>
>> On Tue, Aug 1, 2017 at 2:49 PM, Jon Gordon <[email protected]> wrote:
>>
>>> Refactor how? the system consists of 4 micro-services. In a single
>>> sanity scenario, they are all being called (some sort of a smoke test to
>>> verify all can communicate with each other when spun up). The database is
>>> being written on the last micro-service in line, it's code that being
>>> handled by another group. I can ask them to give me an API to mock the db,
>>> but that still requires me during setup to access the mock, name the entity
>>> and then use that name in the example block. As I don't communicate here
>>> with objects like I do with unit-testings (instead I'm using REST API) I
>>> see no way how I can achieve that?
>>>
>>> Thanks!
>>>
>>> On Tuesday, August 1, 2017 at 9:25:40 AM UTC+3, Jon Rowe wrote:
>>>>
>>>> > As it not allowed by RSpec to use a let value inside a before(:all)
>>>> block.
>>>>
>>>> This is for good reason as it’s a bad idea to share test stare, you
>>>> could assign a constant but it would be better to refactor your code base
>>>> not to depend on an external db for each test like this.
>>>>
>>>> Jon Rowe
>>>> ---------------------------
>>>> [email protected]
>>>> jonrowe.co.uk
>>>>
>>>> On Tuesday, 1 August 2017 at 01:39, Jon Gordon wrote:
>>>>
>>>> Hi again :-)
>>>>
>>>> So I tried to write couple of RSpec test since we last talked, I'll
>>>> mention again I'm writing couple end to end tests to verify all
>>>> micro-services are up and running. Please consider the following example.
>>>> It's an authentication service, that can create users. Before user is being
>>>> created, a schema for the user-type needs to be created on a different
>>>> micro-service:
>>>>
>>>> RSpec.describe 'Authentication' do
>>>> subject { AuthenticationService.new }
>>>> let(:user_schema) { SchemaService.new }
>>>>
>>>> context 'When creating a user that does not exists ' do
>>>> it 'response with status code 200 (success)' do
>>>> schema_name = SecureRandom.uuid.to_s
>>>> user_schema.create_schema(schema_name)
>>>> payload = JSON.parse(File.read('spec/acc
>>>> eptence/fixtures/feature.json'))
>>>> payload['schema_name'] = schema_name
>>>>
>>>> response = subject.create_user(user: 'my_user', payload: payload)
>>>>
>>>> expect(response.code).to eq 200
>>>> end
>>>>
>>>> it 'creates a new user' do
>>>> schema_name = SecureRandom.uuid.to_s
>>>> user_schema.create_schema(schema_name)
>>>> payload = JSON.parse(File.read('spec/acc
>>>> eptence/fixtures/feature.json'))
>>>> payload['schema_name'] = schema_name
>>>>
>>>> subject.create_user(user: 'my_user', payload: payload)
>>>>
>>>> response = subject.get_user_by_category(category: payload[
>>>> 'category'])
>>>> remote_entity = JSON.parse(response.body)
>>>>
>>>> expect(payload.to_json).to eq(
>>>> remote_entity['list'][unique_value]
>>>> )
>>>> end
>>>> end
>>>> end
>>>>
>>>> To keep it dry, I should be moving the whole schema creation into a
>>>> before block:
>>>>
>>>> before
>>>> schema_name = SecureRandom.uuid.to
>>>> user_schema.create_schema(schema_name)
>>>> end
>>>>
>>>> Before makes sense to me over let here, because it's an action.
>>>> However, because the schema is more of a 'pre-condition', I can create it
>>>> just once, and avoid multiple schema in my database. Therefore,
>>>> before(:all) seems like a better option.
>>>>
>>>> before(:all)
>>>> schema_name = SecureRandom.uuid.to_s
>>>> user_schema.create_schema(schema_name)
>>>> end
>>>>
>>>> Now that problem is that when I create user in my examples, I NEED to
>>>> schema name, so I need to share context between the before and it block. It
>>>> makes sense for me to do it like so:
>>>>
>>>> let(:schema_name) { SecureRandom.uuid.to_s }
>>>>
>>>> before(:all)
>>>> user_schema.create_schema(schema_name)
>>>> end
>>>>
>>>> Then I can create easily create user in my example like so:
>>>>
>>>> payload = JSON.parse(File.read('spec/acceptence/fixtures/feature.json'
>>>> ))
>>>> payload['schema_name'] = schema_name
>>>>
>>>> subject.create_user(user: 'my_user', payload: payload)
>>>>
>>>> Alas, this will not work. As it not allowed by RSpec to use a let
>>>> value inside a before(:all) block. So I need to hold a string that can be
>>>> used in both the it and before block. It can solved it by defining a
>>>> Constant or using Instance variable but both methods feels reek to me. I
>>>> mentioned I don't have access to the database (as those are remote machines
>>>> and they don't expose the ip for that database) so I can't truncate the db
>>>> information and avoid the before block here.
>>>>
>>>> Thanks!
>>>>
>>>> On Thursday, July 20, 2017 at 10:57:05 AM UTC+3, Jon Gordon wrote:
>>>>
>>>> Not in Unit-tests of-course, but It seems like the only option in real
>>>> end-to-end testing. The system is quite complex, as it's basically a set of
>>>> couple micro-services. Each has it's own unique database (Can be Postgress,
>>>> Casanda, Oracle...). In a single End to End test, all databases are
>>>> populated with information. Clearing tables between each test can take
>>>> time, and is quite complex. The CI process does re-start the containers at
>>>> the very start of the test-run (so it's like restarting them to a fresh
>>>> state), but not during tests.
>>>>
>>>> if I'll take the list of music instruments in the Faker gem for
>>>> example, It only has around 10 options. So even if I use the unique flag -
>>>> It will run out of options after 10 test-cases. I guess use 'msuic
>>>> instruments' in one spec file, and 'cat-names' on the other to avoid it,
>>>> but that means I need to 'remember' what pool of string I already used in
>>>> previous tests, and that's feel even worse for me.
>>>>
>>>> Thanks.
>>>>
>>>>
>>>>
>>>> but I thought that there no better way around it. Because
>>>>
>>>> On Thursday, July 20, 2017 at 3:29:25 AM UTC+3, Myron Marston wrote:
>>>>
>>>> In general, if you need absolutely unique strings, `SecureRandom.uuid`
>>>> is a good way to get one. UUIDs are universally unique, after all :).
>>>>
>>>> That said, the fact that you are running out of unique random strings
>>>> from faker is concerning. All tests should work off of a "clean slate" in
>>>> your database, either by wrapping each test in a rolled-back transaction,
>>>> or by truncating your DB tables. Are you "leaking" DB records between
>>>> tests?
>>>>
>>>> Myron
>>>>
>>>> On Wed, Jul 19, 2017 at 12:42 PM, Jon Gordon <[email protected]> wrote:
>>>>
>>>> Hi Myron,
>>>>
>>>> I will definitely check the book - looks like it's perfect for RSpec
>>>> beginner. Also, thanks for sharing the example online - that's alone can
>>>> give me a good starting base :)
>>>>
>>>> I will ask another question while posting - for unit-testing I'm using
>>>> Faker gem to fake common strings. However, in end to end tests, we work
>>>> against a database - so even if I'm using the 'unique' method, I'm running
>>>> out of unique strings after a while. I'm using 'securerandom' to generate
>>>> random number, but wondering if there's a better approach for that.
>>>>
>>>> Thanks!
>>>>
>>>> On Wednesday, July 19, 2017 at 6:33:49 PM UTC+3, Myron Marston wrote:
>>>>
>>>> My upcoming book, Effective Testing with RSpec 3
>>>> <https://www.google.com/url?q=https%3A%2F%2Fpragprog.com%2Fbook%2Frspec3%2Feffective-testing-with-rspec-3&sa=D&sntz=1&usg=AFQjCNHGLaAn9OUSvszwbNhLSkP9Ypy-7A>,
>>>> has an example of building a JSON API using end-to-end acceptance tests,
>>>> isolated unit tests, and integration tests. It might fit what you're
>>>> looking for better since you mentioned you're looking for examples of
>>>> end-to-end testing of REST services.
>>>>
>>>> The code for the book is all online <https://github.com/rspec-3-book>,
>>>> as well.
>>>>
>>>> All that said, Xavier's screen cast is very good, and I definitely
>>>> recommend it, particularly if you do better with videos than printed
>>>> materials.
>>>>
>>>> Myron
>>>>
>>>> On Wed, Jul 19, 2017 at 4:30 AM, Jon Gordon <[email protected]> wrote:
>>>>
>>>> Thanks Xavier :)
>>>> I will be checking this course!
>>>>
>>>> Is there perhaps an open-source project with end-to-end spec tests you
>>>> can recommend (REST tests are preferred, not Capybara)? something to get a
>>>> reference from?
>>>> Thank you.
>>>>
>>>> On Wednesday, July 19, 2017 at 2:24:46 AM UTC+3, Xavier Shay wrote:
>>>>
>>>> Obligatory plug for https://www.pluralsight.co
>>>> m/courses/rspec-ruby-application-testing which touches on some of the
>>>> themes you're asking about :)
>>>>
>>>>
>>>> On Tue, Jul 18, 2017, at 04:06 PM, Jon Rowe wrote:
>>>>
>>>> Hi Jon
>>>>
>>>> A couple of tips, firstly you can stub out your external dependencies
>>>> for an end to end test, it just depends on the level of integration you
>>>> want, it’s equally fine to do what you propose. For injecting your endpoint
>>>> (IP, hostname or otherwise) you have a couple of ways of doing it, the
>>>> simplest is to use environment variables e.g. `ENV[‘API_ENDPOINT’]`, or you
>>>> can build yourself a config system like you mention. The reason why you
>>>> don’t see big projects using external configuration files is it is usually
>>>> done at the app level rather than in rspec.
>>>>
>>>> If you chose to go down the config file route, xml, yml or otherwise,
>>>> you’d be better off loading it in a spec_helper or other such support file,
>>>> and assigning it somewhere.
>>>>
>>>> Personally I would go with json fixture files for static json, or a
>>>> generator method if it needs to be dynamic.
>>>>
>>>> Cheers.
>>>> Jon
>>>>
>>>> Jon Rowe
>>>> ---------------------------
>>>> [email protected]
>>>> jonrowe.co.uk
>>>>
>>>> On Wednesday, 19 July 2017 at 01:52, Jon Gordon wrote:
>>>>
>>>>
>>>> Hi everyone,
>>>>
>>>> I'm quite new to RSpec, and I have used it mainly for unit-testing.
>>>> Lately, a need for a small number of end-to-end tests became relevant. When
>>>> writing test-cases, I'm trying to stub all dependencies, but because that's
>>>> not an option when doing integration tests, I need some help to understand
>>>> what's the proper way to do things. Here's couple of questions:
>>>>
>>>> 1. The test requires an IP for remote machine (which is not local and
>>>> sadly can not be). Obviously, I shouldn't supply the IP inside the spec
>>>> file. The simple way is reading an external YML file with the IP (that will
>>>> get created automatically during the CI process with the right IP for
>>>> example) and populate the IP directly from it. But, I was checking couple
>>>> of big project that uses rspec, and I never seen an external configuration
>>>> file, so I'm thinking perhaps there is a better way of doing it
>>>>
>>>> 2. If indeed YML file is the right answer, I'm not sure if reading from
>>>> the YML file every spec file (that uses this service) is the right thing to
>>>> do? Shouldn't I be using hooks instead for that?
>>>>
>>>> 3. The test-object is a REST service, and some of the requests require
>>>> big json object. I have two options:
>>>> a. I can create the json object in the spec file itself (which
>>>> makes all information visible to you from the spec file itself, but
>>>> clutters the spec)
>>>> b. Creating an external default fixture (which is basically a json
>>>> file), read from it during the spec, and re-write the values that are
>>>> relevant for the specific tests.
>>>>
>>>> Thank you!
>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "rspec" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>> To post to this group, send email to [email protected].
>>>> To view this discussion on the web visit https://groups.google.com/d/ms
>>>> gid/rspec/61ac9ade-1045-4211-80d3-441ef01ae7cb%40googlegroups.com
>>>> <https://groups.google.com/d/msgid/rspec/61ac9ade-1045-4211-80d3-441ef01ae7cb%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "rspec" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>> To post to this group, send email to [email protected].
>>>> To view this discussion on the web visit https://groups.google.com/d/ms
>>>> gid/rspec/3FF6FCF2018A482CBDC70C02BAFFB643%40jonrowe.co.uk
>>>> <https://groups.google.com/d/msgid/rspec/3FF6FCF2018A482CBDC70C02BAFFB643%40jonrowe.co.uk?utm_medium=email&utm_source=footer>
>>>> .
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "rspec" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>> To post to this group, send email to [email protected].
>>>> To view this discussion on the web visit https://groups.google.com/d/ms
>>>> gid/rspec/28f3f239-1515-437b-b011-82b2dd163502%40googlegroups.com
>>>> <https://groups.google.com/d/msgid/rspec/28f3f239-1515-437b-b011-82b2dd163502%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "rspec" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>> To post to this group, send email to [email protected].
>>>> To view this discussion on the web visit https://groups.google.com/d/ms
>>>> gid/rspec/c297c4c9-5225-47d9-a6e2-80f461bd1226%40googlegroups.com
>>>> <https://groups.google.com/d/msgid/rspec/c297c4c9-5225-47d9-a6e2-80f461bd1226%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "rspec" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>> To post to this group, send email to [email protected].
>>>> To view this discussion on the web visit https://groups.google.com/d/ms
>>>> gid/rspec/48d88387-8e71-49a5-b25a-850a79fe4181%40googlegroups.com
>>>> <https://groups.google.com/d/msgid/rspec/48d88387-8e71-49a5-b25a-850a79fe4181%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "rspec" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> To post to this group, send email to [email protected].
>>> To view this discussion on the web visit https://groups.google.com/d/ms
>>> gid/rspec/3ae74ab4-f4fd-4f17-b87f-4256656b57d4%40googlegroups.com
>>> <https://groups.google.com/d/msgid/rspec/3ae74ab4-f4fd-4f17-b87f-4256656b57d4%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "rspec" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/rspec/adfd947f-875e-47cf-91e5-a95264f7aedc%40googlegroups.com
> <https://groups.google.com/d/msgid/rspec/adfd947f-875e-47cf-91e5-a95264f7aedc%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"rspec" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/rspec/CADUxQmspz8EvmnHqGeSHwJ6CSgQ_gR18g7BdD0Bc3KHgkbbj7g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
