Re: [rails-oceania] Integration spec failing on CI

[Dave Perrett]

Hey Jim, in my experience there's always a logic behind seemingly 
'random' failures (a test fails but... changing the order of tests 
causes it to pass, adding debugging causes it to pass, works locally but 
fails on CI, works on CI but fails locally, only works in isolation, 
fails in isolation but works within the suite etc). For me, almost 
without exception it has either highlighted edge cases in my code that I 
hadn't considered, or highlighted dependencies in my tests that 
shouldn't be there (the one exception so far being 
https://github.com/markevans/dragonfly/issues/135 which fellow 
rails-oceania-er malc somehow narrowed down and solved). Common causes 
for me have been :

- Timing issues (e.g. a time-based cache key works fine in production, 
but tests run almost instantaneously and intermittently fail).
- Ordering issues (accidentallyrelying on something that was set up by a 
previous test, but are run in a different order on CI).
- Distributed issues (relying on some same-server condition that isn't 
necessarily true on distributed CI - e.g. cache tests will pass locally, 
but fail intermittently on CI unless you specify a common cache across 
machines).
- Assuming a consistent test domain (e.g. www.example.com) or assuming a 
port won't be present in a URL (passing on example.com but failing on 
example.com:12345).
- Assuming an external internet connection that may not be available 
under CI.
- Cookie and domain issues for @javascript / selenium / webkit tests if 
you rely on certain domains or are trying to test subdomains.

I can't hazard any guesses from your gist, and I'm still running one or 
two 'fluke' pass tests myself (with TODO written all over them), so I 
wouldn't take advice from me :) 2c anyway for what it's worth.

Cheers


marsbomber <mailto:jimji...@gmail.com>
20 July 2012 3:12 PM
Hi guys,

I have a problem with one of my capybara-webkit based integration 
tests. It's passing on my local machine (TM) ... but it fails when the 
CI does the build. What's even more interesting is if I run the single 
failing spec on the CI server manually, it passes.

I fluked and fixed it ... but I really want to know why it was failing 
at the first place. Will it have something to do how the 'headless' 
gem fires up and recycles the x server? Anyone had similar issues like 
this?

A strip down version of the spec is at https://gist.github.com/3148406

Thanks in advance.

Cheers,
Jim
--
You received this message because you are subscribed to the Google 
Groups "Ruby or Rails Oceania" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/rails-oceania/-/mrtXhFj--y8J.
To post to this group, send email to rails-oceania@googlegroups.com.
To unsubscribe from this group, send email to 
rails-oceania+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/rails-oceania?hl=en.

--
You received this message because you are subscribed to the Google Groups "Ruby or 
Rails Oceania" group.
To post to this group, send email to rails-oceania@googlegroups.com.
To unsubscribe from this group, send email to 
rails-oceania+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/rails-oceania?hl=en.

<<inline: postbox-contact.jpg>>