Hi John,
Thanks for the elaborate description/review of the available options for
BDD testing in #1251 (https://github.com/ledgersmb/LedgerSMB/issues/1251)!
(Verbatim copy below for those who can't get to the issue easily).
I'll put my comments in a top posting, because that makes the response more
compact. Also, since your text isn't marked as quoted, it makes more clear
what's yours and what is mine.
I like how you run down each of the options we have at the various layers
we need. To summarize (and verify my understanding), these are the layers:
* Testrunner
* Test framework/browser-driver-abstraction [not available for Perl, you
say?]
* Browser driver(s)
* Step definitions
All of these are available for various languages (but not all for Perl), if
I understand you correctly. So, that brings me my first assumption: that we
need to use Perl for our testing. The reason I was operating under the
assumption that we need to use Perl for our testing has a number of reasons
(which I write down here for others to challenge them):
1. We use Travis-CI specifically configured for Perl; I'm not sure which
other languages are available and to what extent
2. With the developers we have at the moment, I'd expect to get up and
running with a few basic scenarios most quickly in Perl
3. Our only option for testing seemed to be Selenium (because of
SauceLabs), which looked "complete" at the Perl side
The reason that we seemed limited to Selenium testing was that I didn't
understand from the Travis-CI docs that we might be able to run other
platforms like PhantomJS. Re-reading the docs with the background you gave
me below, I now think they actually *do* provide PhantomJS on all images:
https://docs.travis-ci.com/user/gui-and-headless-browsers/#Using-PhantomJS .
Your remark that 15 minutes go down to 1minute by switching from Selenium
to PhantomJS caught my attention: I really like the fact that our test
suite provides its feedback as quickly as it does now. Given that we have
loads of tests to add for the in-browser testing, I'd be very much for
using the fastest test platform available.
As far as using PhantomJS with Perl, this article suggests we could simply
drive PhantomJS with the "standard" Selenium driver:
http://blogs.perl.org/users/brian_medley/2013/02/headless-selenium-testing-with-phantomjs.html
Could it be that that driver (or the phantom-framework) provides enough of
an API? (Actually, it looks like PhantomJS integrated the webdriver
protocol in its 1.8 release in 2012?)
That's the technology side of things. Then you raise the question of why we
need BDD, especially since we don't have a library of step definitions
readily usable.
There are multiple reasons to want BDD, even though - as you point out
correctly - it's perfectly fine for the current developers to write tests
directly driving the Selenium or Poltergeist API.
My reasons to want BDD are:
1. If we have BDD, we can solicit contributions to our testing framework
from a much broader audience
2. If we have BDD, we can use that as discussion document format for new
features to be implemented
3. BDD forces the creation of libraries for our testing routines. If you
look at the t/*.t files, you'll find they get hard to follow quite quickly
-- I'd expect the forced separation between step definitions and the actual
steps to greatly help tests keep clean
4. Driving the web-tester APIs can get pretty tedious to do manually
(error-prone); I'm thinking (hoping) that using BDD helps me gain developer
efficiency here.
What about my expectations (3) and (4)? Can you confirm those from your own
experience?
-----
While writing the above, I've tried setting up PhantomJS testing on Travis
CI. Actually, that worked and I've successfully connected to it with the
selenium webdriver!
So, we have a choice now for our testing framework to run against SauceLabs
(real Selenium Browser-based) or against PhantomJS, it seems.
One thing that's built into SauceLabs which I don't know how to do with
PhantomJS is to capture images of a page at the point of test failure. Can
you help there?
-----
Thanks again!
Regards,
Erik.
================== original posting
In IRC, we've been discussing options for adding browser-based tests,
especially in 15 where so much is driven by Javascript
Erik mentioned using Test::BDD::Cucumber for this, based on my enthusiasm
for Behavior Driven Design tests based on what we're doing in Drupal He was
asking me to help get this framework in place
After spending some time evaluating options, it looks like what is
available for us to use in Perl is far less mature than what's available
for Drupal
TL;DR - check out Test::Selenium::Remote::Driver (part of
Selenium::Remote::Driver) That's what we need in place first
Layers/components needed for BDD
The huge head start Drupal has with BDD is a bunch of driver/support
modules that provide > 120 step definitions right out of the gate, along
with all of the integration of the various layers of tools
Test runner - pherkin, cucumber, behat
The top level is there for Perl -- at least at a basic level
Test::BDD::Cucumber provides "pherkin" to run tests, and supports
integration with Test::Builder for running them in an automated way
Cucumber is sort of the gold standard here A Ruby tool, Cucumber seems to
have the most robust functionality available, and supports running tests in
several languages -- though I did not see exactly how you would hook this
up to step definitions written in Perl, and the code templates it outputs
are in Ruby (though I did find references online suggesting it could create
step definition stubs in other languages, but I did not find exactly how to
do this)
Behat is PHP
Really, we could potentially use any of these tools -- if we're doing
browser-side testing, the only reason we would need Perl is for the
scaffolding around more easily inserting test data (eg by calling our own
internal functions)
I actually tried setting up a simple feature to test the loginpl screen,
and all 3 executed this file, with slightly different output:
- All three used the same color-coding to indicate incomplete steps
- Pherkin provided little data about the test run -- only the steps
themselves
- Cucumber and Behat provided more info about line numbers of sources,
and output step definition templates (in Ruby or PHP)
- Behat provided the line number of the beginning of each scenario
Cucumber provided the line number of each individual step
- Behat provides a "behat -dl" option that lists all available,
pre-defined step definitions from all loaded sources (including your
project's Feature Context, where you define your own)
Test framework
The next layer is where Perl seems to fall short
Behat ships with a driver called "mink" which essentially drives the web
browser, and supports a bunch of actual drivers -- Selenium, Goutte,
Zombie, BrowserKit, Sahi, and a couple others Mink handles starting up the
browser, providing an abstraction layer across the various web drivers and
integrating right into Behat
It sounds like the main solution for Cucumber here is "Capybara", which
mainly uses Selenium or PhantomJS (via "Poltergeist"), and provides a
similar role as Mink
For Perl, I didn't find anything on CPAN that looks quite equivalent -- it
looks like much of this is done project-by-project The closest thing I
found is here: https://githubcom/motemen/Wight, which connects to PhantomJS
This layer provides most of the convenience for writing tests that drive
the web browser, but not supporting Gherkin directly -- rather,
Cucumber/Behat sits on top of this layer to drive a variety of actual
browser engines
Browser drivers
Here Perl has plenty to choose from, in the Mech:: namespace, and
Selenium::Remote::Driver At this level you're writing tests for a specific
browser driver, no abstraction layer (other than Selenium itself offers
some cross-browser testing, grid testing, etc)
Here the main decision might be headless or not -- Selenium drives fully
functional browsers, and that makes it slow -- although it also makes it
visible, something you can watch in action
We've started using a new screen-comparison tool that uses PhantomJS, and
it's so much faster -- tests that used to take 15 minutes to run now finish
up in < 1 minute
Step definitions
This is where Drupal is so far ahead The DrupalExtension provides a bunch
of step definitions already available for immediate use, for finding text
on a page, filling out forms, finding text in particular rows of a table,
generating user accounts, auto-logging a user in, creating content, users,
taxonomy terms, and more
You can start writing a feature file in Gherkin and immediately execute it
with much of it passing, not needing any step definitions written
What do we really need?
BDD and Gherkin is very useful for defining the user experience, mainly as
a communication tool that can be tested very quickly -- if you already have
an extensive library of step definitions
Since we don't have that library, there's quite a gap here in implementing
all the pieces necessary to get this working
And given that the main goal here is testing the functionality of the web
side of our application, and most of us (at least for the moment) are
capable of creating these tests in code (especially with some examples to
start with) I think we should start out with tests written using the
Test::Selenium::Remote::Driver module directly, and not attempt to engineer
a whole BDD testing stack to drive those Selenium tests
The alternatives:
-
Build out Wight or equivalent, and get it integrated with
Test::BDD::Cucumber to a sufficient point -- probably quite a bit of effort
to do this
-
Use Cucumber/Capybara/Ruby -- need to set up data manipulation using
Ruby, or figure out how to get Cucumber to run Perl steps for when we want
to enter data Introduces test dependency on Ruby (though if we're
considering Sass, we may end up with that anyway -- Compass seems to be the
de-facto Sass compiler)
-
Use Behat, and write steps in PHP -- we would be able to use many of the
pre-defined steps from DrupalExtension, because they are simply targeting
the DOM in the browser But now we have a PHP/Composer dependency, and any
data population we want to do would need to go through a PHP binding (or
escape to a shell)
-
Possibly try Cucumberjs -- I did not dig deep, but it does look like
there are a ton of tools built on nodejs for this kind of processing, and
it seems like the other vast ecosystem used across languages -- Ruby and
Nodejs seem to be on par for processing tools for testing, CSS/Sass
processing, etc And the dojo build step does introduce a dependency on
nodejs for doing the Javascript build
None of these seem like the right option for a Perl project I've been
scouring the web, and I'm finding very little on actually doing any tests
driving browsers for testing using Perl Just a handful of modules that
don't really talk to each other, relying on a lot of code from the project
(us) to actually get working
—
Reply to this email directly or view it on GitHub
<https://github.com/ledgersmb/LedgerSMB/issues/1251>.
--
Bye,
Erik.
http://efficito.com -- Hosted accounting and ERP.
Robust and Flexible. No vendor lock-in.
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
Ledger-smb-devel mailing list
Ledger-smb-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ledger-smb-devel
