In spite of that fact that it needs a tooling change, I like the added
<mode> tag / prepare steps.
The tooling change should be small and it means no runtime impact on apps.
I love the approach - a very positive step to cleaning up testing.
David Kemp
On Tue, Oct 15, 2013 at 12:18 PM, Michal Mocny <mmo...@chromium.org> wrote:
> Braden, you're right, good catch.
>
> Discussing locally how we could support "prepare --mode=..." in the most
> generalized form, we remembered an old suggestion to just support <mode>
> tags.
>
> The benefits seem to be:
> - No need to add custom tag-prefix/attributes for the combinations of
> js-module mode=, asset mode=, etc etc
> - More visually apparent when reading a plugin.xml file, akin to
> <platforms> tag
>
> The drawbacks seem to be:
> - Not all descendant tags are easy to support for a given mode (ie,
> <dependency>)
>
>
> Summarizing the options currently discussed in this thread:
>
> - new <js-test-module> tag. Not general enough solution to support tests
> bundling <assets>, so -1 from me for this reason.
> - mode="..." attribute for a set of whitelisted tags (<js-module>, <asset>,
> ...)
> - <mode name="..."> tag for a set of whitelisted descendant
> tags (<js-module>, <asset>, ...)
>
> The last two options are really functionally equivalent, but given that we
> opted for <platform> tag instead of attribute, I'm also favoring a <mode>
> tag.
>
> -Michal
>
>
> On Tue, Oct 15, 2013 at 11:43 AM, Braden Shepherdson <bra...@chromium.org
> >wrote:
>
> > It's not true that adding these tests only creates larger binaries. They
> > will be fetched and parsed by the plugin loader code at app startup time.
> > It goes through the list of all plugins in cordova_plugins.js and loads
> > them all. That parses them, and runs the outermost layer, which is the
> > wrapping function function(require, module, exports) { ... }. So there is
> > still runtime memory and CPU impact.
> >
> > I support <js-test-module> tags or <js-module ... test> or whatever.
> > Similarly, prepare for tests. I realize we want to avoid tooling support,
> > but I think tooling support is a lesser evil than production performance
> > impact.
> >
> > Overall approach makes me happy.
> >
> > Braden
> >
> >
> > On Fri, Oct 11, 2013 at 9:43 PM, Michal Mocny <mmo...@chromium.org>
> wrote:
> >
> >> On Fri, Oct 11, 2013 at 9:08 PM, Andrew Grieve <agri...@chromium.org>
> >> wrote:
> >>
> >> > The eval of the jasmine interface deserves mention. Is the motivation
> >> > there that tests can choose to use another testing framework? That's
> why
> >> > you don't just make jasmine functions globals?
> >> >
> >>
> >> I was hoping the plugins would need to depend on anything but CDVTest,
> and
> >> not expect any globals. I guess, though, that CDVTest still expects the
> >> app to provide to a test framework and some other stuff, so in the end
> its
> >> no different. I was hedging on being able to update CDVTest in the
> future
> >> for whatever we need, and all 3rdparty plugins would not need updating.
> >> eval() could be used to do all sorts of clever things if we needed..
> >>
> >>
> >> >
> >> > One nit pick just from reading your email is that this will cause the
> >> test
> >> > js-modules to be injected into apps that use the plugins. I think
> >> probably
> >> > we will want to update the tools to recognize a <js-test-module>. We
> >> > *could* work around it by adding the tests to new plugins that depend
> on
> >> > the thing they are testing, but I think changing the tools would be
> >> nicer.
> >> >
> >>
> >> I also mentioned splitting tests into second plugin but thats overkill
> >> except in extreme circumstances. Note that the tests aren't actually
> >> loaded unless you require them, so its just a matter of larger binaries
> >> which could be filtered out manually.
> >>
> >> My person preference would be to support conditional build types, which
> >> have come up before. ie, cordova prepare debug, cordova prepare
> release,
> >> cordova prepare test -- and plugin.xml could specify a different set of
> >> js-module for either.
> >>
> >> A specific <js-test-module> would be fine, but isnt "0 new tooling".
> >>
> >> Also, I forgot to mention, but we do need to add support for getting the
> >> full list of plugins installed, which should be trivial to add to
> >> modulemapper (maybe there is already a way to reach in, but I don't
> think
> >> we have a documented api for it).
> >>
> >>
> >> > Another nit is that it would be nice if the CordovaTests app didn't
> >> depend
> >> > on any plugins. e.g., don't have it depend on device plugin.
> >> >
> >>
> >> CordovaTests doesn't explicitly depend on device plugin, except that at
> >> the
> >> moment I have it printing the same info that MobileSpec does at startup.
> >> This could be wrapped in a try{}catch in case the require fails, but
> its
> >> good info to have in the common case.
> >>
> >>
> >> >
> >> > Overall, really like the approach!
> >> >
> >>
> >> Thanks!
> >>
> >>
> >> >
> >> >
> >> > On Fri, Oct 11, 2013 at 5:17 PM, Michal Mocny <mmo...@chromium.org>
> >> wrote:
> >> >
> >> >> TLDR; I've implemented a plugin testing strategy that requires 0 new
> >> >> tooling features, can support non-core plugins, and hopefully even
> >> support
> >> >> a variety of methods for running/reporting the test results. Super
> >> alpha
> >> >> preview, but take a look if you like the direction!
> >> >>
> >> >>
> >> >> NEW: CDVTest Plugin: https://github.com/mmocny/CDVTest
> >> >>
> >> >> NEW: CordovaTest App: https://github.com/mmocny/CordovaTests
> >> >>
> >> >> UPDATED: Converted three existing plugins to this "new style". Code
> >> is on
> >> >> feature branches of respective repos (cdvtest branch):
> >> >> org.apache.cordova.device
> >> >> org.apache.cordova.device-motion
> >> >> org.chromium.storage
> >> >> (must clone locally, switch to branch, and plugin add from local
> path)
> >> >>
> >> >>
> >> >> Now, any plugin that wants to join in on the fun needs to provide a
> >> >> <js-module
> >> >> src="..." name="test" /> and use this template:
> >> >>
> >> >> exports.init = function() {
> >> >>
> >> >>
> >>
> eval(require('org.apache.cordova.test.test').injectJasmineInterface(this,
> >> >> 'this'));
> >> >>
> >> >> // TESTS GO HERE
> >> >> describe(..., function() {
> >> >> it(...);
> >> >> });
> >> >> };
> >> >> (The eval is optional but super useful; it will inject the jasmine
> >> >> interface into your scope, so you don't have to type
> jasmine.describe,
> >> >> jasmine.it, etc. Not sure of any cleaner way to do this.)
> >> >>
> >> >>
> >> >> STEPS:
> >> >> 1. create a new cordova project
> >> >> 2. import the CordovaTest app into your www
> >> >> 3. add any or all of the above plugins
> >> >> 4. give it a run, and try out the auto tests (all pass on ipad, some
> >> still
> >> >> fail on android)
> >> >>
> >> >> Lots of work left to do, but hopefully good enough to whet your
> >> appetite.
> >> >>
> >> >> One thing to note, I've attempted to make CDVTest and plugin tests
> >> unaware
> >> >> of the specific jasmine version the app is hosting (so that it can be
> >> >> swapped without changing all plugins), but it must support the new
> >> style
> >> >> interface for async tests (ie, accept a done callback). This is the
> >> style
> >> >> that node-jasmine uses, mocha uses, and jasmine-2.0 is going to use
> >> (I'm
> >> >> using jasmine 2.0 rc3). This means that core plugin tests require
> some
> >> >> code transformations, but the net effect is cleaner tests and more
> >> common
> >> >> style with our node tools' tests.
> >> >>
> >> >> Manual tests are not implemented yet.
> >> >>
> >> >> -Michal
> >> >>
> >> >> On Fri, Oct 11, 2013 at 12:54 PM, Michal Mocny <mmo...@chromium.org>
> >> >> wrote:
> >> >>
> >> >> > I'm throwing something together right now, actually. I'll post my
> >> >> current
> >> >> > progress today so you can take a look.
> >> >> >
> >> >> >
> >> >> > On Fri, Oct 11, 2013 at 12:41 PM, Brian LeRoux <b...@brian.io> wrote:
> >> >> >
> >> >> >> Sorry keep meaning to respond. I like Michal's first step but
> >> growing
> >> >> to a
> >> >> >> full suite of tools. Are you currently tackling this Braden? I
> feel
> >> >> like
> >> >> >> it
> >> >> >> is related to the Medic stuff and maybe we should throw one of our
> >> >> guys on
> >> >> >> the problem fully.
> >> >> >>
> >> >> >>
> >> >> >> On Sep 27, 2013 5:10 PM, "Braden Shepherdson" <
> bra...@chromium.org>
> >> >> >> wrote:
> >> >> >>
> >> >> >> > Which one?
> >> >> >> >
> >> >> >> >
> >> >> >> > On Fri, Sep 27, 2013 at 10:09 AM, Brian LeRoux <b...@brian.io>
> >> wrote:
> >> >> >> >
> >> >> >> > > I really like your proposal as a starting point. Very simple
> but
> >> >> would
> >> >> >> > > allow for in-app testing as well as on the cmd line if we so
> >> wish.
> >> >> >> > >
> >> >> >> > >
> >> >> >> > > On Fri, Sep 27, 2013 at 3:28 PM, Michal Mocny <
> >> mmo...@chromium.org
> >> >> >
> >> >> >> > wrote:
> >> >> >> > >
> >> >> >> > > > I was looking over some old emails from this list on plugin
> >> >> testing,
> >> >> >> > and
> >> >> >> > > an
> >> >> >> > > > idea that was proposed way back was to ship plugin tests as
> a
> >> >> second
> >> >> >> > > > plugin. That way, you can chose to install tests, or not,
> and
> >> >> know
> >> >> >> > > > explicitly if they are being copied into your final project.
> >> >> >> > > >
> >> >> >> > > > An alternative would be to support build targets a la
> >> >> >> "release/debug"
> >> >> >> > and
> >> >> >> > > > have target-specific plugin.xml tags (assets, js-modules,
> >> >> >> > source-file..).
> >> >> >> > > >
> >> >> >> > > > -Michal
> >> >> >> > > >
> >> >> >> > > >
> >> >> >> > > > On Fri, Sep 27, 2013 at 4:52 AM, Brian LeRoux <b...@brian.io>
> >> >> wrote:
> >> >> >> > > >
> >> >> >> > > > > I think this is basically what we've been proposing for a
> >> while
> >> >> >> now.
> >> >> >> > > > >
> >> >> >> > > > >
> >> >> >> > > > > On Thu, Sep 26, 2013 at 8:29 PM, Michal Mocny <
> >> >> >> mmo...@chromium.org>
> >> >> >> > > > wrote:
> >> >> >> > > > >
> >> >> >> > > > > > I would suggest perhaps a simpler approach, which
> doesn't
> >> add
> >> >> >> > > anything
> >> >> >> > > > > new
> >> >> >> > > > > > to cordova-cli/plugman:
> >> >> >> > > > > >
> >> >> >> > > > > > - Each plugin ships with a "tests" js-module, and we
> >> >> document a
> >> >> >> > > > > convention
> >> >> >> > > > > > of where they should live, and what signature it should
> >> have
> >> >> >> (i.e.,
> >> >> >> > > > > > cordova.require('plugin.name.Tests').forEach(...) ).
> >> >> >> > > > > > - Will need a common way to describe/report results
> >> (others
> >> >> >> have
> >> >> >> > > > > > mentioned TAP).
> >> >> >> > > > > > - Any app is free to run those plugin tests in any which
> >> way,
> >> >> >> but
> >> >> >> > we
> >> >> >> > > > > ship a
> >> >> >> > > > > > mobile-spec app which is one opinionated way to do so.
> >> >> >> > > > > > - It attempts to require the test module for each
> >> installed
> >> >> >> > plugin,
> >> >> >> > > > > runs
> >> >> >> > > > > > them, and aggregates results.
> >> >> >> > > > > > - It could report results to some shared server, allow
> >> >> >> toggling
> >> >> >> > of
> >> >> >> > > > > tests,
> >> >> >> > > > > > etc, but no plugin should know or care about those
> >> features.
> >> >> >> > > > > >
> >> >> >> > > > > > Using that as a generic base:
> >> >> >> > > > > >
> >> >> >> > > > > > - We ship a "CDVTests" (or whatever) plugin which has a
> >> >> bunch of
> >> >> >> > > > library
> >> >> >> > > > > > code for creating tests, and plugins can use it to
> >> register
> >> >> >> their
> >> >> >> > > > tests.
> >> >> >> > > > > > - This makes it easier to register manual tests in a
> >> common
> >> >> >> format
> >> >> >> > > for
> >> >> >> > > > > core
> >> >> >> > > > > > plugins, and prevents code duplication for core auto
> >> tests.
> >> >> >> > > > > > - External plugins can chose to use our testing library,
> >> or
> >> >> not.
> >> >> >> > > > > >
> >> >> >> > > > > > -Michal
> >> >> >> > > > > >
> >> >> >> > > > > >
> >> >> >> > > > > > On Thu, Sep 26, 2013 at 10:34 AM, Braden Shepherdson <
> >> >> >> > > > > bra...@chromium.org
> >> >> >> > > > > > >wrote:
> >> >> >> > > > > >
> >> >> >> > > > > > > Here's an off-the-top-of-my-head sketch of how we
> might
> >> do
> >> >> >> > Voltron
> >> >> >> > > > > tests:
> >> >> >> > > > > > >
> >> >> >> > > > > > > - Add a tag to plugin.xml that names each test file:
> >> >> >> > > > > > > <test type="automatic" src="spec/foo.js" name="Foo
> >> >> >> Automated"
> >> >> >> > > />
> >> >> >> > > > > > > <test type="manual" src="spec/bar.js" name="Foo
> >> >> Manual" />
> >> >> >> > > > > > > - Add a new command, cordova test (maybe
> prepare-test),
> >> >> that:
> >> >> >> > > > > > > - Ignores the top-level www.
> >> >> >> > > > > > > - Instead copies in a basic testing index.html
> >> similar
> >> >> to
> >> >> >> the
> >> >> >> > > > > current
> >> >> >> > > > > > > mobile-spec's
> >> >> >> > > > > > > - That index reads a file akin to
> cordova_plugins.js
> >> >> >> > > > > > (cordova_tests.js,
> >> >> >> > > > > > > maybe?) generated by the CLI, containing the info from
> >> the
> >> >> >> <test>
> >> >> >> > > > tags.
> >> >> >> > > > > > > - It has navigation similar to the current
> >> mobile-spec,
> >> >> >> with
> >> >> >> > > > > buttons
> >> >> >> > > > > > > for the automatic and manual sections. Auto has "All"
> >> and
> >> >> then
> >> >> >> > each
> >> >> >> > > > > > module,
> >> >> >> > > > > > > manual just has the list of modules.
> >> >> >> > > > > > >
> >> >> >> > > > > > > Thoughts?
> >> >> >> > > > > > >
> >> >> >> > > > > > > Braden
> >> >> >> > > > > > >
> >> >> >> > > > > > >
> >> >> >> > > > > > > On Thu, Sep 26, 2013 at 6:33 AM, Carlos Santana <
> >> >> >> > > > csantan...@gmail.com
> >> >> >> > > > > > > >wrote:
> >> >> >> > > > > > >
> >> >> >> > > > > > > > I like the idea can we call mobilespec now
> >> >> cordova-voltron
> >> >> >> and
> >> >> >> > be
> >> >> >> > > > DRY
> >> >> >> > > > > > and
> >> >> >> > > > > > > > use the tests form the plugins.
> >> >> >> > > > > > > >
> >> >> >> > > > > > > > Voltron by itself creates an App that tests only
> core,
> >> >> but
> >> >> >> as
> >> >> >> > you
> >> >> >> > > > > > > > use plugman to add plugins to voltron it has more
> test
> >> >> >> cases.
> >> >> >> > > > > > > >
> >> >> >> > > > > > > > It would not be a bad idea to enhance plugin.xml in
> >> the
> >> >> >> future
> >> >> >> > to
> >> >> >> > > > > > include
> >> >> >> > > > > > > > information about testing (i.e. Directory containing
> >> >> tests
> >> >> >> > files,
> >> >> >> > > > > test
> >> >> >> > > > > > > > command, etc..)
> >> >> >> > > > > > > >
> >> >> >> > > > > > > > --Carlos
> >> >> >> > > > > > > >
> >> >> >> > > > > > > > On Thursday, September 26, 2013, Anis KADRI wrote:
> >> >> >> > > > > > > >
> >> >> >> > > > > > > > > What's the challenge of having us use the tests
> that
> >> >> come
> >> >> >> > with
> >> >> >> > > > the
> >> >> >> > > > > > > > > individual plugins ?
> >> >> >> > > > > > > > >
> >> >> >> > > > > > > > > On Thu, Sep 26, 2013 at 8:13 AM, David Kemp <
> >> >> >> > drk...@google.com
> >> >> >> > > > > > > > <javascript:;>>
> >> >> >> > > > > > > > > wrote:
> >> >> >> > > > > > > > > > Currently, the automated test system that we
> have
> >> >> >> running
> >> >> >> > > > > (derived
> >> >> >> > > > > > > from
> >> >> >> > > > > > > > > > Medic) uses only the mobilespec tests. It does
> not
> >> >> yet
> >> >> >> use
> >> >> >> > > > tests
> >> >> >> > > > > > > > > collected
> >> >> >> > > > > > > > > > from the plugins. Its been talked about, but not
> >> gone
> >> >> >> > > anywhere.
> >> >> >> > > > > > > > > >
> >> >> >> > > > > > > > > > David Kemp
> >> >> >> > > > > > > > > >
> >> >> >> > > > > > > > > >
> >> >> >> > > > > > > > > > On Wed, Sep 25, 2013 at 7:58 PM, Jesse <
> >> >> >> > > > purplecabb...@gmail.com
> >> >> >> > > > > > > > <javascript:;>>
> >> >> >> > > > > > > > > wrote:
> >> >> >> > > > > > > > > >
> >> >> >> > > > > > > > > >> Yeah, I have pushed some changes to
> mobile-spec,
> >> and
> >> >> >> when
> >> >> >> > I
> >> >> >> > > > did
> >> >> >> > > > > I
> >> >> >> > > > > > > also
> >> >> >> > > > > > > > > >> copied the tests into the plugin involved.
> >> >> >> > > > > > > > > >> Until we get the magic test runner happening, I
> >> >> think
> >> >> >> we
> >> >> >> > > just
> >> >> >> > > > > keep
> >> >> >> > > > > > > > > >> duplicating.
> >> >> >> > > > > > > > > >>
> >> >> >> > > > > > > > > >> @purplecabbage
> >> >> >> > > > > > > > > >> risingj.com
> >> >> >> > > > > > > > > >>
> >> >> >> > > > > > > > > >>
> >> >> >> > > > > > > > > >> On Wed, Sep 25, 2013 at 4:38 PM, Steven Gill <
> >> >> >> > > > > > > stevengil...@gmail.com
> >> >> >> > > > > > > > <javascript:;>
> >> >> >> > > > > > > > > >
> >> >> >> > > > > > > > > >> wrote:
> >> >> >> > > > > > > > > >>
> >> >> >> > > > > > > > > >> > We copied over tests into plugins when we
> first
> >> >> broke
> >> >> >> > them
> >> >> >> > > > > out,
> >> >> >> > > > > > > but
> >> >> >> > > > > > > > I
> >> >> >> > > > > > > > > >> don't
> >> >> >> > > > > > > > > >> > believe they have been updated.
> >> >> >> > > > > > > > > >> >
> >> >> >> > > > > > > > > >> > I would say for now to just add the tests to
> >> >> mobile
> >> >> >> > spec,
> >> >> >> > > > and
> >> >> >> > > > > > > > > possibly in
> >> >> >> > > > > > > > > >> > the future we go all voltron to build mobile
> >> spec
> >> >> and
> >> >> >> > keep
> >> >> >> > > > > tests
> >> >> >> > > > > > > > with
> >> >> >> > > > > > > > > >> their
> >> >> >> > > > > > > > > >> > corresponding plugins.
> >> >> >> > > > > > > > > >> >
> >> >> >> > > > > > > > > >> >
> >> >> >> > > > > > > > > >> > On Wed, Sep 25, 2013 at 4:22 PM, Joe Bowser <
> >> >> >> > > > > bows...@gmail.com
> >> >> >> > > > > > > > <javascript:;>>
> >> >> >> > > > > > > > > wrote:
> >> >> >> > > > > > > > > >> >
> >> >> >> > > > > > > > > >> > > Hey
> >> >> >> > > > > > > > > >> > >
> >> >> >> > > > > > > > > >> > > Right now, I'm working on a weird file
> issue
> >> >> that
> >> >> >> > > requires
> >> >> >> > > > > me
> >> >> >> > > > > > to
> >> >> >> > > > > > > > > >> > > update mobile-spec, but I'm wondering where
> >> the
> >> >> >> tests
> >> >> >> > > > should
> >> >> >> > > > > > > live.
> >> >> >> > > > > > > > > >> > > Should it all keep living in mobile-spec,
> or
> >> is
> >> >> it
> >> >> >> > with
> >> >> >> > > > the
> >> >> >> > > > > > > > plugins.
> >> >> >> > > > > > > > > >> > > And if it's with the plugins, will there be
> >> >> >> scripts to
> >> >> >> > > > > > assemble
> >> >> >> > > > > > > > > >> > > mobile-spec all Voltron style?
> >> >> >> > > > > > > > > >> > >
> >> >> >> > > > > > > > > >> > > This came up earlier, but I haven't found
> any
> >> >> fix
> >> >> >> that
> >> >> >> > > > > needed
> >> >> >> > > > > > a
> >> >> >> > > > > > > > > >> > > mobile-spec test. (Many that need native
> >> >> testing,
> >> >> >> > like
> >> >> >> > > > > > > recursive
> >> >> >> > > > > > > > > file
> >> >> >> > > > > > > > > >> > > copy, etc). Any thoughts?
> >> >> >> > > > > > > > > >> > >
> >> >> >> > > > > > > > > >> > > Joe
> >> >> >> > > > > > > > > >> > >
> >> >> >> > > > > > > > > >> >
> >> >> >> > > > > > > > > >>
> >> >> >> > > > > > > > >
> >> >> >> > > > > > > >
> >> >> >> > > > > > > >
> >> >> >> > > > > > > > --
> >> >> >> > > > > > > > Carlos Santana
> >> >> >> > > > > > > > <csantan...@gmail.com>
> >> >> >> > > > > > > >
> >> >> >> > > > > > >
> >> >> >> > > > > >
> >> >> >> > > > >
> >> >> >> > > >
> >> >> >> > >
> >> >> >> >
> >> >> >>
> >> >> >
> >> >> >
> >> >>
> >> >
> >> >
> >>
> >
> >
>
