On Wed, 22 Sep 2010 11:00:53 -0700, Brion Vibber wrote: > Hmm, I think you guys are overthinking the details on this; let's step > back a level. > > When you're running tests, you have these tasks: * create a blank-slate > wiki to run tests in * populate the empty wiki with known data * run > tests on this wiki in a known state * clean up in some way > > The parserTests system is designed with some particular constraints: * > run within the local MediaWiki test instance a developer already has, > without altering its contents > * run all tests in a single command-line program * expose no in-test > data to the outside world > > It can do this very easily using temporary tables and a temporary > directory because it only needs to work within that one single test > process; once it's done, it's done and all the temporary data can be > discarded. Nothing needs to be kept across processes or exposed to > clients. > > For Selenium tests, you have a very different set of constraints: * test > data must be exposed to web clients over a web server * test data must > be retained across multiple requests > > The simplest way to accomplish this is to have a dedicated, web-exposed > test wiki instance. A test run would go like this: * (re)initialize test > wiki into known state * run series of tests > > I'd recommend not trying to use the parserTest harness/initialization > for this; it'll be a *lot* simpler to just script creation of a fresh > wiki. (drop database, create database, slurp tables, run update.php) > > Some non-destructive tests can always be run on any existing instance -- > and probably should be! -- and some 'active' tests will be freely > runnable on existing instances that are used for development and > testing, but if you want to work with a blank slate wiki exposed to web > clients, keep things simple and just make a dedicated instance. > > -- brion > > > On Wed, Sep 22, 2010 at 9:47 AM, Dan Nessett <[email protected]> wrote: > >> On Wed, 22 Sep 2010 15:49:40 +0200, Markus Glaser wrote: >> >> > Hi, >> > >> > here are my thoughts about phpunit and selenium testing. >> > >> >> The wiki under test is set up with a master database consisting of a >> >> single objectcache table. The entries of this table specify a test >> >> run identifier as primary key and temporary resource identifiers as >> >> dependent fields. >> > If I understand this correctly, this would not allow to test any >> > wikis that are running on live sites, e.g. intranet wikis. While I >> > agree that regression testing on live sites is not a good idea, I >> > kind of like the notion that after setting up a wiki with all the >> > extensions I like to have, I could do some sort of "everything up and >> > running"-test. With the concept of using separate testing databases >> > and resources, this would be possible without interference with the >> > actual data and could even be done at intervals during, say, >> > maintenance periods. >> >> The problem with testing live sites is tests may alter wiki data >> (consequently, test run reproducibility becomes a problem). If all of >> the tests are read-only, then that isn't a problem, but it means >> developing a whole set of tests that conform to that constraint. >> >> Nevertheless, it wouldn't be hard to design the switching mechanism to >> allow the testing of live sites. There could be an option in test setup >> and cleanup that effectively says "don't switch-in/clean-up temporary >> resources." Personally, I think use of this option is dangerous, but it >> wouldn't be hard to implement. >> >> >> Setup of a test run requires the creation of the test run temporary >> >> resources and a entry in the objectcache table. >> > Are there already mechanisms for this? I haven't done too much work >> > with the objectcache. This is where memcached data is stored, right? >> > So how do I get the data that is needed? This question leads me to >> > another one: How do I get the testind database and resources? As I >> > see this, it should be part of the testing framework to be able to >> > produce the set of data needed from a "normal" MW installation. The >> > whole mechanism would actually be something like a backup, so we >> > might look into any existing solutions for that. >> >> We should use the existing ObjectCache class to manage the object >> cache. However, if there exists some switching-in code, I doubt it has >> corresponding clean-up code. So, we probably need to do some >> development even if we use existing mechanisms. >> >> I think the object cache and memcached are alternative ways of storing >> persistent data. (I also am not an expert in this, so I could be >> wrong). My understanding is memcached uses the memcached daemon >> (http:// memcached.org/), while the object cache uses the underlying >> database. If so, then memcached data disappears after a system crash or >> power outage, whereas object cache data should survive. >> >> You are absolutely correct that we need to figure out how to clone a >> set of temporary resources (db, images directory, perhaps cache data) >> and set them up for use (i.e., so the switch-in logic can copy them for >> the test run). There are a number of problems to solve, e.g., 1) how do >> you package the cloned resources (e.g., tar file), 2) how do you >> efficiently copy them during the switch-in process, 3) security issues, >> 4) resource management issues. >> >> >> When a request is sent to the wiki under test, very early in the >> >> request processing (e.g., immediately after LocalSettings is >> >> processed) a hook is called with the provided state information as >> >> an argument that accesses the objectcache table. The extension >> >> function handling the hook switches in the temporary resources and >> >> returns. >> > Also, this hook might handle the reconfiguration of the wiki, if >> > needed. So for example, testing the PagedTiffHandler requires >> > uploading of tiff files to be enabled. However, there might be some >> > security risks, since it is not directly obvious in the code which >> > settings are changed. So the hook should only be called when >> > $wgEnableSelenium = true. In addition, we could define a >> > DontTouchThisSettings.php, which is called even after the hook and >> > holds some settings that are holy to the admin of the wiki :) The >> > question is, though, would this not become somewhat too complicated? >> >> I have done some research on possible hooks. I think the >> SetupAfterCache hook is a good candidate. The problem with calling a >> (psuedo-)hook in LocalSettings is there is some setup not completed >> when LocatSettings is processed (e.g., the WebRequest object is not yet >> available. the ObjectCache class file is not yet included). Most setup >> is completed before calling the SetupAfterCache hook. >> >> It would be possible to do some configuration in this hook, but I think >> we need to consider efficiency. Preferably, the creation of the base >> resources that are copied during the switch-in will be configured as >> much as possible. For example, I think creating the base for >> PagedTiffHandler should start with a freshly installed wiki and upload >> multipage.tiff. The resulting db could be dumped and the images >> directory as well as other resources copied. The result could them be >> tar'd and uploaded as the base for the PagedTiffHandler test suite. >> This would relieve the test set up code of uploading multipage.tiff on >> each test run. >> >> We may even consider pre-installing the base db so that the switch-in >> code can use db functions to clone it, rather than creating it from the >> base dump for each test run that uses it. The latter could take a >> significant amount of time, thereby severely slowing testing. >> >> >> After the test run completes, the testing application cleans up the >> >> test run by requesting the deletion of the temporary resources and >> >> the objectcache table entry associated with the test run. >> > In some cases, tests will not change any data, e.g. testing dynamic >> > skin elements in vector skin. Would it make sense not to tear down >> > the testing environment in that case in order to save some time when >> > testing repeatedly? I think, there is a conflict between performance >> > and amount of data, but who wins? >> >> There may be ways to make things more efficient by not immediately >> deleting the temporary resources. However, eventually we have to delete >> them. So, there is a question of where the temporary resources >> identifier is stored so it can be used later for a clean-up request. I >> was assuming the switch-in request occurs in test suite start-up and >> the clean-up request occurs in the test suite finishing code. But, we >> should consider alternative implementation strategies. >> >> > In general, it seems to me that we have some similarity with what is >> > called wiki family on mediawiki.org. One could see multiple testing >> > environments as a set of multiple wikis that share a common codebase >> > [1]. Does anybody have experience with wiki families and the object >> > cache rsp. memcached? >> >> I agree. (In fact, I mentioned this previously). No need to reinvent >> the wheel. >> >> > I am not sure whether we can use the same codebase as parsertests. >> > I'd rather think, parser tests are a special case of what we are >> > sketching here. On the other hand, I don't think it is a good idea to >> > have two separate approaches for very similar tasks in the code. Do >> > you think it would be feasible to separate the preparation part from >> > both parser tests and selenium tests and build both of them on a >> > common ground? >> > >> > >> The parserTests code creates some temporary tables in an existing wiki >> database. Currently, these tables are empty and prefixed with a static >> identifier. The requirements for parserTests and selenium tests are >> significantly different. While we may be able to learn from the >> parserTests code, we would have to change the parserTest code >> significantly in order to use it. I think it would actually be less >> work to start from scratch. Of course, as you mention, there may be >> code used to support wiki families that we could use without much >> modification. >> >> -- >> -- Dan Nessett >> >> >> _______________________________________________ Wikitech-l mailing list >> [email protected] >> https://lists.wikimedia.org/mailman/listinfo/wikitech-l >>
100% agreement. Some have mentioned the possibility of using the wiki family logic to help achieve these objectives. Do you have any thoughts on this? If you think it is a good idea, how do we find out more about it? -- -- Dan Nessett _______________________________________________ Wikitech-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikitech-l
