On Wed, 22 Sep 2010 15:49:40 +0200, Markus Glaser wrote: > Hi, > > here are my thoughts about phpunit and selenium testing. > >> The wiki under test is set up with a master database consisting of a >> single objectcache table. The entries of this table specify a test run >> identifier as primary key and temporary resource identifiers as >> dependent fields. > If I understand this correctly, this would not allow to test any wikis > that are running on live sites, e.g. intranet wikis. While I agree that > regression testing on live sites is not a good idea, I kind of like the > notion that after setting up a wiki with all the extensions I like to > have, I could do some sort of "everything up and running"-test. With the > concept of using separate testing databases and resources, this would be > possible without interference with the actual data and could even be > done at intervals during, say, maintenance periods.
The problem with testing live sites is tests may alter wiki data (consequently, test run reproducibility becomes a problem). If all of the tests are read-only, then that isn't a problem, but it means developing a whole set of tests that conform to that constraint. Nevertheless, it wouldn't be hard to design the switching mechanism to allow the testing of live sites. There could be an option in test setup and cleanup that effectively says "don't switch-in/clean-up temporary resources." Personally, I think use of this option is dangerous, but it wouldn't be hard to implement. >> Setup of a test run requires the creation of the test run temporary >> resources and a entry in the objectcache table. > Are there already mechanisms for this? I haven't done too much work with > the objectcache. This is where memcached data is stored, right? So how > do I get the data that is needed? This question leads me to another one: > How do I get the testind database and resources? As I see this, it > should be part of the testing framework to be able to produce the set of > data needed from a "normal" MW installation. The whole mechanism would > actually be something like a backup, so we might look into any existing > solutions for that. We should use the existing ObjectCache class to manage the object cache. However, if there exists some switching-in code, I doubt it has corresponding clean-up code. So, we probably need to do some development even if we use existing mechanisms. I think the object cache and memcached are alternative ways of storing persistent data. (I also am not an expert in this, so I could be wrong). My understanding is memcached uses the memcached daemon (http:// memcached.org/), while the object cache uses the underlying database. If so, then memcached data disappears after a system crash or power outage, whereas object cache data should survive. You are absolutely correct that we need to figure out how to clone a set of temporary resources (db, images directory, perhaps cache data) and set them up for use (i.e., so the switch-in logic can copy them for the test run). There are a number of problems to solve, e.g., 1) how do you package the cloned resources (e.g., tar file), 2) how do you efficiently copy them during the switch-in process, 3) security issues, 4) resource management issues. >> When a request is sent to the wiki under test, very early in the >> request processing (e.g., immediately after LocalSettings is processed) >> a hook is called with the provided state information as an argument >> that accesses the objectcache table. The extension function handling >> the hook switches in the temporary resources and returns. > Also, this hook might handle the reconfiguration of the wiki, if needed. > So for example, testing the PagedTiffHandler requires uploading of tiff > files to be enabled. However, there might be some security risks, since > it is not directly obvious in the code which settings are changed. So > the hook should only be called when $wgEnableSelenium = true. In > addition, we could define a DontTouchThisSettings.php, which is called > even after the hook and holds some settings that are holy to the admin > of the wiki :) The question is, though, would this not become somewhat > too complicated? I have done some research on possible hooks. I think the SetupAfterCache hook is a good candidate. The problem with calling a (psuedo-)hook in LocalSettings is there is some setup not completed when LocatSettings is processed (e.g., the WebRequest object is not yet available. the ObjectCache class file is not yet included). Most setup is completed before calling the SetupAfterCache hook. It would be possible to do some configuration in this hook, but I think we need to consider efficiency. Preferably, the creation of the base resources that are copied during the switch-in will be configured as much as possible. For example, I think creating the base for PagedTiffHandler should start with a freshly installed wiki and upload multipage.tiff. The resulting db could be dumped and the images directory as well as other resources copied. The result could them be tar'd and uploaded as the base for the PagedTiffHandler test suite. This would relieve the test set up code of uploading multipage.tiff on each test run. We may even consider pre-installing the base db so that the switch-in code can use db functions to clone it, rather than creating it from the base dump for each test run that uses it. The latter could take a significant amount of time, thereby severely slowing testing. >> After the test run completes, the testing application cleans up the >> test run by requesting the deletion of the temporary resources and the >> objectcache table entry associated with the test run. > In some cases, tests will not change any data, e.g. testing dynamic skin > elements in vector skin. Would it make sense not to tear down the > testing environment in that case in order to save some time when testing > repeatedly? I think, there is a conflict between performance and amount > of data, but who wins? There may be ways to make things more efficient by not immediately deleting the temporary resources. However, eventually we have to delete them. So, there is a question of where the temporary resources identifier is stored so it can be used later for a clean-up request. I was assuming the switch-in request occurs in test suite start-up and the clean-up request occurs in the test suite finishing code. But, we should consider alternative implementation strategies. > In general, it seems to me that we have some similarity with what is > called wiki family on mediawiki.org. One could see multiple testing > environments as a set of multiple wikis that share a common codebase > [1]. Does anybody have experience with wiki families and the object > cache rsp. memcached? I agree. (In fact, I mentioned this previously). No need to reinvent the wheel. > I am not sure whether we can use the same codebase as parsertests. I'd > rather think, parser tests are a special case of what we are sketching > here. On the other hand, I don't think it is a good idea to have two > separate approaches for very similar tasks in the code. Do you think it > would be feasible to separate the preparation part from both parser > tests and selenium tests and build both of them on a common ground? > The parserTests code creates some temporary tables in an existing wiki database. Currently, these tables are empty and prefixed with a static identifier. The requirements for parserTests and selenium tests are significantly different. While we may be able to learn from the parserTests code, we would have to change the parserTest code significantly in order to use it. I think it would actually be less work to start from scratch. Of course, as you mention, there may be code used to support wiki families that we could use without much modification. -- -- Dan Nessett _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
