Re: [Wikitech-l] using parserTests code for selenium test framework
Hi, Automatic testing with a clean database sounds good, the scheme looks reasonable. Will it be possible to have cleaning of separate databases, used by extensions, added, also? In the context of wiki identification, I was wondering: At the moment I am mainly testing SMW instances having the selenium-server, the application under test, and the tests each on the same machine. Wiki identification is no issue, here. Of course, it would be great to eventually have the WM/extension testing on external (possibly Wikimania) infrastructure, resulting in higher security requirements; Is this planned or much considered in the discussions? Keep up the good work! Regards, Benedikt -- Karlsruhe Institute of Technology (KIT) Institute of Applied Informatics and Formal Description Methods (AIFB) Benedikt Kämpgen Research Associate Kaiserstraße 12 Building 11.40 76131 Karlsruhe, Germany Phone: +49 721 608-7946 Fax: +49 721 608-6580 Email: benedikt.kaemp...@kit.edu Web: http://www.kit.edu/ KIT University of the State of Baden-Wuerttemberg and National Research Center of the Helmholtz Association -Original Message- From: wikitech-l-boun...@lists.wikimedia.org [mailto:wikitech-l-boun...@lists.wikimedia.org] On Behalf Of Markus Glaser Sent: Friday, October 15, 2010 3:54 PM To: Wikimedia developers Subject: Re: [Wikitech-l] using parserTests code for selenium test framework Hi, I recently suggested some scheme for dynamically creating clean wikis for Selenium tests, which can be found here: http://www.mediawiki.org/wiki/SeleniumFramework#Testing_with_a_clean_databas e_and_file_state After some discussion in the Testing group, I would like to elaborate a bit further on some of the steps that need to be done: 2.2 Create temporal resources 2.2.1 create a new database with name se+testID 2.2.2 create a new images folder with name se+testID 2.2.3 populate database and images with template data. there is a standard (vanilla) template, but also, test suites can have their own templates, which then would be used. this test data should be placed in the same folder as the tests, with the same name. 2.3 Create test tracker with timestamp I suggest we use a textfile called se+testID.txt in a folder wiki/seRunningTests. The timestamp would be the creation date of the file. The next important question is, how should the wiki be identified? Brion suggested using a subdomain, e.g. sn+testID.yourwiki.org. If I understand webserver correctly, however, this would need some specific setup. In the selenium testing group we were discussion identifying the wiki via cookie. So the wiki under test would read the cookie and reconfigure accordingly. The reconfiguration would need to take place right after LocalSettings.php, since the following call to Setup.php already assumes some configurations as set. As far as I know, Priyanka already has written some code to do this. 3.1 start testsuites via selenium 3.1.1 First, the SeleniumTestRunner needs to store the testID in order to identify the ressources for teardown. 3.1.2 Start the test suite 5.1 send teardown request fetch the testID stored in 3.1.1 and request the wiki under test to teardown the ressources for that id I would be very happy about comments and thoughts. Are we heading in the right direction? Cheers, Markus -Ursprüngliche Nachricht- Von: wikitech-l-boun...@lists.wikimedia.org [mailto:wikitech-l-boun...@lists.wikimedia.org] Im Auftrag von Markus Glaser Gesendet: Mittwoch, 29. September 2010 20:02 An: Wikimedia developers Betreff: Re: [Wikitech-l] using parserTests code for selenium test framework Hi, since the wiki under test is not neccessarily the wiki running the test, it might be useful to visualize that (I have numbered the individual steps to make reference to them easier in the discussion): testrunner wiki under test -- --- 1.1 start selenium which in turn starts a browser to talk to the wiki under test 1.2 send request for new test with unique test id and tests that will be fired 2.1 create cookie with test id 2.2 create temporal resources according to tests list 2.3 create test tracker with timestamp 2.4 return success code 3.1 start testsuites via selenium 3.2 send a lot of individual requests according to the tests 4.1 testrunner is identified by test id 4.2 reconfigure database and resources according to test id 4.3 ? Do something with memcached ? 4.4 execute request 4.5 update timestamp in test tracker 5.1 send a teardown request 6.1 execute teardown, i.e. delete
Re: [Wikitech-l] using parserTests code for selenium test framework
Hi, I recently suggested some scheme for dynamically creating clean wikis for Selenium tests, which can be found here: http://www.mediawiki.org/wiki/SeleniumFramework#Testing_with_a_clean_database_and_file_state After some discussion in the Testing group, I would like to elaborate a bit further on some of the steps that need to be done: 2.2 Create temporal resources 2.2.1 create a new database with name se+testID 2.2.2 create a new images folder with name se+testID 2.2.3 populate database and images with template data. there is a standard (vanilla) template, but also, test suites can have their own templates, which then would be used. this test data should be placed in the same folder as the tests, with the same name. 2.3 Create test tracker with timestamp I suggest we use a textfile called se+testID.txt in a folder wiki/seRunningTests. The timestamp would be the creation date of the file. The next important question is, how should the wiki be identified? Brion suggested using a subdomain, e.g. sn+testID.yourwiki.org. If I understand webserver correctly, however, this would need some specific setup. In the selenium testing group we were discussion identifying the wiki via cookie. So the wiki under test would read the cookie and reconfigure accordingly. The reconfiguration would need to take place right after LocalSettings.php, since the following call to Setup.php already assumes some configurations as set. As far as I know, Priyanka already has written some code to do this. 3.1 start testsuites via selenium 3.1.1 First, the SeleniumTestRunner needs to store the testID in order to identify the ressources for teardown. 3.1.2 Start the test suite 5.1 send teardown request fetch the testID stored in 3.1.1 and request the wiki under test to teardown the ressources for that id I would be very happy about comments and thoughts. Are we heading in the right direction? Cheers, Markus -Ursprüngliche Nachricht- Von: wikitech-l-boun...@lists.wikimedia.org [mailto:wikitech-l-boun...@lists.wikimedia.org] Im Auftrag von Markus Glaser Gesendet: Mittwoch, 29. September 2010 20:02 An: Wikimedia developers Betreff: Re: [Wikitech-l] using parserTests code for selenium test framework Hi, since the wiki under test is not neccessarily the wiki running the test, it might be useful to visualize that (I have numbered the individual steps to make reference to them easier in the discussion): testrunner wiki under test -- --- 1.1 start selenium which in turn starts a browser to talk to the wiki under test 1.2 send request for new test with unique test id and tests that will be fired 2.1 create cookie with test id 2.2 create temporal resources according to tests list 2.3 create test tracker with timestamp 2.4 return success code 3.1 start testsuites via selenium 3.2 send a lot of individual requests according to the tests 4.1 testrunner is identified by test id 4.2 reconfigure database and resources according to test id 4.3 ? Do something with memcached ? 4.4 execute request 4.5 update timestamp in test tracker 5.1 send a teardown request 6.1 execute teardown, i.e. delete all resources associated with test id 6.2 delete test tracker 6.3 return success code 7.1 stop selenium Now, if something breaks during the test, the test tracker will not be deleted and can serve as as basis for a cleanup procedure that is triggered by a cronjob. Is this something we can all agree on? I assume, steps 2.2 (setting up temporary test data) and 4.2 (find a mechanism to actually use the test data) will be the ones we have to work on now. Regards, Markus -Ursprüngliche Nachricht- Von: wikitech-l-boun...@lists.wikimedia.org [mailto:wikitech-l-boun...@lists.wikimedia.org] Im Auftrag von Ryan Lane Gesendet: Freitag, 24. September 2010 20:22 An: Wikimedia developers Betreff: Re: [Wikitech-l] using parserTests code for selenium test framework Here is all that is required: * a single wildcard entry in Apache configuration * one or two lines in LocalSettings.php to pull a DB name from the hostname/path/CLI parameters. As for cleaning up resources to keep the machine from getting clogged, it's very unlikely that your test wikis will fill up a multi-hundred-gigabyte drive in the middle of a run. If you find that they do, there's still no need to tie cleanup of any particular run to any particular other run. All you need to know
Re: [Wikitech-l] using parserTests code for selenium test framework
Hi, since the wiki under test is not neccessarily the wiki running the test, it might be useful to visualize that (I have numbered the individual steps to make reference to them easier in the discussion): testrunner wiki under test -- --- 1.1 start selenium which in turn starts a browser to talk to the wiki under test 1.2 send request for new test with unique test id and tests that will be fired 2.1 create cookie with test id 2.2 create temporal resources according to tests list 2.3 create test tracker with timestamp 2.4 return success code 3.1 start testsuites via selenium 3.2 send a lot of individual requests according to the tests 4.1 testrunner is identified by test id 4.2 reconfigure database and resources according to test id 4.3 ? Do something with memcached ? 4.4 execute request 4.5 update timestamp in test tracker 5.1 send a teardown request 6.1 execute teardown, i.e. delete all resources associated with test id 6.2 delete test tracker 6.3 return success code 7.1 stop selenium Now, if something breaks during the test, the test tracker will not be deleted and can serve as as basis for a cleanup procedure that is triggered by a cronjob. Is this something we can all agree on? I assume, steps 2.2 (setting up temporary test data) and 4.2 (find a mechanism to actually use the test data) will be the ones we have to work on now. Regards, Markus -Ursprüngliche Nachricht- Von: wikitech-l-boun...@lists.wikimedia.org [mailto:wikitech-l-boun...@lists.wikimedia.org] Im Auftrag von Ryan Lane Gesendet: Freitag, 24. September 2010 20:22 An: Wikimedia developers Betreff: Re: [Wikitech-l] using parserTests code for selenium test framework Here is all that is required: * a single wildcard entry in Apache configuration * one or two lines in LocalSettings.php to pull a DB name from the hostname/path/CLI parameters. As for cleaning up resources to keep the machine from getting clogged, it's very unlikely that your test wikis will fill up a multi-hundred-gigabyte drive in the middle of a run. If you find that they do, there's still no need to tie cleanup of any particular run to any particular other run. All you need to know is which runs have completed and can now be cleaned up. I'd like to add some ideas to this thread that were discussed in the Selenium meeting this morning. The basic plan we discussed (and I'm sure I'll be corrected some on this) is as follows: When a run begins, it registers itself with the wiki and gets a session back. The wiki software, on creating the session, makes a new run specific wiki using the wiki family method. The test will pass both the session cookie, and a test type cookie, which will dynamically configure the wiki as the tests run. When the run is complete, it should notify the wiki that the test run is complete. The wiki software will then destroy the session and the dynamically created resources. If a run doesn't complete for some reason, a cron can clean up resources that haven't been used in some appropriate amount of time. Respectfully, Ryan Lane ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
2010/9/23 Brion Vibber br...@pobox.com: (If using memcached, be sure to clear those out, reinitialize, or otherwise do something that forces old values to be cleared or ignored.) $wgCacheEpoch is a good one for this. The easiest way to change it is to touch LocalSettings.php. Roan Kattouw (Catrope) ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
Dan, I think you're overestimating the difficulty of the basic wiki family method. Here is all that is required: * a single wildcard entry in Apache configuration * one or two lines in LocalSettings.php to pull a DB name from the hostname/path/CLI parameters. As for cleaning up resources to keep the machine from getting clogged, it's very unlikely that your test wikis will fill up a multi-hundred-gigabyte drive in the middle of a run. If you find that they do, there's still no need to tie cleanup of any particular run to any particular other run. All you need to know is which runs have completed and can now be cleaned up. -- brion ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
Here is all that is required: * a single wildcard entry in Apache configuration * one or two lines in LocalSettings.php to pull a DB name from the hostname/path/CLI parameters. As for cleaning up resources to keep the machine from getting clogged, it's very unlikely that your test wikis will fill up a multi-hundred-gigabyte drive in the middle of a run. If you find that they do, there's still no need to tie cleanup of any particular run to any particular other run. All you need to know is which runs have completed and can now be cleaned up. I'd like to add some ideas to this thread that were discussed in the Selenium meeting this morning. The basic plan we discussed (and I'm sure I'll be corrected some on this) is as follows: When a run begins, it registers itself with the wiki and gets a session back. The wiki software, on creating the session, makes a new run specific wiki using the wiki family method. The test will pass both the session cookie, and a test type cookie, which will dynamically configure the wiki as the tests run. When the run is complete, it should notify the wiki that the test run is complete. The wiki software will then destroy the session and the dynamically created resources. If a run doesn't complete for some reason, a cron can clean up resources that haven't been used in some appropriate amount of time. Respectfully, Ryan Lane ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
On Wed, 22 Sep 2010 12:30:35 -0700, Brion Vibber wrote: On Wed, Sep 22, 2010 at 11:09 AM, Dan Nessett dness...@yahoo.com wrote: Some have mentioned the possibility of using the wiki family logic to help achieve these objectives. Do you have any thoughts on this? If you think it is a good idea, how do we find out more about it? I'd just treat it same as any other wiki. Whether you're running one or multiple wiki instances out of one copy of the code base, it just doesn't make a difference here. -- brion Not to hammer this point to hard, but there is another reason for supporting switching in of temporary resources. We have one worked example of a Selenium test - PagedTiffHandler. During the course of its execution, it uploads some tiff files to test some of its functionality. Currently, once an image is uploded, there is no way to completely delete it (short of drastic administrative action involving raw database manipulation). So, running PagedTiffHandler twice doesn't work unless you switch in temporary resources on each run that are then deleted at the end of the run. -- -- Dan Nessett ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
Given a test matrix with multiple OSes, this ain't something individual devs will be running in full over and over as they work. Assume automated batch runs, which can be distributed over as many databases and clients as you like. For small test subsets that are being used during testing the equation still doesn't change much: reset the wiki to known state, run the tests. Keep it simple! -- brion On Thursday, September 23, 2010, Dan Nessett dness...@yahoo.com wrote: On Wed, 22 Sep 2010 12:30:35 -0700, Brion Vibber wrote: On Wed, Sep 22, 2010 at 11:09 AM, Dan Nessett dness...@yahoo.com wrote: Some have mentioned the possibility of using the wiki family logic to help achieve these objectives. Do you have any thoughts on this? If you think it is a good idea, how do we find out more about it? I'd just treat it same as any other wiki. Whether you're running one or multiple wiki instances out of one copy of the code base, it just doesn't make a difference here. -- brion Not to hammer this point to hard, but there is another reason for supporting switching in of temporary resources. We have one worked example of a Selenium test - PagedTiffHandler. During the course of its execution, it uploads some tiff files to test some of its functionality. Currently, once an image is uploded, there is no way to completely delete it (short of drastic administrative action involving raw database manipulation). So, running PagedTiffHandler twice doesn't work unless you switch in temporary resources on each run that are then deleted at the end of the run. -- -- Dan Nessett ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
On Thu, 23 Sep 2010 09:24:18 -0700, Brion Vibber wrote: Given a test matrix with multiple OSes, this ain't something individual devs will be running in full over and over as they work. Assume automated batch runs, which can be distributed over as many databases and clients as you like. For small test subsets that are being used during testing the equation still doesn't change much: reset the wiki to known state, run the tests. Keep it simple! -- brion On Thursday, September 23, 2010, Dan Nessett dness...@yahoo.com wrote: On Wed, 22 Sep 2010 12:30:35 -0700, Brion Vibber wrote: On Wed, Sep 22, 2010 at 11:09 AM, Dan Nessett dness...@yahoo.com wrote: Some have mentioned the possibility of using the wiki family logic to help achieve these objectives. Do you have any thoughts on this? If you think it is a good idea, how do we find out more about it? I'd just treat it same as any other wiki. Whether you're running one or multiple wiki instances out of one copy of the code base, it just doesn't make a difference here. -- brion Not to hammer this point to hard, but there is another reason for supporting switching in of temporary resources. We have one worked example of a Selenium test - PagedTiffHandler. During the course of its execution, it uploads some tiff files to test some of its functionality. Currently, once an image is uploded, there is no way to completely delete it (short of drastic administrative action involving raw database manipulation). So, running PagedTiffHandler twice doesn't work unless you switch in temporary resources on each run that are then deleted at the end of the run. -- -- Dan Nessett ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l I am very much in favor of keeping it simple. I think the issue is whether we will support more than one regression test (or individual test associated with a regression test) running concurrently on the same test wiki. If not, then I agree, no switching logic is necessary. -- -- Dan Nessett ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
Hi. Other good test software is JMeter http://jakarta.apache.org/jmeter/ You could record and replay your http requests -- Lcdo. Wilfredo Rafael Rodríguez Hernández msn,googletalk = wilfre...@gmail.com cv = http://www.wilfredor.co.cc blog = http://wilfredor.blogspot.com fotos = http://picasaweb.google.com/wilfredor/ ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
On Thu, Sep 23, 2010 at 9:46 AM, Dan Nessett dness...@yahoo.com wrote:
I am very much in favor of keeping it simple. I think the issue is
whether we will support more than one regression test (or individual test
associated with a regression test) running concurrently on the same test
wiki. If not, then I agree, no switching logic is necessary.
*nod*
It might be a good idea to divide the test set up into, say 'channels' or
'bundles' which are independent of each other, but whose individual steps
must run in sequence. If the tests are designed well, you should be able to
run tests from multiple 'channels' on the same wiki simultaneously -- just
as in the real world, multiple users are doing multiple things on your wiki
at the same time.
So one test set might be:
* create page A-{{unique-id}} as user One-{{unique-id}}
* open editing page as user One-{{unique-id}}
* open and save the page as user Two-{{unique-id}}
* save the page as user One-{{unique-id}}
* confirm edit conflict / merging behavior was as expected
And another might be:
* register a new user account User--{{unique-id}}
* change skin option in preferences
* confirm that the skin changed as expected
These tests don't interfere with each other -- indeed if they did, that
would be information you'd need to know about a serious bug!
Most test sets should be fairly separate like this; only some that change
global state (say, a site administrator using a global configuration panel
to change the default skin) would need to be run separately.
-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
On Thu, 23 Sep 2010 10:29:58 -0700, Brion Vibber wrote:
On Thu, Sep 23, 2010 at 9:46 AM, Dan Nessett dness...@yahoo.com wrote:
I am very much in favor of keeping it simple. I think the issue is
whether we will support more than one regression test (or individual
test associated with a regression test) running concurrently on the
same test wiki. If not, then I agree, no switching logic is necessary.
*nod*
It might be a good idea to divide the test set up into, say 'channels'
or 'bundles' which are independent of each other, but whose individual
steps must run in sequence. If the tests are designed well, you should
be able to run tests from multiple 'channels' on the same wiki
simultaneously -- just as in the real world, multiple users are doing
multiple things on your wiki at the same time.
So one test set might be:
* create page A-{{unique-id}} as user One-{{unique-id}} * open editing
page as user One-{{unique-id}} * open and save the page as user
Two-{{unique-id}} * save the page as user One-{{unique-id}} * confirm
edit conflict / merging behavior was as expected
And another might be:
* register a new user account User--{{unique-id}} * change skin option
in preferences
* confirm that the skin changed as expected
These tests don't interfere with each other -- indeed if they did, that
would be information you'd need to know about a serious bug!
Most test sets should be fairly separate like this; only some that
change global state (say, a site administrator using a global
configuration panel to change the default skin) would need to be run
separately.
-- brion
After thinking about this some more I think you are right. We should at
least start with something simple and only make it more complex (e.g.,
wiki resource switching) if the simple approach has significant problems.
There is already a way to 'bundle' individual tests together. It is the
selenium test suite. We could use that. We could then break up a
regression test into separate test suites that could run concurrently.
Summarizing, here is my understanding of your proposal:
+ A regression test run comprises a set of test suites, each of which may
run concurrently.
+ If you want to run multiple regression tests concurrently, use
different test wikis (which can run over the same code base, but which
are identified by different URLs - i.e., rely on httpd to multiplex
multiple concurrent regression tests).
+ If you want to run parts of a regression test concurrently, the unit of
concurrency is the test suite.
+ A regression test begins by establishing a fresh wiki. Each test suite
starts by establishing the wiki state it requires (e.g., for
PagedTiffHanlder, loading Multipage.tiff).
+ It is an open question whether test suite or total regression test
cleanup is necessary. It may be possible to elide this step and simply
rely on regression test initialization to cleanup any wiki state left
around by a previous test run.
There are still some open questions:
+ How do you establish a fresh wiki for a URL used previously for a test
run?
+ URLs identify test wikis. Only one regression test can run at time on
any one of these. How do you synchronize regression test initiation so
there is some sort of lock on a test wiki currently running a regression
test?
--
-- Dan Nessett
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
On Thu, Sep 23, 2010 at 1:04 PM, Dan Nessett dness...@yahoo.com wrote: After thinking about this some more I think you are right. We should at least start with something simple and only make it more complex (e.g., wiki resource switching) if the simple approach has significant problems. There is already a way to 'bundle' individual tests together. It is the selenium test suite. We could use that. We could then break up a regression test into separate test suites that could run concurrently. Summarizing, here is my understanding of your proposal: [snip] That all sounds great to me! There are still some open questions: + How do you establish a fresh wiki for a URL used previously for a test run? If reusing wikis, simplest way would be to just remove the old one and reinitialize it: rm -rf /path/to/images/foowiki mysql -e 'drop database foowiki; create database foowiki; use foowiki; source db/tables.sql;' php maintenance/update.php foowiki (If using memcached, be sure to clear those out, reinitialize, or otherwise do something that forces old values to be cleared or ignored.) + URLs identify test wikis. Only one regression test can run at time on any one of these. How do you synchronize regression test initiation so there is some sort of lock on a test wiki currently running a regression test? Simplest way would be to have one wiki (database + URL) for each regression test (test1234wiki), or even for each run of each regression test (test1234run432wiki). These could be created/removed as needed through simple shell scripting. -- brion ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
On Thu, 23 Sep 2010 14:10:24 -0700, Brion Vibber wrote: + URLs identify test wikis. Only one regression test can run at time on any one of these. How do you synchronize regression test initiation so there is some sort of lock on a test wiki currently running a regression test? Simplest way would be to have one wiki (database + URL) for each regression test (test1234wiki), or even for each run of each regression test (test1234run432wiki). These could be created/removed as needed through simple shell scripting. -- brion Not sure I get this. Here is what I understand would happen when a developer checks in a revision: + A script runs that manages the various regression tests run on the revision (e.g., parserTests, PHPUnit tests, the Selenium-based regression test). + The Selenium regression test needs a URL to work with. There are a fixed set of these defined in httpd.conf. + Before assigning one of these to the regression test run, there is a requirement that it isn't currently busy running a regression test for a different revision. So, you need resource access control on the URLs. + Once you have an idle URL, you can initialize the wiki per your previous comments, including loading the revision into the directory associated with the URL. How does this fit into the idea of using a wiki per regression test or regression test run? -- -- Dan Nessett ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
On Thu, Sep 23, 2010 at 2:31 PM, Dan Nessett dness...@yahoo.com wrote: Not sure I get this. Here is what I understand would happen when a developer checks in a revision: + A script runs that manages the various regression tests run on the revision (e.g., parserTests, PHPUnit tests, the Selenium-based regression test). + The Selenium regression test needs a URL to work with. There are a fixed set of these defined in httpd.conf. There's no need to have a fixed set of URLs; just as with Wikimedia's public-hosted sites you can add individually-addressable wikis dynamically at whim without touching any Apache configuration. URL rewriting, or wildcard hostnames, or whatever lets you make as many distinct URLs as you like, funnel them through a single web server and a single codebase, but have them running with different databases. en.wikipedia.org and es.wikipedia.org don't need separate entries, and neither would test1234.r56789.wikimediatesting.org and test1235.r56789.wikimediatesting.org + Before assigning one of these to the regression test run, there is a requirement that it isn't currently busy running a regression test for a different revision. So, you need resource access control on the URLs. If each test run has its own wiki, there's no need to control access -- no other tests would be trying to access it. -- brion ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
On Thu, 23 Sep 2010 14:41:32 -0700, Brion Vibber wrote: On Thu, Sep 23, 2010 at 2:31 PM, Dan Nessett dness...@yahoo.com wrote: Not sure I get this. Here is what I understand would happen when a developer checks in a revision: + A script runs that manages the various regression tests run on the revision (e.g., parserTests, PHPUnit tests, the Selenium-based regression test). + The Selenium regression test needs a URL to work with. There are a fixed set of these defined in httpd.conf. There's no need to have a fixed set of URLs; just as with Wikimedia's public-hosted sites you can add individually-addressable wikis dynamically at whim without touching any Apache configuration. URL rewriting, or wildcard hostnames, or whatever lets you make as many distinct URLs as you like, funnel them through a single web server and a single codebase, but have them running with different databases. -- brion Are there instructions somewhere that describe how to do this? -- -- Dan Nessett ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
On Thu, Sep 23, 2010 at 2:54 PM, Dan Nessett dness...@yahoo.com wrote: On Thu, 23 Sep 2010 14:41:32 -0700, Brion Vibber wrote: There's no need to have a fixed set of URLs; just as with Wikimedia's public-hosted sites you can add individually-addressable wikis dynamically at whim without touching any Apache configuration. URL rewriting, or wildcard hostnames, or whatever lets you make as many distinct URLs as you like, funnel them through a single web server and a single codebase, but have them running with different databases. -- brion Are there instructions somewhere that describe how to do this? http://www.mediawiki.org/wiki/Manual:Wiki_family#Wikimedia_Method -- brion ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
On Thu, 23 Sep 2010 15:50:48 -0700, Brion Vibber wrote: On Thu, Sep 23, 2010 at 2:54 PM, Dan Nessett dness...@yahoo.com wrote: On Thu, 23 Sep 2010 14:41:32 -0700, Brion Vibber wrote: There's no need to have a fixed set of URLs; just as with Wikimedia's public-hosted sites you can add individually-addressable wikis dynamically at whim without touching any Apache configuration. URL rewriting, or wildcard hostnames, or whatever lets you make as many distinct URLs as you like, funnel them through a single web server and a single codebase, but have them running with different databases. -- brion Are there instructions somewhere that describe how to do this? http://www.mediawiki.org/wiki/Manual:Wiki_family#Wikimedia_Method -- brion Thinking about this a bit, we seem to have come full circle. If we use a URL per regression test run, then we need to multiplex wiki resources. When you set up a wiki family, the resources are permanent. But, for a test run, you need to set them up, use them and then reclaim them. The resources are the db, the images directory, cache data, etc. So, either we must use the fixed URL scheme I mentioned previously, or we are back to resource switching (admittedly using the approach already in place for the wikipedia family). The test run can set up the resources and reclaim them, but we also need to handle test runs that fail in the middle (due to e.g. system OS crashes, power outages, etc.). The resources allocated for such runs will become orphans and some sort of garbage collection is required. -- -- Dan Nessett ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
On Thu, Sep 23, 2010 at 4:03 PM, Dan Nessett dness...@yahoo.com wrote: Thinking about this a bit, we seem to have come full circle. If we use a URL per regression test run, then we need to multiplex wiki resources. When you set up a wiki family, the resources are permanent. But, for a test run, you need to set them up, use them and then reclaim them. The resources are the db, the images directory, cache data, etc. Computers can delete files as well as create them. Drop the database and remove the uploads directory, now it's gone *poof magic*. Nothing has to be multiplexed or locked: you create it before you start using it, and if you don't need it you can remove it when you're done with it. Nothing else will try to use it because nothing else has the same test test run ID -- other tests will use *their* dedicated wikis. There is no overlap. This is trivial given the idea of make one dedicated wiki for each test run and a basic understanding of how bulk MediaWiki installations operate. If you're trying to help plan how to run automated regression testing for MediaWiki *without* having done your basic research on how the system works, I strongly recommend you stop and do that first before continuing. -- brion ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
On Thu, 23 Sep 2010 20:13:23 -0700, Brion Vibber wrote: On Thu, Sep 23, 2010 at 7:19 PM, Dan Nessett dness...@yahoo.com wrote: I appreciate your recent help, so I am going to ignore the tone of your last message and focus on issues. While a test run can set up, use and then delete the temporary resources it needs (i.e., db, images directory, etc.), you really haven't answered the question I posed. If the test run ends abnormally, then it will not delete those resources. There has to be a way to garbage collect orphaned dbs, images directories and cache entries. Any introductory Unix sysadmin handbook will include examples of shell scripts to find old directories and remove them, etc. For that matter you could simply delete *all* the databases and files on the test machine every day before test runs start, and not spend even a second of effort worrying about cleaning up individual runs. Since each test database is a fresh slate, there is no shared state between runs -- there is *no* need to clean up immediately between runs or between test sets. My personal view is we should start out simple (as you originally suggested) with a set of fixed URLs that are used serially by test runs. Implementing this is probably the easiest option and would allow us to get something up and running quickly. This approach doesn't require significant development, although it does require a way to control access to the URLs so test runs don't step on each other. What you suggest is more difficult and harder to implement than creating a fresh database for each test run, and gives no clear benefit in exchange. Keep it simple by *not* implementing this idea of a fixed set of URLs which must be locked and multiplexed. Creating a fresh database directory for each run does not require any additional development. It does not require devising any access control. It does not require devising a special way to clean up resources or restore state. -- brion I am authentically sorry that you feel obliged to couch your last 2 replies in an offensive manner. You have done some very good work on the Mediawiki code and deserve a great deal of credit for it. It is clear you do not understand what I am proposing (that is, what I proposed after I accepted your suggestion to keep things simple): + Every test run does create a fresh database (and fresh images directory) for each run. It does this by first dropping the database associated with the last run, recursively deleting the phase3 directory holding the code from the previous run, checking out the revision for the current run (or if this is judged too expensive, we could hold the revisions in tar files and untar them into the directory), adjusting things so that the wiki will work (e.g., recursively chmoding the image directory so it is writable), and installing a LocalSettings file so things like imagemagick, texvc, etc. are locatable and global variables set appropriately. All of this is done in the directory associated with the fixed URL. + Before each test suite of a regression test runs it prepares the wiki. For example, if a prerequisite for the suite is the availability of an image, it uploads it before starting. + The regression test can be guarded by writing a lock file in the images directory (which protects all code and data directories). When the regression test completes, the lock file can be removed and the next regression test started. If there is only one test driver application running, the lock file is unnecessary. If the test driver application crashes for some reason, a simple utility can sweep the directory structures associated with the URLs and remove everything. While this is only a sketch, it is obviously simpler than attempting to set up a test run using the wikipedia family scheme. The previous test run db, images directory, etc. are deleted at the beginning of the next run that uses the fixed URL and its associated directory space. There is no need to Configure your DNS with a wildcard A record, and apache with a server alias (like ServerAlias *.yourdomain.net), something that may not be possible for some sites. There is no need to dynamically edit LocalSettings to fix up the upload directory global variable. As for cleaning up between runs, this simply ensures the database server doesn't become clogged with extraneous databases, that directory space is used efficiently and that memcached doesn't hold useless data. While it may be possible to get by with cleaning up every 24 hours, that the fresh wiki installation process cleans up these resources by default means such a global sweep is completely unnecessary. -- -- Dan Nessett ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
Hi, here are my thoughts about phpunit and selenium testing. The wiki under test is set up with a master database consisting of a single objectcache table. The entries of this table specify a test run identifier as primary key and temporary resource identifiers as dependent fields. If I understand this correctly, this would not allow to test any wikis that are running on live sites, e.g. intranet wikis. While I agree that regression testing on live sites is not a good idea, I kind of like the notion that after setting up a wiki with all the extensions I like to have, I could do some sort of everything up and running-test. With the concept of using separate testing databases and resources, this would be possible without interference with the actual data and could even be done at intervals during, say, maintenance periods. Setup of a test run requires the creation of the test run temporary resources and a entry in the objectcache table. Are there already mechanisms for this? I haven't done too much work with the objectcache. This is where memcached data is stored, right? So how do I get the data that is needed? This question leads me to another one: How do I get the testind database and resources? As I see this, it should be part of the testing framework to be able to produce the set of data needed from a normal MW installation. The whole mechanism would actually be something like a backup, so we might look into any existing solutions for that. When a request is sent to the wiki under test, very early in the request processing (e.g., immediately after LocalSettings is processed) a hook is called with the provided state information as an argument that accesses the objectcache table. The extension function handling the hook switches in the temporary resources and returns. Also, this hook might handle the reconfiguration of the wiki, if needed. So for example, testing the PagedTiffHandler requires uploading of tiff files to be enabled. However, there might be some security risks, since it is not directly obvious in the code which settings are changed. So the hook should only be called when $wgEnableSelenium = true. In addition, we could define a DontTouchThisSettings.php, which is called even after the hook and holds some settings that are holy to the admin of the wiki :) The question is, though, would this not become somewhat too complicated? After the test run completes, the testing application cleans up the test run by requesting the deletion of the temporary resources and the objectcache table entry associated with the test run. In some cases, tests will not change any data, e.g. testing dynamic skin elements in vector skin. Would it make sense not to tear down the testing environment in that case in order to save some time when testing repeatedly? I think, there is a conflict between performance and amount of data, but who wins? In general, it seems to me that we have some similarity with what is called wiki family on mediawiki.org. One could see multiple testing environments as a set of multiple wikis that share a common codebase [1]. Does anybody have experience with wiki families and the object cache rsp. memcached? I am not sure whether we can use the same codebase as parsertests. I'd rather think, parser tests are a special case of what we are sketching here. On the other hand, I don't think it is a good idea to have two separate approaches for very similar tasks in the code. Do you think it would be feasible to separate the preparation part from both parser tests and selenium tests and build both of them on a common ground? Best regards, Markus [1] http://www.mediawiki.org/wiki/Wiki_family#Scenario_2:_Quick_set-up ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
On Wed, 22 Sep 2010 15:49:40 +0200, Markus Glaser wrote: Hi, here are my thoughts about phpunit and selenium testing. The wiki under test is set up with a master database consisting of a single objectcache table. The entries of this table specify a test run identifier as primary key and temporary resource identifiers as dependent fields. If I understand this correctly, this would not allow to test any wikis that are running on live sites, e.g. intranet wikis. While I agree that regression testing on live sites is not a good idea, I kind of like the notion that after setting up a wiki with all the extensions I like to have, I could do some sort of everything up and running-test. With the concept of using separate testing databases and resources, this would be possible without interference with the actual data and could even be done at intervals during, say, maintenance periods. The problem with testing live sites is tests may alter wiki data (consequently, test run reproducibility becomes a problem). If all of the tests are read-only, then that isn't a problem, but it means developing a whole set of tests that conform to that constraint. Nevertheless, it wouldn't be hard to design the switching mechanism to allow the testing of live sites. There could be an option in test setup and cleanup that effectively says don't switch-in/clean-up temporary resources. Personally, I think use of this option is dangerous, but it wouldn't be hard to implement. Setup of a test run requires the creation of the test run temporary resources and a entry in the objectcache table. Are there already mechanisms for this? I haven't done too much work with the objectcache. This is where memcached data is stored, right? So how do I get the data that is needed? This question leads me to another one: How do I get the testind database and resources? As I see this, it should be part of the testing framework to be able to produce the set of data needed from a normal MW installation. The whole mechanism would actually be something like a backup, so we might look into any existing solutions for that. We should use the existing ObjectCache class to manage the object cache. However, if there exists some switching-in code, I doubt it has corresponding clean-up code. So, we probably need to do some development even if we use existing mechanisms. I think the object cache and memcached are alternative ways of storing persistent data. (I also am not an expert in this, so I could be wrong). My understanding is memcached uses the memcached daemon (http:// memcached.org/), while the object cache uses the underlying database. If so, then memcached data disappears after a system crash or power outage, whereas object cache data should survive. You are absolutely correct that we need to figure out how to clone a set of temporary resources (db, images directory, perhaps cache data) and set them up for use (i.e., so the switch-in logic can copy them for the test run). There are a number of problems to solve, e.g., 1) how do you package the cloned resources (e.g., tar file), 2) how do you efficiently copy them during the switch-in process, 3) security issues, 4) resource management issues. When a request is sent to the wiki under test, very early in the request processing (e.g., immediately after LocalSettings is processed) a hook is called with the provided state information as an argument that accesses the objectcache table. The extension function handling the hook switches in the temporary resources and returns. Also, this hook might handle the reconfiguration of the wiki, if needed. So for example, testing the PagedTiffHandler requires uploading of tiff files to be enabled. However, there might be some security risks, since it is not directly obvious in the code which settings are changed. So the hook should only be called when $wgEnableSelenium = true. In addition, we could define a DontTouchThisSettings.php, which is called even after the hook and holds some settings that are holy to the admin of the wiki :) The question is, though, would this not become somewhat too complicated? I have done some research on possible hooks. I think the SetupAfterCache hook is a good candidate. The problem with calling a (psuedo-)hook in LocalSettings is there is some setup not completed when LocatSettings is processed (e.g., the WebRequest object is not yet available. the ObjectCache class file is not yet included). Most setup is completed before calling the SetupAfterCache hook. It would be possible to do some configuration in this hook, but I think we need to consider efficiency. Preferably, the creation of the base resources that are copied during the switch-in will be configured as much as possible. For example, I think creating the base for PagedTiffHandler should start with a freshly installed wiki and upload multipage.tiff. The resulting db could be dumped and the images
Re: [Wikitech-l] using parserTests code for selenium test framework
On Wed, Sep 22, 2010 at 6:47 PM, Dan Nessett dness...@yahoo.com wrote: I think the object cache and memcached are alternative ways of storing persistent data. (I also am not an expert in this, so I could be wrong). My understanding is memcached uses the memcached daemon (http:// memcached.org/), while the object cache uses the underlying database. If so, then memcached data disappears after a system crash or power outage, whereas object cache data should survive. There is no ObjectCache class. Basically, there is a common base class called BagOStuff and from that various classes for various backends are defined such as SqlBagOStuff and MemcacheBagOStuff. To the code outside that class, there is no visible difference. Bryan ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
On Wed, 22 Sep 2010 18:57:12 +0200, Bryan Tong Minh wrote: On Wed, Sep 22, 2010 at 6:47 PM, Dan Nessett dness...@yahoo.com wrote: I think the object cache and memcached are alternative ways of storing persistent data. (I also am not an expert in this, so I could be wrong). My understanding is memcached uses the memcached daemon (http:// memcached.org/), while the object cache uses the underlying database. If so, then memcached data disappears after a system crash or power outage, whereas object cache data should survive. There is no ObjectCache class. Basically, there is a common base class called BagOStuff and from that various classes for various backends are defined such as SqlBagOStuff and MemcacheBagOStuff. To the code outside that class, there is no visible difference. Bryan Thanks for the clarification. I just looked at ObjectCache.php and it appears to provide a set of functions for accessing cache data of any type. Is this correct? How does memcached fit into this? When I looked at BagOStuff, I didn't find a MemcacheBagOStuff class. Is it defined elsewhere? -- -- Dan Nessett ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
2010/9/22 Dan Nessett dness...@yahoo.com: How does memcached fit into this? When I looked at BagOStuff, I didn't find a MemcacheBagOStuff class. Is it defined elsewhere? Either memcached.php, MemCached.php or MWMemcached.php, I forget. The class name is MWMemcached. Roan Kattouw (Catrope) ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
I just looked at SqlBagOStuff. It already has entry expiration logic. So, it seems perfect for use by the switch-in/clean-up functionality. I spoke too soon. Expired entries are deleted automatically when any entry is referenced. Unfortunately, that means there is no opportunity to reclaim the temporary resources identified in the entry before its state is lost. This is a problem. -- -- Dan Nessett ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
Hmm, I think you guys are overthinking the details on this; let's step back a level. When you're running tests, you have these tasks: * create a blank-slate wiki to run tests in * populate the empty wiki with known data * run tests on this wiki in a known state * clean up in some way The parserTests system is designed with some particular constraints: * run within the local MediaWiki test instance a developer already has, without altering its contents * run all tests in a single command-line program * expose no in-test data to the outside world It can do this very easily using temporary tables and a temporary directory because it only needs to work within that one single test process; once it's done, it's done and all the temporary data can be discarded. Nothing needs to be kept across processes or exposed to clients. For Selenium tests, you have a very different set of constraints: * test data must be exposed to web clients over a web server * test data must be retained across multiple requests The simplest way to accomplish this is to have a dedicated, web-exposed test wiki instance. A test run would go like this: * (re)initialize test wiki into known state * run series of tests I'd recommend not trying to use the parserTest harness/initialization for this; it'll be a *lot* simpler to just script creation of a fresh wiki. (drop database, create database, slurp tables, run update.php) Some non-destructive tests can always be run on any existing instance -- and probably should be! -- and some 'active' tests will be freely runnable on existing instances that are used for development and testing, but if you want to work with a blank slate wiki exposed to web clients, keep things simple and just make a dedicated instance. -- brion On Wed, Sep 22, 2010 at 9:47 AM, Dan Nessett dness...@yahoo.com wrote: On Wed, 22 Sep 2010 15:49:40 +0200, Markus Glaser wrote: Hi, here are my thoughts about phpunit and selenium testing. The wiki under test is set up with a master database consisting of a single objectcache table. The entries of this table specify a test run identifier as primary key and temporary resource identifiers as dependent fields. If I understand this correctly, this would not allow to test any wikis that are running on live sites, e.g. intranet wikis. While I agree that regression testing on live sites is not a good idea, I kind of like the notion that after setting up a wiki with all the extensions I like to have, I could do some sort of everything up and running-test. With the concept of using separate testing databases and resources, this would be possible without interference with the actual data and could even be done at intervals during, say, maintenance periods. The problem with testing live sites is tests may alter wiki data (consequently, test run reproducibility becomes a problem). If all of the tests are read-only, then that isn't a problem, but it means developing a whole set of tests that conform to that constraint. Nevertheless, it wouldn't be hard to design the switching mechanism to allow the testing of live sites. There could be an option in test setup and cleanup that effectively says don't switch-in/clean-up temporary resources. Personally, I think use of this option is dangerous, but it wouldn't be hard to implement. Setup of a test run requires the creation of the test run temporary resources and a entry in the objectcache table. Are there already mechanisms for this? I haven't done too much work with the objectcache. This is where memcached data is stored, right? So how do I get the data that is needed? This question leads me to another one: How do I get the testind database and resources? As I see this, it should be part of the testing framework to be able to produce the set of data needed from a normal MW installation. The whole mechanism would actually be something like a backup, so we might look into any existing solutions for that. We should use the existing ObjectCache class to manage the object cache. However, if there exists some switching-in code, I doubt it has corresponding clean-up code. So, we probably need to do some development even if we use existing mechanisms. I think the object cache and memcached are alternative ways of storing persistent data. (I also am not an expert in this, so I could be wrong). My understanding is memcached uses the memcached daemon (http:// memcached.org/), while the object cache uses the underlying database. If so, then memcached data disappears after a system crash or power outage, whereas object cache data should survive. You are absolutely correct that we need to figure out how to clone a set of temporary resources (db, images directory, perhaps cache data) and set them up for use (i.e., so the switch-in logic can copy them for the test run). There are a number of problems to solve, e.g., 1) how do you package the cloned resources (e.g., tar
Re: [Wikitech-l] using parserTests code for selenium test framework
On Wed, 22 Sep 2010 11:00:53 -0700, Brion Vibber wrote: Hmm, I think you guys are overthinking the details on this; let's step back a level. When you're running tests, you have these tasks: * create a blank-slate wiki to run tests in * populate the empty wiki with known data * run tests on this wiki in a known state * clean up in some way The parserTests system is designed with some particular constraints: * run within the local MediaWiki test instance a developer already has, without altering its contents * run all tests in a single command-line program * expose no in-test data to the outside world It can do this very easily using temporary tables and a temporary directory because it only needs to work within that one single test process; once it's done, it's done and all the temporary data can be discarded. Nothing needs to be kept across processes or exposed to clients. For Selenium tests, you have a very different set of constraints: * test data must be exposed to web clients over a web server * test data must be retained across multiple requests The simplest way to accomplish this is to have a dedicated, web-exposed test wiki instance. A test run would go like this: * (re)initialize test wiki into known state * run series of tests I'd recommend not trying to use the parserTest harness/initialization for this; it'll be a *lot* simpler to just script creation of a fresh wiki. (drop database, create database, slurp tables, run update.php) Some non-destructive tests can always be run on any existing instance -- and probably should be! -- and some 'active' tests will be freely runnable on existing instances that are used for development and testing, but if you want to work with a blank slate wiki exposed to web clients, keep things simple and just make a dedicated instance. -- brion On Wed, Sep 22, 2010 at 9:47 AM, Dan Nessett dness...@yahoo.com wrote: On Wed, 22 Sep 2010 15:49:40 +0200, Markus Glaser wrote: Hi, here are my thoughts about phpunit and selenium testing. The wiki under test is set up with a master database consisting of a single objectcache table. The entries of this table specify a test run identifier as primary key and temporary resource identifiers as dependent fields. If I understand this correctly, this would not allow to test any wikis that are running on live sites, e.g. intranet wikis. While I agree that regression testing on live sites is not a good idea, I kind of like the notion that after setting up a wiki with all the extensions I like to have, I could do some sort of everything up and running-test. With the concept of using separate testing databases and resources, this would be possible without interference with the actual data and could even be done at intervals during, say, maintenance periods. The problem with testing live sites is tests may alter wiki data (consequently, test run reproducibility becomes a problem). If all of the tests are read-only, then that isn't a problem, but it means developing a whole set of tests that conform to that constraint. Nevertheless, it wouldn't be hard to design the switching mechanism to allow the testing of live sites. There could be an option in test setup and cleanup that effectively says don't switch-in/clean-up temporary resources. Personally, I think use of this option is dangerous, but it wouldn't be hard to implement. Setup of a test run requires the creation of the test run temporary resources and a entry in the objectcache table. Are there already mechanisms for this? I haven't done too much work with the objectcache. This is where memcached data is stored, right? So how do I get the data that is needed? This question leads me to another one: How do I get the testind database and resources? As I see this, it should be part of the testing framework to be able to produce the set of data needed from a normal MW installation. The whole mechanism would actually be something like a backup, so we might look into any existing solutions for that. We should use the existing ObjectCache class to manage the object cache. However, if there exists some switching-in code, I doubt it has corresponding clean-up code. So, we probably need to do some development even if we use existing mechanisms. I think the object cache and memcached are alternative ways of storing persistent data. (I also am not an expert in this, so I could be wrong). My understanding is memcached uses the memcached daemon (http:// memcached.org/), while the object cache uses the underlying database. If so, then memcached data disappears after a system crash or power outage, whereas object cache data should survive. You are absolutely correct that we need to figure out how to clone a set of temporary resources (db, images directory, perhaps cache data) and set them up for use (i.e., so the switch-in logic can copy them for the test run).
Re: [Wikitech-l] using parserTests code for selenium test framework
On Wed, Sep 22, 2010 at 11:09 AM, Dan Nessett dness...@yahoo.com wrote: Some have mentioned the possibility of using the wiki family logic to help achieve these objectives. Do you have any thoughts on this? If you think it is a good idea, how do we find out more about it? I'd just treat it same as any other wiki. Whether you're running one or multiple wiki instances out of one copy of the code base, it just doesn't make a difference here. -- brion ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
On Wed, 22 Sep 2010 12:30:35 -0700, Brion Vibber wrote: On Wed, Sep 22, 2010 at 11:09 AM, Dan Nessett dness...@yahoo.com wrote: Some have mentioned the possibility of using the wiki family logic to help achieve these objectives. Do you have any thoughts on this? If you think it is a good idea, how do we find out more about it? I'd just treat it same as any other wiki. Whether you're running one or multiple wiki instances out of one copy of the code base, it just doesn't make a difference here. -- brion Oh. Well, perhaps we don't agree exactly 100%. As you suggest, let's step back a bit. Once we get the selenium framework working I assume it will be used for a regression test. This will comprise a set of individual tests. Generally, these tests will write into the wiki db (some may not, but many will). To ensure test reproducibility, the state of the wiki should be the same each time one of these individual tests runs. But, there is a problem. With parserTests, each individual test runs serially. That is fine for parserTests, since (I just ran this on my machine) while there are 610 individual tests, each takes about .08 seconds to run (on average). So, on my machine the whole parserTest takes about 48 seconds. Selenium tests are far more heavy-weight. A rough ball-park figure is each takes about 10 seconds to run (this does not include the time it would take to setup and tear down a clean wiki). So, a selenium-based regression test comprising 180 individual tests would take around 30 minutes. Not too bad. But, things are a bit more complicated. Each individual test runs multiple times, once for every browser/OS combination chosen for the regression test. For example, right now there are 13 configured browser/ OS combinations on the WMF Selenium Grid (see http:// grid.tesla.usability.wikimedia.org/console). So even if you only test 4 of these browser/OS configurations, the regression test (if individual tests run serially) would take 2 hours. If you test 8 of them, it would take 4 hours. This is starting to get onerous. If an individual developer wishes to ensure his modifications don't break things before committing his changes, then waiting 4 hours for a regression test to complete is a pretty heavy penalty. Generally, very few will pay the price. So, running the individual tests of a selenium-based regression test serially is not very attractive. This means you need to achieve some concurrency in the regression test. Since individual tests may interfere with each other, you need a way to protect them from each other. This is what the switching functionality is for. You switch in a base set of temporary resources for each test (or perhaps more likely for a particular test suite comprised of mulitple individual tests) consisting of a db, images directory, etc. This allows tests to run without interfering with each other. -- -- Dan Nessett ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
Dan Nessett wrote: On Sun, 19 Sep 2010 23:42:08 +0200, Platonides wrote: You load originaldb.objectcache, retrieve the specific configuration, and switch into it. For supporting many sumyltaneous configurations, the keyname could have the instance (whatever that cookie is set to) appended, although those dynamic configurations make me a bit nervous. Well, this may work, but consider the following. A nightly build environment (and even a local developer test environment) tests the latest revision using a suite of regression tests. These tests exercise the same wiki code, each parametrized by: + Browser type (e.g., Firefox, IE, Safari, Opera) + Database (e.g., MySQL, Postgres, SQLite) + OS platform (e.g., Linux, BSD unix variant, Windows variant) A particular test environment may not support all permutations of these parameters (in particular a local developer environment may support only one OS), but the code mechanism for supporting the regression tests should. To ensure timely completion of these tests, they will almost certainly run concurrently. So, when a regression test runs, it must not only retrieve the configuration data associated with it, it must create a test run environment (e.g., a test db, a test images directory, test cache data). The creation of this test run environment requires an identifier somewhere so its resources may be reclaimed when the test run completes or after an abnormal end of the test run. Thus, the originaldb must not only hold configuration data with db keys identifying the particular test and its parameters, but also an identifier for the test run that can be used to reclaim resources if the test ends abnormally. The question is whether using a full wiki db for this purpose is advantageous or whether stripping out all of the other tables except the objectcache table is the best implementation strategy. Such originaldb would be empty for an instance used just for regression testing and could in fact only contain the objectcache table. If it's a developer machine he would use the originaldb for local testing, but a nigthly would not need to (in fact, errors trying to access those missing tables would be useful for detecting errors in the isolating system). ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
On Mon, 20 Sep 2010 22:32:24 +0200, Platonides wrote: Dan Nessett wrote: On Sun, 19 Sep 2010 23:42:08 +0200, Platonides wrote: You load originaldb.objectcache, retrieve the specific configuration, and switch into it. For supporting many sumyltaneous configurations, the keyname could have the instance (whatever that cookie is set to) appended, although those dynamic configurations make me a bit nervous. Well, this may work, but consider the following. A nightly build environment (and even a local developer test environment) tests the latest revision using a suite of regression tests. These tests exercise the same wiki code, each parametrized by: + Browser type (e.g., Firefox, IE, Safari, Opera) + Database (e.g., MySQL, Postgres, SQLite) + OS platform (e.g., Linux, BSD unix variant, Windows variant) A particular test environment may not support all permutations of these parameters (in particular a local developer environment may support only one OS), but the code mechanism for supporting the regression tests should. To ensure timely completion of these tests, they will almost certainly run concurrently. So, when a regression test runs, it must not only retrieve the configuration data associated with it, it must create a test run environment (e.g., a test db, a test images directory, test cache data). The creation of this test run environment requires an identifier somewhere so its resources may be reclaimed when the test run completes or after an abnormal end of the test run. Thus, the originaldb must not only hold configuration data with db keys identifying the particular test and its parameters, but also an identifier for the test run that can be used to reclaim resources if the test ends abnormally. The question is whether using a full wiki db for this purpose is advantageous or whether stripping out all of the other tables except the objectcache table is the best implementation strategy. Such originaldb would be empty for an instance used just for regression testing and could in fact only contain the objectcache table. If it's a developer machine he would use the originaldb for local testing, but a nigthly would not need to (in fact, errors trying to access those missing tables would be useful for detecting errors in the isolating system). Sounds reasonable. Using this approach, here is how I see the logical flow of a test run. (NB: by a test run, I mean an execution of a test in the regression test set). The wiki under test is set up with a master database consisting of a single objectcache table. The entries of this table specify a test run identifier as primary key and temporary resource identifiers as dependent fields. Setup of a test run requires the creation of the test run temporary resources and a entry in the objectcache table. A cookie or other state is returned to the testing application. This state is provided by the testing application during the test run on each request to the wiki under test. When a request is sent to the wiki under test, very early in the request processing (e.g., immediately after LocalSettings is processed) a hook is called with the provided state information as an argument that accesses the objectcache table. The extension function handling the hook switches in the temporary resources and returns. After the test run completes, the testing application cleans up the test run by requesting the deletion of the temporary resources and the objectcache table entry associated with the test run. In order to handle prematurely abandoned test runs, the objectcache table entry probably needs an entry that specifies its lifetime. If this lifetime expires, the temporary resources associated with the entry are reclaimed and the entry is delete. -- -- Dan Nessett ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
On Fri, 17 Sep 2010 19:13:33 +, Dan Nessett wrote: On Fri, 17 Sep 2010 18:40:53 +, Dan Nessett wrote: I have been tasked to evaluate whether we can use the parserTests db code for the selenium framework. I just looked it over and have serious reservations. I would appreciate any comments on the following analysis. The environment for selenium tests is different than that for parserTests. It is envisioned that multiple concurrent tests could run using the same MW code base. Consequently, each test run must: + Use a db that if written to will not destroy other test wiki information. + Switch in a new images and math directory so any writes do not interfere with other tests. + Maintain the integrity of the cache. Note that tests would *never* run on a production wiki (it may be possible to do so if they do no writes, but safety considerations suggest they should always run on a test data, not production data). In fact production wikis should always retain the setting $wgEnableSelenium = false, to ensure selenium test are disabled. Given this background, consider the following (and feel free to comment on it): parserTests temporary table code: A fixed set of tables are specified in the code. parserTests creates temporary tables with the same name, but using a different static prefix. These tables are used for the parserTests run. Problems using this approach for selenium tests: + Selenium tests on extensions may require use of extension specific tables, the names of which cannot be elaborated in the code. + Concurrent test runs of parserTests are not supported, since the temporary tables have fixed names and therefore concurrent writes to them by parallel test runs would cause interference. + Clean up from aborted runs requires dropping fossil tables. But, if a previous run tested an extension with extension-specific tables, there is no way for a test of some other functionality to figure out which tables to drop. For these reasons, I don't think we can reuse the parserTests code. However, I am open to arguments to the contrary. After reflection, here are some other problems. + Some tests assume the existence of data in the db. For example, the PagedTiffHandler tests assume the image Multipage.tiff is already loaded. However, this requires an entry in the image table. You could modify the test to clone the existing image table, but that means you have problems with: + Some tests assume certain data is *not* in the db. PagedTiffHandler has tests that upload images. These cannot already be in the images table. So, you can't simply clone the images table. All of this suggests to me that a better strategy is: + When the test run begins, clone a db associated with the test suite. + Switch the wiki to use this db and return a cookie or some other state information that identifies this test run configuration. + When the test suite runs, each wiki access supplies this state so the wiki code can switch in the correct db. + Cleanup of test runs requires removing the cloned db. + To handled aborted runs, there needs to be a mechanism to time out cloned dbs and the state associated with the test run. Regardless of how we implement the persistent storage for managing test runs, there needs to be a way to trigger it use. To minimize the changes to core, we need a hook that runs after processing LocalSettings (and by implication DefaultSettings), but before any wiki state is accessed (e.g., before accessing the db, the images directory, any cached data). I looked at the existing hooks, but so far have not found one that appears suitable. So, either we need to identify an appropriate existing hook, or we need to add a hook that meets the requirements. -- -- Dan Nessett ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
On Sun, 19 Sep 2010 02:47:00 +0200, Platonides wrote: Dan Nessett wrote: What about memcached? (that would be a key based on the original db name) The storage has to be persistent to accommodate wiki crashes (e.g., httpd crash, server OS crash, power outage). It might be possible to use memcachedb, but as far as I am aware that requires installing Berkeley DB, which complicated deployment. Why not employ the already installed DB software used by the wiki? That provides persistent storage and requires no additional software. My original idea was to use whatever ObjectCache the wiki used, but it could be forced to use the db as backend (that's the objectcache table). My familiarity with the ObjectCache is casual. I presume it holds data that is set on particular wiki access requests and that data is then used on subsequent requests to make them more efficient. If so, then using a common ObjectCache for all concurrent test runs would cause interference between them. To ensure such interference doesn't exist, we would need to switch in a per-test-run ObjectCache (which takes us back to the idea of using a per-test-run db, since the ObjectCache is implemented using the objectcache table). -- -- Dan Nessett ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
Dan Nessett wrote: Platonides wrote: Dan Nessett wrote: What about memcached? (that would be a key based on the original db name) The storage has to be persistent to accommodate wiki crashes (e.g., httpd crash, server OS crash, power outage). It might be possible to use memcachedb, but as far as I am aware that requires installing Berkeley DB, which complicated deployment. Why not employ the already installed DB software used by the wiki? That provides persistent storage and requires no additional software. My original idea was to use whatever ObjectCache the wiki used, but it could be forced to use the db as backend (that's the objectcache table). My familiarity with the ObjectCache is casual. I presume it holds data that is set on particular wiki access requests and that data is then used on subsequent requests to make them more efficient. If so, then using a common ObjectCache for all concurrent test runs would cause interference between them. To ensure such interference doesn't exist, we would need to switch in a per-test-run ObjectCache (which takes us back to the idea of using a per-test-run db, since the ObjectCache is implemented using the objectcache table). You load originaldb.objectcache, retrieve the specific configuration, and switch into it. For supporting many sumyltaneous configurations, the keyname could have the instance (whatever that cookie is set to) appended, although those dynamic configurations make me a bit nervous. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
On Sun, 19 Sep 2010 23:42:08 +0200, Platonides wrote: Dan Nessett wrote: Platonides wrote: Dan Nessett wrote: What about memcached? (that would be a key based on the original db name) The storage has to be persistent to accommodate wiki crashes (e.g., httpd crash, server OS crash, power outage). It might be possible to use memcachedb, but as far as I am aware that requires installing Berkeley DB, which complicated deployment. Why not employ the already installed DB software used by the wiki? That provides persistent storage and requires no additional software. My original idea was to use whatever ObjectCache the wiki used, but it could be forced to use the db as backend (that's the objectcache table). My familiarity with the ObjectCache is casual. I presume it holds data that is set on particular wiki access requests and that data is then used on subsequent requests to make them more efficient. If so, then using a common ObjectCache for all concurrent test runs would cause interference between them. To ensure such interference doesn't exist, we would need to switch in a per-test-run ObjectCache (which takes us back to the idea of using a per-test-run db, since the ObjectCache is implemented using the objectcache table). You load originaldb.objectcache, retrieve the specific configuration, and switch into it. For supporting many sumyltaneous configurations, the keyname could have the instance (whatever that cookie is set to) appended, although those dynamic configurations make me a bit nervous. Well, this may work, but consider the following. A nightly build environment (and even a local developer test environment) tests the latest revision using a suite of regression tests. These tests exercise the same wiki code, each parametrized by: + Browser type (e.g., Firefox, IE, Safari, Opera) + Database (e.g., MySQL, Postgres, SQLite) + OS platform (e.g., Linux, BSD unix variant, Windows variant) A particular test environment may not support all permutations of these parameters (in particular a local developer environment may support only one OS), but the code mechanism for supporting the regression tests should. To ensure timely completion of these tests, they will almost certainly run concurrently. So, when a regression test runs, it must not only retrieve the configuration data associated with it, it must create a test run environment (e.g., a test db, a test images directory, test cache data). The creation of this test run environment requires an identifier somewhere so its resources may be reclaimed when the test run completes or after an abnormal end of the test run. Thus, the originaldb must not only hold configuration data with db keys identifying the particular test and its parameters, but also an identifier for the test run that can be used to reclaim resources if the test ends abnormally. The question is whether using a full wiki db for this purpose is advantageous or whether stripping out all of the other tables except the objectcache table is the best implementation strategy. -- -- Dan Nessett ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
Dan Nessett wrote: On Sat, 18 Sep 2010 00:53:04 +0200, Platonides wrote: + Switch the wiki to use this db and return a cookie or some other state information that identifies this test run configuration. I think you mean for remote petitions, not just for internal queries, where do you expect to store that data? Not sure what you mean by remote petitions. What you are calling going through the web portal, as opposed to parser tests and most phpunit tests, which are done in one run. Selenium requests always come through the web portal from the selenium server. So, no internal queries are involved. Note that although easier, other types of tests should also be confined. Where to store the data is an open question, one that requires consultation with others. However, here are some thoughts: + The data must be persistent. If the wiki crashes for some reason, there may be cloned dbs and test-specific copies of images and images/math hanging around. (Depending how we handle the cache information, there may also be fossil cache data). This requires cleanup after a wiki crash. + It would be possible to store the data in a file or in a master db table. Which is best (or if something else is better) is a subject for discussion. What about memcached? (that would be a key based on the original db name) ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
On Sun, 19 Sep 2010 00:28:42 +0200, Platonides wrote: Where to store the data is an open question, one that requires consultation with others. However, here are some thoughts: + The data must be persistent. If the wiki crashes for some reason, there may be cloned dbs and test-specific copies of images and images/math hanging around. (Depending how we handle the cache information, there may also be fossil cache data). This requires cleanup after a wiki crash. + It would be possible to store the data in a file or in a master db table. Which is best (or if something else is better) is a subject for discussion. What about memcached? (that would be a key based on the original db name) The storage has to be persistent to accommodate wiki crashes (e.g., httpd crash, server OS crash, power outage). It might be possible to use memcachedb, but as far as I am aware that requires installing Berkeley DB, which complicated deployment. Why not employ the already installed DB software used by the wiki? That provides persistent storage and requires no additional software. -- -- Dan Nessett ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
Dan Nessett wrote: What about memcached? (that would be a key based on the original db name) The storage has to be persistent to accommodate wiki crashes (e.g., httpd crash, server OS crash, power outage). It might be possible to use memcachedb, but as far as I am aware that requires installing Berkeley DB, which complicated deployment. Why not employ the already installed DB software used by the wiki? That provides persistent storage and requires no additional software. My original idea was to use whatever ObjectCache the wiki used, but it could be forced to use the db as backend (that's the objectcache table). ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
Dan Nessett wrote: Given this background, consider the following (and feel free to comment on it): parserTests temporary table code: A fixed set of tables are specified in the code. parserTests creates temporary tables with the same name, but using a different static prefix. These tables are used for the parserTests run. Problems using this approach for selenium tests: + Selenium tests on extensions may require use of extension specific tables, the names of which cannot be elaborated in the code. The extensions could list their table names. No problem there. + Concurrent test runs of parserTests are not supported, since the temporary tables have fixed names and therefore concurrent writes to them by parallel test runs would cause interference. So it gets changed to a random name with a fixed prefix... What concerns me is + Clean up from aborted runs requires dropping fossil tables. But, if a previous run tested an extension with extension-specific tables, there is no way for a test of some other functionality to figure out which tables to drop. Run a script dropping all tables with a fixed prefix (the shared part of the tests) when you have no tests running. For these reasons, I don't think we can reuse the parserTests code. However, I am open to arguments to the contrary. There may be other issues with that code, and using a separate db would be preferable if you have enough permissions, but this doesn't seem like real problems. What concerns me is that Oracle is using (r58669) a different prefix for the parsertests table. If it has some restriction on [medium-large] table names, there may not be possible to run the tests there using the long table names that we could produce. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
On Fri, 17 Sep 2010 18:40:53 +, Dan Nessett wrote: I have been tasked to evaluate whether we can use the parserTests db code for the selenium framework. I just looked it over and have serious reservations. I would appreciate any comments on the following analysis. The environment for selenium tests is different than that for parserTests. It is envisioned that multiple concurrent tests could run using the same MW code base. Consequently, each test run must: + Use a db that if written to will not destroy other test wiki information. + Switch in a new images and math directory so any writes do not interfere with other tests. + Maintain the integrity of the cache. Note that tests would *never* run on a production wiki (it may be possible to do so if they do no writes, but safety considerations suggest they should always run on a test data, not production data). In fact production wikis should always retain the setting $wgEnableSelenium = false, to ensure selenium test are disabled. Given this background, consider the following (and feel free to comment on it): parserTests temporary table code: A fixed set of tables are specified in the code. parserTests creates temporary tables with the same name, but using a different static prefix. These tables are used for the parserTests run. Problems using this approach for selenium tests: + Selenium tests on extensions may require use of extension specific tables, the names of which cannot be elaborated in the code. + Concurrent test runs of parserTests are not supported, since the temporary tables have fixed names and therefore concurrent writes to them by parallel test runs would cause interference. + Clean up from aborted runs requires dropping fossil tables. But, if a previous run tested an extension with extension-specific tables, there is no way for a test of some other functionality to figure out which tables to drop. For these reasons, I don't think we can reuse the parserTests code. However, I am open to arguments to the contrary. After reflection, here are some other problems. + Some tests assume the existence of data in the db. For example, the PagedTiffHandler tests assume the image Multipage.tiff is already loaded. However, this requires an entry in the image table. You could modify the test to clone the existing image table, but that means you have problems with: + Some tests assume certain data is *not* in the db. PagedTiffHandler has tests that upload images. These cannot already be in the images table. So, you can't simply clone the images table. All of this suggests to me that a better strategy is: + When the test run begins, clone a db associated with the test suite. + Switch the wiki to use this db and return a cookie or some other state information that identifies this test run configuration. + When the test suite runs, each wiki access supplies this state so the wiki code can switch in the correct db. + Cleanup of test runs requires removing the cloned db. + To handled aborted runs, there needs to be a mechanism to time out cloned dbs and the state associated with the test run. -- -- Dan Nessett ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
On Fri, 17 Sep 2010 21:05:12 +0200, Platonides wrote: Dan Nessett wrote: Given this background, consider the following (and feel free to comment on it): parserTests temporary table code: A fixed set of tables are specified in the code. parserTests creates temporary tables with the same name, but using a different static prefix. These tables are used for the parserTests run. Problems using this approach for selenium tests: + Selenium tests on extensions may require use of extension specific tables, the names of which cannot be elaborated in the code. The extensions could list their table names. No problem there. + Concurrent test runs of parserTests are not supported, since the temporary tables have fixed names and therefore concurrent writes to them by parallel test runs would cause interference. So it gets changed to a random name with a fixed prefix... What concerns me is + Clean up from aborted runs requires dropping fossil tables. But, if a previous run tested an extension with extension-specific tables, there is no way for a test of some other functionality to figure out which tables to drop. Run a script dropping all tables with a fixed prefix (the shared part of the tests) when you have no tests running. For these reasons, I don't think we can reuse the parserTests code. However, I am open to arguments to the contrary. There may be other issues with that code, and using a separate db would be preferable if you have enough permissions, but this doesn't seem like real problems. What concerns me is that Oracle is using (r58669) a different prefix for the parsertests table. If it has some restriction on [medium-large] table names, there may not be possible to run the tests there using the long table names that we could produce. The strategy you suggest is reasonable. But, I think it requires significant changes to the parserTests code. The question then is: is it simpler to modify this code or just write something new? -- -- Dan Nessett ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
Dan Nessett wrote: After reflection, here are some other problems. + Some tests assume the existence of data in the db. For example, the PagedTiffHandler tests assume the image Multipage.tiff is already loaded. However, this requires an entry in the image table. You could modify the test to clone the existing image table, but that means you have problems with: + Some tests assume certain data is *not* in the db. PagedTiffHandler has tests that upload images. These cannot already be in the images table. So, you can't simply clone the images table. Interesting. I haven't seen PagedTiffHandler tests. What normal parsertest do is to upload existing images and add the needed articles to the empty tables. Previously, the image tables entries were added directly by SQL. I changed it in r70917 to use recordUpload2() instead. All of this suggests to me that a better strategy is: + When the test run begins, clone a db associated with the test suite. Having another database would be the optimal solution, but it's not always possible. OTOH MaxSem replied to my concern in r58669: Oracle table names are limited to 32 characters. Mysql limit is of 64 characters* which gives more margin. Our longest table is msg_resource_links with 18 characters. If we choose a prefix like mwtest_ we could use another underscore plus 6 random digits for identifying the instance and still be Oracle compliant. * http://dev.mysql.com/doc/refman/5.1/en/identifiers.html + Switch the wiki to use this db and return a cookie or some other state information that identifies this test run configuration. I think you mean for remote petitions, not just for internal queries, where do you expect to store that data? ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
On Sat, 18 Sep 2010 00:53:04 +0200, Platonides wrote: + Switch the wiki to use this db and return a cookie or some other state information that identifies this test run configuration. I think you mean for remote petitions, not just for internal queries, where do you expect to store that data? Not sure what you mean by remote petitions. Selenium requests always come through the web portal from the selenium server. So, no internal queries are involved. Where to store the data is an open question, one that requires consultation with others. However, here are some thoughts: + The data must be persistent. If the wiki crashes for some reason, there may be cloned dbs and test-specific copies of images and images/math hanging around. (Depending how we handle the cache information, there may also be fossil cache data). This requires cleanup after a wiki crash. + It would be possible to store the data in a file or in a master db table. Which is best (or if something else is better) is a subject for discussion. We may be able to use the mechanisms in the code that supports access to different language versions of a wiki (e.g., Wikipedia) using the same code. I am not familiar with these mechanisms, so this approach requires help from someone who is. -- -- Dan Nessett ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
