Re: [Wikitech-l] Testing Framework (was Selenium Framework - Question on coding conventions)
On Fri, Aug 6, 2010 at 2:27 PM, Trevor Parscal tpars...@wikimedia.org wrote: I think /tests/unit and /tests/acceptance would be reasonable places to put things, and if they are both within maintenance or in the root doesn't really matter to me. Remember Selenium is a framework for doing acceptance testing, not unit testing. I don't quite see the purpose of specifying the framework name in our directory structure. Are we planning on using more than one unit or acceptance testing framework? I think it's best to pick whatever term will be the most intelligible. The idea of unit tests is very widely known, so tests/unit/ should be readily understood by anyone who knows much about automated software testing at all. Would tests/selenium/ or tests/acceptance/ be more comprehensible to the average person? I had never heard the term acceptance test before now, and the Wikipedia article on it makes it sound like a pretty vague term. Either way, we should probably throw parser tests in the same directory. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Testing Framework (was Selenium Framework - Question on coding conventions)
Hello, You are right that having unit and selenium folders in tests does not quite fit. However, acceptance I don't find appropriate, either. IMHO, Selenium actually is a framework for system testing (as it evaluates the system functionalities from a user perspective). So, how about having a folder system? Regards, Benedikt -- Karlsruhe Institute of Technology (KIT) Institute of Applied Informatics and Formal Description Methods (AIFB) Benedikt Kämpgen Research Associate Kaiserstraße 12 Building 11.40 76131 Karlsruhe, Germany Phone: +49 721 608-7946 Fax: +49 721 608-6580 Email: benedikt.kaemp...@kit.edu Web: http://www.kit.edu/ KIT - University of the State of Baden-Wuerttemberg and National Research Center of the Helmholtz Association -Original Message- From: wikitech-l-boun...@lists.wikimedia.org [mailto:wikitech-l-boun...@lists.wikimedia.org] On Behalf Of Trevor Parscal Sent: Friday, August 06, 2010 8:27 PM To: wikitech-l@lists.wikimedia.org Subject: Re: [Wikitech-l] Testing Framework (was Selenium Framework - Question on coding conventions) I think /tests/unit and /tests/acceptance would be reasonable places to put things, and if they are both within maintenance or in the root doesn't really matter to me. Remember Selenium is a framework for doing acceptance testing, not unit testing. I don't quite see the purpose of specifying the framework name in our directory structure. Are we planning on using more than one unit or acceptance testing framework? My 2 cents. - Trevor On 8/5/10 3:55 PM, Chad wrote: On Thu, Aug 5, 2010 at 6:47 PM, Mark A. Hershbergerm...@everybody.org wrote: Markus Glasergla...@hallowelt.biz writes: 1) Where are the tests located? I suggest for core to put them into maintenance/tests/selenium. That is where they are now. For extensions I propse a similar structure, that isextensiondir/tests/selenium. Sounds fine. In the same way, since maintenance/tests contains tests that should be run using PHPUnit, we can say thatextensiondir/tests will contain tests that should be run using PHPUnit. I would prefer moving them to a subdirectory of /tests/. As we hopefully amass more unit tests, keeping them in the top-level will get a bit confusing when trying to distinguish them from supporting code (shared setUp and tearDown code, the bootstrap stuff, etc) Something like /maintenance/tests/unit/ to mirror /maintenance/tests/ selenium/ would make the most sense. Consistency and thinking ahead is nice :) -Chad ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Testing Framework (was Selenium Framework - Question on coding conventions)
On 8/7/10 9:58 AM, Benedikt Kaempgen wrote: Hello, You are right that having unit and selenium folders in tests does not quite fit. However, acceptance I don't find appropriate, either. IMHO, Selenium actually is a framework for system testing (as it evaluates the system functionalities from a user perspective). So, how about having a folder system? Regards, Benedikt -- Karlsruhe Institute of Technology (KIT) Institute of Applied Informatics and Formal Description Methods (AIFB) Benedikt Kämpgen Research Associate Kaiserstraße 12 Building 11.40 76131 Karlsruhe, Germany Phone: +49 721 608-7946 Fax: +49 721 608-6580 Email: benedikt.kaemp...@kit.edu Web: http://www.kit.edu/ KIT - University of the State of Baden-Wuerttemberg and National Research Center of the Helmholtz Association -Original Message- From: wikitech-l-boun...@lists.wikimedia.org [mailto:wikitech-l-boun...@lists.wikimedia.org] On Behalf Of Trevor Parscal Sent: Friday, August 06, 2010 8:27 PM To: wikitech-l@lists.wikimedia.org Subject: Re: [Wikitech-l] Testing Framework (was Selenium Framework - Question on coding conventions) I think /tests/unit and /tests/acceptance would be reasonable places to put things, and if they are both within maintenance or in the root doesn't really matter to me. Remember Selenium is a framework for doing acceptance testing, not unit testing. I don't quite see the purpose of specifying the framework name in our directory structure. Are we planning on using more than one unit or acceptance testing framework? My 2 cents. - Trevor On 8/5/10 3:55 PM, Chad wrote: On Thu, Aug 5, 2010 at 6:47 PM, Mark A. Hershbergerm...@everybody.org wrote: Markus Glasergla...@hallowelt.biz writes: 1) Where are the tests located? I suggest for core to put them into maintenance/tests/selenium. That is where they are now. For extensions I propse a similar structure, that isextensiondir/tests/selenium. Sounds fine. In the same way, since maintenance/tests contains tests that should be run using PHPUnit, we can say thatextensiondir/tests will contain tests that should be run using PHPUnit. I would prefer moving them to a subdirectory of /tests/. As we hopefully amass more unit tests, keeping them in the top-level will get a bit confusing when trying to distinguish them from supporting code (shared setUp and tearDown code, the bootstrap stuff, etc) Something like /maintenance/tests/unit/ to mirror /maintenance/tests/ selenium/ would make the most sense. Consistency and thinking ahead is nice :) -Chad ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l I'm not /totally/ opposed to breaking away from the standard terminology of unit/integration/acceptance testing... We could call it something more descriptive than system though - perhaps client... - Trevor ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Testing Framework (was Selenium Framework - Question on coding conventions)
On Thu, 05 Aug 2010 18:47:58 -0400, Mark A. Hershberger wrote: Markus Glaser gla...@hallowelt.biz writes: 1) Where are the tests located? I suggest for core to put them into maintenance/tests/selenium. That is where they are now. For extensions I propse a similar structure, that is extensiondir/tests/selenium. Sounds fine. In the same way, since maintenance/tests contains tests that should be run using PHPUnit, we can say that extensiondir/tests will contain tests that should be run using PHPUnit. Alternatively, we could use the word Selenium somewhere in there in order to be able to divide between unit and selenium tests. I think putting them in the selenium directory (or the “Se” directory) is sufficient. 3) How does the framework know there are tests? Can I suggest that the framework can see that an extension has tests simply by the presence of the extensiondir/tests directory containing a Extension*TestSuite.php file? The extensiondir/tests/ExtensionTestSuite.php file should define a class using the name ExtensionTestSuite which has a static method suite(). See the PHPUnit documentation at http://bit.ly/b9L50r for how this is set up. This static suite() method should take care of letting the autoloader know about any test classes so the test classes are only available during testing. So, for your example using PagedTiffHandler, there would be the files: PagedTiffHandler/tests/PagedTiffHandlerTestSuite.php PagedTiffHandler/tests/PagedTiffHandlerUploadsTestSuite.php 4) Which tests should be executed? By default all the test suites in extensiondir/tests should be run. It is should be possible to specify which particular test to run by using whatever command line arguments to the CLI. This seems better to me than defining a new global. If some tests should only be run rarely, that information can be put in the TestSuite class for te extension. In this way, I think it is possible to remove all the $wgSelenium* variables from the DefaultSettings.php file. (I plan to do something similar with the $wgParserTest* variables as well — these sorts of things don't seem like they belong in Core.) Mark. I don't think walking through all the extensions looking for test subdirectories and then running all tests therein is a good idea. First, in a large installation with many extensions, this takes time and delays the test execution. Second, if a developer is working on a particular extension or part of the core, it will be common to run the tests associated with that for regression purposes. Making the developer specify the extension or core tests to run on the RunSeleniumTests command line is irritating (at least, it would irritate me). Specifying the test suite(s) to be run in LocalSettings.php is a set and forget approach that allows the developer to get on with the work. However, I do agree that the number of global variables associated with the selenium framework is getting large and has the potential of growing over time. One solution is to use a multi-dimensional associative array (much like $wgGroupPermissions). We could use a global variable $wgSelenium and move all selenium framework values into it. For example: $wgSelenium['wiki']['host'] = 'localhost'; $wgSelenium['wiki']['wikiurl'] = false; $wgSelenium['wiki']['loginname'] = 'Wikiuser; $wgSelenium['wiki']['password'] = ''; $wgSelenium['server']['port'] = ; etc. The only global we may wish to keep separate is $wgEnableSelenium, since it specifies whether $wgSelenium is used. -- -- Dan Nessett ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Testing Framework (was Selenium Framework - Question on coding conventions)
On Fri, 06 Aug 2010 12:38:39 -0400, Aryeh Gregor wrote: On Thu, Aug 5, 2010 at 6:47 PM, Mark A. Hershberger m...@everybody.org wrote: Can I suggest that the framework can see that an extension has tests simply by the presence of the extensiondir/tests directory containing a Extension*TestSuite.php file? IMO, the way parser tests do it is smarter. When you install the extension, it adds the location of the test files to $wgParserTestFiles. That way, only the tests associated with installed extensions will run. If you want to force only particular tests to run all the time, you can also modify the variable in LocalSettings.php as usual. We are doing something similar. In the extension require() file, the test suite is added to $wgAutoloadClasses. Right now the entry in $wgSeleniumTestSuites is pushed in LocalSettings. However, we could establish the convention that it is pushed in the extension require() file as well. Then all extensions with test suites would automatically load them. To tailor this, the entries in $wgSeleniumTestSuites could be removed in LocalSettings. -- -- Dan Nessett ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Testing Framework (was Selenium Framework - Question on coding conventions)
I think /tests/unit and /tests/acceptance would be reasonable places to put things, and if they are both within maintenance or in the root doesn't really matter to me. Remember Selenium is a framework for doing acceptance testing, not unit testing. I don't quite see the purpose of specifying the framework name in our directory structure. Are we planning on using more than one unit or acceptance testing framework? My 2 cents. - Trevor On 8/5/10 3:55 PM, Chad wrote: On Thu, Aug 5, 2010 at 6:47 PM, Mark A. Hershbergerm...@everybody.org wrote: Markus Glasergla...@hallowelt.biz writes: 1) Where are the tests located? I suggest for core to put them into maintenance/tests/selenium. That is where they are now. For extensions I propse a similar structure, that isextensiondir/tests/selenium. Sounds fine. In the same way, since maintenance/tests contains tests that should be run using PHPUnit, we can say thatextensiondir/tests will contain tests that should be run using PHPUnit. I would prefer moving them to a subdirectory of /tests/. As we hopefully amass more unit tests, keeping them in the top-level will get a bit confusing when trying to distinguish them from supporting code (shared setUp and tearDown code, the bootstrap stuff, etc) Something like /maintenance/tests/unit/ to mirror /maintenance/tests/ selenium/ would make the most sense. Consistency and thinking ahead is nice :) -Chad ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] Testing Framework (was Selenium Framework - Question on coding conventions)
Markus Glaser gla...@hallowelt.biz writes: 1) Where are the tests located? I suggest for core to put them into maintenance/tests/selenium. That is where they are now. For extensions I propse a similar structure, that is extensiondir/tests/selenium. Sounds fine. In the same way, since maintenance/tests contains tests that should be run using PHPUnit, we can say that extensiondir/tests will contain tests that should be run using PHPUnit. Alternatively, we could use the word Selenium somewhere in there in order to be able to divide between unit and selenium tests. I think putting them in the selenium directory (or the “Se” directory) is sufficient. 3) How does the framework know there are tests? Can I suggest that the framework can see that an extension has tests simply by the presence of the extensiondir/tests directory containing a Extension*TestSuite.php file? The extensiondir/tests/ExtensionTestSuite.php file should define a class using the name ExtensionTestSuite which has a static method suite(). See the PHPUnit documentation at http://bit.ly/b9L50r for how this is set up. This static suite() method should take care of letting the autoloader know about any test classes so the test classes are only available during testing. So, for your example using PagedTiffHandler, there would be the files: PagedTiffHandler/tests/PagedTiffHandlerTestSuite.php PagedTiffHandler/tests/PagedTiffHandlerUploadsTestSuite.php 4) Which tests should be executed? By default all the test suites in extensiondir/tests should be run. It is should be possible to specify which particular test to run by using whatever command line arguments to the CLI. This seems better to me than defining a new global. If some tests should only be run rarely, that information can be put in the TestSuite class for te extension. In this way, I think it is possible to remove all the $wgSelenium* variables from the DefaultSettings.php file. (I plan to do something similar with the $wgParserTest* variables as well — these sorts of things don't seem like they belong in Core.) Mark. -- http://hexmode.com/ Embrace Ignorance. Just don't get too attached. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Testing Framework (was Selenium Framework - Question on coding conventions)
On Thu, Aug 5, 2010 at 6:47 PM, Mark A. Hershberger m...@everybody.org wrote: Markus Glaser gla...@hallowelt.biz writes: 1) Where are the tests located? I suggest for core to put them into maintenance/tests/selenium. That is where they are now. For extensions I propse a similar structure, that is extensiondir/tests/selenium. Sounds fine. In the same way, since maintenance/tests contains tests that should be run using PHPUnit, we can say that extensiondir/tests will contain tests that should be run using PHPUnit. I would prefer moving them to a subdirectory of /tests/. As we hopefully amass more unit tests, keeping them in the top-level will get a bit confusing when trying to distinguish them from supporting code (shared setUp and tearDown code, the bootstrap stuff, etc) Something like /maintenance/tests/unit/ to mirror /maintenance/tests/ selenium/ would make the most sense. Consistency and thinking ahead is nice :) -Chad ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
