Re: [Wikitech-l] Sites using InstantCommons (was: wikimedia commons slow lately?)
On 22 September 2010 07:49, Erik Moeller e...@wikimedia.org wrote: Unless there's another obvious source of this information If it were an extension, we would have a little [1]. There has been some talk about implementing optional statistics collection for MediaWiki installations (like some configuration variables and installed extensions), but nothing more thus far. [1] http://s23.org/wikistats/mediawiki_extensions.php -- Niklas Laxström ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] API architecture
Hi! I am developing my API extension. It's enumerating revisions, but in a different way (not like in ApiQueryRevisions class). Also, it can optionally create xml dumps via WikiExporter, like API action=queryexportexportnowrap (so I need to change output printer to raw mode sometimes). Because it's enumerating the lists (however not titles) I've choosed list in non-generator mode (derived from ApiQueryBase). However, after initial development I've figured out that I cannot change default printer in such case, because it's not being called at all. The reason is explained in following message by Roan Kattouw: https://bugzilla.wikimedia.org/show_bug.cgi?id=25232 Query submodules can be called in conjunction with other query submodules In this case, if your module would switch to a custom printer, the others would quite likely freak out. But, I don't need to query in conjunction. Ok, I've derived from ApiBase class (my own API action). Then, it starts to fail on huge amount of generally useful methods, like SQL query building methods (addJoinConds, addWhereRange) and list continuation methods (setContinueEnumParameter), because these are not defined in ApiBase. I understand that not every ApiBase derived class needs these, but many could have. Why not make inheritance chain like this: ApiBase - ApiListBase - ApiQueryBase where ApiListBase would have at least these nice SQL query buliding methods but also the possibility to override the default printer? Why these methods are limited to action=query lists and generators only? Dmitriy ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] MediaWiki 1.17 release target (Re: UsabilityInitiative extension...)
On 21/09/10 23:03, Max Semenik wrote: snip The only viable technical method of feature freeze is branching, otherwise you'll have to come up with messages like Okay, I declare a feature freeze. Whomever commits a new feature to trunk whill have their fingers smashed with a hammer. Not very practical. That is actually the aim. The frozen date can be seen as a dead line, a day by wich you must have something 'ready to show'. If it is not ready, you have missed the deadline and the feature will made it in the next release. Keep the feature in your branch. The freeze itself might just last a couple weeks. Some developers don't read wikitech-l, some are weeks behind on reading their mails. Should they keep commit right to trunk ? Maybe some patches could just be sent by emails for review. I personally found it hard to commit anything without being on IRC or I will occasionaly face a silent revert. -- Ashar Voultoiz ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Balancing MediaWiki Core/Extensions
On 22/09/10 05:34, Trevor Parscal wrote: argument #2 against merging 2. Extensions encourage modularity and are easier to learn and work on because they are smaller sets of code organized in discreet bundles. That is probably the most important point. It ease code review, any new comer can easily understand code. Arguments I have made/observed *for* merging things into core include: 1. MediaWiki should be awesome out-of-the-box, so extensions that would be good for virtually everyone seem silly to bury deep within the poorly organized depths of the extensions folder. What about shipping the Mediawiki release with some of the best extensions ? You do not need to actually merge them with trunk, just chip mediawiki + some extensions preconfigured. Hooks in the installer would let user tweak the configuration. I had a little script which, given a revision number would : - installed trunk and stable branch - copy my favorite extensions in the extension folders - append to local settings my extension configuration files snip If we are to develop guidelines for when to push things in/pull things out of core, ... Would you move as extensions : squid support, math, languages, external stores, external auth, API, chinese support, Tidy, Antivirus ... They are all awesome features but not necessarily needed in core. -- Ashar Voultoiz % mv hashar /mw/extensions/ ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] MediaWiki 1.17 release target (Re: UsabilityInitiative extension...)
On Wed, Sep 22, 2010 at 4:21 PM, Ashar Voultoiz hashar+...@free.fr wrote: Some developers don't read wikitech-l, some are weeks behind on reading their mails. Should they keep commit right to trunk ? Maybe some patches could just be sent by emails for review. I personally found it hard to commit anything without being on IRC or I will occasionaly face a silent revert. -- Ashar Voultoiz Silent reverting shouldn't happen, anyone with commit access should have allow others to email me activated on their mw wiki account so that code review can send them emails when people comment or take action on code, although some older accounts might not have this enabled it is now a requirement to get access[1] and people don't/shouldn't be reverting without commenting on why anyway. Sending patches via email isn't a really good idea, because it isn't as easy to track them, easily lost/forgotten. Where as when directly submitted/attached to Bugzilla report it means that they can be easily searched and found if required and commented on, then just bugging someone on IRC to get it applied (Yes I know there is a backlog of patches on Bugzilla needing to be looked at[2]). [1]. http://www.mediawiki.org/wiki/Commit_access#Requesting_commit_access [2]. https://bugzilla.wikimedia.org/buglist.cgi?cmdtype=doremremaction=runnamedcmd=Patches%3A%20Open%20and%20Need%20Reviewingsharer_id=9593 ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Balancing MediaWiki Core/Extensions
On Tue, Sep 21, 2010 at 8:34 PM, Trevor Parscal tpars...@wikimedia.org wrote: In response to recent comments in our code review tool about whether some extensions should be merged into core MediaWiki, or not. I would like to try and initiate a productive conversation about this topic in hopes that we can collaboratively define a set of guidelines for evaluating when to move features out of core and into an extension or out of an extension and into core. unintended bias Arguments I have made/observed *against* merging things into core include: 1. Fewer developers have commit access to core, which pushes people off into branches who would have otherwise been able to contribute directly to trunk, inhibiting entry-level contribution. People can get access to core. This is a poor argument for not integrating things into core. If this is a problem, then it is a problem with us, and our processes, and should be fixed. 2. Extensions encourage modularity and are easier to learn and work on because they are smaller sets of code organized in discreet bundles. 3. We should be looking to make core less inclusive, not more. The line between Wikipedia and MediaWiki is already blurry enough as it is. I think this should be the case, but there are certain sets of functionality that people expect to be installed in a default installation. A usable editing interface is definitely one of those things. Arguments I have made/observed *for* merging things into core include: 1. MediaWiki should be awesome out-of-the-box, so extensions that would be good for virtually everyone seem silly to bury deep within the poorly organized depths of the extensions folder. I totally agree with this to an extent. I don't think we should be including everything and the kitchen sink, but things that make MediaWiki simply feel broken when they are missing should be integrated. ParserFunctions is an amazing example of this. MediaWiki simply doesn't work without ParserFunctions. You can tell me it does, and that people can live without it, but I refuse to believe that. We get support issues extremely frequently that end in install ParserFunctions. I'd also like to mention that something I'm seeing in even greater frequency now is how do I get Vector to work like it does on Wikipedia? or How do I make the sidebar collapse, or why doesn't my search bar work like it does on Wikipedia? Why did we expend all of this effort on usability if we are going to make it a pain to use if you aren't Wikimedia? That is very anti-usability. 2. When an extension is unable to do what it needs to do because it's dependent on a limited set of hooks, none of which quite do what it needs. This goes back to people needing core access to add hooks. In reality, they can send patches, or ask a dev with core access to do so. I've had a number of hooks added this way. 3. Because someone said so. Agreed. This is never a reasonable argument. /unintended bias obvious bias I will respond to these three pro-integration points; mostly because I am generally biased against integration and would like to state why. I realize that there are probably additional pro-integration points that are far less biased than the three I've listed, but I am basing these on arguments I've actually seen presented. 1. This is a very valid and important goal, but am unconvinced and merging extensions into core is the only way to achieve it. We can, for instance, take advantage the new installer that demon is working on which has the ability to automate the installation of extensions at setup-time. One problem I have every time I upgrade MediaWiki is extension compatibility. Extension management in MediaWiki is simply broken. Until we have a reasonable way to handle extension compatibility and some reasonable means for people to upgrade extensions, this is a painful course of action. 2. This seems like a call for better APIs/a more robust set of hooks. Integration for this sort of reason is more likely to introduce cruft and noise than improve the software in any way. This would help. 3. Noting that so-and-so said I should integrate this into core is not going to magically absolve anyone of having to stand behind their decision to proceed with such action and support it with logic and reason. Agreed. We need to be picky about what goes into core. /obvious bias If we are to develop guidelines for when to push things in/pull things out of core, it's going to be important that we reach some general consensus on the merits of these points. This mailing list is not historically known for its efficacy in consensus building, but I still feel like this conversation is going to be better off here than in a series of disjointed code review comments. Well, to restate my above opinions: I think in general we don't integrate
Re: [Wikitech-l] Balancing MediaWiki Core/Extensions
Trevor Parscal wrote: In response to recent comments in our code review tool about whether some extensions should be merged into core MediaWiki, or not. I would like to try and initiate a productive conversation about this topic in hopes that we can collaboratively define a set of guidelines for evaluating when to move features out of core and into an extension or out of an extension and into core. I know I mentioned this in a CodeReview comment, but I'll mention it again here: http://mediawiki.org/wiki/List_of_extensions_to_be_merged_to_the_core I think developing a set of guidelines for this issue might be nice, though I'm not sure there's really all that much disagreement about what should and shouldn't be in core currently. MZMcBride ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Balancing MediaWiki Core/Extensions
It seems like there are two dimensions to this problem. X: Should it be part of stock MediaWiki or not Y: Is it better to write features as discreet modules or spread them out the way core is currently done The problem is when you move across X, you are forced to cross Y; If we were to start moving extensions into core, it would mean breaking up otherwise well bundled units of code into lots of tiny parts and spreading them out. But why must this be the case? Modular feature development being unique to extensions points out a significant flaw in the design of MediaWiki core. There's no reason we can't convert existing core features to discreet components, much like how extensions are written, while leaving them in stock MediaWiki. This sort of design would also make light work of adding extensions to core. - Trevor On 9/22/10 12:11 AM, Ryan Lane wrote: On Tue, Sep 21, 2010 at 8:34 PM, Trevor Parscaltpars...@wikimedia.org wrote: In response to recent comments in our code review tool about whether some extensions should be merged into core MediaWiki, or not. I would like to try and initiate a productive conversation about this topic in hopes that we can collaboratively define a set of guidelines for evaluating when to move features out of core and into an extension or out of an extension and into core. unintended bias Arguments I have made/observed *against* merging things into core include: 1. Fewer developers have commit access to core, which pushes people off into branches who would have otherwise been able to contribute directly to trunk, inhibiting entry-level contribution. People can get access to core. This is a poor argument for not integrating things into core. If this is a problem, then it is a problem with us, and our processes, and should be fixed. 2. Extensions encourage modularity and are easier to learn and work on because they are smaller sets of code organized in discreet bundles. 3. We should be looking to make core less inclusive, not more. The line between Wikipedia and MediaWiki is already blurry enough as it is. I think this should be the case, but there are certain sets of functionality that people expect to be installed in a default installation. A usable editing interface is definitely one of those things. Arguments I have made/observed *for* merging things into core include: 1. MediaWiki should be awesome out-of-the-box, so extensions that would be good for virtually everyone seem silly to bury deep within the poorly organized depths of the extensions folder. I totally agree with this to an extent. I don't think we should be including everything and the kitchen sink, but things that make MediaWiki simply feel broken when they are missing should be integrated. ParserFunctions is an amazing example of this. MediaWiki simply doesn't work without ParserFunctions. You can tell me it does, and that people can live without it, but I refuse to believe that. We get support issues extremely frequently that end in install ParserFunctions. I'd also like to mention that something I'm seeing in even greater frequency now is how do I get Vector to work like it does on Wikipedia? or How do I make the sidebar collapse, or why doesn't my search bar work like it does on Wikipedia? Why did we expend all of this effort on usability if we are going to make it a pain to use if you aren't Wikimedia? That is very anti-usability. 2. When an extension is unable to do what it needs to do because it's dependent on a limited set of hooks, none of which quite do what it needs. This goes back to people needing core access to add hooks. In reality, they can send patches, or ask a dev with core access to do so. I've had a number of hooks added this way. 3. Because someone said so. Agreed. This is never a reasonable argument. /unintended bias obvious bias I will respond to these three pro-integration points; mostly because I am generally biased against integration and would like to state why. I realize that there are probably additional pro-integration points that are far less biased than the three I've listed, but I am basing these on arguments I've actually seen presented. 1. This is a very valid and important goal, but am unconvinced and merging extensions into core is the only way to achieve it. We can, for instance, take advantage the new installer that demon is working on which has the ability to automate the installation of extensions at setup-time. One problem I have every time I upgrade MediaWiki is extension compatibility. Extension management in MediaWiki is simply broken. Until we have a reasonable way to handle extension compatibility and some reasonable means for people to upgrade extensions, this is a painful course of action. 2. This seems like a call for better APIs/a more robust set of
Re: [Wikitech-l] Sites using InstantCommons (was: wikimedia commons slow lately?)
On 22 September 2010 07:49, Erik Moeller e...@wikimedia.org wrote: Unless there's another obvious source of this information Nope, thanks for doing that :) If it were an extension, we would have a little [1]. There has been some talk about implementing optional statistics collection for MediaWiki installations (like some configuration variables and installed extensions), but nothing more thus far. [1] http://s23.org/wikistats/mediawiki_extensions.php I threw around some ideas with Jeroen during GSoC this year, but nothing concrete ever came out of it. It's something I'd like to see happen (1.18?). Fully optional, of course. -Chad ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] API architecture
2010/9/22 Dmitriy Sintsov ques...@rambler.ru: But, I don't need to query in conjunction. Ok, I've derived from ApiBase class (my own API action). Then, it starts to fail on huge amount of generally useful methods, like SQL query building methods (addJoinConds, addWhereRange) and list continuation methods (setContinueEnumParameter), because these are not defined in ApiBase. I understand that not every ApiBase derived class needs these, but many could have. Why not make inheritance chain like this: ApiBase - ApiListBase - ApiQueryBase where ApiListBase would have at least these nice SQL query buliding methods but also the possibility to override the default printer? Why these methods are limited to action=query lists and generators only? That's a very good suggestion. Feel free to implement it or, if you can't or don't want to, file a bug requesting it in Bugzilla. Roan Kattouw (Catrope) ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] API architecture
2010/9/22 Roan Kattouw roan.katt...@gmail.com: I understand that not every ApiBase derived class needs these, but many could have. Why not make inheritance chain like this: ApiBase - ApiListBase - ApiQueryBase Actually, let's just put that stuff in ApiBase so every module can use it easily. Creating an intermediate class that modules then have to extend serves no purpose. Roan Kattouw (Catrope) ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] API architecture
On Wed, Sep 22, 2010 at 8:17 AM, Roan Kattouw roan.katt...@gmail.com wrote: 2010/9/22 Roan Kattouw roan.katt...@gmail.com: I understand that not every ApiBase derived class needs these, but many could have. Why not make inheritance chain like this: ApiBase - ApiListBase - ApiQueryBase Actually, let's just put that stuff in ApiBase so every module can use it easily. Creating an intermediate class that modules then have to extend serves no purpose. I disagree. Things that work currently with ApiBase would continue to, and things extending ApiQueryBase wouldn't notice a difference if a few of their methods were moved up a level to a new ApiListBase. -Chad ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] API architecture
2010/9/22 Chad innocentkil...@gmail.com: I disagree. Things that work currently with ApiBase would continue to, and things extending ApiQueryBase wouldn't notice a difference if a few of their methods were moved up a level to a new ApiListBase. Moving a few methods from ApiQueryBase up to ApiBase won't break any other ApiBase subclasses. There is no advantage in creating a new class over simply moving this stuff to ApiBase. Roan Kattouw (Catrope) ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] API architecture
On Wed, Sep 22, 2010 at 8:28 AM, Roan Kattouw roan.katt...@gmail.com wrote: 2010/9/22 Chad innocentkil...@gmail.com: I disagree. Things that work currently with ApiBase would continue to, and things extending ApiQueryBase wouldn't notice a difference if a few of their methods were moved up a level to a new ApiListBase. Moving a few methods from ApiQueryBase up to ApiBase won't break any other ApiBase subclasses. There is no advantage in creating a new class over simply moving this stuff to ApiBase. I thought keeping ApiBase clean of methods most implementations don't need would be a benefit. But I'm willing to concede here :) -Chad ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Balancing MediaWiki Core/Extensions
2010/9/22 Trevor Parscal tpars...@wikimedia.org: Modular feature development being unique to extensions points out a significant flaw in the design of MediaWiki core. There's no reason we can't convert existing core features to discreet components, much like how extensions are written, while leaving them in stock MediaWiki. This sort of design would also make light work of adding extensions to core. Making MediaWiki more modular won't magically make it possible (or even desirable) to write any old feature as an extension. Some things will still have to go in core and some things we'll simply /want/ to put in core because making them extensions would be ridiculous for some reason. It's true that more modularity will lower the difference in awkwardness between doing certain things as extensions vs. in core (and is generally desirable), but I also don't see how it would make (much) lighter work of adding extensions to core: most extensions should either be easy to add to core (Vector and WikiEditor for instance, any special page extension) or cause some bumps because they were written in an awkward way to begin with, probably because it'd have been better to put them in core. Roan Kattouw (Catrope) ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Balancing MediaWiki Core/Extensions
On Tue, Sep 21, 2010 at 11:34 PM, Trevor Parscal tpars...@wikimedia.org wrote: when to move features out of core and into an extension or out of an extension and into core. I don't think anyone's commented on the former (everyone's been talking about pushing in, not pulling out). IMO, the conditions for splitting something into an extension A) Not a lot of people use it anyway (hard to gauge) B) It probably shouldn't have been in core in the first place (eg: AskSQL) So far, the only successful case I can think of offhand for splitting an extension out was AskSQL, but it's a perfect example of what should happen. 2. When an extension is unable to do what it needs to do because it's dependent on a limited set of hooks, none of which quite do what it needs. I think this also makes a good case for keeping something in core and not removing it. If we need to add 6 new hooks to keep it working, then it might be best to keep it in core (single purpose hooks are lame) 1. This is a very valid and important goal, but am unconvinced and merging extensions into core is the only way to achieve it. We can, for instance, take advantage the new installer that demon is working on which has the ability to automate the installation of extensions at setup-time. Quick note on the installer. It only enables extensions that already reside in your extensions folder. Since we don't distribute any with the default package, this might not be terribly useful. More awesome is Jeroen's GSoC work on extension management. Something to look at post-1.17 branch though. 3. Because someone said so. If enough people say so ;-) -Chad ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] Problem with maintenance/dumpBackup.php
The script to dump pages fails with the following error:
zbys...@escher:/srv/www/eduwiki$ sudo -u www-data php
maintenance/dumpBackup.php --full /tmp/dump-full2
PHP Fatal error: Call to undefined method DumpFilter::DumpFilter() in
/home/srv/mediawiki/mediawiki-trunk/maintenance/backup.inc on line 306
This behaviour is the same in 1.16-wmf4 and tr...@72382.
This is easily corrected with the following patch:
diff --git a/maintenance/backup.inc b/maintenance/backup.inc
index 9b2ff89..88da915 100644
--- a/maintenance/backup.inc
+++ b/maintenance/backup.inc
@@ -303,7 +303,7 @@ class BackupDumper {
class ExportProgressFilter extends DumpFilter {
function ExportProgressFilter( $sink, $progress ) {
- parent::DumpFilter( $sink );
+ parent::__construct( $sink );
$this-progress = $progress;
}
HTH,
Asia Jedrzejewska-Szmek
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Problem with maintenance/dumpBackup.php
My fault. Will apply as soon as I get in front of a
computer if no one beats me to it.
On Sep 22, 2010 9:43 AM, Asia Jedrzejewska-Szmek a...@fuw.edu.pl wrote:
The script to dump pages fails with the following error:
zbys...@escher:/srv/www/eduwiki$ sudo -u www-data php
maintenance/dumpBackup.php --full /tmp/dump-full2
PHP Fatal error: Call to undefined method DumpFilter::DumpFilter() in
/home/srv/mediawiki/mediawiki-trunk/maintenance/backup.inc on line 306
This behaviour is the same in 1.16-wmf4 and tr...@72382.
This is easily corrected with the following patch:
diff --git a/maintenance/backup.inc b/maintenance/backup.inc
index 9b2ff89..88da915 100644
--- a/maintenance/backup.inc
+++ b/maintenance/backup.inc
@@ -303,7 +303,7 @@ class BackupDumper {
class ExportProgressFilter extends DumpFilter {
function ExportProgressFilter( $sink, $progress ) {
- parent::DumpFilter( $sink );
+ parent::__construct( $sink );
$this-progress = $progress;
}
HTH,
Asia Jedrzejewska-Szmek
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
Hi, here are my thoughts about phpunit and selenium testing. The wiki under test is set up with a master database consisting of a single objectcache table. The entries of this table specify a test run identifier as primary key and temporary resource identifiers as dependent fields. If I understand this correctly, this would not allow to test any wikis that are running on live sites, e.g. intranet wikis. While I agree that regression testing on live sites is not a good idea, I kind of like the notion that after setting up a wiki with all the extensions I like to have, I could do some sort of everything up and running-test. With the concept of using separate testing databases and resources, this would be possible without interference with the actual data and could even be done at intervals during, say, maintenance periods. Setup of a test run requires the creation of the test run temporary resources and a entry in the objectcache table. Are there already mechanisms for this? I haven't done too much work with the objectcache. This is where memcached data is stored, right? So how do I get the data that is needed? This question leads me to another one: How do I get the testind database and resources? As I see this, it should be part of the testing framework to be able to produce the set of data needed from a normal MW installation. The whole mechanism would actually be something like a backup, so we might look into any existing solutions for that. When a request is sent to the wiki under test, very early in the request processing (e.g., immediately after LocalSettings is processed) a hook is called with the provided state information as an argument that accesses the objectcache table. The extension function handling the hook switches in the temporary resources and returns. Also, this hook might handle the reconfiguration of the wiki, if needed. So for example, testing the PagedTiffHandler requires uploading of tiff files to be enabled. However, there might be some security risks, since it is not directly obvious in the code which settings are changed. So the hook should only be called when $wgEnableSelenium = true. In addition, we could define a DontTouchThisSettings.php, which is called even after the hook and holds some settings that are holy to the admin of the wiki :) The question is, though, would this not become somewhat too complicated? After the test run completes, the testing application cleans up the test run by requesting the deletion of the temporary resources and the objectcache table entry associated with the test run. In some cases, tests will not change any data, e.g. testing dynamic skin elements in vector skin. Would it make sense not to tear down the testing environment in that case in order to save some time when testing repeatedly? I think, there is a conflict between performance and amount of data, but who wins? In general, it seems to me that we have some similarity with what is called wiki family on mediawiki.org. One could see multiple testing environments as a set of multiple wikis that share a common codebase [1]. Does anybody have experience with wiki families and the object cache rsp. memcached? I am not sure whether we can use the same codebase as parsertests. I'd rather think, parser tests are a special case of what we are sketching here. On the other hand, I don't think it is a good idea to have two separate approaches for very similar tasks in the code. Do you think it would be feasible to separate the preparation part from both parser tests and selenium tests and build both of them on a common ground? Best regards, Markus [1] http://www.mediawiki.org/wiki/Wiki_family#Scenario_2:_Quick_set-up ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Balancing MediaWiki Core/Extensions
On 22 September 2010 13:52, Chad innocentkil...@gmail.com wrote: Quick note on the installer. It only enables extensions that already reside in your extensions folder. Since we don't distribute any with the default package, this might not be terribly useful. More awesome is Jeroen's GSoC work on extension management. Something to look at post-1.17 branch though. Note from the perspective of a tarball user: yes please. Though code management may be much more sensible with things organised into the extensions folder, having a basic set of extensions included by default in the tarball would be most helpful to the sysadmin whose user just asked for a new wiki. (I'm thinking by analogy with operating systems - a base set of extensions which are a good working set to start with, but which you can strip right down to core if you really want to.) - d. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Problem with maintenance/dumpBackup.php
On Wed, Sep 22, 2010 at 9:44 AM, Chad innocentkil...@gmail.com wrote:
My fault. Will apply as soon as I get in front of a
computer if no one beats me to it.
On Sep 22, 2010 9:43 AM, Asia Jedrzejewska-Szmek a...@fuw.edu.pl wrote:
The script to dump pages fails with the following error:
zbys...@escher:/srv/www/eduwiki$ sudo -u www-data php
maintenance/dumpBackup.php --full /tmp/dump-full2
PHP Fatal error: Call to undefined method DumpFilter::DumpFilter() in
/home/srv/mediawiki/mediawiki-trunk/maintenance/backup.inc on line 306
This behaviour is the same in 1.16-wmf4 and tr...@72382.
This is easily corrected with the following patch:
diff --git a/maintenance/backup.inc b/maintenance/backup.inc
index 9b2ff89..88da915 100644
--- a/maintenance/backup.inc
+++ b/maintenance/backup.inc
@@ -303,7 +303,7 @@ class BackupDumper {
class ExportProgressFilter extends DumpFilter {
function ExportProgressFilter( $sink, $progress ) {
- parent::DumpFilter( $sink );
+ parent::__construct( $sink );
$this-progress = $progress;
}
HTH,
Asia Jedrzejewska-Szmek
This is fixed in trunk in r73529.
-Chad
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Balancing MediaWiki Core/Extensions
On Wed, Sep 22, 2010 at 6:33 PM, Trevor Parscal tpars...@wikimedia.org wrote: Modular feature development being unique to extensions points out a significant flaw in the design of MediaWiki core. There's no reason we can't convert existing core features to discreet components, much like how extensions are written, while leaving them in stock MediaWiki. This sort of design would also make light work of adding extensions to core. I think that I would support allowing core features to use hooks instead of writing the code in all over the place. All you would have to do is initialise $wgHooks somewhere before LocalSettings.php is run. Giving structural proximity to functionally-related code would make things much less confusing for new developers (and simpler for those with experience). It would allow us to declutter our core code (things like user permission checking come to mind), and it would make light work of merging extensions into core. -- Andrew Garrett http://werdn.us/ ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] API architecture
* Roan Kattouw roan.katt...@gmail.com [Wed, 22 Sep 2010 14:16:06 +0200]: ApiBase - ApiListBase - ApiQueryBase where ApiListBase would have at least these nice SQL query buliding methods but also the possibility to override the default printer? Why these methods are limited to action=query lists and generators only? That's a very good suggestion. Feel free to implement it or, if you can't or don't want to, file a bug requesting it in Bugzilla. I probably can implement it, that should be easy. However, 1. I need to have it working with 1.15.x, the customer uses. So I've made a hack, I extend ApiQueryBase instead of ApiBase, but make constructor like ApiBase. Such way, I have both addWhere*() and setContinueEnumParameter() methods working without being forced to use action=query. I've also overrided getDB(), because ApiQueryBase::getDB is unsuitable to ApiBase code (ApiQueryBase::getQuery() and all methods which use ApiQueryBase::getQuery() are also make no sense, thus I've overrided these throwing MWException() ). 2. I don't have core commit access only extension commit access, however I don't feel like MW guru, so I don't even want to request it. I'll probably file a bug. Too bad it will never come back to 1.15.x, though. Dmitriy ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] Slow file downloads from commons...
Right now I'm seeing that image downloads on wikimedia commons have slowed down by a factor of ten or so since the last weekend. I used to download about 700 images an hour, so it's possible that my bandwidth is being throttled, but it might be something affecting other people too. Something I notice is that sometimes I can download a file very quickly: [p...@haruhi ~]$ curl -O http://upload.wikimedia.org/wikipedia/commons/7/7b/Barabinsk_station1.jpg % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 807k 100 807k0 0 797k 0 0:00:01 0:00:01 --:--:-- 863k Then sometimes the download takes orders of magnitude longer: [p...@haruhi ~]$ curl -O http://upload.wikimedia.org/wikipedia/commons/7/7b/Barabinsk_station1.jpg % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 807k 100 807k0 0 16431 0 0:00:50 0:00:50 --:--:-- 14436 I'm wondering if there's some specific problem here, if I'm being throttled, or if there's just a general need for more bandwidth servers. In any case I'd be pretty happy to make a donation that would cover any operational costs that my activities incur and then some. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
On Wed, 22 Sep 2010 15:49:40 +0200, Markus Glaser wrote: Hi, here are my thoughts about phpunit and selenium testing. The wiki under test is set up with a master database consisting of a single objectcache table. The entries of this table specify a test run identifier as primary key and temporary resource identifiers as dependent fields. If I understand this correctly, this would not allow to test any wikis that are running on live sites, e.g. intranet wikis. While I agree that regression testing on live sites is not a good idea, I kind of like the notion that after setting up a wiki with all the extensions I like to have, I could do some sort of everything up and running-test. With the concept of using separate testing databases and resources, this would be possible without interference with the actual data and could even be done at intervals during, say, maintenance periods. The problem with testing live sites is tests may alter wiki data (consequently, test run reproducibility becomes a problem). If all of the tests are read-only, then that isn't a problem, but it means developing a whole set of tests that conform to that constraint. Nevertheless, it wouldn't be hard to design the switching mechanism to allow the testing of live sites. There could be an option in test setup and cleanup that effectively says don't switch-in/clean-up temporary resources. Personally, I think use of this option is dangerous, but it wouldn't be hard to implement. Setup of a test run requires the creation of the test run temporary resources and a entry in the objectcache table. Are there already mechanisms for this? I haven't done too much work with the objectcache. This is where memcached data is stored, right? So how do I get the data that is needed? This question leads me to another one: How do I get the testind database and resources? As I see this, it should be part of the testing framework to be able to produce the set of data needed from a normal MW installation. The whole mechanism would actually be something like a backup, so we might look into any existing solutions for that. We should use the existing ObjectCache class to manage the object cache. However, if there exists some switching-in code, I doubt it has corresponding clean-up code. So, we probably need to do some development even if we use existing mechanisms. I think the object cache and memcached are alternative ways of storing persistent data. (I also am not an expert in this, so I could be wrong). My understanding is memcached uses the memcached daemon (http:// memcached.org/), while the object cache uses the underlying database. If so, then memcached data disappears after a system crash or power outage, whereas object cache data should survive. You are absolutely correct that we need to figure out how to clone a set of temporary resources (db, images directory, perhaps cache data) and set them up for use (i.e., so the switch-in logic can copy them for the test run). There are a number of problems to solve, e.g., 1) how do you package the cloned resources (e.g., tar file), 2) how do you efficiently copy them during the switch-in process, 3) security issues, 4) resource management issues. When a request is sent to the wiki under test, very early in the request processing (e.g., immediately after LocalSettings is processed) a hook is called with the provided state information as an argument that accesses the objectcache table. The extension function handling the hook switches in the temporary resources and returns. Also, this hook might handle the reconfiguration of the wiki, if needed. So for example, testing the PagedTiffHandler requires uploading of tiff files to be enabled. However, there might be some security risks, since it is not directly obvious in the code which settings are changed. So the hook should only be called when $wgEnableSelenium = true. In addition, we could define a DontTouchThisSettings.php, which is called even after the hook and holds some settings that are holy to the admin of the wiki :) The question is, though, would this not become somewhat too complicated? I have done some research on possible hooks. I think the SetupAfterCache hook is a good candidate. The problem with calling a (psuedo-)hook in LocalSettings is there is some setup not completed when LocatSettings is processed (e.g., the WebRequest object is not yet available. the ObjectCache class file is not yet included). Most setup is completed before calling the SetupAfterCache hook. It would be possible to do some configuration in this hook, but I think we need to consider efficiency. Preferably, the creation of the base resources that are copied during the switch-in will be configured as much as possible. For example, I think creating the base for PagedTiffHandler should start with a freshly installed wiki and upload multipage.tiff. The resulting db could be dumped and the images
Re: [Wikitech-l] Balancing MediaWiki Core/Extensions
On Tue, Sep 21, 2010 at 11:34 PM, Trevor Parscal tpars...@wikimedia.org wrote: In response to recent comments in our code review tool about whether some extensions should be merged into core MediaWiki, or not. I would like to try and initiate a productive conversation about this topic in hopes that we can collaboratively define a set of guidelines for evaluating when to move features out of core and into an extension or out of an extension and into core. Partly this is a matter of preference. I personally write things almost entirely for core -- I've never written an extension, so my only extension commits are to things that other people wrote as extensions. Other people prefer to write extensions instead of core features. As long as the code gets written and works either way, I don't think it's usually a huge deal. A lot of stuff will work fine either in core or as an extension -- this doesn't have to be made a religious issue or one where we have detailed official policies. What I think is clear is that anything that the large majority of MediaWikis would benefit from should be enabled by default, which given our current setup means it has to be in core. I haven't looked closely, but I believe this includes a significant part of the usability stuff (although not all). MediaWiki should work as well as possible for as many users as possible out of the box, and whatever that implies for how the code is structured is what we should do. I cannot conceive of any reason why most of the features of the Vector extension should not be in core, for instance -- why would you *not* want them? Why would you ever want to disable them? (Admins should have choice is not a good reason by itself. Cf.: http://weblogs.mozillazine.org/bz/archives/020683.html) Other than things you want to enable by default, here are some reasons to put some features in core rather than extensions: * Core features are much easier for site admins to enable than extensions. It would be great if this were fixed, but currently, it's not, so it can't be dismissed. You can dismiss this reason (basically your #1) as soon as it's become irrelevant in practice, but not before. * For smaller features, it will take a lot more code to write them as extensions than as part of core. You can often add a useful core feature in less than ten lines of code -- extensions are much heavier-weight. This is unavoidable, because extensions have to have a whole bunch of metadata to specify where their code executes and so on; with core code you can just put it in the right place. * It's easier to understand how core code works in the absence of extensions, because the code is explicitly written out where it occurs. When extensions are involved, you have a wfRunHooks() line in core, which basically means absolutely anything could happen at this point. When you run into a hook when reading code, it doesn't only complicate understanding how the code will run in practice, it also makes it much harder to change. If you want to refactor, you have to write a lot of painstaking and ugly logic to preserve the semantics of the hooks, or else just give up and get rid of them. On Wed, Sep 22, 2010 at 4:33 AM, Trevor Parscal tpars...@wikimedia.org wrote: Modular feature development being unique to extensions points out a significant flaw in the design of MediaWiki core. There's no reason we can't convert existing core features to discreet components, much like how extensions are written, while leaving them in stock MediaWiki. This can be accomplished by regular old object-oriented programming. Keep all the related code in one class, and make as much possible private so that classes can only call a limited number of well-defined, well-documented, easily-understood entry points. I don't think extensions are a good model for this -- to the contrary, they're antithetical to modularity, since any extension can drastically change how any part of the program behaves (given enough hooks). On Wed, Sep 22, 2010 at 10:17 AM, Andrew Garrett agarr...@wikimedia.org wrote: I think that I would support allowing core features to use hooks instead of writing the code in all over the place. All you would have to do is initialise $wgHooks somewhere before LocalSettings.php is run. Giving structural proximity to functionally-related code would make things much less confusing for new developers (and simpler for those with experience). It would allow us to declutter our core code (things like user permission checking come to mind), and it would make light work of merging extensions into core. This is reminiscent of COMEFROM. It makes it really hard to understand what's going on if you have the code scattered everywhere. Modularity is great, but it needs to happen along clear boundaries to keep the code comprehensible -- not anything can happen at this point. ___ Wikitech-l mailing list
Re: [Wikitech-l] using parserTests code for selenium test framework
On Wed, Sep 22, 2010 at 6:47 PM, Dan Nessett dness...@yahoo.com wrote: I think the object cache and memcached are alternative ways of storing persistent data. (I also am not an expert in this, so I could be wrong). My understanding is memcached uses the memcached daemon (http:// memcached.org/), while the object cache uses the underlying database. If so, then memcached data disappears after a system crash or power outage, whereas object cache data should survive. There is no ObjectCache class. Basically, there is a common base class called BagOStuff and from that various classes for various backends are defined such as SqlBagOStuff and MemcacheBagOStuff. To the code outside that class, there is no visible difference. Bryan ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] API architecture
2010/9/22 Dmitriy Sintsov ques...@rambler.ru: 2. I don't have core commit access only extension commit access, however I don't feel like MW guru, so I don't even want to request it. I'll probably file a bug. You can submit patches on Bugzilla too :) Roan Kattouw (Catrope) ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
On Wed, 22 Sep 2010 18:57:12 +0200, Bryan Tong Minh wrote: On Wed, Sep 22, 2010 at 6:47 PM, Dan Nessett dness...@yahoo.com wrote: I think the object cache and memcached are alternative ways of storing persistent data. (I also am not an expert in this, so I could be wrong). My understanding is memcached uses the memcached daemon (http:// memcached.org/), while the object cache uses the underlying database. If so, then memcached data disappears after a system crash or power outage, whereas object cache data should survive. There is no ObjectCache class. Basically, there is a common base class called BagOStuff and from that various classes for various backends are defined such as SqlBagOStuff and MemcacheBagOStuff. To the code outside that class, there is no visible difference. Bryan Thanks for the clarification. I just looked at ObjectCache.php and it appears to provide a set of functions for accessing cache data of any type. Is this correct? How does memcached fit into this? When I looked at BagOStuff, I didn't find a MemcacheBagOStuff class. Is it defined elsewhere? -- -- Dan Nessett ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] MediaWiki 1.17 release target (Re: UsabilityInitiative extension...)
On Tue, Sep 21, 2010 at 2:59 PM, Rob Lanphier ro...@wikimedia.org wrote: So, let's assume that the existing trunk (ResourceLoader and all) is a required part of 1.17. What's the earliest reasonable time we could expect to branch point for 1.17? Whenever the code is all reviewed. If Wikimedia deployed enough resources to reviewing code, I don't know, maybe that could be a month or two from now. If the status quo on code review is kept, it will probably be never. Of course, we could just accept that review won't keep up with trunk, if we want to just give up on ever getting volunteer code deployed in a timely fashion again. In that case, we could branch from the last reviewed revision, and base both production and 1.17 off that. But I'm kind of hoping that's not necessary or desired by the powers that be. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
2010/9/22 Dan Nessett dness...@yahoo.com: How does memcached fit into this? When I looked at BagOStuff, I didn't find a MemcacheBagOStuff class. Is it defined elsewhere? Either memcached.php, MemCached.php or MWMemcached.php, I forget. The class name is MWMemcached. Roan Kattouw (Catrope) ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] ResourceLoader, now in trunk!
Another bugs: * https://bugzilla.wikimedia.org/show_bug.cgi?id=25251 * https://bugzilla.wikimedia.org/show_bug.cgi?id=25252 * https://bugzilla.wikimedia.org/show_bug.cgi?id=25253 -- Leinad 2010/9/22 Daniel ~ Leinad danny.lei...@gmail.com: Please fix https://bugzilla.wikimedia.org/show_bug.cgi?id=25239 ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Balancing MediaWiki Core/Extensions
On 9/22/10 9:49 AM, Aryeh Gregor wrote: Partly this is a matter of preference. I personally write things almost entirely for core -- I've never written an extension, so my only extension commits are to things that other people wrote as extensions. Other people prefer to write extensions instead of core features. As long as the code gets written and works either way, I don't think it's usually a huge deal. A lot of stuff will work fine either in core or as an extension -- this doesn't have to be made a religious issue or one where we have detailed official policies. I certainly never mentioned anything about making this a religious issue (metaphor understood) but I think being able to point to a simple guideline on the wiki would be an improvement on the current situation. This can be accomplished by regular old object-oriented programming. Keep all the related code in one class, and make as much possible private so that classes can only call a limited number of well-defined, well-documented, easily-understood entry points. I don't think extensions are a good model for this -- to the contrary, they're antithetical to modularity, since any extension can drastically change how any part of the program behaves (given enough hooks). And what about javascript and css files? What about messages? Generally speaking lots of the MediaWiki PHP code is modular, but it's the rest of the stuff that goes into making a feature work that is not. Extensions bundle this all together, and that proximity makes maintenance and review easier. - Trevor ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
I just looked at SqlBagOStuff. It already has entry expiration logic. So, it seems perfect for use by the switch-in/clean-up functionality. I spoke too soon. Expired entries are deleted automatically when any entry is referenced. Unfortunately, that means there is no opportunity to reclaim the temporary resources identified in the entry before its state is lost. This is a problem. -- -- Dan Nessett ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
Hmm, I think you guys are overthinking the details on this; let's step back a level. When you're running tests, you have these tasks: * create a blank-slate wiki to run tests in * populate the empty wiki with known data * run tests on this wiki in a known state * clean up in some way The parserTests system is designed with some particular constraints: * run within the local MediaWiki test instance a developer already has, without altering its contents * run all tests in a single command-line program * expose no in-test data to the outside world It can do this very easily using temporary tables and a temporary directory because it only needs to work within that one single test process; once it's done, it's done and all the temporary data can be discarded. Nothing needs to be kept across processes or exposed to clients. For Selenium tests, you have a very different set of constraints: * test data must be exposed to web clients over a web server * test data must be retained across multiple requests The simplest way to accomplish this is to have a dedicated, web-exposed test wiki instance. A test run would go like this: * (re)initialize test wiki into known state * run series of tests I'd recommend not trying to use the parserTest harness/initialization for this; it'll be a *lot* simpler to just script creation of a fresh wiki. (drop database, create database, slurp tables, run update.php) Some non-destructive tests can always be run on any existing instance -- and probably should be! -- and some 'active' tests will be freely runnable on existing instances that are used for development and testing, but if you want to work with a blank slate wiki exposed to web clients, keep things simple and just make a dedicated instance. -- brion On Wed, Sep 22, 2010 at 9:47 AM, Dan Nessett dness...@yahoo.com wrote: On Wed, 22 Sep 2010 15:49:40 +0200, Markus Glaser wrote: Hi, here are my thoughts about phpunit and selenium testing. The wiki under test is set up with a master database consisting of a single objectcache table. The entries of this table specify a test run identifier as primary key and temporary resource identifiers as dependent fields. If I understand this correctly, this would not allow to test any wikis that are running on live sites, e.g. intranet wikis. While I agree that regression testing on live sites is not a good idea, I kind of like the notion that after setting up a wiki with all the extensions I like to have, I could do some sort of everything up and running-test. With the concept of using separate testing databases and resources, this would be possible without interference with the actual data and could even be done at intervals during, say, maintenance periods. The problem with testing live sites is tests may alter wiki data (consequently, test run reproducibility becomes a problem). If all of the tests are read-only, then that isn't a problem, but it means developing a whole set of tests that conform to that constraint. Nevertheless, it wouldn't be hard to design the switching mechanism to allow the testing of live sites. There could be an option in test setup and cleanup that effectively says don't switch-in/clean-up temporary resources. Personally, I think use of this option is dangerous, but it wouldn't be hard to implement. Setup of a test run requires the creation of the test run temporary resources and a entry in the objectcache table. Are there already mechanisms for this? I haven't done too much work with the objectcache. This is where memcached data is stored, right? So how do I get the data that is needed? This question leads me to another one: How do I get the testind database and resources? As I see this, it should be part of the testing framework to be able to produce the set of data needed from a normal MW installation. The whole mechanism would actually be something like a backup, so we might look into any existing solutions for that. We should use the existing ObjectCache class to manage the object cache. However, if there exists some switching-in code, I doubt it has corresponding clean-up code. So, we probably need to do some development even if we use existing mechanisms. I think the object cache and memcached are alternative ways of storing persistent data. (I also am not an expert in this, so I could be wrong). My understanding is memcached uses the memcached daemon (http:// memcached.org/), while the object cache uses the underlying database. If so, then memcached data disappears after a system crash or power outage, whereas object cache data should survive. You are absolutely correct that we need to figure out how to clone a set of temporary resources (db, images directory, perhaps cache data) and set them up for use (i.e., so the switch-in logic can copy them for the test run). There are a number of problems to solve, e.g., 1) how do you package the cloned resources (e.g., tar
Re: [Wikitech-l] using parserTests code for selenium test framework
On Wed, 22 Sep 2010 11:00:53 -0700, Brion Vibber wrote: Hmm, I think you guys are overthinking the details on this; let's step back a level. When you're running tests, you have these tasks: * create a blank-slate wiki to run tests in * populate the empty wiki with known data * run tests on this wiki in a known state * clean up in some way The parserTests system is designed with some particular constraints: * run within the local MediaWiki test instance a developer already has, without altering its contents * run all tests in a single command-line program * expose no in-test data to the outside world It can do this very easily using temporary tables and a temporary directory because it only needs to work within that one single test process; once it's done, it's done and all the temporary data can be discarded. Nothing needs to be kept across processes or exposed to clients. For Selenium tests, you have a very different set of constraints: * test data must be exposed to web clients over a web server * test data must be retained across multiple requests The simplest way to accomplish this is to have a dedicated, web-exposed test wiki instance. A test run would go like this: * (re)initialize test wiki into known state * run series of tests I'd recommend not trying to use the parserTest harness/initialization for this; it'll be a *lot* simpler to just script creation of a fresh wiki. (drop database, create database, slurp tables, run update.php) Some non-destructive tests can always be run on any existing instance -- and probably should be! -- and some 'active' tests will be freely runnable on existing instances that are used for development and testing, but if you want to work with a blank slate wiki exposed to web clients, keep things simple and just make a dedicated instance. -- brion On Wed, Sep 22, 2010 at 9:47 AM, Dan Nessett dness...@yahoo.com wrote: On Wed, 22 Sep 2010 15:49:40 +0200, Markus Glaser wrote: Hi, here are my thoughts about phpunit and selenium testing. The wiki under test is set up with a master database consisting of a single objectcache table. The entries of this table specify a test run identifier as primary key and temporary resource identifiers as dependent fields. If I understand this correctly, this would not allow to test any wikis that are running on live sites, e.g. intranet wikis. While I agree that regression testing on live sites is not a good idea, I kind of like the notion that after setting up a wiki with all the extensions I like to have, I could do some sort of everything up and running-test. With the concept of using separate testing databases and resources, this would be possible without interference with the actual data and could even be done at intervals during, say, maintenance periods. The problem with testing live sites is tests may alter wiki data (consequently, test run reproducibility becomes a problem). If all of the tests are read-only, then that isn't a problem, but it means developing a whole set of tests that conform to that constraint. Nevertheless, it wouldn't be hard to design the switching mechanism to allow the testing of live sites. There could be an option in test setup and cleanup that effectively says don't switch-in/clean-up temporary resources. Personally, I think use of this option is dangerous, but it wouldn't be hard to implement. Setup of a test run requires the creation of the test run temporary resources and a entry in the objectcache table. Are there already mechanisms for this? I haven't done too much work with the objectcache. This is where memcached data is stored, right? So how do I get the data that is needed? This question leads me to another one: How do I get the testind database and resources? As I see this, it should be part of the testing framework to be able to produce the set of data needed from a normal MW installation. The whole mechanism would actually be something like a backup, so we might look into any existing solutions for that. We should use the existing ObjectCache class to manage the object cache. However, if there exists some switching-in code, I doubt it has corresponding clean-up code. So, we probably need to do some development even if we use existing mechanisms. I think the object cache and memcached are alternative ways of storing persistent data. (I also am not an expert in this, so I could be wrong). My understanding is memcached uses the memcached daemon (http:// memcached.org/), while the object cache uses the underlying database. If so, then memcached data disappears after a system crash or power outage, whereas object cache data should survive. You are absolutely correct that we need to figure out how to clone a set of temporary resources (db, images directory, perhaps cache data) and set them up for use (i.e., so the switch-in logic can copy them for the test run).
Re: [Wikitech-l] using parserTests code for selenium test framework
On Wed, Sep 22, 2010 at 11:09 AM, Dan Nessett dness...@yahoo.com wrote: Some have mentioned the possibility of using the wiki family logic to help achieve these objectives. Do you have any thoughts on this? If you think it is a good idea, how do we find out more about it? I'd just treat it same as any other wiki. Whether you're running one or multiple wiki instances out of one copy of the code base, it just doesn't make a difference here. -- brion ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] using parserTests code for selenium test framework
On Wed, 22 Sep 2010 12:30:35 -0700, Brion Vibber wrote: On Wed, Sep 22, 2010 at 11:09 AM, Dan Nessett dness...@yahoo.com wrote: Some have mentioned the possibility of using the wiki family logic to help achieve these objectives. Do you have any thoughts on this? If you think it is a good idea, how do we find out more about it? I'd just treat it same as any other wiki. Whether you're running one or multiple wiki instances out of one copy of the code base, it just doesn't make a difference here. -- brion Oh. Well, perhaps we don't agree exactly 100%. As you suggest, let's step back a bit. Once we get the selenium framework working I assume it will be used for a regression test. This will comprise a set of individual tests. Generally, these tests will write into the wiki db (some may not, but many will). To ensure test reproducibility, the state of the wiki should be the same each time one of these individual tests runs. But, there is a problem. With parserTests, each individual test runs serially. That is fine for parserTests, since (I just ran this on my machine) while there are 610 individual tests, each takes about .08 seconds to run (on average). So, on my machine the whole parserTest takes about 48 seconds. Selenium tests are far more heavy-weight. A rough ball-park figure is each takes about 10 seconds to run (this does not include the time it would take to setup and tear down a clean wiki). So, a selenium-based regression test comprising 180 individual tests would take around 30 minutes. Not too bad. But, things are a bit more complicated. Each individual test runs multiple times, once for every browser/OS combination chosen for the regression test. For example, right now there are 13 configured browser/ OS combinations on the WMF Selenium Grid (see http:// grid.tesla.usability.wikimedia.org/console). So even if you only test 4 of these browser/OS configurations, the regression test (if individual tests run serially) would take 2 hours. If you test 8 of them, it would take 4 hours. This is starting to get onerous. If an individual developer wishes to ensure his modifications don't break things before committing his changes, then waiting 4 hours for a regression test to complete is a pretty heavy penalty. Generally, very few will pay the price. So, running the individual tests of a selenium-based regression test serially is not very attractive. This means you need to achieve some concurrency in the regression test. Since individual tests may interfere with each other, you need a way to protect them from each other. This is what the switching functionality is for. You switch in a base set of temporary resources for each test (or perhaps more likely for a particular test suite comprised of mulitple individual tests) consisting of a db, images directory, etc. This allows tests to run without interfering with each other. -- -- Dan Nessett ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Balancing MediaWiki Core/Extensions
I suppose I have a foot in each camp here. Ryan Lane wrote: ParserFunctions is an amazing example of this. MediaWiki simply doesn't work without ParserFunctions. You can tell me it does, and that people can live without it, but I refuse to believe that. We get support issues extremely frequently that end in install ParserFunctions. I wrote ParserFunctions under protest, and I hoped at the time that it would not be essential or widely used. That's why it was an extension. Maybe the time has come to merge it in. I also wrote the ImageMap extension, which some might argue should be integrated. Aryeh Gregor wrote: Partly this is a matter of preference. I personally write things almost entirely for core -- I've never written an extension, so my only extension commits are to things that other people wrote as extensions. I agree that there is an element of personal preference. Brion was certainly in favour of putting lots of functionality in extensions, and, like Trevor, used the argument that at some point in the future, extensions will be easier to install and/or bundled with core releases. Brion wrote the Poem extension, for instance, which is easily as trivial as Trevor's ArticleEmblems extension. My view is somewhat more moderate. I think that: * Trivial, uncontroversial things like EditWarning should be in the core. * MediaWiki should be usable without extensions, some basic set of functionality is required. * Complex functionality, large code size, complex installation and external dependencies are all arguments for splitting a feature out to an extension. Trevor Parscal wrote: In response to recent comments in our code review tool about whether some extensions should be merged into core MediaWiki, or not. I would like to try and initiate a productive conversation about this topic in hopes that we can collaboratively define a set of guidelines for evaluating when to move features out of core and into an extension or out of an extension and into core. I think there is a need for judgement, and a definitive set of guidelines will be difficult to produce. 3. Noting that so-and-so said I should integrate this into core is not going to magically absolve anyone of having to stand behind their decision to proceed with such action and support it with logic and reason. Yes, however, community consensus should be respected. The logic and reason you've been expressing in this thread does not appear to be particularly more convincing than anyone else's. Aryeh Gregor wrote: * It's easier to understand how core code works in the absence of extensions, because the code is explicitly written out where it occurs. When extensions are involved, you have a wfRunHooks() line in core, which basically means absolutely anything could happen at this point. When you run into a hook when reading code, it doesn't only complicate understanding how the code will run in practice, it also makes it much harder to change. If you want to refactor, you have to write a lot of painstaking and ugly logic to preserve the semantics of the hooks, or else just give up and get rid of them. I agree with this. The point of modularity is to separate concerns, and a hook-based model is opposed to this. A well-designed modular architecture has a simple, well-documented interface between the modules. Some of our extensions are a long way from this. Nowadays I prefer adding object-oriented extension interfaces, which work by having the extension subclass a core base class. See for instance MediaHandler, FileRepo, LBFactory and PoolCounter. Trevor Parscal wrote: And what about javascript and css files? What about messages? Generally speaking lots of the MediaWiki PHP code is modular, but it's the rest of the stuff that goes into making a feature work that is not. Extensions bundle this all together, and that proximity makes maintenance and review easier. There are security problems with having web-accessible files like JavaScript and CSS in the same directory as non-web-accessible files like PHP includes and command-line scripts. Perhaps the resource loader will fix this, and we'll be able to organise such files by module in the future. There is a significant performance advantage to splitting up messages by language. However it is possible to have module-specific message files in the core -- we have one now for the new installer. -- Tim Starling ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
