Re: [Toolserver-l] [Labs-l] A proposal for better tool discoverability
Thanks Husky,
I've added WDQ to try it. At what interval does it refresh?
It might be helpful if I could structure (group) tools on the wiki page.
Does the directory ignore not-well-formatted lines (e.g. headings)?
Cheers,
Magnus
On Wed, Aug 13, 2014 at 4:15 PM, Hay (Husky) hus...@gmail.com wrote:
Hey everyone,
for those who attended Wikimania, i hope you all had a wonderful time!
One of the things that i've talked about at Mania with a few people is
the current way of 'tool discovery', or rather, the lack thereof.
We've seen really cool things done by the community, but i have a
feeling many of the tools go by unnoticed, simply because nobody knows
that they exist. Maybe they've been send around here or posted on IRC,
but after the initial interest many of them are forgotten and lead a
lonely life on our virtual servers.
I think this is especially problematic for non-community members for
whom the tools might be very useful. For example, as a Wikipedian in
Residence and GLAM advocate i have sent numerous mails with links to
the various GLAM tools made by Magnus, stats.grok.se, etcetera.
But, you might say, we run all these knowledge sites called wikis
right? And there are some pages there that list tools, don't they?
You're right, the community has put some efforts in a few pages (1).
However, IMHO, a wiki doesn't lend itself to easy tool discovery:
* The format on how to describe a tool is unclear, thereby leading to
both very long and very short descriptions.
* It's not possible to easily search for tools in a certain category
(e.g. 'GLAM' or 'editcount').
* The categorization of tools leads to much clicking around,
distracting and confusing users.
* There's no way to sort results, for example by the number of people
using the tool.
The biggest problem of all is the disconnection between the actual
code of the tool itself (hosted in a Git repo), and the description
advertising it. People update their tool to do something different (or
deprecate it), but they forget the documentation on the wiki, don't
remove it when the tool doesn't work anymore, etcetera.
I believe we can, and should do better.
THE PROPOSAL
My proposal is to use the same mechanisms that already exist for
package managers such as NPM (2) or 'app stores', such as the Chrome
store (3). Basically this involes adding a small JSON file to your
project, including a few key properties such as 'title' and
'description'. These files get indexed, and an easy to use frontend to
search to all the tools is provided to end users. You could imagine a
'toolinfo.json' file to look something like this:
{
name : WikiDataQuery,
description : An API for Wikidata items and properties.,
url : http://wdq.wmflabs.org/;,
keywords : wikidata, api, query,
author : Magnus Manske
}
If you have a web-hosted tool, simply stick it in the root of your
tools directory so that it's reachable by the crawler. Whenever your
tool data changes, just update the file and the directory will
automatically update the directory site.
The link to your toolinfo.json could be added to a Wiki page so that
it's easy to remove your tool from the directory or change the URL to
the JSON file.
I'm a firm believer in putting code where your mouth is, so i've
hacked up a working tool directory here:
http://tools.wmflabs.org/hay/directory
Try searching for stuff, clicking on the labels. To add your own tool,
scroll the page down for instructions.
Current this only lists my tools, but i hope that this directory will
soon grow with everything the Wikimedia community has to offer.
I'm interested in your opinion in this proposal and, if you like it,
add a toolinfo.json to your project!
Kind regards,
-- Hay / [[User:Husky]]
1: https://en.wikipedia.org/wiki/Wikipedia:Tools
2: https://www.npmjs.org/
3: https://developer.chrome.com/apps/manifest
___
Labs-l mailing list
lab...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list:
https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] [Labs-l] A proposal for better tool discoverability
Also: Can the directory detect arrays? For example, in the wikidata-todo
tool, I have many smaller tools. I could make a file for each, but that
seems overkill.
Directory tool could do object=single tool, array=multiple tools in
one file.
Cheers,
Magnus
On Wed, Aug 13, 2014 at 4:53 PM, Magnus Manske magnusman...@googlemail.com
wrote:
Thanks Husky,
I've added WDQ to try it. At what interval does it refresh?
It might be helpful if I could structure (group) tools on the wiki page.
Does the directory ignore not-well-formatted lines (e.g. headings)?
Cheers,
Magnus
On Wed, Aug 13, 2014 at 4:15 PM, Hay (Husky) hus...@gmail.com wrote:
Hey everyone,
for those who attended Wikimania, i hope you all had a wonderful time!
One of the things that i've talked about at Mania with a few people is
the current way of 'tool discovery', or rather, the lack thereof.
We've seen really cool things done by the community, but i have a
feeling many of the tools go by unnoticed, simply because nobody knows
that they exist. Maybe they've been send around here or posted on IRC,
but after the initial interest many of them are forgotten and lead a
lonely life on our virtual servers.
I think this is especially problematic for non-community members for
whom the tools might be very useful. For example, as a Wikipedian in
Residence and GLAM advocate i have sent numerous mails with links to
the various GLAM tools made by Magnus, stats.grok.se, etcetera.
But, you might say, we run all these knowledge sites called wikis
right? And there are some pages there that list tools, don't they?
You're right, the community has put some efforts in a few pages (1).
However, IMHO, a wiki doesn't lend itself to easy tool discovery:
* The format on how to describe a tool is unclear, thereby leading to
both very long and very short descriptions.
* It's not possible to easily search for tools in a certain category
(e.g. 'GLAM' or 'editcount').
* The categorization of tools leads to much clicking around,
distracting and confusing users.
* There's no way to sort results, for example by the number of people
using the tool.
The biggest problem of all is the disconnection between the actual
code of the tool itself (hosted in a Git repo), and the description
advertising it. People update their tool to do something different (or
deprecate it), but they forget the documentation on the wiki, don't
remove it when the tool doesn't work anymore, etcetera.
I believe we can, and should do better.
THE PROPOSAL
My proposal is to use the same mechanisms that already exist for
package managers such as NPM (2) or 'app stores', such as the Chrome
store (3). Basically this involes adding a small JSON file to your
project, including a few key properties such as 'title' and
'description'. These files get indexed, and an easy to use frontend to
search to all the tools is provided to end users. You could imagine a
'toolinfo.json' file to look something like this:
{
name : WikiDataQuery,
description : An API for Wikidata items and properties.,
url : http://wdq.wmflabs.org/;,
keywords : wikidata, api, query,
author : Magnus Manske
}
If you have a web-hosted tool, simply stick it in the root of your
tools directory so that it's reachable by the crawler. Whenever your
tool data changes, just update the file and the directory will
automatically update the directory site.
The link to your toolinfo.json could be added to a Wiki page so that
it's easy to remove your tool from the directory or change the URL to
the JSON file.
I'm a firm believer in putting code where your mouth is, so i've
hacked up a working tool directory here:
http://tools.wmflabs.org/hay/directory
Try searching for stuff, clicking on the labels. To add your own tool,
scroll the page down for instructions.
Current this only lists my tools, but i hope that this directory will
soon grow with everything the Wikimedia community has to offer.
I'm interested in your opinion in this proposal and, if you like it,
add a toolinfo.json to your project!
Kind regards,
-- Hay / [[User:Husky]]
1: https://en.wikipedia.org/wiki/Wikipedia:Tools
2: https://www.npmjs.org/
3: https://developer.chrome.com/apps/manifest
___
Labs-l mailing list
lab...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list:
https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] [Labs-l] A proposal for better tool discoverability
On Wed, Aug 13, 2014 at 5:16 PM, Hay (Husky) hus...@gmail.com wrote: Hey Magnus, thanks! WDQ is now added to the directory. The cronjob should run every hour. Having a list of tools as an array is something i've been thinking about as well, will probably add that. Thanks, I'll hold off some of my tools then, until this is done. 'Grouping' tools can be done by simply adding a category. Or do you mean something else? I mean, visually separating groups on your wiki page, within the source tag. Easier to check if I have a tool added there if mine are all in a section. But not really a show-stopper :-) This is great! Cheers, Magnus ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Need Tool Labs migration support?
It is annoying, but to be fair, it's only been this bad since yesterday or so. Usually it's OK. On Thu, Oct 10, 2013 at 11:50 AM, Christian Thiele ap...@apper.de wrote: Hi, Am 09.10.2013, 14:55 Uhr, schrieb Magnus Manske magnusman...@googlemail.com: There are some tools from e.g. apper that would probably move quite well. I just started moving some simple tools a few days ago, but the availability seems very bad to me. Sometimes the tools work, sometimes not - at the moment nothing works for me, not only my tools but also e.g. http://tools.wmflabs.org/add-**information/http://tools.wmflabs.org/add-information/or http://tools.wmflabs.org/**catfood/ http://tools.wmflabs.org/catfood/ It was told that the toolserver will die one year after Tool Labs has all necessary features to replace the Toolserver.. One feature is availability - and for now it's impossible for me to move tools if they only work 50% of the day. So either the availability will be at nearly 100% or the shutdown will be delayed. Regards, Christian / APPER __**_ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.**orgToolserver-l@lists.wikimedia.org ) https://lists.wikimedia.org/**mailman/listinfo/toolserver-lhttps://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/** view/Mailing_list_etiquettehttps://wiki.toolserver.org/view/Mailing_list_etiquette -- undefined ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Need Tool Labs migration support?
On Thu, Oct 10, 2013 at 5:58 PM, Marc A. Pelletier m...@uberbox.org wrote: On 10/10/2013 12:32 PM, Silke Meyer wrote: ok, I consulted Nosy, DaB. and Mette about this proposal and we will not make the home directories accessible to others. They can contain all sorts of private data, user names, e-mail addresses, ssh keys, who knows what else - maybe even gpg keys, so handing the complete package over is no option. I wouldn't want to intrude into the people's stuff. And we'd get in trouble with the law here, as the German data protection act is binding for us. I agree with this as well, for those very reasons. This is the reason why Tool Labs insists that any real tool is run from multi-maintainer project equivalents, and with clear licensing. I would not allow user's own home directories to be turned over to a third party under any circumstances, this is why we want no code running there we might want to salvage in the future. And, likewise, I'm pretty sure the foundation wouldn't /want/ to be given custody over data that may or may not be private from contributors who have not given explicit leave to do so. To be clear, I was not suggesting to just dump all home directories on bittorrent ;-) public_html and cgi-bin should contain most code, and for small tools, could be checked quickly by hand for personal/secret data. Also, of course I prefer people moving their own tools! Just in the important-tool-will-die-with-toolserver scenario... Magnus ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Need Tool Labs migration support?
Interesting. Most of this seems to be related to OSM/geo-something. Not sure how much could actually be ported to Labs at the moment. There are some tools from e.g. apper that would probably move quite well. Also, there is geohack, which I had moved to Labs quite a while ago ( http://tools.wmflabs.org/geohack/ ). Not sure why it's not redirected/URLs on Wikipedia haven't changed yet. On Wed, Oct 9, 2013 at 1:26 PM, Marlen Caemmerer marlen.caemme...@wikimedia.de wrote: Hello, reading this I feel forced to publish my short Top50 usage of yesterday URLs from Toolserver. Its just a quick hack on my console but I guess it is more clear output then probably a webalizer could offer. I just counted the lines in my access log from yesterday of how often a user dir was called. This is my command line: zcat access.log.1.gz|cut -d \/ -f 4,5|cut -d \? -f 1|sort |uniq -c|sort -rn 4410444 tiles/hikebike 3410590 ~cmarqu/hill 3030592 ~dschwen/wma 1568706 tiles/osm 651872 tiles/osm-labels-ru 452436 (should be toolserver.org) 242462 ~dapete/ime 191536 ~kolossos/openlayers 172239 ~para/geoip.fcgi HTTP 146808 ~kolossos/geoworld 117295 ~dschwen/iip 114200 tiles/bw-mapnik 112167 ~para/region.php 104377 tiles/osm-no-labels 99926 ~apper/pd 85028 ~osm/libs 63087 ~apper/sc 61585 ~dispenser/temp 59166 ~dapete/random 53725 ~dispenser/cgi-bin 50095 ~para/GeoCommons 48399 ~geohack/geohack.php 46208 ~lvova/cgi-bin 43747 images/wikimedia-toolserver-**button.png HTTP 30192 ~tparis/autoedits 26460 ~geohack/siteicon.png HTTP 24401 ~master/osmjson 23565 ~dispenser/ghct 22924 ~cmarqu/opentiles.com 22199 ~mzmcbride/redirector 22089 ~daniel/potd 21928 geohack/geohack.php 20984 ~para/geoip.fcgi 20635 ~chm/whois.php HTTP 16343 ~quentinv57/sulinfo 15717 ~krinkle/I18N 11955 ~daniel/WikiSense 11533 ~soxred93/images 11007 ~authoritycontrol/redirect 9610 ~dapete/rss 8677 favicon.ico HTTP/1.1 200 1150 - 8492 %7Egeohack/geohack.php 7913 tsthumb/tsthumb 7707 tiles/osm-labels-de 7460 ~rhaworth/os 7363 ~dispenser/resources 6789 ~para/cgi-bin 6561 tiles/osm-labels-en 6543 ~dapete/wikinews-rss 6205 ~dapete/ImageMapEdit __**_ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.**orgToolserver-l@lists.wikimedia.org ) https://lists.wikimedia.org/**mailman/listinfo/toolserver-lhttps://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/** view/Mailing_list_etiquettehttps://wiki.toolserver.org/view/Mailing_list_etiquette -- undefined ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Need Tool Labs migration support?
On Tue, Oct 8, 2013 at 7:47 PM, DaB. t...@dabpunkt.eu wrote: We do not force people to license their stuff as open source (and when I read something like we force them to move, then I’m glad we didn’t). Because then we could just make a copy of the tool in that nasty open-source way of Wikipedia, instead of having it die with the toolserver? ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Need Tool Labs migration support?
Maybe this just needs a little more planning. * Look at the server logs and find the most used tools on toolserver * Contact those tool authors directly * Make copies of the source of those tools publicly available (they should all be open source, but some directories are not accessibly for other users from shell IIRC) * Keep a Foundation-readable snapshot of the toolserver home directories, user-store, DB dumps etc. * Forcibly port the most important, unmaintained tools to Labs I have already started with one or two, though I'd appreciate it if I don't have to do it all by myself ;-) On Mon, Oct 7, 2013 at 8:31 PM, Maarten Dammers maar...@mdammers.nl wrote: Hi Silke, I'm afraid you're effort won't have the intended outcome. What I expect will happen: * Some tool authors will move in the next couple of months * Some tool authors will move at the last moment * A lot of tools willl just be abandoned. * You'll get a lot of upset people when you shut down the Toolserver Don't say I didn't warn you, Maarten Op 7-10-2013 14:03, Silke Meyer schreef: Hey all! We had set a deadline (Sept 30th) to ask WMDE for support to migrate your tools to Tool Labs. As very few of you made use of this until now, Johannes still has time to help you. So we are still accepting support requests. Please think about whether you'll have the time and the know-how to migrate on your own. Your tools shall have left the toolserver until June 30th, 2014. This deadline is serious. If you would like Johannes to give you a hand, please tell us until November 15th 2013. * Write to me or to directly to johannes.kr...@wikimedia.de. * Give some details about your tool: Did you try to migrate already but something fails to work? What works? What doesn't? What exactly do you want Johannes to do? Best, Silke __**_ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.**orgToolserver-l@lists.wikimedia.org ) https://lists.wikimedia.org/**mailman/listinfo/toolserver-lhttps://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/** view/Mailing_list_etiquettehttps://wiki.toolserver.org/view/Mailing_list_etiquette -- undefined ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] TS and/vs. Labs
On Tue, Aug 13, 2013 at 6:51 AM, Maarten Dammers maar...@mdammers.nlwrote: Hi DrTrigon, Op 10-8-2013 11:36, Dr. Trigon schreef: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hello DaB! Could you please give me a short update - the end of 2013 is approaching, what is the current state for TS resp. Labs? If you don't care about losing your (nfs) file system every once in a while and having very slow databases, you can probably start moving stuff to toollabs. I'm waiting for these bugs to be fixed. It's been more than 3 months now and no clear estimate when it will be fixed. I don't think that's a fair representation. Despite some teething problems, Labs has been at least (probably more) reliable than the toolserver in the time I've been there. I have ported 25 tools to Labs now, including some heavy-duty ones, and they all seem to work fine. Yes, there is the occasional bug (imagelinks table on cawiki is not working at the moment; roots are en route from Wikimania;-) but I don't see the traditional toolserver slowness, and Labs doesn't occasionally vanish behind 504 errors for hours at a time either. ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] TS and/vs. Labs
If you're asking is it safe to move tools to Labs, my experience so far say yes. I can't speak for all unique toolserver features, though. Cheers, Magnus On Sat, Aug 10, 2013 at 10:36 AM, Dr. Trigon dr.tri...@surfeu.ch wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hello DaB! Could you please give me a short update - the end of 2013 is approaching, what is the current state for TS resp. Labs? Thanks and Greetings DrTrigon -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.13 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlIGCboACgkQAXWvBxzBrDAKPwCgjsEcGfRtbKaw/eDwTxNhUFfo XgwAoLzV6J+TCdJZ3lvwvkqsMWihQmr+ =1dN+ -END PGP SIGNATURE- ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette -- undefined ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Status update on Tool Labs
Slightly OT: Sorry if I missed that in the discussion, but is there a for-dummies howto for creating a labs git for the ported tool? I know they use gerrit, but that's the extend of my knowledge. On Sun, Jun 2, 2013 at 9:22 PM, Merlijn van Deen valhall...@arctus.nlwrote: On 2 June 2013 21:12, Tim Landscheidt t...@tim-landscheidt.de wrote: Maciej Jaros e...@wp.pl wrote: Is there any how-to on moving SVN repo. along with it's history to TL? Is it possible/planned at all? As DaB. explained, the roots can provide you with a dump of your SVN repository. You can then import (load) this into a repository in your home directory and access it via ssh Note: you can also use svnsync to do this from your own computer - see https://github.com/pywikibot/svn2git/blob/master/scripts-pwb/sync for an example. Merlijn ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Status update on Tool Labs
Are they up somewhere else? Otherwise, redirects won't be useful. I tried to have a look at interiot's tools, but both web and shell access are blocked. There might be volunteers to port abandoned tools to labs (as I did for [1]), but only if they are still sought-after, and if the source code is accessible and properly licensed. [1] http://tools.wmflabs.org/joanjoc/ On Fri, May 31, 2013 at 9:19 AM, Federico Leva (Nemo) nemow...@gmail.comwrote: Would .htaccess work also for expired accounts then, and could operators/roots do something about them? We still have tons of users entering, say, interiot and soxred's tools and getting slightly unhelpful results. Nemo __**_ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.**orgToolserver-l@lists.wikimedia.org ) https://lists.wikimedia.org/**mailman/listinfo/toolserver-lhttps://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/** view/Mailing_list_etiquettehttps://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Status update on Tool Labs
.htaccess is your friend :-) On Tue, May 28, 2013 at 7:59 AM, Federico Leva (Nemo) nemow...@gmail.comwrote: A bot to update URLs is a nice thing (except that some stupid wikis will surely block it), but a true solution would be setting up a system where redirects from toolserver to new URLs can be set up without too much pain. Nemo __**_ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.**orgToolserver-l@lists.wikimedia.org ) https://lists.wikimedia.org/**mailman/listinfo/toolserver-lhttps://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/** view/Mailing_list_etiquettehttps://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Status update on Tool Labs
Database HOWTO: https://wikitech.wikimedia.org/wiki/User:Magnus_Manske/Migrating_from_toolserver#Database_access On Mon, May 27, 2013 at 4:27 PM, Silke Meyer silke.me...@wikimedia.dewrote: Hello to all tool maintainers! It was good to see several of you in person this weekend! Faces ftw! :) Here is a status update with what is now working in Tool Labs: First of all (looking at DaB.s survey and your comments): You _don't_ have to care about virtual instances, puppet or any of this. This is underneath/behind Tool Labs but you can just use the login server without having to know about it. What you get: * login server made to resemble the toolserver * database replicas from s1-s6, s7 is following in June. Access to all data you can access as a registered, unprivileged user. * possibility to create your own user databases * shared project storage, home directories (nfs) * possibility to add maintainers to your project via the web interface on http://tools.wmflabs.org/ * possibility to restore your files to the state they were in 1, 2, 3 hours ago, 1, 2, 3 days ago and on the last 2 Sundays. * infrastructure to run web services * job system: OpenGridengine * two different environments to develop tools in: Tools project which is a stable environment, Bots project which can be used for experiments with the environment, root access is possible in the latter. * bug tracking in WMF's Bugzilla (product: Wikimedia Labs, components: bots and tools for the two projects) * more stuff that is mentioned in the road map (link below). What we are still on: * documentation (let's crowdsource this!) * make Merlissimo's MerlLinkBot hunt down deprecated weblinks and point them to your tools' new URL. Looking for things you can help with? * Write down you experiences like some already did: Magnus Manske: https://wikitech.wikimedia.org/wiki/User:Magnus_Manske/Migrating_from_toolserver or Russell Blau: https://wikitech.wikimedia.org/wiki/User:Russell_Blau/Using_pywikibot_on_Labs * Help to improve the user interface of tools.wmflabs.org! Contact Coren/Marc-Andre Pelletier. Links: Tool Labs help: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help FAQ (ongoing collection of what you ask): http://www.mediawiki.org/wiki/Wikimedia_Labs/Migration_of_Toolserver_tools Road map: http://www.mediawiki.org/wiki/Tool_Labs/Roadmap_en (English) and http://www.mediawiki.org/wiki/Tool_Labs/Roadmap_de (German) What's next? Please come and try out the new database replication! If your tool is up and running in Tool Labs, _please consider documenting_ what you missed in the docs. By doing so you will help others a lot! Don't hesitate to ask! The best places to ask questions: * IRC (#wikimedia-labs), where Coren and petan (as Tool Labs admins) and many other experienced people hang around * the list labs-l (https://lists.wikimedia.org/mailman/listinfo/labs-l) Do your tools need more adaptations than you think you can do? Wouldn't mind a helping hand? Ask me and I'll try to find you a helper! Best, Silke -- Silke Meyer Internes IT-Management und Projektmanagement Toolserver Wikimedia Deutschland e.V. | Obentrautstr. 72 | 10963 Berlin Tel. (030) 219 158 260 http://wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985. ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
[Toolserver-l] Migrating to labs from the toolserver
As I boldly ssh in where few toolserver users have ssh'ed in yet, I try to keep a log of what I'm doing, what issues arise, and where I had to stop and scratch my head. Behold: https://wikitech.wikimedia.org/wiki/User:Magnus_Manske/Migrating_from_toolserver Feel free to improve the page as you see fit. ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Migrating to labs from the toolserver
.htaccess redirects? On Sat, May 4, 2013 at 2:10 PM, rupert THURNER rupert.thur...@gmail.comwrote: What happens to the zillions of published toolserver links when a tool switches to labs? Rupert Am 04.05.2013 14:18 schrieb Magnus Manske magnusman...@googlemail.com: As I boldly ssh in where few toolserver users have ssh'ed in yet, I try to keep a log of what I'm doing, what issues arise, and where I had to stop and scratch my head. Behold: https://wikitech.wikimedia.org/wiki/User:Magnus_Manske/Migrating_from_toolserver Feel free to improve the page as you see fit. ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Wikidata tables
FWIW, I have implemented a query-able stand-alone web server that keeps all of the wikidata property-item-links in memory. This uses the wikidata dumps which appear to be rather frequent. I'll try do deploy a test version on wikilabs (once I figure out how all that works); it seems to be more favourable to such services than the toolserver. On Fri, Apr 19, 2013 at 9:29 AM, Platonides platoni...@gmail.com wrote: On 19/04/13 01:19, DaB. wrote: as you may know there is a rev_text_id-field in the revision-table. This field points to the text-table where the actual text is – or should be. Because the WMF doesn’t store the text here, but only a pointer (DB://cluster25/11458305 for example). If you query different wikis you will see that most of them point to the same cluster or one with a number short by. That says me (and I was also told so before) that all text of all wmf-projects are stored together. The task would now to separate wikidata from the rest – but the storage-area has no clue from where a text is which makes the separating very hard. And there is another problem: Deleted texts are also in this area, so even more filtering would be needed. I very doubt that this situation will change at the TS and I also doubt that it will be different for WikiLabs. So I guess your best bet is the API here. Sincerely, DaB. I think the only hope would be if wikidata was stored under its own cluster (for easier differenciation) and at least one server of that group (the master?) only had that (so toolserver could get its binlogs). ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
[Toolserver-l] Wikidata tables
Just wondering what the status of exposing all wikidata tables on the toolserver is. Currently, there are a few wb_* tables with item labels, descriptions, aliases, and language links. But the tables (whatever they are called) containing item-to-item connections appear to be missing. Maybe because they were added later? Magnus ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Wikidata tables
Huh. That could be ... problematic in the future. Thanks, Magnus On Thu, Apr 18, 2013 at 10:21 AM, Lydia Pintscher lydia.pintsc...@wikimedia.de wrote: On Thu, Apr 18, 2013 at 9:52 AM, Magnus Manske magnusman...@googlemail.com wrote: Just wondering what the status of exposing all wikidata tables on the toolserver is. Currently, there are a few wb_* tables with item labels, descriptions, aliases, and language links. But the tables (whatever they are called) containing item-to-item connections appear to be missing. Maybe because they were added later? As far as I know they're only saved in JSON where usually the article text is stored and not in separate tables. Cheers Lydia -- Lydia Pintscher - http://about.me/lydia.pintscher Community Communications for Wikidata Wikimedia Deutschland e.V. Obentrautstr. 72 10963 Berlin www.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985. ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
[Toolserver-l] Toolserver down?
Can't get http connection, and ssh says: ssh: Could not resolve hostname login.toolserver.org: nodename nor servname provided, or not known ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
[Toolserver-l] Can't log into toolserver
Keeps asking for password, but doesn't accept it. login.toolserver.org, also ortelius, probably more. ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Tool request
Why would you want to upload downsized images if you can just have MediaWiki generate downsized thumbnails? On Fri, Mar 22, 2013 at 7:05 PM, John phoenixoverr...@gmail.com wrote: This is slightly outside the scope of this mailing list, But could someone write a web tool that someone can pass a filename, lang, project and the script then gets the associated image, lets the user modify the size (and has a few pre-configured resolutions including 0.1MP ) then download it so it can be re-uploaded (or uploaded via upload by url if the wiki has it enabled.) I know this would help a lot of users. ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
[Toolserver-l] FYI: TS wikivoyage is missing a few pages
https://jira.toolserver.org/browse/TS-1598 ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Result of the general member meeting of WMDE
Doubleplusgood for DaB staying in the fight for another round!!! On Mon, Dec 3, 2012 at 12:49 AM, DaB. w...@daniel.baur4.info wrote: Hello all, Am Sonntag 25 November 2012, 23:44:05 schrieb DaB.: I have not decided yet if I will remain as root under this circumstances for 2013 – I will tell you my decision until next Sunday. I promised you a decision until Sunday and while it is not Sunday anymore where I live, for many of you still is. First of all I would like to thank all people who have sent me emails: Thanks. During the last week I thought about my decision several times and to be honest I switched from staying to leave and back several times. The toolserver consumes a big part of my free time and so the idea of leaving was alluring. It would also de-stress me, because I would not longer need to argue with people from WMDE and WMF. Maybe even the personal relationship with some people of WMDE would normalize again. On the other hand the toolserver is a important project into which I invested much time already. Many people would dislike me if I go down without a fight too. The toolserver is also important for WMDE and its relationship to WMF – even if most people do not see that at the moment. Another path of thought was optimism versus pessimism. The pessimist in me says, that it is very sure that WMDE will find a cheap way out in 6 months when it becomes clear that WikiLabs can not replace the toolserver – but I have no proof for that. The optimist in me says that that is most likely but it COULD be that WMDE is fair and the 6 months of waiting are worth that. The third point is the question of motivation. I really like to help people – that's the reason I started editing and sysoping in Wikipedia 9 years ago. The idea that the toolserver helps people to do their work is the basic motivation for me to administrate the toolserver. But when you know that all you do is for the birds because some other people decided that the toolserver is not welcome anymore that's not very motivational. Than again helping people for 6-12 months is better than nothing. The forth, least significant and last point is the question of pride. Leaving would be a form of loosing; WMDE and WMF would win. I don't like loosing and I prefer to fight till the end for normal. To summarize (people who scrolls down to find the result should stop here): I will stay for another 12 months and see where it goes. The toolserver and you all are worth my time, the toolserver is still helpful, and there is always the possibility that WMDE will do a fair decision in summer. The way will be rocky and I'm sure that we will see some drawbacks. Good night everybody. Sincerely, DaB. -- Userpage: [[:w:de:User:DaB.]] — PGP: 2B255885 ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
[Toolserver-l] Anyone else having TS trouble?
Started a few minutes ago. getting alternating errors on reloading a page; 500 (on a static HTML page, not CGI, mind you!), This webpage is not available: The connection to toolserver.org was interrupted., 404. ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Anyone else having TS trouble?
Works for me now. On Sun, Nov 25, 2012 at 6:19 PM, Krinkle krinklem...@gmail.com wrote: On Nov 25, 2012, at 5:06 PM, Platonides platoni...@gmail.com wrote: On 25/11/12 16:42, Magnus Manske wrote: Started a few minutes ago. getting alternating errors on reloading a page; 500 (on a static HTML page, not CGI, mind you!), This webpage is not available: The connection to toolserver.org http://toolserver.org was interrupted., 404. The same thing happpened here. The servers didn't allow logins, and accessing the web pages return 404s. Apparently ldap was down. It recovered now. I don't know if it there was a human involved, if it solved itself or if it was turnera taking over damiana automatically. Seems fine here: https://toolserver.org/~krinkle/mwSnapshots/#!/mediawiki-core/master Maybe HTTP or Geo related? Is anyone still having this problem? -- Krinkle ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] 502
Thanks all. Must be my line then. On Mon, Nov 12, 2012 at 3:38 PM, Mardetanha mardetanha.w...@gmail.comwrote: no problem from Iran and through vpn connection from UK Mardetanha On Mon, Nov 12, 2012 at 7:03 PM, Simon Walker stwalkers...@googlemail.com wrote: No problems from Edinburgh (Scotland) Simon On Nov 12, 2012 3:26 PM, Magnus Manske magnusman...@googlemail.com wrote: I suddenly see a lot of 502 error on the toolserver from the UK. Anyone else? (can't find a IRC client for OSX 10.6 quickly...) ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Welcome: wikidata
Fantastic! Thank you! On Tue, Oct 30, 2012 at 3:00 PM, DaB. w...@daniel.baur4.info wrote: Hello all, wikidata has made it on the toolserver today. The database is called wikidatawiki_p (don't ask me…) and its is home on s3. Because I guessed that many joins to it will happen in future, I created also copies on all clusters with the exception of s6 and s7 (will happen later); so you can join it in the same way as commons. Sincerely, DaB. -- Userpage: [[:w:de:User:DaB.]] — PGP: 2B255885 ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Web services are down
Me too. Occasionally, it's 502 bad gateway though. Just in the middle of testing stuff and thought it was my broken code. Frustration. On Sun, Sep 2, 2012 at 4:33 PM, emijrp emi...@gmail.com wrote: I'm getting Internal Server Error (500) and pages dont load. 2012/8/29 Platonides platoni...@gmail.com On 29/08/12 09:22, Marlen Caemmerer wrote: Hello, wolfsbane ran out of space which means one of the web servers was not able to deliver any more web pages which has the effect of sometimes the page works - sometimes not. Should be fixed now - if you still see errors please let us know. Cheers nosy Thanks nosy. Shouldn't the balancer have automatically considered wolfsbane down and direct all requests to ortelius? Cheers ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette -- Emilio J. Rodríguez-Posada. E-mail: emijrp AT gmail DOT com Pre-doctoral student at the University of Cádiz (Spain) Projects: AVBOT | StatMediaWiki | WikiEvidens | WikiPapers | WikiTeam Personal website: https://sites.google.com/site/emijrp/ ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
[Toolserver-l] What's up with enwp10?
Hi all, the Wikipedia 1.0 tool gives a 403 forbidden error: http://toolserver.org/~enwp10/bin/list2.fcgi?run=yesprojecta=Computational_Biologynamespace=pagename=quality=importance=score=limit=1500offset=1sorta=Importancesortb=Quality+%28reverse%29 Does anyone know what's going on, or how to fix this? Cheers, Magnus ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] What's up with enwp10?
Thanks! On Wed, Aug 29, 2012 at 4:02 PM, DaB. w...@daniel.baur4.info wrote: Hello, At Wednesday 29 August 2012 17:01:58 DaB. wrote: Might read through the [Toolserver-l] Web services are down thread. no, read the we-disabled-our-own-tool-thread at [1] ;-). Sincerely, DaB. [1] https://en.wikipedia.org/wiki/Wikipedia_talk:Version_1.0_Editorial_Team/Index#WP_1.0_bot_temporarily_down_2012-8-7 -- Userpage: [[:w:de:User:DaB.]] — PGP: 2B255885 ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Wpipe
You can access the asset metadata in the u_magnus_wpipe_p database,
table asset, if you wish. The files are stored in
/mnt/user-store/wpipe/
Cheers,
Magnus
On Sun, Jul 22, 2012 at 10:51 AM, Dr. Trigon dr.tri...@surfeu.ch wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
To me as someone always struggling with sql queries (since I am not an
expert at all) this sound somehow promising. May be there will be a
comprehensive set of queries going to pipes one day. Then I would simply
have to pick up the pipe and connect it further. (I like this flavour
of UNIX philosophy ;)
Greetings
DrTrigon
On 19.07.2012 22:34, Magnus Manske wrote:
On Thu, Jul 19, 2012 at 8:06 PM, Platonides platoni...@gmail.com
wrote:
I'm not convinced about its utility. What tools would need
combining? If I just need the results of a SQL query, it may be
easier for me than using this system. Maybe a better interface
would help.
The use case I see more interesting is for taking a tool which
outputs a list of pages and provide for input of another tool.
Some page/user to work with seems to be the most common input.
Maybe we should just standarize the input parameters and let some
tools chain to another, simply using a special format parameter.
For instance I usually use names like: art, lang, project such
as: $_REQUEST += array('art'='', 'lang'='en',
'project'='wikipedia' );
I believe we mean the same thing; maybe I didn't describe the
asset thing very well.
It's not for a single page run of some tool; one reason I chose
my CatScan rewrite as demo ist that it can run for a long time
(two-digit number of minutes), and generate a vast list of results
(tens of thousands of pages), depending on the query. The idea is
that (a) you're not blocking while waiting for that to finish,
before you can do something else; (b) you can access the results of
the run again, maybe if the subsequent tool fails, or you want to
try a different filter or subset, or a different subsequent tool
altogether; (c) you can define new data sources, maybe a tool where
you just paste in page titles, or another tool that gets the newest
1.000 articles, or 1.000 random articles, or the last 1.000
articles you edited, or /insert crazy idea here/, and all
subsequent tools will just run with it.
And you can chain tools together via a single number; no file path
that the other guy doesn't have access to, no sql query that runs
for a few minutes every time (that is, /if/ your tool can be
reduced to that...), no massive paste orgy, no loss of meta-data
between tools.
I also envision longer chains: Give me all articles that are in
both these two category trees; remove the ones that have images
(except template symbol icons, if possible); remove the ones that
have language links; remove the ones that had an edit less than a
month old; render that as wikitext. There's a subject-specific
needs work finder from simple components. UNIX philosophy at its
finest :-)
Magnus Manske wrote:
Right now, a tool is started by this page via nohup ; that
could change to the job submission system, if that's possible
from the web servers, but right now it seems overly complicated
(runtime estimation? memory estimation? sql server access?
whatnot) The web page then returns the reserved output asset
ID, while the actual tool is running; another tool could thus
be watching asynchronously, by pulling the status every few
seconds.
Yes, it can be called. I use it in a script for scheduling a
cleanup of the created temporary files.
The relevant code:
$dt = new DateTime( now, new DateTimeZone( UTC ) ); $tmpdir
= dirname( __FILE__ ) . /tmp; @mkdir( $tmpdir, 0711 ); $shell
= mktemp -d --tmpdir= . escapeshellarg($tmpdir) .
catdown.;
$tmpdir2 = trim( `$shell` ); // Program the folder destruction
// Note that qsub is 'slow' to return, so we perform it in the
background $dt-add( new DateInterval( PT1H ) ); exec(
SGE_ROOT=/sge/GE qsub -a . $dt-format(YmdHi.s) . -wd
. escapeshellarg( $tmpdir ) . -j y -b y /bin/rm -r .
escapeshellarg( $tmpdir2 ) . 21 );
Thanks, that looks interesting. I'll play with it, thou I still
face the problem of estimating resource requirements for a tool by
a generic wrapper. /Shudder/
Cheers, Magnus
___ Toolserver-l
mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting
guidelines for this list:
https://wiki.toolserver.org/view/Mailing_list_etiquette
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAlALzSAACgkQAXWvBxzBrDD+TgCfWRF59s6oaGaRANJW+NscTix3
Jl8AoOIoaqBPwV/NWw4TeIZhqvj14/Qx
=t1Fk
-END PGP SIGNATURE-
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo
[Toolserver-l] Wpipe
Occasionally, toolserver people (both programmers and users) talk about joining up tools. Wouldn't it be great if we could use one or several toolserver tools, and mash-up their output to create something new and useful? And wouldn't it be even better if the users could do this directly, across tools, without programmers hard-connecting tools? Some tools already support machine-readable output, and some tools already use others to perform a specific function. But these are hardcoded, out formats are often crude (tabbed text, not that there's anything wrong with that in principle), runtimes add up, and so on. So, I went ahead and, as a first step towards a pipeline setup called wpipe (w for wiki, as you no doubt have guessed), implemented an asset tracker. Here, an asset is a dataset, ideally a JSON file (I also created a simple structure to hold lists of wiki pages, along with arbitrary metadata). Each asset is tracked in a database, accessible via a unique numeric identifier, and its data is stored in a file. Assets can be created and queried via toolserver command line, as well as via a web interface. The usual steps in asset creation involve: 1. reserve (gets a new, unique ID) 2. start (set a flag that the asset data creation has begun) 3. done (store data, either by creating a file directly, or passing the data to be stored) 4. fail (if there was an error during data creation) Some points: * All assets and associated data are public. * Creation and last-access time (write or read the actual data) are tracked, so unused assets can be removed to conserve storage. * Currently, data creation is limited to toolserver IP adresses (yes, I know, it can be gamed. Like the rest of the wiki world.) * The suggested JSON format should be flexible enough for most tools dealing with lists of wiki pages, but any text-based format will work for specialist uses. * The asset system can be used by command-line tools and web tools alike. * Existing tools should be simple to adapt; if a tool takes a list of page names, and language/project information, then using asset IDs as an alternative source should be straightforward. * A pipeline of tools could be started asynchronously, and their progress could be tracked via JavaScript; once a tool has finished, the next one in the chain could be run, all from the user's browser. The main web API, and documentation page, is here: http://toolserver.org/~magnus/wpipe/ That page also links to a generic asset browser : http://toolserver.org/~magnus/wpipe/asset_info.php As an appetizer, and feasibility demo, I adapted my own CatScan rewrite to use assets as an optional output. This can be done by copying the normal CatScan rewrite URL parameters and pasting them here: http://toolserver.org/~magnus/wpipe/toolwrap.php This is intended as a generic starter page for command-line-enabled tools. At the moment, only CatScan rewrite is available, but I plan to add others. If you have tools you would like to add, I'll be happy to help you set that up. Right now, a tool is started by this page via nohup ; that could change to the job submission system, if that's possible from the web servers, but right now it seems overly complicated (runtime estimation? memory estimation? sql server access? whatnot) The web page then returns the reserved output asset ID, while the actual tool is running; another tool could thus be watching asynchronously, by pulling the status every few seconds. Of course, this whole shebang doesn't make sense unless others are willing to join in, with work on this core, or at least by enabling some of their tools; so please, if you are even slightly interested in a generic data exchange mechanism between tools, potentially leading to a pipeline-able ecosystem, by all means step forward! Cheers, Magnus ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Wpipe
On Thu, Jul 19, 2012 at 4:44 PM, Petr Onderka gsv...@gmail.com wrote: * Currently, data creation is limited to toolserver IP adresses (yes, I know, it can be gamed. Like the rest of the wiki world.) This doesn't seem to be working. I was able to create asset 12 directly from my home computer. I'll look into that. Not a big issue, I just don't want to make it too easy for the script kiddies to store warez here... Also, action=fail doesn't work: Error : Unknown action fail It seems it's not listed in the if in index.php starting at line 23. Thanks, should be fixed now. Cheers, Magnus ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Wpipe
On Thu, Jul 19, 2012 at 8:06 PM, Platonides platoni...@gmail.com wrote:
I'm not convinced about its utility. What tools would need combining?
If I just need the results of a SQL query, it may be easier for me than
using this system. Maybe a better interface would help.
The use case I see more interesting is for taking a tool which outputs a
list of pages and provide for input of another tool. Some page/user to
work with seems to be the most common input.
Maybe we should just standarize the input parameters and let some tools
chain to another, simply using a special format parameter.
For instance I usually use names like: art, lang, project such as:
$_REQUEST += array('art'='', 'lang'='en', 'project'='wikipedia' );
I believe we mean the same thing; maybe I didn't describe the asset
thing very well.
It's not for a single page run of some tool; one reason I chose my
CatScan rewrite as demo ist that it can run for a long time (two-digit
number of minutes), and generate a vast list of results (tens of
thousands of pages), depending on the query. The idea is that (a)
you're not blocking while waiting for that to finish, before you can
do something else; (b) you can access the results of the run again,
maybe if the subsequent tool fails, or you want to try a different
filter or subset, or a different subsequent tool altogether; (c) you
can define new data sources, maybe a tool where you just paste in page
titles, or another tool that gets the newest 1.000 articles, or 1.000
random articles, or the last 1.000 articles you edited, or /insert
crazy idea here/, and all subsequent tools will just run with it.
And you can chain tools together via a single number; no file path
that the other guy doesn't have access to, no sql query that runs for
a few minutes every time (that is, /if/ your tool can be reduced to
that...), no massive paste orgy, no loss of meta-data between tools.
I also envision longer chains: Give me all articles that are in both
these two category trees; remove the ones that have images (except
template symbol icons, if possible); remove the ones that have
language links; remove the ones that had an edit less than a month
old; render that as wikitext. There's a subject-specific needs work
finder from simple components. UNIX philosophy at its finest :-)
Magnus Manske wrote:
Right now, a tool is started by this page via nohup ; that could
change to the job submission system, if that's possible from the web
servers, but right now it seems overly complicated (runtime
estimation? memory estimation? sql server access? whatnot)
The web page then returns the reserved output asset ID, while the
actual tool is running; another tool could thus be watching
asynchronously, by pulling the status every few seconds.
Yes, it can be called. I use it in a script for scheduling a cleanup of
the created temporary files.
The relevant code:
$dt = new DateTime( now, new DateTimeZone( UTC ) );
$tmpdir = dirname( __FILE__ ) . /tmp;
@mkdir( $tmpdir, 0711 );
$shell = mktemp -d --tmpdir= . escapeshellarg($tmpdir) .
catdown.;
$tmpdir2 = trim( `$shell` );
// Program the folder destruction
// Note that qsub is 'slow' to return, so we perform it in the background
$dt-add( new DateInterval( PT1H ) );
exec( SGE_ROOT=/sge/GE qsub -a . $dt-format(YmdHi.s) . -wd .
escapeshellarg( $tmpdir ) . -j y -b y /bin/rm -r . escapeshellarg(
$tmpdir2 ) . 21 );
Thanks, that looks interesting. I'll play with it, thou I still face
the problem of estimating resource requirements for a tool by a
generic wrapper. /Shudder/
Cheers,
Magnus
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list:
https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Toolserver user phe account expired
On Sat, May 26, 2012 at 8:47 AM, Danny B. wikipedia.dann...@email.cz wrote: Perhaps we may be able to make it a cross-tool, which can either be embedded in MediaWiki or standalone (I'm not aware of any extension doing that, but it's something I have considered several times myself). CategoryTree was/is sort of that... So was GeoHack. ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Toolserver user phe account expired
On Tue, May 22, 2012 at 10:22 AM, Merlijn van Deen valhall...@arctus.nl wrote: On 21 May 2012 23:52, Lars Aronsson l...@aronsson.se wrote: This madness has to stop. Wikisource depends on this service. I understand the German toolserver is beyond repair, but phe's statistics scripts should be moved to proper WMF servers. This madness has to stop. Wikis should stop depending on services that are not hosted in multi-maintainer projects! I don't think that's realistic. There is barely enough of us (volunteer programmers) as it is; co-maintaining someone else's code would add a significant workload. I have repeatedly offered all my tools for MMPs - so far, only CommonsHelper2 and geohack are MMPs. Magnus ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Toolserver user phe account expired
On Tue, May 22, 2012 at 1:30 PM, Platonides platoni...@gmail.com wrote: On 22/05/12 11:22, Merlijn van Deen wrote: How do we make sure tools do not disappear? By making multi-maintainer projects for them. However, we see that this doesn't happen enough - see also the thread about the expiration of soxred93's account. Options for improvement: - better communication with wikis - which tools are used a lot and *thus* should be moved to mmp's? - easier creation of mmp's? I can imaging people don't move their tools because it takes time to organise everything. It's relatively hard to create a MMP. Compare that with the complexity of doing a mkdir for creating a project in your account. Add to that the relatively low interest of other people for maintaining external projects (as shown by Magnus mail). There's little reason to create a MMP in advance. Plus, each of is coding using different languages, conventions and frameworks (helper functions). Maybe we should use a model where stable tools are available in a repository where all users can commit. The code can only be updated through that. As an alternative, each project could be in either open-gate or closed-gate model. In the first one, anyone can commit there. In the second one, there's just a subset of users which can directly commit (commits by others must be approved by a project member). If the accounts for all the project members expire (it gets orphan), the tool automatically changes to open-gate mode. ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette An other idea, albeit one requiring more planning, coding, and cooperation (and we are notoriously bad at two of these) would be to separate front-end and back-end. If we could set up a MMP that presents an API for the queries and data storage of several tools, anyone (at the very least, anyone on the toolserver) could write a front-end; also, taking over a front-end from someone else might be more maintenance-friendly. We could even offer consistent stylesheets for toolserver tools (hey, one can dream?) Cheers, Magnus ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Toolserver user phe account expired
On Tue, May 22, 2012 at 6:16 PM, DaB. w...@daniel.baur4.info wrote: In my eyes the toolserver developed behind this original idea and is now a good place to host stuff even when it is done Just like Wikipedia was to Nupedia. Symmetry! :-) ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] External authentication?
On Thu, Feb 9, 2012 at 1:08 PM, Victor Vasiliev vasi...@gmail.com wrote: On Thu, Feb 9, 2012 at 3:37 PM, Magnus Manske magnusman...@googlemail.com wrote: Hi all, on my talk page, [[User:Pathoschild]] raised the idea of allowing OpenID authentication to operate toolserver tools that currently rely on TUSC. While I'd rather go for browserID [1] (not mutually exclusive), it raised the point of which authentication is good enough for using some toolserver tools, especially those that edit or upload on Wiki(m|p)edia projects. Would these non-TUSC accounts need to be linked to Wiki(m|p)edia user names? If so, how would this be done? If Wiki(m|p)edia were to provide openID/browserID authentication, it would be a non-issue, but as it stands, this would need to be done on the toolserver in some form, which would most likely be more cumbersome than the current TUSC account creation. Ideas? Magnus Well, Wikimedia should become OpenID/OAuth provider for that. So, no change before 2020 then? ;-) ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] DNS-problems
Page loads fine in the UK, but I am quite familiar with the 502 error, which, on certain days, shows up on every second request or so. Might be coincidence, though. Magnus On Wed, Nov 30, 2011 at 6:02 PM, AleXXw alexx...@gmail.com wrote: Hi! Right now: 502 Bad Gateway, when I'm in I get a SQL-Error 28 (http://toolserver.org/~alexxw/Denkmalliste/index.php). Location: Austria, Provider: Kabelsignal Same Problem? Greets Alex On Wed, Nov 30, 2011 at 18:39, DaB. w...@daniel.baur4.info wrote: Hello all, a part of the german community (all member of the same big ISP as far as I see) reported to me problems with the DNS of the toolserver. I doubt that the problems are on our end, but just to be sure: Have had anyone problems to reach the toolserver in the time since 20. Nov. until now, that are related to DNS (and is not a customer by german vodafon or arcor). I would be especialy interrested in non-germany-users. Sincerly, DaB. -- Userpage: [[:w:de:User:DaB.]] — PGP: 2B255885 ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] CrontabCr
On Fri, Nov 25, 2011 at 9:45 AM, John phoenixoverr...@gmail.com wrote: Mine is betacommand And one more (magnus): Unable to run job: got no response from JSV script /sge62/default/common/jsv.sh. Exiting. ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Queues
Thanks! On Wed, Nov 16, 2011 at 9:13 AM, Marlen Caemmerer marlen.caemme...@wikimedia.de wrote: Hi Magnus, I guess you query these DBs: 'dewiki_p' , 'enwiki_p' , 'frwiki_p' , 'plwiki_p' dewiki and plwiki have not always been available until Monday so it is possible that your script failed for this reason. Everything should be back since yesterday so if there is still a problem please let me know. Cheers nosy On Tue, 15 Nov 2011, Magnus Manske wrote: Date: Tue, 15 Nov 2011 19:29:38 From: Magnus Manske magnusman...@googlemail.com Reply-To: toolserver-l@lists.wikimedia.org To: toolserver-l@lists.wikimedia.org Subject: Re: [Toolserver-l] Queues Hi nosy, first one (which I had missed) is from Nov 12, 5:15pm (UTC, I guess). Seems to be a daily job. Mail title is: Cron magnus@nightshade cd /home/magnus/cronjobs ; ./all_most_wanted.pl Cheers, Magnus On Tue, Nov 15, 2011 at 5:15 PM, Marlen Caemmerer marlen.caemme...@wikimedia.de wrote: Hey Magnus, since when are they coming? Are they related to the databases s2 or s5? Cheers nosy On Tue, 15 Nov 2011, Magnus Manske wrote: Date: Tue, 15 Nov 2011 18:09:59 From: Magnus Manske magnusman...@googlemail.com Reply-To: toolserver-l@lists.wikimedia.org To: toolserver-l@lists.wikimedia.org Subject: [Toolserver-l] Queues I am now getting mails like this: Unable to run job: error: no suitable queues. Exiting. Haven't touched my cron settings for months... ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Queues
On Wed, Nov 16, 2011 at 7:09 PM, Marlen Caemmerer marlen.caemme...@wikimedia.de wrote: Hi, Which script does generate which error? I got three messages on three days (it seems to have stopped now), all like this: Cron magnus@nightshade cd /home/magnus/cronjobs ; ./all_most_wanted.pl to magnus Unable to run job: error: no suitable queues. Exiting. Unable to run job: error: no suitable queues. Exiting. It thus appears that /home/magnus/cronjobs/all_most_wanted.pl did not run on these days. Cheers, Magnus ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
[Toolserver-l] Queues
I am now getting mails like this: Unable to run job: error: no suitable queues. Exiting. Haven't touched my cron settings for months... ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Queues
Hi nosy, first one (which I had missed) is from Nov 12, 5:15pm (UTC, I guess). Seems to be a daily job. Mail title is: Cron magnus@nightshade cd /home/magnus/cronjobs ; ./all_most_wanted.pl Cheers, Magnus On Tue, Nov 15, 2011 at 5:15 PM, Marlen Caemmerer marlen.caemme...@wikimedia.de wrote: Hey Magnus, since when are they coming? Are they related to the databases s2 or s5? Cheers nosy On Tue, 15 Nov 2011, Magnus Manske wrote: Date: Tue, 15 Nov 2011 18:09:59 From: Magnus Manske magnusman...@googlemail.com Reply-To: toolserver-l@lists.wikimedia.org To: toolserver-l@lists.wikimedia.org Subject: [Toolserver-l] Queues I am now getting mails like this: Unable to run job: error: no suitable queues. Exiting. Haven't touched my cron settings for months... ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] performance issues
On Wed, Oct 26, 2011 at 10:18 PM, Platonides platoni...@gmail.com wrote: Marlen Caemmerer wrote: Rosemary (one of the enwiki-DB-Hosts) seems to bring the maximum of the I/O that is possible, disk graphs are clipping there. In the MySQL traffic graph you can see there is clipping too. Strange thing about this is that this phenomenon started in the middle of september. Can anyone remember any important change in this time? That is not the case with thyme (other enwiki DB Host) so it'd be a question wether we could share the load better. Also I enabled query caching and will see if this is useful in any way. Maybe there's some job/tool which started at that time which loaded rosemary so much? In the last three hours, I got no less than 21 immensely useless mails from the query killer. The following query, apparently, was killed after running for a whooping 71 minutes: SELECT /* SLOW_OK */ /* GLAMOROUS */ gil_wiki,gil_page_title,gil_page_namespace,gil_to from globalimagelinks,page,categorylinks where gil_to=page_title and cl_to =Images_from_the_National_Archives_and_Records_Administration AND page_id=cl_from AND page_namespace=6 AND gil_page_namespace= This queries, and others like it, have not caused problems for the last one (two? I can't remember) years they've been in place. Something has changed for the far, far worse. Magnus ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
[Toolserver-l] Toolserver down
The server at www.toolserver.org is taking too long to respond. Also, no ssh (I tried nightshade). Magnus ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Cronjob failing - help?
2011/9/22 René Kijewski kijew...@inf.fu-berlin.de: Am Wed, 21 Sep 2011 20:48:58 -0400 schrieb Hersfold hersfoldw...@gmail.com: I've gotten this failure message the last two time my bot has tried to run. Does any one know what might be causing this? I don't recognize the script it mentions. Unable to run job: got no response from JSV script /sge62/default/common/jsv.sh. Exiting. Hi, same here. My scripts worked well for some month now, so I guess it's a new problem at server side. aolme too/aol Magnus ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] A new tool request - find a translator
There you go: http://toolserver.org/~magnus/transparent.php Magnus On Mon, Sep 12, 2011 at 8:18 PM, Ryan Kaldari kald...@gmail.com wrote: Could someone write a tool that allows me to search for the most recently active users on a certain project who speak a certain language (or combination of languages) natively? The user language categories seem to exist on most projects, so it seem like you could use those: http://en.wikipedia.org/wiki/Category:User_ja-N Ryan Kaldari ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Better toolserver status?
On Wed, Aug 17, 2011 at 3:33 PM, Casey Brown li...@caseybrown.org wrote: On Wed, Aug 17, 2011 at 10:20 AM, Daniel Schwen li...@schwen.de wrote: Sure, I was a bit sloppy with the terminology, but the message still stands. Even in an emergency, please think about _priorities_. Five minutes taken to write a notification I think this needs a little bit of clarifying. A notification should be written if the downtime is going to be more extended or if you stop working on it for some amount of time (you can't do it, you go to bed, or something like that). If you're busy working on fixing it, that's your first priority and I don't think anyone wants you to stop prioritizing that. :-) 100% agree. Wikimedia ops are the same way -- it's the sysadmins first priority to fix up the techie stuff, that's their job. If someone's not doing anything at the moment and has some free time, *then* they should make communicating about the issue a priority. In the meantime, anyone watching what's going on in #wikimedia-toolserver can share what they're seeing with confused endusers. Idea: How about some one-line message detailing current issues (if any) with the toolserver that I could transclude from a single location on top of my tools? That would alert all my users that there is a problem with the toolserver (and not my tool specifically), and that it's being worked on. Magnus ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Query-Killer is back in action
On Mon, Aug 15, 2011 at 11:39 PM, K. Peachey p858sn...@gmail.com wrote: On Tue, Aug 16, 2011 at 7:03 AM, Maarten Dammers maar...@mdammers.nl wrote: Ah, you are the one who killed the categorization bot. Thanks for announcing that beforehand (not!). It's very annoying that you suddenly just decide to deploy a new toy the screws up our tools. Maarten It was re-enabling a service that has been around for a long time that [accidently] broke at one stage and because of the TS performance not being degraded was never really noticed or cared about. Recently a TS box went down and caused major issues (Replag was easily over 2 days at one stage for those databases) for the backup box handling the effected databases, DaB was monitoring this by hand and manually killing the processes to keep this somewhat saner then re-wrote and brought the tool back online to handle this when they noticed it was [the tool] was not functioning. This was a tool that was already meant to be running but by [your] luck wasn't, and was already documented on wiki as DaB pointed out over a year ago. Perhaps you should consider apologizing to DaB in regards to the tone of your email. While it's wrong do insult DaB (or anyone here :-) I feel the annoyance. It doesn't really matter if it was supposed to be running but wasn't for some reason we mere mortals can't comprehend - queries were fine, and then they suddenly they were not. I understand there are reasons behind this (hardware failure, miser mode), but they weren't obvious from the original mail. So now I (and others, I suspect) are inserting /* SLOW_OK */ into queries. That doesn't make the queries less or faster, but it returns service to normal (for the tool writers, and the tool users). German speakers might be familiar with the phrase wenn's denn der Wahrheitsfindung dient... Should we (the tool authors) make our queries faster? Absolutely. Can we? Sometimes, maybe. Usually not. That might be due to personal qualification, or system constraints. The local version of the work smarter, not harder mantra is unlikely to improve the situation. Cheers, Magnus ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Wikimedia projects pageviews stats
On Thu, Aug 11, 2011 at 11:35 PM, RYU Cheol rch...@gmail.com wrote: Hi, folks! Pagecounts is available on toolserver database? I found Magnus have it for en and de in u_magnus_popular_p to serve http://toolserver.org/~magnus/toptopics.php. These are just a tiny subset. The popular articles, as chance has it. If not, let me ask a question. Is it feasible to create one if we use the primary key with page_id rather than page_title? Putting all that data into MySQL might strain the poor surviving DB servers significantly. Cheers, Magnus ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Query-Killer is back in action
On Sun, Aug 14, 2011 at 5:42 PM, DaB. w...@daniel.baur4.info wrote: Hello all, during the last week I kept an eye on the database-servers and killed queries which had run for too long. In the past we had a programm for that, but somewhen it stoped working and nobody fixed it AFAIK. Because watching the mytop is a boring thing and I have to sleep too, I used the last 2-3 days to write a substitute of the old query-kiler. This program is working now. It does more or less the same as I did, but it send you an eMail when it killed one of your query. I think that is an improvment and I can spend my time with other things. You can find the parameters of when and if the killer works at [1]. Of corse there is always the possiblity of adjustments. Feel free to open bugs at JIRA or discuss on teh mailinglist. So the max run-time is dependent on replag. But, considering this: Replag s3 6050h 10m 44s I currently get bombarded with killed-query-mails :-( Magnus ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Query-Killer is back in action
On Sun, Aug 14, 2011 at 8:12 PM, DaB. w...@daniel.baur4.info wrote: At Sunday 14 August 2011 21:10:26 DaB. wrote: Does anybody have a good suggestion what wiki to use for s3? dawiki is the largest wikipedia on it, but I think it will have very little edits during the night in Europe. I use dawiki and the hightest replag in the last 24h was 1978s. That should be more or less correct. Update: I seem to be getting the vast majority of mails about z-dat-s4-a. Replag is 1000s but it seems strange that that one shows up so predominantly. ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Toolserver status
On Tue, Aug 9, 2011 at 1:39 AM, MZMcBride z...@mzmcbride.com wrote: K. Peachey wrote: On Tue, Aug 9, 2011 at 10:28 AM, MZMcBride z...@mzmcbride.com wrote: * http://status.toolserver.org/ is broken s/broken/hasn't been manually updated/ No, it's broken. You can click edit this page and view the source. It was updated last week, but it currently takes any status and outputs ok, regardless of user input. The changes are pending and need to be approved by an administrator. /me ducks ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Toolserver status
On Tue, Aug 9, 2011 at 11:32 AM, Krinkle krinklem...@gmail.com wrote: On Aug 9, 2011, at 10:20 AM, Magnus Manske wrote: On Tue, Aug 9, 2011 at 1:39 AM, MZMcBride z...@mzmcbride.com wrote: K. Peachey wrote: On Tue, Aug 9, 2011 at 10:28 AM, MZMcBride z...@mzmcbride.com wrote: * http://status.toolserver.org/ is broken s/broken/hasn't been manually updated/ No, it's broken. You can click edit this page and view the source. It was updated last week, but it currently takes any status and outputs ok, regardless of user input. The changes are pending and need to be approved by an administrator. /me ducks /me thought it was only editable by administrators... I was talking about the secret cabal meta-admins... ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
[Toolserver-l] Toolserver offline?
Can't lon into nightshade or willow (ssh keeps asking for password). http://toolserver.org/ does not responsive. ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Toolserver offline?
Back for me as well. On Mon, Aug 8, 2011 at 8:13 PM, Béria Lima berial...@gmail.com wrote: Jus tested and both works for me :D _ Béria Lima (351) 925 171 484 Imagine um mundo onde é dada a qualquer pessoa a possibilidade de ter livre acesso ao somatório de todo o conhecimento humano. É isso o que estamos a fazer. On 8 August 2011 19:15, Magnus Manske magnusman...@googlemail.com wrote: Can't lon into nightshade or willow (ssh keeps asking for password). http://toolserver.org/ does not responsive. ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Cronjob starting cronsub
On Tue, Mar 1, 2011 at 12:49 AM, MZMcBride z...@mzmcbride.com wrote: Magnus Manske wrote: I understand that, and I didn't say lfs was better (it has plenty of its own quirks), or that SGE was a bad choice - it seems to run well enough, all things considered. It's just that things like SGE might interpret comments in your scripts as commands and die or you can't run qsub from within qsub are types of behaviour that are unusual to say the least, and the toolserver users are left to the mercy of these issues (I'd call them bugs) without warning or documentation. (Yes, there is a wiki pages that says it can interpret special comments in your script. That's somewhat different from it will reinterpret your Perl script and fall over). Having learned some lessons today, you should write improved documentation on the wiki. :-) I might, once I've recovered from said learning experience... Magnus ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Cronjob starting cronsub
On Sun, Feb 27, 2011 at 1:10 PM, River Tarnell r.tarn...@ieee.org wrote: In article slrnimkfjp.fte.sa...@saper.info, Marcin Cieslak sa...@saper.info wrote: Magnus Manske magnusman...@googlemail.com wrote: I have a wrapper script that starts a handful of jobs on the toolserver (via qsub). But, when I put that wrapper script as a normal cronjob, I get some error messages about qsub not found. Starting the wrapper via cronsub doesn't work either. You should first run cronsub from the command-line with the same arguments. If that still doesn't work, post the entire command line and any output from it. Once you have it working from the command-line then it should be fine fromc ron. Actually cronsub does a litle bit more than sourcing /sge62/default/common/settings.sh file, so I don't know why it's not working for you. Maybe you could share some output? cronsub makes it easier to submit longrun jobs, automatically sets the default output file, runs the script from $HOME and prevents duplicate jobs being submitted. Otherwise it's identical to qsub. OK, so I gave up on starting my perl script using cronsub, which seems to work only if I give the perl script a parameter (WTF?), and instead altered it to be run manually, generating a shell script that could then start via cronsub. Script ( /home/magnus/cronjobs/most_wanted_wrapper.sh ) : #!/bin/ksh qsub -N mw_dewiki_p -j Y -o /home/magnus/cronjobs/update_most_wanted.dewiki_p.out -l sqlprocs-s5=1 /home/magnus/cronjobs/update_most_wanted.pl dewiki_p qsub -N mw_enwiki_p -j Y -o /home/magnus/cronjobs/update_most_wanted.enwiki_p.out -l sqlprocs-s1=1 /home/magnus/cronjobs/update_most_wanted.pl enwiki_p qsub -N mw_frwiki_p -j Y -o /home/magnus/cronjobs/update_most_wanted.frwiki_p.out -l sqlprocs-s6=1 /home/magnus/cronjobs/update_most_wanted.pl frwiki_p qsub -N mw_plwiki_p -j Y -o /home/magnus/cronjobs/update_most_wanted.plwiki_p.out -l sqlprocs-s2=1 /home/magnus/cronjobs/update_most_wanted.pl plwiki_p Running this script, or any of its individual lines, directly from shell, works absolutely fine. However, on cronsub mostwanted /home/magnus/cronjobs/most_wanted_wrapper.sh I get (in the error file): /home/magnus/cronjobs/most_wanted_wrapper.sh[2]: 7274 Segmentation Fault(coredump) /home/magnus/cronjobs/most_wanted_wrapper.sh[3]: 7289 Segmentation Fault(coredump) /home/magnus/cronjobs/most_wanted_wrapper.sh[4]: 7290 Segmentation Fault(coredump) /home/magnus/cronjobs/most_wanted_wrapper.sh[5]: 7298 Segmentation Fault(coredump) I tried replacing qsub with cronsub, only to be met with a flood of errors relating to all the options it doesn't know, including the sql slot reservation (I guess that's my punishment for trying to be a good citizen...). Removing basically all options, using only lines like this: cronsub -s mw_dewiki_p /home/magnus/cronjobs/update_most_wanted.pl dewiki_p in the wrapper script, again runs peachy from shell, but barfs coredumps again when run via cronsub. So, as far as I am concerned, it's broken. Cheers, Magnus ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Cronjob starting cronsub
On Mon, Feb 28, 2011 at 11:37 AM, River Tarnell r.tarn...@ieee.org wrote: In article aanlktiknfjfxqdvgzcj63hz90v_xpmkqvu145yj+p...@mail.gmail.com, Magnus Manske magnusman...@googlemail.com wrote: OK, so I gave up on starting my perl script using cronsub, which seems to work only if I give the perl script a parameter (WTF?), and instead altered it to be run manually, generating a shell script that could then start via cronsub. This should not be the case. Please provide the exact script and command line you used and the error message. cronsub update_dupes /home/magnus/update_dupes.pl qsub: Unknown option qsub: Unknown option qsub: Unknown option ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Cronjob starting cronsub
On Mon, Feb 28, 2011 at 1:59 PM, River Tarnell r.tarn...@ieee.org wrote: In article AANLkTi=cdpcjx_znz04wxmqiw6ssvz3nphywu7-oj...@mail.gmail.com, Magnus Manske magnusman...@googlemail.com wrote: cronsub update_dupes /home/magnus/update_dupes.pl qsub: Unknown option qsub: Unknown option qsub: Unknown option What is the output from this? $ ksh -x /opt/local/bin/cronsub update_dupes /home/magnus/update_dupes.pl + . /sge62/default/common/settings.sh + SGE_ROOT=/sge62 + export SGE_ROOT + + /sge62/util/arch ARCH=sol-amd64 + + /sge62/util/arch -m DEFAULTMANPATH=/usr/share/man + + /sge62/util/arch -mt MANTYPE=man + SGE_CELL=default + export SGE_CELL + SGE_CLUSTER_NAME=toolserver + export SGE_CLUSTER_NAME + unset SGE_QMASTER_PORT + unset SGE_EXECD_PORT + [ /sge62/man:/opt/ts/gnu/share/man:/opt/ts/share/man:/opt/SUNWspro/man:/usr/postgres/8.3/man:/opt/ts/mysql/5.1/share/man:/opt/ts/python/2.6/share/man:/opt/ts/perl/5.10/share/man:/opt/ts/ruby/1.8/share/man:/usr/share/man:/usr/sfw/share/man:/usr/cluster/man:/opt/jobserver/man = ] + MANPATH=/sge62/man:/sge62/man:/opt/ts/gnu/share/man:/opt/ts/share/man:/opt/SUNWspro/man:/usr/postgres/8.3/man:/opt/ts/mysql/5.1/share/man:/opt/ts/python/2.6/share/man:/opt/ts/perl/5.10/share/man:/opt/ts/ruby/1.8/share/man:/usr/share/man:/usr/sfw/share/man:/usr/cluster/man:/opt/jobserver/man + export MANPATH + PATH=/sge62/bin/sol-amd64:/sge62/bin/sol-amd64:/opt/local/bin:/opt/ts/gnu/bin:/opt/ts/bin:/opt/ts/mysql/5.1/bin:/opt/ts/perl/5.12/bin:/opt/ts/perl/5/bin:/opt/ts/python/2.7/bin:/opt/ts/php/5.3/bin:/opt/ts/ruby/1.8/bin:/opt/ts/mono/2.0/bin:/opt/ts/tcl/8.5/bin:/usr/ccs/bin:/sge62/bin/sol-amd64:/usr/bin:/usr/sbin:/usr/sfw/bin:/usr/postgres/8.3/bin:/usr/cluster/bin:/opt/jobserver/bin + export PATH + unset ARCH DEFAULTMANPATH MANTYPE + PATH=/usr/bin:/sge62/bin/sol-amd64:/sge62/bin/sol-amd64:/opt/local/bin:/opt/ts/gnu/bin:/opt/ts/bin:/opt/ts/mysql/5.1/bin:/opt/ts/perl/5.12/bin:/opt/ts/perl/5/bin:/opt/ts/python/2.7/bin:/opt/ts/php/5.3/bin:/opt/ts/ruby/1.8/bin:/opt/ts/mono/2.0/bin:/opt/ts/tcl/8.5/bin:/usr/ccs/bin:/sge62/bin/sol-amd64:/usr/bin:/usr/sbin:/usr/sfw/bin:/usr/postgres/8.3/bin:/usr/cluster/bin:/opt/jobserver/bin + sflag= + QUEUE=all.q + OPTS= + hrt= + getopts slh f + shift 0 + [[ 2 -ge 1 ]] + JOBNAME=update_dupes + shift + [ -z ] + [[ 1 -ge 1 ]] + SCRIPT=/home/magnus/update_dupes.pl + + mktemp TMPF=/tmp/tmp.5jaiw2 + chmod 700 /tmp/tmp.5jaiw2 + trap rm -f /tmp/tmp.5jaiw2 0 + cat + 1 /tmp/tmp.5jaiw2 0 #! /bin/sh cd $HOME exec $@ exit 1 + grep ^#\$ + 0 /home/magnus/update_dupes.pl 1 /tmp/tmp.5jaiw2 + grep Full jobname:[ \t]*update_dupes$ + 1 /dev/null + qstat -r + qsub -q all.q -N update_dupes /tmp/tmp.5jaiw2 /home/magnus/update_dupes.pl + 1 /dev/null qsub: Unknown option qsub: Unknown option qsub: Unknown option + rm -f /tmp/tmp.5jaiw2 ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Cronjob starting cronsub
On Mon, Feb 28, 2011 at 2:28 PM, Marcin Cieslak sa...@saper.info wrote: Magnus Manske magnusman...@googlemail.com wrote: On Mon, Feb 28, 2011 at 11:37 AM, River Tarnell r.tarn...@ieee.org wrote: In article aanlktiknfjfxqdvgzcj63hz90v_xpmkqvu145yj+p...@mail.gmail.com, Magnus Manske magnusman...@googlemail.com wrote: OK, so I gave up on starting my perl script using cronsub, which seems to work only if I give the perl script a parameter (WTF?), and instead altered it to be run manually, generating a shell script that could then start via cronsub. This should not be the case. Please provide the exact script and command line you used and the error message. cronsub update_dupes /home/magnus/update_dupes.pl qsub: Unknown option qsub: Unknown option qsub: Unknown option This is because you have in your script #$cmd = ./cron_sql.sh unless -e $cmd ; and this is recognized by qsub(1) as the embedded qsub option. If you change it to: # $cmd = it works again. Thanks! Wow, what a mess. Why does cronsub/qsub try to divine hidden vodoo meaning from my scripts (which predate qsub availability) anyway? If that is desired for some cases, why is that activated by default? Is there at least a qsub option to turn it off? Will that option work with cronsub? In my day job, I work with a large compute farm (lfs, ~10K cores), so I'm more than happy to work with, and migrate existing cronjobs to, this system, but it would help if it weren't a deliberate minefield... Cheers, Magnus ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Cronjob starting cronsub
On Mon, Feb 28, 2011 at 3:01 PM, Marcin Cieslak sa...@saper.info wrote: Magnus Manske magnusman...@googlemail.com wrote: On Mon, Feb 28, 2011 at 11:43 AM, Ilmari Karonen nos...@vyznev.net wrote: On 02/28/2011 01:00 PM, Magnus Manske wrote: OK, so I gave up on starting my perl script using cronsub, which seems to work only if I give the perl script a parameter (WTF?) Any parameter? I smell a line break encoding issue. Try making sure there are no carriage returns in your crontab or in any of the scripts involved. Well, since that happens also when I start cronsub fron the shell, I doubt it... It seems like qsub dumps core when executed from inside of the SGE job. /sge62/default/spool/wolfsbane/job_scripts/210756[3]: 1253 Segmentation Fault(coredump) I have filed https://jira.toolserver.org/browse/TS-963 about this. Really strange. Maybe some safety mechanism to prevent infinite recursion gone bad? ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
[Toolserver-l] Cronjob starting cronsub
Hi, I have a wrapper script that starts a handful of jobs on the toolserver (via qsub). But, when I put that wrapper script as a normal cronjob, I get some error messages about qsub not found. Starting the wrapper via cronsub doesn't work either. Anyone: What do I have to put in my .environment file (or whatever) so that my normal cronjob can start qsub/cronsub jobs? Thanks, Magnus ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Google Art Project ripping
On Sat, Feb 5, 2011 at 7:57 AM, Kolossos tim.al...@s2002.tu-chemnitz.de wrote: You can make your own pictures if you have a 7 Gigapixel camera ;-) I can make my own pictures if I have a 6 megapixel camera as well. I just can't make /the same/ pictures as Google, at least not easily. The question is if leeching a couple of 7 gigapixel images to Commons is worth p*g off the GLAMs (and Google). I think it's not, and the exchange on the cultural-partners mailing list about this topic seems to agree with me. These few pictures could set back any months (years even) of efforts for good collaborations with the GLAMs. Aaah, politics... ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Google Art Project ripping
I see NPG, reloaded coming our way... Note that Commons already has plenty of images from at least some of these museums, e.g.: http://commons.wikimedia.org/wiki/Van_Gogh_Museum For the rest, we could just ask them now that Google has put images online, can we too?, then go and take our own pictures. Too radical? ;-) Magnus On Fri, Feb 4, 2011 at 7:15 AM, Beao at Toolserver.org b...@toolserver.org wrote: Hello everybody! I've been working on a bash script to rip the high quality images from the Google Art Project. From what I understand, all depictions of the original works go under PD-Art, even though Google claims otherwise in their FAQ. What I'm wondering is whether anyone is interested in helping me rip all these images. There are a lot of images to rip, and each rip takes at least an hour. All you need is a GNU/Linux OS and some basic knowledge on using the terminal. -- Beao ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
[Toolserver-l] Monthly pageviews
Hi, I know there are lots'o'files for daily (hourly?) pageview stats on the toolserver. Are there aggregated counts for the whole month? So I only have to check 1 file instead of hundreds (the aggregated file would, of course, be smaller than the concatenated hourly ones). Or maybe even as a database? (Onecan dream...) If not, does anyone volunteer to generate them? They'd really help with my GLAM tools, increase Wikimedia outreach etc. Cheers, Magnus ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Reminder: move your projects off stable -Geohack
On Sat, Jan 2, 2010 at 4:38 PM, Platonides platoni...@gmail.com wrote: Which extension is it? http://svn.wikimedia.org/viewvc/mediawiki/trunk/extensions/geo/ ? Actually, most of it seems to be in http://svn.wikimedia.org/viewvc/mediawiki/trunk/extensions/gis/ The first step should be to make the scripts available before moving to anything. Kolossos says he can't access geohack-scripts, https://fisheye.toolserver.org/browse/geohack shows an empty svn folder... (*ahem*, https://jira.toolserver.org/browse/TS-187 ) Yes, this is a mess, but IMHO this is largely not our (geohack maintainers) fault; rather, things kept/keep changing on the toolserver, stable or not, and things are generally not in sync. For example, https://svn.toolserver.org/svnroot/geohack/ has a version of geohack, but fisheye does not show it. I am not sure how up-to-date this is, either; and I can't commit to (or even developer-checkout) that repository, neither as magnus nor as geohack. For now, I have put the current sources from stable into my own toolserver svn: https://fisheye.toolserver.org/browse/Magnus/geohack If anyone can update the real repository, and convince fisheye to show them too, that'd be peachy :-) Cheers, Magnus ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] More advanced watchlists
On Tue, Nov 3, 2009 at 5:13 PM, Lars Aronsson l...@aronsson.se wrote:
==Watch a category of articles==
2010 is election year in Sweden, so I want to keep an eye on all
articles in category:Swedish politicians, in recursive levels.
I now do this on a weekly basis, using CatScan,
http://toolserver.org/~daniel/WikiSense/CategoryIntersect.php
I set wiki = sv.wikipedia.org
search in category = svenska politiker
with depth = 7
for changes in last 168 hours (= 7 x 24 = one week)
save as CSV format
I then filter the output using the UNIX command line
awk '$1==0 {print $2}'|sort|uniq -c|sort -nr
to get a listing of which articles got the most edits.
It would be nice to intersect this search with category:Living
people, so dead politicians are filtered out.
Are there already any smarter tools to do this?
Sorted by date, with intersection:
http://toolserver.org/~magnus/catscan_rewrite.php?language=svdepth=7categories=Svenska+politiker%0D%0ALevande+personerafter=20091026sortby=datesortorder=descending
Cheers,
Magnus
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list:
https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] sql maintenance, Monday 14th
On Mon, Sep 14, 2009 at 9:03 AM, River Tarnell ri...@loreley.flyingparchment.org.uk wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 this maintenance is now finished and sql is available again. nightshade doesn't seem to agree: mag...@nightshade:~$ sql dewiki_p ERROR 2003 (HY000): Can't connect to MySQL server on 'sql-toolserver' (111) /usr/local/bin/sql: no such database dewiki_p Magnus ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Change onwer by multi-maintainer-project
On Wed, Sep 9, 2009 at 2:58 PM, River Tarnell ri...@loreley.flyingparchment.org.uk wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jan Luca: How can I run become in a SFTP-Client? you need to upload the files as your own user then copy them to the project account. Or become user, make sure everything's group-read/writable, then operate the STFP client under your normal user name, which should be in the shared tool group (type group on the shell, it show the shared tool name among others). Cheers, Magnus ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
[Toolserver-l] Stange Perl (?) problem
While I was on vacation, my CommonsHelper stopped working for direct upload. Here's what happens: * CommonsHelper (PHP) downloads an image to /tmp * CommonsHelper invokes upload bot written in Perl * Perl dies: Can't open file /tmp/chHHaikY-dir/dummy.jpg : No such file or directory at /opt/ts/perl/5.10/lib/vendor_perl/5.10/LWP/UserAgent.pm line 394 However, the file exists at that point in time (this one still does on wolfsbane, feel free to check), and is readable for everyone. Any idea what's going on? Magnus ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Stange Perl (?) problem
On Fri, Sep 4, 2009 at 12:07 PM, River Tarnell ri...@loreley.flyingparchment.org.uk wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Marco Schuster: River, can you please take a look at this? we have 5 Toolserver admins who are quite capable of investigating problems; not everything has to be done by me. If it's nothing obvious (probably related to the Solaris switch), I'll file a JIRA bug. Cheers, Magnus ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Wikimedia Foundation Grants $40, 000 to the Toolserver
YES! Fantastic! On Wed, Jul 29, 2009 at 2:53 PM, Daniel Kinzlerdan...@brightbyte.de wrote: Hi all. The Wikimedia Foundation has approved a grant of $40,000 towards improving the Toolserver’s reliability. With that money, we'll by three database servers, so we can keep two copies of each wiki database instead of just one. This should greatly improve the availability of up to date data in the wiki dbs. A blog post with some more details is at http://journal.toolserver.org/entry/2009/07/29/wikimedia-foundation-grants-4-to-the-toolserver/. You can find the original grant reqwuest at http://meta.wikimedia.org/wiki/Wikimedia_chapters/WMF_grants/WM_DE/Improve_toolserver_reliability. -- daniel ___ Toolserver-l mailing list Toolserver-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/toolserver-l ___ Toolserver-l mailing list Toolserver-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Re: [Toolserver-l] multi-maintainer tools
On Mon, Jul 27, 2009 at 1:12 PM, DaB.w...@daniel.baur4.info wrote: Hello, Am Montag 27 Juli 2009 14:03:32 schrieb Magnus Manske: Email to ts-adm...@toolserver.org, April 17, titled Stable TS: Am Montag 27 Juli 2009 14:04:06 schrieb Magnus Manske: Also, http://de.wikipedia.org/wiki/Benutzer_Diskussion:DaB.#SVN_repo and both is wrong. Use the TS-Project on Jira please. And since April, noone could invest 5 secs to reply to these two queries, even if it's use JIRA ... why? Magnus ___ Toolserver-l mailing list Toolserver-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Re: [Toolserver-l] multi-maintainer tools
On Mon, Jul 27, 2009 at 2:04 PM, DaB.w...@daniel.baur4.info wrote: Hello, Am Montag 27 Juli 2009 14:24:32 schrieb Magnus Manske: And since April, noone could invest 5 secs to reply to these two queries, even if it's use JIRA ... why? ok, it was my fault. Is now everybody happy? I wasn't trying to assign blame, merely pointing out where communications could be improved. I've submitted JIRA requests, all is well now. Cheers, Magnus ___ Toolserver-l mailing list Toolserver-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Re: [Toolserver-l] [Toolserver-announce] [stable] new web server
geohack appears to be working .I'm traveling so no in-depth tests from me this week... Magnus On Fri, May 22, 2009 at 10:41 PM, River Tarnell ri...@loreley.flyingparchment.org.uk wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 just a quick reminder, i'm still waiting for confirmation from these projects: * geohack * admstats * wmfgcbot * poty * nubio - river. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (HP-UX) iEYEARECAAYFAkoXDhQACgkQIXd7fCuc5vK+MACgkJ6fY0PpaJwBVUFSp+dAoSWb ++0AnRhgsZtv7zQBxpjwVtTpK3rm5AKo =4R+2 -END PGP SIGNATURE- ___ Toolserver-l mailing list Toolserver-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/toolserver-l ___ Toolserver-l mailing list Toolserver-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Re: [Toolserver-l] Experimental: Live template value search
On Tue, Apr 21, 2009 at 10:13 AM, Daniel Kinzler dan...@brightbyte.de wrote: Magnus Manske schrieb: I agree about Semantic MediaWiki, which is a different beast (and might one day be used on Wikipedia). That's really the question. Should we work *now* on making it usable for wikipedia, or should we focus on something simpler? IMHO we should try to harvest the data that is already in Wikipedia first. Semantic Wikipedia, technical issues aside, relies heavily on users learning a new syntax, which is a community (read: political;-) decision. And it will be fought about much harder and longer than the license question... The question seems to be scalability.Extrapolating from my sample data set, just the key/value pairs of templates directly included in articles would come to over 200 million rows for en.wikipedia at the moment. A MediaWiki-internal solution would want to store templates included in templates as well, which can be a lot for complicated meta-templates. I think a billion rows for the current English Wikipedia is not too far-fetched in that model. The table would be both constantly updated (potentially hundeds of writes for a single article update) and heavily searched (with LIKE %stuff%, no less). Would the RDF extension be up to that? It would in a way: it just wouldn't store all parameters. It would store only things explicitly defined to be RDF values. That would greatly reduce the number of parameters to store, since all the templates used maintenance, formatting, styling and navigation can be omitted. It would be used nearly exclusively for infobox-type templates, image meta-info, and cross-links like the PND template. Or at least, that'S the idea. It also does away with problems caused by the various names a parameters with the same meaning may have in different templates (and different wikis). Nice! I was thinking along the lines of a template whitelist/blacklist, but yours would be much more efficient. And it would hide most of the technical ugliness in the templates. ___ Toolserver-l mailing list Toolserver-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Re: [Toolserver-l] Simple Geohack-like tool for CAS numbers
Wrote that a year ago:
http://svn.wikimedia.org/viewvc/mediawiki/trunk/extensions/TemplateLink/
But, as with all my extensions, I don't have the time and will to push
24/7, and noone else cares, so...
Magnus
On Mon, Nov 10, 2008 at 10:15 AM, Conrad Irwin
[EMAIL PROTECTED] wrote:
That would actually be a great idea. If mediawiki supported urls
something like [[Special:Template/Cool|Alphabet|Grandplan|NamedParam=Amazing]]
which would produce the same output as
{{Cool|Alphabet|Grandplan|NamedParam=Amazing}} but on a standalone
page then I feel that could have a lot of uses.
Another related idea would be to allow the preload= parameter to have
variable substitution - that way creating page content remotely could
also be made easier.
Yours, hoping someone else has the time to do this.
Conrad
2008/11/10 Nikola Smolenski [EMAIL PROTECTED]:
Annabel wrote:
I guess it is a good idea to contact user Beetstra on the English
wikipedia. He has been busy in the past with writing a plugin for mediawiki
for this task. It seemed a hard job to implement such a thing practically
in mediawiki. I do not know the current state of his project, but you may
want to ask him (if there is not a direct chance that it will be
implemented and included in mediawiki, I guess that the code could be
relatively easy converted to a toolserver tool)
In fact, I think it would be easy to generalize this into a system where
you can pair any page (template) with any parameter; that could even be
used to replace Special:BookSources . I already have something similar.
- Original Message
From: Guillaume Paumier [EMAIL PROTECTED]
To: Toolserver mailing list toolserver-l@lists.wikimedia.org
Sent: Sunday, November 9, 2008 8:19:43 PM
Subject: [Toolserver-l] Simple Geohack-like tool for CAS numbers
Hello,
CAS numbers are used to identify chemical compounds [1]. Someone on
fr.wp has created a template for CAS numbers; it uses the CAS number
as a parameter and outputs some links to several resources/databases
[2]. The thing is, although these links might be useful, the articles
aren't really the good place for them.
I was wondering if someone from this list would be willing to write a
simple tool on the toolserver à la GeoHack. The tool would do exactly
what the template is doing right now, i.e. it would check the validity
of the input and would output a few links [3]. The template would then
be modified to simply link to this tool.
This should be pretty easy (I think) and it would avoid displaying all
these links inside the articles. Anyone interested?
Many thanks in advance.
[1] http://en.wikipedia.org/wiki/CAS_registry_number
[2] example: http://fr.wikipedia.org/wiki/Acrylamide ; look at N°
CAS in the infobox and click on the [+] symbol to expand the
template.
[3] current code of the template :
http://fr.wikipedia.org/w/index.php?title=Mod%C3%A8le:CASaction=edit
___
Toolserver-l mailing list
Toolserver-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
___
Toolserver-l mailing list
Toolserver-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
___
Toolserver-l mailing list
Toolserver-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Re: [Toolserver-l] help test a new web server
My tools work fine there, as far as I can see. Magnus On Thu, Aug 28, 2008 at 6:14 PM, River Tarnell [EMAIL PROTECTED] wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 hi, i'm currently evaluating a new web server on hemlock. please, help test it (especially with complex .htaccess, CGI/PHP scripts, etc). to use the test server, replace toolserver.org in the URL with toolserver.org:81, so http://toolserver.org/~jsmith/cgi-bin/myscript becomes: http://toolserver.org:81/~jsmith/cgi-bin/myscript please, report any problems. (please, don't distribute the toolserver.org:81 URLs, they are only for testing and will stop working soon.) - river. PS: the new server does _not_ support RewriteRules in .htaccess; however, it does support Alias and Redirect. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (SunOS) iEYEARECAAYFAki23PIACgkQIXd7fCuc5vJqhwCfTg0j+/9Gu4Sq6fH4a6lITCHm zjsAnjOtCvA2D/SOvvIrBEJqSBYUin6z =CEpI -END PGP SIGNATURE- ___ Toolserver-l mailing list Toolserver-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/toolserver-l ___ Toolserver-l mailing list Toolserver-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Re: [Toolserver-l] Toolserver is very slow
On Tue, Jul 22, 2008 at 8:50 AM, Daniel Kinzler [EMAIL PROTECTED] wrote: James R. schrieb: All bots and other tools are now run on nightshade, leaving hemlock for all apache jobs. There is no problem with nightshade's speed, it is perfect :) Regards, James R. That is not so helpful when response time for all web tools is over one minute... Confirmed - noticable slower than usual in the UK. Magnus ___ Toolserver-l mailing list Toolserver-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/toolserver-l
[Toolserver-l] PHP problems
(I tried to reply to the new PHP thingy mail before, but my mails bounced. Resubscribed and trying again) Since the switch to the new FastCGI PHP wrapper, I observe some problems with my tools: 1. When using the gzip output handler (default on my tools), output dies after a screen or two with: Content Encoding Error The page you are trying to view cannot be shown because it uses an invalid or unsupported form of compression. 2. Intermittent, output of longer-running scripts dies (again, in the middle of the run): Internal Server Error (500) The server encountered an internal error and is unable to complete your request at this time. If the problem persists, please contact the owner of the tool you are trying to use and inform them of this error, quoting the following information: * Request host: toolserver.org * Request path: GET /~magnus/flickr2commons.php 3. Accessing external sites from the toolserver (e.g., copying files from flickr to reupload them on Commons) dies in 50% of the cases. So, yes, I do see significant problems with the wrapper as it is now. Magnus ___ Toolserver-l mailing list Toolserver-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/toolserver-l
