Re: [Toolserver-l] Grid Engine config change
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04.03.2012 01:06, Merlissimo wrote: In both cases the old behavior was without -m a -b y, so 'cronsub [jobname] [command]' has become 'qcronsub -l h_rt=06:00:00 -l virtual_free=100M -N [jobname] [command] 'cronsub -l [jobname] [command]' has become 'qcronsub -l h_rt=INFINITY -l virtual_free=100M -N [jobname] [command] The -b y option is mostly useful for binaries, e.g. if you don't submit the python script itself, but call the binary interpreter (python) with an argument. It is just an option if the submitted script file should be copied to a local filesystem on execution server (which increases performance, makes nfs error impossible and was always the default setting) or executed directly from your home (if you use -b y). In most cases this option isn't needed and copying is the best for most shell scripts. Thanks for that info! I thought to remember once there was a comment about cronsub copying (like -b y does) the script...?!? Anyway I had to use '-b y' else my script closed directly after execution. What am I doing wrong here? (my script do read and write files in my home, e.g. config and log files - may be this is related?) Greetings DrTrigon -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9TPfcACgkQAXWvBxzBrDDQMwCffVmUUDwH2bkPgTedGhk2W/7f uq4AoMqopPOTZfz4WnmpsSUAWoUcx8Ve =v1y9 -END PGP SIGNATURE- ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Grid Engine config change
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04.03.2012 01:21, Platonides wrote: On 04/03/12 00:17, Dr. Trigon wrote: And here my key question arises; you mentioned 'qacct' to get more info (thanks for this hint) and this is one of the biggest problem I had with the whole SGE stuff; I was not able to get a complete docu whether on the toolserver nor else. At the moment, on the toolserver commands like 'qstat' or 'qdel' are not covered anymore. I (we) would like to know more about this great system. ? They are documented in the server man pages. Just run man qacct Or PAGER=less man qacct as I find it a nicer one. No this was not my point. If I know there is a command like e.g. 'qacct' I know how to get help. My question was where do I get a list of all SGE commands, options and backgrund infos? (it does not need to be a reference book, but a user manual would be nice) I searched the net several times for SGE infos and found some in several places but that have been small parts of the whole docu only. But I assume there has to be something... ;) Greetings DrTrigon -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9TPuAACgkQAXWvBxzBrDCs8gCg2gZ0dDQh/GRykzsZ+h7h6roC jFoAoL6RkEoc7+8LBQKb9TFpDlJfh7+x =tAG2 -END PGP SIGNATURE- ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Grid Engine config change
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04.03.2012 01:06, Merlissimo wrote: Some weeks ago i installed a script that removes empty log files for standard error/output stream after job execution. Many people used this option to prevent that their homedir contain so many empty error logs. Could you may be explain how to use this script you installed? Another question I have; Is there a way to finish and start a script without having to wait until a queue becomes free by re-using to old? E.g. if I like to restart my ircbot (running for INFTY time) I have to wait some time until the needed queue becomes available, but what about just re-using the queue it already had before? Or is there an alternative way to have such a ircbot script running continuously without having to wait (e.g. more than 1 minute) during a restart? Can I trigger a restart (reading the new script from hdd) while keeping it in the queue? Thanks again! Greetings DrTrigon -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9TQoYACgkQAXWvBxzBrDDaagCgvt5IyYPxdi0yXG90qgdLG87E UkYAoLQ+Zz+tJC8Q8wX8So+slSK6X1mV =5XZz -END PGP SIGNATURE- ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
[Toolserver-l] Nightshade timing out?
Is nightshade.toolserver.org timing out for anyone else? -- User:Hersfold hersfoldw...@gmail.com ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Nightshade timing out?
Hello, At Sunday 04 March 2012 14:53:13 DaB. wrote: Is nightshade.toolserver.org timing out for anyone else? yes, since ~2 weeks. It was announced here and the status-page tells the same. Sincerly, DaB. -- Userpage: [[:w:de:User:DaB.]] — PGP: 2B255885 signature.asc Description: This is a digitally signed message part. ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
[Toolserver-l] Debian (Linux) is coming back
Hello all, as you may noticed nightshade is not re-setup yet and also yarrow is not back for use. That have a reason: We decided to get rid of Solaris on the userland- server again, because no real solaris-expert is left in the root-group; but there are 2 Debian-users in the group so we choose Debian as new operating system. The original plan was to use yarrow as a playground and when everything is fine there to switch willow. For some unknown reason, nightshade lost its solaris- installation shortly after or during the last colo-visit. So the plan was changed in the following manner: We use yarrow as playground, but try to speed up everything, and when we fine with it, we switch *nightshade* over. I hope to make all changes to puppet (our server-managment- system) during the next 7 days, but it may take few days longer. If you are a Debian-user too, you can help me a little bit: Insert needed packages at [1] and help other users to find packages for their needed liberies. Willow will keep Solaris for several more months (if nothing bad happens with it), so there will be no problems with non-running tools. At first everything will keep running on willow and when you are ready you can switch over your tools to Debian (if you use SGE, that will be very easy because only 1 flag has to set – more details later) – for most tools there should be no (big) problems. Just as a information for you what is going on at the moment. Sincerley, DaB. [1] https://wiki.toolserver.org/view/User:Dab/Debian-Packages -- Userpage: [[:w:de:User:DaB.]] — PGP: 2B255885 signature.asc Description: This is a digitally signed message part. ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Grid Engine config change
On 04/03/12 01:06, Merlissimo wrote: In both cases the old behavior was without -m a -b y, so (...) The -b y option is mostly useful for binaries, e.g. if you don't submit the python script itself, but call the binary interpreter (python) with an argument. (...) Actually, cronsub works as if providing -b y. It doesn't provide -b y, but submits a new script which then calls the calling script, so it's like submitting the script with -b y. ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Nightshade timing out?
Thanks, didn't check my email that far back. I usually use willow, but for some reason I've still got one thing set up for nightshade. User:Hersfold hersfoldw...@gmail.com On 3/4/2012 8:54 AM, Hydriz Wikipedia wrote: Its currently (still) unavailable due to hardware issue. See https://jira.toolserver.org/browse/MNT-1198 On Sun, Mar 4, 2012 at 9:22 PM, Hersfold hersfoldw...@gmail.com mailto:hersfoldw...@gmail.com wrote: Is nightshade.toolserver.org http://nightshade.toolserver.org timing out for anyone else? -- User:Hersfold hersfoldw...@gmail.com mailto:hersfoldw...@gmail.com ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org mailto:Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette -- Regards, Hydriz We've created the greatest collection of shared knowledge in history. Help protect Wikipedia. Donate now: http://donate.wikimedia.org http://donate.wikimedia.org/ ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Grid Engine config change
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04.03.2012 16:25, Platonides wrote: On 04/03/12 01:06, Merlissimo wrote: In both cases the old behavior was without -m a -b y, so (...) The -b y option is mostly useful for binaries, e.g. if you don't submit the python script itself, but call the binary interpreter (python) with an argument. (...) Actually, cronsub works as if providing -b y. It doesn't provide -b y, but submits a new script which then calls the calling script, so it's like submitting the script with -b y. Aaa-ha! That explains a lot! Thanks for the hint! Greetings -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9Tnk4ACgkQAXWvBxzBrDBAgwCeLKRGxQm+VtNY8WweJx1167Ar 1HIAmwdqIzlRo8arZz4P0G3eAURhSUnI =STMa -END PGP SIGNATURE- ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] About cron
On 04/03/12 17:42, محمد الجداوي wrote: Hi there. I made a cron for operating /clean_sandbox.py/ every 6 hours, I made a modified copy from /clean_sandbox.py/ for me and uploaded it to my account on toolserver (outside the pywikipedia folder). The problem i face is that i can't write the proper code, I made this code: #!/bin/sh #$ -j y #$ -o /dev/null $HOME python clean_sandbox.py -lang:ar -family:wikipedia But it didn't work. In which folder? I suspect you are getting that run in the wrong folder. Moreover, that $HOME there seems useless. So, let's assume it's in at /home/name/local_clean/clean_sandbox.py First step, check manually that it works: cd /home/name/local_clean/ python clean_sandbox.py -lang:ar -family:wikipedia Does it run? Do you have any problems for eg. not finding the rest of pywikipediabot? Then, when creating the script, make it run in that folder: #!/bin/sh #$ -j y #$ -o /dev/null #$ -l h_rt=00:10:00 #$ -l virtual_free=20M #$ -wd /home/name/local_clean/clean_sandbox.py python clean_sandbox.py -lang:ar -family:wikipedia I'd also recommend you to not run it with -o /dev/null the first time, so you can see the output files if something were wrong. (I also added there a time limit of 10 minutes to clean the sandbox, and an arbitrary memory size of 20M, in line with Merlissimo guidelines) ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Debian (Linux) is coming back
Zawartość nagłówka [Followup-To: gmane.org.wikimedia.toolserver.] DaB. w...@daniel.baur4.info wrote: --===3758331293830646509== Content-Type: multipart/signed; boundary=nextPart244.Px5et4TFKU; protocol=application/pgp-signature; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit --nextPart244.Px5et4TFKU Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hello all, as you may noticed nightshade is not re-setup yet and also yarrow is not ba= ck=20 for use. That have a reason: We decided to get rid of Solaris on the userla= nd- server again, because no real solaris-expert is left in the root-group; but= =20 there are 2 Debian-users in the group so we choose Debian as new operating= =20 system. The original plan was to use yarrow as a playground and when everything is = fine=20 there to switch willow. For some unknown reason, nightshade lost its solari= s- installation shortly after or during the last colo-visit. DaB, Please let me know if you need some help with Solaris. //Saper ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Virtual memory limits for interwiki.py
All jobs must at least contain informations about maximum runtime (h_rt) and peak memory usage (virtual_free). This information may get obligatory in future. Currently only a warning message is shown. You also have to request other resources like sql connections, free temp space, etc. if these are needed by your job. Please read documentation on toolserverwiki i have updated today: https://wiki.toolserver.org/view/Job_scheduling This currently contains the main informations you need to know, but maybe i add some more examples later. Thank you for your efforts. It would be sad to see SGE go because lack of proper maintenance. I already got used to it and currently I submit all my (even on-off) SQL queries by qsub. I have one question that is vaguely related problem related to the virtual memory size. Does anyone have some reasonable values for virtual memory limits (and other resources as well) for interwiki.py? On tools.wikimedia.pl (a small toolserver box run by Wikimedia Poland) we run some interwiki bots and I am slowly getting crazy to figure out correct values for this. As far as I can see, for interwiki.py the sky is the limit. Any hints how to gauge virtual memory usage given some set of input data? We run bots not only for wikipedia but also for wiktionary. //Saper ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Grid Engine issues
By the way, did you have a chance to check how mail is delivered to job submitters? We have tickets: https://jira.toolserver.org/browse/TS-862 I don't receive SGE job started/ended mails anymore and related (if not duplicated): https://jira.toolserver.org/browse/TS-955 willow local MTA does not seem to accept mail from SGE jobs There were some issues in configuring MTAs/SGE so that email works just right, i.e. is delivered to the submitter (saper@submittedhost) and not to the localuser on the host running the task (saper@runninghost). Another interesting issue (which probably can be solved only by software upgrade) is that attempt to submit a job from another job results in segmentation fault: https://jira.toolserver.org/browse/TS-963 qsub segfaults when running from within SGE job //Saper ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] About cron
Platonides wrote: On 04/03/12 17:42, محمد الجداوي wrote: Hi there. I made a cron for operating /clean_sandbox.py/ every 6 hours, I made a modified copy from /clean_sandbox.py/ for me and uploaded it to my account on toolserver (outside the pywikipedia folder). The problem i face is that i can't write the proper code, I made this code: #!/bin/sh #$ -j y #$ -o /dev/null $HOME python clean_sandbox.py -lang:ar -family:wikipedia But it didn't work. In which folder? I suspect you are getting that run in the wrong folder. Moreover, that $HOME there seems useless. So, let's assume it's in at /home/name/local_clean/clean_sandbox.py First step, check manually that it works: cd /home/name/local_clean/ python clean_sandbox.py -lang:ar -family:wikipedia Does it run? Do you have any problems for eg. not finding the rest of pywikipediabot? Then, when creating the script, make it run in that folder: #!/bin/sh #$ -j y #$ -o /dev/null #$ -l h_rt=00:10:00 #$ -l virtual_free=20M #$ -wd /home/name/local_clean/clean_sandbox.py python clean_sandbox.py -lang:ar -family:wikipedia I'd also recommend you to not run it with -o /dev/null the first time, so you can see the output files if something were wrong. (I also added there a time limit of 10 minutes to clean the sandbox, and an arbitrary memory size of 20M, in line with Merlissimo guidelines) -wd specifies the working _directory_ and not a file. It's needed if you use relative path names as your script does. Joining the error stream into the standard output stream and writing both to /dev/null isn't a good idea if you are searching for an error cause. Merlissimo ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] About cron
On 04/03/12 23:45, Merlissimo wrote: -wd specifies the working _directory_ and not a file. It's needed if you use relative path names as your script does. Joining the error stream into the standard output stream and writing both to /dev/null isn't a good idea if you are searching for an error cause. Merlissimo Whoops. Copy paste error. I had intended to remove clean_sandbox.py from that line. Thanks for noticing. ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Debian (Linux) is coming back
DaB. w...@daniel.baur4.info wrote: [...] If you are a Debian-user too, you can help me a little bit: Insert needed packages at [1] and help other users to find packages for their needed liberies. [...] Does this mean that all the accumulated JIRA requests for Perl modules Co. will get scrapped? :-) I believe we had several issues already in the past when the installed soft- ware differed between the Solaris servers. Which brings me to: Does anyone know an established format a) in which pro- jects could write down their requirements and b) that covers both Debian and Solaris? So when admins need to (re-)in- stall a server, they wouldn't have to guess which packages are (still) required, but could just collect all $HOME/.requirements for active accounts and when one of these could not be satisfied, there would also be a person to contact before tools get broken. Tim ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Debian (Linux) is coming back
On Sun, Mar 4, 2012 at 22:05, Tim Landscheidt t...@tim-landscheidt.de wrote: Which brings me to: Does anyone know an established format a) in which pro- jects could write down their requirements and b) that covers both Debian and Solaris? So when admins need to (re-)in- stall a server, they wouldn't have to guess which packages are (still) required, but could just collect all $HOME/.requirements for active accounts and when one of these could not be satisfied, there would also be a person to contact before tools get broken. I assume this is one of the reasons to use puppet? Puppet manifests can have comments and the roots could establish a standardized way of writing (inline) why a package is needed. (e.g., a) assumed to be widely used like sed, grep or b) needed by the roots or a process that doesn't belong to a particular user or MMP or c) needed by users/MMP foo, baz, bar or some combination of those.) Of course most people (whether roots or users or anyone else) won't do a very thorough job of enumerating dependencies when installing, updating, hacking software unless they first do an installation of that version on a brand new Debian install with a limited set of installed packages. i.e. most people won't notice that a package is needed or not already picked up some other way until they see something is broken because it's missing. I'm not sure if I like ~/.requirements (and maybe it could be something like ~/.package-depends instead?) in place of puppet but at least it could be used as a failsafe if a root wanted to check after installing a new box or before removing a package. This all got me thinking: can SGE be told that a job needs a certain package, choose a box that has it already installed, and keep relevant stats so that roots know if a package should be installed somewhere else as well? -Jeremy ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
[Toolserver-l] What happened to user_properties_anonym?
Hey all, I used to periodically run some stats on preference usage (mostly skin-preferences and gadget usage), but it appears the custom user_properties_anonym table view Toolserver used to have no longer exists ? Testing here: https://toolserver.org/~krinkle/tmp/user_properties_anonym.php -- Krinkle ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Debian (Linux) is coming back
DaB, On Sun, Mar 4, 2012 at 4:04 PM, Marcin Cieslak sa...@saper.info wrote: Please let me know if you need some help with Solaris. Let me know if you need Solaris or other help. Thanks, Gerald ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
