Hi again,

I have looked into this matter a little bit more, and it looks like this is happening:


- tasked job is split
- tasks commands are sent to workers (I am running 8-core high cpu extra large workers on EC2)
- per task, worker runs env.sh for the respective tool
- per task, worker runs scripts/extract_dataset_part.py
- this scripts issues import statements (ones forsimplejson and galaxy.model.mapping have caused me problems) - which lead to unzipping .so libraries from python eggs into the nodes' /home/galaxy/.python-eggs - this runs into lib/pkg_resources.py and its _bypass_ensure_directory method that creates the temporary dir for the egg unzip - since there are 8 processes on the node, sometimes this method tries to mkdir a directory that was just made by the previous process after the isdir.

That last point is my guessing. I don't really know how to solve this in a non-hackish way, so until someone finds out, I may use reading from a 'eggs_extracted.txt' file to determine if the eggs have been extracted. And locking the file when writing to it of course.

cheers,
jorrit

On 09/14/2012 10:57 AM, Jorrit Boekel wrote:
Dear list,

I am running galaxy-dist on Amazon EC2 through Cloudman, and am using the enable_tasked_jobs to run jobs in parallel. Yes, I know it's not recommended in production. My jobs usually get split in 72 parts, and sometimes (but not always, maybe in 30-50% of cases), errors are returned concerning the python egg cache, usually:

[Errno 17] File exists: '/home/galaxy/.python-eggs'

or something like

[Errno 17] File exists: '/home/galaxy/.python-eggs/simplejson-2.1.1-py2.7-linux-x86_64-ucs4.egg-tmp'

The errors arise AFAIK from when scripts/extract_dataset_part.py is run. I am guessing that the tmp python egg dir is created for every task of the mentioned 72, that they sometimes coincide and that this leads to an error.

I would like to solve this problem, but before doing so, I'd like to know if someone else has already fixed it in a galaxy-central changeset.

cheers,
jorrit
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

 http://lists.bx.psu.edu/

Reply via email to