For completeness, here's two tracebacks (there were more similar ones)
from the same job:
/mnt/galaxyData/tmp/job_working_directory/000/75/task_4:
Traceback (most recent call last):
File "./scripts/extract_dataset_part.py", line 25, in <module>
import galaxy.model.mapping #need to load this before we unpickle, in order
to setup properties assigned by the mappers
File "/mnt/galaxyTools/galaxy-central/lib/galaxy/model/__init__.py", line 13, in
<module>
import galaxy.datatypes.registry
File "/mnt/galaxyTools/galaxy-central/lib/galaxy/datatypes/registry.py", line 8, in
<module>
from display_applications.application import DisplayApplication
File
"/mnt/galaxyTools/galaxy-central/lib/galaxy/datatypes/display_applications/application.py",
line 9, in <module>
from util import encode_dataset_user
File
"/mnt/galaxyTools/galaxy-central/lib/galaxy/datatypes/display_applications/util.py",
line 3, in <module>
from Crypto.Cipher import Blowfish
File
"/mnt/galaxyTools/galaxy-central/eggs/pycrypto-2.5-py2.7-linux-x86_64-ucs4.egg/Crypto/Cipher/Blowfish.py",
line 7, in <module>
File
"/mnt/galaxyTools/galaxy-central/eggs/pycrypto-2.5-py2.7-linux-x86_64-ucs4.egg/Crypto/Cipher/Blowfish.py",
line 4, in __bootstrap__
File "/mnt/galaxyTools/galaxy-central/lib/pkg_resources.py", line 882, in
resource_filename
self, resource_name
File "/mnt/galaxyTools/galaxy-central/lib/pkg_resources.py", line 1351, in
get_resource_filename
self._extract_resource(manager, self._eager_to_zip(name))
File "/mnt/galaxyTools/galaxy-central/lib/pkg_resources.py", line 1373, in
_extract_resource
self.egg_name, self._parts(zip_path)
File "/mnt/galaxyTools/galaxy-central/lib/pkg_resources.py", line 962, in
get_cache_path
self.extraction_error()
File "/mnt/galaxyTools/galaxy-central/lib/pkg_resources.py", line 928, in
extraction_error
raise err
pkg_resources.ExtractionError: Can't extract file(s) to egg cache
The following error occurred while trying to extract file(s) to the Python egg
cache:
[Errno 17] File exists:
'/home/galaxy/.python-eggs/pycrypto-2.5-py2.7-linux-x86_64-ucs4.egg-tmp'
The Python egg cache directory is currently set to:
/home/galaxy/.python-eggs
Perhaps your account does not have write access to this directory? You can
change the cache directory by setting the PYTHON_EGG_CACHE environment
variable to point to an accessible directory.
/mnt/galaxyData/tmp/job_working_directory/000/75/task_5:
Traceback (most recent call last):
File "./scripts/extract_dataset_part.py", line 22, in <module>
import simplejson
File
"/mnt/galaxyTools/galaxy-central/eggs/simplejson-2.1.1-py2.7-linux-x86_64-ucs4.egg/simplejson/__init__.py",
line 111, in <module>
File
"/mnt/galaxyTools/galaxy-central/eggs/simplejson-2.1.1-py2.7-linux-x86_64-ucs4.egg/simplejson/decoder.py",
line 7, in <module>
File
"/mnt/galaxyTools/galaxy-central/eggs/simplejson-2.1.1-py2.7-linux-x86_64-ucs4.egg/simplejson/scanner.py",
line 10, in <module>
File
"/mnt/galaxyTools/galaxy-central/eggs/simplejson-2.1.1-py2.7-linux-x86_64-ucs4.egg/simplejson/scanner.py",
line 6, in _import_c_make_scanner
File
"/mnt/galaxyTools/galaxy-central/eggs/simplejson-2.1.1-py2.7-linux-x86_64-ucs4.egg/simplejson/_speedups.py",
line 7, in <module>
File
"/mnt/galaxyTools/galaxy-central/eggs/simplejson-2.1.1-py2.7-linux-x86_64-ucs4.egg/simplejson/_speedups.py",
line 4, in __bootstrap__
File "/mnt/galaxyTools/galaxy-central/lib/pkg_resources.py", line 882, in
resource_filename
self, resource_name
File "/mnt/galaxyTools/galaxy-central/lib/pkg_resources.py", line 1351, in
get_resource_filename
self._extract_resource(manager, self._eager_to_zip(name))
File "/mnt/galaxyTools/galaxy-central/lib/pkg_resources.py", line 1373, in
_extract_resource
self.egg_name, self._parts(zip_path)
File "/mnt/galaxyTools/galaxy-central/lib/pkg_resources.py", line 962, in
get_cache_path
self.extraction_error()
File "/mnt/galaxyTools/galaxy-central/lib/pkg_resources.py", line 928, in
extraction_error
raise err
pkg_resources.ExtractionError: Can't extract file(s) to egg cache
The following error occurred while trying to extract file(s) to the Python egg
cache:
[Errno 17] File exists: '/home/galaxy/.python-eggs'
The Python egg cache directory is currently set to:
/home/galaxy/.python-eggs
Perhaps your account does not have write access to this directory? You can
change the cache directory by setting the PYTHON_EGG_CACHE environment
variable to point to an accessible directory.
On 09/18/2012 05:24 PM, James Taylor wrote:
Interesting. If I'm reading this correctly the problem is happening
inside pkg_resources? (galaxy.eggs unzips eggs, but I think it does so
on install [fetch_eggs] time not run time which would avoid this). If
so this would seem to be a locking bug in pkg_resources. Dannon, we
could put a guard around the imports in extract_dataset_part.py as an
(overly aggressive and hacky) fix.
-- jt
On Tue, Sep 18, 2012 at 10:37 AM, Jorrit Boekel
<[email protected]> wrote:
- which lead to unzipping .so libraries from python eggs into the nodes'
/home/galaxy/.python-eggs
- this runs into lib/pkg_resources.py and its _bypass_ensure_directory
method that creates the temporary dir for the egg unzip
- since there are 8 processes on the node, sometimes this method tries to
mkdir a directory that was just made by the previous process after the
isdir.
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
