Hi, Tassilo, Have you solved this problem? I encounter the same problem at present without any idea for it. I just run some separate programs in the same time then the function compiling outputs the similar problem. Thanks a lot for any idea.
在 2015年3月9日星期一 UTC-4下午1:51:30，Tassilo Klein写道： > > Hi Fred, > > I just wanted to follow up on this issue as I am sort of deadlocked with > this problem now. Do you see any solution? > Btw, the non-thread safety issue of Theano has not been fixed yet, right? > Because it sounds somewhat related to that. > > -Tassilo > > On Mon, Mar 2, 2015 at 7:07 PM, Tassilo Klein <[email protected] > <javascript:>> wrote: > >> Hi Fred, >> >> yes, master and slave have the same directory. Sometimes it works better, >> sometimes worse - feels like a race condition. It is weird. Here are >> stack-traces for different error outputs. >> >> Cheers, >> Tassilo >> >> >> 15/03/02 18:55:57 WARN TaskSetManager: Lost task 64.0 in stage 7.19 (TID >> 17710, node004.cm.cluster): org.apache.spark.api.python.PythonException: >> Traceback (most recent call last): >> File "/scratch/users/tjklein/215178/spark/python/pyspark/worker.py", line >> 101, in main >> process() >> File "/scratch/users/tjklein/215178/spark/python/pyspark/worker.py", line >> 96, in process >> serializer.dump_stream(func(split_index, iterator), outfile) >> File "/scratch/users/tjklein/215178/spark/python/pyspark/serializers.py", >> line 236, in dump_stream >> vs = list(itertools.islice(iterator, batch)) >> File "/home/tjklein/cnn-3d/spark_LORM_refactored.py", line 1046, in >> distributed_gradient_computation >> return broadcast_ADMM_gradient_function.value(*param_list) >> File "/scratch/users/tjklein/215178/spark/python/pyspark/broadcast.py", >> line 106, in value >> self._value = self.load(self._path) >> File "/scratch/users/tjklein/215178/spark/python/pyspark/broadcast.py", >> line 95, in load >> return cPickle.loads(data) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/compile/function_module.py", >> line 747, in _constructor_Function >> f = maker.create(input_storage, trustme = True) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/compile/function_module.py", >> line 1415, in create >> input_storage=input_storage_lists) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/link.py", >> line 525, in make_thunk >> output_storage=output_storage)[:3] >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/vm.py", line >> 897, in make_all >> no_recycling)) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/op.py", line >> 1002, in make_thunk >> compute_map, no_recycling) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/op.py", line >> 739, in make_thunk >> output_storage=node_output_storage) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", line >> 1072, in make_thunk >> keep_lock=keep_lock) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", line >> 1014, in __compile__ >> keep_lock=keep_lock) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", line >> 1441, in cthunk_factory >> key=key, lnk=self, keep_lock=keep_lock) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", >> line 1076, in module_from_key >> module = lnk.compile_cmodule(location) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", line >> 1353, in compile_cmodule >> preargs=preargs) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", >> line 1889, in compile_str >> raise MissingGXX("g++ not available! We can't compile c code.") >> MissingGXX: (MissingGXX('The following error happened while compiling the >> node', Elemwise{neg,no_inplace}(y), '\n', "g++ not available! We can't >> compile c code.", '[Elemwise{neg,no_inplace}(y)]'), <function >> _constructor_Function at 0x2aaacc32d230>, >> (<theano.compile.function_module.FunctionMaker object at 0x2aab2a53ff10>, >> [<None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, >> <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, >> <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, >> <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, >> <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, >> <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, >> <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, <array([[ >> 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>], [None, None, None, None, None, None, >> None, None, None, None, None, None, None, None, None, None, None, None, >> None, None, None, None, None, None, None, None, None, None, None, None, >> None, None, None, None, None, None, None, None, None, None, None, None, >> None, None, None, None, None, None, None, None, None, None, None, None, >> None, None, None, None, None, None, None, None, array([[ 0., 0., 0., 0., >> 0.], >> >> >> aceback (most recent call last): >> File "/scratch/users/tjklein/215178/spark/python/pyspark/worker.py", line >> 101, in main >> process() >> File "/scratch/users/tjklein/215178/spark/python/pyspark/worker.py", line >> 96, in process >> serializer.dump_stream(func(split_index, iterator), outfile) >> File "/scratch/users/tjklein/215178/spark/python/pyspark/serializers.py", >> line 236, in dump_stream >> vs = list(itertools.islice(iterator, batch)) >> File "/home/tjklein/cnn-3d/spark_LORM_refactored.py", line 1046, in >> distributed_gradient_computation >> return broadcast_ADMM_gradient_function.value(*param_list) >> File "/scratch/users/tjklein/215178/spark/python/pyspark/broadcast.py", >> line 106, in value >> self._value = self.load(self._path) >> File "/scratch/users/tjklein/215178/spark/python/pyspark/broadcast.py", >> line 95, in load >> return cPickle.loads(data) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/compile/function_module.py", >> line 747, in _constructor_Function >> f = maker.create(input_storage, trustme = True) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/compile/function_module.py", >> line 1415, in create >> input_storage=input_storage_lists) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/link.py", >> line 525, in make_thunk >> output_storage=output_storage)[:3] >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/vm.py", line >> 897, in make_all >> no_recycling)) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/op.py", line >> 1002, in make_thunk >> compute_map, no_recycling) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/op.py", line >> 739, in make_thunk >> output_storage=node_output_storage) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", line >> 1072, in make_thunk >> keep_lock=keep_lock) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", line >> 1014, in __compile__ >> keep_lock=keep_lock) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", line >> 1441, in cthunk_factory >> key=key, lnk=self, keep_lock=keep_lock) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", >> line 1054, in module_from_key >> module = self._get_from_hash(module_hash, key, keep_lock=keep_lock) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", >> line 954, in _get_from_hash >> module = self._get_from_key(None, key_data) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", >> line 949, in _get_from_key >> return self._get_module(name) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", >> line 632, in _get_module >> self.module_from_name[name] = dlimport(name) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", >> line 293, in dlimport >> rval = __import__(module_name, {}, {}, [module_name]) >> ImportError: (ImportError('The following error happened while compiling the >> node', Elemwise{add,no_inplace}(TensorConstant{(1,) of -1}, >> AdvancedSubtensor1.0), '\n', 'No module named >> tmpuawGWX.7ecbb18ed585719993b1efa9e2a60ff5'), <function >> _constructor_Function at 0x2aaacc3570c8>, >> (<theano.compile.function_module.FunctionMaker object at 0x2aab2a314110>, >> [<None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, >> <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, >> <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, >> <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, >> <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, >> <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, >> <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, <array([[ >> 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >> [ 1., 0., 0., 0., 0.], >> [ 1., 1., 0., 0., 0.], >> [ 1., 1., 1., 0., 0.], >> [ 1., 1., 1., 1., 0.], >> [ 1., 1., 1., 1., 1.]])>] >> >> >> 15/03/02 18:56:42 WARN TaskSetManager: Lost task 56.1 in stage 7.19 (TID >> 17719, node004.cm.cluster): org.apache.spark.api.python.PythonException: >> Traceback (most recent call last): >> File "/scratch/users/tjklein/215178/spark/python/pyspark/worker.py", line >> 101, in main >> process() >> File "/scratch/users/tjklein/215178/spark/python/pyspark/worker.py", line >> 96, in process >> serializer.dump_stream(func(split_index, iterator), outfile) >> File "/scratch/users/tjklein/215178/spark/python/pyspark/serializers.py", >> line 236, in dump_stream >> vs = list(itertools.islice(iterator, batch)) >> File "/home/tjklein/cnn-3d/spark_LORM_refactored.py", line 1046, in >> distributed_gradient_computation >> return broadcast_ADMM_gradient_function.value(*param_list) >> File "/scratch/users/tjklein/215178/spark/python/pyspark/broadcast.py", >> line 106, in value >> self._value = self.load(self._path) >> File "/scratch/users/tjklein/215178/spark/python/pyspark/broadcast.py", >> line 95, in load >> return cPickle.loads(data) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/compile/function_module.py", >> line 747, in _constructor_Function >> f = maker.create(input_storage, trustme = True) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/compile/function_module.py", >> line 1415, in create >> input_storage=input_storage_lists) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/link.py", >> line 525, in make_thunk >> output_storage=output_storage)[:3] >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/vm.py", line >> 897, in make_all >> no_recycling)) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/op.py", line >> 1002, in make_thunk >> compute_map, no_recycling) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/op.py", line >> 739, in make_thunk >> output_storage=node_output_storage) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", line >> 1072, in make_thunk >> keep_lock=keep_lock) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", line >> 1014, in __compile__ >> keep_lock=keep_lock) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", line >> 1441, in cthunk_factory >> key=key, lnk=self, keep_lock=keep_lock) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", >> line 1045, in module_from_key >> module = self._get_from_key(key) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", >> line 949, in _get_from_key >> return self._get_module(name) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", >> line 632, in _get_module >> self.module_from_name[name] = dlimport(name) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", >> line 293, in dlimport >> rval = __import__(module_name, {}, {}, [module_name]) >> ImportError: (ImportError('The following error happened while compiling the >> node', Elemwise{add,no_inplace}(TensorConstant{(1,) of -1}, >> AdvancedSubtensor1.0), '\n', 'No module named >> tmpuawGWX.7ecbb18ed585719993b1efa9e2a60ff5'), < >> >> >> 15/03/02 18:57:02 INFO TaskSetManager: Lost task 51.1 in stage 7.19 (TID >> 17724) on executor node004.cm.cluster: >> org.apache.spark.api.python.PythonException (Traceback (most recent call >> last): >> File "/scratch/users/tjklein/215178/spark/python/pyspark/worker.py", line >> 101, in main >> process() >> File "/scratch/users/tjklein/215178/spark/python/pyspark/worker.py", line >> 96, in process >> serializer.dump_stream(func(split_index, iterator), outfile) >> File "/scratch/users/tjklein/215178/spark/python/pyspark/serializers.py", >> line 236, in dump_stream >> vs = list(itertools.islice(iterator, batch)) >> File "/home/tjklein/cnn-3d/spark_LORM_refactored.py", line 1046, in >> distributed_gradient_computation >> return broadcast_ADMM_gradient_function.value(*param_list) >> File "/scratch/users/tjklein/215178/spark/python/pyspark/broadcast.py", >> line 106, in value >> self._value = self.load(self._path) >> File "/scratch/users/tjklein/215178/spark/python/pyspark/broadcast.py", >> line 95, in load >> return cPickle.loads(data) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/compile/function_module.py", >> line 747, in _constructor_Function >> f = maker.create(input_storage, trustme = True) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/compile/function_module.py", >> line 1415, in create >> input_storage=input_storage_lists) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/link.py", >> line 525, in make_thunk >> output_storage=output_storage)[:3] >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/vm.py", line >> 897, in make_all >> no_recycling)) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/op.py", line >> 739, in make_thunk >> output_storage=node_output_storage) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", line >> 1072, in make_thunk >> keep_lock=keep_lock) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", line >> 1014, in __compile__ >> keep_lock=keep_lock) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", line >> 1441, in cthunk_factory >> key=key, lnk=self, keep_lock=keep_lock) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", >> line 1054, in module_from_key >> module = self._get_from_hash(module_hash, key, keep_lock=keep_lock) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", >> line 957, in _get_from_hash >> key_data.add_key(key, save_pkl=bool(key[0])) >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", >> line 467, in add_key >> self.save_pkl() >> File >> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", >> line 484, in save_pkl >> with open(self.key_pkl, 'wb') as f: >> IOError: [Errno [Errno 2] No such file or directory: >> '/scratch/users/tjklein/215178/theano/compiledir_Linux-2.6-el6.x86_64-x86_64-with-redhat-6.6-Santiago-x86_64-2.7.9-64/tmpQrM4nE/key.pkl'] >> >> >> >> On Mon, Mar 2, 2015 at 11:21 AM, Frédéric Bastien <[email protected] >> <javascript:>> wrote: >> >>> Hi, >>> >>> Can you give the full stack trace of the error? >>> >>> Do you use the same compiledir for the master and all process when it is >>> on the same compute node? >>> >>> Fred >>> >>> On Sun, Mar 1, 2015 at 5:17 PM, Tassilo Klein <[email protected] >>> <javascript:>> wrote: >>> >>>> Hi Fred, >>>> >>>> I got the latest version of Theano from the repository. I ran it on a >>>> single machine and also tried it on a distributed cluster. But still have >>>> the same issues as before, in both scenarios. >>>> >>>> I get many: >>>> >>>> INFO (theano.gof.compilelock): To manually release the lock, delete >>>> /scratch/users/tjklein/STANDALONE/theano/compiledir_Linux-2.6-el6.x86_64-x86_64-with-redhat-6.6-Santiago-x86_64-2.7.9-64/lock_dir >>>> >>>> INFO (theano.gof.compilelock): Waiting for existing lock by process >>>> '252635' (I am process '249796') >>>> >>>> INFO (theano.gof.compilelock): To manually release the lock, delete >>>> /scratch/users/tjklein/STANDALONE/theano/compiledir_Linux-2.6-el6.x86_64-x86_64-with-redhat-6.6-Santiago-x86_64-2.7.9-64/lock_dir >>>> >>>> INFO (theano.gof.compilelock): Waiting for existing lock by process >>>> '252635' (I am process '238059') >>>> >>>> INFO (theano.gof.compilelock): To manually release the lock, delete >>>> /scratch/users/tjklein/STANDALONE/theano/compiledir_Linux-2.6-el6.x86_64-x86_64-with-redhat-6.6-Santiago-x86_64-2.7.9-64/lock_dir >>>> >>>> INFO (theano.gof.compilelock): Waiting for existing lock by process >>>> '252635' (I am process '237960') >>>> >>>> INFO (theano.gof.compilelock): To manually release the lock, delete >>>> /scratch/users/tjklein/STANDALONE/theano/compiledir_Linux-2.6-el6.x86_64-x86_64-with-redhat-6.6-Santiago-x86_64-2.7.9-64/lock_dir >>>> >>>> INFO (theano.gof.compilelock): Waiting for existing lock by process >>>> '252635' (I am process '238179') >>>> >>>> INFO (theano.gof.compilelock): To manually release the lock, delete >>>> /scratch/users/tjklein/STANDALONE/theano/compiledir_Linux-2.6-el6.x86_64-x86_64-with-redhat-6.6-Santiago-x86_64-2.7.9-64/lock_dir >>>> >>>> INFO (theano.gof.compilelock): Waiting for existing lock by process >>>> '252635' (I am process '247811') >>>> >>>> INFO (theano.gof.compilelock): To manually release the lock, delete >>>> /scratch/users/tjklein/STANDALONE/theano/compiledir_Linux-2.6-el6.x86_64-x86_64-with-redhat-6.6-Santiago-x86_64-2.7.9-64/lock_dir >>>> >>>> INFO (theano.gof.compilelock): Waiting for existing lock by process >>>> '252635' (I am process '237684') >>>> >>>> INFO (theano.gof.compilelock): To manually release the lock, delete >>>> /scratch/users/tjklein/STANDALONE/theano/compiledir_Linux-2.6-el6.x86_64-x86_64-with-redhat-6.6-Santiago-x86_64-2.7.9-64/lock_dir >>>> >>>> INFO (theano.gof.compilelock): Waiting for existing lock by process >>>> '252635' (I am process '251284') >>>> >>>> and >>>> >>>> IOError: [Errno [Errno 2] No such file or directory: >>>> '/scratch/users/STANDALONE/theano/compiledir_Linux-2.6-el6.x86_64-x86_64-with-redhat-6.6-Santiago-x86_64-2.7.9-64/tmptlW68q/key.pkl'] >>>> >>>> <function _constructor_Function at 0x2aaacc1b7b90>: >>>> (<theano.compile.function_module.FunctionMaker object at 0x2aab24139b50>, >>>> >>>> >>>> >>>> Let me know if you want me to try out something else. >>>> >>>> Best, >>>> >>>> Tassilo >>>> >>>> On Wed, Feb 25, 2015 at 5:39 PM, Frédéric Bastien < >>>> [email protected] <javascript:>> wrote: >>>> >>>>> This could cause duplicate compilation and warning related to that. >>>>> Can you try with up to date Theano without your work around? >>>>> Le 22 févr. 2015 09:16, "György Solymosi" <[email protected] >>>>> <javascript:>> a écrit : >>>>> >>>>>> Hi Fred, >>>>>> >>>>>> Sorry to jump in, but I had some experiences with the same problem, >>>>>> so I simply switched off the lock and I had no issues -perhaps >>>>>> seemingly- >>>>>> with multiprocessing some -even 4 another- Theano tasks. Is it possible >>>>>> or >>>>>> it was just an illusion and it generated some issues I didn't see in >>>>>> background? >>>>>> >>>>>> George >>>>>> >>>>>> -- >>>>>> >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "theano-users" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected] <javascript:>. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> -- >>>>> >>>>> --- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "theano-users" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/theano-users/Pi4zQpfn5Ts/unsubscribe >>>>> . >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> [email protected] <javascript:>. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>>> >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "theano-users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected] <javascript:>. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> >>> --- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "theano-users" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/theano-users/Pi4zQpfn5Ts/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected] <javascript:>. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> > -- --- You received this message because you are subscribed to the Google Groups "theano-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
