Thank you anyway. 在 2016年8月5日星期五 UTC-4下午6:24:07，Tassilo Klein写道： > > Unless something was dramatically changed in the architecture, I am afraid > there is no solution to this the problem. I had to change the framework to > get it working. > > On Fri, Aug 5, 2016 at 10:30 PM, <[email protected] <javascript:>> wrote: > >> Hi, Tassilo, >> Have you solved this problem? >> I encounter the same problem at present without any idea for it. >> I just run some separate programs in the same time then the function >> compiling outputs the similar problem. >> Thanks a lot for any idea. >> >> 在 2015年3月9日星期一 UTC-4下午1:51:30，Tassilo Klein写道： >>> >>> Hi Fred, >>> >>> I just wanted to follow up on this issue as I am sort of deadlocked with >>> this problem now. Do you see any solution? >>> Btw, the non-thread safety issue of Theano has not been fixed yet, >>> right? Because it sounds somewhat related to that. >>> >>> -Tassilo >>> >>> On Mon, Mar 2, 2015 at 7:07 PM, Tassilo Klein <[email protected]> wrote: >>> >>>> Hi Fred, >>>> >>>> yes, master and slave have the same directory. Sometimes it works >>>> better, sometimes worse - feels like a race condition. It is weird. Here >>>> are stack-traces for different error outputs. >>>> >>>> Cheers, >>>> Tassilo >>>> >>>> >>>> 15/03/02 18:55:57 WARN TaskSetManager: Lost task 64.0 in stage 7.19 (TID >>>> 17710, node004.cm.cluster): org.apache.spark.api.python.PythonException: >>>> Traceback (most recent call last): >>>> File "/scratch/users/tjklein/215178/spark/python/pyspark/worker.py", >>>> line 101, in main >>>> process() >>>> File "/scratch/users/tjklein/215178/spark/python/pyspark/worker.py", >>>> line 96, in process >>>> serializer.dump_stream(func(split_index, iterator), outfile) >>>> File >>>> "/scratch/users/tjklein/215178/spark/python/pyspark/serializers.py", line >>>> 236, in dump_stream >>>> vs = list(itertools.islice(iterator, batch)) >>>> File "/home/tjklein/cnn-3d/spark_LORM_refactored.py", line 1046, in >>>> distributed_gradient_computation >>>> return broadcast_ADMM_gradient_function.value(*param_list) >>>> File "/scratch/users/tjklein/215178/spark/python/pyspark/broadcast.py", >>>> line 106, in value >>>> self._value = self.load(self._path) >>>> File "/scratch/users/tjklein/215178/spark/python/pyspark/broadcast.py", >>>> line 95, in load >>>> return cPickle.loads(data) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/compile/function_module.py", >>>> line 747, in _constructor_Function >>>> f = maker.create(input_storage, trustme = True) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/compile/function_module.py", >>>> line 1415, in create >>>> input_storage=input_storage_lists) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/link.py", >>>> line 525, in make_thunk >>>> output_storage=output_storage)[:3] >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/vm.py", >>>> line 897, in make_all >>>> no_recycling)) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/op.py", >>>> line 1002, in make_thunk >>>> compute_map, no_recycling) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/op.py", >>>> line 739, in make_thunk >>>> output_storage=node_output_storage) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", >>>> line 1072, in make_thunk >>>> keep_lock=keep_lock) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", >>>> line 1014, in __compile__ >>>> keep_lock=keep_lock) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", >>>> line 1441, in cthunk_factory >>>> key=key, lnk=self, keep_lock=keep_lock) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", >>>> line 1076, in module_from_key >>>> module = lnk.compile_cmodule(location) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", >>>> line 1353, in compile_cmodule >>>> preargs=preargs) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", >>>> line 1889, in compile_str >>>> raise MissingGXX("g++ not available! We can't compile c code.") >>>> MissingGXX: (MissingGXX('The following error happened while compiling the >>>> node', Elemwise{neg,no_inplace}(y), '\n', "g++ not available! We can't >>>> compile c code.", '[Elemwise{neg,no_inplace}(y)]'), <function >>>> _constructor_Function at 0x2aaacc32d230>, >>>> (<theano.compile.function_module.FunctionMaker object at 0x2aab2a53ff10>, >>>> [<None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, >>>> <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, >>>> <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, >>>> <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, >>>> <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, >>>> <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, >>>> <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, <array([[ >>>> 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>], [None, None, None, None, None, None, >>>> None, None, None, None, None, None, None, None, None, None, None, None, >>>> None, None, None, None, None, None, None, None, None, None, None, None, >>>> None, None, None, None, None, None, None, None, None, None, None, None, >>>> None, None, None, None, None, None, None, None, None, None, None, None, >>>> None, None, None, None, None, None, None, None, array([[ 0., 0., 0., >>>> 0., 0.], >>>> >>>> >>>> aceback (most recent call last): >>>> File "/scratch/users/tjklein/215178/spark/python/pyspark/worker.py", >>>> line 101, in main >>>> process() >>>> File "/scratch/users/tjklein/215178/spark/python/pyspark/worker.py", >>>> line 96, in process >>>> serializer.dump_stream(func(split_index, iterator), outfile) >>>> File >>>> "/scratch/users/tjklein/215178/spark/python/pyspark/serializers.py", line >>>> 236, in dump_stream >>>> vs = list(itertools.islice(iterator, batch)) >>>> File "/home/tjklein/cnn-3d/spark_LORM_refactored.py", line 1046, in >>>> distributed_gradient_computation >>>> return broadcast_ADMM_gradient_function.value(*param_list) >>>> File "/scratch/users/tjklein/215178/spark/python/pyspark/broadcast.py", >>>> line 106, in value >>>> self._value = self.load(self._path) >>>> File "/scratch/users/tjklein/215178/spark/python/pyspark/broadcast.py", >>>> line 95, in load >>>> return cPickle.loads(data) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/compile/function_module.py", >>>> line 747, in _constructor_Function >>>> f = maker.create(input_storage, trustme = True) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/compile/function_module.py", >>>> line 1415, in create >>>> input_storage=input_storage_lists) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/link.py", >>>> line 525, in make_thunk >>>> output_storage=output_storage)[:3] >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/vm.py", >>>> line 897, in make_all >>>> no_recycling)) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/op.py", >>>> line 1002, in make_thunk >>>> compute_map, no_recycling) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/op.py", >>>> line 739, in make_thunk >>>> output_storage=node_output_storage) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", >>>> line 1072, in make_thunk >>>> keep_lock=keep_lock) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", >>>> line 1014, in __compile__ >>>> keep_lock=keep_lock) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", >>>> line 1441, in cthunk_factory >>>> key=key, lnk=self, keep_lock=keep_lock) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", >>>> line 1054, in module_from_key >>>> module = self._get_from_hash(module_hash, key, keep_lock=keep_lock) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", >>>> line 954, in _get_from_hash >>>> module = self._get_from_key(None, key_data) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", >>>> line 949, in _get_from_key >>>> return self._get_module(name) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", >>>> line 632, in _get_module >>>> self.module_from_name[name] = dlimport(name) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", >>>> line 293, in dlimport >>>> rval = __import__(module_name, {}, {}, [module_name]) >>>> ImportError: (ImportError('The following error happened while compiling >>>> the node', Elemwise{add,no_inplace}(TensorConstant{(1,) of -1}, >>>> AdvancedSubtensor1.0), '\n', 'No module named >>>> tmpuawGWX.7ecbb18ed585719993b1efa9e2a60ff5'), <function >>>> _constructor_Function at 0x2aaacc3570c8>, >>>> (<theano.compile.function_module.FunctionMaker object at 0x2aab2a314110>, >>>> [<None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, >>>> <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, >>>> <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, >>>> <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, >>>> <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, >>>> <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, >>>> <None>, <None>, <None>, <None>, <None>, <None>, <None>, <None>, <array([[ >>>> 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>, <array([[ 0., 0., 0., 0., 0.], >>>> [ 1., 0., 0., 0., 0.], >>>> [ 1., 1., 0., 0., 0.], >>>> [ 1., 1., 1., 0., 0.], >>>> [ 1., 1., 1., 1., 0.], >>>> [ 1., 1., 1., 1., 1.]])>] >>>> >>>> >>>> 15/03/02 18:56:42 WARN TaskSetManager: Lost task 56.1 in stage 7.19 (TID >>>> 17719, node004.cm.cluster): org.apache.spark.api.python.PythonException: >>>> Traceback (most recent call last): >>>> File "/scratch/users/tjklein/215178/spark/python/pyspark/worker.py", >>>> line 101, in main >>>> process() >>>> File "/scratch/users/tjklein/215178/spark/python/pyspark/worker.py", >>>> line 96, in process >>>> serializer.dump_stream(func(split_index, iterator), outfile) >>>> File >>>> "/scratch/users/tjklein/215178/spark/python/pyspark/serializers.py", line >>>> 236, in dump_stream >>>> vs = list(itertools.islice(iterator, batch)) >>>> File "/home/tjklein/cnn-3d/spark_LORM_refactored.py", line 1046, in >>>> distributed_gradient_computation >>>> return broadcast_ADMM_gradient_function.value(*param_list) >>>> File "/scratch/users/tjklein/215178/spark/python/pyspark/broadcast.py", >>>> line 106, in value >>>> self._value = self.load(self._path) >>>> File "/scratch/users/tjklein/215178/spark/python/pyspark/broadcast.py", >>>> line 95, in load >>>> return cPickle.loads(data) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/compile/function_module.py", >>>> line 747, in _constructor_Function >>>> f = maker.create(input_storage, trustme = True) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/compile/function_module.py", >>>> line 1415, in create >>>> input_storage=input_storage_lists) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/link.py", >>>> line 525, in make_thunk >>>> output_storage=output_storage)[:3] >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/vm.py", >>>> line 897, in make_all >>>> no_recycling)) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/op.py", >>>> line 1002, in make_thunk >>>> compute_map, no_recycling) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/op.py", >>>> line 739, in make_thunk >>>> output_storage=node_output_storage) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", >>>> line 1072, in make_thunk >>>> keep_lock=keep_lock) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", >>>> line 1014, in __compile__ >>>> keep_lock=keep_lock) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", >>>> line 1441, in cthunk_factory >>>> key=key, lnk=self, keep_lock=keep_lock) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", >>>> line 1045, in module_from_key >>>> module = self._get_from_key(key) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", >>>> line 949, in _get_from_key >>>> return self._get_module(name) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", >>>> line 632, in _get_module >>>> self.module_from_name[name] = dlimport(name) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", >>>> line 293, in dlimport >>>> rval = __import__(module_name, {}, {}, [module_name]) >>>> ImportError: (ImportError('The following error happened while compiling >>>> the node', Elemwise{add,no_inplace}(TensorConstant{(1,) of -1}, >>>> AdvancedSubtensor1.0), '\n', 'No module named >>>> tmpuawGWX.7ecbb18ed585719993b1efa9e2a60ff5'), < >>>> >>>> >>>> 15/03/02 18:57:02 INFO TaskSetManager: Lost task 51.1 in stage 7.19 (TID >>>> 17724) on executor node004.cm.cluster: >>>> org.apache.spark.api.python.PythonException (Traceback (most recent call >>>> last): >>>> File "/scratch/users/tjklein/215178/spark/python/pyspark/worker.py", >>>> line 101, in main >>>> process() >>>> File "/scratch/users/tjklein/215178/spark/python/pyspark/worker.py", >>>> line 96, in process >>>> serializer.dump_stream(func(split_index, iterator), outfile) >>>> File >>>> "/scratch/users/tjklein/215178/spark/python/pyspark/serializers.py", line >>>> 236, in dump_stream >>>> vs = list(itertools.islice(iterator, batch)) >>>> File "/home/tjklein/cnn-3d/spark_LORM_refactored.py", line 1046, in >>>> distributed_gradient_computation >>>> return broadcast_ADMM_gradient_function.value(*param_list) >>>> File "/scratch/users/tjklein/215178/spark/python/pyspark/broadcast.py", >>>> line 106, in value >>>> self._value = self.load(self._path) >>>> File "/scratch/users/tjklein/215178/spark/python/pyspark/broadcast.py", >>>> line 95, in load >>>> return cPickle.loads(data) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/compile/function_module.py", >>>> line 747, in _constructor_Function >>>> f = maker.create(input_storage, trustme = True) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/compile/function_module.py", >>>> line 1415, in create >>>> input_storage=input_storage_lists) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/link.py", >>>> line 525, in make_thunk >>>> output_storage=output_storage)[:3] >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/vm.py", >>>> line 897, in make_all >>>> no_recycling)) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/op.py", >>>> line 739, in make_thunk >>>> output_storage=node_output_storage) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", >>>> line 1072, in make_thunk >>>> keep_lock=keep_lock) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", >>>> line 1014, in __compile__ >>>> keep_lock=keep_lock) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", >>>> line 1441, in cthunk_factory >>>> key=key, lnk=self, keep_lock=keep_lock) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", >>>> line 1054, in module_from_key >>>> module = self._get_from_hash(module_hash, key, keep_lock=keep_lock) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", >>>> line 957, in _get_from_hash >>>> key_data.add_key(key, save_pkl=bool(key[0])) >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", >>>> line 467, in add_key >>>> self.save_pkl() >>>> File >>>> "/home/tjklein/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", >>>> line 484, in save_pkl >>>> with open(self.key_pkl, 'wb') as f: >>>> IOError: [Errno [Errno 2] No such file or directory: >>>> '/scratch/users/tjklein/215178/theano/compiledir_Linux-2.6-el6.x86_64-x86_64-with-redhat-6.6-Santiago-x86_64-2.7.9-64/tmpQrM4nE/key.pkl'] >>>> >>>> >>>> >>>> On Mon, Mar 2, 2015 at 11:21 AM, Frédéric Bastien < >>>> [email protected]> wrote: >>>> >>>>> Hi, >>>>> >>>>> Can you give the full stack trace of the error? >>>>> >>>>> Do you use the same compiledir for the master and all process when it >>>>> is on the same compute node? >>>>> >>>>> Fred >>>>> >>>>> On Sun, Mar 1, 2015 at 5:17 PM, Tassilo Klein <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Fred, >>>>>> >>>>>> I got the latest version of Theano from the repository. I ran it on a >>>>>> single machine and also tried it on a distributed cluster. But still >>>>>> have >>>>>> the same issues as before, in both scenarios. >>>>>> >>>>>> I get many: >>>>>> >>>>>> INFO (theano.gof.compilelock): To manually release the lock, delete >>>>>> /scratch/users/tjklein/STANDALONE/theano/compiledir_Linux-2.6-el6.x86_64-x86_64-with-redhat-6.6-Santiago-x86_64-2.7.9-64/lock_dir >>>>>> >>>>>> INFO (theano.gof.compilelock): Waiting for existing lock by process >>>>>> '252635' (I am process '249796') >>>>>> >>>>>> INFO (theano.gof.compilelock): To manually release the lock, delete >>>>>> /scratch/users/tjklein/STANDALONE/theano/compiledir_Linux-2.6-el6.x86_64-x86_64-with-redhat-6.6-Santiago-x86_64-2.7.9-64/lock_dir >>>>>> >>>>>> INFO (theano.gof.compilelock): Waiting for existing lock by process >>>>>> '252635' (I am process '238059') >>>>>> >>>>>> INFO (theano.gof.compilelock): To manually release the lock, delete >>>>>> /scratch/users/tjklein/STANDALONE/theano/compiledir_Linux-2.6-el6.x86_64-x86_64-with-redhat-6.6-Santiago-x86_64-2.7.9-64/lock_dir >>>>>> >>>>>> INFO (theano.gof.compilelock): Waiting for existing lock by process >>>>>> '252635' (I am process '237960') >>>>>> >>>>>> INFO (theano.gof.compilelock): To manually release the lock, delete >>>>>> /scratch/users/tjklein/STANDALONE/theano/compiledir_Linux-2.6-el6.x86_64-x86_64-with-redhat-6.6-Santiago-x86_64-2.7.9-64/lock_dir >>>>>> >>>>>> INFO (theano.gof.compilelock): Waiting for existing lock by process >>>>>> '252635' (I am process '238179') >>>>>> >>>>>> INFO (theano.gof.compilelock): To manually release the lock, delete >>>>>> /scratch/users/tjklein/STANDALONE/theano/compiledir_Linux-2.6-el6.x86_64-x86_64-with-redhat-6.6-Santiago-x86_64-2.7.9-64/lock_dir >>>>>> >>>>>> INFO (theano.gof.compilelock): Waiting for existing lock by process >>>>>> '252635' (I am process '247811') >>>>>> >>>>>> INFO (theano.gof.compilelock): To manually release the lock, delete >>>>>> /scratch/users/tjklein/STANDALONE/theano/compiledir_Linux-2.6-el6.x86_64-x86_64-with-redhat-6.6-Santiago-x86_64-2.7.9-64/lock_dir >>>>>> >>>>>> INFO (theano.gof.compilelock): Waiting for existing lock by process >>>>>> '252635' (I am process '237684') >>>>>> >>>>>> INFO (theano.gof.compilelock): To manually release the lock, delete >>>>>> /scratch/users/tjklein/STANDALONE/theano/compiledir_Linux-2.6-el6.x86_64-x86_64-with-redhat-6.6-Santiago-x86_64-2.7.9-64/lock_dir >>>>>> >>>>>> INFO (theano.gof.compilelock): Waiting for existing lock by process >>>>>> '252635' (I am process '251284') >>>>>> >>>>>> and >>>>>> >>>>>> IOError: [Errno [Errno 2] No such file or directory: >>>>>> '/scratch/users/STANDALONE/theano/compiledir_Linux-2.6-el6.x86_64-x86_64-with-redhat-6.6-Santiago-x86_64-2.7.9-64/tmptlW68q/key.pkl'] >>>>>> >>>>>> <function _constructor_Function at 0x2aaacc1b7b90>: >>>>>> (<theano.compile.function_module.FunctionMaker object at >>>>>> 0x2aab24139b50>, >>>>>> >>>>>> >>>>>> >>>>>> Let me know if you want me to try out something else. >>>>>> >>>>>> Best, >>>>>> >>>>>> Tassilo >>>>>> >>>>>> On Wed, Feb 25, 2015 at 5:39 PM, Frédéric Bastien < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> This could cause duplicate compilation and warning related to that. >>>>>>> Can you try with up to date Theano without your work around? >>>>>>> Le 22 févr. 2015 09:16, "György Solymosi" <[email protected]> a >>>>>>> écrit : >>>>>>> >>>>>>>> Hi Fred, >>>>>>>> >>>>>>>> Sorry to jump in, but I had some experiences with the same problem, >>>>>>>> so I simply switched off the lock and I had no issues -perhaps >>>>>>>> seemingly- >>>>>>>> with multiprocessing some -even 4 another- Theano tasks. Is it >>>>>>>> possible or >>>>>>>> it was just an illusion and it generated some issues I didn't see in >>>>>>>> background? >>>>>>>> >>>>>>>> George >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> --- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "theano-users" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> -- >>>>>>> >>>>>>> --- >>>>>>> You received this message because you are subscribed to a topic in >>>>>>> the Google Groups "theano-users" group. >>>>>>> To unsubscribe from this topic, visit >>>>>>> https://groups.google.com/d/topic/theano-users/Pi4zQpfn5Ts/unsubscribe >>>>>>> . >>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>> [email protected]. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "theano-users" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>>>> >>>>> --- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "theano-users" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/theano-users/Pi4zQpfn5Ts/unsubscribe >>>>> . >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> [email protected]. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>> -- >> >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "theano-users" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/theano-users/Pi4zQpfn5Ts/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected] <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > >
-- --- You received this message because you are subscribed to the Google Groups "theano-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
