I found out that it is use CPU by running the program on a machine where all GUPs are already in use. On our setup if all GPUs are already in use the 'import theano' causes the program to crash. When I remove the * from cuda* it does use GPU, but then the theano functions do not compile and dumps long header files and error descriptions on the screen. My suspicion is that this may be due to the fact I am using Lasagne libarary that seems not to be compatible with the new backend. I am OK using the old backend but it does not work with cuda-8. import theano gives me the following warning ------------------------------- Using gpu device 0: GeForce GTX 1080 Ti (CNMeM is enabled with initial size: 95.0% of memory, cuDNN 6021) /var/local/miniconda2/lib/python2.7/site-packages/theano/sandbox/cuda/__init__.py:631: UserWarning: Your cuDNN version is more recent than the one Theano officially supports. If you see any problems, try updating Theano or downgrading cuDNN to version 5.1. -------------------------------- And once again theano functions do not compile. Long error messages with some header files are dumped as part of error. However when I the following settings
CUDA_ROOT =/usr/local/cuda-7.5 LD_PATH=/usr/local/cuda-8/lib64:... PATH=/usr/local/cuda-8/bin: ... Thing work but after certain number of training epochs (reducing the error nicely) all training parameters suddenly become nan. I have a complex network using 4 Bi-GRUs, Conv1DLayer, MaxPool1DLayer from lasagne and some attention layers I implemented. I have spent numerous hours trying to make sure gradients remain bounded, by using 'theano.gradient.grad_clip' at various stages of computation and by using 'lasagne.updates.norm_constraint' and still have not been able to pin down the cause of parameters suddenly becoming 'nan'. I just would like to be sure that this is not happening because of my usage of cuda-8 and cuda-7.5 at the same time. I appreciate your help in resolving this. On Monday, July 3, 2017 at 6:13:09 PM UTC-4, Pascal Lamblin wrote: > > How did you determine it is using the CPU? > > On Monday, July 3, 2017 at 10:20:40 AM UTC-4, [email protected] > <javascript:> wrote: >> >> Changing it to cuda* results is CPU usage and not GPU >> >> On Friday, June 30, 2017 at 4:20:22 PM UTC-4, nouiz wrote: >>> >>> You should not mix cuda version... >>> >>> Do you still use the old gpu back-end (device=gpu*) or the new back-end >>> (device=cuda*)? >>> >>> Fred >>> >>> On Fri, Jun 30, 2017 at 9:57 AM <[email protected]> wrote: >>> >>>> I trying to understand some unexplained behavior of my code. >>>> To be sure that the problem is with my code and not with software >>>> incompatibility I would like to sure about the correctness of my setup >>>> I have: >>>> theano version 0.9 >>>> >>>> CUDA_ROOT =/usr/local/cuda-7.5 >>>> LD_PATH=/usr/local/cuda-8/lib64:... >>>> PATH=/usr/local/cuda-8/bin: ... >>>> >>>> Essentially I am using some parts of cuda-8 and some of cuda-7.5. >>>> >>>> With CUDA_ROOT =/usr/local/cuda-8, I cannot compile the theano >>>> functions. >>>> >>>> Thanks >>>> >>>> >>>> >>>> >>>> ******************************************************************************* >>>> >>>> This e-mail and any of its attachments may contain Interactions >>>> Corporation proprietary information, which is privileged, confidential, or >>>> subject to copyright belonging to the Interactions Corporation. This >>>> e-mail >>>> is intended solely for the use of the individual or entity to which it is >>>> addressed. If you are not the intended recipient of this e-mail, you are >>>> hereby notified that any dissemination, distribution, copying, or action >>>> taken in relation to the contents of and attachments to this e-mail is >>>> strictly prohibited and may be unlawful. If you have received this e-mail >>>> in error, please notify the sender immediately and permanently delete the >>>> original and any copy of this e-mail and any printout. Thank You. >>>> >>>> >>>> ******************************************************************************* >>>> >>>> >>>> >>>> -- >>>> >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "theano-users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> -- ******************************************************************************* This e-mail and any of its attachments may contain Interactions Corporation proprietary information, which is privileged, confidential, or subject to copyright belonging to the Interactions Corporation. This e-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this e-mail is strictly prohibited and may be unlawful. If you have received this e-mail in error, please notify the sender immediately and permanently delete the original and any copy of this e-mail and any printout. Thank You. ******************************************************************************* -- --- You received this message because you are subscribed to the Google Groups "theano-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
