I set CUDA_ROOT =/usr/local/cuda-8 LD_PATH=/usr/local/cuda-8/lib64:... PATH=/usr/local/cuda-8/bin: ... delete .theano dir and ran my program. I get the errors as shown in the attached file. The error takes place when theano.function() is called. My .theanorc file looks like -------------------- [global] floatX = float32 device = gpu exception_verbosity = high
[nvcc] optimizer_including=alloc_empty_to_zeros use_fast_math=False [lib] cnmem = 1 #gpuarray.preallocate = 1 ------------------------ I am using old backend, because it seems Lasagne is not compatible with new backend. What am I doing wrong? On Wednesday, July 5, 2017 at 6:02:58 PM UTC-4, nouiz wrote: > > You are mixing cuda 7.5 and 8 in your end variable. Never do this. Delete > your Theano cache. It is not safe to reuse it. > > The new backend support cuda 8. Your warning is about cudnn version. This > is not the same. > > If in a clean environment,you still have problems, give the error you have. > > Fred > > Le mer. 5 juil. 2017 08:51, <[email protected] <javascript:>> a > écrit : > >> I found out that it is use CPU by running the program on a machine where >> all GUPs are already in use. >> On our setup if all GPUs are already in use the 'import theano' causes >> the program to crash. >> When I remove the * from cuda* it does use GPU, but then the theano >> functions >> do not compile and dumps long header files and error descriptions on the >> screen. >> My suspicion is that this may be due to the fact I am using Lasagne >> libarary that >> seems not to be compatible with the new backend. >> I am OK using the old backend but it does not work with cuda-8. import >> theano >> gives me the following warning >> ------------------------------- >> Using gpu device 0: GeForce GTX 1080 Ti (CNMeM is enabled with initial >> size: 95.0% of memory, cuDNN 6021) >> /var/local/miniconda2/lib/python2.7/site-packages/theano/sandbox/cuda/__init__.py:631: >> >> UserWarning: Your cuDNN version is more recent than the one Theano >> officially supports. If you see any problems, try updating Theano or >> downgrading cuDNN to version 5.1. >> -------------------------------- >> And once again theano functions do not compile. Long error messages with >> some header files are dumped as part of error. >> However when I the following settings >> >> >> CUDA_ROOT =/usr/local/cuda-7.5 >> LD_PATH=/usr/local/cuda-8/lib64:... >> PATH=/usr/local/cuda-8/bin: ... >> >> Thing work but after certain number of training epochs (reducing the >> error nicely) >> all training parameters suddenly become nan. I have a complex network >> using 4 Bi-GRUs, Conv1DLayer, >> MaxPool1DLayer from lasagne and some attention layers I implemented. >> I have spent numerous hours trying to make sure gradients remain bounded, >> by using 'theano.gradient.grad_clip' >> at various stages of computation and by using >> 'lasagne.updates.norm_constraint' and still have >> not been able to pin down the cause of parameters suddenly becoming >> 'nan'. >> >> I just would like to be sure that this is not happening because of my >> usage of cuda-8 and cuda-7.5 at >> the same time. >> I appreciate your help in resolving this. >> >> >> >> >> On Monday, July 3, 2017 at 6:13:09 PM UTC-4, Pascal Lamblin wrote: >> >>> How did you determine it is using the CPU? >>> >>> On Monday, July 3, 2017 at 10:20:40 AM UTC-4, [email protected] >>> wrote: >>>> >>>> Changing it to cuda* results is CPU usage and not GPU >>>> >>>> On Friday, June 30, 2017 at 4:20:22 PM UTC-4, nouiz wrote: >>>>> >>>>> You should not mix cuda version... >>>>> >>>>> Do you still use the old gpu back-end (device=gpu*) or the new >>>>> back-end (device=cuda*)? >>>>> >>>>> Fred >>>>> >>>>> On Fri, Jun 30, 2017 at 9:57 AM <[email protected]> wrote: >>>>> >>>>>> I trying to understand some unexplained behavior of my code. >>>>>> To be sure that the problem is with my code and not with software >>>>>> incompatibility I would like to sure about the correctness of my setup >>>>>> I have: >>>>>> theano version 0.9 >>>>>> >>>>>> CUDA_ROOT =/usr/local/cuda-7.5 >>>>>> LD_PATH=/usr/local/cuda-8/lib64:... >>>>>> PATH=/usr/local/cuda-8/bin: ... >>>>>> >>>>>> Essentially I am using some parts of cuda-8 and some of cuda-7.5. >>>>>> >>>>>> With CUDA_ROOT =/usr/local/cuda-8, I cannot compile the theano >>>>>> functions. >>>>>> >>>>>> Thanks >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> ******************************************************************************* >>>>>> >>>>>> This e-mail and any of its attachments may contain Interactions >>>>>> Corporation proprietary information, which is privileged, confidential, >>>>>> or >>>>>> subject to copyright belonging to the Interactions Corporation. This >>>>>> e-mail >>>>>> is intended solely for the use of the individual or entity to which it >>>>>> is >>>>>> addressed. If you are not the intended recipient of this e-mail, you are >>>>>> hereby notified that any dissemination, distribution, copying, or action >>>>>> taken in relation to the contents of and attachments to this e-mail is >>>>>> strictly prohibited and may be unlawful. If you have received this >>>>>> e-mail >>>>>> in error, please notify the sender immediately and permanently delete >>>>>> the >>>>>> original and any copy of this e-mail and any printout. Thank You. >>>>>> >>>>>> >>>>>> ******************************************************************************* >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "theano-users" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >> >> ******************************************************************************* >> >> This e-mail and any of its attachments may contain Interactions >> Corporation proprietary information, which is privileged, confidential, or >> subject to copyright belonging to the Interactions Corporation. This e-mail >> is intended solely for the use of the individual or entity to which it is >> addressed. If you are not the intended recipient of this e-mail, you are >> hereby notified that any dissemination, distribution, copying, or action >> taken in relation to the contents of and attachments to this e-mail is >> strictly prohibited and may be unlawful. If you have received this e-mail >> in error, please notify the sender immediately and permanently delete the >> original and any copy of this e-mail and any printout. Thank You. >> >> >> ******************************************************************************* >> >> >> >> -- >> >> --- >> You received this message because you are subscribed to the Google Groups >> "theano-users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > -- ******************************************************************************* This e-mail and any of its attachments may contain Interactions Corporation proprietary information, which is privileged, confidential, or subject to copyright belonging to the Interactions Corporation. This e-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this e-mail is strictly prohibited and may be unlawful. If you have received this e-mail in error, please notify the sender immediately and permanently delete the original and any copy of this e-mail and any printout. Thank You. ******************************************************************************* -- --- You received this message because you are subscribed to the Google Groups "theano-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
error
Description: Binary data
