I figured out what happened in my case, it might or might not apply to others. starting nvidia-smi will slow down gpu computation on a headless (no monitor, no X11 running) GPU server (a reference https://devtalk.nvidia.com/default/topic/482848/multiple-gpus-listing-devices-very-slow/)
and at the beginning, I was watching GPU usage using watch-n 0.1 nvidia-smi which caused the slowness and once I switch to nvidia-smi -l 1 It is under normal speed. On Friday, 12 May 2017 00:20:27 UTC-4, nouiz wrote: > > Try to use Theano profiling to know where the time is spent. > > Le jeu. 11 mai 2017 15:48, Gavin Weiguang Ding <[email protected] > <javascript:>> a écrit : > >> hi daksh, >> >> have you figured out the difference? i have the same issue here. >> >> Thanks, >> Gavin >> >> >> On Friday, 17 February 2017 02:37:00 UTC-5, Daksh Varshneya wrote: >>> >>> Hi, >>> >>> I have a piece of code which basically trains a GRU to learn from some >>> sequences. I am training the neural network on each individual >>> sequence(batch size = 1). When I am running the training code on my local >>> machine which has a Nvidia gtx 960m, it takes about 3.09 secs for ~ 10 data >>> points. But when I run the same code on aws ec2 p2.xlarge machine which has >>> a Nvidia Tesla K80 GPU, surprisingly the time taken for exactly same >>> sequences is around 4.93 seconds. This is very disheartening right now. I >>> have tried switching AMI's as well. Originally I used a clean install of >>> ubuntu 14.04 and installed cuda, cudnn and theano myself. The second image >>> I took from amazon marketplace which had all these things installed. >>> The code is exactly the same with no difference at all. Any pointers >>> which can help? >>> >>> Thanks, >>> Daksh >>> >> -- >> >> --- >> You received this message because you are subscribed to the Google Groups >> "theano-users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > -- --- You received this message because you are subscribed to the Google Groups "theano-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
