Hi, I've been facing memory issues for the past few weeks with my code. I have a few general doubts/requests regarding the working of Theano -
1. How do I find the number of parameters for my model? It is a model with custom designed layers, and built on top of the 3D-R2N2 (https://github.com/chrischoy/3D-R2N2/) In these some pre-existing function (like model.count_params() in Keras), or could someone help me out in writing it? I have seen the "pydotprint" function, but, I need something that is easier to understand like the count_params() or model.summary() in Keras. 2. I need to find out the GPU usage of my model. I read somewhere that Theano allocates the whole of the GPU memory for the process, at first, and then scales down depending on usage. How does it exactly work? 3. Based on the tests I ran on the R2N2 code, it "appears" as though my WHOLE dataset gets loaded into the GPU memory and then it splits into batches for training and validation. Since my dataset is huge, this is highly inefficient and leads to an error requesting for more memory. Although it my intuition that this is what is happening, I am unable to prove it. Based on the documentation, it is only the "theano.shared", "gpuArray" and certain frequently accessed variables that go into the GPU memory. The size of my weights file is about 250 MB, and I am using a 11GB 1080Ti to train the model. I shouldn't be getting memory allocation errors. If I can print the params, and find out how much of the data is occupied by them, I can prove that it is indeed my data that is getting loaded into the GPU memory and is causing issues. 4. Moreover, can someone tell me that if it is customary for theano to load ALL of the the data (from my dataset) into the GPU memory? If so, how do I avoid it? I don't mind slowing it down by making multiple disk I/Os, as long as it fits in the memory and runs. 5. In Theano, specifically, and in any library, in general, what goes into the system RAM, and what into the GPU RAM? My system RAM usage for the above seems to be close to 50GB. While that is not a bottleneck now, I'd like to reduce it, all the same. I know I'm asking a lot. But, I couldn't find any resource that explains all of the above. Thanks, Abbhinav P.S: I'm posting a snapshot of my memory error here: https://pastebin.com/FJy11p3s -- --- You received this message because you are subscribed to the Google Groups "theano-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
