[GitHub] KaiserSozo opened a new issue #11101: Gluon Performance and memory conumption

[GitBox]

KaiserSozo opened a new issue #11101: Gluon Performance and memory conumption
URL: https://github.com/apache/incubator-mxnet/issues/11101
 
 
   Working under gpu, I have next code:
   
   for i, (data) in enumerate(trainingInputs):
                   calcT = time.time()
                   data = data.as_in_context(ctx)
                   output, win_index, delta, mask = netSom(data)
                   calc += time.time() - calcT
                   copyT = time.time()            
                   weightsData = weights.data()
                   ratesData = rates.data()
                   ratesData[win_index] += 1
                   weightsData[win_index] += delta
                   ratesData.wait_to_read()
                   weightsData.wait_to_read()
                   train_accuracy += output.asscalar()
                   copy += time.time() - copyT
   
   Calculation time that is in calc variable is 5 times less than copy time 
that is in copy variable. Why o an how it can be reduced?
   Also I noted that if I remove calling of wait_to_read() function then copy 
time is 0, but memory consumption always increasing and leads to memory 
allocation failure. And almost the same behaviour I see in next code using 
gluon:
   
   for data, label in itertools.izip(trainingInputs, trainingOutputs):
               calcT = time.time()
               data = data.as_in_context(ctx)
               label = label.as_in_context(ctx)
               output, win_index, delta, mask = netSom(data)
               data = data.reshape((-1,inputsCount))
               with autograd.record():
                   args = (data, mask)
                   output = net(*args)
                   l2loss = loss(output, label)
               l2loss.backward()
               calc += time.time() - calcT
               copyT = time.time()    
               trainer.step(data.shape[0])
               copy += time.time() - copyT
               i+=1
   
           testT = time.time()
           test_accuracy = evaluate_accuracyMLP(testInputs, testOutputs, net, 
netSom, inputsCount, activeNeuronsCount)
           test += time.time() - testT
   
   Here calculation and copying (gradient adjusting) times are almost equal 
each other. And I also searching the way of decreasing copy time, it should be 
serious less than calculation time. And here also persists strange behaviour, 
if I remove 'evaluate_accuracyMLP' call, memory consumption become increase 
till error.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services