lixiangchun opened a new issue #8024: Memory usage increase significantly in applying trained model to multiple rec files URL: https://github.com/apache/incubator-mxnet/issues/8024 I ran the following code and observed that memory usage increased gradually until all of it was used out (total RAM is 64Gb), and the program stop without any error. Any comments are appreciated. ```python from __future__ import print_function import glob import sys import os import mxnet as mx batch_size=64 ctx=mx.gpu(1) resnet_model = get_checkpoint('resnet18',32, image_size=299, batch_size = batch_size, ctx=ctx) fls = glob.glob('../rec/*.rec') # length(fls) = 1269 for fl in fls: p = predict_on_rec_v1(resnet_model, path_imgrec=fl, ctx=ctx, batch_size=batch_size) ``` When I applied the trained model to only 1 rec file, the memory usage is just about 3 Gb. ```python resnet_model = get_checkpoint('resnet18',32, image_size=299, batch_size = batch_size, ctx=ctx) p = predict_on_rec_v1(resnet_model, path_imgrec=fls[0], ctx=ctx, batch_size=batch_size) ``` Code for get_checkpoint: ```python def get_checkpoint(prefix, epoch, ctx=mx.gpu(0), batch_size=32, image_size=299): height, width = image_size, image_size channels = 3 sym, arg_params, aux_params = mx.model.load_checkpoint(prefix, epoch) mod = mx.mod.Module(symbol=sym, context=ctx, label_names=None) mod.bind(for_training=False, data_shapes=[('data', (batch_size, channels, height, width))], label_shapes=mod._label_shapes) mod.set_params(arg_params, aux_params, allow_missing=True) return mod ``` Code for predict_on_rec_v1: ```python def predict_on_rec_v1(model=None, path_imgrec=None, p_cutoff=0.9, ctx=mx.gpu(0), batch_size=32, data_shape=(3,299,299)): classIdx2label = {0:'cancer',1:'normal'} if path_imgrec is not None: data_iter = mx.io.ImageRecordIter(path_imgrec=path_imgrec, data_shape=data_shape, round_batch=False, \ batch_size=batch_size, mean_r=123.68, mean_g=116.779, mean_b=103.939, shuffle=False) # set round_batch=False p = model.predict(data_iter) p = p.asnumpy() k_cancer, k_normal = sum(p[:,0] > p_cutoff), sum(p[:,1] > p_cutoff) print("cancer vs. normal, {}, {} vs. {}.".format(os.path.basename(path_imgrec), \ k_cancer, k_normal), file=sys.stderr) return p else: raise ValueError("path_imgrec is None.") return None ``` OS: ubuntu 16.04 mxnet package: mxnet_cu80-0.11.1b20170910-py2.py3-none-manylinux1_x86_64.whl ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
With regards, Apache Git Services
