thomasmooon opened a new issue #7475: Paradox VRAM demand as a function of batch size: Low batch size, high VRAM demand URL: https://github.com/apache/incubator-mxnet/issues/7475 Dear community, I'm running mxnet on the environment as mentioned below with the following hardware: - 32 GB RAM - 8 x NVIDIA 1080 Ti, 8 GB VRAM - 12 CPUs MXnet is utlized to train a stacked denoising autoencoder reconstruction of an array with dimension about 62.000 x 100. I was looking for a batch size with minimum VRAM demand. Therefore I measured the VRAM demand per single Card for a given combination of "number of GPUs" and "array.batch.size". The results are shown in the attached table: Columns indicate array.batch.size and rows indicate number of GPUs. Each entry indicates the GPU memory usage per Card.  As one can see if array.batch.size is below or above a particular value, VRAM demand rises. Given an array.batch.size of 12, computation crashes due to "cuda memory allocation error" (red style). Given a array.batch.size of 128 (2 and 4 GPUS) or 256 (6) results in a minimum VRAM demand of about 3GB per card (green style). My question is: **Why does VRAM hunger rise below a particular array.batch.size value?** ## Environment info Operating System: REDHAT 7.3 Package used (Python/R/Scala/Julia): R MXNet version: 0.1.0.1 R `sessionInfo()`: R version 3.4.0 (2017-04-21) Platform: x86_64-redhat-linux-gnu (64-bit) Running under: Red Hat Enterprise Linux Server 7.3 (Maipo) Matrix products: default BLAS/LAPACK: /usr/lib64/R/lib/libRblas.so locale: [1] LC_CTYPE=en_GB.UTF-8 LC_NUMERIC=C LC_TIME=en_GB.UTF-8 LC_COLLATE=en_GB.UTF-8 LC_MONETARY=en_GB.UTF-8 LC_MESSAGES=en_GB.UTF-8 LC_PAPER=en_GB.UTF-8 [8] LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=en_GB.UTF-8 LC_IDENTIFICATION=C attached base packages: [1] stats graphics grDevices utils datasets methods base loaded via a namespace (and not attached): [1] compiler_3.4.0 tools_3.4.0 ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
With regards, Apache Git Services
