wkcn edited a comment on issue #18501:
URL:
https://github.com/apache/incubator-mxnet/issues/18501#issuecomment-643524295
Hi @leezu , I am not sure that it costs less than 3GB memory to build
indexing_op.o.
By building with ninja in multiple threads, It takes less than 12 GB memory.
However, it takes more than 16GB memory to build indexing_op.o by ‘make’ in
single thread. I need to confirm it.
I think the compiler flags may be different between original ‘Makefile’ and
‘CMakeList’.
I remember that the original Makefile will be deprecated . If so, it is not
a block issue.
I modify `indexing_op.cc` and re-build it.
```
Ninja
`cgmemtime ninja -v -j 1`
/usr/bin/ccache /usr/lib/ccache/bin/c++ -DDMLC_CORE_USE_CMAKE
-DDMLC_LOG_FATAL_THROW=1 -DDMLC_LOG_STACK_TRACE_SIZE=0
-DDMLC_MODERN_THREAD_LOCAL=0 -DDMLC_STRICT_CXX11 -DDMLC_USE_CXX11
-DDMLC_USE_CXX11=1 -DDMLC_USE_CXX14 -DMSHADOW_INT64_TENSOR_SIZE=0
-DMSHADOW_IN_CXX11 -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_CUDA=0
-DMSHADOW_USE_MKL=0 -DMSHADOW_USE_SSE -DMXNET_USE_BLAS_OPEN=1
-DMXNET_USE_LAPACK=1 -DMXNET_USE_LIBJPEG_TURBO=0 -DMXNET_USE_MKLDNN=1
-DMXNET_USE_OPENCV=1 -DMXNET_USE_OPENMP=1 -DMXNET_USE_OPERATOR_TUNING=1
-DMXNET_USE_SIGNAL_HANDLER=1 -DNDEBUG=1 -D__USE_XOPEN2K8 -Dmxnet_EXPORTS
-I../3rdparty/mkldnn/include -I3rdparty/mkldnn/include -I../include -I../src
-I../3rdparty/nvidia_cub -I../3rdparty/tvm/nnvm/include
-I../3rdparty/tvm/include -I../3rdparty/dmlc-core/include
-I../3rdparty/dlpack/include -I../3rdparty/mshadow
-I../3rdparty/mkldnn/src/../include -I3rdparty/dmlc-core/include -isystem
/usr/include/opencv4 -Wall -Wno-sign-compare -O3 -fopenmp -fPIC
-Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3
-mf16c -std=gnu++17 -MD -MT
CMakeFiles/mxnet.dir/src/operator/tensor/indexing_op.cc.o -MF
CMakeFiles/mxnet.dir/src/operator/tensor/indexing_op.cc.o.d -o
CMakeFiles/mxnet.dir/src/operator/tensor/indexing_op.cc.o -c
../src/operator/tensor/indexing_op.cc
Child user: 275.350 s
Child sys : 8.356 s
Child wall: 292.649 s
Child high-water RSS : 11403896 KiB
Recursive and acc. high-water RSS+CACHE : 11448656 KiB
```
```
Makefile
g++ -std=c++17 -c -DMSHADOW_FORCE_STREAM -Wall -Wsign-compare
-DDMLC_USE_CXX11=1 -DDMLC_USE_CXX11=1 -DDMLC_USE_CXX14=1
-DDMLC_MODERN_THREAD_LOCAL=0 -DDMLC_LOG_STACK_TRACE_SIZE=0
-DDMLC_LOG_FATAL_THROW=1 -O3 -DNDEBUG=1
-I/mnt/wkcn/proj/incubator-mxnet/3rdparty/mshadow/
-I/mnt/wkcn/proj/incubator-mxnet/3rdparty/dmlc-core/include -fPIC
-I/mnt/wkcn/proj/incubator-mxnet/3rdparty/tvm/nnvm/include
-I/mnt/wkcn/proj/incubator-mxnet/3rdparty/dlpack/include
-I/mnt/wkcn/proj/incubator-mxnet/3rdparty/tvm/include -Iinclude -funroll-loops
-Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3
-mf16c -DMSHADOW_USE_CUDA=0 -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0
-DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0
-DMXNET_USE_OPENCV=1 -I/usr/include/opencv4 -fopenmp -DMXNET_USE_OPENMP=1
-DMXNET_USE_OPERATOR_TUNING=1 -DMSHADOW_INT64_TENSOR_SIZE=0 -DMXNET_USE_LAPACK
-DMXNET_USE_BLAS_ATLAS=1 -fno-builtin-malloc -fno-builtin-calloc
-fno-builtin-realloc -fno-builtin-free -DMXNET_USE_NVML=0 -DMXNET_USE_NCCL=0
-DMXNET_USE_LIBJPEG_TURBO=0 -MMD -c src/operator/tensor/indexing_op.cc -o
build/src/operator/tensor/indexing_op.o
```
The difference between ninja and makefile:
```
Make - Ninja
{'-DMSHADOW_USE_PASCAL=0', '-DMXNET_USE_BLAS_ATLAS=1', '-fno-builtin-free',
'-DDMLC_USE_CXX14=1', '-std=c++17', 'g++', '-DMSHADOW_RABIT_PS=0',
'-DMXNET_USE_LAPACK', '-funroll-loops', '-DMSHADOW_FORCE_STREAM',
'-fno-builtin-calloc', '-fno-builtin-realloc',
'src/operator/tensor/indexing_op.cc', '-DMXNET_USE_NCCL=0',
'-fno-builtin-malloc', 'build/src/operator/tensor/indexing_op.o',
'-Wsign-compare', '-MMD', '-DMXNET_USE_NVML=0', '-DMSHADOW_DIST_PS=0'}
Ninja - make
{'-DMSHADOW_USE_SSE', '-DMXNET_USE_SIGNAL_HANDLER=1',
'/usr/include/opencv4', '-MD', '-DMXNET_USE_LAPACK=1',
'/usr/lib/ccache/bin/c++', '-MF', '-DDMLC_USE_CXX14', '-std=gnu++17',
'CMakeFiles/mxnet.dir/src/operator/tensor/indexing_op.cc.o.d',
'/usr/bin/ccache', '-DMXNET_USE_MKLDNN=1', '-DMSHADOW_IN_CXX11', '-MT',
'../src/operator/tensor/indexing_op.cc', '-Dmxnet_EXPORTS',
'CMakeFiles/mxnet.dir/src/operator/tensor/indexing_op.cc.o',
'-D__USE_XOPEN2K8', '-Wno-sign-compare', '-DDMLC_USE_CXX11',
'-DMXNET_USE_BLAS_OPEN=1', '-DDMLC_STRICT_CXX11', '-isystem'}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
