Hello,
do you know about the "high" memory usage during build of mlpack?
With version 3.3.2 a machine with guaranteed physical memory of 4.8G
when using 2 CPUs has been enough.
Now when compiling 3.4.2 a machine with 8G physical memory and using 2
CPUs is going in OOM.
[ 5447s] cd /home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build &&
/usr/bin/cmake -E cmake_depends "Unix Makefiles"
/home/abuild/rpmbuild/BUILD/mlpack-3.4.2
/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/src/mlpack
/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build
/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build/src/mlpack
/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build/src/mlpack/CMakeFiles/python_configure.dir/DependInfo.cmake
[ 5447s] Scanning dependencies of target python_configure
[ 5447s] make[2]: Leaving directory
'/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build'
[ 5447s] /usr/bin/make -f
src/mlpack/CMakeFiles/python_configure.dir/build.make
src/mlpack/CMakeFiles/python_configure.dir/build
[ 5447s] make[2]: Entering directory
'/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build'
[ 5447s] [ 39%] Configuring setup.py...
[ 5447s] cd /home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build/src/mlpack &&
/usr/bin/cmake -D
SETUP_PY_IN=/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/src/mlpack/bindings/python/setup.py.in
-D
SETUP_PY_OUT=/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build/src/mlpack/bindings/python/setup.py
-D PACKAGE_VERSION="3.4.2" -D
Boost_SERIALIZATION_LIBRARY="/usr/lib64/libboost_serialization.so" -D
Boost_LIBRARY_DIRS="/usr/lib64" -D
ARMADILLO_LIBRARIES="/usr/lib64/libarmadillo.so" -D
MLPACK_LIBRARY=/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build/lib/libmlpack.so
-D MLPACK_LIBDIR=/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build/lib -D
MLPACK_PYXS="arma_numpy.pyx test_python_binding.pyx adaboost.pyx
approx_kfn.pyx bayesian_linear_regression.pyx cf.pyx dbscan.pyx
decision_stump.pyx decision_tree.pyx det.pyx emst.pyx fastmks.pyx
gmm_train.pyx gmm_generate.pyx gmm_probability.pyx hmm_train.pyx
hmm_loglik.pyx hmm_viterbi.pyx hmm_generate.pyx hoeffding_tree.pyx
kde.pyx kernel_pca.pyx kmeans.pyx lars.pyx linear_regression.pyx
linear_svm.pyx lmnn.pyx local_coordinate_coding.pyx
logistic_regression.pyx lsh.pyx mean_shift.pyx nbc.pyx nca.pyx knn.pyx
kfn.pyx nmf.pyx pca.pyx perceptron.pyx preprocess_split.pyx
preprocess_binarize.pyx preprocess_describe.pyx preprocess_scale.pyx
preprocess_one_hot_encoding.pyx image_converter.pyx radical.pyx
random_forest.pyx krann.pyx softmax_regression.pyx sparse_coding.pyx" -D
OpenMP_CXX_FLAGS="-fopenmp" -D DISABLE_CFLAGS="" -D
CYTHON_INCLUDE_DIRECTORIES="/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/src
/usr/include /usr/include /usr/include/stb /usr/include
/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build/src/mlpack/.." -D
CMAKE_CXX_FLAGS=-fmessage-length=0\ -grecord-gcc-switches\ -O2\ -Wall\
-D_FORTIFY_SOURCE=2\ -fstack-protector-strong\ -funwind-tables\
-fasynchronous-unwind-tables\ -fstack-clash-protection\ -DNDEBUG\ -Wall\
-Wextra\ -ftemplate-depth=1000\ -Wno-unused-function\ -O3\ -fopenmp -D
OUTPUT_DIR=/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build -P
/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/src/mlpack/bindings/python/ConfigureSetup.cmake
[ 5447s] -- Run with /usr/lib64/libboost_serialization.so.
[ 5447s] make[2]: Leaving directory
'/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build'
[ 5447s] make[1]: Entering directory
'/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build'
[ 5447s] [ 39%] Built target python_configure
[ 5447s] make[1]: Leaving directory
'/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build'
[ 6994s] [ 6977.215743] Out of memory: Killed process 10354 (cc1plus)
total-vm:8193592kB, anon-rss:5490544kB, file-rss:0kB, shmem-rss:0kB
[ 6995s] make[2]: Entering directory
'/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build'
[ 6995s] [ 38%] Building CXX object
src/mlpack/tests/CMakeFiles/mlpack_catch_test.dir/feedforward_network_test.cpp.o
[ 6995s] cd
/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build/src/mlpack/tests &&
/usr/bin/c++ -DARMA_NO_DEBUG -DBOOST_TEST_DYN_LINK -DHAS_OPENMP
-DHAS_STB -DNDEBUG -I/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/src
-I/usr/include/stb
-I/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build/src/mlpack/..
-fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2
-fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables
-fstack-clash-protection -DNDEBUG -Wall -Wextra -ftemplate-depth=1000
-Wno-unused-function -O3 -fopenmp -O2 -g -DNDEBUG -Winvalid-pch
-include
/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build/src/mlpack/tests/cotire/mlpack_catch_test_CXX_prefix.hxx
-std=gnu++11 -o
CMakeFiles/mlpack_catch_test.dir/feedforward_network_test.cpp.o -c
/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/src/mlpack/tests/feedforward_network_test.cpp
[ 6995s] c++: internal compiler error: Killed (program cc1plus)
[ 6995s] make[2]: ***
[src/mlpack/tests/CMakeFiles/mlpack_catch_test.dir/build.make:518:
src/mlpack/tests/CMakeFiles/mlpack_catch_test.dir/feedforward_network_test.cpp.o]
Error 4
[ 6995s] make[2]: Leaving directory
'/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build'
[ 6995s] make[2]: *** Waiting for unfinished jobs....
What is being done here during build that needs so much memory?
In relation to the size of mlpack library this looks strange.
Is the memory usage ok or is there anything wrong in the build process?
Thanks
Markus
_______________________________________________
mlpack mailing list
[email protected]
http://knife.lugatgt.org/cgi-bin/mailman/listinfo/mlpack
