Re: [Discussion] MXNet CMake build - raise minimal required version
yes that flag can be removed. profiler should always be. built in On Thu, Nov 22, 2018 at 7:44 AM Anton Chernov wrote: > You can find relevant information regarding the profiling flag here: > > https://github.com/apache/incubator-mxnet/issues/11563 > > > чт, 22 нояб. 2018 г. в 16:06, Chris Olivier : > > > what is meant by: > > > > > > *Profiling* > > The profiler is always on even for production release builds, because > MXNet > > can not be build without it [2]. ? > > > > you mean it is always built or it is turned on (ie recording and saving > > profiling information)? I am not aware of it being turned on by default. > > > > > > profiler has no overhead when built in but not turned on. > > > > > > On Thu, Nov 22, 2018 at 2:35 AM Anton Chernov > wrote: > > > > > Dear MXNet community, > > > > > > I propose to raise the minimal required cmake version that is needed to > > > build MXNet to 3.10 which was tagged on March 16 2018 [1]. > > > > > > The effort of repairing cmake scripts in general is targeting to > > deprecate > > > make and maintain only 1 build system. > > > > > > *Need* > > > > > > The build system is the foundation of every software project. It's > > quality > > > is directly impacting the quality of the project. The MXNet build > system > > is > > > fragile, partially broken and not maintained. > > > > > > Users of MXNet and developers are confused by the fact that 2 build > > systems > > > exist at the same time: make and CMake. > > > > > > The main functional areas which are impacted by the current state of > the > > > cmake files are: > > > > > > *OpenMP* > > > The current CMake files mix OpenMP libraries from different compliers > > which > > > is undefined behaviour. It leads to indeterministic crashes on some > > > platforms. Build and deployment are very hard. No evidence exists that > > > proves that there is any benefit of having llvm OpenMP library as a > > > submodule in MXNet. > > > > > > *BLAS and LAPACK* > > > Basic math library usage is mixed up. It is hard and confusing to > > configure > > > and the choosing logic of the most optimal library is not present. MKL > > and > > > OpenBLAS are intermixed in an unpredictable manner. > > > > > > *Profiling* > > > The profiler is always on even for production release builds, because > > MXNet > > > can not be build without it [2]. > > > > > > *CUDA* > > > CUDA is detected by 3 different files in the current cmake scripts and > > the > > > choice of those is based on a obscure logic with involves different > > > versions of cmake and platforms which it's building on > > > > > > * CMakeLists.txt > > > * cmake/FirstClassLangCuda.cmake > > > * 3rdparty/mshadow/cmake/Cuda.cmake > > > > > > > > > *Confusing and misleading cmake user options* > > > For example, USE_CUDA / USE_OLDCMAKECUDA. Some of them will do or not > do > > > what they supposed to based on cmake generator version and version of > > cmake > > > [3]. > > > There are currently more than 30 build parameters for MXNet none of > them > > > documented. Some of them not even located in the main CMakeLists.txt > > file, > > > for example 'BLAS'. > > > > > > > > > *Issues* > > > There is a significant amount of github issues related to cmake or > build > > in > > > general. New tickets are issued frequently. > > > > > > * #8702 (https://github.com/apache/incubator-mxnet/issues/8702) > > > [DISCUSSION] Should we deprecate Makefile and only use CMake? > > > * #5079 (https://github.com/apache/incubator-mxnet/issues/5079) > > troubles > > > building python interface on raspberry pi 3 > > > * #1722 (https://github.com/apache/incubator-mxnet/issues/1722) > > problem: > > > compile mxnet with hdfs > > > * #11549 (https://github.com/apache/incubator-mxnet/issues/11549) Pip > > > package can be much faster (OpenCV version?) > > > * #11417 (https://github.com/apache/incubator-mxnet/issues/11417) > > > libomp.so > > > dependency (need REAL fix) > > > * #8532 (https://github.com/apache/incubator-mxnet/issues/8532) > > > mxnet-mkl > > > (v0.12.0) crash when using (conda-installed) numpy with MKL // > > (indirectly) > > > * #11131 (https://github.com/apache/incubator-mxnet/issues/11131) > > > mxnet-cu92 low efficiency // (indirectly) > > > * #10743 (https://github.com/apache/incubator-mxnet/issues/10743) CUDA > > > 9.1.xx failed if not set OLDCMAKECUDA on cmake 3.10.3 with unix > makefile > > or > > > Ninja generator > > > * #10742 (https://github.com/apache/incubator-mxnet/issues/10742) typo > > in > > > cpp-package/CMakeLists.txt > > > * #10737 (https://github.com/apache/incubator-mxnet/issues/10737) > Cmake > > is > > > running again when execute make install > > > * #10543 (https://github.com/apache/incubator-mxnet/issues/10543) > Failed > > > to > > > build from source when set USE_CPP_PACKAGE = 1, fatal error C1083: > unabel > > > to open file: “mxnet-cpp/op.h”: No such file or directory > > > * #10217 (https://github.com/apache/incubator-mxnet/issues/10217) > > Building > > >
Re: [Discussion] MXNet CMake build - raise minimal required version
Thanks Anton for putting this together and your efforts here. I think it's crucial that we maintain and bring the CMake system forward. I have spent a lot of time dealing with CMake issues on different platforms, we really increase developer productivity and platform support by having a streamlined build system. On Thu, Nov 22, 2018 at 4:06 PM Chris Olivier wrote: > > what is meant by: > > > *Profiling* > The profiler is always on even for production release builds, because MXNet > can not be build without it [2]. ? > > you mean it is always built or it is turned on (ie recording and saving > profiling information)? I am not aware of it being turned on by default. > > > profiler has no overhead when built in but not turned on. > > > On Thu, Nov 22, 2018 at 2:35 AM Anton Chernov wrote: > > > Dear MXNet community, > > > > I propose to raise the minimal required cmake version that is needed to > > build MXNet to 3.10 which was tagged on March 16 2018 [1]. > > > > The effort of repairing cmake scripts in general is targeting to deprecate > > make and maintain only 1 build system. > > > > *Need* > > > > The build system is the foundation of every software project. It's quality > > is directly impacting the quality of the project. The MXNet build system is > > fragile, partially broken and not maintained. > > > > Users of MXNet and developers are confused by the fact that 2 build systems > > exist at the same time: make and CMake. > > > > The main functional areas which are impacted by the current state of the > > cmake files are: > > > > *OpenMP* > > The current CMake files mix OpenMP libraries from different compliers which > > is undefined behaviour. It leads to indeterministic crashes on some > > platforms. Build and deployment are very hard. No evidence exists that > > proves that there is any benefit of having llvm OpenMP library as a > > submodule in MXNet. > > > > *BLAS and LAPACK* > > Basic math library usage is mixed up. It is hard and confusing to configure > > and the choosing logic of the most optimal library is not present. MKL and > > OpenBLAS are intermixed in an unpredictable manner. > > > > *Profiling* > > The profiler is always on even for production release builds, because MXNet > > can not be build without it [2]. > > > > *CUDA* > > CUDA is detected by 3 different files in the current cmake scripts and the > > choice of those is based on a obscure logic with involves different > > versions of cmake and platforms which it's building on > > > > * CMakeLists.txt > > * cmake/FirstClassLangCuda.cmake > > * 3rdparty/mshadow/cmake/Cuda.cmake > > > > > > *Confusing and misleading cmake user options* > > For example, USE_CUDA / USE_OLDCMAKECUDA. Some of them will do or not do > > what they supposed to based on cmake generator version and version of cmake > > [3]. > > There are currently more than 30 build parameters for MXNet none of them > > documented. Some of them not even located in the main CMakeLists.txt file, > > for example 'BLAS'. > > > > > > *Issues* > > There is a significant amount of github issues related to cmake or build in > > general. New tickets are issued frequently. > > > > * #8702 (https://github.com/apache/incubator-mxnet/issues/8702) > > [DISCUSSION] Should we deprecate Makefile and only use CMake? > > * #5079 (https://github.com/apache/incubator-mxnet/issues/5079) troubles > > building python interface on raspberry pi 3 > > * #1722 (https://github.com/apache/incubator-mxnet/issues/1722) problem: > > compile mxnet with hdfs > > * #11549 (https://github.com/apache/incubator-mxnet/issues/11549) Pip > > package can be much faster (OpenCV version?) > > * #11417 (https://github.com/apache/incubator-mxnet/issues/11417) > > libomp.so > > dependency (need REAL fix) > > * #8532 (https://github.com/apache/incubator-mxnet/issues/8532) > > mxnet-mkl > > (v0.12.0) crash when using (conda-installed) numpy with MKL // (indirectly) > > * #11131 (https://github.com/apache/incubator-mxnet/issues/11131) > > mxnet-cu92 low efficiency // (indirectly) > > * #10743 (https://github.com/apache/incubator-mxnet/issues/10743) CUDA > > 9.1.xx failed if not set OLDCMAKECUDA on cmake 3.10.3 with unix makefile or > > Ninja generator > > * #10742 (https://github.com/apache/incubator-mxnet/issues/10742) typo in > > cpp-package/CMakeLists.txt > > * #10737 (https://github.com/apache/incubator-mxnet/issues/10737) Cmake is > > running again when execute make install > > * #10543 (https://github.com/apache/incubator-mxnet/issues/10543) Failed > > to > > build from source when set USE_CPP_PACKAGE = 1, fatal error C1083: unabel > > to open file: “mxnet-cpp/op.h”: No such file or directory > > * #10217 (https://github.com/apache/incubator-mxnet/issues/10217) Building > > with OpenCV causes link errors > > * #10175 (https://github.com/apache/incubator-mxnet/issues/10175) MXNet > > MKLDNN build dependency/flow discussion > > * #10009 (https://github.com/apache/incubator-mxnet/issues/10009) > > [CMAKE][IoT]
Re: [Discussion] MXNet CMake build - raise minimal required version
You can find relevant information regarding the profiling flag here: https://github.com/apache/incubator-mxnet/issues/11563 чт, 22 нояб. 2018 г. в 16:06, Chris Olivier : > what is meant by: > > > *Profiling* > The profiler is always on even for production release builds, because MXNet > can not be build without it [2]. ? > > you mean it is always built or it is turned on (ie recording and saving > profiling information)? I am not aware of it being turned on by default. > > > profiler has no overhead when built in but not turned on. > > > On Thu, Nov 22, 2018 at 2:35 AM Anton Chernov wrote: > > > Dear MXNet community, > > > > I propose to raise the minimal required cmake version that is needed to > > build MXNet to 3.10 which was tagged on March 16 2018 [1]. > > > > The effort of repairing cmake scripts in general is targeting to > deprecate > > make and maintain only 1 build system. > > > > *Need* > > > > The build system is the foundation of every software project. It's > quality > > is directly impacting the quality of the project. The MXNet build system > is > > fragile, partially broken and not maintained. > > > > Users of MXNet and developers are confused by the fact that 2 build > systems > > exist at the same time: make and CMake. > > > > The main functional areas which are impacted by the current state of the > > cmake files are: > > > > *OpenMP* > > The current CMake files mix OpenMP libraries from different compliers > which > > is undefined behaviour. It leads to indeterministic crashes on some > > platforms. Build and deployment are very hard. No evidence exists that > > proves that there is any benefit of having llvm OpenMP library as a > > submodule in MXNet. > > > > *BLAS and LAPACK* > > Basic math library usage is mixed up. It is hard and confusing to > configure > > and the choosing logic of the most optimal library is not present. MKL > and > > OpenBLAS are intermixed in an unpredictable manner. > > > > *Profiling* > > The profiler is always on even for production release builds, because > MXNet > > can not be build without it [2]. > > > > *CUDA* > > CUDA is detected by 3 different files in the current cmake scripts and > the > > choice of those is based on a obscure logic with involves different > > versions of cmake and platforms which it's building on > > > > * CMakeLists.txt > > * cmake/FirstClassLangCuda.cmake > > * 3rdparty/mshadow/cmake/Cuda.cmake > > > > > > *Confusing and misleading cmake user options* > > For example, USE_CUDA / USE_OLDCMAKECUDA. Some of them will do or not do > > what they supposed to based on cmake generator version and version of > cmake > > [3]. > > There are currently more than 30 build parameters for MXNet none of them > > documented. Some of them not even located in the main CMakeLists.txt > file, > > for example 'BLAS'. > > > > > > *Issues* > > There is a significant amount of github issues related to cmake or build > in > > general. New tickets are issued frequently. > > > > * #8702 (https://github.com/apache/incubator-mxnet/issues/8702) > > [DISCUSSION] Should we deprecate Makefile and only use CMake? > > * #5079 (https://github.com/apache/incubator-mxnet/issues/5079) > troubles > > building python interface on raspberry pi 3 > > * #1722 (https://github.com/apache/incubator-mxnet/issues/1722) > problem: > > compile mxnet with hdfs > > * #11549 (https://github.com/apache/incubator-mxnet/issues/11549) Pip > > package can be much faster (OpenCV version?) > > * #11417 (https://github.com/apache/incubator-mxnet/issues/11417) > > libomp.so > > dependency (need REAL fix) > > * #8532 (https://github.com/apache/incubator-mxnet/issues/8532) > > mxnet-mkl > > (v0.12.0) crash when using (conda-installed) numpy with MKL // > (indirectly) > > * #11131 (https://github.com/apache/incubator-mxnet/issues/11131) > > mxnet-cu92 low efficiency // (indirectly) > > * #10743 (https://github.com/apache/incubator-mxnet/issues/10743) CUDA > > 9.1.xx failed if not set OLDCMAKECUDA on cmake 3.10.3 with unix makefile > or > > Ninja generator > > * #10742 (https://github.com/apache/incubator-mxnet/issues/10742) typo > in > > cpp-package/CMakeLists.txt > > * #10737 (https://github.com/apache/incubator-mxnet/issues/10737) Cmake > is > > running again when execute make install > > * #10543 (https://github.com/apache/incubator-mxnet/issues/10543) Failed > > to > > build from source when set USE_CPP_PACKAGE = 1, fatal error C1083: unabel > > to open file: “mxnet-cpp/op.h”: No such file or directory > > * #10217 (https://github.com/apache/incubator-mxnet/issues/10217) > Building > > with OpenCV causes link errors > > * #10175 (https://github.com/apache/incubator-mxnet/issues/10175) MXNet > > MKLDNN build dependency/flow discussion > > * #10009 (https://github.com/apache/incubator-mxnet/issues/10009) > > [CMAKE][IoT] Remove pthread from android_arm64 build > > * #9944 (https://github.com/apache/incubator-mxnet/issues/9944) MXNet > > MinGW-w64 build error // (indirectly) > >
Re: [Discussion] MXNet CMake build - raise minimal required version
i have not seen any proof that any crashes are due to llvm openmp usage. On Thu, Nov 22, 2018 at 2:35 AM Anton Chernov wrote: > Dear MXNet community, > > I propose to raise the minimal required cmake version that is needed to > build MXNet to 3.10 which was tagged on March 16 2018 [1]. > > The effort of repairing cmake scripts in general is targeting to deprecate > make and maintain only 1 build system. > > *Need* > > The build system is the foundation of every software project. It's quality > is directly impacting the quality of the project. The MXNet build system is > fragile, partially broken and not maintained. > > Users of MXNet and developers are confused by the fact that 2 build systems > exist at the same time: make and CMake. > > The main functional areas which are impacted by the current state of the > cmake files are: > > *OpenMP* > The current CMake files mix OpenMP libraries from different compliers which > is undefined behaviour. It leads to indeterministic crashes on some > platforms. Build and deployment are very hard. No evidence exists that > proves that there is any benefit of having llvm OpenMP library as a > submodule in MXNet. > > *BLAS and LAPACK* > Basic math library usage is mixed up. It is hard and confusing to configure > and the choosing logic of the most optimal library is not present. MKL and > OpenBLAS are intermixed in an unpredictable manner. > > *Profiling* > The profiler is always on even for production release builds, because MXNet > can not be build without it [2]. > > *CUDA* > CUDA is detected by 3 different files in the current cmake scripts and the > choice of those is based on a obscure logic with involves different > versions of cmake and platforms which it's building on > > * CMakeLists.txt > * cmake/FirstClassLangCuda.cmake > * 3rdparty/mshadow/cmake/Cuda.cmake > > > *Confusing and misleading cmake user options* > For example, USE_CUDA / USE_OLDCMAKECUDA. Some of them will do or not do > what they supposed to based on cmake generator version and version of cmake > [3]. > There are currently more than 30 build parameters for MXNet none of them > documented. Some of them not even located in the main CMakeLists.txt file, > for example 'BLAS'. > > > *Issues* > There is a significant amount of github issues related to cmake or build in > general. New tickets are issued frequently. > > * #8702 (https://github.com/apache/incubator-mxnet/issues/8702) > [DISCUSSION] Should we deprecate Makefile and only use CMake? > * #5079 (https://github.com/apache/incubator-mxnet/issues/5079) troubles > building python interface on raspberry pi 3 > * #1722 (https://github.com/apache/incubator-mxnet/issues/1722) problem: > compile mxnet with hdfs > * #11549 (https://github.com/apache/incubator-mxnet/issues/11549) Pip > package can be much faster (OpenCV version?) > * #11417 (https://github.com/apache/incubator-mxnet/issues/11417) > libomp.so > dependency (need REAL fix) > * #8532 (https://github.com/apache/incubator-mxnet/issues/8532) > mxnet-mkl > (v0.12.0) crash when using (conda-installed) numpy with MKL // (indirectly) > * #11131 (https://github.com/apache/incubator-mxnet/issues/11131) > mxnet-cu92 low efficiency // (indirectly) > * #10743 (https://github.com/apache/incubator-mxnet/issues/10743) CUDA > 9.1.xx failed if not set OLDCMAKECUDA on cmake 3.10.3 with unix makefile or > Ninja generator > * #10742 (https://github.com/apache/incubator-mxnet/issues/10742) typo in > cpp-package/CMakeLists.txt > * #10737 (https://github.com/apache/incubator-mxnet/issues/10737) Cmake is > running again when execute make install > * #10543 (https://github.com/apache/incubator-mxnet/issues/10543) Failed > to > build from source when set USE_CPP_PACKAGE = 1, fatal error C1083: unabel > to open file: “mxnet-cpp/op.h”: No such file or directory > * #10217 (https://github.com/apache/incubator-mxnet/issues/10217) Building > with OpenCV causes link errors > * #10175 (https://github.com/apache/incubator-mxnet/issues/10175) MXNet > MKLDNN build dependency/flow discussion > * #10009 (https://github.com/apache/incubator-mxnet/issues/10009) > [CMAKE][IoT] Remove pthread from android_arm64 build > * #9944 (https://github.com/apache/incubator-mxnet/issues/9944) MXNet > MinGW-w64 build error // (indirectly) > * #9868 (https://github.com/apache/incubator-mxnet/issues/9868) MKL and > CMake > * #9516 (https://github.com/apache/incubator-mxnet/issues/9516) cmake > cuda arch issues > * #9105 (https://github.com/apache/incubator-mxnet/issues/9105) > libmxnet.so load path error > * #9096 (https://github.com/apache/incubator-mxnet/issues/9096) MXNet > built with GPerftools crashes > * #8786 (https://github.com/apache/incubator-mxnet/issues/8786) Link > failure on DEBUG=1 (static member symbol not defined) // (indirectly) > * #8729 (https://github.com/apache/incubator-mxnet/issues/8729) Build > amalgamation using a docker // (indirectly) > * #8667
[Discussion] MXNet CMake build - raise minimal required version
Dear MXNet community, I propose to raise the minimal required cmake version that is needed to build MXNet to 3.10 which was tagged on March 16 2018 [1]. The effort of repairing cmake scripts in general is targeting to deprecate make and maintain only 1 build system. *Need* The build system is the foundation of every software project. It's quality is directly impacting the quality of the project. The MXNet build system is fragile, partially broken and not maintained. Users of MXNet and developers are confused by the fact that 2 build systems exist at the same time: make and CMake. The main functional areas which are impacted by the current state of the cmake files are: *OpenMP* The current CMake files mix OpenMP libraries from different compliers which is undefined behaviour. It leads to indeterministic crashes on some platforms. Build and deployment are very hard. No evidence exists that proves that there is any benefit of having llvm OpenMP library as a submodule in MXNet. *BLAS and LAPACK* Basic math library usage is mixed up. It is hard and confusing to configure and the choosing logic of the most optimal library is not present. MKL and OpenBLAS are intermixed in an unpredictable manner. *Profiling* The profiler is always on even for production release builds, because MXNet can not be build without it [2]. *CUDA* CUDA is detected by 3 different files in the current cmake scripts and the choice of those is based on a obscure logic with involves different versions of cmake and platforms which it's building on * CMakeLists.txt * cmake/FirstClassLangCuda.cmake * 3rdparty/mshadow/cmake/Cuda.cmake *Confusing and misleading cmake user options* For example, USE_CUDA / USE_OLDCMAKECUDA. Some of them will do or not do what they supposed to based on cmake generator version and version of cmake [3]. There are currently more than 30 build parameters for MXNet none of them documented. Some of them not even located in the main CMakeLists.txt file, for example 'BLAS'. *Issues* There is a significant amount of github issues related to cmake or build in general. New tickets are issued frequently. * #8702 (https://github.com/apache/incubator-mxnet/issues/8702) [DISCUSSION] Should we deprecate Makefile and only use CMake? * #5079 (https://github.com/apache/incubator-mxnet/issues/5079) troubles building python interface on raspberry pi 3 * #1722 (https://github.com/apache/incubator-mxnet/issues/1722) problem: compile mxnet with hdfs * #11549 (https://github.com/apache/incubator-mxnet/issues/11549) Pip package can be much faster (OpenCV version?) * #11417 (https://github.com/apache/incubator-mxnet/issues/11417) libomp.so dependency (need REAL fix) * #8532 (https://github.com/apache/incubator-mxnet/issues/8532) mxnet-mkl (v0.12.0) crash when using (conda-installed) numpy with MKL // (indirectly) * #11131 (https://github.com/apache/incubator-mxnet/issues/11131) mxnet-cu92 low efficiency // (indirectly) * #10743 (https://github.com/apache/incubator-mxnet/issues/10743) CUDA 9.1.xx failed if not set OLDCMAKECUDA on cmake 3.10.3 with unix makefile or Ninja generator * #10742 (https://github.com/apache/incubator-mxnet/issues/10742) typo in cpp-package/CMakeLists.txt * #10737 (https://github.com/apache/incubator-mxnet/issues/10737) Cmake is running again when execute make install * #10543 (https://github.com/apache/incubator-mxnet/issues/10543) Failed to build from source when set USE_CPP_PACKAGE = 1, fatal error C1083: unabel to open file: “mxnet-cpp/op.h”: No such file or directory * #10217 (https://github.com/apache/incubator-mxnet/issues/10217) Building with OpenCV causes link errors * #10175 (https://github.com/apache/incubator-mxnet/issues/10175) MXNet MKLDNN build dependency/flow discussion * #10009 (https://github.com/apache/incubator-mxnet/issues/10009) [CMAKE][IoT] Remove pthread from android_arm64 build * #9944 (https://github.com/apache/incubator-mxnet/issues/9944) MXNet MinGW-w64 build error // (indirectly) * #9868 (https://github.com/apache/incubator-mxnet/issues/9868) MKL and CMake * #9516 (https://github.com/apache/incubator-mxnet/issues/9516) cmake cuda arch issues * #9105 (https://github.com/apache/incubator-mxnet/issues/9105) libmxnet.so load path error * #9096 (https://github.com/apache/incubator-mxnet/issues/9096) MXNet built with GPerftools crashes * #8786 (https://github.com/apache/incubator-mxnet/issues/8786) Link failure on DEBUG=1 (static member symbol not defined) // (indirectly) * #8729 (https://github.com/apache/incubator-mxnet/issues/8729) Build amalgamation using a docker // (indirectly) * #8667 (https://github.com/apache/incubator-mxnet/issues/8667) Compiler/linker error while trying to build from source on Mac OSX Sierra 10.12.6 * #8295 (https://github.com/apache/incubator-mxnet/issues/8295) Building with cmake - error * #7852 (https://github.com/apache/incubator-mxnet/issues/7852) Trouble installing MXNet on Raspberry Pi 3 * #13303
