ctcyang edited a comment on issue #10696: [MXNET-366]Extend MXNet Distributed Training by AllReduce URL: https://github.com/apache/incubator-mxnet/pull/10696#issuecomment-401432358 Thanks for your help @threeleafzerg I was able to build it with `USE_ALLREDUCE_DIST_KVSTORE = 1`. On AWS EC2 instances using the Deep Learning AMI, you need to do these additional steps: ``` wget https://github.com/google/protobuf/releases/download/v3.5.1/protobuf-cpp-3.5.1.tar.gz && tar --no-same-owner -zxf protobuf-cpp-3.5.1.tar.gz cd protobuf-3.5.1 && export CFLAGS=-fPIC && export CXXFLAGS=-fPIC && ./configure -prefix=/usr && sudo make -j16 && sudo make -j16 install conda remove protobuf conda remove libprotobuf rm -rf ~/anaconda3/bin/proto* && rm -rf ~/anaconda3/lib/libproto* sudo apt-get --assume-yes remove libprotobuf-dev sudo apt-get --assume-yes remove libprotobuf-lite9v5 sudo apt-get --assume-yes remove libprotobuf9v5 sudo apt-get --assume-yes remove libprotoc9v5 sudo apt-get --assume-yes install mpich sudo apt-get --assume-yes remove libopenmpi-dev sudo apt-get --assume-yes remove libopenmpi1.10 sudo apt-get --assume-yes remove mpi-default-bin sudo apt-get --assume-yes remove mpi-default-dev sudo apt-get --assume-yes purge openmpi-bin sudo apt-get --assume-yes remove openmpi-common sudo rm -rf /usr/local/mpi sudo ln -s /usr/lib/mpich/lib/libmpich.so.12 /usr/lib/mpich/lib/libmpi.so sudo ln -s /usr/lib/mpich/lib/libmpich.so.12 /usr/lib/mpich/lib/libmpi.so.12 sudo ldconfig ``` When finished, do `ldconfig -p` and verify no more occurrences of `libprotoc.so.9` and `libprotobuf.so.9` occur in the output. Due to ABI incompatibility between different protobuf versions--preinstalled version that comes with Deep Learning AMI (Ubuntu apt), preinstalled version that comes with Anaconda and version that gets auto-installed by the Makefile--you need to uninstall the 2 versions that come with apt and Anaconda.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
