Hi, Yogi, I see you are trying to run Norne with 144 processes. Did you see the problem with much less processes, for example 4 or 8 processes?
In my opinion, with the current approach for domain decomposition, it can be challenging to run Norne with so many processes, considering the relatively small size of the Norne model and the long wells existing in this model. I am not totally sure though. Best Regards, Kai Bao ________________________________ From: Opm <[email protected]> on behalf of Yogi Pandey <[email protected]> Sent: Tuesday, March 10, 2020 10:15 PM To: [email protected] <[email protected]> Subject: [Opm] OPM Flow multi-node simulations stuck at domain decomposition step All, I am trying to run OPM Flow simulations on multiple nodes. I have built OPM Flow from source on Oracle Linux 7 OS (binary compatible with RHEL) with: . GCC-8.3.1 . openmpi-4.0.2 (built from source) . boost-1.72.0 (built from source) . cmake-3.16.4 (built from source) . parmetis-4.0.3 (built from source) . dune-2.6.0: dune-common, dune-geometry, dune-grid, dune-istl (built from source) . Zoltan-3.83 (built from source) . OPM Flow modules are built using following commads: o cmake -DCMAKE_BUILD_TYPE=Release -DUSE_MPI=ON -DUSE_OPENMP=ON -DBLAS_LIBRARIES=/usr/lib64 -DCMAKE_INSTALL_PREFIX=/usr/local .. o sudo make For Norne data set, following is the input file (params) content: ecl-deck-file-name=NORNE_ATW2013.DATA output-dir=out_parallel output-mode=none output-interval=1000000 enable-opm-rst-file=false threads-per-process=1 Simulation is being run on 4 nodes with 32 processors each using following command: mpirun --display-map -mca btl self -x UCX_TLS=rc,self,sm -x HCOLL_ENABLE_MCAST_ALL=0 -mca coll_hcoll_enable 0 -x UCX_IB_TRAFFIC_CLASS=105 -x UCX_IB_GID_INDEX=3 --cpu-set 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35 -np 144 --hostfile /etc/opt/rdma/hostfile /mnt/nfs-share/etc/opm-flow/opm-simulators/build/bin/flow --parameter-file=/mnt/nfs-share/data/norne/params The simulation get stuck indefinitely at the domain decomposition step. I am able to finish a parallel run up to 3 nodes, but always getting stuck at 4 nodes. I have also created some customized simulation decks with about 11 million cells to rule-out that fewer number of cells in the Norne model may be a reason, but the simulation gets stuck as soon as I scale from 1 node to 2 nodes. Can someone please help me understand, what might be causing it? Thank you, Yogi _______________________________________________ Opm mailing list [email protected] https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fopm-project.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fopm&data=02%7C01%7Ckai.bao%40sintef.no%7C1ac6994dae0842f9a4a008d7c5387102%7Ce1f00f39604145b0b309e0210d8b32af%7C1%7C1%7C637194718508875070&sdata=dk%2BNtwSBcDA9%2F2ubjoNpmQcLmp1tt63ftJkHkBcUsq0%3D&reserved=0 _______________________________________________ Opm mailing list [email protected] https://opm-project.org/cgi-bin/mailman/listinfo/opm
