Rick, That ko2iblnd-opa was the cause, thanks.
In the default ko2iblnd.conf that was provided, there has already the ko2iblnd-opa there appeared to be aliased to back to ko2iblnd, I didn't think twice about changing the alias and the -opa lines initially which is why it did not work by just modifying the map_on_demand parameter but not the rest in that file. Thank you all. Regards, - Pak My current ko2iblnd.conf that is working: options ko2iblnd peer_credits=128 peer_credits_hiw=64 credits=1024 ntx=2048 map_on_demand=16 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=4 install ko2iblnd /usr/sbin/ko2iblnd-probe The default ko2iblnd.conf: # Currently it isn't possible to auto-tune the o2iblnd parameters optimally # inside the kernel since the OFED API hides the details from us. # Unfortunately, there isn't a single set of parameters that provide optimal # performance on different HCA/HFI types, so this file provides optimized # tunables based on the installed IB interface cards. # # Card detection and tunable selection is handled via /usr/sbin/ko2iblnd-probe # at runtime when the ko2iblnd module is installed, either at boot or when # Lustre is first mounted. alias ko2iblnd-opa ko2iblnd options ko2iblnd-opa peer_credits=128 peer_credits_hiw=64 credits=1024 ntx=2048 map_on_demand=256 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=4 install ko2iblnd /usr/sbin/ko2iblnd-probe On Tue, Sep 4, 2018 at 9:40 AM, Mohr Jr, Richard Frank (Rick Mohr) < [email protected]> wrote: > > > On Sep 4, 2018, at 12:12 PM, Pak Lui <[email protected]> wrote: > > > > I have tried "map_on_demand=16" to the "/etc/modprobe.d/ko2iblnd.conf" > that was suggested. Also tried "map_on_demand=0" as suggested here: > http://wiki.lustre.org/Optimizing_o2iblnd_Performance > > > > /etc/modprobe.d/ko2iblnd.conf > > alias ko2iblnd-opa ko2iblnd > > # tried, as suggested in http://wiki.lustre.org/ > Optimizing_o2iblnd_Performance > > #options ko2iblnd-opa peer_credits=128 peer_credits_hiw=64 credits=1024 > ntx=2048 map_on_demand=0 fmr_pool_size=2048 fmr_flush_trigger=512 > fmr_cache=1 conns_per_peer=4 > > options ko2iblnd-opa peer_credits=128 peer_credits_hiw=64 credits=1024 > ntx=2048 map_on_demand=16 fmr_pool_size=2048 fmr_flush_trigger=512 > fmr_cache=1 conns_per_peer=4 > > install ko2iblnd /usr/sbin/ko2iblnd-probe > > The lines with ko2iblnd-opa are intended for OmniPath hardware. Since you > are using IB, you will want to just set your options like this: > > options ko2iblnd peer_credits=…, etc. > > Have you verified that the firewall is not running? It’s possible a > firewall might be allowing ping traffic but blocking the port needed by > Lustre. > > -- > Rick Mohr > Senior HPC System Administrator > National Institute for Computational Sciences > http://www.nics.tennessee.edu > > -- Regards, - Pak
_______________________________________________ lustre-discuss mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
