Re: [lustre-discuss] Lustre poor performance

2017-08-22 Thread Riccardo Veraldi 
On 8/22/17 9:22 AM, Mannthey, Keith wrote:
>
> You may want to file a jira ticket if ko2iblnd-opa setting were being
> automatically used on your Mellanox setup.  That is not expected.
>
yes they are automatically used on my Mellanox and the script
ko2iblnd-probe seems like not working properly.
>
>  
>
> On another note:  As you note you NVMe backend is much faster than QRD
> link speed.  You may want to look at using the new Multi-rall lnet
> feature to boost network bandwidth.  You can add a 2^nd QRD HCA/Port
> and get more Lnet bandwith from your OSS server.   It is a new feature
> that is a bit of work to use but if you are chasing bandwith it might
> be worth the effort.
>
I have a dual infiniband card so I was thinking to bond them to have
more bandwidth. Is this that you mean when you are talking about the
Muti-rail feature boost ?

thanks

Rick


>  
>
> Thanks,
>
> Keith
>
>  
>
> *From:*lustre-discuss [mailto:lustre-discuss-boun...@lists.lustre.org]
> *On Behalf Of *Chris Horn
> *Sent:* Monday, August 21, 2017 12:40 PM
> *To:* Riccardo Veraldi ; Arman
> Khalatyan 
> *Cc:* lustre-discuss@lists.lustre.org
> *Subject:* Re: [lustre-discuss] Lustre poor performance
>
>  
>
> The ko2iblnd-opa settings are tuned specifically for Intel OmniPath.
> Take a look at the /usr/sbin/ko2iblnd-probe script to see how OPA
> hardware is detected and the “ko2iblnd-opa” settings get used.
>
>  
>
> Chris Horn
>
>  
>
> *From: *lustre-discuss  > on behalf of
> Riccardo Veraldi  >
> *Date: *Saturday, August 19, 2017 at 5:00 PM
> *To: *Arman Khalatyan >
> *Cc: *"lustre-discuss@lists.lustre.org
> "
> >
> *Subject: *Re: [lustre-discuss] Lustre poor performance
>
>  
>
> I ran again my Lnet self test and  this time adding --concurrency=16 
> I can use all of the IB bandwith (3.5GB/sec).
>
> the only thing I do not understand is why ko2iblnd.conf is not loaded
> properly and I had to remove the alias in the config file to allow
> the proper peer_credit settings to be loaded.
>
> thanks to everyone for helping
>
> Riccardo
>
> On 8/19/17 8:54 AM, Riccardo Veraldi wrote:
>
>
> I found out that ko2iblnd is not getting settings from
> /etc/modprobe/ko2iblnd.conf
> alias ko2iblnd-opa ko2iblnd
> options ko2iblnd-opa peer_credits=128 peer_credits_hiw=64
> credits=1024 concurrent_sends=256 ntx=2048 map_on_demand=32
> fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=4
>
> install ko2iblnd /usr/sbin/ko2iblnd-probe
>
> but if I modify ko2iblnd.conf like this, then settings are loaded:
>
> options ko2iblnd peer_credits=128 peer_credits_hiw=64 credits=1024
> concurrent_sends=256 ntx=2048 map_on_demand=32 fmr_pool_size=2048
> fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=4
>
> install ko2iblnd /usr/sbin/ko2iblnd-probe
>
> Lnet tests show better behaviour but still I Would expect more
> than this.
> Is it possible to tune parameters in /etc/modprobe/ko2iblnd.conf
> so that Mellanox ConnectX-3 will work more efficiently ?
>
> [LNet Rates of servers]
> [R] Avg: 2286 RPC/s Min: 0RPC/s Max: 4572 RPC/s
> [W] Avg: 3322 RPC/s Min: 0RPC/s Max: 6643 RPC/s
> [LNet Bandwidth of servers]
> [R] Avg: 625.23   MiB/s Min: 0.00 MiB/s Max: 1250.46  MiB/s
> [W] Avg: 1035.85  MiB/s Min: 0.00 MiB/s Max: 2071.69  MiB/s
> [LNet Rates of servers]
> [R] Avg: 2286 RPC/s Min: 1RPC/s Max: 4571 RPC/s
> [W] Avg: 3321 RPC/s Min: 1RPC/s Max: 6641 RPC/s
> [LNet Bandwidth of servers]
> [R] Avg: 625.55   MiB/s Min: 0.00 MiB/s Max: 1251.11  MiB/s
> [W] Avg: 1035.05  MiB/s Min: 0.00 MiB/s Max: 2070.11  MiB/s
> [LNet Rates of servers]
> [R] Avg: 2291 RPC/s Min: 0RPC/s Max: 4581 RPC/s
> [W] Avg: 3329 RPC/s Min: 0RPC/s Max: 6657 RPC/s
> [LNet Bandwidth of servers]
> [R] Avg: 626.55   MiB/s Min: 0.00 MiB/s Max: 1253.11  MiB/s
> [W] Avg: 1038.05  MiB/s Min: 0.00 MiB/s Max: 2076.11  MiB/s
> session is ended
> ./lnet_test.sh: line 17: 23394 Terminated  lst stat
> servers
>
>
>
>
> On 8/19/17 4:20 AM, Arman Khalatyan wrote:
>
> just minor comment,
>
> you should push up performance of your nodes,they are not
> running in the max cpu frequencies.Al tests might be
> inconsistent. in order to get most of ib run following:
>
> tuned-adm profile latency-performance
>
> for more options use:
>
> tuned-adm list
>
>  
>
> It will be interesting to see the

Re: [lustre-discuss] nodes crash during ior test

2017-08-22 Thread Vicker, Darby (JSC-EG311) 
Any more info on this?  I’m running into the same thing.  I tried to find and 
LU on this but didn’t see anything directly related.
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Lustre poor performance

2017-08-22 Thread Mannthey, Keith 
You may want to file a jira ticket if ko2iblnd-opa setting were being 
automatically used on your Mellanox setup.  That is not expected.

On another note:  As you note you NVMe backend is much faster than QRD link 
speed.  You may want to look at using the new Multi-rall lnet feature to boost 
network bandwidth.  You can add a 2nd QRD HCA/Port and get more Lnet bandwith 
from your OSS server.   It is a new feature that is a bit of work to use but if 
you are chasing bandwith it might be worth the effort.

Thanks,
Keith

From: lustre-discuss [mailto:lustre-discuss-boun...@lists.lustre.org] On Behalf 
Of Chris Horn
Sent: Monday, August 21, 2017 12:40 PM
To: Riccardo Veraldi ; Arman Khalatyan 

Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] Lustre poor performance

The ko2iblnd-opa settings are tuned specifically for Intel OmniPath. Take a 
look at the /usr/sbin/ko2iblnd-probe script to see how OPA hardware is detected 
and the “ko2iblnd-opa” settings get used.

Chris Horn

From: lustre-discuss 
>
 on behalf of Riccardo Veraldi 
>
Date: Saturday, August 19, 2017 at 5:00 PM
To: Arman Khalatyan >
Cc: "lustre-discuss@lists.lustre.org" 
>
Subject: Re: [lustre-discuss] Lustre poor performance

I ran again my Lnet self test and  this time adding --concurrency=16  I can use 
all of the IB bandwith (3.5GB/sec).

the only thing I do not understand is why ko2iblnd.conf is not loaded properly 
and I had to remove the alias in the config file to allow
the proper peer_credit settings to be loaded.

thanks to everyone for helping

Riccardo

On 8/19/17 8:54 AM, Riccardo Veraldi wrote:

I found out that ko2iblnd is not getting settings from 
/etc/modprobe/ko2iblnd.conf
alias ko2iblnd-opa ko2iblnd
options ko2iblnd-opa peer_credits=128 peer_credits_hiw=64 credits=1024 
concurrent_sends=256 ntx=2048 map_on_demand=32 fmr_pool_size=2048 
fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=4

install ko2iblnd /usr/sbin/ko2iblnd-probe

but if I modify ko2iblnd.conf like this, then settings are loaded:

options ko2iblnd peer_credits=128 peer_credits_hiw=64 credits=1024 
concurrent_sends=256 ntx=2048 map_on_demand=32 fmr_pool_size=2048 
fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=4

install ko2iblnd /usr/sbin/ko2iblnd-probe

Lnet tests show better behaviour but still I Would expect more than this.
Is it possible to tune parameters in /etc/modprobe/ko2iblnd.conf so that 
Mellanox ConnectX-3 will work more efficiently ?

[LNet Rates of servers]
[R] Avg: 2286 RPC/s Min: 0RPC/s Max: 4572 RPC/s
[W] Avg: 3322 RPC/s Min: 0RPC/s Max: 6643 RPC/s
[LNet Bandwidth of servers]
[R] Avg: 625.23   MiB/s Min: 0.00 MiB/s Max: 1250.46  MiB/s
[W] Avg: 1035.85  MiB/s Min: 0.00 MiB/s Max: 2071.69  MiB/s
[LNet Rates of servers]
[R] Avg: 2286 RPC/s Min: 1RPC/s Max: 4571 RPC/s
[W] Avg: 3321 RPC/s Min: 1RPC/s Max: 6641 RPC/s
[LNet Bandwidth of servers]
[R] Avg: 625.55   MiB/s Min: 0.00 MiB/s Max: 1251.11  MiB/s
[W] Avg: 1035.05  MiB/s Min: 0.00 MiB/s Max: 2070.11  MiB/s
[LNet Rates of servers]
[R] Avg: 2291 RPC/s Min: 0RPC/s Max: 4581 RPC/s
[W] Avg: 3329 RPC/s Min: 0RPC/s Max: 6657 RPC/s
[LNet Bandwidth of servers]
[R] Avg: 626.55   MiB/s Min: 0.00 MiB/s Max: 1253.11  MiB/s
[W] Avg: 1038.05  MiB/s Min: 0.00 MiB/s Max: 2076.11  MiB/s
session is ended
./lnet_test.sh: line 17: 23394 Terminated  lst stat servers




On 8/19/17 4:20 AM, Arman Khalatyan wrote:
just minor comment,
you should push up performance of your nodes,they are not running in the max 
cpu frequencies.Al tests might be inconsistent. in order to get most of ib run 
following:
tuned-adm profile latency-performance
for more options use:
tuned-adm list

It will be interesting to see the difference.

Am 19.08.2017 3:57 vorm. schrieb "Riccardo Veraldi" 
>:
Hello Keith and Dennis, these are the test I ran.

  *   obdfilter-survey, shows that I Can saturate disk performance, the 
NVMe/ZFS backend is performing very well and it is faster then my Infiniband 
network

pool  alloc   free   read  write   read  write
  -  -  -  -  -  -
drpffb-ost01  3.31T  3.19T  3  35.7K  16.0K  7.03G
  raidz1  3.31T  3.19T  3  35.7K  16.0K  7.03G
nvme0n1   -  -  1  5.95K  7.99K  1.17G
nvme1n1   -  -  0  6.01K  0  1.18G
nvme2n1   -  -  0  5.93K  0  1.17G
nvme3n1   -  -  0  5.88K  0  1.16G
nvme4n1   -  -  1