date:20170818

[lustre-discuss] Lustre/OFED compatibility ?

2017-08-18 Thread john casu


I'm having issues compiling the lustre-2.8 src rpm vs mellanox ofed 4.1-1.0.2.0

Is there anywhere I can find what versions of Lustre are compatible with 
specific versions of OFED ?

thanks,
-john c.
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Lustre poor performance

2017-08-18 Thread Riccardo Veraldi

On 8/18/17 7:05 PM, Dennis Nelson wrote:
> If all four servers are identical and all have IB, why are you
> specifying tcp when mounting the client?
because the MDS does not have InfiniBand but just ethernet connection.
Only the OSSes have Infiniband on ib0 interface.

this is my ldev.conf

psdrp-tst-mds01 - mgs zfs:drpffb-mgs/mgs
psdrp-tst-mds01 - mdt0 zfs:drpffb-mdt0/mdt0
#
drp-tst-ffb01 - OST01 zfs:drpffb-ost01/ost01
drp-tst-ffb02 - OST02 zfs:drpffb-ost02/ost02

this is my lustre.conf on the OSSes and Lustre client

options lnet networks=o2ib5(ib0),tcp5(enp1s0f0)

this is my lustre.conf on the MDS

options lnet networks=tcp5(eth0)






>
> Sent from my iPhone
>
> On Aug 18, 2017, at 8:57 PM, Riccardo Veraldi
> >
> wrote:
>
>> Hello Keith and Dennis, these are the test I ran.
>>
>>   * obdfilter-survey, shows that I Can saturate disk performance, the
>> NVMe/ZFS backend is performing very well and it is faster then my
>> Infiniband network
>>
>> *pool  alloc   free   read  write   read  write**
>> **  -  -  -  -  -  -**
>> **drpffb-ost01  3.31T  3.19T  3  35.7K  16.0K  7.03G**
>> **  raidz1  3.31T  3.19T  3  35.7K  16.0K  7.03G**
>> **nvme0n1   -  -  1  5.95K  7.99K  1.17G**
>> **nvme1n1   -  -  0  6.01K  0  1.18G**
>> **nvme2n1   -  -  0  5.93K  0  1.17G**
>> **nvme3n1   -  -  0  5.88K  0  1.16G**
>> **nvme4n1   -  -  1  5.95K  7.99K  1.17G**
>> **nvme5n1   -  -  0  5.96K  0  1.17G**
>> **  -  -  -  -  -  -*
>>
>> this are the tests results
>>
>> Fri Aug 18 16:54:48 PDT 2017 Obdfilter-survey for case=disk from
>> drp-tst-ffb01
>> ost  1 sz 10485760K rsz 1024K obj1 thr1
>> write*7633.08   *  SHORT rewrite 7558.78 SHORT
>> read 3205.24 [3213.70, 3226.78]
>> ost  1 sz 10485760K rsz 1024K obj1 thr2
>> write*7996.89 *SHORT rewrite 7903.42 SHORT
>> read 5264.70 SHORT
>> ost  1 sz 10485760K rsz 1024K obj2 thr2 write
>> *7718.94* SHORT rewrite 7977.84 SHORT read
>> 5802.17 SHORT
>>
>>   * Lnet self test, and here I see the problems. For reference
>> 172.21.52.[83,84] are the two OSSes 172.21.52.86 is the
>> reader/writer. Here is the script that I ran
>>
>> #!/bin/bash
>> export LST_SESSION=$$
>> lst new_session read_write
>> lst add_group servers 172.21.52.[83,84]@o2ib5
>> lst add_group readers 172.21.52.86@o2ib5
>> lst add_group writers 172.21.52.86@o2ib5
>> lst add_batch bulk_rw
>> lst add_test --batch bulk_rw --from readers --to servers \
>> brw read check=simple size=1M
>> lst add_test --batch bulk_rw --from writers --to servers \
>> brw write check=full size=1M
>> # start running
>> lst run bulk_rw
>> # display server stats for 30 seconds
>> lst stat servers & sleep 30; kill $!
>> # tear down
>> lst end_session
>>
>>
>> here the results
>>
>> SESSION: read_write FEATURES: 1 TIMEOUT: 300 FORCE: No
>> 172.21.52.[83,84]@o2ib5 are added to session
>> 172.21.52.86@o2ib5 are added to session
>> 172.21.52.86@o2ib5 are added to session
>> Test was added successfully
>> Test was added successfully
>> bulk_rw is running now
>> [LNet Rates of servers]
>> [R] Avg: 1751 RPC/s Min: 0RPC/s Max: 3502 RPC/s
>> [W] Avg: 2525 RPC/s Min: 0RPC/s Max: 5050 RPC/s
>> [LNet Bandwidth of servers]
>> [R] Avg: 488.79   MiB/s Min: 0.00 MiB/s Max: 977.59   MiB/s
>> [W] Avg: 773.99   MiB/s Min: 0.00 MiB/s Max: 1547.99  MiB/s
>> [LNet Rates of servers]
>> [R] Avg: 1718 RPC/s Min: 0RPC/s Max: 3435 RPC/s
>> [W] Avg: 2479 RPC/s Min: 0RPC/s Max: 4958 RPC/s
>> [LNet Bandwidth of servers]
>> [R] Avg: 478.19   MiB/s Min: 0.00 MiB/s Max: 956.39   MiB/s
>> [W] Avg: 761.74   MiB/s Min: 0.00 MiB/s Max: 1523.47  MiB/s
>> [LNet Rates of servers]
>> [R] Avg: 1734 RPC/s Min: 0RPC/s Max: 3467 RPC/s
>> [W] Avg: 2506 RPC/s Min: 0RPC/s Max: 5012 RPC/s
>> [LNet Bandwidth of servers]
>> [R] Avg: 480.79   MiB/s Min: 0.00 MiB/s Max: 961.58   MiB/s
>> [W] Avg: 772.49   MiB/s Min: 0.00 MiB/s Max: 1544.98  MiB/s
>> [LNet Rates of servers]
>> [R] Avg: 1722 RPC/s Min: 0RPC/s Max: 3444 RPC/s
>> [W] Avg: 2486 RPC/s Min: 0RPC/s Max: 4972 RPC/s
>> [LNet Bandwidth of servers]
>> [R] Avg: 479.09   MiB/s Min: 0.00 MiB/s Max: 958.18   MiB/s
>> [W] Avg: 764.19   MiB/s Min: 0.00 MiB/s Max: 1528.38  MiB/s
>> [LNet Rates of servers]
>> [R] Avg: 1741 RPC/s Min: 0RPC/s Max: 3482 RPC/s
>> [W] Avg: 2513 RPC/s Min: 0RPC/s Max: 5025 RPC/s
>> [LNet Bandwidth of servers]
>> [R] Avg: 484.59   MiB/s Min: 0.00 MiB/s Max: 969.19   MiB/s
>> [W] Avg: 771.94   MiB/s Min: 0.00 MiB/s Max: 1543.87

Re: [lustre-discuss] Lustre poor performance

2017-08-18 Thread Dennis Nelson

If all four servers are identical and all have IB, why are you specifying tcp 
when mounting the client?

Sent from my iPhone

On Aug 18, 2017, at 8:57 PM, Riccardo Veraldi 
> wrote:

Hello Keith and Dennis, these are the test I ran.


  *   obdfilter-survey, shows that I Can saturate disk performance, the 
NVMe/ZFS backend is performing very well and it is faster then my Infiniband 
network

pool  alloc   free   read  write   read  write
  -  -  -  -  -  -
drpffb-ost01  3.31T  3.19T  3  35.7K  16.0K  7.03G
  raidz1  3.31T  3.19T  3  35.7K  16.0K  7.03G
nvme0n1   -  -  1  5.95K  7.99K  1.17G
nvme1n1   -  -  0  6.01K  0  1.18G
nvme2n1   -  -  0  5.93K  0  1.17G
nvme3n1   -  -  0  5.88K  0  1.16G
nvme4n1   -  -  1  5.95K  7.99K  1.17G
nvme5n1   -  -  0  5.96K  0  1.17G
  -  -  -  -  -  -

this are the tests results

Fri Aug 18 16:54:48 PDT 2017 Obdfilter-survey for case=disk from drp-tst-ffb01
ost  1 sz 10485760K rsz 1024K obj1 thr1 write 7633.08 SHORT 
rewrite 7558.78 SHORT read 3205.24 [3213.70, 3226.78]
ost  1 sz 10485760K rsz 1024K obj1 thr2 write 7996.89 SHORT 
rewrite 7903.42 SHORT read 5264.70 SHORT
ost  1 sz 10485760K rsz 1024K obj2 thr2 write 7718.94 SHORT 
rewrite 7977.84 SHORT read 5802.17 SHORT


  *   Lnet self test, and here I see the problems. For reference 
172.21.52.[83,84] are the two OSSes 172.21.52.86 is the reader/writer. Here is 
the script that I ran

#!/bin/bash
export LST_SESSION=$$
lst new_session read_write
lst add_group servers 172.21.52.[83,84]@o2ib5
lst add_group readers 172.21.52.86@o2ib5
lst add_group writers 172.21.52.86@o2ib5
lst add_batch bulk_rw
lst add_test --batch bulk_rw --from readers --to servers \
brw read check=simple size=1M
lst add_test --batch bulk_rw --from writers --to servers \
brw write check=full size=1M
# start running
lst run bulk_rw
# display server stats for 30 seconds
lst stat servers & sleep 30; kill $!
# tear down
lst end_session


here the results

SESSION: read_write FEATURES: 1 TIMEOUT: 300 FORCE: No
172.21.52.[83,84]@o2ib5 are added to session
172.21.52.86@o2ib5 are added to session
172.21.52.86@o2ib5 are added to session
Test was added successfully
Test was added successfully
bulk_rw is running now
[LNet Rates of servers]
[R] Avg: 1751 RPC/s Min: 0RPC/s Max: 3502 RPC/s
[W] Avg: 2525 RPC/s Min: 0RPC/s Max: 5050 RPC/s
[LNet Bandwidth of servers]
[R] Avg: 488.79   MiB/s Min: 0.00 MiB/s Max: 977.59   MiB/s
[W] Avg: 773.99   MiB/s Min: 0.00 MiB/s Max: 1547.99  MiB/s
[LNet Rates of servers]
[R] Avg: 1718 RPC/s Min: 0RPC/s Max: 3435 RPC/s
[W] Avg: 2479 RPC/s Min: 0RPC/s Max: 4958 RPC/s
[LNet Bandwidth of servers]
[R] Avg: 478.19   MiB/s Min: 0.00 MiB/s Max: 956.39   MiB/s
[W] Avg: 761.74   MiB/s Min: 0.00 MiB/s Max: 1523.47  MiB/s
[LNet Rates of servers]
[R] Avg: 1734 RPC/s Min: 0RPC/s Max: 3467 RPC/s
[W] Avg: 2506 RPC/s Min: 0RPC/s Max: 5012 RPC/s
[LNet Bandwidth of servers]
[R] Avg: 480.79   MiB/s Min: 0.00 MiB/s Max: 961.58   MiB/s
[W] Avg: 772.49   MiB/s Min: 0.00 MiB/s Max: 1544.98  MiB/s
[LNet Rates of servers]
[R] Avg: 1722 RPC/s Min: 0RPC/s Max: 3444 RPC/s
[W] Avg: 2486 RPC/s Min: 0RPC/s Max: 4972 RPC/s
[LNet Bandwidth of servers]
[R] Avg: 479.09   MiB/s Min: 0.00 MiB/s Max: 958.18   MiB/s
[W] Avg: 764.19   MiB/s Min: 0.00 MiB/s Max: 1528.38  MiB/s
[LNet Rates of servers]
[R] Avg: 1741 RPC/s Min: 0RPC/s Max: 3482 RPC/s
[W] Avg: 2513 RPC/s Min: 0RPC/s Max: 5025 RPC/s
[LNet Bandwidth of servers]
[R] Avg: 484.59   MiB/s Min: 0.00 MiB/s Max: 969.19   MiB/s
[W] Avg: 771.94   MiB/s Min: 0.00 MiB/s Max: 1543.87  MiB/s
session is ended
./lnet_test.sh: line 17:  4940 Terminated  lst stat servers


so looks like Lnet is really under performing  going at least half and less 
than InfiniBand capabilities.
How can I find out what is causing this ?

running perf tools tests with infiniband tools I have good results:



* Waiting for client to connect... *


---
Send BW Test
 Dual-port   : OFFDevice : mlx4_0
 Number of qps   : 1Transport type : IB
 Connection type : RCUsing SRQ  : OFF
 RX depth: 512
 CQ Moderation   : 100
 Mtu : 2048[B]
 Link type   : IB
 Max inline data : 0[B]
 rdma_cm QPs : OFF
 Data ex. method : Ethernet

Re: [lustre-discuss] Lustre poor performance

2017-08-18 Thread Riccardo Veraldi

Hello Keith and Dennis, these are the test I ran.

  * obdfilter-survey, shows that I Can saturate disk performance, the
NVMe/ZFS backend is performing very well and it is faster then my
Infiniband network

*pool  alloc   free   read  write   read  write**
**  -  -  -  -  -  -**
**drpffb-ost01  3.31T  3.19T  3  35.7K  16.0K  7.03G**
**  raidz1  3.31T  3.19T  3  35.7K  16.0K  7.03G**
**nvme0n1   -  -  1  5.95K  7.99K  1.17G**
**nvme1n1   -  -  0  6.01K  0  1.18G**
**nvme2n1   -  -  0  5.93K  0  1.17G**
**nvme3n1   -  -  0  5.88K  0  1.16G**
**nvme4n1   -  -  1  5.95K  7.99K  1.17G**
**nvme5n1   -  -  0  5.96K  0  1.17G**
**  -  -  -  -  -  -*

this are the tests results

Fri Aug 18 16:54:48 PDT 2017 Obdfilter-survey for case=disk from
drp-tst-ffb01
ost  1 sz 10485760K rsz 1024K obj1 thr1
write*7633.08   *  SHORT rewrite 7558.78 SHORT read
3205.24 [3213.70, 3226.78]
ost  1 sz 10485760K rsz 1024K obj1 thr2
write*7996.89 *SHORT rewrite 7903.42 SHORT read
5264.70 SHORT
ost  1 sz 10485760K rsz 1024K obj2 thr2 write
*7718.94* SHORT rewrite 7977.84 SHORT read
5802.17 SHORT

  * Lnet self test, and here I see the problems. For reference
172.21.52.[83,84] are the two OSSes 172.21.52.86 is the
reader/writer. Here is the script that I ran

#!/bin/bash
export LST_SESSION=$$
lst new_session read_write
lst add_group servers 172.21.52.[83,84]@o2ib5
lst add_group readers 172.21.52.86@o2ib5
lst add_group writers 172.21.52.86@o2ib5
lst add_batch bulk_rw
lst add_test --batch bulk_rw --from readers --to servers \
brw read check=simple size=1M
lst add_test --batch bulk_rw --from writers --to servers \
brw write check=full size=1M
# start running
lst run bulk_rw
# display server stats for 30 seconds
lst stat servers & sleep 30; kill $!
# tear down
lst end_session


here the results

SESSION: read_write FEATURES: 1 TIMEOUT: 300 FORCE: No
172.21.52.[83,84]@o2ib5 are added to session
172.21.52.86@o2ib5 are added to session
172.21.52.86@o2ib5 are added to session
Test was added successfully
Test was added successfully
bulk_rw is running now
[LNet Rates of servers]
[R] Avg: 1751 RPC/s Min: 0RPC/s Max: 3502 RPC/s
[W] Avg: 2525 RPC/s Min: 0RPC/s Max: 5050 RPC/s
[LNet Bandwidth of servers]
[R] Avg: 488.79   MiB/s Min: 0.00 MiB/s Max: 977.59   MiB/s
[W] Avg: 773.99   MiB/s Min: 0.00 MiB/s Max: 1547.99  MiB/s
[LNet Rates of servers]
[R] Avg: 1718 RPC/s Min: 0RPC/s Max: 3435 RPC/s
[W] Avg: 2479 RPC/s Min: 0RPC/s Max: 4958 RPC/s
[LNet Bandwidth of servers]
[R] Avg: 478.19   MiB/s Min: 0.00 MiB/s Max: 956.39   MiB/s
[W] Avg: 761.74   MiB/s Min: 0.00 MiB/s Max: 1523.47  MiB/s
[LNet Rates of servers]
[R] Avg: 1734 RPC/s Min: 0RPC/s Max: 3467 RPC/s
[W] Avg: 2506 RPC/s Min: 0RPC/s Max: 5012 RPC/s
[LNet Bandwidth of servers]
[R] Avg: 480.79   MiB/s Min: 0.00 MiB/s Max: 961.58   MiB/s
[W] Avg: 772.49   MiB/s Min: 0.00 MiB/s Max: 1544.98  MiB/s
[LNet Rates of servers]
[R] Avg: 1722 RPC/s Min: 0RPC/s Max: 3444 RPC/s
[W] Avg: 2486 RPC/s Min: 0RPC/s Max: 4972 RPC/s
[LNet Bandwidth of servers]
[R] Avg: 479.09   MiB/s Min: 0.00 MiB/s Max: 958.18   MiB/s
[W] Avg: 764.19   MiB/s Min: 0.00 MiB/s Max: 1528.38  MiB/s
[LNet Rates of servers]
[R] Avg: 1741 RPC/s Min: 0RPC/s Max: 3482 RPC/s
[W] Avg: 2513 RPC/s Min: 0RPC/s Max: 5025 RPC/s
[LNet Bandwidth of servers]
[R] Avg: 484.59   MiB/s Min: 0.00 MiB/s Max: 969.19   MiB/s
[W] Avg: 771.94   MiB/s Min: 0.00 MiB/s Max: 1543.87  MiB/s
session is ended
./lnet_test.sh: line 17:  4940 Terminated  lst stat servers

so looks like Lnet is really under performing  going at least half and
less than InfiniBand capabilities.
How can I find out what is causing this ?

running perf tools tests with infiniband tools I have good results:



* Waiting for client to connect... *


---
Send BW Test
 Dual-port   : OFFDevice : mlx4_0
 Number of qps   : 1Transport type : IB
 Connection type : RCUsing SRQ  : OFF
 RX depth: 512
 CQ Moderation   : 100
 Mtu : 2048[B]
 Link type   : IB
 Max inline data : 0[B]
 rdma_cm QPs : OFF
 Data ex. method : Ethernet
---
 local address: LID 0x07 QPN 0x020f PSN 0xacc37a
 remote address: LID 0x0a QPN 0x020f PSN 0x91a069

Re: [lustre-discuss] Lustre poor performance

2017-08-18 Thread Riccardo Veraldi

On 8/18/17 1:13 PM, Mannthey, Keith wrote:
> Is Selinux enabled on the client or server? 
the first thing I always to is to disable SElinux.
it's not running.

>
> Thanks,
>  Keith 
> -Original Message-
> From: Riccardo Veraldi [mailto:riccardo.vera...@cnaf.infn.it] 
> Sent: Friday, August 18, 2017 11:31 AM
> To: Mannthey, Keith ; Dennis Nelson 
> ; lustre-discuss@lists.lustre.org
> Subject: Re: [lustre-discuss] Lustre poor performance
>
>
> thank you Keith,
> I will do all this. the single thread dd tests shows 1GB/sec. I will do the 
> other tests
>
>
> On 8/18/17 9:05 AM, Mannthey, Keith wrote:
>> I would suggest you a few other tests to help isolate where the issue might 
>> be.  
>>
>> 1. What is the single thread "DD" write speed?
>>  
>> 2. Lnet_selfttest:  Please see " Chapter 28. Testing Lustre Network 
>> Performance (LNet Self-Test)" in the Lustre manual if this is a new test for 
>> you. 
>> This will help show how much Lnet bandwith you have from your single client. 
>>  There are tunable in the lnet later that can affect things.  Which QRD HCA 
>> are you using?
>>
>> 3. OBDFilter_survey :  Please see " 29.3. Testing OST Performance 
>> (obdfilter-survey)" in the Lustre manual.  This test will help demonstrate 
>> what the backed NVMe/ZFS setup can do at the OBD layer in Lustre.  
>>
>> Thanks,
>>  Keith
>> -Original Message-
>> From: lustre-discuss [mailto:lustre-discuss-boun...@lists.lustre.org] 
>> On Behalf Of Riccardo Veraldi
>> Sent: Thursday, August 17, 2017 10:48 PM
>> To: Dennis Nelson ; lustre-discuss@lists.lustre.org
>> Subject: Re: [lustre-discuss] Lustre poor performance
>>
>> this is my lustre.conf
>>
>> [drp-tst-ffb01:~]$ cat /etc/modprobe.d/lustre.conf options lnet 
>> networks=o2ib5(ib0),tcp5(enp1s0f0)
>>
>> data transfer is over infiniband
>>
>> ib0: flags=4163  mtu 65520
>> inet 172.21.52.83  netmask 255.255.252.0  broadcast 
>> 172.21.55.255
>>
>>
>> On 8/17/17 10:45 PM, Riccardo Veraldi wrote:
>>> On 8/17/17 9:22 PM, Dennis Nelson wrote:
 It appears that you are running iozone on a single client?  What kind of 
 network is tcp5?  Have you looked at the network to make sure it is not 
 the bottleneck?

>>> yes the data transfer is on ib0 interface and I did a memory to 
>>> memory test through InfiniBand QDR  resulting in 3.7GB/sec.
>>> tcp is used to connect to the MDS. It is tcp5 to differentiate it 
>>> from my other many Lustre clusters. I could have called it tcp but it 
>>> does not make any difference performance wise.
>>> I ran the test from one single node yes, I ran the same test also 
>>> locally on a zpool identical to the one on the Lustre OSS.
>>>  Ihave 4 identical servers each of them with the aame nvme disks:
>>>
>>> server1: OSS - OST1 Lustre/ZFS  raidz1
>>>
>>> server2: OSS - OST2 Lustre/ZFS  raidz1
>>>
>>> server3: local ZFS raidz1
>>>
>>> server4: Lustre client
>>>
>>>
>>>
>>> ___
>>> lustre-discuss mailing list
>>> lustre-discuss@lists.lustre.org
>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>> ___
>> lustre-discuss mailing list
>> lustre-discuss@lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>
>

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Best way to run serverside 2.8 w. MOFED 4.1 on Centos 7.2

2017-08-18 Thread Patrick Farrell

I would strongly suggest make -j something for parallelism, unless you want to 
have time to go out for your coffee.

From: lustre-discuss  on behalf of 
Christopher Johnston 
Sent: Friday, August 18, 2017 3:45:39 PM
To: Jeff Johnson
Cc: lustre-discuss
Subject: Re: [lustre-discuss] Best way to run serverside 2.8 w. MOFED 4.1 on 
Centos 7.2

Get coffee somewhere in between 

On Aug 18, 2017 1:08 PM, "Jeff Johnson" 
> wrote:
John,

You can rebuild 2.8 against MOFED. 1) Install MOFED version of choice. 2) Pull 
down the 2.8 Lustre source and configure with 
'--with-o2ib=/usr/src/ofa_kernel/default'. 3) `make rpms` 4) Install. 5) Profit.

--Jeff

On Fri, Aug 18, 2017 at 9:41 AM, john casu 
> wrote:
I have an existing 2.8 install that broke when we added MOFED into the mix.

Nothing I do wrt installing 2.8 rpms works to fix this, and I get a couple of 
missing symbole, when I install lustre-modules:
depmod: WARNING: 
/lib/modules/3.10.0-327.3.1.el7_lustre.x86_64/extra/kernel/net/lustre/ko2iblnd.ko
 needs unknown symbol ib_query_device
depmod: WARNING: 
/lib/modules/3.10.0-327.3.1.el7_lustre.x86_64/extra/kernel/net/lustre/ko2iblnd.ko
 needs unknown symbol ib_alloc_pd

I'm assuming the issue is that lustre 2.8 is built using the standard Centos 
7.2 infiniband drivers.

I can't move to Centos 7.3, at this time.  Is there any way to get 2.8 up & 
running w. mofed without rebuilding lustre rpms?

If I have to rebuild, it'd probably be easier to go to 2.10 (and zfs 0.7.1). Is 
that a correct assumption?
Or will the 2.10 rpms work on Centps 7.2?

thanks,
-john c
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

--
--
Jeff Johnson
Co-Founder
Aeon Computing

jeff.john...@aeoncomputing.com
www.aeoncomputing.com
t: 858-412-3810 x1001   f: 
858-412-3845
m: 619-204-9061

4170 Morena Boulevard, Suite D - San Diego, CA 92117

High-Performance Computing / Lustre Filesystems / Scale-out Storage

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Best way to run serverside 2.8 w. MOFED 4.1 on Centos 7.2

2017-08-18 Thread Christopher Johnston

Get coffee somewhere in between 

On Aug 18, 2017 1:08 PM, "Jeff Johnson" 
wrote:

> John,
>
> You can rebuild 2.8 against MOFED. 1) Install MOFED version of choice. 2)
> Pull down the 2.8 Lustre source and configure with
> '--with-o2ib=/usr/src/ofa_kernel/default'. 3) `make rpms` 4) Install. 5)
> Profit.
>
> --Jeff
>
> On Fri, Aug 18, 2017 at 9:41 AM, john casu 
> wrote:
>
>> I have an existing 2.8 install that broke when we added MOFED into the
>> mix.
>>
>> Nothing I do wrt installing 2.8 rpms works to fix this, and I get a
>> couple of missing symbole, when I install lustre-modules:
>> depmod: WARNING: /lib/modules/3.10.0-327.3.1.el
>> 7_lustre.x86_64/extra/kernel/net/lustre/ko2iblnd.ko needs unknown symbol
>> ib_query_device
>> depmod: WARNING: /lib/modules/3.10.0-327.3.1.el
>> 7_lustre.x86_64/extra/kernel/net/lustre/ko2iblnd.ko needs unknown symbol
>> ib_alloc_pd
>>
>> I'm assuming the issue is that lustre 2.8 is built using the standard
>> Centos 7.2 infiniband drivers.
>>
>> I can't move to Centos 7.3, at this time.  Is there any way to get 2.8 up
>> & running w. mofed without rebuilding lustre rpms?
>>
>> If I have to rebuild, it'd probably be easier to go to 2.10 (and zfs
>> 0.7.1). Is that a correct assumption?
>> Or will the 2.10 rpms work on Centps 7.2?
>>
>> thanks,
>> -john c
>> ___
>> lustre-discuss mailing list
>> lustre-discuss@lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>
>
>
>
> --
> --
> Jeff Johnson
> Co-Founder
> Aeon Computing
>
> jeff.john...@aeoncomputing.com
> www.aeoncomputing.com
> t: 858-412-3810 x1001 <(858)%20412-3810>   f: 858-412-3845
> <(858)%20412-3845>
> m: 619-204-9061 <(619)%20204-9061>
>
> 4170 Morena Boulevard, Suite D - San Diego, CA 92117
>
> High-Performance Computing / Lustre Filesystems / Scale-out Storage
>
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
>
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Lustre poor performance

2017-08-18 Thread Mannthey, Keith

Is Selinux enabled on the client or server? 

Thanks,
 Keith 
-Original Message-
From: Riccardo Veraldi [mailto:riccardo.vera...@cnaf.infn.it] 
Sent: Friday, August 18, 2017 11:31 AM
To: Mannthey, Keith ; Dennis Nelson 
; lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] Lustre poor performance


thank you Keith,
I will do all this. the single thread dd tests shows 1GB/sec. I will do the 
other tests


On 8/18/17 9:05 AM, Mannthey, Keith wrote:
> I would suggest you a few other tests to help isolate where the issue might 
> be.  
>
> 1. What is the single thread "DD" write speed?
>  
> 2. Lnet_selfttest:  Please see " Chapter 28. Testing Lustre Network 
> Performance (LNet Self-Test)" in the Lustre manual if this is a new test for 
> you. 
> This will help show how much Lnet bandwith you have from your single client.  
> There are tunable in the lnet later that can affect things.  Which QRD HCA 
> are you using?
>
> 3. OBDFilter_survey :  Please see " 29.3. Testing OST Performance 
> (obdfilter-survey)" in the Lustre manual.  This test will help demonstrate 
> what the backed NVMe/ZFS setup can do at the OBD layer in Lustre.  
>
> Thanks,
>  Keith
> -Original Message-
> From: lustre-discuss [mailto:lustre-discuss-boun...@lists.lustre.org] 
> On Behalf Of Riccardo Veraldi
> Sent: Thursday, August 17, 2017 10:48 PM
> To: Dennis Nelson ; lustre-discuss@lists.lustre.org
> Subject: Re: [lustre-discuss] Lustre poor performance
>
> this is my lustre.conf
>
> [drp-tst-ffb01:~]$ cat /etc/modprobe.d/lustre.conf options lnet 
> networks=o2ib5(ib0),tcp5(enp1s0f0)
>
> data transfer is over infiniband
>
> ib0: flags=4163  mtu 65520
> inet 172.21.52.83  netmask 255.255.252.0  broadcast 
> 172.21.55.255
>
>
> On 8/17/17 10:45 PM, Riccardo Veraldi wrote:
>> On 8/17/17 9:22 PM, Dennis Nelson wrote:
>>> It appears that you are running iozone on a single client?  What kind of 
>>> network is tcp5?  Have you looked at the network to make sure it is not the 
>>> bottleneck?
>>>
>> yes the data transfer is on ib0 interface and I did a memory to 
>> memory test through InfiniBand QDR  resulting in 3.7GB/sec.
>> tcp is used to connect to the MDS. It is tcp5 to differentiate it 
>> from my other many Lustre clusters. I could have called it tcp but it 
>> does not make any difference performance wise.
>> I ran the test from one single node yes, I ran the same test also 
>> locally on a zpool identical to the one on the Lustre OSS.
>>  Ihave 4 identical servers each of them with the aame nvme disks:
>>
>> server1: OSS - OST1 Lustre/ZFS  raidz1
>>
>> server2: OSS - OST2 Lustre/ZFS  raidz1
>>
>> server3: local ZFS raidz1
>>
>> server4: Lustre client
>>
>>
>>
>> ___
>> lustre-discuss mailing list
>> lustre-discuss@lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Lustre poor performance

2017-08-18 Thread Riccardo Veraldi


thank you Keith,
I will do all this. the single thread dd tests shows 1GB/sec. I will do
the other tests


On 8/18/17 9:05 AM, Mannthey, Keith wrote:
> I would suggest you a few other tests to help isolate where the issue might 
> be.  
>
> 1. What is the single thread "DD" write speed?
>  
> 2. Lnet_selfttest:  Please see " Chapter 28. Testing Lustre Network 
> Performance (LNet Self-Test)" in the Lustre manual if this is a new test for 
> you. 
> This will help show how much Lnet bandwith you have from your single client.  
> There are tunable in the lnet later that can affect things.  Which QRD HCA 
> are you using?
>
> 3. OBDFilter_survey :  Please see " 29.3. Testing OST Performance 
> (obdfilter-survey)" in the Lustre manual.  This test will help demonstrate 
> what the backed NVMe/ZFS setup can do at the OBD layer in Lustre.  
>
> Thanks,
>  Keith 
> -Original Message-
> From: lustre-discuss [mailto:lustre-discuss-boun...@lists.lustre.org] On 
> Behalf Of Riccardo Veraldi
> Sent: Thursday, August 17, 2017 10:48 PM
> To: Dennis Nelson ; lustre-discuss@lists.lustre.org
> Subject: Re: [lustre-discuss] Lustre poor performance
>
> this is my lustre.conf
>
> [drp-tst-ffb01:~]$ cat /etc/modprobe.d/lustre.conf options lnet 
> networks=o2ib5(ib0),tcp5(enp1s0f0)
>
> data transfer is over infiniband
>
> ib0: flags=4163  mtu 65520
> inet 172.21.52.83  netmask 255.255.252.0  broadcast 172.21.55.255
>
>
> On 8/17/17 10:45 PM, Riccardo Veraldi wrote:
>> On 8/17/17 9:22 PM, Dennis Nelson wrote:
>>> It appears that you are running iozone on a single client?  What kind of 
>>> network is tcp5?  Have you looked at the network to make sure it is not the 
>>> bottleneck?
>>>
>> yes the data transfer is on ib0 interface and I did a memory to memory 
>> test through InfiniBand QDR  resulting in 3.7GB/sec.
>> tcp is used to connect to the MDS. It is tcp5 to differentiate it from 
>> my other many Lustre clusters. I could have called it tcp but it does 
>> not make any difference performance wise.
>> I ran the test from one single node yes, I ran the same test also 
>> locally on a zpool identical to the one on the Lustre OSS.
>>  Ihave 4 identical servers each of them with the aame nvme disks:
>>
>> server1: OSS - OST1 Lustre/ZFS  raidz1
>>
>> server2: OSS - OST2 Lustre/ZFS  raidz1
>>
>> server3: local ZFS raidz1
>>
>> server4: Lustre client
>>
>>
>>
>> ___
>> lustre-discuss mailing list
>> lustre-discuss@lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Best way to run serverside 2.8 w. MOFED 4.1 on Centos 7.2

2017-08-18 Thread Jeff Johnson

John,

You can rebuild 2.8 against MOFED. 1) Install MOFED version of choice. 2)
Pull down the 2.8 Lustre source and configure with
'--with-o2ib=/usr/src/ofa_kernel/default'. 3) `make rpms` 4) Install. 5)
Profit.

--Jeff

On Fri, Aug 18, 2017 at 9:41 AM, john casu  wrote:

> I have an existing 2.8 install that broke when we added MOFED into the mix.
>
> Nothing I do wrt installing 2.8 rpms works to fix this, and I get a couple
> of missing symbole, when I install lustre-modules:
> depmod: WARNING: /lib/modules/3.10.0-327.3.1.el
> 7_lustre.x86_64/extra/kernel/net/lustre/ko2iblnd.ko needs unknown symbol
> ib_query_device
> depmod: WARNING: /lib/modules/3.10.0-327.3.1.el
> 7_lustre.x86_64/extra/kernel/net/lustre/ko2iblnd.ko needs unknown symbol
> ib_alloc_pd
>
> I'm assuming the issue is that lustre 2.8 is built using the standard
> Centos 7.2 infiniband drivers.
>
> I can't move to Centos 7.3, at this time.  Is there any way to get 2.8 up
> & running w. mofed without rebuilding lustre rpms?
>
> If I have to rebuild, it'd probably be easier to go to 2.10 (and zfs
> 0.7.1). Is that a correct assumption?
> Or will the 2.10 rpms work on Centps 7.2?
>
> thanks,
> -john c
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>



-- 
--
Jeff Johnson
Co-Founder
Aeon Computing

jeff.john...@aeoncomputing.com
www.aeoncomputing.com
t: 858-412-3810 x1001   f: 858-412-3845
m: 619-204-9061

4170 Morena Boulevard, Suite D - San Diego, CA 92117

High-Performance Computing / Lustre Filesystems / Scale-out Storage
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] Best way to run serverside 2.8 w. MOFED 4.1 on Centos 7.2

2017-08-18 Thread john casu


I have an existing 2.8 install that broke when we added MOFED into the mix.

Nothing I do wrt installing 2.8 rpms works to fix this, and I get a couple of 
missing symbole, when I install lustre-modules:
depmod: WARNING: /lib/modules/3.10.0-327.3.1.el7_lustre.x86_64/extra/kernel/net/lustre/ko2iblnd.ko needs unknown symbol 
ib_query_device

depmod: WARNING: 
/lib/modules/3.10.0-327.3.1.el7_lustre.x86_64/extra/kernel/net/lustre/ko2iblnd.ko
 needs unknown symbol ib_alloc_pd

I'm assuming the issue is that lustre 2.8 is built using the standard Centos 
7.2 infiniband drivers.

I can't move to Centos 7.3, at this time.  Is there any way to get 2.8 up & 
running w. mofed without rebuilding lustre rpms?

If I have to rebuild, it'd probably be easier to go to 2.10 (and zfs 0.7.1). Is 
that a correct assumption?
Or will the 2.10 rpms work on Centps 7.2?

thanks,
-john c
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Lustre poor performance

2017-08-18 Thread Mannthey, Keith

I would suggest you a few other tests to help isolate where the issue might be. 
 

1. What is the single thread "DD" write speed?
 
2. Lnet_selfttest:  Please see " Chapter 28. Testing Lustre Network Performance 
(LNet Self-Test)" in the Lustre manual if this is a new test for you. 
This will help show how much Lnet bandwith you have from your single client.  
There are tunable in the lnet later that can affect things.  Which QRD HCA are 
you using?

3. OBDFilter_survey :  Please see " 29.3. Testing OST Performance 
(obdfilter-survey)" in the Lustre manual.  This test will help demonstrate what 
the backed NVMe/ZFS setup can do at the OBD layer in Lustre.  

Thanks,
 Keith 
-Original Message-
From: lustre-discuss [mailto:lustre-discuss-boun...@lists.lustre.org] On Behalf 
Of Riccardo Veraldi
Sent: Thursday, August 17, 2017 10:48 PM
To: Dennis Nelson ; lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] Lustre poor performance

this is my lustre.conf

[drp-tst-ffb01:~]$ cat /etc/modprobe.d/lustre.conf options lnet 
networks=o2ib5(ib0),tcp5(enp1s0f0)

data transfer is over infiniband

ib0: flags=4163  mtu 65520
inet 172.21.52.83  netmask 255.255.252.0  broadcast 172.21.55.255


On 8/17/17 10:45 PM, Riccardo Veraldi wrote:
> On 8/17/17 9:22 PM, Dennis Nelson wrote:
>> It appears that you are running iozone on a single client?  What kind of 
>> network is tcp5?  Have you looked at the network to make sure it is not the 
>> bottleneck?
>>
> yes the data transfer is on ib0 interface and I did a memory to memory 
> test through InfiniBand QDR  resulting in 3.7GB/sec.
> tcp is used to connect to the MDS. It is tcp5 to differentiate it from 
> my other many Lustre clusters. I could have called it tcp but it does 
> not make any difference performance wise.
> I ran the test from one single node yes, I ran the same test also 
> locally on a zpool identical to the one on the Lustre OSS.
>  Ihave 4 identical servers each of them with the aame nvme disks:
>
> server1: OSS - OST1 Lustre/ZFS  raidz1
>
> server2: OSS - OST2 Lustre/ZFS  raidz1
>
> server3: local ZFS raidz1
>
> server4: Lustre client
>
>
>
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] Lustre/OFED compatibility ?

Re: [lustre-discuss] Lustre poor performance

Re: [lustre-discuss] Lustre poor performance

Re: [lustre-discuss] Lustre poor performance

Re: [lustre-discuss] Lustre poor performance

Re: [lustre-discuss] Best way to run serverside 2.8 w. MOFED 4.1 on Centos 7.2

Re: [lustre-discuss] Best way to run serverside 2.8 w. MOFED 4.1 on Centos 7.2

Re: [lustre-discuss] Lustre poor performance

Re: [lustre-discuss] Lustre poor performance

Re: [lustre-discuss] Best way to run serverside 2.8 w. MOFED 4.1 on Centos 7.2

[lustre-discuss] Best way to run serverside 2.8 w. MOFED 4.1 on Centos 7.2

Re: [lustre-discuss] Lustre poor performance

12 matches

Site Navigation

Mail list logo

Footer information