Re: [lustre-discuss] Lustre and OFED

[Harald van Pee]

Hi Eli,

we are running lustre without OFED on debian client and server. 
With lustre 2.4.0 on client and servers no problem at all since years.
With lustre 2.5.3 on servers and 2.6. 92 no problems at least for monthes.
with lustre 2.5.3 on servers and 2.7 on clients allways ib connection loss.
Here I'm wondering if a more recent OFED version could help?

We are mostly interested in a rock solid lustre version, lustre 2.6 is fast 
enough for us, but has a memory leak caused by cache usage, lustre 2.7 was 
perfect for us in tests with a small number of machines, but fails completly 
for the full cluster and/or certain tasks.

Best
Harald


On Donnerstag, 27. Juli 2017 22:55:33 CEST E.S. Rosenberg wrote:
> Hi all,
> 
> How 'needed' is OFED for Lustre? In the LUG talks it is mentioned every
> once in a while and that got me thinking a bit.
> 
> What things are gained by installing OFED? Performance? Accurate traffic
> reports?
> 
> Currently I am using a lustre system without OFED but our IB hardware is
> from the FDR generation so not bleeding edge and probably doesn't need OFED
> because of that
> 
> Thanks,
> Eli
> 
> Tech specs:
> Servers: CentOS 6.8 + Lustre 2.8 (kernel from Lustre RPMs)
> Clients: Debian + kernel 4.2 + Lustre 2.8
> IB: ConnectX-3 FDR


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Lustre and OFED

[E.S. Rosenberg]

Jeff (and Grigory - offlist),

Thanks for your fast replies!
On Fri, Jul 28, 2017 at 12:09 AM, Jeff Johnson <
jeff.john...@aeoncomputing.com> wrote:

> Eli,
>
> The biggest driver is usually the drivers. Newer Mellanox hardware not yet
> supported, or supported well, by kernel IB. Way back in the days of old
> there were some interoperability issues where everything (clients and
> servers) needed to be the same drivers and libraries but much of that was
> cleaned up. There could be situations where OFED is needed on the server
> side to support something under the Lustre layer like OST or MDT block
> devices via iSER, SRP, NVMeF, etc.
>
> There may be other reasons but those are off the top of my head.
>
So currently everything seems to be working just fine without OFED, my only
complaint is that the normal Linux interface counters don't report traffic
properly which means I have to write my own perfquery wrappers for tools
like zabbix etc.

I may try adding OFED if I have time at some point but I hope by then to at
least have moved our servers to CentOS 7.3 + Lustre 2.9/10.

Has anyone ever run benchmarks of vanilla vs. OFED?
Thanks again,
Eli

>
> --Jeff
>
> On Thu, Jul 27, 2017 at 4:55 PM, E.S. Rosenberg <
> esr+lus...@mail.hebrew.edu> wrote:
>
>> Hi all,
>>
>> How 'needed' is OFED for Lustre? In the LUG talks it is mentioned every
>> once in a while and that got me thinking a bit.
>>
>> What things are gained by installing OFED? Performance? Accurate traffic
>> reports?
>>
>> Currently I am using a lustre system without OFED but our IB hardware is
>> from the FDR generation so not bleeding edge and probably doesn't need OFED
>> because of that
>>
>> Thanks,
>> Eli
>>
>> Tech specs:
>> Servers: CentOS 6.8 + Lustre 2.8 (kernel from Lustre RPMs)
>> Clients: Debian + kernel 4.2 + Lustre 2.8
>> IB: ConnectX-3 FDR
>>
>> ___
>> lustre-discuss mailing list
>> lustre-discuss@lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>
>>
>
>
> --
> --
> Jeff Johnson
> Co-Founder
> Aeon Computing
>
> jeff.john...@aeoncomputing.com
> www.aeoncomputing.com
> t: 858-412-3810 x1001 <(858)%20412-3810>   f: 858-412-3845
> <(858)%20412-3845>
> m: 619-204-9061 <(619)%20204-9061>
>
> 4170 Morena Boulevard, Suite D - San Diego, CA 92117
>
> High-Performance Computing / Lustre Filesystems / Scale-out Storage
>
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Lustre and OFED

[Jeff Johnson]

Eli,

The biggest driver is usually the drivers. Newer Mellanox hardware not yet
supported, or supported well, by kernel IB. Way back in the days of old
there were some interoperability issues where everything (clients and
servers) needed to be the same drivers and libraries but much of that was
cleaned up. There could be situations where OFED is needed on the server
side to support something under the Lustre layer like OST or MDT block
devices via iSER, SRP, NVMeF, etc.

There may be other reasons but those are off the top of my head.

--Jeff

On Thu, Jul 27, 2017 at 4:55 PM, E.S. Rosenberg 
wrote:

> Hi all,
>
> How 'needed' is OFED for Lustre? In the LUG talks it is mentioned every
> once in a while and that got me thinking a bit.
>
> What things are gained by installing OFED? Performance? Accurate traffic
> reports?
>
> Currently I am using a lustre system without OFED but our IB hardware is
> from the FDR generation so not bleeding edge and probably doesn't need OFED
> because of that
>
> Thanks,
> Eli
>
> Tech specs:
> Servers: CentOS 6.8 + Lustre 2.8 (kernel from Lustre RPMs)
> Clients: Debian + kernel 4.2 + Lustre 2.8
> IB: ConnectX-3 FDR
>
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
>


-- 
--
Jeff Johnson
Co-Founder
Aeon Computing

jeff.john...@aeoncomputing.com
www.aeoncomputing.com
t: 858-412-3810 x1001   f: 858-412-3845
m: 619-204-9061

4170 Morena Boulevard, Suite D - San Diego, CA 92117

High-Performance Computing / Lustre Filesystems / Scale-out Storage
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] Lustre and OFED

[E.S. Rosenberg]

Hi all,

How 'needed' is OFED for Lustre? In the LUG talks it is mentioned every
once in a while and that got me thinking a bit.

What things are gained by installing OFED? Performance? Accurate traffic
reports?

Currently I am using a lustre system without OFED but our IB hardware is
from the FDR generation so not bleeding edge and probably doesn't need OFED
because of that

Thanks,
Eli

Tech specs:
Servers: CentOS 6.8 + Lustre 2.8 (kernel from Lustre RPMs)
Clients: Debian + kernel 4.2 + Lustre 2.8
IB: ConnectX-3 FDR
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] High MDS load, but no activity

[Robin Humble]

Hi Kevin,

On Thu, Jul 27, 2017 at 08:18:04AM -0400, Kevin M. Hildebrand wrote:
>We recently updated to Lustre 2.8 on our cluster, and have started seeing
>some unusal load issues.
>Last night our MDS load climbed to well over 100, and client performance
>dropped to almost zero.
>Initially this appeared to be related to a number of jobs that were doing
>large numbers of opens/closes, but even after killing those jobs, the MDS
>load did not recover.
>
>Looking at stats in /proc/fs/lustre/mdt/scratch-MDT/exports showed
>little to no activity on the MDS.  Looking at iostat showed almost no disk
>activity to the MDT (or to any device, for that matter), and minimal IO wait.
>Memory usage (the machine has 128GB) showed over half of that memory free.

sounds like VM spinning to me. check /proc/zoneinfo, /proc/vmstat etc.

do you have zone_reclaim_mode=0? that's an olde, but important to have
set to zero.
 sysctl vm.zone_reclaim_mode

failing that (and assuming you have a 2 or more numa zone server) I
would guess it's all the zone affinity stuff in lustre these days.
you can turn most of it off with a modprobe option
  options libcfs cpu_npartitions=1

what happens by default is that a bunch of lustre threads are bound to
numa zones and preferentially and agressively allocate kernel ram in
those zones. in practice this usually means that the zone where IB card
is physically attached fills up, and then the machine is (essentially)
out of ram and spinning hard trying to reclaim, even though all the ram
in the other zone(s) is almost all unused.

I tried to talk folks out of having affinity on by default in
  https://jira.hpdd.intel.com/browse/LU-5050
but didn't succeed.

even if it wasn't unstable to have affinity on, IMHO having 2x the ram
available for caching on the MDS and OSS's is #1, and tiny performance
increases from having that ram next to the IB card is a distant #2.

cheers,
robin

>I eventually ended up unmounting the MDT and failing it over to a backup
>MDS, which promptly recovered and now has a load of near zero.
>
>Has anyone seen this before?  Any suggestions for what I should look at if
>this happens again?
>
>Thanks!
>Kevin
>
>--
>Kevin Hildebrand
>University of Maryland, College Park
>Division of IT

>___
>lustre-discuss mailing list
>lustre-discuss@lists.lustre.org
>http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] LAD'17 - Last call!

[DEGREMONT Aurelien]

Hello Lustre community!

These are the last days to send abstracts for LAD'17, do not wait! We 
will be pleased to hear about your Lustre experiences. Not only 
developers but also from sites or admins presenting their Lustre 
deployment and experiences with it.


https://easychair.org/conferences/?conf=lad17


*ABOUT*

EOFS and OpenSFS are happy to announce the 7th LAD will be held in 
Paris, France, at Salon des Arts et Métiers! This will be a 2-day event, 
from 4th to 5th of October, 2017. This will be a great opportunity for 
worldwide Lustre administrators and developers to gather and exchange 
their experiences, developments, tools, good practices and more.


**https://www.eofs.eu/events/lad17

*CALL FOR PAPERS
*

We invite community members to send proposals for presentation.
Those talks could cover, by example, tools, administration experiences, 
configuration setup, developments, tunings, tweaking, ...
No proceeding is required, just an abstract for a 30-min technical 
presentation (including Q).


Please send your abstracts using EasyChair, before July 31th, 2017.
https://easychair.org/conferences/?conf=lad17

*REGISTRATION
*Registration for the workshop is open: http://lad.eofs.eu/
*
DATE CHANGE!*
Important! Please note that LAD dates have shifted by one day from what 
was originally announced!
LAD will now take place from October 4th to October 5th, which means 
from Wednesday to Thursday.


*SOCIAL EVENT*
On Wednesday evening, a dinner will take place at the top of Tour 
Montparnasse, Paris highest skyscraper! A limited number of spouses can 
also attend (on a first-come, first-served basis).


*SPONSORS*
We are very pleased this event is organized thanks to the following 
generous sponsors:

 ATOS, CEA, DDN, INTEL and SEAGATE

For any other information, please contact l...@eofs.eu

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] High MDS load, but no activity

[Kevin M. Hildebrand]

We recently updated to Lustre 2.8 on our cluster, and have started seeing
some unusal load issues.
Last night our MDS load climbed to well over 100, and client performance
dropped to almost zero.
Initially this appeared to be related to a number of jobs that were doing
large numbers of opens/closes, but even after killing those jobs, the MDS
load did not recover.

Looking at stats in /proc/fs/lustre/mdt/scratch-MDT/exports showed
little to no activity on the MDS.  Looking at iostat showed almost no disk
activity to the MDT (or to any device, for that matter), and minimal IO
wait.
Memory usage (the machine has 128GB) showed over half of that memory free.

I eventually ended up unmounting the MDT and failing it over to a backup
MDS, which promptly recovered and now has a load of near zero.

Has anyone seen this before?  Any suggestions for what I should look at if
this happens again?

Thanks!
Kevin

--
Kevin Hildebrand
University of Maryland, College Park
Division of IT
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org