Hi Anna,
This isn’t Lustre specific, but last time I spoke to Mellanox when I was
worried about over subscription between InfiniBand L1 and L2 switches. They
said that the IB ASICs monitor congestion on the HBAs of the connected channel
and will throttle back HBA channel transfer rates if congestion on one or both
is detected.
This may help to smear out some of the imbalances, but you would probably still
get Lustre ‘waiting’ type warnings.
Cheers
Marc
From: lustre-discuss <lustre-discuss-boun...@lists.lustre.org> on behalf of
lustre-discuss-requ...@lists.lustre.org
<lustre-discuss-requ...@lists.lustre.org>
Date: Friday, 7 July 2023 at 21:12
To: lustre-discuss@lists.lustre.org <lustre-discuss@lists.lustre.org>
Subject: lustre-discuss Digest, Vol 208, Issue 6
Send lustre-discuss mailing list submissions to
lustre-discuss@lists.lustre.org
To subscribe or unsubscribe via the World Wide Web, visit
https://linkprotect.cudasvc.com/url?a=http%3a%2f%2flists.lustre.org%2flistinfo.cgi%2flustre-discuss-lustre.org&c=E,1,udOzn8uiss34fCFAWaJTKs9MrvTzb7uSTeisCdxzZ76x5DnzLe2J3JMmFIyIv-IJtUaBMKZaiBbYuF1c5RF6m8rF-jHYOcR0evY3ZEutI-rZ&typo=1
or, via email, send a message with subject or body 'help' to
lustre-discuss-requ...@lists.lustre.org
You can reach the person managing the list at
lustre-discuss-ow...@lists.lustre.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of lustre-discuss digest..."
Today's Topics:
1. Imbalanced incoming and outgoing network load (Anna Fuchs)
2. Re: Imbalanced incoming and outgoing network load
(Kulyavtsev, Alex Ivanovich)
----------------------------------------------------------------------
Message: 1
Date: Fri, 7 Jul 2023 13:48:34 +0200
From: Anna Fuchs <anna.fu...@uni-hamburg.de>
To: <lustre-discuss@lists.lustre.org>
Subject: [lustre-discuss] Imbalanced incoming and outgoing network
load
Message-ID: <ad173ee0-101a-29ee-6995-fa1a66aa5...@uni-hamburg.de>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Dear all,
I have some questions regarding the following scenario:
?- A large HPC system.
- Let's assume that Job X is running on 1 compute node and is reading a
very large file with a stripecount (>>1)..-1. Alternatively, tons of
files are read at once with smaller striping each, but distributed
across all OSS/OSTs.
- The compute node is connected, for example, with a 100Gb/s link, and
there are 50 servers, each with a 200Gb/s link. This generates a network
load of 50x200Gb/s, which is processed at 100Gb/s.
- Job Y, which requires the same network and potentially doesn't even
perform I/O, suffers a lot as a result.
Does this scenario sound familiar to you?
Is the sequence of events correct?
What could be done in this situation?
To avoid:
a) having such single/few-nodes jobs
b) striping large files with up to -1
c) reading millions of files at once
One could try, but I have concerns that the users will persist in doing
it, either intentionally or accidentally, and it would only shift the
problem, rather than solving it.
One could tweak the network design, reconfigure it, separate I/O from
communication, but it would hardly optimize all use cases. Virtual lanes
could potentially be a solution as well. Though, that might not help if
the Job Y also involves some I/O.
Wouldn't it be better if Lustre somehow recognized this imbalance
between incoming and outgoing network traffic and loaded the
file(s)/data gradually rather than all at once, saturating or slightly
overloading the consumer 100Gb/s connection rather than by a factor of
100? Does this sound reasonable, and is there already a solution for it?
I would appreciate any opinions.
Best regards
Anna
--
Anna Fuchs
Universit?t Hamburg
https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwr.informatik.uni-hamburg.de%2fpeople%2fanna_fuchs&c=E,1,5lU4ZdOq0v8v03GynIpPmNbhvtS_2QTSByMPxhQ3oiVaSAIfEjlmOuf0py53AEnmokksRCIU8P50mJcXWmHPYkklq_Gcbhq8AQUcD3kGWcq9rDuUR_K1pk96&typo=1
------------------------------
Message: 2
Date: Fri, 7 Jul 2023 16:18:55 +0000
From: "Kulyavtsev, Alex Ivanovich" <ale...@anl.gov>
To: Anna Fuchs <anna.fu...@uni-hamburg.de>
Cc: "lustre-discuss@lists.lustre.org"
<lustre-discuss@lists.lustre.org>
Subject: Re: [lustre-discuss] Imbalanced incoming and outgoing network
load
Message-ID: <5f635719-e080-4ca0-be2e-55ed330c6...@anl.gov>
Content-Type: text/plain; charset="utf-8"
There is QoS in lustre, the feature called NRS - Network Request Scheduler.
It is possible to set different policies.
Will it address the issue ?
The manual has entry and there were few presentations on LUG/LAD.
I did not use NRS myself but I would like to learn.
Alex.
> On Jul 7, 2023, at 06:48, Anna Fuchs via lustre-discuss
> <lustre-discuss@lists.lustre.org> wrote:
>
> Dear all,
>
> I have some questions regarding the following scenario:
> - A large HPC system.
> - Let's assume that Job X is running on 1 compute node and is reading a very
> large file with a stripecount (>>1)..-1. Alternatively, tons of files are
> read at once with smaller striping each, but distributed across all OSS/OSTs.
> - The compute node is connected, for example, with a 100Gb/s link, and there
> are 50 servers, each with a 200Gb/s link. This generates a network load of
> 50x200Gb/s, which is processed at 100Gb/s.
> - Job Y, which requires the same network and potentially doesn't even perform
> I/O, suffers a lot as a result.
>
> Does this scenario sound familiar to you?
> Is the sequence of events correct?
> What could be done in this situation?
>
> To avoid:
> a) having such single/few-nodes jobs
> b) striping large files with up to -1
> c) reading millions of files at once
> One could try, but I have concerns that the users will persist in doing it,
> either intentionally or accidentally, and it would only shift the problem,
> rather than solving it.
> One could tweak the network design, reconfigure it, separate I/O from
> communication, but it would hardly optimize all use cases. Virtual lanes
> could potentially be a solution as well. Though, that might not help if the
> Job Y also involves some I/O.
>
> Wouldn't it be better if Lustre somehow recognized this imbalance between
> incoming and outgoing network traffic and loaded the file(s)/data gradually
> rather than all at once, saturating or slightly overloading the consumer
> 100Gb/s connection rather than by a factor of 100? Does this sound
> reasonable, and is there already a solution for it?
> I would appreciate any opinions.
>
> Best regards
> Anna
>
> --
> Anna Fuchs
> Universit?t Hamburg
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwr.informatik.uni-hamburg.de%2fpeople%2fanna_fuchs&c=E,1,tE0uhLIlDzWctIlGAdMLaXv5rg1mMzZb43E_JUBZ-5wIPT2fsxnmMiVa4Nnjp9_x58lz7HSfjkbJOnQWnmQBPcdZ1ickmZDSMUA98IQ_VQ,,&typo=1
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> https://linkprotect.cudasvc.com/url?a=http%3a%2f%2flists.lustre.org%2flistinfo.cgi%2flustre-discuss-lustre.org&c=E,1,RdA3hxSQ2vRMTbs0l--PQyom2TjdaN7ziaZ-dcCmFkL565YonIZRF6_wWL8RFV4Pgeb4uSMCtgWb2NMFa9yzfNdU1uaZKrsuICISLwNNYiIDOxt6qFSXuhVa&typo=1
------------------------------
Subject: Digest Footer
_______________________________________________
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
https://linkprotect.cudasvc.com/url?a=http%3a%2f%2flists.lustre.org%2flistinfo.cgi%2flustre-discuss-lustre.org&c=E,1,dcSAQBOQ0m4cALCt1T7jYhggyItOaWAFRfKYaJFwekxVjZ9jzl7BcKTulPJhqJpr2AKONUYh8Zq_y2mRYMnVb61m_i6sAo6VphEn2T2aIY9LsCiRAhdT&typo=1
------------------------------
End of lustre-discuss Digest, Vol 208, Issue 6
**********************************************
_______________________________________________
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
