Re: [lustre-discuss] Lustre with ZFS Install

[Degremont, Aurelien via lustre-discuss]

> configure: WARNING: GSS keyring backend requires libkeyutils

The configure command clearly says that libkeyutils should be installed.
Did you try to install it?

Under Rhel, this is probably: dnf install libkeyutils-devel


Aurélien

De : Nick dan 
Date : mardi 24 janvier 2023 à 10:41
À : "Degremont, Aurelien" , 
"lustre-discuss@lists.lustre.org" , 
"lustre-discuss-requ...@lists.lustre.org" 
, 
"lustre-discuss-ow...@lists.lustre.org" 
Objet : RE: [EXTERNAL][lustre-discuss] Lustre with ZFS Install


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.


Hi

I have attached the text file. I have got the following error on ./configure.
configure: error: Cannot enable gss_keyring. See above for details.

Can you help with this?

On Tue, 24 Jan 2023 at 14:47, Degremont, Aurelien 
mailto:degre...@amazon.fr>> wrote:
Hi

It looks like the ‘./configure’ command was not successful. Did you check it?
Also, please copy/paste terminal output as text and not as a picture.

Aurélien

De : lustre-discuss 
mailto:lustre-discuss-boun...@lists.lustre.org>>
 au nom de Nick dan via lustre-discuss 
mailto:lustre-discuss@lists.lustre.org>>
Répondre à : Nick dan mailto:nickdan2...@gmail.com>>
Date : mardi 24 janvier 2023 à 09:31
À : "lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org>" 
mailto:lustre-discuss@lists.lustre.org>>, 
"lustre-discuss-requ...@lists.lustre.org<mailto:lustre-discuss-requ...@lists.lustre.org>"
 
mailto:lustre-discuss-requ...@lists.lustre.org>>,
 
"lustre-discuss-ow...@lists.lustre.org<mailto:lustre-discuss-ow...@lists.lustre.org>"
 
mailto:lustre-discuss-ow...@lists.lustre.org>>
Objet : [EXTERNAL] [lustre-discuss] Lustre with ZFS Install


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.


Hi,

We are trying to use ZFS with Lustre referring to the link:
https://wiki.lustre.org/Lustre_with_ZFS_Install#Build_ZFS

We are using the following steps to do so and getting error while making rpms.

git clone 
git://git.whamcloud.com/fs/lustre-release.git<http://git.whamcloud.com/fs/lustre-release.git>
cd lustre-release/
sh ./autogen.sh
./configure --disable-ldiskfs

make rpms (When we are doing make rpms, we are getting the following error) 
(Error attached in ss below)
[root@sv01 lustre-release]# make rpms
make: *** No rule to make target 'rpms'.  Stop.
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Lustre with ZFS Install

[Degremont, Aurelien via lustre-discuss]

Hi

It looks like the ‘./configure’ command was not successful. Did you check it?
Also, please copy/paste terminal output as text and not as a picture.

Aurélien

De : lustre-discuss  au nom de Nick 
dan via lustre-discuss 
Répondre à : Nick dan 
Date : mardi 24 janvier 2023 à 09:31
À : "lustre-discuss@lists.lustre.org" , 
"lustre-discuss-requ...@lists.lustre.org" 
, 
"lustre-discuss-ow...@lists.lustre.org" 
Objet : [EXTERNAL] [lustre-discuss] Lustre with ZFS Install


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.


Hi,

We are trying to use ZFS with Lustre referring to the link:
https://wiki.lustre.org/Lustre_with_ZFS_Install#Build_ZFS

We are using the following steps to do so and getting error while making rpms.

git clone 
git://git.whamcloud.com/fs/lustre-release.git
cd lustre-release/
sh ./autogen.sh
./configure --disable-ldiskfs

make rpms (When we are doing make rpms, we are getting the following error) 
(Error attached in ss below)
[root@sv01 lustre-release]# make rpms
make: *** No rule to make target 'rpms'.  Stop.
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Lustre Support for Postgres

[Degremont, Aurelien via lustre-discuss]

Hi Dan,

There is no, a priori, incompatibilities between Lustre and Postgres. Don’t 
bother configuring some clients in RW and some in RO before having done 
properly your Postgres setup.

However, this is a Lustre mailing-list and you’re asking for Postgres setup. 
This is not the right place.
You should ask those questions to a Postgres community.

Aurélien

De : lustre-discuss  au nom de Nick 
dan via lustre-discuss 
Répondre à : Nick dan 
Date : jeudi 19 janvier 2023 à 12:23
À : "lustre-discuss@lists.lustre.org" , 
"lustre-discuss-requ...@lists.lustre.org" 
, 
"lustre-discuss-ow...@lists.lustre.org" 
Objet : [EXTERNAL] [lustre-discuss] Lustre Support for Postgres


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.


Hi

We have mounted the same data directory on all the lustre clients and have 
started the postgres service on all clients.
Our requirement is as follows:
1 client as read-write node
Other clients as read only

We want to know if Postgres is compatible with Lustre and this case is 
achievable.
Can you please let us know if this is possible?

Regards,
Nick Dan
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] OST not freeing space for deleted files?

[Degremont, Aurelien via lustre-discuss]

In the past, when I've seen such issue this is was really because there was 
more thread adding new stuff to that queue and the MDT was able to empty it.
- Verify how many sync_in_flight you have?
- You're talking about Robinhood. Is Robinhood deleting lots of files?
- You're saying your destroy queue is not emptying, is there a steady UNLINK 
load coming to your MDT?
- Verify how many new requests is coming to your MDT

lctl set_param mdt.lfsc -MDT.md_stats=clear
sleep 10
lctl get_param mdt.lfsc -MDT.md_stats


Aurélien

Le 12/01/2023 18:38, « lustre-discuss au nom de Daniel Szkola via 
lustre-discuss »  a écrit :

CAUTION: This email originated from outside of the organization. Do not 
click links or open attachments unless you can confirm the sender and know the 
content is safe.



I’m not seeing anything obvious. Today, the inode counts are increased and 
the group has reached their hard limit.
We have Robinhood running and the numbers there seem accurate but the quota 
numbers are still high.

I’m seeing things like this on the MDS node in dmesg:

[Wed Jan 11 11:39:07 2023] LustreError: 
39308:0:(osp_dev.c:1682:osp_iocontrol()) lfsc-OST0004-osc-MDT: unrecognized 
ioctl 0xc00866e6 by lctl
[Wed Jan 11 11:39:14 2023] LustreError: 
39314:0:(class_obd.c:465:class_handle_ioctl()) OBD ioctl : No Device -12066
[Wed Jan 11 11:39:38 2023] LustreError: 
39385:0:(class_obd.c:465:class_handle_ioctl()) OBD ioctl : No Device -12066
[Wed Jan 11 11:39:38 2023] LustreError: 
39385:0:(class_obd.c:465:class_handle_ioctl()) Skipped 1 previous similar 
message
[Wed Jan 11 12:06:12 2023] LustreError: 41360:0:(lod_dev.c:1551:lod_sync()) 
lfsc-MDT-mdtlov: can't sync ost 4: rc = -110
[Wed Jan 11 12:06:12 2023] LustreError: 41360:0:(lod_dev.c:1551:lod_sync()) 
Skipped 1 previous similar message
[Wed Jan 11 12:09:30 2023] LustreError: 41362:0:(lod_dev.c:1551:lod_sync()) 
lfsc-MDT-mdtlov: can't sync ost 4: rc = -110
[Wed Jan 11 16:18:27 2023] LustreError: 41360:0:(lod_dev.c:1551:lod_sync()) 
lfsc-MDT-mdtlov: can't sync ost 4: rc = -110

Only seeing this for OST4 though and not 5, both of which seem to be having 
the problem. So, these may be harmless.

I still don’t know why the destroys_in_flight are over 13 million and not 
decreasing. Any ideas?

—
Dan Szkola
FNAL



> On Jan 12, 2023, at 2:59 AM, Degremont, Aurelien  
wrote:
>
> Hello Daniel,
>
> You should also check if there is not some user workload that is 
triggering that load, like a constant load of SYNC to files on those OSTs by 
example.
>
> Aurélien
>
> Le 11/01/2023 22:37, « lustre-discuss au nom de Daniel Szkola via 
lustre-discuss »  a écrit :
>
>CAUTION: This email originated from outside of the organization. Do 
not click links or open attachments unless you can confirm the sender and know 
the content is safe.
>
>
>
>We recently had to take an OSS node that hosts two OSTs out of service 
to test the hardware as it was randomly power cycling.
>
>I migrated all files off of the two OSTs and after some testing we 
brought the node back into service after recreating the ZFS pools
>and the two OSTs. Since then it’s been mostly working fine, however 
we’ve noticed a few group quotas reporting file usage that doesn’t
>seem to match what is actually on the filesystem. The inode counts 
seem to be correct, but the space used is way too high.
>
>After lots of poking around I am seeing this on the two OSTS:
>
>osp.lfsc-OST0004-osc-MDT.sync_changes=13802381
>osp.lfsc-OST0005-osc-MDT.sync_changes=13060667
>
>I upped the max_rpcs_in_progress and max_rpcs_in_flight for the two 
OSTs, but that just caused the numbers to dip slightly.
>All other OSTs have 0 for that value. Also destroys_in_flight show 
similar numbers for the two OSTs.
>
>Any ideas how I can remedy this?
>
>Lustre 2.12.8
>ZFS 0.7.13
>
>—
>Dan Szkola
>FNAL
>
>
>
>
>
>___
>lustre-discuss mailing list
>lustre-discuss@lists.lustre.org
>
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddiscuss-2Dlustre.org=DwIGaQ=gRgGjJ3BkIsb5y6s49QqsA=e9DXjyTaQ786Tg7WH7oIVaQOA1YDRqyxHOUaYU2_LQw=DBtSEnlwRKd7IUYAtj21XR88qwWp8PCksiUQy7Mn0imnzYiq8OhdYUVdjx3aGoyR=T29TaXoWSYBTh5eRNhMflhEe2YEQu8M1CDqrp_NSNMg=
>

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Lustre Client

[Degremont, Aurelien via lustre-discuss]

Did you try? :)


But the answer is yes, ‘-o ro’ is supported for client mounts

Aurélien
De : lustre-discuss  au nom de Nick 
dan via lustre-discuss 
Répondre à : Nick dan 
Date : vendredi 13 janvier 2023 à 10:48
À : "lustre-discuss@lists.lustre.org" , 
"lustre-discuss-requ...@lists.lustre.org" 
, 
"lustre-discuss-ow...@lists.lustre.org" 
Objet : [EXTERNAL] [lustre-discuss] Lustre Client


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.


Hi

I wanted to ask if Lustre client can be mounted as read-only client or not

Regards,
Nick Dan
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] OST not freeing space for deleted files?

[Degremont, Aurelien via lustre-discuss]

Hello Daniel,

You should also check if there is not some user workload that is triggering 
that load, like a constant load of SYNC to files on those OSTs by example.

Aurélien

Le 11/01/2023 22:37, « lustre-discuss au nom de Daniel Szkola via 
lustre-discuss »  a écrit :

CAUTION: This email originated from outside of the organization. Do not 
click links or open attachments unless you can confirm the sender and know the 
content is safe.



We recently had to take an OSS node that hosts two OSTs out of service to 
test the hardware as it was randomly power cycling.

I migrated all files off of the two OSTs and after some testing we brought 
the node back into service after recreating the ZFS pools
and the two OSTs. Since then it’s been mostly working fine, however we’ve 
noticed a few group quotas reporting file usage that doesn’t
seem to match what is actually on the filesystem. The inode counts seem to 
be correct, but the space used is way too high.

After lots of poking around I am seeing this on the two OSTS:

osp.lfsc-OST0004-osc-MDT.sync_changes=13802381
osp.lfsc-OST0005-osc-MDT.sync_changes=13060667

I upped the max_rpcs_in_progress and max_rpcs_in_flight for the two OSTs, 
but that just caused the numbers to dip slightly.
All other OSTs have 0 for that value. Also destroys_in_flight show similar 
numbers for the two OSTs.

Any ideas how I can remedy this?

Lustre 2.12.8
ZFS 0.7.13

—
Dan Szkola
FNAL





___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Problem in confugration of MGS and MDT server in lustre.

[Degremont, Aurelien via lustre-discuss]

Hi

Yes, both parameters at the same time are valid.

Regarding the RPMS you installed. You picked a new kernel, did you reboot to 
use it? If not you should.
You are missing those 2 packages, I’m even surprised Yum did not complain about 
missing deps.
kmod-lustre-2.15.1-1.el8.x86_64.rpm
kmod-lustre-osd-ldiskfs-2.15.1-1.el8.x86_64.rpm

lustre-ldiskfs-dkms: you installed ldiskfs support as a DKMS package but not 
Lustre as a DKMS too. I recommend using the same for every package, so here, 
install the osd package above instead of your DKMS one.

Probably the best for you is to setup 
https://downloads.whamcloud.com/public/lustre/latest-release/el8.6/server/ as 
YUM additional repository.

Aurélien

De : lustre-discuss  au nom de Taner 
KARAGÖL via lustre-discuss 
Répondre à : Taner KARAGÖL 
Date : mardi 20 décembre 2022 à 07:12
À : Nick dan 
Cc : "lustre-discuss@lists.lustre.org" 
Objet : RE: [EXTERNAL][lustre-discuss] Problem in confugration of MGS and MDT 
server in lustre.


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.


UNCLASSIFIED

Hi,

Is it valid to use two parameters at the same time?  “--mgs –mdt”   I dont 
think so.

Regards,

From: lustre-discuss  On Behalf Of 
Nick dan via lustre-discuss
Sent: Tuesday, December 20, 2022 7:17 AM
To: lustre-discuss@lists.lustre.org
Subject: [lustre-discuss] Problem in confugration of MGS and MDT server in 
lustre.

Hi,
Nick here Trying to configure Lustre MGS and MDT on Redhat 8.6.
Referred from  https://www.lustre.org/
Successfully downloaded and installed the following packages by using yum 
install command:

yum install 
https://downloads.whamcloud.com/public/lustre/latest-release/el8.6/server/RPMS/x86_64/kernel-4.18.0-372.9.1.el8_lustre.x86_64.rpm
yum install 
https://downloads.whamcloud.com/public/lustre/latest-release/el8.6/server/RPMS/x86_64/lustre-2.15.1-1.el8.x86_64.rpm
yum install 
https://downloads.whamcloud.com/public/lustre/latest-release/el8.6/server/RPMS/x86_64/lustre-ldiskfs-dkms-2.15.1-1.el8.noarch.rpm
yum install 
https://downloads.whamcloud.com/public/lustre/latest-release/el8.6/server/RPMS/x86_64/lustre-osd-ldiskfs-mount-2.15.1-1.el8.x86_64.rpm
yum install 
https://downloads.whamcloud.com/public/e2fsprogs/latest/el8/RPMS/x86_64/e2fsprogs-1.46.2.wc5-0.el8.x86_64.rpm
yum install 
https://downloads.whamcloud.com/public/e2fsprogs/latest/el8/RPMS/x86_64/e2fsprogs-libs-1.46.2.wc5-0.el8.x86_64.rpm
yum install 
https://downloads.whamcloud.com/public/e2fsprogs/latest/el8/RPMS/x86_64/libcom_err-1.46.2.wc5-0.el8.x86_64.rpm
yum install 
https://downloads.whamcloud.com/public/e2fsprogs/latest/el8/RPMS/x86_64/libss-1.46.2.wc5-0.el8.x86_64.rpm
After this trying to do mkfs.lustre on my available NVMe-disk(nvme1n1)Mentioned 
in the below Fig
[cid:image001.png@01D9145E.95FCE300]

And did the mkfs.lusre on it and getting the error .


mkfs.lustre --fsname=lustre  --mgs --mdt --index=0  /dev/nvme1n1

   Permanent disk data:
Target: lustre:MDT
Index:  0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:  0x65
  (MDT MGS first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters:

checking for existing Lustre data: not found
device size = 953869MB
formatting backing filesystem ldiskfs on /dev/nvme1n1
target name   lustre:MDT
kilobytes 976762584
options-J size=4096 -I 1024 -i 2560 -q -O 
dirdata,uninit_bg,^extents,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg
 -E lazy_journal_init="0",lazy_itable_init="0" -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT  -J size=4096 -I 1024 -i 2560 -q 
-O 
dirdata,uninit_bg,^extents,dir_nlink,quota,project,huge_file,ea_inode,large_dir,^fast_commit,flex_bg
 -E lazy_journal_init="0",lazy_itable_init="0" -F /dev/nvme1n1 976762584k
mkfs.lustre: Unable to mount /dev/nvme1n1: No such device
Is the ldiskfs module available?
mkfs.lustre FATAL: failed to write local files
mkfs.lustre: exiting with 19 (No such device)

Can you help where am I going wrong?

Regards,

Nick


Dikkat:

Bu elektronik posta mesaji kisisel ve ozeldir. Eger size gonderilmediyse lutfen 
gondericiyi bilgilendirip mesaji siliniz. Firmamiza gelen ve giden mesajlar 
virus taramasindan gecirilmekte, guvenlik nedeni ile kontrol edilerek 
saklanmaktadir. Mesajdaki gorusler ve bakis acisi gondericiye ait olup Aselsan 
A.S. resmi gorusu olmak zorunda degildir.


Attention:

This e-mail message is privileged and confidential. If you are not the intended 
recipient please delete the message and notify the sender. E-mails to 

Re: [lustre-discuss] Patched or Patch less kernel

[Degremont, Aurelien via lustre-discuss]

Hi Christopher,

As far as I know, this will only prevent you from using few features that 
requires either a recent Linux kernel or a kernel with the appropriate 
backports for them to work. From the top of my head, I'm thinking as project 
quotas for ldiskfs, but you are using ZFS, or client side encryption but this 
is not yet available in 2.12. Other may know better, but you should be pretty 
safe going with the stock kernel if using Lustre 2.12 and ZFS w.r.t patched 
kernel.


Aurélien

Le 19/10/2022 19:35, « lustre-discuss au nom de Mountford, Christopher J. 
(Dr.) via lustre-discuss »  a écrit :

CAUTION: This email originated from outside of the organization. Do not 
click links or open attachments unless you can confirm the sender and know the 
content is safe.



Is there any disadvantage to using a stock distribution kernel on our 
Centos 7 lustre servers instead of the _lustre kernel provided in the lustre 
release (version 2.12.9)?

We build spl/zfs and the lustre-zfs kernel modules using dkms and 
standardized on the kernel in the lustre server release a while ago.

Kind Regards,
Christopher.
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Lustre recycle bin

[Degremont, Aurelien via lustre-discuss]

Hi François

[root@server1 ~] rm: cannot remove ‘Logs: Cannot send after transport endpoint 
shutdown
[root@server1 ~] mv: cannot move /test/lustre/structure1 to 
‘/test/lustre/structure2’: Input/output error

These 2 error messages are typical from a client eviction issue. Your client 
was evicted (likely from the MDT) and you should expect ESHUTDOWN or EIO in 
that situation. Look at the kernel logs (dmesg) from both client and server to 
try understand. The client usually reconnect quickly when this happens and 
following access should not return error.

Aurélien

De : lustre-discuss  au nom de 
"Cloete, F. (Francois) via lustre-discuss" 
Répondre à : "Cloete, F. (Francois)" 
Date : jeudi 20 octobre 2022 à 08:48
À : "Spitz, Cory James" , Alastair Basden 

Cc : "lustre-discuss@lists.lustre.org" 
Objet : RE: [EXTERNAL][lustre-discuss] Lustre recycle bin


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.


Hi Cory,
They are both running the same versions on client and mgs server.

lfs 2.12.8_6_g5457c37

Not sure if this could be related but our lustre environment started behaving 
starngly 2/3 weeks ago.

Also seeing the below when doing rsync of folders to new destinations.
failed: Cannot send after transport endpoint shutdown (108)


[root@server1 ~] rm: cannot remove ‘Logs: Cannot send after transport endpoint 
shutdown
[root@server1 ~] mv: cannot move /test/lustre/structure1 to 
‘/test/lustre/structure2’: Input/output error

[root@server1 ~] ~] ll
[root@server1 ~] ls: cannot open directory .: Cannot send after transport 
endpoint shutdown

Regards

From: Spitz, Cory James 
Sent: Monday, 17 October 2022 20:56
To: Cloete, F. (Francois) ; Alastair Basden 

Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] Lustre recycle bin


You don't often get email from cory.sp...@hpe.com. 
Learn why this is important

CAUTION - EXTERNAL SENDER - Please be careful when opening links and 
attachments. Nedbank - IT Information Security Department (ISD)
What version(s) are you using?  Do you have an old client and a new-ish server?

Very old client versions will disagree with the MDSes about how to clean up 
objects, resulting in orphans.

-Cory

On 10/17/22, 3:44 AM, "lustre-discuss" 
mailto:lustre-discuss-boun...@lists.lustre.org>>
 wrote:

Thank-you!

-Original Message-
From: Alastair Basden mailto:a.g.bas...@durham.ac.uk>>
Sent: Monday, 17 October 2022 10:13
To: Cloete, F. (Francois) 
mailto:francois...@nedbank.co.za>>
Cc: Andreas Dilger mailto:adil...@whamcloud.com>>; 
lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] Lustre recycle bin

[You don't often get email from 
a.g.bas...@durham.ac.uk. Learn why this is 
important at https://aka.ms/LearnAboutSenderIdentification   ]


CAUTION - EXTERNAL SENDER -  Please be careful when opening links and 
attachments. Nedbank - IT Information Security Department (ISD)


Hi Francois,

We had something similar a few months back - I suspect a bug somewhere.

Basically files weren't getting removed from the OST.  Eventually, we mounted 
as ext, and removed them manually, I think.

A reboot of the file system meant that rm operations then proceeded correctly 
after that.

Cheers,
Alastair.

On Mon, 17 Oct 2022, Cloete, F. (Francois) via lustre-discuss wrote:

> [EXTERNAL EMAIL]
> Hi Andreas,
> Our OSTs still display high file-system usage after removing folders.
>
> Are there any commands that could be run to confirm if the allocated space 
> which was used by those files have been released successfully ?
>
> Thanks
> Francois
>
> From: Andreas Dilger mailto:adil...@whamcloud.com>>
> Sent: Saturday, 15 October 2022 00:20
> To: Cloete, F. (Francois) 
> mailto:francois...@nedbank.co.za>>
> Cc: lustre-discuss@lists.lustre.org
> Subject: Re: [lustre-discuss] Lustre recycle bin
>
> You don't often get email from
> adil...@whamcloud.com>.
>  Learn why this is
> important  >
> CAUTION - EXTERNAL SENDER - Please be careful when opening links and
> attachments. Nedbank - IT Information Security Department (ISD) There isn't a 
> recycle bin, but filenames are deleted from the filesystem quickly and the 
> data objects are deleted 

Re: [lustre-discuss] Compiling lustre 2.12.X with linux kernel 5.10.X and above

[Degremont, Aurelien via lustre-discuss]

Lustre 2.14.0 supports Linux kernel up to 5.4.
Lustre 2.15.0 which will be released in the coming days, supports up to Linux 
5.11 according to Changelog, but supports clients up to 5.14 according to this 
ticket https://jira.whamcloud.com/browse/LU-15220

This ticket is tracking effort to support Linux 5.15 
https://jira.whamcloud.com/browse/LU-15420


Aurélien

Le 12/05/2022 06:23, « lustre-discuss au nom de Tung-Han Hsieh » 
 a écrit :

CAUTION: This email originated from outside of the organization. Do not 
click links or open attachments unless you can confirm the sender and know the 
content is safe.



Dear All,

We tried to compile Lustre-2.12.6, 2.12.8, and 2.14.0 with Linux
kernel 5.10.114 and 5.15.38, the newest releases of the longterm
series of Linux kernel in Linux Kernel Archives:

https://www.kernel.org/

but all failed in configure state. When running this command in:

./configure --prefix=/opt/lustre --with-linux=/usr/src/linux-5.4.192 \
--with-o2ib=no --disable-server --enable-mpitests=no

it prompted with the error message:


==
checking for /usr/src/linux-5.10.114/include/generated/autoconf.h... yes
checking for /usr/src/linux-5.10.114/include/linux/version.h... no
checking for 
/usr/src/linux-5.10.114/include/generated/uapi/linux/version.h... yes
checking for /usr/src/linux-5.10.114/include/linux/kconfig.h... yes
checking for external module build target... configure: error: unknown; 
check config.log for details

==

The error logs in config.log are attached below.

I am wondering whether there is a plan to port Lustre to Linux kernel
version 5 ? at least the Lustre client part. Upgrading to Linux kernel
version 5 is necessary for us, because the drivers of the embedded
ethernet cards of some newly purchased hardware only available in
Linux kernel version 5.

Thanks very much for your reply in advance.

Best Regards,

T.H.Hsieh


==
configure:10681: checking for external module build target
configure:10709: cp conftest.c build && make -d 
/usr/src/lustre-2.12.6/build LUSTRE_KERNEL_TEST=conftest.i LDFLAGS= 
LD=/usr/bin/ld -m elf_x86_64 CC=gcc -f /usr/src/lustre-2.12.6/build/Makefile 
LUSTRE_LINUX_CONFIG=/usr/src/linux-5.10.114/.config LINUXINCLUDE= 
-I/usr/src/linux-5.10.114/arch/x86/include -Iinclude 
-Iarch/x86/include/generated -I/usr/src/linux-5.10.114/include -Iinclude2 
-I/usr/src/linux-5.10.114/include/uapi -Iinclude/generated 
-I/usr/src/linux-5.10.114/arch/x86/include/uapi 
-Iarch/x86/include/generated/uapi -I/usr/src/linux-5.10.114/include/uapi 
-Iinclude/generated/uapi -include 
/usr/src/linux-5.10.114/include/linux/kconfig.h -o tmp_include_depends -o 
scripts -o include/config/MARKER -C /usr/src/linux-5.10.114 
EXTRA_CFLAGS=-Werror-implicit-function-declaration -g 
-I/usr/src/lustre-2.12.6/libcfs/include -I/usr/src/lustre-2.12.6/lnet/include 
-I/usr/src/lustre-2.12.6/lustre/include/uapi 
-I/usr/src/lustre-2.12.6/lustre/include -Wno-format-truncation 
-Wno-stringop-truncation
  -Wno-stringop-overflow SUBDIRS=/usr/src/lustre-2.12.6/build
configure:10712: $? = 0
configure:10714: test -s build/conftest.i
configure:10717: $? = 1
configure: failed program was:
| /* confdefs.h */
| #define PACKAGE_NAME "Lustre"
| #define PACKAGE_TARNAME "lustre"
| #define PACKAGE_VERSION "2.12.6"
| #define PACKAGE_STRING "Lustre 2.12.6"
| #define PACKAGE_BUGREPORT "https://jira.whamcloud.com/;
| #define PACKAGE_URL ""
| #define PACKAGE "lustre"
| #define VERSION "2.12.6"
| #define STDC_HEADERS 1
| #define HAVE_SYS_TYPES_H 1
| #define HAVE_SYS_STAT_H 1
| #define HAVE_STDLIB_H 1
| #define HAVE_STRING_H 1
| #define HAVE_MEMORY_H 1
| #define HAVE_STRINGS_H 1
| #define HAVE_INTTYPES_H 1
| #define HAVE_STDINT_H 1
| #define HAVE_UNISTD_H 1
| #define HAVE_DLFCN_H 1
| #define LT_OBJDIR ".libs/"
| #define LUSTRE_MAJOR 2
| #define LUSTRE_MINOR 12
| #define LUSTRE_PATCH 6
| #define LUSTRE_FIX 0
| #define LUSTRE_VERSION_STRING "2.12.6"
| #define SIZEOF_UNSIGNED_LONG_LONG 8
| /* end confdefs.h.  */
|
| #include 
| #include 
|
| int
| main (void)
| {
|
|   ;
|   return 0;
| };
| MODULE_LICENSE("GPL");
configure:10746: cp conftest.c build && make -d 
_module_/usr/src/lustre-2.12.6/build LUSTRE_KERNEL_TEST=conftest.i LDFLAGS= 
LD=/usr/bin/ld -m elf_x86_64 CC=gcc -f /usr/src/lustre-2.12.6/build/Makefile 
LUSTRE_LINUX_CONFIG=/usr/src/linux-5.10.114/.config LINUXINCLUDE= 
-I/usr/src/linux-5.10.114/arch/x86/include -Iinclude 
-Iarch/x86/include/generated 

Re: [lustre-discuss] project quota problem on existing directories from filesystem created on zfs 0.7 pool

[Degremont, Aurelien via lustre-discuss]

Hi

I'm not specialist of project quota, but I have a more generic comment.
I see you said you upgraded to 2.14.58? Is that a version you pick on purpose?

2.14.58 is not intended for production at all. This is an alpha version of what 
will be Lustre 2.15. 

If you want a production-compatible version you should use 2.12.8 or 2.14.0.
Never pick a version where last digit is > 50. That means "development release".


Aurélien

Le 14/01/2022 15:12, « lustre-discuss au nom de Chen Wei via lustre-discuss » 
 a écrit :

CAUTION: This email originated from outside of the organization. Do not 
click links or open attachments unless you can confirm the sender and know the 
content is safe.



Hi,

We have a lustre filesystem created by Lustre 2.12.8 on zfs 0.7.13 pool.
Since "Project quotas are not supported on zfs versions earlier than
0.8", it has been recently upgraded to Lustre 2.14.58 with zfs 2.0.7.
After upgrade the zfs pool to enable project quota feature and enable
project quota in Lustre, project quota works on new directory. However,
for existing directories, set project quota command fail with:

# lfs project -p 3000 andy
lfs: failed to set xattr for 'andy': No such device or address

Strace of above command:

...
open("andy", O_RDONLY|O_NOCTTY|O_NONBLOCK) = 3
ioctl(3, 0x801c581f, 0x7ffe6fa35d30)= 0
ioctl(3, 0x401c5820, 0x7ffe6fa35d30)= -1 ENXIO (No such device or 
address)
write(2, "lfs: failed to set xattr for 'an"..., 63lfs: failed to set xattr 
for 'andy': No such device or address
) = 63
close(3)= 0
exit_group(1)   = ?


Is there a way to enable project quota on existing directories?


Thanks


--
Wei Chen
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] build help with 5.15 kernel

[Degremont, Aurelien via lustre-discuss]

Hello Michael

Lustre 2.12.8 does not support Linux 5.15 
More recent Lustre versions support up to Linux 5.11 but not further.
See these tickets for 5.12 and 5.14 support.
https://jira.whamcloud.com/browse/LU-14651
https://jira.whamcloud.com/browse/LU-15220

It is possible to manually backport patches to support some 5.x kernels with 
2.12 but this is not trivial. 

I don't know what is your current project but it will be much easier for you if 
you can target an older Linux kernel and focus on kernel used by major distro. 
Ubuntu 20.04 is using 5.11 by example.


Aurélien


Le 31/12/2021 01:34, « lustre-discuss au nom de Hebenstreit, Michael via 
lustre-discuss »  a écrit :

CAUTION: This email originated from outside of the organization. Do not 
click links or open attachments unless you can confirm the sender and know the 
content is safe.



Some additional info. I extracted the make command and ran it against the 2 
kernel versions. Old kernel works, new kernel fails

$ make _module_/tmp/lustre-2.12.5/build LUSTRE_KERNEL_TEST=conftest.i 
LDFLAGS=' LD=/usr/bin/ld -m elf_x86_64' CC=gcc   -f 
/tmp/lustre-2.12.5/build/Makefile 
LUSTRE_LINUX_CONFIG=/admin/src/4.18.0-240.22.1.el8_3.crt6.x86_64.withib//.config
 
LINUXINCLUDE='-I/admin/src/4.18.0-240.22.1.el8_3.crt6.x86_64.withib//arch/x86/include
 -Iinclude -Iarch/x86/include/generated 
-I/admin/src/4.18.0-240.22.1.el8_3.crt6.x86_64.withib//include -Iinclude2 
-I/admin/src/4.18.0-240.22.1.el8_3.crt6.x86_64.withib//include/uapi 
-Iinclude/generated 
-I/admin/src/4.18.0-240.22.1.el8_3.crt6.x86_64.withib//arch/x86/include/uapi 
-Iarch/x86/include/generated/uapi 
-I/admin/src/4.18.0-240.22.1.el8_3.crt6.x86_64.withib//include/uapi 
-Iinclude/generated/uapi -include 
/admin/src/4.18.0-240.22.1.el8_3.crt6.x86_64.withib//include/linux/kconfig.h' 
-o tmp_include_depends -o scripts -o include/config/MARKER -C 
/admin/src/4.18.0-240.22.1.el8_3.crt6.x86_64.withib/ 
EXTRA_CFLAGS='-Werror-implicit-function-declaration -g 
-I/tmp/lustre-2.12.5/libcfs/include -I/tmp/lustre-2.12.5/lnet/include 
-I/tmp/lustre-2.12.5/lustre/include/uapi -I/tmp/lustre-2.12.5/lustre/include 
-Wno-format-truncation -Wno-stringop-truncation -Wno-stringop-overflow' 
M=/tmp/lustre-2.12.5/build
make: Entering directory 
'/global/panfs01/admin/src/4.18.0-240.22.1.el8_3.crt6.x86_64.withib'
  CC [M]  /tmp/lustre-2.12.5/build/conftest.o
  CPP [M] /tmp/lustre-2.12.5/build/conftest.i
make: Leaving directory 
'/global/panfs01/admin/src/4.18.0-240.22.1.el8_3.crt6.x86_64.withib'

$ touch /tmp/lustre-2.12.5/build/conftest.c
$ make _module_/tmp/lustre-2.12.5/build LUSTRE_KERNEL_TEST=conftest.i 
LDFLAGS=' LD=/usr/bin/ld -m elf_x86_64' CC=gcc   -f 
/tmp/lustre-2.12.5/build/Makefile 
LUSTRE_LINUX_CONFIG=/global/panfs01/admin/src/5.15.0-spr.bkc.pc.1.21.0.x86_64/.config
 
LINUXINCLUDE='-I/global/panfs01/admin/src/5.15.0-spr.bkc.pc.1.21.0.x86_64/arch/x86/include
 -Iinclude -Iarch/x86/include/generated 
-I/global/panfs01/admin/src/5.15.0-spr.bkc.pc.1.21.0.x86_64/include -Iinclude2 
-I/global/panfs01/admin/src/5.15.0-spr.bkc.pc.1.21.0.x86_64/include/uapi 
-Iinclude/generated 
-I/global/panfs01/admin/src/5.15.0-spr.bkc.pc.1.21.0.x86_64/arch/x86/include/uapi
 -Iarch/x86/include/generated/uapi 
-I/global/panfs01/admin/src/5.15.0-spr.bkc.pc.1.21.0.x86_64/include/uapi 
-Iinclude/generated/uapi -include 
/global/panfs01/admin/src/5.15.0-spr.bkc.pc.1.21.0.x86_64/include/linux/kconfig.h'
 -o tmp_include_depends -o scripts -o include/config/MARKER -C 
/global/panfs01/admin/src/5.15.0-spr.bkc.pc.1.21.0.x86_64 
EXTRA_CFLAGS='-Werror-implicit-function-declaration -g 
-I/tmp/lustre-2.12.5/libcfs/include -I/tmp/lustre-2.12.5/lnet/include 
-I/tmp/lustre-2.12.5/lustre/include/uapi -I/tmp/lustre-2.12.5/lustre/include 
-Wno-format-truncation -Wno-stringop-truncation -Wno-stringop-overflow' 
M=/tmp/lustre-2.12.5/build
make: Entering directory 
'/global/panfs01/admin/src/5.15.0-spr.bkc.pc.1.21.0.x86_64'
make: *** No rule to make target '_module_/tmp/lustre-2.12.5/build'.  Stop.
make: Leaving directory 
'/global/panfs01/admin/src/5.15.0-spr.bkc.pc.1.21.0.x86_64'

-Original Message-
From: lustre-discuss  On Behalf Of 
Hebenstreit, Michael via lustre-discuss
Sent: Thursday, December 30, 2021 5:07 PM
To: lustre-discuss@lists.lustre.org
Subject: [lustre-discuss] build help with 5.15 kernel

Hello

I'm trying to build the Lustre 2.12.8 client on a 5.15 kernel and already 
failing in the configure step. Looks to me like something in the build process 
has changed. The failure occurs in configure line 14390. From the log:

configure:14390: cp conftest.c build && make -d 
_module_/tmp/lustre-2.12.8/build LUSTRE_KERNEL_TEST=conftest.i LDFLAGS= 
LD=/usr/bin/ld -m elf_x86_64 CC=gcc -f
make: *** No rule to make target '_module_/tmp/lustre-2.12.8/build'.  Stop.
configure:14393: $? = 2

For some reasons the construct "make -d _module_/${PWD} .." does 

Re: [lustre-discuss] unmount FS when endpoint is gone

[Degremont, Aurelien via lustre-discuss]

Hello Florin,

As the filesystem servers do not exist anymore as you deleted it previously, 
the client could not reach them to complete the unmount process.

Try unmounting them using '-f' flag, ie: 'umount -f '


You should also reach out to AWS support and check that with them.

Aurélien



Le 21/12/2021 00:54, « lustre-discuss au nom de Florin Andrei » 
 a 
écrit :

CAUTION: This email originated from outside of the organization. Do not 
click links or open attachments unless you can confirm the sender and know the 
content is safe.



We've created a few Lustre FS endpoints in AWS. They were mounted on a
system. The Lustre endpoints got terminated soon after that, and others
were created instead.

Now the old Lustre filesystems appear to be mounted on that node, and
there's automation trying to unmount them, resulting in a very large
number of umount processes just hanging. In dmesg I see this message
repeated many, many times:

Lustre: 919:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request
sent has failed due to network error:

What is the recommended procedure to unmount those FSs? Just running
umount manually also hangs indefinitely. I would prefer to not reboot
that node.

--
Florin Andrei
https://florin.myip.org/
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] eviction timeout

[Degremont, Aurelien via lustre-discuss]

Hello

This message is appearing during MDT recovery, likely after a MDS restart. MDT 
tries to reconnect first all existing client when it stopped.
It seems all these clients have been also rebooted. To avoid this message, try 
to stop your clients before the servers.

If not possible, you can abort the recovery, either at start time 
(https://doc.lustre.org/lustre_manual.xhtml#lustremaint.abortRecovery) or when 
recovery is running with the following commands on the MDS host:

lctl --device lustre-MDT abort_recovery


Aurélien

De : lustre-discuss  au nom de Sid 
Young via lustre-discuss 
Répondre à : Sid Young 
Date : lundi 11 octobre 2021 à 03:16
À : lustre-discuss 
Objet : [EXTERNAL] [lustre-discuss] eviction timeout


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.



I'm seeing a lot of these messages:

Oct 11 11:12:09 hpc-mds-02 kernel: Lustre: lustre-MDT: Denying connection 
for new client b6df7eda-8ae1-617c-6ff1-406d1ffb6006 (at 10.140.90.82@tcp), 
waiting for 6 known clients (0 recovered, 0 in progress, and 0 evicted) to 
recover in 2:42

It seems to be a 3minute timeout, is it possible to shorten this and even not 
log this message?

Sid Young

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] how to optimize write performances

[Degremont, Aurelien via lustre-discuss]

Hello

Direct I/O is impacting the whole I/O path, from client down to ZFS. Agreed ZFS 
does not support it, but all the rest of I/O path is.

Could you provide you fio command line?
As I said, you need to do _large I/O_ of multiple MB size. If you are just 
doing 1 MB I/O (assuming stripesize is 1MB), you application will just send 1 
RPC at a time to 1 OST, wait for the reply and send the next one. The client 
cache will help at the beginning, until it is full (32MB max_dirty_mb per OST 
by default). 
What about rpc_stats?


Aurélien

Le 04/10/2021 18:32, « Riccardo Veraldi »  a 
écrit :

CAUTION: This email originated from outside of the organization. Do not 
click links or open attachments unless you can confirm the sender and know the 
content is safe.



Hello Aurelien,

I Am using ZFS as lustre backend. ZFS does not support direct I/O.

Ok Lustre does but anyway the performance with direct I/O are worse when
using ZFS backend at least during my tests.

Best

Riccardo


On 10/1/21 2:22 AM, Degremont, Aurelien wrote:
> Hello
>
> To achieve higher throughput with a single threaded process, you should 
try to limit latencies and parallelize under the hood.
> Try checking the following parameters:
> - Stripe your file across multiple OSTs
> - Do large I/O, multiple MB per write, to let Lustre send multiple RPC to 
different OSTs
> - Try testing with and without Direct I/O.
>
> What is your 'dd' test command?
> Clear and check rpc stats (sudo lctl set_param osc.*.rpc_stats=clear; 
sudo lctl get_param osc.*.rpc_stats). Check you are sending large RPCs (pages 
per rpc).
>
> Aurélien
>
> Le 30/09/2021 18:11, « lustre-discuss au nom de Riccardo Veraldi » 
 a écrit :
>
>  CAUTION: This email originated from outside of the organization. Do 
not click links or open attachments unless you can confirm the sender and know 
the content is safe.
>
>
>
>  Hello,
>
>  I wanted to ask some hint on how I may increase single process
>  sequential write performance on Lustre.
>
>  I am using Lustre 2.12.7 on RHEL 7.9
>
>  I have a number of OSSes with SAS SSDs in raidz. 3 OST per oss and 
each
>  OST is made by 8 SSD in raidz.
>
>  On a local test with multiple writes I can write and read from the 
zpool
>  at 7GB/s per OSS.
>
>  With Lustre/ZFS backend I can reach peak writes of 5.5GB/s per OSS 
which
>  is ok.
>
>  This anyway happens only with several multiple writes at once on the
>  filesystem.
>
>  A single write cannot perform more than 800MB-1GB/s
>
>  Changing the underlying hardware and moving to MVMe slightly improve
>  single write performance but just slightly.
>
>  What is preventing a single write pattern to perform better ? They 
are
>  XTC files.
>
>  Each single SSD has a 500MB/s write capability by factory specs. So
>  seems like that with a single write it is not possible to take 
advantage
>  of the
>
>  zpool parallelism. I tried also striping but that does not really 
help much.
>
>  Any hint is really appreciated.
>
>  Best
>
>  Riccardo
>
>
>
>  ___
>  lustre-discuss mailing list
>  lustre-discuss@lists.lustre.org
>  http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] how to optimize write performances

[Degremont, Aurelien via lustre-discuss]

Hello

To achieve higher throughput with a single threaded process, you should try to 
limit latencies and parallelize under the hood.
Try checking the following parameters:
- Stripe your file across multiple OSTs
- Do large I/O, multiple MB per write, to let Lustre send multiple RPC to 
different OSTs
- Try testing with and without Direct I/O.

What is your 'dd' test command?
Clear and check rpc stats (sudo lctl set_param osc.*.rpc_stats=clear; sudo lctl 
get_param osc.*.rpc_stats). Check you are sending large RPCs (pages per rpc).

Aurélien

Le 30/09/2021 18:11, « lustre-discuss au nom de Riccardo Veraldi » 
 a écrit :

CAUTION: This email originated from outside of the organization. Do not 
click links or open attachments unless you can confirm the sender and know the 
content is safe.



Hello,

I wanted to ask some hint on how I may increase single process
sequential write performance on Lustre.

I am using Lustre 2.12.7 on RHEL 7.9

I have a number of OSSes with SAS SSDs in raidz. 3 OST per oss and each
OST is made by 8 SSD in raidz.

On a local test with multiple writes I can write and read from the zpool
at 7GB/s per OSS.

With Lustre/ZFS backend I can reach peak writes of 5.5GB/s per OSS which
is ok.

This anyway happens only with several multiple writes at once on the
filesystem.

A single write cannot perform more than 800MB-1GB/s

Changing the underlying hardware and moving to MVMe slightly improve
single write performance but just slightly.

What is preventing a single write pattern to perform better ? They are
XTC files.

Each single SSD has a 500MB/s write capability by factory specs. So
seems like that with a single write it is not possible to take advantage
of the

zpool parallelism. I tried also striping but that does not really help much.

Any hint is really appreciated.

Best

Riccardo



___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Full OST

[Degremont, Aurelien via lustre-discuss]

print-fid
>>> Print the FID with the path.
>>>
>>>  -c, --print-link
>>> Print the current link number with each pathname or parent 
directory.
>>>
>>>  -l, --link=LINK
>>> If a file has multiple hard links, then print only the 
specified LINK, starting at link 0.
>>> If multiple FIDs are given, but only one pathname is needed 
for each file, use --link=0.
>>>
>>> EXAMPLES
>>>  $ lfs fid2path /mnt/testfs [0x20403:0x11f:0x0]
>>> /mnt/testfs/etc/hosts
>>>
>>>
>>> On Sep 3, 2021, at 14:51, Alastair Basden 
mailto:a.g.bas...@durham.ac.uk>> wrote:
>>>
>>> Hi,
>>>
>>> lctl get_param mdt.*.exports.*.open_files  returns:
>>> mdt.snap8-MDT.exports.172.18.180.21@o2ib.open_files=
>>> [0x2b90e:0x10aa:0x0]
>>> mdt.snap8-MDT.exports.172.18.180.22@o2ib.open_files=
>>> [0x2b90e:0x21b3:0x0]
>>> mdt.snap8-MDT.exports.172.18.181.19@o2ib.open_files=
>>> [0x2b90e:0x21b3:0x0]
>>> [0x2b90e:0x21b4:0x0]
>>> [0x2b90c:0x1574:0x0]
>>> [0x2b90c:0x1575:0x0]
>>> [0x2b90c:0x1576:0x0]
>>>
>>> Doesn't seem to be many open, so I don't think it's a problem of open 
files.
>>>
>>> Not sure which bit of this I need to use with lfs fid2path either...
>>>
>>> Cheers,
>>> Alastair.
>>>
>>>
>>> On Fri, 3 Sep 2021, Andreas Dilger wrote:
>>>
>>> [EXTERNAL EMAIL]
>>> You can also check "mdt.*.exports.*.open_files" on the MDTs for a list 
of FIDs open on each client, and use "lfs fid2path" to resolve them to a 
pathname.
>>>
>>> On Sep 3, 2021, at 02:09, Degremont, Aurelien via lustre-discuss 
mailto:lustre-discuss@lists.lustre.org><mailto:lustre-discuss@lists.lustre.org>>
 wrote:
>>>
>>> Hi
>>>
>>> It could be a bug, but most of the time, this is due to an 
open-unlinked file, typically a log file which is still in use and some 
processes keep writing to it until it fills the OSTs it is using.
>>>
>>> Look for such files on your clients (use lsof).
>>>
>>> Aurélien
>>>
>>>
>>> Le 03/09/2021 09:50, « lustre-discuss au nom de Alastair Basden » 
mailto:lustre-discuss-boun...@lists.lustre.org><mailto:lustre-discuss-boun...@lists.lustre.org>
 au nom de 
a.g.bas...@durham.ac.uk<mailto:a.g.bas...@durham.ac.uk><mailto:a.g.bas...@durham.ac.uk>>
 a écrit :
>>>
>>> CAUTION: This email originated from outside of the organization. Do not 
click links or open attachments unless you can confirm the sender and know the 
content is safe.
>>>
>>>
>>>
>>> Hi,
>>>
>>> We have a file system where each OST is a single SSD.
>>>
>>> One of those is reporting as 100% full (lfs df -h /snap8):
>>> snap8-OST004d_UUID  5.8T2.0T3.5T  37% 
/snap8[OST:77]
>>> snap8-OST004e_UUID  5.8T5.5T7.5G 100% 
/snap8[OST:78]
>>> snap8-OST004f_UUID  5.8T2.0T3.4T  38% 
/snap8[OST:79]
>>>
>>> However, I can't find any files on it:
>>> lfs find --ost snap8-OST004e /snap8/
>>> returns nothing.
>>>
>>> I guess that it has filled up, and that there is some bug or other that 
is
>>> now preventing proper behaviour - but I could be wrong.
>>>
>>> Does anyone have any suggestions?
>>>
>>> Essentially, I'd like to find some of the files and delete or migrate
>>> some, and thus return it to useful production.
>>>
>>> Cheers,
>>> Alastair.
>>> ___
>>> lustre-discuss mailing list
>>> 
lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org><mailto:lustre-discuss@lists.lustre.org>
>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>>
>>> ___
>>> lustre-discuss mailing list
>>> 
lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org><mailto:lustre-discuss@lists.lustre.org>
>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>>
>>> Cheers, Andreas
>>> --
>>> Andreas Dilger
>>> Lustre Principal Architect
>>> Whamcloud
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Cheers, Andreas
>>> --
>>> Andreas Dilger
>>> Lustre Principal Architect
>>> Whamcloud
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] OST mount issue

[Degremont, Aurelien via lustre-discuss]

I see that you are using dkms version of lustre module.
I would check that, it is possible that the dkms build trigger a client-only 
build of Lustre ? (missing deps, etc...)
Could you check the list of Lustre modules built and installed by dkms between 
the 2 working OSSes and the 4 others?


Aurélien

Le 26/04/2021 18:27, « Steve Thompson »  a écrit :

CAUTION: This email originated from outside of the organization. Do not 
click links or open attachments unless you can confirm the sender and know the 
content is safe.



On Mon, 26 Apr 2021, Degremont, Aurelien wrote:

> Could you provide more debugging information, like 'rpm -qa | grep 
lustre' on both hosts?
> The actual mount command, etc...
>
> There must be something different, as the result is different...

Yes, I believe that something must be different; I just cannot find it. I
now have six OST systems. All were installed the same way; two work fine
and four do not. The rpm list:

# rpm -qa | grep lustre
lustre-osd-zfs-mount-2.12.6-1.el7.x86_64
lustre-2.12.6-1.el7.x86_64
lustre-zfs-dkms-2.12.6-1.el7.noarch

# the mount command example:
# grep lustre /etc/fstab
fs1/ost1/mnt/fs1/ost1   lustre defaults,_netdev_  0 0

and all are the same on all six systems. I currently have ZFS 0.8.5
installed, but I have tried with ZFS 0.7.13, and the results are
the same.

Steve
--

Steve Thompson E-mail:  smt AT vgersoft DOT com
Voyager Software LLC   Web: http://www DOT vgersoft DOT com
3901 N Charles St  VSW Support: support AT vgersoft DOT com
Baltimore MD 21218
   "186,282 miles per second: it's not just a good idea, it's the law"


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] OST mount issue

[Degremont, Aurelien via lustre-discuss]

Could you provide more debugging information, like 'rpm -qa | grep lustre' on 
both hosts?
The actual mount command, etc...

There must be something different, as the result is different...



Le 26/04/2021 16:25, « Steve Thompson »  a écrit :

CAUTION: This email originated from outside of the organization. Do not 
click links or open attachments unless you can confirm the sender and know the 
content is safe.



On Mon, 26 Apr 2021, Degremont, Aurelien wrote:

> Le 26/04/2021 09:34, ? Degremont, Aurelien ?  a 
?crit :
>
>This message appears when you are using Lustre modules built with
>only client support, with server support disabled. This message is
>quite new and only appears in very recent Lustre releases.
>
> Actually I double-checked that and this is not true. It does exist with 
Lustre 2.12.6
>
> Just looks like you installed a client-only version of Lustre RPMs...

Unfortunately this isn't true. I installed the RPM's on the working and
non-working systems from the same place:

[lustre-server]
name=lustre-server

baseurl=https://downloads.whamcloud.com/public/lustre/latest-release/el7/server
gpgcheck=0

Steve
--

Steve Thompson E-mail:  smt AT vgersoft DOT com
Voyager Software LLC   Web: http://www DOT vgersoft DOT com
3901 N Charles St  VSW Support: support AT vgersoft DOT com
Baltimore MD 21218
   "186,282 miles per second: it's not just a good idea, it's the law"


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] OST mount issue

[Degremont, Aurelien via lustre-discuss]

Le 26/04/2021 09:34, « Degremont, Aurelien »  a écrit :


This message appears when you are using Lustre modules built with only 
client support, with server support disabled.
This message is quite new and only appears in very recent Lustre releases.

Actually I double-checked that and this is not true. It does exist with Lustre 
2.12.6

Just looks like you installed a client-only version of Lustre RPMs...

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] OST mount issue

[Degremont, Aurelien via lustre-discuss]

Hello Steve,

This message appears when you are using Lustre modules built with only client 
support, with server support disabled.
This message is quite new and only appears in very recent Lustre releases. What 
Lustre version are you using, this error does not exist in 2.12.6 as far as I 
know.

Could you double check the Lustre version and RPMs you installed on that host, 
compare with the other and ensure they are the same?
Could you simply try: 'modprobe mdt' and see if that works?

Check also: 'lctl get_param version'

Aurélien

Le 25/04/2021 15:10, « lustre-discuss au nom de Steve Thompson » 
 a écrit :

CAUTION: This email originated from outside of the organization. Do not 
click links or open attachments unless you can confirm the sender and know the 
content is safe.



Two CentOS 7.9 hosts with kernel 3.10.0-1160.21.1.el7.x86_64, using ZFS
0.8.5 with lustre 2.12.6. One system works perfectly, whereas the second
host fails to mount the OST, with this message from the mount:

mount.lustre: mount fs1/ost1 at /mnt/fs1/ost1 failed: Invalid argument

and this confusing message in the log:

Apr 25 08:56:18 fs1 kernel: LustreError: 
27722:0:(obd_mount.c:1597:lustre_fill_super())
This is client-side-only module, cannot handle server mount.

Can someone please point me to the error? BTW, the two systems were both
kickstarted from the same repo, so as far as I can tell, they are
identical. The lustre RPMs were installed from the server repo.

Steve

--

Steve Thompson E-mail:  smt AT vgersoft DOT com
Voyager Software LLC   Web: http://www DOT vgersoft DOT com
3901 N Charles St  VSW Support: support AT vgersoft DOT com
Baltimore MD 21218
   "186,282 miles per second: it's not just a good idea, it's the law"

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] servicenode /failnode

[Degremont, Aurelien via lustre-discuss]

You're totally correct!


De : lustre-discuss  au nom de Sid 
Young via lustre-discuss 
Répondre à : Sid Young 
Date : vendredi 26 février 2021 à 00:45
À : lustre-discuss 
Objet : [EXTERNAL] [lustre-discuss] servicenode /failnode


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.


G'Day all,

I'm rebuilding my Lustre cluster again and in doing so I am trying to 
understand the role of the --servicenode option when creating an OST. There is 
an example in the doco shown as this:

[root@rh7z-oss1 system]# mkfs.lustre --ost \

>   --fsname demo \

>   --index 0 \

>   --mgsnode 192.168.227.11@tcp1 \

>   --mgsnode 192.168.227.12@tcp1 \

>   --servicenode 192.168.227.21@tcp1 \

>   --servicenode 192.168.227.22@tcp1 \

>   /dev/dm-3
But its not clear what the service node actually is.

Am I correct in saying the service nodes are the IP's of the two OSS servers 
that can manage this particular OST (the HA pair)?



Sid Young

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] need to always manually add network after reboot

[Degremont, Aurelien via lustre-discuss]

Hello

If I understand correctly, you're telling that you have 2 configuration files:

/etc/modprobe.d/lnet.conf
options lnet networks=tcp

[root@hpc-oss-03 ~]# cat /etc/modprobe.d/lustre.conf
options lnet networks="tcp(ens2f0)"
options lnet ip2nets="tcp(ens2f0) 10.140.93.*

That means you are declaring twice the "networks" option for "lnet" kernel 
module. I don't know how 'modprobe' will behave regarding that.
If you have a very simple configuration, where your nodes only have one 
Ethernet interface "ens2f0", you only need the following lines, from the 3 
above:

options lnet networks="tcp(ens2f0)"

If this interface is the only Ethernet interface on your host, you don't even 
need a network specific setup. By default, when loading Lustre, in the absence 
of a network configuration, Lustre will automatically setup the only ethernet 
interface to use it for "tcp".

Aurélien


De : lustre-discuss  au nom de Sid 
Young via lustre-discuss 
Répondre à : Sid Young 
Date : mardi 23 février 2021 à 06:59
À : lustre-discuss 
Objet : [EXTERNAL] [lustre-discuss] need to always manually add network after 
reboot


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.



G'Day all,
I'm finding that when I reboot any node in our new HPC, I need to keep manually 
adding the network using lnetctl net add --net tcp --if ens2f0
Then I can do an lnetctl net show and see the tcp part active...

I have options in  /etc/modprobe.d/lnet.conf
options lnet networks=tcp

and

[root@hpc-oss-03 ~]# cat /etc/modprobe.d/lustre.conf
options lnet networks="tcp(ens2f0)"
options lnet ip2nets="tcp(ens2f0) 10.140.93.*

I've read the doco and tried to understand the correct parameters for a simple 
Lustre config so this is what I worked out is needed... but I suspect its still 
wrong.

Any help appreciated :)



Sid Young

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Determining server version from client

[Degremont, Aurelien]

Hi Andrew,

There are no such things as a filesystem version, as a filesystem is made of 
MDTs, OSTs and clients. Each of them could have different Lustre versions (even 
if running too much different lustre versions between MDT and OST is not really 
supported).
So, to get the Lustre version, you should check server versions, with this 
command by example:

$ lctl get_param *.*.import | grep target_version

The local Lustre client version could be read with


$ lctl get_param version


Aurélien

Le 19/01/2021 05:56, « lustre-discuss au nom de Andrew Elwell » 
 a 
écrit :

CAUTION: This email originated from outside of the organization. Do not 
click links or open attachments unless you can confirm the sender and know the 
content is safe.



Hi All,

Is there a trivial command to determine the server side version of
lustre (in my case, trying to confirm what types of quotas are allowed
(project - 2.10+, default - 2.12+) ?

I was hoping there'd be something in lfs, such as lfs getname
--version which would ideally spit out something like

$ lfs getname --version
fs1-9920dde7d000 /fs1 2.10.4
testfs-992073597800 /testfs 2.12.5

but that's wishful thinking :-) as lfs --version merely gives me the
client version as expected

Is this something that's fairly trivial and I'll open a jira ticket
for the request - I know it's done at mount time as the kernel can log
kernel: Lustre: Server MGS version (2.5.1.0) is much older than
client. Consider upgrading server (2.12.5)

Many thanks

Andrew
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Robinhood scan time

[Degremont, Aurelien]

There could be lots of difference between these 2 systems.
- What is the backend FS type? (ZFS or LDiskfs)
- How many MDT do you have?
- Is 2 threads enough to maximum your scan throughput? Stephane said he used 4 
and 8 of them.
- What the workload running on the MDT at the same time, is it overloaded 
already by your users' jobs?

Robinhood is also dumping its pipeline stats regularly in the logs. You can 
spot which step of the pipeline is slowing you down. 

Aurélien

Le 07/12/2020 20:59, « Kumar, Amit »  a écrit :


Hi Stephane & Aurélien

Here are the stats that I see in my logs:

Below is the best and worst avg. speed I noted in the log, with 
nb_threads_scan=2:
2020/11/03 16:51:04 [4850/3] STATS |  avg. speed  (effective):
618.32 entries/sec (3.23 ms/entry/thread)
2020/11/25 18:06:10 [4850/3] STATS |  avg. speed  (effective):
187.93 entries/sec (10.62 ms/entry/thread)

Finally the full scan results are below:
2020/11/25 17:13:41 [4850/4] FS_Scan | Full scan of /scratch completed, 
369729104 entries found (123 errors). Duration = 1964257.21s

Stephane, now I wonder what could have caused poor scanning performance. 
Once I kicked off my initial scan during the LAD with same number of threads(2) 
my scan along with some users jobs in the following days caused opening and 
closing of file 150-200 million file operations and as a result filled up my 
change log too soon than I expected.  I had to cancel the first initial scan to 
bring the situation under control. After I cleared change log, I asked 
Robinhood to perform a new full scan. I am not sure if this cancel and restart 
could have caused delays with additional lookup into database for existing 
entries of already scanned 200millions files by then? Other thing your point 
out is you have RAID10 SSD, on our end I have RAID-5 3.6TB of SSD's, this 
probably explains the slowness?

I wasn't sure of the impact of the scan hence chose only 2 threads, I am 
guessing I could bump that up to 4 next times to see if the benefits my scan 
times.

Thank you,
Amit

-Original Message-
From: Stephane Thiell 
Sent: Monday, December 7, 2020 11:43 AM
To: Degremont, Aurelien 
Cc: Kumar, Amit ; Russell Dekema ; 
lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] Robinhood scan time

Hi Amit,

Your number is very low indeed.

At our site, we're seeing ~100 million files/day during a Robinhood scan 
with nb_threads_scan =4 and on hardware using Intel based CPUs:

2020/11/16 07:29:46 [126653/2] STATS |  avg. speed  (effective):   
1207.06 entries/sec (3.31 ms/entry/thread)

2020/11/16 07:31:44 [126653/29] FS_Scan | Full scan of /oak completed, 
1508197871 entries found (65 errors). Duration = 1249490.23s

In that case, our Lustre MDS and Robinhood server are running all on 2 x 
CPU E5-2643 v3 @ 3.40GHz.
The Robinhood server has 768GB of RAM and 7TB of SSDs in RAID-10 for the DB.

On another filesystem, using AMD Naples -based CPUs and a dedicated 
Robinhood DB, hosted a different server with AMD Rome CPUs, we’re seeing a rate 
of 266M/day during a Robinhood scan with nb_threads_scan = 8:

2020/09/20 21:43:46 [25731/4] FS_Scan | Full scan of /fir completed, 
877905438 entries found (744 errors). Duration = 284564.88s


Best,

Stephane

> On Dec 7, 2020, at 4:49 AM, Degremont, Aurelien  
wrote:
>
> Hi Amit,
>
> Thanks for this data point, that's interesting.
> Robinhood prints a scan summary in its logfile at the end of scan. It 
could be nice if you can copy/paste it, for further reference.
>
> Aurélien
>
> Le 04/12/2020 23:39, « lustre-discuss au nom de Kumar, Amit » 
 a 
écrit :
>
>CAUTION: This email originated from outside of the organization. Do 
not click links or open attachments unless you can confirm the sender and know 
the content is safe.
>
>
>
>Dual Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz;
>256GB RAM
>System x3650 M5
>Storage for MDT is from NetApp EF560.
>
>Best regards,
>Amit
>
>-Original Message-
>From: Russell Dekema 
>Sent: Friday, December 4, 2020 4:27 PM
>To: Kumar, Amit 
>Cc: lustre-discuss@lists.lustre.org
>Subject: Re: [lustre-discuss] Robinhood scan time
>
>Greetings,
>
>What kind of hardware are you running on your metadata array?
>
>Cheers,
>Rusty Dekema
>
>On Fri, Dec 4, 2020 at 5:12 PM Kumar, Amit  
wrote:
>>
>> HI All,
>>
>>
>>
>> During LAD’20 Andreas mentioned if I could share the Robinhood scan time 
for the 369millions files we have. So here it is. It took ~23 days for me to 
c

Re: [lustre-discuss] Robinhood scan time

[Degremont, Aurelien]

Hi Amit,

Thanks for this data point, that's interesting.
Robinhood prints a scan summary in its logfile at the end of scan. It could be 
nice if you can copy/paste it, for further reference.

Aurélien

Le 04/12/2020 23:39, « lustre-discuss au nom de Kumar, Amit » 
 a 
écrit :

CAUTION: This email originated from outside of the organization. Do not 
click links or open attachments unless you can confirm the sender and know the 
content is safe.



Dual Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz;
256GB RAM
System x3650 M5
Storage for MDT is from NetApp EF560.

Best regards,
Amit

-Original Message-
From: Russell Dekema 
Sent: Friday, December 4, 2020 4:27 PM
To: Kumar, Amit 
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] Robinhood scan time

Greetings,

What kind of hardware are you running on your metadata array?

Cheers,
Rusty Dekema

On Fri, Dec 4, 2020 at 5:12 PM Kumar, Amit  wrote:
>
> HI All,
>
>
>
> During LAD’20 Andreas mentioned if I could share the Robinhood scan time 
for the 369millions files we have. So here it is. It took ~23 days for me to 
complete initial scan of all 369 million files, on a dedicated robinhood server 
that has 384GB RAM. I had it setup with all tweaks for database and client that 
was mentioned in Robinhood document. I only used 2 threads for this scan. Hope 
this reference helps.
>
>
>
> Thank you,
>
> Amit
>
>
>
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


IF CLASSIFICATION START

IF CLASSIFICATION END
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lnet routing issue - 2.12.5 client with 2.10.3 server

[Degremont, Aurelien]

This is a known issue, see https://jira.whamcloud.com/browse/LU-11840 and 
https://jira.whamcloud.com/browse/LU-13548

Aurélien

De : lustre-discuss  au nom de Mark 
Lundie 
Date : mardi 1 décembre 2020 à 13:16
À : fırat yılmaz 
Cc : "lustre-discuss@lists.lustre.org" 
Objet : RE: [EXTERNAL] [lustre-discuss] lnet routing issue - 2.12.5 client with 
2.10.3 server


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.


Hi Firat,

Thanks for your reply. Apologies if I am being silly here, but there is no 
route configured for that network. We have the networks tcp (10.110.0.0/16) and 
tcp1 (10.10.0.0/16). The servers have interfaces on both, but the clients only 
have an interface on tcp1. I'm not sure why the client is trying to route to 
10.110.0.21@tcp:

client # mount /net/lustre/
mount.lustre: mount hmeta1@tcp1:hmeta2@tcp1:/lustre at /net/lustre failed: 
Input/output error
Is the MGS running?

hmeta1 resolves to 10.10.0.91, on tcp1.

Thanks,

Mark

From: fırat yılmaz 
Sent: 01 December 2020 11:55
To: Mark Lundie 
Cc: lustre-discuss@lists.lustre.org 
Subject: Re: [lustre-discuss] lnet routing issue - 2.12.5 client with 2.10.3 
server

Hi Mark,

[Tue Dec  1 11:07:55 2020] LNetError: 
2127:0:(lib-move.c:1999:lnet_handle_find_routed_path()) no route to 
10.110.0.21@tcp from 

I would suggest checking  lnetctl routing show and remove the route to  
10.110.0.21@tcp and try to mount.
https://wiki.lustre.org/LNet_Router_Config_Guide



On Tue, Dec 1, 2020 at 2:41 PM Mark Lundie 
mailto:mark.lun...@manchester.ac.uk>> wrote:
Hi all,

I've just run in to an issue mounting on a newly upgraded client running 2.12.5 
with 2.10.3 servers. Just to give some background, we're about to replace our 
existing Lustre storage, but will run it concurrently with the replacement for 
a couple of months. We'll be running 2.12.5 server on the new MDS and OSSs and 
I plan to update all clients to the same version. I would like to avoid 
updating the existing servers though.

The problem is this. The servers have two tcp LNET networks, tcp and tcp1, on 
separate subnets and VLANs. The clients only see tcp1 (a small number are also 
on tcp3, routed via 2 lnet routers), which has been fine until now. With the 
2.12.5 client, however, it is trying to mount from tcp. 2.10.3 to 2.12.5 is 
obviously a bit of a jump, but does anyone have any ideas on what has changed 
and what I could do here please?

meta# lnetctl net show
net:
- net type: lo
  local NI(s):
- nid: 0@lo
  status: up
- net type: tcp
  local NI(s):
- nid: 10.110.0.21@tcp
  status: up
  interfaces:
  0: bond0.22
- net type: tcp1
  local NI(s):
- nid: 10.10.0.91@tcp1
  status: up
  interfaces:
  0: bond0

meta# lnetctl route show
route:
- net: tcp2
  gateway: 10.10.0.254@tcp1
- net: tcp3
  gateway: 10.10.0.254@tcp1


client# lnetctl net show
net:
- net type: lo
  local NI(s):
- nid: 0@lo
  status: up
- net type: o2ib
  local NI(s):
- nid: 10.12.170.47@o2ib
  status: up
  interfaces:
  0: ib0
- net type: tcp1
  local NI(s):
- nid: 10.10.170.47@tcp1
  status: up
  interfaces:
  0: em1

[Tue Dec  1 11:07:55 2020] LNetError: 
2127:0:(lib-move.c:1999:lnet_handle_find_routed_path()) no route to 
10.110.0.21@tcp from 
[Tue Dec  1 11:08:01 2020] LustreError: 
1792:0:(mgc_request.c:249:do_config_log_add()) MGC10.10.0.91@tcp1: failed 
processing log, type 1: rc = -5
[Tue Dec  1 11:08:08 2020] LustreError: 2169:0:(mgc_request.c:599:do_requeue()) 
failed processing log: -5
[Tue Dec  1 11:08:19 2020] LNetError: 
2127:0:(lib-move.c:1999:lnet_handle_find_routed_path()) no route to 
10.110.0.22@tcp from 
[Tue Dec  1 11:08:30 2020] LustreError: 15c-8: MGC10.10.0.91@tcp1: The 
configuration from log 'lustre-client' failed (-5). This may be the result of 
communication errors between this node and the MGS, a bad configuration, or 
other errors. See the syslog for more information.

client# lctl ping 10.10.0.91@tcp1
12345-0@lo
12345-10.110.0.21@tcp
12345-10.10.0.91@tcp1

Any suggestions will be greatly appreciated!

Many thanks,

Mark

Dr Mark Lundie | Research IT Systems Administrator | Research IT | Directorate 
of IT Services | B39, Sackville Street Building | The University of Manchester 
| Manchester | M1 3WE | 0161 275 8403 | ri.itservices.manchester.ac.uk


Working Hours: Tues - Thurs 0730-1730; Fri 0730-1630
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list

Re: [lustre-discuss] lnetctl fails to recreate exact config when importing exported lnet.conf

[Degremont, Aurelien]

Hi Angelos,

Bug reports could be made at  https://jira.whamcloud.com/


Aurélien

Le 04/09/2020 06:11, « lustre-discuss au nom de Angelos Ching » 
 a écrit :

CAUTION: This email originated from outside of the organization. Do not 
click links or open attachments unless you can confirm the sender and know the 
content is safe.



Dear all,

I think I've encountered a bug in lnetctl but not sure where to submit a
bug report:

Summary:
It's expected that the Lnet config on a node can be recreated on
lnet.service start up by saving the config using: lnetctl export
--backup > /etc/lnet.conf
But ordering within ymal file causes extraneous NIDs to be created when
used in combination with routing, thus breaking Lnet routing / node
communication, with server side dmesg showing "Bad dest nid n.n.n.n@o2ib
(it's my nid but on a different network)"

Environment:
Client: CentOS 7.8, Lustre 2.12.5-ib, MLNX OFED 4.9-0.1.7.1
Lnet router + server: CentOS 7.7, Lustre 2.12.4-ib, MLNX OFED 4.7-3.2.9.0

Steps to reproduce:
(Listing 1) Server side Lnet config (peer list omitted for conciseness):
https://pastebin.com/DH6HAt5a
(Listing 2) Full command listing and output on client side is reproduced
here: https://pastebin.com/h3wHyCM7

All steps below carried out on Lustre client:

1. Restart lnet service with empty /etc/lnet.conf
2. lnetctl net add: TCP network using Ethernet
3. lnetctl peer add: 2 peers with "Lnet router + server"@o2ib,tcp NIDs
4. lnetctl route add: 2 gateways to o2ib network using "Lnet router +
server"@TCP NID
5. lnetctl export: with --backup to /etc/lnet.conf; check the saved file
and confirm Lnet is configured with 2 peers and 2 gateways (Listing 2:
37-47)
6. Mount o2ib exported Lustre volume and confirm volume functioning
correctly; unmount volume
7. Restart lnet.service and check lnet configuration; finds 2 extra peer
entries that reference only TCP NID of the "Lnet router + server" along
with 2 manually configured peers that reference both o2ib and tcp NIDs
(Listing 2: 75-93)
8. Client fails to mount o2ib exported volume; server side kernel
message shows "Bad dest nid n.n.n.n@o2ib (it's my nid but on a different
network)"

9. If we reorder the peer list to go before the route list in
/etc/lnet.conf (Listing 2: 16), then lnet would be properly configured
with 2 peers on service restart and everything works as expected.

Best regards,

--
Angelos Ching
ClusterTech Limited

Tel : +852-2655-6138
Fax : +852-2994-2101
Address : Unit 211-213, Lakeside 1, 8 Science Park West Ave., Shatin, Hong 
Kong

Got praises or room for improvements? http://bit.ly/TellAngelos



The information contained in this e-mail and its attachments is 
confidential and
intended solely for the specified addressees. If you have received this 
email in
error, please do not read, copy, distribute, disclose or use any 
information of
this email in any way and please immediately notify the sender and delete 
this
email. Thank you for your cooperation.



___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] some clients dmesg filled up with "dirty page discard"

[Degremont, Aurelien]

Do you mean you've never hit client eviction or dirty page discard before?
What was your previous Lustre version?
Dirty page discard warning exists for a long time, since Lustre 2.4.

Eviction happens for lots of reason. Evictions just mean this client did not 
respond in 100 sec to this OSS. It could be due to network being overloaded, 
hardware issue on one of these hosts, client not responding due to CPU being 
overloaded, or indeed a bug. You should first try to understand why these 
eviction happened.
Verify the server and client load at that time (CPU, network, etc…). Verify the 
impacted files and the application accessing them. What's the application I/O 
pattern? Is it putting a strong pressure on these files? File name could be 
obtained using the FID and 'lfs fid2path MOUNT_POINT FID'.


Aurélien

De : lustre-discuss  au nom de 肖正刚 

Date : dimanche 30 août 2020 à 07:41
À : Andreas Dilger 
Cc : lustre-discuss 
Objet : RE: [EXTERNAL] [lustre-discuss] some clients dmesg filled up with 
"dirty page discard"


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.


Hi, Andreas，
Thanks for your reply.
Maybe this is a bug?
We never hit this before update client to 2.12.5

Andreas Dilger mailto:adil...@whamcloud.com>> 
于2020年8月29日周六 下午6:37写道：
On Aug 25, 2020, at 17:42, 肖正刚 
mailto:guru.nov...@gmail.com>> wrote:

no, on oss we found only the client who reported " dirty page discard  " being 
evicted.
we hit this again last night, and on oss we can see logs like:
"
[Tue Aug 25 23:40:12 2020] LustreError: 
14278:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired 
after 100s: evicting client at 10.10.3.223@o2ib  ns: 
filter-public1-OST_UUID lock: 9f1f91cba880/0x3fcc67dad1c65842 lrc: 
3/0,0 mode: PR/PR res: [0xde2db83:0x0:0x0].0x0 rrc: 3 type: EXT 
[0->18446744073709551615] (req 0->270335) flags: 0x6400020020 nid: 
10.10.3.223@o2ib remote: 0xd713b7b417045252 expref: 7081 pid: 25923 timeout: 
21386699 lvb_type: 0

It isn't clear what the question is here.  The "dirty page discard" message 
means that unwritten data from the client was discarded because the client was 
evicted and the lock covering this data was revoked by the server because the 
client was not responsive.


Anymore , we exec lfsck on all servers,  result is

There is no need for LFSCK in this case.  The file data was not written, but a 
client eviction does not result in the filesystem becoming inconsistent.

Cheers, Andreas
--
Andreas Dilger
Principal Lustre Architect
Whamcloud





___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Lustre client modules

[Degremont, Aurelien]

Hi Phil,

There are conflicts with your already installed Lustre 2.9.0 packages.
Based on the output you provide, you should remove 'kmod-lustre-client-tests' 
first.

Actually, only kmod-lustre-client and lustre-client are the required. You 
probably don't need the other ones (lustre-iokit, lustre-client-debuginfo, ...).

Remove all the other Lustre packages except for these 2 and try again.
 

Aurélien

Le 28/05/2020 10:57, « lustre-discuss au nom de Phill Harvey-Smith » 
 a écrit :

CAUTION: This email originated from outside of the organization. Do not 
click links or open attachments unless you can confirm the sender and know the 
content is safe.



On 27/05/2020 19:26, Leonardo Saavedra wrote:
> On 5/26/20 5:47 PM, Phill Harvey-Smith wrote:
> echo "%_topdir  $HOME/rpmbuild" >> .rpmmacros
> wget -c
> 
https://downloads.whamcloud.com/public/lustre/lustre-2.12.4/el7/client/SRPMS/lustre-2.12.4-1.src.rpm
> rpmbuild --clean  --rebuild --without servers --without lustre_tests
> lustre-2.12.4-1.src.rpm
> cd $HOME/rpmbuild/RPMS/x86_64

Right that worked and I have the following rpms in
$HOME/rpmbuild/RPMS/x86_64 :

# ls
kmod-lustre-client-2.12.4-1.el7.x86_64.rpm
lustre-client-2.12.4-1.el7.x86_64.rpm
lustre-client-debuginfo-2.12.4-1.el7.x86_64.rpm
lustre-iokit-2.12.4-1.el7.x86_64.rpm

However trying to install them with yum I get :

Loaded plugins: fastestmirror, langpacks
Examining kmod-lustre-client-2.12.4-1.el7.x86_64.rpm:
kmod-lustre-client-2.12.4-1.el7.x86_64
Marking kmod-lustre-client-2.12.4-1.el7.x86_64.rpm as an update to
kmod-lustre-client-2.9.0-1.el7.x86_64
Examining lustre-client-2.12.4-1.el7.x86_64.rpm:
lustre-client-2.12.4-1.el7.x86_64
Marking lustre-client-2.12.4-1.el7.x86_64.rpm as an update to
lustre-client-2.9.0-1.el7.x86_64
Examining lustre-client-debuginfo-2.12.4-1.el7.x86_64.rpm:
lustre-client-debuginfo-2.12.4-1.el7.x86_64
Marking lustre-client-debuginfo-2.12.4-1.el7.x86_64.rpm to be installed
Examining lustre-iokit-2.12.4-1.el7.x86_64.rpm:
lustre-iokit-2.12.4-1.el7.x86_64
Marking lustre-iokit-2.12.4-1.el7.x86_64.rpm as an update to
lustre-iokit-2.9.0-1.el7.x86_64
Resolving Dependencies
--> Running transaction check
---> Package kmod-lustre-client.x86_64 0:2.9.0-1.el7 will be updated
--> Processing Dependency: kmod-lustre-client = 2.9.0 for package:
lustre-client-tests-2.9.0-1.el7.x86_64
Loading mirror speeds from cached hostfile
  * base: centos.serverspace.co.uk
  * epel: lon.mirror.rackspace.com
  * extras: centos.serverspace.co.uk
  * updates: centos.mirrors.nublue.co.uk
--> Processing Dependency: ksym(class_find_client_obd) = 0x7fc892aa for
package: kmod-lustre-client-tests-2.9.0-1.el7.x86_64
--> Processing Dependency: ksym(class_name2obd) = 0x2a2fe6c0 for
package: kmod-lustre-client-tests-2.9.0-1.el7.x86_64
--> Processing Dependency: ksym(class_register_type) = 0xc4cc2c4f for
package: kmod-lustre-client-tests-2.9.0-1.el7.x86_64
--> Processing Dependency: ksym(llog_cat_add) = 0xc5e4acf5 for package:
kmod-lustre-client-tests-2.9.0-1.el7.x86_64
--> Processing Dependency: ksym(llog_cat_cancel_records) = 0x72fd39ee
for package: kmod-lustre-client-tests-2.9.0-1.el7.x86_64
--> Processing Dependency: ksym(llog_cat_close) = 0xf83a61a8 for
package: kmod-lustre-client-tests-2.9.0-1.el7.x86_64
--> Processing Dependency: ksym(llog_cat_process) = 0x79b2c569 for
package: kmod-lustre-client-tests-2.9.0-1.el7.x86_64
--> Processing Dependency: ksym(llog_cat_reverse_process) = 0xd7510c21
for package: kmod-lustre-client-tests-2.9.0-1.el7.x86_64
--> Processing Dependency: ksym(llog_cleanup) = 0x0632eadc for package:
kmod-lustre-client-tests-2.9.0-1.el7.x86_64
--> Processing Dependency: ksym(llog_close) = 0xa6f1cf8b for package:
kmod-lustre-client-tests-2.9.0-1.el7.x86_64
--> Processing Dependency: ksym(__llog_ctxt_put) = 0xe1c19687 for
package: kmod-lustre-client-tests-2.9.0-1.el7.x86_64
--> Processing Dependency: ksym(llog_destroy) = 0xe12c11de for package:
kmod-lustre-client-tests-2.9.0-1.el7.x86_64
--> Processing Dependency: ksym(llog_exist) = 0xa6594d74 for package:
kmod-lustre-client-tests-2.9.0-1.el7.x86_64
--> Processing Dependency: ksym(llog_init_handle) = 0xe2107196 for
package: kmod-lustre-client-tests-2.9.0-1.el7.x86_64
--> Processing Dependency: ksym(llog_open) = 0x9ba55f56 for package:
kmod-lustre-client-tests-2.9.0-1.el7.x86_64
--> Processing Dependency: ksym(llog_open_create) = 0xd4bdcea7 for
package: kmod-lustre-client-tests-2.9.0-1.el7.x86_64
--> Processing Dependency: ksym(llog_osd_ops) = 0x034860f6 for package:
kmod-lustre-client-tests-2.9.0-1.el7.x86_64
--> Processing Dependency: ksym(llog_process) = 0x18a1b423 for package:

Re: [lustre-discuss] "no space on device"

[Degremont, Aurelien]

Hi Lana,

Lustre dispatches the data across several servers, MDTs and OSTs. It is likely 
that one of this OST is full.
To see the usage per sub-component, you should check:

lfs df -h
lfs df -ih

See if this reports one OSTs or MDT is full.

Aurélien

De : lustre-discuss  au nom de Lana 
Deere 
Date : jeudi 19 mars 2020 à 19:08
À : "lustre-discuss@lists.lustre.org" 
Objet : [EXTERNAL] [lustre-discuss] "no space on device"


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.


I have a Lustre 2.12 setup running on CentOS 7.  It has been working fine for 
some months but last night one of my users tried to untar a large file, which 
(among other things) created a single directory containing several million 
subdirectories.  At that point the untar failed, reporting "no space on 
device".  All attempts to create a file on this Lustre system now produce the 
same error message, but "df" and "df -i" indicate there is plenty of space and 
inodes left.  I checked the mount point on the metadata node and it appears to 
have plenty of space left too.

I can list directories and view files on this filesystem.  I can delete files 
or directories on it.  But even after removing a few files and a directory I 
cannot create a new file.

If anyone can offer some help here it would be appreciated.

.. Lana (lana.de...@gmail.com)

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lnet_peer_ni_add_to_recoveryq

[Degremont, Aurelien]

What's the impact of being in recovery mode with LNET health?


Le 06/03/2020 21:12, « lustre-discuss au nom de Chris Horn » 
 a écrit :

> lneterror: 10164:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked())
> lpni  added to recovery queue.  Health = 900

The message means that the health value of a remote peer interface has been 
decremented, and as a result, the interface has been put into recovery mode. 
This mechanism is part of the LNet health feature.

Health values are decremented when a PUT or GET fails. Usually there are 
other messages in the log that can tell you more about the specific failure. 
Depending on your network type you should probably see messages from socklnd or 
o2iblnd. Network congestion could certainly lead to message timeouts, which 
would in turn result in interfaces being placed into recovery mode.

Chris Horn

On 3/6/20, 8:59 AM, "lustre-discuss on behalf of Michael Di Domenico" 
 
wrote:

along the aforementioned error i also see these at the same time

lustreerror: 9675:0:(obd_config.c:1428:class_modify_config())
<...>-clilov-<...>; failed to send uevent qos_threshold_rr=100

On Fri, Mar 6, 2020 at 9:39 AM Michael Di Domenico
 wrote:
>
> On Fri, Mar 6, 2020 at 9:36 AM Degremont, Aurelien 
 wrote:
> >
> > Did you see any actual error on your system?
> >
> > Because there is a patch that is just decreasing the verbosity 
level of such messages, which looks like could be ignored.
> > 
https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.whamcloud.com_browse_LU-2D13071=DwICAg=C5b8zRQO1miGmBeVZ2LFWg=hIaFpo9yRyCwkkAs6y0c7W-QqT7uZMMSOkAIByhcA-I=ByOR33WN61jv0rEVZTtNhUgN313iSqbgrdfakY-TAjc=jp8DpDcylEQYlbd9-s3efysfDy2KdLvBrptsplqR1ks=
> > 
https://urldefense.proofpoint.com/v2/url?u=https-3A__review.whamcloud.com_-23_c_37718_=DwICAg=C5b8zRQO1miGmBeVZ2LFWg=hIaFpo9yRyCwkkAs6y0c7W-QqT7uZMMSOkAIByhcA-I=ByOR33WN61jv0rEVZTtNhUgN313iSqbgrdfakY-TAjc=8EUQ5wHRCuFFbd4PKxQCnTB_L9IgffvkzFw4_v6MEHg=
>
> thanks.  it's not entirely clear just yet.  i'm trying to track down a
> "slow jobs" issue.  i see these messages everywhere, so it might be a
> non issue or a sign of something more pressing.
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org

https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddiscuss-2Dlustre.org=DwICAg=C5b8zRQO1miGmBeVZ2LFWg=hIaFpo9yRyCwkkAs6y0c7W-QqT7uZMMSOkAIByhcA-I=ByOR33WN61jv0rEVZTtNhUgN313iSqbgrdfakY-TAjc=d36yZXUxMDJOjluQt2LUPivEkfLhScuCLIQT6Fl-Qhs=





___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lnet_peer_ni_add_to_recoveryq

[Degremont, Aurelien]

Did you see any actual error on your system?

Because there is a patch that is just decreasing the verbosity level of such 
messages, which looks like could be ignored.
https://jira.whamcloud.com/browse/LU-13071
https://review.whamcloud.com/#/c/37718/


Aurélien

Le 06/03/2020 15:17, « lustre-discuss au nom de Michael Di Domenico » 
 a 
écrit :

CAUTION: This email originated from outside of the organization. Do not 
click links or open attachments unless you can confirm the sender and know the 
content is safe.



i'm seeing this error show up on many of my nodes many times (usually
in big spurts).  i suspect i'm having some network congestion issues,
but i haven't narrowed it down yet.  but i'm unclear what the error
actually signifies

lneterror: 10164:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked())
lpni  added to recovery queue.  Health = 900
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] old Lustre 2.8.0 panic'ing continously

[Degremont, Aurelien]

I understand you want to avoid deploying kdump, but you should focus on saving 
your console history somewhere. It will be difficult to help without the panic 
message.

For 2.10, maybe I was a bit optimistic. I think you should be able to build the 
RPMs from sources, but pre-build packages are not available.


Aurélien

Le 05/03/2020 11:47, « Torsten Harenberg »  
a écrit :

Hi Aurélien,

thanks for your quick reply, really appreciate it.

Am 05.03.20 um 10:20 schrieb Degremont, Aurelien:
> - What is the exact error message when the panic happens? Could you 
copy/paste few log messages from this panic message?

a running ssh shell only shows

kernel panic - not synced.

No trace, nothing. I will try to capture the console next time that happens.

> - Did you try searching for this pattern onto jira.whamcloud.com, to see 
if this is an already known bug.

yes.. though not deeply. Was difficult, as kdump is not installed on
that one and I was a bit afraid to deeply touch the system as we bought
it pre-installed and the kdump-tools are not installed and /var/crash is
empty, so I fear kdump was not configured :-( So my only knowledge is
that the kernel panics :-(

> - It seems related to quota. Is disabling quota an option for you?

not easily, the system is always full and enforcing group quotas was our
way so far to keep the system from physically reach 100%. But of course
we could talk to the users and ask them to behave nicely.

> - Lustre 2.10.8 supports CentOS 6 and was a LTS Lustre version, it got a 
lot more fixes and is very stable. It could be an easy upgrade path for you 
before getting your new system?

That would be great!! Where do you find the server RPMs for this
version? I couldn't find them on whamcloud:

https://downloads.whamcloud.com/public/lustre/lustre-2.10.8/

only contains server RPMS for el7 as far as I can see.

Thanks again

 Torsten


-- 
Dr. Torsten Harenberg harenb...@physik.uni-wuppertal.de
Bergische Universitaet
Fakultät 4 - Physik   Tel.: +49 (0)202 439-3521
Gaussstr. 20  Fax : +49 (0)202 439-2811
42097 Wuppertal



___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] old Lustre 2.8.0 panic'ing continously

[Degremont, Aurelien]

Hello Torsten,

- What is the exact error message when the panic happens? Could you copy/paste 
few log messages from this panic message?
- Did you try searching for this pattern onto jira.whamcloud.com, to see if 
this is an already known bug.
- It seems related to quota. Is disabling quota an option for you?
- Lustre 2.10.8 supports CentOS 6 and was a LTS Lustre version, it got a lot 
more fixes and is very stable. It could be an easy upgrade path for you before 
getting your new system?

Aurélien

Le 05/03/2020 08:49, « lustre-discuss au nom de Torsten Harenberg » 
 a écrit :

CAUTION: This email originated from outside of the organization. Do not 
click links or open attachments unless you can confirm the sender and know the 
content is safe.



Dear all,

I know it's dared to ask for help for such an old system.

We still run a CentOS 6 based Lustre 2.8.0 system
(kernel-2.6.32-573.12.1.el6_lustre.x86_64,
lustre-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64).

It's out of warrenty and about to be replaced. The approval process for
the new grant took over a year and we're currently preparing an EU wide
tender, all of that takes and took much more time than we expected.

The problem is:

one OSS server is always running into a kernel panic. It seems that this
kernel panic is related to one of the OSS mount points. If we mount the
LUNs of that server (all data is on a 3par SAN) to a different server,
this one is panic'ing, too.

We always run file system checks after such a panic but these show only
minor issues that you would expect after a crashed machine like

[QUOTA WARNING] Usage inconsistent for ID 2901:actual (757747712, 217)
!= expected (664182784, 215)

We would love to avoid an upgrade to CentOS 7 with these old machines,
but these crashes happen really often meanwhile and yesterday it
panic'ed after only 30mins.

Now we're running out of ideas.

If anyone has an idea how we could identify the source of the problem,
we would really appreciate it.

Kind regards

  Torsten


--
Dr. Torsten Harenberg harenb...@physik.uni-wuppertal.de
Bergische Universitaet
Fakultät 4 - Physik   Tel.: +49 (0)202 439-3521
Gaussstr. 20  Fax : +49 (0)202 439-2811
42097 Wuppertal



___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] LNET ports and connections

[Degremont, Aurelien]

Very nice one! Thanks!
I was able to reproduce and confirm that with the proper sequence, the LNET 
connections will be initiated from server to client 988 port.

Thanks all!

De : Steve Crusan 
Date : vendredi 21 février 2020 à 03:53
À : NeilBrown 
Cc : "Degremont, Aurelien" , 
"lustre-discuss@lists.lustre.org" 
Objet : Re: [lustre-discuss] LNET ports and connections

Can’t you also use tcpkill to kill the connections? I’ve used it to kill 
“stuck” NFS connections before due to MTU related issues, etc.

It’s distributed normally with the dsniff package if you are using an rpm based 
distribution.

-Steve

On Thu, Feb 20, 2020 at 20:39 NeilBrown mailto:ne...@suse.com>> 
wrote:

I haven't tried this, but you might be able to close a socket by:

1/ decrease  /proc/sys/net/ipv4/tcp_keepalive_time
  so that keep-alives get sent sooner.  Maybe set to 60,
  set ..._intvl to 5, and _probes to 3.

2/ create a rule with iptables to drop all messages sent
  on the particular connection.
   iptables -A OUTPUT -m multiport --dports ... -sports .. -j DROP


Given the suggested keep alive settings, you should only have to wait 75
seconds after creating the IP tables rule before the connection is
broken.

NeilBrown


On Thu, Feb 20 2020, Degremont, Aurelien wrote:

> Thanks. It feels like the theory is valid.
> Ideally to confirm I would need a way to manually force close the socklnd 
> socket to force the other peer to re-established it.
> Could not find a way to do it for socket opened by kernel threads.
>
> Le 19/02/2020 23:12, « NeilBrown » mailto:ne...@suse.com>> a 
> écrit :
>
>
> When LNet wants to send a message over a SOCKLND interface,
> ksocknal_launch_packet() is called.
>
> This calls ksocknal_launch_all_connections_locked()
> This loops over all "routes" to the "peer" to make sure they all have
> "connections".
> If it finds a route without a connection (returned by
> ksocknal_find_connectable_route_locked()) it calls
> ksocknal_launch_connection_locked() which adds the connection request to
> ksnd_connd_routes, and wakes up the connd.  The connd thread will then
> make the connection.
>
> Hope that helps.
>
> NeilBrown
>
>
>
> On Wed, Feb 19 2020, Degremont, Aurelien wrote:
>
> > Thanks! That's really interesting.
> > Do you have a code pointer that could show where the code will 
> establish this connection if missing?
> >
> > Le 18/02/2020 23:34, « NeilBrown » 
> mailto:ne...@suse.com>> a écrit :
> >
> >
> > It is not true that:
> >LNET will established connections only if asked for by upper 
> layers.
> >
> > or at least, not in the sense that the upper layers ask for a
> > connection.
> > Lustre knows nothing about connections.  Even LNet doesn't really 
> know
> > about connections. It is only at the socklnd level that connections 
> mean
> > much.
> >
> > Lustre and LNet are message-passing protocols.
> > Lustre asks LNet to send a message to a given peer, and gives some
> > details of the sort of reply to expect.
> > LNet chooses a route and thus a network interface, and asked the 
> LND to
> > send the message.
> > The socklnd LND will see if it already has a TCP connection.  If it
> > does, it will use it.  If not, it will create one.
> >
> > So yes : it is exactly:
> >   possible that the server in this case opens the connection itself
> >   without waiting for the client to reconnect?
> >
> > NeilBrown
> >
> >
> > On Tue, Feb 18 2020, Aurelien Degremont wrote:
> >
> > > Thanks for your reply.
> > > I think I have a good enough understanding of LNET itself. My 
> question was more about how LNET is being used by Lustre itself.
> > >
> > > LNET will established connections only if asked for by upper 
> layers.
> > > When I was talking about client and server, I was talking about 
> how Lustre was using it.
> > >
> > > As far as I understood, Lustre server only contact clients when 
> they need to send LDLM callbacks.
> > > They do so through the socket already opened by the client 
> (reverse import).
> > > What happened if the socket is closed is what I'm not sure. I 
> though the server is rather waiting for the client to reconnect and if not, 
> is more or less evicting it.
> > > Could it b

Re: [lustre-discuss] LNET ports and connections

[Degremont, Aurelien]

Thanks. It feels like the theory is valid.
Ideally to confirm I would need a way to manually force close the socklnd 
socket to force the other peer to re-established it.
Could not find a way to do it for socket opened by kernel threads.

Le 19/02/2020 23:12, « NeilBrown »  a écrit :


When LNet wants to send a message over a SOCKLND interface,
ksocknal_launch_packet() is called.

This calls ksocknal_launch_all_connections_locked()
This loops over all "routes" to the "peer" to make sure they all have
"connections".
If it finds a route without a connection (returned by
ksocknal_find_connectable_route_locked()) it calls
ksocknal_launch_connection_locked() which adds the connection request to
ksnd_connd_routes, and wakes up the connd.  The connd thread will then
make the connection.

Hope that helps.

NeilBrown



On Wed, Feb 19 2020, Degremont, Aurelien wrote:

> Thanks! That's really interesting.
> Do you have a code pointer that could show where the code will establish 
this connection if missing?
>
> Le 18/02/2020 23:34, « NeilBrown »  a écrit :
>
> 
> It is not true that:
>LNET will established connections only if asked for by upper 
layers.
> 
> or at least, not in the sense that the upper layers ask for a
> connection.
> Lustre knows nothing about connections.  Even LNet doesn't really know
> about connections. It is only at the socklnd level that connections 
mean
> much.
> 
> Lustre and LNet are message-passing protocols.
> Lustre asks LNet to send a message to a given peer, and gives some
> details of the sort of reply to expect.
> LNet chooses a route and thus a network interface, and asked the LND 
to
> send the message.
> The socklnd LND will see if it already has a TCP connection.  If it
> does, it will use it.  If not, it will create one.
> 
> So yes : it is exactly:
>   possible that the server in this case opens the connection itself
>   without waiting for the client to reconnect?
> 
> NeilBrown
> 
> 
> On Tue, Feb 18 2020, Aurelien Degremont wrote:
> 
> > Thanks for your reply.
> > I think I have a good enough understanding of LNET itself. My 
question was more about how LNET is being used by Lustre itself.
> >
> > LNET will established connections only if asked for by upper 
layers. 
> > When I was talking about client and server, I was talking about how 
Lustre was using it.
> >
> > As far as I understood, Lustre server only contact clients when 
they need to send LDLM callbacks.
> > They do so through the socket already opened by the client (reverse 
import).
> > What happened if the socket is closed is what I'm not sure. I 
though the server is rather waiting for the client to reconnect and if not, is 
more or less evicting it.
> > Could it be possible that the server in this case opens the 
connection itself without waiting for the client to reconnect?
> >
> >
> > Aurélien
> >
> > Le 18/02/2020 05:42, « NeilBrown »  a écrit :
> >
> > 
> > LNet is a peer-to-peer protocol, it has no concept of client 
and server.
> > If one host needs to send a message to another but doesn't 
already have
> > a connection, it creates a new connection.
> > I don't yet know enough specifics of the lustre protocol to be 
certain
> > of the circumstances when a lustre server will need to initiate 
a message
> > to a client, but I imagine that recalling a lock might be one.
> > 
> > I think you should assume that any LNet node might receive a 
connection
> > from any other LNet node (for which they share an LNet 
network), and
>     >     that the connection could come from any port between 512 and 
1023
> > (LNET_ACCEPTOR_MIN_PORT to LNET_ACCEPTOR_MAX_PORT).
> > 
> > NeilBrown
> > 
> > 
> > 
> > On Mon, Feb 17 2020, Degremont, Aurelien wrote:
> > 
> > > Hi all,
> > >
> > > From what I've understood so far, LNET listens on port 988 by 
default and peers connect to it using 1021-1023 TCP ports as source ports.
> > > At 

Re: [lustre-discuss] LNET ports and connections

[Degremont, Aurelien]

Hi Cory

I'm with 2.10 there and idle disconnect is not available in 2.10.
The steps I'm thinking of so far are:

1. - Client connects, opens a TCP socket to server
2. - Client acquires a LDLM lock
3. - TCP connection gets broken
4. - Soon after a conflicting lock is enqueued. Server needs to cancel the lock 
from Client.
5. - Server tries to send a LDLM callback through LNET
6. - LNET initiates a TCP connection in the other way, as the existing socket 
is no more there.

If time between 2. and 4. is long enough, either the client will have time to 
reconnect (next ping?) or either the server will evict the client because it 
did not get any message from it for a while.

This is difficult to reproduce as I don't know how to force close the socket 
socklnd has opened.


Aurélien

Le 19/02/2020 18:19, « Spitz, Cory James »  a écrit :

Hello, Aurélien.  I'm guessing that if you have modern Lustre then idle 
clients may disconnect, and so you might regularly see Lustre servers initiate 
the socket connection again.   I'm not sure how to show that that it is the 
case or not.  Perhaps someone else can chime in on whether that could be it and 
if so, how to prove it.

-Cory


On 2/19/20, 2:35 AM, "lustre-discuss on behalf of Degremont, Aurelien" 
 
wrote:

Thanks! That's really interesting.
Do you have a code pointer that could show where the code will 
establish this connection if missing?

Le 18/02/2020 23:34, « NeilBrown »  a écrit :


It is not true that:
   LNET will established connections only if asked for by upper 
layers.

or at least, not in the sense that the upper layers ask for a
connection.
Lustre knows nothing about connections.  Even LNet doesn't really 
know
about connections. It is only at the socklnd level that connections 
mean
much.

Lustre and LNet are message-passing protocols.
Lustre asks LNet to send a message to a given peer, and gives some
details of the sort of reply to expect.
LNet chooses a route and thus a network interface, and asked the 
LND to
send the message.
The socklnd LND will see if it already has a TCP connection.  If it
does, it will use it.  If not, it will create one.

So yes : it is exactly:
  possible that the server in this case opens the connection itself
  without waiting for the client to reconnect?

NeilBrown


On Tue, Feb 18 2020, Aurelien Degremont wrote:

> Thanks for your reply.
> I think I have a good enough understanding of LNET itself. My 
question was more about how LNET is being used by Lustre itself.
>
> LNET will established connections only if asked for by upper 
layers. 
> When I was talking about client and server, I was talking about 
how Lustre was using it.
>
> As far as I understood, Lustre server only contact clients when 
they need to send LDLM callbacks.
> They do so through the socket already opened by the client 
(reverse import).
> What happened if the socket is closed is what I'm not sure. I 
though the server is rather waiting for the client to reconnect and if not, is 
more or less evicting it.
> Could it be possible that the server in this case opens the 
connection itself without waiting for the client to reconnect?
>
>
> Aurélien
>
> Le 18/02/2020 05:42, « NeilBrown »  a écrit :
>
> 
> LNet is a peer-to-peer protocol, it has no concept of client 
and server.
> If one host needs to send a message to another but doesn't 
already have
> a connection, it creates a new connection.
> I don't yet know enough specifics of the lustre protocol to 
be certain
> of the circumstances when a lustre server will need to 
initiate a message
> to a client, but I imagine that recalling a lock might be one.
> 
> I think you should assume that any LNet node might receive a 
connection
> from any other LNet node (for which they share an LNet 
network), and
> that the connection could come from any port between 512 and 
1023
> (LNET_ACCEPTOR_MIN_PORT to LNET_ACCEPTOR_MAX_PORT).
> 
> NeilBrown
    > 
> 
> 
> On Mon, Feb 17 2020, Degremont, Aurelien wrote:
> 
   

Re: [lustre-discuss] LNET ports and connections

[Degremont, Aurelien]

Thanks! That's really interesting.
Do you have a code pointer that could show where the code will establish this 
connection if missing?

Le 18/02/2020 23:34, « NeilBrown »  a écrit :


It is not true that:
   LNET will established connections only if asked for by upper layers.

or at least, not in the sense that the upper layers ask for a
connection.
Lustre knows nothing about connections.  Even LNet doesn't really know
about connections. It is only at the socklnd level that connections mean
much.

Lustre and LNet are message-passing protocols.
Lustre asks LNet to send a message to a given peer, and gives some
details of the sort of reply to expect.
LNet chooses a route and thus a network interface, and asked the LND to
send the message.
The socklnd LND will see if it already has a TCP connection.  If it
does, it will use it.  If not, it will create one.

So yes : it is exactly:
  possible that the server in this case opens the connection itself
  without waiting for the client to reconnect?

NeilBrown


On Tue, Feb 18 2020, Aurelien Degremont wrote:

> Thanks for your reply.
> I think I have a good enough understanding of LNET itself. My question 
was more about how LNET is being used by Lustre itself.
>
> LNET will established connections only if asked for by upper layers. 
> When I was talking about client and server, I was talking about how 
Lustre was using it.
>
> As far as I understood, Lustre server only contact clients when they need 
to send LDLM callbacks.
> They do so through the socket already opened by the client (reverse 
import).
> What happened if the socket is closed is what I'm not sure. I though the 
server is rather waiting for the client to reconnect and if not, is more or 
less evicting it.
> Could it be possible that the server in this case opens the connection 
itself without waiting for the client to reconnect?
>
>
> Aurélien
>
> Le 18/02/2020 05:42, « NeilBrown »  a écrit :
>
> 
> LNet is a peer-to-peer protocol, it has no concept of client and 
server.
> If one host needs to send a message to another but doesn't already 
have
> a connection, it creates a new connection.
> I don't yet know enough specifics of the lustre protocol to be certain
> of the circumstances when a lustre server will need to initiate a 
message
> to a client, but I imagine that recalling a lock might be one.
> 
> I think you should assume that any LNet node might receive a 
connection
> from any other LNet node (for which they share an LNet network), and
> that the connection could come from any port between 512 and 1023
> (LNET_ACCEPTOR_MIN_PORT to LNET_ACCEPTOR_MAX_PORT).
>     
> NeilBrown
> 
> 
> 
> On Mon, Feb 17 2020, Degremont, Aurelien wrote:
> 
> > Hi all,
> >
> > From what I've understood so far, LNET listens on port 988 by 
default and peers connect to it using 1021-1023 TCP ports as source ports.
> > At Lustre level, servers listen on 988 and clients connect to them 
using the same source ports 1021-1023.
> > So only accepting connections to port 988 on server side sounded 
pretty safe to me. However, I've seen connections from 1021-1023 to 988, from 
server hosts to client hosts sometimes.
> > I can't understand what mechanism could trigger these connections. 
Did I miss something?
> >
> > Thanks
> >
> > Aurélien
> >
> > ___
> > lustre-discuss mailing list
> > lustre-discuss@lists.lustre.org
> > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> 


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] LNET ports and connections

[Degremont, Aurelien]

Thanks for your reply.
I think I have a good enough understanding of LNET itself. My question was more 
about how LNET is being used by Lustre itself.

LNET will established connections only if asked for by upper layers. 
When I was talking about client and server, I was talking about how Lustre was 
using it.

As far as I understood, Lustre server only contact clients when they need to 
send LDLM callbacks.
They do so through the socket already opened by the client (reverse import).
What happened if the socket is closed is what I'm not sure. I though the server 
is rather waiting for the client to reconnect and if not, is more or less 
evicting it.
Could it be possible that the server in this case opens the connection itself 
without waiting for the client to reconnect?


Aurélien

Le 18/02/2020 05:42, « NeilBrown »  a écrit :


LNet is a peer-to-peer protocol, it has no concept of client and server.
If one host needs to send a message to another but doesn't already have
a connection, it creates a new connection.
I don't yet know enough specifics of the lustre protocol to be certain
of the circumstances when a lustre server will need to initiate a message
to a client, but I imagine that recalling a lock might be one.

I think you should assume that any LNet node might receive a connection
from any other LNet node (for which they share an LNet network), and
that the connection could come from any port between 512 and 1023
(LNET_ACCEPTOR_MIN_PORT to LNET_ACCEPTOR_MAX_PORT).

NeilBrown



On Mon, Feb 17 2020, Degremont, Aurelien wrote:

> Hi all,
>
> From what I've understood so far, LNET listens on port 988 by default and 
peers connect to it using 1021-1023 TCP ports as source ports.
> At Lustre level, servers listen on 988 and clients connect to them using 
the same source ports 1021-1023.
> So only accepting connections to port 988 on server side sounded pretty 
safe to me. However, I've seen connections from 1021-1023 to 988, from server 
hosts to client hosts sometimes.
> I can't understand what mechanism could trigger these connections. Did I 
miss something?
>
> Thanks
>
> Aurélien
>
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] LNET ports and connections

[Degremont, Aurelien]

Hi all,

From what I've understood so far, LNET listens on port 988 by default and peers 
connect to it using 1021-1023 TCP ports as source ports.
At Lustre level, servers listen on 988 and clients connect to them using the 
same source ports 1021-1023.
So only accepting connections to port 988 on server side sounded pretty safe to 
me. However, I've seen connections from 1021-1023 to 988, from server hosts to 
client hosts sometimes.
I can't understand what mechanism could trigger these connections. Did I miss 
something?

Thanks

Aurélien

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Poor read performance when using o2ib nets over RoCE

[Degremont, Aurelien]

I don't have idea of what could be the problem, but you should try benchmarking 
your network bandwidth with lnet_selftest, with o2ib and tcp and compare the 
value. You will see if the problem is a related to Lustre network layer or 
something else.

http://wiki.lustre.org/LNET_Selftest

De : lustre-discuss  au nom de 
Christian Kuntz 
Date : mercredi 12 février 2020 à 04:46
À : "lustre-discuss@lists.lustre.org" 
Objet : [lustre-discuss] Poor read performance when using o2ib nets over RoCE

Hello,

I've been running into a strange issue where my writes are blazingly fast (5.5 
GB/s) over RoCE with Mellanox MCX516A-CCAT cards all running together over 
o2ib, but read performance tanks to roughly 100 MB/s. During mixed read/write 
situations write performance also plummets to sub 100MB/s.

Curiously, when using tcp these problems disappear and everyone is happy, 
hovering around 1.5 GB/s read, 3 GB/s write.

I'm wondering if anyone else has run into this and what the solution may be?
My setup is:
Debian 10.3, lustre 2.13.0, zfs 0.8.2 with two OSS/OST pairs, a single mgs/mdt 
node and a single client node connected over o2ib. Everyone is cabled together 
via 100g fiber through a mellanox switch that's configured for roce and 
bonding, and they all hit about 98 Gb/s to each other via ib_send_bw, and 
simple testing of network file transfers via NFSoRDMA didn't experience the 
slowdown that lustre seems to be seeing.

I'd be happy to provide more diagnostic information if that helps, as well as 
trace information if needed.

Best,
Christian

[Image supprimée par l'expéditeur.]
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] LUSTRE - Installation on DEBIAN 10.x ????

[Degremont, Aurelien]

I did not try for Debian, but Ubuntu support works quite well.
You should try building from sources and hopefully it will work.
I know some sites in Germany (GSI) which are using Debian for their cluster. So 
it must work.

Try downloading 2.12.4 sources, then

sh ./autogen.sh
./configure
make debs


Aurélien

Le 11/02/2020 18:02, « lustre-discuss au nom de Spitz, Cory James » 
 a écrit :

No, you are not the only one.  There is help for Debian users at 
https://wiki.whamcloud.com/display/PUB/Building+Lustre+from+Source.

Unfortunately, the guide at http://wiki.debian.org/Lustre is ancient old.  
https://wiki.whamcloud.com/display/PUB/Build+Lustre+MASTER+client+on+Debian+10.1.0+from+Git
 seems to be what you are looking for.

BTW, there is a lot of Ubuntu support out there for Lustre.  LTS Ubuntu is 
even declared a supported client as you can see here: 
https://wiki.whamcloud.com/display/PUB/Lustre+Support+Matrix.

Good luck,
-Cory



On 2/11/20, 9:49 AM, "lustre-discuss on behalf of Matthias Krawutschke" 
 wrote:

Hey Thomas,

thank you very much for your quick answer.
I would like to install LUSTRE on DEBIAN 10 - named: BUSTER in our 
Cluster.
So what I missing is, that I can´t find so much documents over 
Installation, etc. for DEBIAN and
no Software - Packet for this.

My question is: which Software - Packet should I use for installation 
on DEBIAN 10?
Or is it true, that we both are the pioneer for DEBIAN - Platform with 
this Software, because you had answered me only ☹

Best regards….



Matthias Krawutschke, Dipl. Inf.

Universität Potsdam
ZIM - Zentrum für Informationstechnologie und Medienmanagement
Team High-Performance-Computing on Cluster - Environment
Campus Am Neuen Palais: Am Neuen Palais 10 | 14469 Potsdam
Tel: +49 331 977-, Fax: +49 331 977-1750

Internet: https://www.uni-potsdam.de/de/zim/angebote-loesungen/hpc.html 


-Ursprüngliche Nachricht-
Von: lustre-discuss  Im 
Auftrag von Thomas Roth
Gesendet: Donnerstag, 6. Februar 2020 15:51
An: lustre-discuss@lists.lustre.org
Betreff: Re: [lustre-discuss] LUSTRE - Installation on DEBIAN 10.x 

Hi Matthias,

last time we tried this (Debian Wheezy + Lustre 2.5) we had to use the 
Redhat kernel _in_ Debian.
However, nowadays there is the patchless-ldiskfs-server, and of course 
the server based on ZFS, both of which would require only to compile the proper 
modules.
My guess is that you have better chances with the newer Lustre 
versions...

Regards
Thomas


On 05.02.20 11:50, Matthias Krawutschke wrote:
> Hello everybody,
> 
>   
> 
> I am new here on this platform and I have a question about LUSTRE – 
> Installation on DEBIAN 10.x ….. 
> 
> It is possible to install this on this LINUX – Version & if yes: 
which version of LUSTRE does it work and how?
> 
>   
> 
> Best regards….
> 
>   
> 
>   
> 
>   
> 
> Matthias Krawutschke, Dipl. Inf.
> 
>   
> 
> Universität Potsdam
> ZIM - Zentrum für Informationstechnologie und Medienmanagement Team 
> High-Performance-Computing
> 
> Campus Am Neuen Palais: Am Neuen Palais 10 | 14469 Potsdam
> Tel: +49 331 977-, Fax: +49 331 977-1750
> 
>   
> 
> Internet:  
>  
> https://www.uni-potsdam.de/de/zim/angebote-loesungen/hpc.html 
> 
>   
> 
>   
> 
> 
> 
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org 
> 

--

Thomas Roth
Department: IT

GSI Helmholtzzentrum für Schwerionenforschung GmbH Planckstraße 1, 
64291 Darmstadt, Germany, http://www.gsi.de 

Commercial Register / Handelsregister: Amtsgericht Darmstadt, HRB 1528 
Managing Directors / Geschäftsführung:
Professor Dr. Paolo Giubellino, Jörg Blaurock Chairman of the 
Supervisory Board / Vorsitzender des GSI-Aufsichtsrats:
State Secretary / Staatssekretär Dr. Volkmar Dietz 
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org

Re: [lustre-discuss] reflecting state of underlying store in Lustre with HSM

[Degremont, Aurelien]

Hi Kris,

As people said each HSM has its own copytool with its own constraints. To work 
properly with S3, additional metadata is stored in Lustre when files are 
imported from a S3 bucket.
Questions specific to Amazon FSx For Lustre would be better asked to AWS 
support. 

Aurélien

Le 08/01/2020 14:14, « Matt Rásó-Barnett »  a écrit :

Hi Kris,
I assume you are using Amazon's FSx for Lustre product 
(https://aws.amazon.com/fsx/lustre/) for this, rather than rolling 
Lustre HSM to S3 on AWS yourself?

I'm afraid I don't know any more than you do on this product but it's 
something I've been keen to play with as it sounds really interesting.

However the process or policy-engine by which Amazon are mapping S3 
objects to files in Lustre is not part of Lustre itself so you'd need 
input from Amazon on this - maybe Aurelien (in BCC) can comment here?

 From their overview documenation my understanding is that the Lustre FS 
is meant to be short lived in this model, so you create it populated 
with the contents of the bucket *at that time*, run your workload, then 
archive results back and delete the FS.

So if your bucket has been updated post Lustre FS creation, you may have 
to destroy the Lustre FS and recreate it to notice the changes. HSM 
isn't designed to be a mechanism for synchronising two endpoints that 
change independent of the other, so any change to the backend not via 
the filesystem will not be tracked.

This is just me guessing without having used it however, I'll be 
interested to hear if you learn more about this from Amazon.

Kind regards,

Matt

On Tue, Jan 07, 2020 at 03:18:39PM -0800, Kristian Kvilekval wrote:
>We have Lustre <- HSM -> S3
>
>We have direct modifications to S3 that occur after the Lustre filesystem
>is created
>I was  wondering if there is any way to register a new/deleted file at the
>Lustre level using HSM or other commands
>
>Say a user uploads a file to S3, and I know  the mapped path in Lustre,
>I would like to do
>lfs hsm_register /path/to/file/in/S3/ # Create a metadata entry in
>Lustre
>lfs hsm_restore /path/to/file/in/S3   # Fetch file from S3 into Lustre
>
>Thx
>
>
>
>
>
>
>On Tue, Jan 7, 2020 at 8:04 AM Colin Faber  wrote:
>
>> Can you provide an example of what you're attempting to accomplish?  Am I
>> understanding correctly, that you've got a lustre file system, you're 
then
>> writing data into this file system?
>>
>> On Mon, Jan 6, 2020 at 10:02 PM Kristian Kvilekval  wrote:
>>
>>> We are using Lustre on AWS backed by S3 buckets.
>>> When creating a new Lustre filesystem, S3 metadata can be automatically
>>> imported  into Lustre.  When changes occur to the underlying S3 store,
>>> these changes are not automatically reflected.
>>>
>>> Is it possible to indicate the creation / deletion of the underlying S3
>>> files after filesystem creation using HSM?
>>> Is it possible to reimport the underlying metadata after creation?
>>>
>>> Any pointers appreciated.
>>>
>>> Thanks,
>>> Kris
>>>
>>> --
>>> Kris Kvilekval, Ph.D.
>>> ViQi Inc
>>> (805)-699-6081
>>> ___
>>> lustre-discuss mailing list
>>> lustre-discuss@lists.lustre.org
>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>>
>>
>
>-- 
>Kris Kvilekval, Ph.D.
>ViQi Inc
>(805)-699-6081

>___
>lustre-discuss mailing list
>lustre-discuss@lists.lustre.org
>http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] frequent Connection lost, Connection restored to mdt

[Degremont, Aurelien]

Hi

These messages means the client thinks it has lost the communication with the 
server and reconnect. The server only sees the reconnection and never thought 
the client was gone.

It could be related to lots of things. The server could be receiving RPCs from 
this client but not processing them fast enough. Is there other errors on your 
server? Is there any high load?
Same on your clients? Is there any high load that could prevent your client 
from communicating with your server properly?

Do you correlate that with some specific load running on your clients?

Aurélien

De : lustre-discuss  au nom de David 
Cohen 
Date : dimanche 22 décembre 2019 à 17:08
À : "lustre-discuss@lists.lustre.org" 
Objet : [lustre-discuss] frequent Connection lost, Connection restored to mdt

Hi,
We are running 2.10.5 on the servers and 2.10.8 on the clients.
Every few minutes, we see:

On client side:

Dec 22 15:26:34 gftp kernel: Lustre: 
439834:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed 
out for slow reply: [sent 1577021187/real 1577021187]  req@88160be9c6c0 
x1653620348981536/t0(0) 
o36->lustre-MDT-mdc-8817d9776c00@10.0.0.1@tcp:12/10 lens 608/4768 e 0 
to 1 dl 1577021194 ref 2 fl Rpc:X/0/ rc 0/-1
Dec 22 15:26:34 gftp kernel: Lustre: 
439834:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 3 previous similar 
messages
Dec 22 15:26:34 gftp kernel: Lustre: lustre-MDT-mdc-8817d9776c00: 
Connection to lustre-MDT (at 10.0.0.1@tcp) was lost; in progress operations 
using this service will wait for recovery to complete
Dec 22 15:26:34 gftp kernel: Lustre: Skipped 3 previous similar messages
Dec 22 15:26:34 gftp kernel: Lustre: lustre-MDT-mdc-8817d9776c00: 
Connection restored to 10.0.0.1@tcp (at 192.114.101.153@tcp)
Dec 22 15:26:34 gftp kernel: Lustre: Skipped 3 previous similar messages

On server side:

Dec 22 15:26:34 oss03 kernel: Lustre: lustre-MDT: Client 
38d6eef1-e146-be41-bab9-409b272d0d4f (at 10.0.0.10@tcp) reconnecting
Dec 22 15:26:34 oss03 kernel: Lustre: lustre-MDT: Connection restored to 
ec2cdfce-353f-583a-c970-fde3f5d5189c (at 10.0.0.10@tcp)

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Lustre 2.12 quota

[Degremont, Aurelien]

Quota is not enabled by default, you need to enable it. Space accounting is on 
by default.
Did you check the doc?
See 25.2.1: http://doc.lustre.org/lustre_manual.xhtml#configuringquotas


Aurélien

De : lustre-discuss  au nom de Parag 
Khuraswar 
Date : vendredi 29 novembre 2019 à 04:10
À : 'Lustre discussion' 
Objet : [lustre-discuss] Lustre 2.12 quota

Hi,

I am facing issues with quota on Lustre version 2.12.0.
Is quota by default enabled in 2.12.0 version ? At client side I am using 
Lustre version 2.11 (OS 7.4).
I have set quota limit for one test user and ' lfs quota -u   is showing the defined values as well, but quota is not working. I have 
not enabled the quota hoping that it is by default enabled.
The file system is live.
DO I have to enable quota manually  ?

Below is the partial output of " lctl get_param osd-*.*.quota_slave.info "
===
[root@pfs-srv-1 ~]# lctl get_param osd-*.*.quota_slave.info
osd-ldiskfs.pfs-MDT.quota_slave.info=
target name:pfs-MDT
pool ID:0
type:   md
quota enabled:  none
conn to master: setup
space acct: ug
user uptodate:  glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
project uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.pfs-OST.quota_slave.info=
==
Please share the command if I have to manually enable quota.


Regards,
Parag

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Limit to number of OSS?

[Degremont, Aurelien]

Thanks for this info. But actually I was really looking at the number of OSS, 
not OSTs :)
This is really more how Lustre client nodes and MDT will cope with very large 
number of OSSes.

De : Andreas Dilger 
Date : vendredi 4 octobre 2019 à 04:54
À : "Degremont, Aurelien" 
Cc : "lustre-discuss@lists.lustre.org" 
Objet : Re: [lustre-discuss] Limit to number of OSS?

On Oct 3, 2019, at 07:55, Degremont, Aurelien 
mailto:degre...@amazon.com>> wrote:

Hello all!

This doc from the wiki says "Lustre can support up to 2000 OSS per file system" 
(http://wiki.lustre.org/Lustre_Server_Requirements_Guidelines).

I'm a bit surprised by this statement. Does somebody has information about the 
upper limit to the number of OSSes?
Or what could be the scaling limitator for this number of OSS? Network limit? 
Memory consumption? Other?

That's likely a combination of a bit of confusion and a bit of safety on the 
part of Intel writing that document.

The Lustre Operations Manual writes:
Although a single file can only be striped over 2000 objects, Lustre file 
systems can have thousands of OSTs. The I/O bandwidth to access a single file 
is the aggregated I/O bandwidth to the objects in a file, which can be as much 
as a bandwidth of up to 2000 servers. On systems with more than 2000 OSTs, 
clients can do I/O using multiple files to utilize the full file system 
bandwidth.
I think PNNL once tested up to 4000 OSTs, and I think the compile-time limit 
is/was 8000 OSTs (maybe it was made dynamic, I don't recall offhand), but the 
current code could _probably_ handle up to 65000 OSTs without significant 
problems.  Beyond that, there is the 16-bit OST index limit in the filesystem 
device labels and the __u16 lov_user_md_v1->lmm_stripe_offset to specify the 
starting OST index for "lfs setstripe", but that could be overcome with some 
changes.

Given OSTs are starting to approach 1PB with large drives and 
declustered-parity RAID, this would get us in the range 8-65EB, which is over 
2^64 bytes (16EB), so I don't think it is an immediate concern.  Let me know if 
you have any trouble with a 9000-OST filesystem... :-)

Cheers, Andreas
--
Andreas Dilger
Principal Lustre Architect
Whamcloud





___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] Limit to number of OSS?

[Degremont, Aurelien]

Hello all!

This doc from the wiki says "Lustre can support up to 2000 OSS per file system" 
(http://wiki.lustre.org/Lustre_Server_Requirements_Guidelines).

I'm a bit surprised by this statement. Does somebody has information about the 
upper limit to the number of OSSes?
Or what could be the scaling limitator for this number of OSS? Network limit? 
Memory consumption? Other?

Thanks

Aurélien

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] changing inode size on MDT

[Degremont, Aurelien]

As Andreas said "it is not relevant for ZFS since ZFS dynamically allocates 
inodes and blocks as needed"

"as needed" is the important part. In your example, your MDT is almost empty, 
so 17G inodes for an empty MDT seems pretty sufficient.
As you will create new files and use these inodes, you will see the total 
number of inodes change.

But I don't what is the maximum number of inodes you can estimate for a 8.3 TiB 
MDT…

De : lustre-discuss  au nom de 
"Hebenstreit, Michael" 
Date : jeudi 3 octobre 2019 à 13:04
À : Andreas Dilger 
Cc : "lustre-discuss@lists.lustre.org" 
Objet : Re: [lustre-discuss] changing inode size on MDT

So you are saying on a zfs based Lustre there is no way to increase the number 
of available inodes? I have 8TB MDT with roughly 17G inodes

[root@elfsa1m1 ~]# df -h
Filesystem   Size  Used Avail Use% Mounted on
mdt  8.3T  256K  8.3T   1% /mdt

[root@elfsa1m1 ~]# df -i
Filesystem   Inodes  IUsed   IFree IUse% Mounted on
mdt 17678817874  6 176788178681% /mdt

Formating under Lustre 2.10.8

mkfs.lustre --mdt --backfstype=zfs --fsname=lfsarc01 --index=0 
--mgsnid="36.101.92.22@tcp" --reformat mdt/mdt

this translates to only 948M inodes on the Lustre FS.

[root@elfsa1m1 ~]# df -i
Filesystem   Inodes  IUsed   IFree IUse% Mounted on
mdt 17678817874  6 176788178681% /mdt
mdt/mdt   948016092263   9480158291% /lfs/lfsarc01/mdt

[root@elfsa1m1 ~]# df -h
Filesystem   Size  Used Avail Use% Mounted on
mdt  8.3T  256K  8.3T   1% /mdt
mdt/mdt  8.2T   24M  8.2T   1% /lfs/lfsarc01/mdt

and there is no reasonable option to provide more file entries except for 
adding another MDT?

Thanks
Michael

From: Andreas Dilger 
Sent: Wednesday, October 02, 2019 18:49
To: Hebenstreit, Michael 
Cc: Mohr Jr, Richard Frank ; lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] changing inode size on MDT

There are several confusing/misleading comments on this thread that need to be 
clarified...

On Oct 2, 2019, at 13:45, Hebenstreit, Michael 
mailto:michael.hebenstr...@intel.com>> wrote:

http://wiki.lustre.org/Lustre_Tuning#Number_of_Inodes_for_MDS

Note that I've updated this page to reflect current defaults.  The Lustre 
Operations Manual has a much better description of these parameters.


and I'd like to use --mkfsoptions='-i 1024' to have more inodes in the MDT. We 
already run out of inodes on that FS (probably due to an ZFS bug in early IEEL 
version) - so I'd like to increase #inodes if possible.

The "-i 1024" option (bytes-per-inode ratio) is only needed for ldiskfs since 
it statically allocates the inodes at mkfs time, it is not relevant for ZFS 
since ZFS dynamically allocates inodes and blocks as needed.

On Oct 2, 2019, at 14:00, Colin Faber 
mailto:cfa...@gmail.com>> wrote:
With 1K inodes you won't have space to accommodate new features, IIRC the 
current minimal limit on modern lustre is 2K now. If you're running out of MDT 
space you might consider DNE and multiple MDT's to accommodate that larger name 
space.

To clarify, since Lustre 2.10 any new ldiskfs MDT will allocate 1024 bytes for 
the inode itself (-I 1024).  That allows enough space *within* the inode to 
efficiently store xattrs for more complex layouts (PFL, FLR, DoM).  If xattrs 
do not fit inside the inode itself then they will be stored in an external 4KB 
inode block.

The MDT is formatted with a bytes-per-inode *ratio* of 2.5KB, which means 
(approximately) one inode will be created for every 2.5kB of the total MDT 
size.  That 2.5KB of space includes the 1KB for the inode itself, plus space 
for a directory entry (or multiple if hard-linked), extra xattrs, the journal 
(up to 4GB for large MDTs), Lustre recovery logs, ChangeLogs, etc.  Each 
directory inode will have at least one 4KB block allocated.

So, it is _possible_ to reduce the inode *ratio* below 2.5KB if you know what 
you are doing (e.g. 2KB/inode or 1.5KB/inode, this can be an arbitrary number 
of bytes, it doesn't have to be an even multiple of anything) but it definitely 
isn't possible to have 1KB inode size and 1KB per inode ratio, as there 
wouldn't be *any* space left for directories, log files, journal, etc.

Cheers, Andreas
--
Andreas Dilger
Principal Lustre Architect
Whamcloud





___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] which zfs for lustre 2.12.3?

[Degremont, Aurelien]

Hi Bernd,

Lustre is trying to update to support by default ZFS 0.8.1. During the tests of 
Lustre 2.13 development branch, there was an increase of test failures. Default 
ZFS version was reverted to 0.7.13 until this is clear if the problem comes 
from that or something else. Based on that, the official ZFS version for 2.13 
is not yet decided and could or could not be 0.8.1.
If you can wait this is sorted out, you will see what's the conclusion.

If you don't want to wait, 0.7.13 is probably the safest choice for now.
Lustre will move to 0.8 eventually anyway. This is just that this could be 
delayed a bit at short term.


Aurélien

Le 27/09/2019 11:45, « lustre-discuss au nom de Bernd Melchers » 
 
a écrit :

Hi all,
is there already a recommendation for the zfs-on-linux version for the
upcoming lustre 2.12.3? zfs 0.8.x includes the spl part of zfs 0.7.x
and i am not sure how to migrate from 0.7.x to 0.8.x  and if this is the
right choice...

Mit freundlichen Grüßen
Bernd Melchers

-- 
Archiv- und Backup-Service | fab-serv...@zedat.fu-berlin.de
Freie Universität Berlin   | Tel. +49-30-838-55905
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lustre client on centos 7.7

[Degremont, Aurelien]

The Lustre 2.10 branch don't support CentOS 7.7
Lustre 2.12.3 and Lustre 2.13 will.

However Lustre 2.10.8 can built on CentOS 7.7 if you can afford to miss 
Infiniband support.
This is the part of the code that creates this problem.  ./configure 
--with-ofed=no will disable it.

Aurélien

De : lustre-discuss  au nom de Arman 
Khalatyan 
Date : dimanche 22 septembre 2019 à 16:26
À : Lustre discussion 
Objet : [lustre-discuss] lustre client on centos 7.7

Hello,
are there any lustre client with the CentOS 7.7?
i was trying to compile the listre 2.10 with the recent centos 7.7 it was 
failing with the error:
rpmbuild  --rebuild --without servers --with lnet-dlc lustre-2.10.8-1.src.rpm

/root/rpmbuild/BUILD/lustre-2.10.8/lnet/klnds/o2iblnd/o2iblnd.h:69:27: fatal 
error: linux/pci-dma.h: No such file or directory
 #include 
   ^
compilation terminated.

looks like something changed in the Kernel drivers:(

thank you beforehand,
Arman
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] CPU soft lockup on mkfs.lustre

[Degremont, Aurelien]

I saw the same issue and downgraded my kernel back to RHEL kernel 3.10.0-957.

But you probably wants to keep 7.7 kernel ? :)

De : lustre-discuss  au nom de Tamas 
Kazinczy 
Organisation : Governmental Agency for IT Development
Date : mardi 10 septembre 2019 à 14:39
À : "lustre-discuss@lists.lustre.org" 
Objet : [lustre-discuss] CPU soft lockup on mkfs.lustre


Hi,

I've successfully compiled Lustre after the

'LU-12457 kernel: RHEL 7.7 server support' change

but when I try to create an MGS with LDISKFS backend

on RHEL 7.7 I get CPU soft lockup:

 kernel:NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
[mkfs.lustre:31220]



Looked into logs for more details:
Sep  6 10:41:00 mgs1 kernel: Call Trace:

Sep  6 10:41:00 mgs1 kernel: [] 
queued_spin_lock_slowpath+0xb/0xf
Sep  6 10:41:00 mgs1 kernel: [] _raw_spin_lock+0x20/0x30
Sep  6 10:41:00 mgs1 kernel: [] igrab+0x1e/0x60
Sep  6 10:41:00 mgs1 kernel: [] ldiskfs_quota_off+0x3b/0x130 
[ldiskfs]
Sep  6 10:41:00 mgs1 kernel: [] ldiskfs_put_super+0x4d/0x400 
[ldiskfs]
Sep  6 10:41:00 mgs1 kernel: [] 
generic_shutdown_super+0x6d/0x100
Sep  6 10:41:00 mgs1 kernel: [] kill_block_super+0x27/0x70
Sep  6 10:41:00 mgs1 kernel: [] 
deactivate_locked_super+0x4e/0x70
Sep  6 10:41:00 mgs1 kernel: [] deactivate_super+0x46/0x60
Sep  6 10:41:00 mgs1 kernel: [] cleanup_mnt+0x3f/0x80
Sep  6 10:41:00 mgs1 kernel: [] __cleanup_mnt+0x12/0x20
Sep  6 10:41:00 mgs1 kernel: [] task_work_run+0xbb/0xe0
Sep  6 10:41:00 mgs1 kernel: [] do_notify_resume+0xa5/0xc0
Sep  6 10:41:00 mgs1 kernel: [] int_signal+0x12/0x17
Sep  6 10:41:00 mgs1 kernel: Code: 47 fe ff ff 66 2e 0f 1f 84 00 00 00 00 00 0f 
1f 44 00 00 55 48 89 e5 66 90 b9 01 00 00 00 8b 17 85 d2 74 0d 83 fa 03 74 08 
f3 90 <8b> 17 85 d2 75 f3 89 d0 f0 0f b1 0f 39 c2 75 e3 5d 66 90 c3 0f



Has anyone else also experienced this issue?

Is there a way to make this work?



Thanks,

--

Tamás Kazinczy
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] Do not recreate OST objects on OST replacement

[Degremont, Aurelien]

Hello

When an OST dies and you have no choice but replacing it with a newly freshly 
formatted one (using mkfs --replace), Lustre runs a resynchronization 
mechanisms between the MDT and the OST.
The MDT will sent the last object ID it knows for this OST and the OST will 
compare this value with its own counter (0 for a freshly formatted OST).
If the difference is greater than 100,000 objects, it will recreate only the 
last 10,000, if not, it will recreate all the missing objects.

I would like it to avoid recreating any objects. The missing ones are lost and 
just start recreating new ones. Is there a way to achieve that?

Thanks

Aurélien 


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] cannot create file for unknown uid

[Degremont, Aurelien]

Hello

I think that's expected. Lustre runs an identity upcall command on the MDT to 
get the user secondary groups. If it fails, Lustre returns a permission error.
See: http://doc.lustre.org/lustre_manual.xhtml#dbdoclet.l_getidentity

Try to disable it, confirm this is the reason of your problem.

If yes, and you need to be able resolve supplementary groups for known users 
but accept any unknown user ID, you will probably have to wrap this script to 
handle both cases.


Aurélien

Le 30/08/2019 15:14, « lustre-discuss au nom de Bernd Melchers » 
 
a écrit :

Dear all,
we use lustre 2.12.2 (CentOS-7.6) and observe a problem where file
creation for unknown userids is not possible. Background is that
we export our lustre file system with ganesha-nfs (nfs vers. 3) to
nfs clients with userids unknown to the (nfs-)server.

Attached is a short C Program to reproduce the problem.
Process runs as root, changes effective user id to an unknown user id
and creates a file:

/tmp is xfs, /scratch is lustre :
# ls -ld /tmp /scratch/tmp
drwxrwxrwt9 root root 25600 Jun  3 14:47 /scratch/tmp
drwxrwxrwt. 132 root root 16384 Aug 30 15:06 /tmp

[working for xfs:] # ./debug1_lustre /tmp/testfile && echo success
success

[not working for lustre:] # ./debug1_lustre /scratch/tmp/testfile
/scratch/tmp/testfile: Permission denied

If i change the uid to a known uid, it works.

Is this a bug in lustre?

Mit freundlichen Grüßen
Bernd Melchers

-- 
Archiv- und Backup-Service | fab-serv...@zedat.fu-berlin.de
Freie Universität Berlin   | Tel. +49-30-838-55905


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] osc and llite stats files are not exported to clients

[Degremont, Aurelien]

Lustre is transitionning /proc files to /sys per kernel developer requests.

Instead of chasing where these files are located now depending on your Lustre 
version, 'lctl get_param' is the official way to read them.
This command will figure out where to properly read the files.

De : lustre-discuss  au nom de U 
Sawangwit 
Date : jeudi 25 juillet 2019 à 13:15
À : "lustre-discuss@lists.lustre.org" 
Objet : [lustre-discuss] osc and llite stats files are not exported to clients

Dear all,

   I am having trouble with lustre monitoring using Ganglia gmond_python_modules
(https://github.com/ganglia/gmond_python_modules/tree/master/lustre).

Upon closer inspections, I found that the stats files in 
/proc/fs/lustre/osc/*/
and /proc/fs/lustre/llite are missing (among other similar files in these 
directories).

 However, lctl get_param osc.*.stats and lctl get_param llite.*.stats return
parameters and values as you would expect. The stats files are what the
gmond_python_modules wants hence the problem monitoring with Ganglia.

After many searching around, could not figure out what I could have done to
have stats export normally to clients and remounting did not help. On MDS and 
OSS,
I can see the obdfiler/*/export/*/stats or mdt/*/export/*/stats

The lustre manual also said that the exporting of these info to client 
should be by
default.

  Thank you in advance.

Cheers,
Utane

---
Utane Sawangwit, PhD.
Research Astronomer
National Astronomical Research Institute of Thailand (NARIT)
Sirindhon AstroPark, 260, Mue 4, Donkaew
Mae-Rim, Chiang Mai, 50180, Thailand

ดร. อุเทน แสวงวิทย์
นักวิจัยชำนาญการ
สถาบันวิจัยดาราศาสตร์แห่งชาติ(องค์การมหาชน)
อุทยานดาราศาสตร์สิรินธร 260 หมู่ 4 ต.ดอนแก้ว
อ.แม่ริม จ.เชียงใหม่ 50180
---

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] [External] Re: obdfilter/mdt stats meaning ?

[Degremont, Aurelien]

>   read  | write
>disk I/Os in flightios   % cum % |  ios % cum %
>1:   211177215  61  61   | 29305564  97  97
>2:41332944  11  72   | 498260   1  99
>[..]
>Does these lines means :
>Since last snapshot there was 211177215x1 and read 41332944x2 I/O in flight ?

It means (since the last time the statistics were cleared)

  *   11% of the time, 2 READ I/O requests were "in-flights" to disk, meaning 2 
I/O were sent to disks and not yet commit/acknowledged
  *   61 % of the time, only 1 READ I/O request.

Same principle for write.

What this means here is that your workload is not feeding the disks with lots 
of write (97% with 1 I/O in flight), but a bit more reads.
Disks and especially disk arrays are reordering I/O and distributing them 
across the various drives they are composed of to optimized bandwith. To really 
take benefits of all the possible bandwith/throughput your hardward can offer, 
you often need to be able to have lots of big I/O and possible multiple I/O in 
flights.
Few I/O in flight could means:

  *   your workload is not really big
  *   your hardward is fast compared to the throughput coming to this server 
(ratio disk BW vs network BW by example)
This could also help you identify bad performance numbers and find from where 
the bottleneck comes from.


De : lustre-discuss  au nom de Louis 
Bailleul 
Date : mardi 16 juillet 2019 à 17:49
À : lustre-discuss 
Objet : Re: [lustre-discuss] [External] Re: obdfilter/mdt stats meaning ?

Hi Aurélien,

Thanks for the prompt reply.
For the ost stats, any idea what the preprw and commitrw mean ?
And why there are two entries with different values for statfs ?

For brw_stats even with the doc I still struggle to read this.
For example how do you make sense of disk I/O in flight ?
   read  | write
disk I/Os in flightios   % cum % |  ios % cum %
1:   211177215  61  61   | 29305564  97  97
2:41332944  11  72   | 498260   1  99
[..]
Does these lines means :
Since last snapshot there was 211177215x1 and read 41332944x2 I/O in flight ?

Best regards,
Louis
On 16/07/2019 15:50, Degremont, Aurelien wrote:
Hi Louis,

About brw_stats, there are a bit of explanation in the Lustre Doc (not that 
detailed, but still)
http://doc.lustre.org/lustre_manual.xhtml#dbdoclet.50438271_55057

> Last thing, is there any way to get the name of the filesystem an OST is part 
> of by using lctl ?

I don't know what you want exactly, but the OST names are self explanatory, 
there always are like: fsname-OST
Where fsname is the lustre filesystem they are part of.

For obdfilter stats, these are mostly action to OST objects or client 
connection management RPCs.

setattr: changing an OST object attributes (owner, group, ...)
punch: mostly used for truncate (theorically can do holes in files, like 
truncate with a start and length)
sync: straighforward, sync OST to disk
destroy: delete an OST object (mostly when a file is deleted)
create: create an OST object
statfs: like 'df' for this specific OST (used by 'lfs df' by example)
(re)connect: when a client connect/reconnect to this OST
ping: when a client ping this OST.


Aurélien

De : lustre-discuss 
<mailto:lustre-discuss-boun...@lists.lustre.org>
 au nom de Louis Bailleul 
<mailto:louis.baill...@pgs.com>
Date : mardi 16 juillet 2019 à 16:38
À : lustre-discuss 
<mailto:lustre-discuss@lists.lustre.org>
Objet : [lustre-discuss] obdfilter/mdt stats meaning ?

Hi all,

I am trying to make sense of some of the OST/MDT stats for 2.12.
Can anybody point me to the doc that explain what the metrics are ?
The wiki only mention read/write/get_info : 
http://wiki.lustre.org/Lustre_Monitoring_and_Statistics_Guide<https://urldefense.proofpoint.com/v2/url?u=http-3A__wiki.lustre.org_Lustre-5FMonitoring-5Fand-5FStatistics-5FGuide=DwMGaQ=KV_I7O14pmwRcmAVyJ1eg4Jwb8Y2JAxuL5YgMGHpjcQ=FTXmt89oLXmbXfP78w86-PxB1XdLYgxG8hEoAnZvCvs=UC1t7z9tgmxUE2FWaTFHFT_Y69z_VMH0dEYF1VXadX0=cdXTUStD_NPwj3GtNYBqJA2nkJ1Ec53F9aD5UxFo5tw=>
But the list I get is quite different :
obdfilter.OST001.stats=
snapshot_time 1563285450.647120173 secs.nsecs
read_bytes340177708 samples [bytes] 4096 4194304 
396712660910080
write_bytes   30008856 samples [bytes] 24 4194304 78618271501667
setattr   1755 samples [reqs]
punch 73463 samples [reqs]
sync  50606 samples [reqs]
destroy   31990 samples [reqs]
create956 samples [reqs]
statfs75378743 samples [reqs]
connect   5798 samples [reqs]
reconnect 3242 samples [reqs]
disconnect5820 samples [reqs]
statfs3737980 samples

Re: [lustre-discuss] ZFS tunings

[Degremont, Aurelien]

Nobody on this topic?
I'm pretty sure they are lots of people running Lustre on ZFS with various 
tuning applied. Don't be shy 


De : "Degremont, Aurelien" 
Date : mercredi 10 juillet 2019 à 10:35
À : "lustre-discuss@lists.lustre.org" 
Objet : ZFS tunings

Hi all,

I know good default tunings for ZFS when used with Lustre is a big topic. There 
are several pages on wiki or LUG/LAD slides which are few years old and it is 
difficult to know which ones are still relevant when used with recent Lustre 
and ZFS versions.

Does anybody have insight about which tunings are important. Especially for:

  *   zfs atime (does DMU when used by Lustre also updates atime ?)
  *   redundant_metadata=most
  *   metaslab_debug_unload=1
  *   zfs_prefetch_disable=1
  *   zfs_vdev_scheduler=deadline

Lustre already sets ‘xattr=sa’, ‘dnodesize=auto’ and ‘recordsize=1M’ 
automatically since Lustre 2.11.

Thanks

Aurélien


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] obdfilter/mdt stats meaning ?

[Degremont, Aurelien]

Hi Louis,

About brw_stats, there are a bit of explanation in the Lustre Doc (not that 
detailed, but still)
http://doc.lustre.org/lustre_manual.xhtml#dbdoclet.50438271_55057

> Last thing, is there any way to get the name of the filesystem an OST is part 
> of by using lctl ?

I don't know what you want exactly, but the OST names are self explanatory, 
there always are like: fsname-OST
Where fsname is the lustre filesystem they are part of.

For obdfilter stats, these are mostly action to OST objects or client 
connection management RPCs.

setattr: changing an OST object attributes (owner, group, ...)
punch: mostly used for truncate (theorically can do holes in files, like 
truncate with a start and length)
sync: straighforward, sync OST to disk
destroy: delete an OST object (mostly when a file is deleted)
create: create an OST object
statfs: like 'df' for this specific OST (used by 'lfs df' by example)
(re)connect: when a client connect/reconnect to this OST
ping: when a client ping this OST.


Aurélien

De : lustre-discuss  au nom de Louis 
Bailleul 
Date : mardi 16 juillet 2019 à 16:38
À : lustre-discuss 
Objet : [lustre-discuss] obdfilter/mdt stats meaning ?

Hi all,

I am trying to make sense of some of the OST/MDT stats for 2.12.
Can anybody point me to the doc that explain what the metrics are ?
The wiki only mention read/write/get_info : 
http://wiki.lustre.org/Lustre_Monitoring_and_Statistics_Guide
But the list I get is quite different :
obdfilter.OST001.stats=
snapshot_time 1563285450.647120173 secs.nsecs
read_bytes340177708 samples [bytes] 4096 4194304 
396712660910080
write_bytes   30008856 samples [bytes] 24 4194304 78618271501667
setattr   1755 samples [reqs]
punch 73463 samples [reqs]
sync  50606 samples [reqs]
destroy   31990 samples [reqs]
create956 samples [reqs]
statfs75378743 samples [reqs]
connect   5798 samples [reqs]
reconnect 3242 samples [reqs]
disconnect5820 samples [reqs]
statfs3737980 samples [reqs]
preprw370186566 samples [reqs]
commitrw  370186557 samples [reqs]
ping  882096292 samples [reqs]
For the MDT, most are pretty much self explanatory, but I'll still be happy to 
be pointed to some doc.
mdt.MDT.md_stats=
snapshot_time 1563287416.006001068 secs.nsecs
open  3174644054 samples [reqs]
close 3174494603 samples [reqs]
mknod 107564 samples [reqs]
unlink99625 samples [reqs]
mkdir 199643 samples [reqs]
rmdir 45021 samples [reqs]
rename12728 samples [reqs]
getattr   50227431 samples [reqs]
setattr   103435 samples [reqs]
getxattr  9051470 samples [reqs]
setxattr  14 samples [reqs]
statfs7525513 samples [reqs]
sync  20597 samples [reqs]
samedir_rename207 samples [reqs]
crossdir_rename   12521 samples [reqs]
And anyone knows how to read the OST brw_stats ?
obdfilter.OST0014.brw_stats=
snapshot_time: 1563287631.511085465 (secs.nsecs)

   read  | write
pages per bulk r/w rpcs  % cum % |  rpcs% cum %
1:   231699298  66  66   | 180944   0   0
2:  855611   0  67   | 322359   1   1
4:  541749   0  67   | 5539716  18  20
8: 1281219   0  67   | 67837   0  20
16: 637808   0  67   | 114546   0  20
32:1342813   0  68   | 3099780  10  31
64:1559834   0  68   | 173166   0  31
128:   1583127   0  69   | 211512   0  32
256:  10627583   3  72   | 499978   1  34
512:   3909601   1  73   | 1029686   3  37
1K:   92141161  26 100   | 18788597  62 100

   read  | write
discontiguous pagesrpcs  % cum % |  rpcs% cum %
0:   346179839 100 100   | 180946   0   0
1:   0   0 100   | 322363   1   1
2:   0   0 100   | 5521062  18  20
3:   0   0 100   | 18650   0  20
4:   0   0 100   | 18159   0  20
5:   0   0 100   | 26664   0  20
6:   0   0 100   | 10830   0  20
7:   0   0 100   | 12189   0  20
8:   0   0 100   | 11365   0  20
9:   0   0 100   | 10253   0  20
10:  0   0 100   | 8810   0  20
11:  0   0 100   | 9825   0  20
12:  0   0 100   | 16740   0  20
13: 

[lustre-discuss] ZFS tunings

[Degremont, Aurelien]

Hi all,

I know good default tunings for ZFS when used with Lustre is a big topic. There 
are several pages on wiki or LUG/LAD slides which are few years old and it is 
difficult to know which ones are still relevant when used with recent Lustre 
and ZFS versions.

Does anybody have insight about which tunings are important. Especially for:

  *   zfs atime (does DMU when used by Lustre also updates atime ?)
  *   redundant_metadata=most
  *   metaslab_debug_unload=1
  *   zfs_prefetch_disable=1
  *   zfs_vdev_scheduler=deadline

Lustre already sets ‘xattr=sa’, ‘dnodesize=auto’ and ‘recordsize=1M’ 
automatically since Lustre 2.11.

Thanks

Aurélien


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] a question about max_create_count

[Degremont, Aurelien]

Hello Kurt,

max_create_count is a tunable per OST. It controls how many object 
pre-creations could be done on each OST, by the MDT. If you set this to 0 for 
one OST, no additional object will be allocated there and no new file will have 
file stripes on it.

With the appropriate hardware, Lustre can create much more than 20,000 files 
simultaneously.

Aurélien

De : lustre-discuss  au nom de Kurt 
Strosahl 
Date : mercredi 19 juin 2019 à 22:43
À : "lustre-discuss@lists.lustre.org" 
Objet : [lustre-discuss] a question about max_create_count

Good Afternoon,

I'm in the process of testing a new lustre file system at 2.12, and in 
going through the documentation I saw that to stop writes from going to the 
system we now set the max_create_count to 0 on the mdt.

   I looked at that value on my system and the default seems to be 2, am I 
correct in thinking that this is the maximum number of simultaneous creates 
that can happen on an OST?

w/r,

Kurt J. Strosahl
System Administrator: Lustre, HPC
Scientific Computing Group, Thomas Jefferson National Accelerator Facility
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] Connection restored to ...

[Degremont, Aurelien]

Hi folks!

I'm seeing a lot of "Connection restored to " messages in a situation where 
everything looks fine, with 2.10.5 filesystems.
Looking at the code, it looks like a client side thing but I'm seeing that on 
server side.

Does this mean something is bad on the system or this message is not relevant? 
Does somebody know what this message exactly means?

Lustre: lustre-MDT: Connection restored to 
591cfb0f-848a-6508-553b-46941e66ca13 (at 0@lo)
Lustre: MGS: Connection restored to b554cb80-6eb6-8135-b9dd-b331ccbd0352 (at 
172.20.2.18@tcp)


Thanks

Aurélien

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] ZFS tuning for MDT/MGS

[Degremont, Aurelien]

This is very unlikely.
The only reason that could happened is this hardware is acknowledging I/O to 
Lustre that it did not really commit to disk like writeback cache, or a Lustre 
bug. 

Le 02/04/2019 14:11, « lustre-discuss au nom de Hans Henrik Happe » 
 a écrit :

Isn't there a possibility that the MDS falsely tells the client that a
transaction has been committed to disk. After that the client might not
be able to replay, if the MDS dies.

Cheers,
Hans Henrik

On 19/03/2019 21.32, Andreas Dilger wrote:
> You would need to lose the MDS within a few seconds after the client to
> lose filesystem operations, since the clients will replay their
> operations if the MDS crashes, and ZFS commits the current transaction
> every 1s, so this setting only really affects "sync" from the client. 
> 
> Cheers, Andreas
> 
> On Mar 19, 2019, at 12:43, George Melikov  > wrote:
> 
>> Can you explain the reason about 'zfs set sync=disabled mdt0'? Are you
>> ready to lose last transaction on that mdt during power failure? What
>> did I miss?
>>
>> 14.03.2019, 01:00, "Riccardo Veraldi" > >:
>>> these are the zfs settings I use on my MDSes
>>>
>>>  zfs set mountpoint=none mdt0
>>>  zfs set sync=disabled mdt0
>>>
>>>  zfs set atime=off amdt0
>>>  zfs set redundant_metadata=most mdt0
>>>  zfs set xattr=sa mdt0
>>>
>>> if youor MDT partition is on a 4KB sector disk then you can use
>>> ashift=12 when you create the filesystem but zfs is pretty smart and
>>> in my case it recognized it automatically and used ashift=12
>>> automatically.
>>>
>>> also here are the zfs kernel modules parameters i use to ahve better
>>> performance. I use it on both MDS and OSSes
>>>
>>> options zfs zfs_prefetch_disable=1
>>> options zfs zfs_txg_history=120
>>> options zfs metaslab_debug_unload=1
>>> #
>>> options zfs zfs_vdev_scheduler=deadline
>>> options zfs zfs_vdev_async_write_active_min_dirty_percent=20
>>> #
>>> options zfs zfs_vdev_scrub_min_active=48
>>> options zfs zfs_vdev_scrub_max_active=128
>>> #options zfs zfs_vdev_sync_write_min_active=64
>>> #options zfs zfs_vdev_sync_write_max_active=128
>>> #
>>> options zfs zfs_vdev_sync_write_min_active=8
>>> options zfs zfs_vdev_sync_write_max_active=32
>>> options zfs zfs_vdev_sync_read_min_active=8
>>> options zfs zfs_vdev_sync_read_max_active=32
>>> options zfs zfs_vdev_async_read_min_active=8
>>> options zfs zfs_vdev_async_read_max_active=32
>>> options zfs zfs_top_maxinflight=320
>>> options zfs zfs_txg_timeout=30
>>> options zfs zfs_dirty_data_max_percent=40
>>> options zfs zfs_vdev_async_write_min_active=8
>>> options zfs zfs_vdev_async_write_max_active=32
>>>
>>> some people may disagree with me anyway after years of trying
>>> different options I reached this stable configuration.
>>>
>>> then there are a bunch of other important Lustre level optimizations
>>> that you can do if you are looking for performance increase.
>>>
>>> Cheers
>>>
>>> Rick
>>>
>>> On 3/13/19 11:44 AM, Kurt Strosahl wrote:

 Good Afternoon,


 I'm reviewing the zfs parameters for a new metadata system and I
 was looking to see if anyone had examples (good or bad) of zfs
 parameters?  I'm assuming that the MDT won't benefit from a
 recordsize of 1MB, and I've already set the ashift to 12.  I'm using
 an MDT/MGS made up of a stripe across mirrored ssds.


 w/r,

 Kurt


 ___
 lustre-discuss mailing list
 lustre-discuss@lists.lustre.org 

 http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.o… 

>>>
>>>
>>> ___
>>> lustre-discuss mailing list
>>> lustre-discuss@lists.lustre.org
>>> 
>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.o…
>>> 
>>>
>>
>>
>> 
>> Sincerely,
>> George Melikov
>>
>> ___
>> lustre-discuss mailing list
>> lustre-discuss@lists.lustre.org 
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> 
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> 

Re: [lustre-discuss] EINVAL error when writing to a PFL file (lustre 2.12.0)

[Degremont, Aurelien]

Another thought I just had while re-reading LU-9341 is whether it would be 
better to have the MDS always create files opened with O_APPEND with 
stripe_count=1?  There is no write parallelism for O_APPEND files, so having 
multiple stripes doesn't help the writer.  Because the writer also always locks 
the whole file [0,EOF] then there is no read-write parallelism either, so 
creating only a single file stripe simplifies things significantly with no real 
loss.

Having several stripes if still useful if you want to distribute space usage 
among several OSTs and also it could help if you have multiple readers later 
for this file.



Aurélien



Cheers, Andreas

On Feb 22, 2019, at 10:09, LEIBOVICI Thomas  wrote:
> 
> Hello Patrick,
> 
> Thank you for the quick reply.
> No, I have no particular use-case in mind, I'm just playing around with 
PFL.
> 
> If this is currently not properly supported, a quick fix could be to 
prevent the user from creating such incomplete layouts?
> 
> Regards,
> Thomas
> 
> On 2/22/19 5:33 PM, Patrick Farrell wrote:
>> Thomas,
>> 
>> This is expected, but it's also something we'd like to fix - See LU-9341.
>> 
>> Basically, append tries to instantiate the layout from 0 to infinity, 
and it fails because your layout is incomplete (ie doesn't go to infinity).
>> 
>> May I ask why you're creating a file with an incomplete layout?  Do you 
have a use case in mind?
>> 
>> - Patrick
>> From: lustre-discuss  on behalf 
of LEIBOVICI Thomas 
>> Sent: Friday, February 22, 2019 10:27:48 AM
>> To: lustre-discuss@lists.lustre.org
>> Subject: [lustre-discuss] EINVAL error when writing to a PFL file 
(lustre 2.12.0)
>>  
>> Hello,
>> 
>> Is it expected to get an error when appending a PFL file made of 2 
>> regions [0 - 1M] and [1M to 6M]
>> even if writing in this range?
>> 
>> I get an error when appending it, even when writting in the very first 
>> bytes:
>> 
>> [root@vm0]# lfs setstripe  -E 1M -c 1 -E 6M -c 2 /mnt/lustre/m_fou3
>> 
>> [root@vm0]# lfs getstripe /mnt/lustre/m_fou3
>> /mnt/lustre/m_fou3
>>lcm_layout_gen:2
>>lcm_mirror_count:  1
>>lcm_entry_count:   2
>>  lcme_id: 1
>>  lcme_mirror_id:  0
>>  lcme_flags:  init
>>  lcme_extent.e_start: 0
>>  lcme_extent.e_end:   1048576
>>lmm_stripe_count:  1
>>lmm_stripe_size:   1048576
>>lmm_pattern:   raid0
>>lmm_layout_gen:0
>>lmm_stripe_offset: 3
>>lmm_objects:
>>- 0: { l_ost_idx: 3, l_fid: [0x10003:0x9cf:0x0] }
>> 
>>  lcme_id: 2
>>  lcme_mirror_id:  0
>>  lcme_flags:  0
>>  lcme_extent.e_start: 1048576
>>  lcme_extent.e_end:   6291456
>>lmm_stripe_count:  2
>>lmm_stripe_size:   1048576
>>lmm_pattern:   raid0
>>lmm_layout_gen:0
>>lmm_stripe_offset: -1
>> 
>> [root@vm0]# stat -c %s /mnt/lustre/m_fou3
>> 14
>> 
>> * append fails:
>> 
>> [root@vm0]# echo qsdkjqslkdjkj >> /mnt/lustre/m_fou3
>> bash: echo: write error: Invalid argument
>> 
>> # strace indicates that write() gets the error:
>> 
>> write(1, "qsdkjqslkdjkj\n", 14) = -1 EINVAL (Invalid argument)
>> 
>> * no error in case of an open/truncate:
>> 
>> [root@vm0]# echo qsdkjqslkdjkj > /mnt/lustre/m_fou3
>> 
>> OK
>> 
>> Is it expected or should I open a ticket?
>> 
>> Thomas
>> 
>> ___
>> lustre-discuss mailing list
>> lustre-discuss@lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> 
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Cheers, Andreas
---
Andreas Dilger
CTO Whamcloud




___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] ZFS tuning for MDT/MGS

[Degremont, Aurelien]

Also, if you’re not using Lustre 2.11 or 2.12, do not forget dnodesize=auto and 
recordsize=1M for OST

zfs set dnodesize=auto mdt0
zfs set dnodesize=auto ostX

https://jira.whamcloud.com/browse/LU-8342

(useful for 2.10 LTS. Automatically done by Lustre for 2.11+)

De : lustre-discuss  au nom de 
"Carlson, Timothy S" 
Date : mercredi 13 mars 2019 à 23:07
À : Riccardo Veraldi , Kurt Strosahl 
, "lustre-discuss@lists.lustre.org" 

Objet : Re: [lustre-discuss] ZFS tuning for MDT/MGS

+1 on

options zfs zfs_prefetch_disable=1


Might not be as critical now, but that was a must-have on Lustre 2.5.x

Tim

From: lustre-discuss  On Behalf Of 
Riccardo Veraldi
Sent: Wednesday, March 13, 2019 3:00 PM
To: Kurt Strosahl ; lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] ZFS tuning for MDT/MGS

these are the zfs settings I use on my MDSes

 zfs set mountpoint=none mdt0
 zfs set sync=disabled mdt0
 zfs set atime=off amdt0
 zfs set redundant_metadata=most mdt0
 zfs set xattr=sa mdt0

if youor MDT partition is on a 4KB sector disk then you can use ashift=12 when 
you create the filesystem but zfs is pretty smart and in my case it recognized 
it automatically and used ashift=12 automatically.

also here are the zfs kernel modules parameters i use to ahve better 
performance. I use it on both MDS and OSSes

options zfs zfs_prefetch_disable=1
options zfs zfs_txg_history=120
options zfs metaslab_debug_unload=1
#
options zfs zfs_vdev_scheduler=deadline
options zfs zfs_vdev_async_write_active_min_dirty_percent=20
#
options zfs zfs_vdev_scrub_min_active=48
options zfs zfs_vdev_scrub_max_active=128
#options zfs zfs_vdev_sync_write_min_active=64
#options zfs zfs_vdev_sync_write_max_active=128
#
options zfs zfs_vdev_sync_write_min_active=8
options zfs zfs_vdev_sync_write_max_active=32
options zfs zfs_vdev_sync_read_min_active=8
options zfs zfs_vdev_sync_read_max_active=32
options zfs zfs_vdev_async_read_min_active=8
options zfs zfs_vdev_async_read_max_active=32
options zfs zfs_top_maxinflight=320
options zfs zfs_txg_timeout=30
options zfs zfs_dirty_data_max_percent=40
options zfs zfs_vdev_async_write_min_active=8
options zfs zfs_vdev_async_write_max_active=32

some people may disagree with me anyway after years of trying different options 
I reached this stable configuration.

then there are a bunch of other important Lustre level optimizations that you 
can do if you are looking for performance increase.

Cheers

Rick

On 3/13/19 11:44 AM, Kurt Strosahl wrote:

Good Afternoon,



I'm reviewing the zfs parameters for a new metadata system and I was 
looking to see if anyone had examples (good or bad) of zfs parameters?  I'm 
assuming that the MDT won't benefit from a recordsize of 1MB, and I've already 
set the ashift to 12.  I'm using an MDT/MGS made up of a stripe across mirrored 
ssds.



w/r,

Kurt




___

lustre-discuss mailing list

lustre-discuss@lists.lustre.org

http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Disable identity_upcall and ACL

[Degremont, Aurelien]

Thanks for the clarifications.

Aurélien

Le 14/01/2019 01:36, « Andreas Dilger »  a écrit :

On Jan 10, 2019, at 04:52, Degremont, Aurelien  wrote:
> 
> 
> Le 09/01/2019 21:39, « Andreas Dilger »  a écrit :
> 
>> If admins completely trust the client nodes (e.g. they are on a secure
>> network) or they completely _distrust_ them (e.g. subdirectory mounts
>> with nodemaps/idmaps and Kerberos/SSK to identify them), or the data
>> just isn't that secret, then allowing the client to handle the group
>> lookups instead of the MDS is mostly OK.  
>> 
>> The main issue is for new, uncached lookups from the client.  Since the
>> RPC only includes the UID, GID, and maybe one supplementary GID, it is
>> possible that the MDS (without the identity_upcall) may deny the lookup
>> because the request does not contain any IDs that would allow file 
access.
> 
> According to struct mdt_body there is room for only one suppgid.
> But the value is not always packed in mdc, depending on the call.
> So that means that hopefully between 0 and 1 supplementary group will be 
passed to MDT, if I read the code correctly.
> 
>> I guess the other question is why you are interested to get rid of it,
>> or what issue you are seeing with it enabled?
> 
> If identity_upcall is enabled, you need an up to date group database 
available on MDS.  This is not always the case. I'm trusting the clients in 
this case. I would be interesting in having the MDT doing no credential checks 
and letting the clients (VFS) do all the validations. MDT is already trusting 
client when it is sending uid and gid.
> 
> So, coming back to my original question, the ACL warning message in MDT 
is not really limited to ACL but more generally to any supplementary groups 
checks. Some accesses could be denied if they rely on supplementary groups 
(likely not the first one) and could be wrongly granted or denied if based on 
ACL. Correct?

Correct.

> Permission checks for primary uid/gid is always correct, whatever 
identity_upcall value?

Yes, definitely.  If the client "knows" the right suppgid then it will send 
it to the MDS as well, but otherwise it just picks the first one.

Cheers, Andreas
---
Andreas Dilger
Principal Lustre Architect
Whamcloud









___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Kernel Module Build

[Degremont, Aurelien]

ake[1]: Entering directory '/nfshome/attaufer/lustre-release'
>> if test -d "lustre-2.12.0"; then find "lustre-2.12.0" -type d ! -perm 
-200 -exec chmod u+w {} ';' && rm -rf "lustre-2.12.0" || { sleep 5 && rm -rf 
"lustre-2.12.0"; }; else :; fi
>> test -d "lustre-2.12.0" || mkdir "lustre-2.12.0"
>> (cd ldiskfs && make  top_distdir=../lustre-2.12.0 
distdir=../lustre-2.12.0/ldiskfs \
>>am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: 
distdir)
>> make[2]: Entering directory '/nfshome/attaufer/lustre-release/ldiskfs'
>> make[2]: Leaving directory '/nfshome/attaufer/lustre-release/ldiskfs'
>> (cd lustre-iokit && make  top_distdir=../lustre-2.12.0 
distdir=../lustre-2.12.0/lustre-iokit \
> 
> I don't think we need this whole spew each time...  Usually the errors
> are in the last 20 or so lines.
> 
> You never really mentioned it, but what distro are you using?  I've 
recently built on both RHEL6 and RHEL7 using this process without problems (the 
build mechanism for the full release packages is somewhat different and more 
complex), so it seems likely that there is something strange with your 
environment.
> 
> Based on the 4.14 kernel, I'd assume RHEL7.5 alt?  Any strange RPM macros 
that might be affecting the build?  Have you tried on a different system?
> 
> Cheers, Andreas
> 
>> tardir=lustre-2.12.0 && false | GZIP=--best gzip -c >lustre-2.12.0.tar.gz
>> make[1]: Leaving directory '/nfshome/attaufer/lustre-release'
>> if test -d "lustre-2.12.0"; then find "lustre-2.12.0" -type d ! -perm 
-200 -exec chmod u+w {} ';' && rm -rf "lustre-2.12.0" || { sleep 5 && rm -rf 
"lustre-2.12.0"; }; else :; fi
>> rpmbuilddir=`mktemp -t -d rpmbuild-lustre-$USER-`; \
>> make  \
>>   rpmbuilddir="$rpmbuilddir" rpm-local || exit 1; \
>> cp ./rpm/* .; \
>> /usr/bin/rpmbuild \
>>   --define "_tmppath $rpmbuilddir/TMP" \
>>   --define "_topdir $rpmbuilddir" \
>>   --define "dist %{nil}" \
>>   -ts lustre-2.12.0.tar.gz || exit 1; \
>> cp $rpmbuilddir/SRPMS/lustre-2.12.0-*.src.rpm . || exit 1; \
>> rm -rf $rpmbuilddir
>> make[1]: Entering directory '/nfshome/attaufer/lustre-release'
>> make[1]: Leaving directory '/nfshome/attaufer/lustre-release'
>> error: Failed to read spec file from lustre-2.12.0.tar.gz
>> autoMakefile:1189: recipe for target 'srpm' failed
> 
>> make: *** [srpm] Error 1
>> 
>> Andrew Tauferner
>> 1-952-562-4944 (office)
>> 
>> -Original Message-
>> From: Andreas Dilger [mailto:adil...@whamcloud.com] 
>> Sent: Thursday, January 10, 2019 3:39 PM
>> To: Tauferner, Andrew T 
>> Cc: Degremont, Aurelien ; 
lustre-discuss@lists.lustre.org
>> Subject: Re: [lustre-discuss] Kernel Module Build
>> 
>> Did you run "sh autogen.sh && ./configure" before your "make rpms"?  The 
lustre.spec file should be generated by configure.  You can check this before 
running the "make rpms", and check the tarball to see if it is included. 
>> 
>> Cheers, Andreas
>> 
>>> On Jan 10, 2019, at 14:19, Tauferner, Andrew T 
 wrote:
>>> 
>>> Hmm.  Same error with v2_12_0.  It would seem that the 4.14 kernel 
isn't the issue.
>>> 
>>> attaufer@head-2:~/lustre-release> git status
>>> On branch 2.12.0
>>> nothing to commit, working tree clean
>>> attaufer@head-2:~/lustre-release> make rpms
>>> make  dist-gzip am__post_remove_distdir='@:'
>>> make[1]: Entering directory '/nfshome/attaufer/lustre-release'
>>> :
>>> tardir=lustre-2.12.0 && false | GZIP=--best gzip -c 
>lustre-2.12.0.tar.gz
>>> make[1]: Leaving directory '/nfshome/attaufer/lustre-release'
>>> if test -d "lustre-2.12.0"; then find "lustre-2.12.0" -type d ! -perm 
-200 -exec chmod u+w {} ';' && rm -rf "lustre-2.12.0" || { sleep 5 && rm -rf 
"lustre-2.12.0"; }; else :; fi
>>> rpmbuilddir=`mktemp -t -d rpmbuild-lustre-$USER-`; \
>>> make  \
>>>  rpmbuilddir="$rpmbuilddir" rpm-local || exit 1; \
>>> cp ./rpm/* .; \
>>> /usr/bin/rpmbuild \
>>>  --define "_tmppath $rpmbuilddir/TMP"

Re: [lustre-discuss] Kernel Module Build

[Degremont, Aurelien]

2.10.6 does not support Linux 4.14.
There is several patches that needs to be cherry picked to have it working.

2.12 is fine though.


Aurélien 

Le 09/01/2019 18:43, « lustre-discuss au nom de Tauferner, Andrew T » 
 a écrit :

I configured like so:

attaufer@head-2:~/lustre-release> ./configure --disable-server 
--with-linux=/nfshome/attaufer/kernel/usr/src/kernels/4.14.54-mos-tc3-20181113

The make failed as follows:

attaufer@head-2:~/lustre-release> make rpms
make  dist-gzip am__post_remove_distdir='@:'
make[1]: Entering directory '/nfshome/attaufer/lustre-release'
grep -v config.h.in config.h.in > undef.h
if test -d "lustre-2.10.6"; then find "lustre-2.10.6" -type d ! -perm -200 
-exec chmod u+w {} ';' && rm -rf "lustre-2.10.6" || { sleep 5 && rm -rf 
"lustre-2.10.6"; }; else :; fi
test -d "lustre-2.10.6" || mkdir "lustre-2.10.6"
 (cd ldiskfs && make  top_distdir=../lustre-2.10.6 
distdir=../lustre-2.10.6/ldiskfs \
 am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: 
distdir)
make[2]: Entering directory '/nfshome/attaufer/lustre-release/ldiskfs'
make[2]: Leaving directory '/nfshome/attaufer/lustre-release/ldiskfs'
 (cd lustre-iokit && make  top_distdir=../lustre-2.10.6 
distdir=../lustre-2.10.6/lustre-iokit \
 am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: 
distdir)
make[2]: Entering directory '/nfshome/attaufer/lustre-release/lustre-iokit'
 (cd obdfilter-survey && make  top_distdir=../../lustre-2.10.6 
distdir=../../lustre-2.10.6/lustre-iokit/obdfilter-survey \
 am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: 
distdir)
make[3]: Entering directory 
'/nfshome/attaufer/lustre-release/lustre-iokit/obdfilter-survey'
make[3]: Leaving directory 
'/nfshome/attaufer/lustre-release/lustre-iokit/obdfilter-survey'
 (cd sgpdd-survey && make  top_distdir=../../lustre-2.10.6 
distdir=../../lustre-2.10.6/lustre-iokit/sgpdd-survey \
 am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: 
distdir)
make[3]: Entering directory 
'/nfshome/attaufer/lustre-release/lustre-iokit/sgpdd-survey'
make[3]: Leaving directory 
'/nfshome/attaufer/lustre-release/lustre-iokit/sgpdd-survey'
 (cd ost-survey && make  top_distdir=../../lustre-2.10.6 
distdir=../../lustre-2.10.6/lustre-iokit/ost-survey \
 am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: 
distdir)
make[3]: Entering directory 
'/nfshome/attaufer/lustre-release/lustre-iokit/ost-survey'
make[3]: Leaving directory 
'/nfshome/attaufer/lustre-release/lustre-iokit/ost-survey'
 (cd ior-survey && make  top_distdir=../../lustre-2.10.6 
distdir=../../lustre-2.10.6/lustre-iokit/ior-survey \
 am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: 
distdir)
make[3]: Entering directory 
'/nfshome/attaufer/lustre-release/lustre-iokit/ior-survey'
make[3]: Leaving directory 
'/nfshome/attaufer/lustre-release/lustre-iokit/ior-survey'
 (cd mds-survey && make  top_distdir=../../lustre-2.10.6 
distdir=../../lustre-2.10.6/lustre-iokit/mds-survey \
 am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: 
distdir)
make[3]: Entering directory 
'/nfshome/attaufer/lustre-release/lustre-iokit/mds-survey'
make[3]: Leaving directory 
'/nfshome/attaufer/lustre-release/lustre-iokit/mds-survey'
 (cd stats-collect && make  top_distdir=../../lustre-2.10.6 
distdir=../../lustre-2.10.6/lustre-iokit/stats-collect \
 am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: 
distdir)
make[3]: Entering directory 
'/nfshome/attaufer/lustre-release/lustre-iokit/stats-collect'
make[3]: Leaving directory 
'/nfshome/attaufer/lustre-release/lustre-iokit/stats-collect'
make[2]: Leaving directory '/nfshome/attaufer/lustre-release/lustre-iokit'
 (cd libcfs && make  top_distdir=../lustre-2.10.6 
distdir=../lustre-2.10.6/libcfs \
 am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: 
distdir)
make[2]: Entering directory '/nfshome/attaufer/lustre-release/libcfs'
 (cd libcfs && make  top_distdir=../../lustre-2.10.6 
distdir=../../lustre-2.10.6/libcfs/libcfs \
 am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: 
distdir)
make[3]: Entering directory '/nfshome/attaufer/lustre-release/libcfs/libcfs'
 (cd linux && make  top_distdir=../../../lustre-2.10.6 
distdir=../../../lustre-2.10.6/libcfs/libcfs/linux \
 am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: 
distdir)
make[4]: Entering directory 
'/nfshome/attaufer/lustre-release/libcfs/libcfs/linux'
make[4]: Leaving directory 
'/nfshome/attaufer/lustre-release/libcfs/libcfs/linux'
 (cd util && make  top_distdir=../../../lustre-2.10.6 
distdir=../../../lustre-2.10.6/libcfs/libcfs/util \
 am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: 
distdir)

Re: [lustre-discuss] Disable identity_upcall and ACL

[Degremont, Aurelien]

Hi Daniel!

The secondary group thing was ok to me. I got this idea even if there is some 
weird results during my tests. Looks like you can overwrite MDT checks if user 
and group is properly defined on client node. Cache effects?

ACL is really the thing I was interested in. Who is validating the ACLs? MDT, 
client or both? Do you think ACL could be properly applied if user/groups are 
only defined on client side and identity_upcall is disabled on MDT side?

Thanks

Aurélien

Le 09/01/2019 12:22, « lustre-discuss au nom de Daniel Kobras » 
 a 
écrit :

Hi Aurélien!

Am 09.01.19 um 11:48 schrieb Degremont, Aurelien:
> When disabling identity_upcall on a MDT, you get this message in system
> logs:
> 
>   lustre-MDT: disable "identity_upcall" with ACL enabled maybe cause
> unexpected "EACCESS"
> 
> I’m trying to understand what could be a scenario that shows this problem?
> What is the implication, or rather, how identity_upcall works?

Without an identity_upcall, all Lustre users effectively lose their
secondary group memberships. These are not passed in the RPCs, but
evaluated on the MDS instead. The default l_getidentity receives a
numeric uid, queries NSS to obtain the corresponding account's list of
gids, and passes the list back to the kernel. As a test scenario, just
try to access a file or directory from an account that only has access
permissions via one of its secondardy groups. (The log message is a bit
misleading--you don't actually need to use ACLs, ordinary group
permissions are sufficient.)

Kind regards,

Daniel
-- 
Daniel Kobras
Principal Architect
Puzzle ITC Deutschland
+49 7071 14316 0
www.puzzle-itc.de

-- 
Puzzle ITC Deutschland GmbH
Sitz der Gesellschaft:  Jurastr. 27/1, 72072 
Tübingen
Eingetragen am Amtsgericht Stuttgart HRB 765802
Geschäftsführer: 
Lukas Kallies, Daniel Kobras, Mark Pröhl

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] Disable identity_upcall and ACL

[Degremont, Aurelien]

Hello all,

When disabling identity_upcall on a MDT, you get this message in system logs:

  lustre-MDT: disable "identity_upcall" with ACL enabled maybe cause 
unexpected "EACCESS"

I’m trying to understand what could be a scenario that shows this problem?
What is the implication, or rather, how identity_upcall works?

Thanks for your help

Aurélien

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] LAD'18: Last days to benefit from early bird rates

[DEGREMONT Aurelien]

*LAD'18 - Lustre Administrators and Developers Workshop*
September 24th-25th, 2018
Marriott Champs Elysées Hotel, Paris - France

These are the last days to benefit from early bird rates for LAD!
Register before August 26th and get a discount on registration fees.
 http://lad.eofs.org/register.php

Look at the very interesting topics that will be presented during these 
2 days: http://www.eofs.eu/events/lad18



We are very pleased this event is organized thanks to the following 
generous sponsors:

ATOS, CEA, CRAY, DDN, INTEL

For any other information, please contact l...@eofs.eu
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lfs data_version $filename

[DEGREMONT Aurelien]

Hello

This is feature is primarily used by HSM, internally, but is not limited to.
data_version is a number computed from each OST objects the file is made of.

Two different values for data_version mean the file content has changed. 
Two identical values mean the content is identical. There is not 
relationship like older or more recent. There is no order between values.

Values from 2 different files cannot be compared. This has no sense.


Aurélien


Le 12/04/2018 à 17:28, E.S. Rosenberg a écrit :
This is just a guess - could it be related to ZFS backends? Allowing 
for snapshots etc.


On Thu, Apr 5, 2018 at 10:53 PM, Ms. Megan Larko > wrote:


Greetings List!

What is the number from the command "lfs data_version $filename "
telling me?

I do not see "data_version" documented in lfs -h, man lfs, nor in
lustre manual..

I do know that if I have a zero-length file my robinhood scan of
my Lustre mount point indicates that "lfs get_version" failed.  As
the file in-question is zero length, I have no problem with this
(other than lines in my robinhood lustre.log file making it large).

I understand that Lustre File System uses three numbers to
validate data integrity.  How does the number returned from "lfs
data_version" come into play?  What reference number is that one?

Thank you in advance for the education.
Cheers,
megan

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org

http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org





___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Announce: Lustre Systems Administration Guide

[DEGREMONT Aurelien]

Hi

My current understanding of Documentation status is that it is hard to 
update and we want to make this easy to encourage people to improve it. 
Correct? I clearly understand this point and it makes sense.


My point of view on this topic, is that MediaWiki is not a good fit for 
an official documentation.


* It is hard to organize pages in MediaWiki. Pages do not have a logical 
flow. Hard to have a sequence of pages or tasks. You just have pages and 
categories. This is quickly a mess.
* Hard to export pages, hard to convert to another format. Documentation 
will be only browsable online. No more external PDF (useful when you do 
not have Internet access on your compute cluster). Plugins exists but 
are hard to install.
* I do not have in mind a good project documentation which is using 
MediaWiki to maintain it.


My understanding of the current workflow for editing doc is:
- create a JIRA ticket
- git clone the source
- edit the docbook doc
- push modification to Gerrit
- wait for approval

My proposal/Features I like with readthedocs-like system are:
- Nice looking HTML documentation (https://docs.readthedocs.io/en/latest/)
- Easy to download in nice looking PDF or HTML version (see links at 
bottom left)
- Could be edited online thanks to "Edit on GitHub" link (top right), 
which will transform your edit into a PR. This is few clicks and online 
editing, in your browser, no need for an explicit git-clone and a terminal.

- Changes could be moderated (thanks to PR)
- Documentation has clear versions (see v1.0.2 link at bottom left 
http://graphite.readthedocs.io/en/1.0.2/ ).
- Documentation is automatically updated when new commits are made, 
thanks to callbacks.


Indeed, this requires a Git repo for documentation management. I agree 
this is similar to what we currently have, but I think we can drop JIRA 
ticket requirement.


Regarding documentation syntax, Sphinx (backend behind readthedocs) 
supports Markdown and ReStructuredText (rst). Both formats are really 
popular at the moment and, even if I used to use docbook in the past and 
was fine with it, I know a lot of people is complaining about its XML 
style syntax. Markdown and RST have a wiki-style syntax which is much 
more user friendly.


It is also possible to deploy our own Sphinx runtime on lustre.org 
servers if we can manage to setup the same feature readthedocs.org put 
on their own.


I hope this helps clarify my point of view.


This does not blame the really good content that was recently put on 
wiki.lustre.org, just the format, not the content :)



Aurélien

Le 15/12/2017 à 09:32, Dilger, Andreas a écrit :

On Dec 13, 2017, at 09:02, DEGREMONT Aurelien <aurelien.degrem...@cea.fr> wrote:

Hello

My recommendation would be to go for something like Documentation-as-a-code.
It is now very easy to deploy an infrastructure to automatically generate nice 
looking HTML and PDF versions of a rst or markdown documentation thanks to 
readthedocs.io

Maybe I'm missing something, but it seems like the Sphynix markup is just a 
different form of inline text markup compared to DocBook XML that the current 
manual is using?  Also, we already host the manual repo in Git, and 
automatically build PDF, HTML, and ePub formats.

My main thought about moving to a wiki format is that this makes it easier for users to 
contribute to the manual, since the text could be edited "in place" in a web 
browser.  Hopefully that would reduce the barrier to entry, and more people would update 
the manual and improve the places where it is outdated.


readthedocs.io uses Sphinx as backend, likewise kernel.org 
(https://www.kernel.org/doc/html/v4.11/doc-guide/sphinx.html)

A github project could be easy plugged to readthedocs, but you could use your 
own hooks.

We started with DOxygen markup for the code (I don't know why it wasn't the 
same as the kernel), but it would probably be possible to write a script to 
convert the existing code markup to Sphynx if we wanted.


If you use GitHub to store your source code, you can easily edit the 
documentation source code and create a pull request. I know this is not in line 
with Gerrit usage, but this is an interesting workflow.

Could you please explain?  It isn't clear to me how this is significantly 
different from Git+Gerrit?  It is also possible to edit source files directly 
in Gerrit, though it isn't clear to me if it is possible to _start_ editing a 
file and submit a patch vs. being able to edit an existing change.  In either 
case, one still needs to use the inline markup language.  Does Github have a 
WYSIWYG editor for the manual?


I really think we should really uses such technologies which produce very nice 
looking documentation, which could be easily exports, versionned and coupled 
with a git repository. See how kernel.org uses it.

I'm not necessarily against this, but it would be a considerable amount of work 
to convert the existing manual, and there would have 

Re: [lustre-discuss] Announce: Lustre Systems Administration Guide

[DEGREMONT Aurelien]

Hello

My recommendation would be to go for something like Documentation-as-a-code.
It is now very easy to deploy an infrastructure to automatically 
generate nice looking HTML and PDF versions of a rst or markdown 
documentation thanks to readthedocs.io


readthedocs.io uses Sphinx as backend, likewise kernel.org 
(https://www.kernel.org/doc/html/v4.11/doc-guide/sphinx.html)


A github project could be easy plugged to readthedocs, but you could use 
your own hooks.


If you use GitHub to store your source code, you can easily edit the 
documentation source code and create a pull request. I know this is not 
in line with Gerrit usage, but this is an interesting workflow.


I really think we should really uses such technologies which produce 
very nice looking documentation, which could be easily exports, 
versionned and coupled with a git repository. See how kernel.org uses it.


MediaWiki is not a good fit for doc manual like Lustre one. Not easy to 
browse, not easy to export.



My 2 cents

Aurélien

Le 17/11/2017 à 23:03, Dilger, Andreas a écrit :

On Nov 16, 2017, at 22:41, Cowe, Malcolm J  wrote:

I am pleased to announce the availability of a new systems administration guide 
for the Lustre file system, which has been published to wiki.lustre.org. The 
content can be accessed directly from the front page of the wiki, or from the 
following URL:
  
http://wiki.lustre.org/Category:Lustre_Systems_Administration
  
The guide is intended to provide comprehensive instructions for the installation and configuration of production-ready Lustre storage clusters. Topics covered:
  
	• Introduction to Lustre

• Lustre File System Components
• Lustre Software Installation
• Lustre Networking (LNet)
• LNet Router Configuration
• Lustre Object Storage Devices (OSDs)
• Creating Lustre File System Services
• Mounting a Lustre File System on Client Nodes
• Starting and Stopping Lustre Services
• Lustre High Availability
  
Refer to the front page of the guide for the complete table of contents.

Malcolm,
thanks so much for your work on this.  It is definitely improving the
state of the documentation available today.

I was wondering if people have an opinion on whether we should remove
some/all of the administration content from the Lustre Operations Manual,
and make that more of a reference manual that contains details of
commands, architecture, features, etc. as a second-level reference from
the wiki admin guide?

For that matter, should we export the XML Manual into the wiki and
leave it there?  We'd have to make sure that the wiki is being indexed
by Google for easier searching before we could do that.

Cheers, Andreas


In addition, for people who are new to Lustre, there is a high-level 
introduction to Lustre concepts, available as a PDF download:
  
http://wiki.lustre.org/images/6/64/LustreArchitecture-v4.pdf
  
  
Malcolm Cowe

High Performance Data Division
  
Intel Corporation | www.intel.com
  
___

lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Intel Corporation







___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] LAD'17 agenda is online

[DEGREMONT Aurelien]

*LAD'17 - Lustre Administrators and Developers Workshop*
October 4th-5th, 2017
Salon des Arts et Métiers, Paris - France


We are pleased to announce that agenda for Lustre Administrator and 
Developer Workshop 2017 is now available online:

http://www.eofs.eu/events/lad17

Please register before August 27st to benefit from early bird rate.
http://lad.eofs.org/register.php


We are very pleased this event is organized thanks to the following 
generous sponsors:

ATOS, CEA, DDN, INTEL and SEAGATE

For any other information, please contact l...@eofs.eu
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] LAD'17 - Last call!

[DEGREMONT Aurelien]

Hello Lustre community!

These are the last days to send abstracts for LAD'17, do not wait! We 
will be pleased to hear about your Lustre experiences. Not only 
developers but also from sites or admins presenting their Lustre 
deployment and experiences with it.


https://easychair.org/conferences/?conf=lad17


*ABOUT*

EOFS and OpenSFS are happy to announce the 7th LAD will be held in 
Paris, France, at Salon des Arts et Métiers! This will be a 2-day event, 
from 4th to 5th of October, 2017. This will be a great opportunity for 
worldwide Lustre administrators and developers to gather and exchange 
their experiences, developments, tools, good practices and more.


**https://www.eofs.eu/events/lad17

*CALL FOR PAPERS
*

We invite community members to send proposals for presentation.
Those talks could cover, by example, tools, administration experiences, 
configuration setup, developments, tunings, tweaking, ...
No proceeding is required, just an abstract for a 30-min technical 
presentation (including Q).


Please send your abstracts using EasyChair, before July 31th, 2017.
https://easychair.org/conferences/?conf=lad17

*REGISTRATION
*Registration for the workshop is open: http://lad.eofs.eu/
*
DATE CHANGE!*
Important! Please note that LAD dates have shifted by one day from what 
was originally announced!
LAD will now take place from October 4th to October 5th, which means 
from Wednesday to Thursday.


*SOCIAL EVENT*
On Wednesday evening, a dinner will take place at the top of Tour 
Montparnasse, Paris highest skyscraper! A limited number of spouses can 
also attend (on a first-come, first-served basis).


*SPONSORS*
We are very pleased this event is organized thanks to the following 
generous sponsors:

 ATOS, CEA, DDN, INTEL and SEAGATE

For any other information, please contact l...@eofs.eu

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] LAD'17 - Call for paper

[DEGREMONT Aurelien]

*LAD'17 - Lustre Administrators and Developers Workshop*

*October**4th-5th, 2017
Salon des Arts et Métiers**, Paris - France*

EOFS and OpenSFS are happy to announce the 7th LAD will be held in 
Paris, France, at Salon des Arts et Métiers! This will be a 2-day event, 
from 4th to 5th of October, 2017. This will be a great opportunity for 
worldwide Lustre administrators and developers to gather and exchange 
their experiences, developments, tools, good practices and more.


We are waiting for your registration and presentation proposals!


*WEB PAGE
*https://www.eofs.eu/events/lad17

*PRESENTATION*
We invite community members to send proposals for presentation.
Those talks could cover, by example, tools, administration experiences, 
configuration setup, developments, tunings, tweaking, ...
No proceeding is required, just an abstract of a 30-min (technical) 
presentation.


Please send your abstracts using EasyChair, before July 30th, 2017.
https://easychair.org/conferences/?conf=lad17

*DATE CHANGE!*
Important! Please note that LAD dates have shifted by one day!
LAD will now take place from October 4th to October 5th, which means 
from Wednesday to Thursday.


*SPONSORS*
We are very pleased this event is organized thanks to the following 
generous sponsors:

ATOS, CEA, DDN and INTEL

For any other information, please contact l...@eofs.eu
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] LAD'17 - Save the date!

[DEGREMONT Aurelien]

*LAD'17 - Lustre Administrator and Developer Workshop**

**October 3rd - 4th, 2017**
**Hôtel des Arts et Métiers, Paris - France*

EOFS and OpenSFS are happy to announce the 7th LAD will be held in 
Paris, France, at the Salon de l'Hôtel des Arts et Métiers! This will be 
a 2-day event, from 3rd to 4th of October, 2017. This will be a great 
opportunity for worldwide Lustre administrators and developers to gather 
and exchange their experiences, developments, tools, good practices and 
more.


Please save the date!

*WEB PAGE*
All information on http://www.eofs.eu/events/lad17

*SPONSORS*
This event is organized thanks to the following generous sponsors:
Atos, CEA, Intel


For any other information, please contact l...@eofs.eu
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] LAD'16 agenda is online

[DEGREMONT Aurelien]

*LAD'16 - Lustre Administrator and Developer Workshop*
September 20th - 21st, 2016
Hotel Renaissance Trocadero, Paris - France


We are pleased to announce that agenda for Lustre Administrator and 
Developer Workshop 2016 is now available online:

http://www.eofs.eu/?id=lad16

Please register before August 21st to benefit from early bird rate.
http://lad.eofs.org/register.php


We are very pleased this event is organized thanks to the following 
generous sponsors:

ATOS, CEA, CRAY, DDN, INTEL and SEAGATE

For any other information, please contact l...@eofs.eu
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] LAD'15 agenda is online

[DEGREMONT Aurelien]

*LAD'15 - Lustre Administrator and Developer Workshop*
September 22th - 23th, 2015
Paris Marriott Champs Elysees Hotel, Paris - France

We are pleased to announce that agenda for Lustre Administrator and 
Developer Workshop 2015 is now available online:

http://www.eofs.eu/?id=lad15

Please register before August 23rd to benefit from early bird rate.
http://lad.eofs.org/register.php


We are very pleased this event is organized thanks to the following 
generous sponsors:

ATOS, CEA, CRAY, DDN, INTEL and SEAGATE

For any other information, please contact l...@eofs.eu


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] LAD'15: Call for papers extended up to August 4th

[DEGREMONT Aurelien]

*LAD'15 - Lustre Administrator and Developer Workshop*
September 22th - 23th, 2015
Paris Marriott Champs Elysees Hotel, Paris - France

*CALL FOR PAPERS*
We are extending call for papers up to August 4th, 2015. You have one 
more week to send your abstract!
We are inviting community members to send proposals for presentations at 
this event. No proceeding is required, just an abstract of a 30-min 
(technical) presentation.

Please send this to l...@eofs.eu

Topics may include (but are not limited to): site updates or future 
projects, Lustre administration, monitoring and tools, Lustre feature 
overview, Lustre client performance (benefits of hardware evolution to 
Lustre (like SSD, many-cores…), comparison between Lustre and other 
parallel file systems (perf. and/or features), Lustre and Exascale I/O, 
tunings, etc.


*REGISTRATION*
Registration for the workshop is open:
http://lad.eofs.org/register.php

*WEB SITE*
Get all details on http://www.eofs.eu/?id=lad15

*SPONSORS*
We are very pleased this event is organized thanks to the following 
generous sponsors:

ATOS, CEA, CRAY, DDN, INTEL and SEAGATE

For any other information, please contact l...@eofs.eu
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] LAD'15 - Call for paper and registration is open!

[DEGREMONT Aurelien]

*LAD'15 - Lustre Administrator and Developer Workshop*

*September 22th - 23th, 2015**
**Paris Marriott Champs Elysees Hotel, Paris - France*

EOFS and OpenSFS are happy to announce the 5th LAD will be held in 
Paris, France, at Marriott Hotel on Champs-Elysees Avenue! This will be 
a 2-day event, from 22nd to 23rd of September, 2015. This will be a 
great opportunity for worldwide Lustre administrators and developers to 
gather and exchange their experiences, developments, tools, good 
practices and more.


We are waiting for your registration and presentation proposals!


*WEB PAGE*
http://www.eofs.eu/?id=lad15

*PRESENTATION*
We invite community members to send proposals for presentation.
Those talks could cover, by example, tools, administration experiences, 
configuration setup, developments, tunings, tweaking, ...
No proceeding is required, just an abstract of a 30-min (technical) 
presentation.

Please send this to la...@eofs.eu before July 24th, 2015.

*REGISTRATION*
Registration for the workshop is now open:

http://lad.eofs.org/register.php

*SOCIAL EVENT*
On Tuesday evening, there will be a dinner at Eiffel Tower. A limited 
number of spouses can attend too (on a first-come, first-served basis).


*SPONSORS*
We are very pleased this event is organized thanks to the following 
generous sponsors:

ATOS, CEA, DDN and INTEL

For any other information, please contact la...@eofs.eu
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[Lustre-discuss] LAD'13 - Agenda and end of early bird rate

[DEGREMONT Aurelien]

*LAD'13 - Lustre Administrators and Developers Workshop*

*September 16th - 17th, 2013**
**Hotel Lutetia, Paris - France*


AGENDA
The LAD'13 agenda is now available online!
Check LAD web page: http://www.eofs.eu/?id=lad13

REGISTRATION
There is only very few days left to benefit from from the early bird 
rate, up to August, 10th

http://lad.eofs.org/register.php

We are waiting for you! If you have already registered, tell your 
friends and colleagues to do the same! :)



SOCIAL EVENT
On Monday evening, there will be a dinner and a boat tour on the Seine 
river, aboard the Excellence Yacht. A limited number of spouses can 
attend too (on a first-come, first-served basis). Register quickly!


SPONSORS
This event is organized thanks to the following generous sponsors:
Bull, CEA, DataDirect Networks, Intel, NetApp and Xyratex.

For any other information, please contact la...@eofs.eu
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] lustre startup sequence Re: OSTs not activating following MGS/MDS move

[DEGREMONT Aurelien]

Hello

AFAIK there is 2 orders:
  - If you are started your filesystem for the first time (or using 
--writeconf), order is :
MGS, MDS, OST, Clients
  - On normal start
MGS, OST, MDS, Clients

There is a patch on some recent Lustre release to be able to use the 
first order any time but I would advise to use the second one anyway as 
it avoids starting MDS first, lacking connection to OST, and then 
reconnecting to them when they are really started.


Aurélien


Le 07/03/2013 17:48, Colin Faber a écrit :
 Hi Yes,

 Thanks for finding this Alex. The manual should be updated with the
 correct order.

 -cf



 On 03/07/2013 09:39 AM, Alex Kulyavtsev wrote:
 Hi Colin.
 This is not what the manual says.

 Shall it be corrected then? Or, add description for startup sequence
 in different situations (first start, restart).

 The manual (or online information) does not describe graceful shutdown
 sequence for separate MGS/MDT configuration, it will be nice to add
 that too.

 Alex.

 E.g.
 http://wiki.lustre.org/manual/LustreManual20_HTML/LustreOperations.html#50438194_24122
 and similar
 http://build.whamcloud.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#dbdoclet.50438194_24122

  13.2 Starting Lustre

 The startup order of Lustre components depends on whether you have a
 combined MGS/MDT or these components are separate.

* If you have a combined MGS/MDT, the recommended startup order is
  OSTs, then the MGS/MDT, and then clients.

* If the MGS and MDT are separate, the recommended startup order
  is: *MGS, then OSTs, then the MDT, and then clients.*



 On Mar 7, 2013, at 9:51 AM, Colin Faber wrote:

 Hi Christopher,

 In general this can happen when your initial remount of the various
 services is in thewrong order.

 Such as MGS - OST - MDT - Client. or MGS - MDT - Clients - OST,
 etc.

 During initial mount and registration it's critical that your mount be
 in the correct order:

 MGS - MDT - OST(s) - Client(s)

 CATALOG corruption, or out of order sequence is more rare on active file
 system, but is possible. The simple fix here as described below is to
 just truncate it and all should be well again.

 -cf

 ailing list
 Lustre-discuss@lists.lustre.org mailto:Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss


 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss
 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] Object index

[DEGREMONT Aurelien]

Le 24/07/2012 20:10, Daniel Kobras a écrit :

 Is this the troglodyte type of OST that started its life in times of 
 prehistoric versions of Lustre? We see this on old files that were created in 
 the early ages of Lustre 1.6, before the trusted.fid EA was introduced.
No, this filesystem was formatted with Lustre 2.0
By the way, does someone remember the incompatibility with 2.0/2.1 which 
prevent a target, formatted with Lustre 2.1 to 
be downgraded to Lustre 2.0 ?

 Other than that, these objects could have been preallocated, but never 
 actually used. Do these objects contain any data at all (blockcount != 0)?
I was rather thinking of that. But I'm surprised that so many objects are 
preallocated.

 -Some of them have good results, and the man page says that
 For objects with MDT parent sequence numbers above 0x2, this 
 indicates that the FID needs to be mapped via the
 MDT Object Index (OI) file on the MDT.
 How do I do this mapping? I found some iam utilities but they do not seems 
 to be ok, and I'm afraid IAM userspace code
 has been deactivated.
 lfs fid2path (on any client) should do what you're looking for.
It does not. Moreover, lfs does not support this kind of fid
[0x20a5df05f:0x4874:0x0]
[0x20a6e8d8c:0x27b4:0x1]

Lustre Manual said The idx field shows the stripe number of this OST object in 
the Lustre RAID-0 striped file. 

Which seems true as I've got several files where idx  0.
But, lfs fid sanity check is :
static inline int fid_is_sane(const struct lu_fid *fid)
{
 return
 fid != NULL 
 ((fid_seq(fid) = FID_SEQ_START  fid_oid(fid) != 0
 fid_ver(fid) == 0) ||
 fid_is_igif(fid));
}

And so complains when fid_ver != 0

I'm not sure at all lfs fid2patch expect fid coming from OST.

  From my experience, a small amount of object leakage is not too uncommon on 
 real-world systems, so if lfs find doesn't show up any objects anymore, most 
 likely you're good to take this OST down.
I agree on that, but I consider that more than 4k objects is not a small 
amount :)

 (Hey, and you can double-check with rbh-report --dump-ost, of course! ;-)
Sure, but I did not have an rbh DB for that FS available (a pity as 
rbh-report is few minutes in worst cases, lfs 
find was 15 hours :))
By the way, using a Lustre tool helps me to be sure the remaining objects were 
not related to a possible robinhood bug :)


Aurélien
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] Object index

[DEGREMONT Aurelien]

Hello

I'm trying to analyze an OST with few thousands of object and find where they 
belong to.
Mounting this OST with ldiskfs and using ll_decode_filter_fid tells me that.

-Most of these object do not have a fid EA back pointer. Does that means they 
are not used?
-Some of them have good results, and the man page says that
For objects with MDT parent sequence numbers above 0x2, this indicates 
that the FID needs to be mapped via the 
MDT Object Index (OI) file on the MDT.
How do I do this mapping? I found some iam utilities but they do not seems to 
be ok, and I'm afraid IAM userspace code 
has been deactivated.

How can I know if those files could be removed without risk.
I previously checked that lfs find did not find any other files with object 
on this specific OST I'm working on.

Thanks

Aurélien Degrémont
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] python lustre os.stat(path)

[DEGREMONT Aurelien]

Mark Day a écrit :
 robinhood supplies this list, if you are using lustre 2.0 it directly 
 queries the mdt.

 http://sourceforge.net/apps/trac/robinhood

 On this subject does anyone know of a tool/app that tracks folder sizes?
Robinhood can also do this.


Aurélien

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] reorder OSTs?

[DEGREMONT Aurelien]

Liam Forbes a écrit :
 I just built a lustre filesystem, and it appears to be working fine.
 However, I think I'd like to reorder the OSTs in order to spread the
 work across the OSSes better.  Is this something I can do on the fly,
 or do I have to rebuild the filesystem in a different order?  If the
 latter, I need to do that now before we put any users on it.  Or maybe
 the ordering doesn't matter?
   
You cannot reorder them but you do not need too.
If your file are striped, Lustre will automatically balanced the load, 
choosing OSTs on different OSS in priority.


Aurélien

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] European Lustre Workshop - Registration is now open!

[DEGREMONT Aurelien]

European Lustre Workshop 2011
September 26th - 27th 2011
Hotel Pullman Paris Bercy - Paris, France

EOFS is organizing the first European Lustre Workshop, in Paris, at 
Hotel Pullman Paris Bercy during 2 days, 26th and 27th of September, 2011.
This will be a great opportunity for Lustre(tm) administrators and 
developers from Europe and worldwide to gather and exchange their 
experiences, developments, tools, good practices.
This event will take place in the venue of the Pullman Paris Bercy Hotel.
http://www.eofs.org/index.php?id=5

REGISTRATION
Registration for the European Lustre Workshop is now open!
http://www.eofs.org/workshop2011/

PRESENTATION
It is still possible to submit proposal for presentation during this 
workshop until the 1st of September.
No proceeding is required, just an abstract of a 30-min presentation. 
Please send this to workshop2...@eofs.org .

SPONSORS
This is event is organized thanks to the generous sponsoring of BULL and 
XYRATEX.

For any other information, please contact workshop2...@eofs.org
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] Too many client eviction

[DEGREMONT Aurelien]

Nathan Rutman a écrit :
 On May 3, 2011, at 10:09 AM, DEGREMONT Aurelien wrote:

 Server and client cooperate together for the adaptive timeouts.
Ok they cooperate, the client will change its timeout through this 
cooperation, but will also do the same ?
If yes, obd_timeout and ldlm_timeout are very rarely used now?

   I don't remember which bug the ORNL settings were in, maybe 14071, 
 bugzilla's not responding at the moment.  
Bugzilla is ok now, 14071 is the AT feature bug. The ORNL settings are 
not there.



Aurélien
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] Too many client eviction

[DEGREMONT Aurelien]

Hello

Andreas Dilger a écrit :
 On May 3, 2011, at 13:41, Nathan Rutman wrote:
   
 On May 3, 2011, at 10:09 AM, DEGREMONT Aurelien wrote:
 
 Correct me if I'm wrong, but when I'm looking at Lustre manual, it said 
 that client is adapting its timeout, but not the server. I'm understood 
 that server-client RPC still use the old mechanism, especially for our 
 case where it seems server is revoking a client lock (ldlm_timeout is 
 used for that?) and client did not respond.
   
 Server and client cooperate together for the adaptive timeouts.  I don't 
 remember which bug the ORNL settings were in, maybe 14071, bugzilla's not 
 responding at the moment.  But a big question here is why 25315 seconds for 
 a callback - that's well beyond anything at_max should allow...
 

 I assume that the 25315s is from a bug (fixed in 1.8.5 I think, not sure if 
 it was ported to 2.x) that calculated the wrong time when printing this error 
 message for LDLM lock timeouts.
   
I did not find the bug for that.
 I forgot to say that we have LNET routers also involved for some cases.
   
 If there are routers they can cause dropped RPCs from the server to the 
 client, and the client will be evicted for unresponsiveness even though it is 
 not at fault.  At one time Johann was working on a patch (or at least 
 investigating) the ability to have servers resend RPCs before evicting 
 clients.  The tricky part is that you don't want to send 2 RPCs each with 1/2 
 the timeout interval, since that may reduce stability instead of increasing 
 it.
   
How can I track those dropped RPCs on routers?
Is this an expected behaviour? How could I protect my filesystem from 
that? If I increase the timeout this won't change anything if 
client/server do not re-send their RPC.

 I think the bugzilla bug was called limited server-side resend or similar, 
 filed by me several years ago.
   
Did not find either :)

Aurélien
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] Too many client eviction

[DEGREMONT Aurelien]

Johann Lombardi a écrit :
 On Wed, May 04, 2011 at 01:37:14PM +0200, DEGREMONT Aurelien wrote:
   
 I assume that the 25315s is from a bug
   
 BTW, do you see this problem with both extent  inodebits locks?
   
Yes both. But more often on MDS.
 How can I track those dropped RPCs on routers?
 

 I don't think routers can drop RPCs w/o a good reason. It is just that a 
 router failure can lead to packet loss and given that servers don't resend 
 local callbacks, this can result in client evictions.
   
Currently I do not see any issue with the routers.
Logs are very silent and load is very low. Nothing looks like router 
failure.
If LNET decides to drop packet for some buggy reason, I would expect to 
have it, at least, say something in kernel log (omg i've drop 2 
packets, please expect evictions :))

 if client/server do not re-send their RPC.
 
 To be clear, clients go through a disconnect/reconnect cycle and eventually 
 resend RPCs.
   
I'm not sure I understand clearly what happens there.
If client did not respond to server ast, it will be evicted by the 
server. Server do not seem to send a message to tell it (why bother as 
it seems it is unresponsive or dead anyway?).
Client realizes at next obd_ping that connection does not exist anymore 
(rc=-107 ENOTCONN).
Then it try to reconnect, and at that time, server tells it, it is 
really evicted. Client says in progress operation will fail. AFAIK, 
this means dropping all locks, all dirty pages. Async I/O are lost. 
Connection status becomes EVICTED. I/O during this window will receive 
-108, ESHUTDOWN, (kernel log said @@@ IMP_INVALID, see 
ptlrpc_import_delay_req()).
Then client reconnects, but some I/O were lost, user program could have 
experienced errors from I/O syscall.

This is not the same as a connection timeout, where client will try a 
failover and do a disconnect/recovery cycle, everything is ok.

Is this correct?

 That's bug 3622. Fanyong also used to work on a patch, see 
 http://review.whamcloud.com/#change,125.
   
This looks very interesting as it seems to match our issue. But 
unfortunately, no news since 2 months.



Aurélien

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] Too many client eviction

[DEGREMONT Aurelien]

Hello

We often see some of our Lustre clients being evicted abusively (clients 
seem healthy).
The pattern is always the same:

All of this on Lustre 2.0, with adaptative timeout enabled

1 - A server complains about a client :
### lock callback timer expired... after 25315s...
(nothing on client)

(few seconds later)

2 - The client receives -107 to a obd_ping for this target
(server says @@@processing error 107)

3 - Client realize its connection was lost.
Client notices it was evicted.
It reconnects.

(To be sure) When client is evicted, all undergoing I/O are lost, no 
recovery will be done for that?

We are thinking to increase timeout to give more time to clients to 
answer the ldlm revocation.
(maybe it is just too loaded)
- Is ldlm_timeout enough to do so?
- Do we need to also change obd_timeout in accordance? Is there a risk 
to trigger new timeouts if we just change ldlm_timeout (cascading timeout).

Any feedback in this area is welcomed.

Thank you

Aurélien Degrémont
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] Too many client eviction

[DEGREMONT Aurelien]

Correct me if I'm wrong, but when I'm looking at Lustre manual, it said 
that client is adapting its timeout, but not the server. I'm understood 
that server-client RPC still use the old mechanism, especially for our 
case where it seems server is revoking a client lock (ldlm_timeout is 
used for that?) and client did not respond.

I forgot to say that we have LNET routers also involved for some cases.

Thank you

Aurélien

Andreas Dilger a écrit :
 I don't think ldlm_timeout and obd_timeout have much effect when AT is 
 enabled. I believe that LLNL has some adjusted tunables for AT that might 
 help for you (increased at_min, etc).

 Hopefully Chris or someone at LLNL can comment. I think they were also 
 documented in bugzilla, though I don't know the bug number. 

 Cheers, Andreas

 On 2011-05-03, at 6:59 AM, DEGREMONT Aurelien aurelien.degrem...@cea.fr 
 wrote:

   
 Hello

 We often see some of our Lustre clients being evicted abusively (clients 
 seem healthy).
 The pattern is always the same:

 All of this on Lustre 2.0, with adaptative timeout enabled

 1 - A server complains about a client :
 ### lock callback timer expired... after 25315s...
 (nothing on client)

 (few seconds later)

 2 - The client receives -107 to a obd_ping for this target
 (server says @@@processing error 107)

 3 - Client realize its connection was lost.
 Client notices it was evicted.
 It reconnects.

 (To be sure) When client is evicted, all undergoing I/O are lost, no 
 recovery will be done for that?

 We are thinking to increase timeout to give more time to clients to 
 answer the ldlm revocation.
 (maybe it is just too loaded)
 - Is ldlm_timeout enough to do so?
 - Do we need to also change obd_timeout in accordance? Is there a risk 
 to trigger new timeouts if we just change ldlm_timeout (cascading timeout).

 Any feedback in this area is welcomed.

 Thank you

 Aurélien Degrémont
 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss
 

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] Lustre 2.0 client cache size

[DEGREMONT Aurelien]

Jason Rappleye a écrit :
 On Mar 18, 2011, at 1:07 AM, DEGREMONT Aurelien wrote:
   
 Yes, that would totally make sense to do regardless of other methods.

 Hmm... I do not want to patch 'cp' or 'dd' :)
 

 You might want to have a look at this:

 http://code.google.com/p/pagecache-mangagement/

 We use it when copying crash dumps off of our OSSes, as we don't want to 
 disturb cached metadata.
   
Interesting project. I will have a look. Thank you.


Aurélien

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] Lustre 2.0 client cache size

[DEGREMONT Aurelien]

Hello

I'm trying to control the amount of cache memory used by Lustre clients 
with Lustre 2.0
There is a /proc tuning called max_cache_mb in llite which is 
documented in Lustre manual.
Unfortunately, after testing it and checking the source code it seems 
this variable is present but it not used anymore in Lustre code.
This is not the case in 1.8

I did not find if this was removed or this was partially included in 
Lustre 2.0.
What's the current status of this and how can I tell to my client to 
avoid caching too many data?
Those clients do a lot of read and write in Lustre filesystems but 
thoses files will not be re-read soon, so it is useless to fill memory 
with it. Moreover, when the client memory is full, Lustre performance 
are really impacted.

Thanks in advance

Aurélien Degrémont
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] clients gets EINTR from time to time

[DEGREMONT Aurelien]

Hello

 From my understanding, Lustre can return EINTR for some I/O error cases.
I think that when a client gets evicted in the middle of one of its RPC, 
it can returns EINTR to the caller.
Is this can explain your issue?

Can your verify your clients where not evicted at the same time?

Aurélien

Francois Chassaing a écrit :
 OK, thanks it makes it more clear.
 I indeed messed up my mind (and words) between signals and error return codes.
 I did understood that the write()/pwrite() system was returning the EINTR 
 error code because it received a signal, but I supposed that the signal was 
 sent because of an error condition somewhere in the FS. 
 This is where I now think I'm wrong. 
  
 As for your questions :
 - I have to mention that I always had had this issue, and this is why I've 
 upgraded from 1.8.4 to 1.8.5, hoping this would solve it.
 - I will try to have that SA_RESTART flag set in the app... if I can find 
 where the signal handler is set.
 - How can I see that lustre is returning EINTR for any other reason ? As I 
 said no logs shows nothing neither on MDS or OSSs, but I didn't go through 
 examining lctl debug_kernel yet... which I'm going to do right away...

 my last question is : how can I tell which signal I am receiving ? because my 
 app doesn't say, it just dumps outs the write/pwrite error code. 
 And if there is no signal handler, then it should follow the standard 
 actions (as of man 7 signal). On the other hand, my app does not stop or dump 
 core, and is not ignored, so it has to be handled in the code. Correct me if 
 I'm wrong...

 At that point, you realize that I didn't write the app, nor am I a good Linux 
 guru ;-)

 Tnaks a lot.

 weborama  lineFrançois Chassaing Directeur Technique - CTO 

 - Mail Original -
 De: Ken Hornstein k...@cmf.nrl.navy.mil
 À: Francois Chassaing f...@weborama.com
 Cc: lustre-discuss@lists.lustre.org
 Envoyé: Jeudi 24 Février 2011 15h54:24 GMT +01:00 Amsterdam / Berlin / Berne 
 / Rome / Stockholm / Vienne
 Objet: Re: [Lustre-discuss] clients gets EINTR from time to time

   
 OK, the app is used to deal with standard disks, that is why it is not
 handling the EINTR signal propoerly.
 

 I think you're misunderstanding what a signal is in the Unix sense.

 EINTR isn't a signal; it's a return code from the write() system call
 that says, Hey, you got a signal in the middle of this write() call
 and it didn't complete.  It doesn't mean that there was an error
 writing the file; if that was happening, you'd get a (presumably
 different) error code.  Signals can be sent by the operating system,
 but those signals are things like SIGSEGV, which basically means, you're
 program screwed up.  Programs can also send signals to each other,
 with kill(2) and the like.

 Now, NORMALLY systems calls like write() are interrupted by signals
 when you're writing to slow devices, like network sockets.  According
 to the signal(7) man page, disks are not normally considered slow
 devices, so I can understand the application not being used to handling
 this.  And you know, now that I think about it I'm not even sure that
 network filesystems SHOULD allow I/O system calls to be interrupted by
 signals ... I'd have to think more about it.

 I suspect what happened is that something changed between 1.8.5 and the
 previous version of Lustre that you were using that allowed some operations
 to be interruptable by signals.  Some things to try:

 - Check to see if you are, in fact, receiving a signal in your application
   and Lustre isn't returning EINTR for some other reason.
 - If you are receiving a signal, when you set the signal handler for it
   you could use the SA_RESTART flag to restart the interrupted I/O; I think
   that would make everything work like it did before.

 --Ken
 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss
   

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] ne2scan and fsfind

[DEGREMONT Aurelien]

Hello

I advise you to have a look at Robinhood, which has a ton of features to 
handle Lustre file management:
http://robinhood.sf.net/ http://robinhood.sourceforge.net

And if you think this does not fit your needs I will be very interested 
to know why.

Regards,

Aurélien Degrémont

Andrus, Brian Contractor a écrit :

 Hello all!

 I have been googling around trying to find some good tools to handle 
 the large fileystems we have grown here and see reference to ‘ne2scan’ 
 and ‘fsfind’.

 They sound great, but I am not finding a place to get them.

 Does anyone out there know where these gems are hidden?

 Brian Andrus

 ITACS/Research Computing

 Naval Postgraduate School

 Monterey, California

 voice: 831-656-6238

 

 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss
   

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] Lustre community build server

[DEGREMONT Aurelien]

Hi

Nathan Rutman a écrit :
 Hi Aurelien, Robert  - 

 We also use Hudson and are interested in using it to do Lustre builds 
 and testing.
 Hi

 Robert Read a écrit :
  Hi Aurélien,
 
  Yes, we've noticed Hudson's support for testing is not quite what we need, 
  so 
  we're planning to use Hudson to trigger our testing system, but not 
  necessarily to manage it.  We'd definitely be interested in learning more 
  about your experiences, though. 

 I do not know what you mean by triggering your testing system. But here 
 is what I set up.
 Hudson has only 1 slave node dedicated to testing Lustre 2.
 Hudson will launch a shell script through ssh to it.

 This script:
  - retrieves Lustre source (managed by Hudson git plugin)
  - compiles it.
  - launches acceptance-small with several parameters.
  - acceptance-small will connect to other nodes dedicated for these tests.

 acc-sm have been patched:
 - to be more error resilient (does not stop at first failure)
 - to generate a test report in JUNIT format.
 
 Is this the yaml acc-sm that Robert was talking about, or an older one?
I think so.
We modified the current test-framework.sh  and acceptance-small.sh, from 
master.
We reused the call introduced to produce a result.yml script to produce 
a junit-report.xml script

 Hudson fetch the junit report and parse it thanks to its plugin.
 Hudson can display in its interface all tests successes and failures.

 Everything goes fine as long as:
  - the testsuite leaves the node in a good shape. It is difficult to 
 have a automatic way to put the node back. Currently, we need to manualy 
 fix that.
 
 Would it be helpful to run the test in a VM?  Hudson has a 
 libvirt-slave plugin that
 seems like it can start and stop a VM for you.  Another point I like 
 about VM's is
 that they can be suspended and shipped to an investigator for local 
 debugging.
I notice this plugin for Hudson and also had the same analysis.
I did not try to setup it yet, but this as some side effects anyway and 
won't be a perfect solution for sure.

I find your idea to send the VM interesting. Not sure it could be done 
easily, but could be a great help for debugging if this is doable.
  - Hudson does not know about the other nodes used by acc-sm. And so can 
 trigger tests even if some sattelites nodes are unavailable.
 
 Don't know if libvirt-slave can handle multiple
 VM's for a multi-node Lustre test, but maybe it can be extended.
Even without testing Lustre I will be very pleased if Hudson can 
allocate several nodes for 1 run and give the node list to the test 
script through variable by example.
This could be very interesting for us, as we are working on clustering 
stuff, we always need group of nodes whatever the tests are. This is not 
easy to handle in Hudson. We have to cheat :p

 How is you do this on your side?
 
 It seems that like you, we are more interested in reporting the results
 within Hudson as opposed to a different custom tool.
In this case, I'm quite sure we can converge toward the same patch for that.

Robert's ideas are more ambitious and want to handle more generic uses. 
I will not be displeased by that as long as the test results could be 
analyzed by Hudson plugins.



Aurelien
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] split OSTs from single OSS in 2 networks

[DEGREMONT Aurelien]

Hello

If you want to register different interfaces for different OST on the 
same OSS, you should use --network options, introduced in patch 
https://bugzilla.lustre.org/show_bug.cgi?id=22078

Regards,

Aurélien

Haisong Cai a écrit :
 I have a storage server that has two QPI contolllers,
 each controller contols half of the I/O slots. In the first half is
 a raid controller and 10GbE card, in the second half is a raid controller
 and a 10GbE card.

 4 raid arrays for 4 OSTs of single Lustre filesystem.

 2 10GbE, each configured with their own IP and switches,
 IPSUBNET1 and IPSUBNET2.

 I want to register OSTs on this OSS with different addresses.
 That is, OST1  OST2 in IPSUBNET1, OST3  OST4 in IPSUBNET2,
 and using policy-based routing directing traffic.

 The idea is to not do the bonding and get as much of bandwidth
 out of 10GbE NICs as possible.

 Is it possible?

   

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] git server down?

[DEGREMONT Aurelien]

It seems Oracle Broomfield labs is down today, this impacted not just
the git repo, but other services such as bugzilla.
Monday was a holiday in US.
The business will resume today, I expect that the lab service will be 
restored today


Aurélien

Wojciech Turek a écrit :
 I am afraid that lustre wiki and bugzilla are not working either.



 On 18 January 2011 10:29, Götz Waschk goetz.was...@gmail.com 
 mailto:goetz.was...@gmail.com wrote:

 Hi everyone,

 I have tried to access the git server but have received this error:

 % git clone git://git.lustre.org/prime/lustre.git
 http://git.lustre.org/prime/lustre.git
 Initialized empty Git repository in /tmp/lustre/.git/
 fatal: read error: Connection reset by peer

 Is this a known problem, will it be fixed soon?

 Regards, Götz Waschk
 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.org
 mailto:Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss



 

 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss
   

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss