Re: [lustre-discuss] lfs_migrate

2017-03-20 Thread Dilger, Andreas 
If you mark the OSTs degraded and the MDS will still avoid them for new 
allocations, though you should make a second scanning pass to verify.

Deactivating the OSTs on the MDS will allow it to delete the (now unused) OST 
objects.

Cheers, Andreas

On Mar 20, 2017, at 17:03, E.S. Rosenberg 
> wrote:



On Mon, Mar 20, 2017 at 10:59 PM, Dilger, Andreas 
> wrote:
If you've marked the OST inactive on the MDS then that is not surprising. See 
https://jira.hpdd.intel.com/browse/LU-4825 and the comments in my previous 
email.
Ah OK.
But if I re-activate the OST will lfs_migrate still move them away from the 
device?

Cheers, Andreas

On Mar 20, 2017, at 16:56, E.S. Rosenberg 
> wrote:



On Mon, Mar 20, 2017 at 10:50 PM, E.S. Rosenberg 
> wrote:


On Mon, Mar 20, 2017 at 10:19 PM, Dilger, Andreas 
> wrote:
The underlying "lfs migrate" command (not the "lfs_migrate" script) in newer 
Lustre versions (2.9) is capable of migrating files that are in use by using 
the "--block" option, which prevents other processes from accessing or 
modifying the file during migration.

Unfortunately, "lfs_migrate" doesn't pass that argument on, though it wouldn't 
be hard to change the script. Ideally, the "lfs_migrate" script would pass all 
unknown options to "lfs migrate".


The other item of note is that setting the OST inactive on the MDS will prevent 
the MDS from deleting objects on the OST (see 
https://jira.hpdd.intel.com/browse/LU-4825 for details).  In Lustre 2.9 and 
later it is possible to set on the MDS:

   mds# lctl set_param osp..create_count=0

to stop MDS allocation of new objects on that OST. On older versions it is 
possible to set on the OSS:

  oss# lctl set_param obdfilter..degraded=1

so that it tells the MDS to avoid it if possible, but this isn't a hard 
exclusion.

It is also possible to use a testing hack to mark an OST as out of inodes, but 
that only works for one OST per OSS and it sounds like that won't be useful in 
this case.

Cheers, Andreas
You're making me want Lustre 2.9 more :) but for now I'm still stuck on 2.8 and 
because this is very much production these days I'm more careful with the 
update (hoping to finally get hw allocated for a test env soon to test the 
update).
Thanks,
Eli
Another related question:
The migration has been running for several hours now on one OST but I am yet to 
see 1 block being freed from the OSS point of view, is this not mv but cp as 
far as the original OST is concerned?
(also man lfs has no lfs migrate that was added in 2.9?)
Thanks,
Eli

On Mar 20, 2017, at 13:11, Brett Lee 
> wrote:

Hi Eli,

I believe that is still the case with lfs_migrate.  If otherwise, we'll 
probably hear soon.

You should be able to disable those OSTs while keeping the file system active - 
via a command on the MDS(s) as well as the clients.  My notes have the command 
as shown below, but please confirm via the appropriate Lustre manual:

lctl set_param osc.--*.active=0

Brett
--
Protect Yourself Against Cybercrime
PDS Software Solutions LLC
https://www.TrustPDS.com

On Mon, Mar 20, 2017 at 10:43 AM, E.S. Rosenberg 
> wrote:
In the man page it says the following:

Because  lfs_migrate  is  not yet closely integrated with the MDS, it cannot 
determine whether a file is currently open and/or in-use by other applications 
or nodes.  That makes it UNSAFE for use on files that might be modified by 
other applications, since the migrated file is only a copy of the current file, 
and this will result in the old file becoming an open-unlinked file and any  
modifications to that file will be lost.

Is this still the case?
Is there a better way to disable OSTs while keeping the filesystem live?

Background:
We need to take a OSS enclosure that hosts multiple OSTs offline for hardware 
maintenance, I'd like to do this without bringing Lustre as a whole down. I 
made sure there is enough space on the other OSTs to house the contents of the 
machine going offline and am now about to move things.

Thanks,
Eli

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org



___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org

Re: [lustre-discuss] lfs_migrate

2017-03-20 Thread Dilger, Andreas 
If you've marked the OST inactive on the MDS then that is not surprising. See 
https://jira.hpdd.intel.com/browse/LU-4825 and the comments in my previous 
email.

Cheers, Andreas

On Mar 20, 2017, at 16:56, E.S. Rosenberg 
> wrote:



On Mon, Mar 20, 2017 at 10:50 PM, E.S. Rosenberg 
> wrote:


On Mon, Mar 20, 2017 at 10:19 PM, Dilger, Andreas 
> wrote:
The underlying "lfs migrate" command (not the "lfs_migrate" script) in newer 
Lustre versions (2.9) is capable of migrating files that are in use by using 
the "--block" option, which prevents other processes from accessing or 
modifying the file during migration.

Unfortunately, "lfs_migrate" doesn't pass that argument on, though it wouldn't 
be hard to change the script. Ideally, the "lfs_migrate" script would pass all 
unknown options to "lfs migrate".


The other item of note is that setting the OST inactive on the MDS will prevent 
the MDS from deleting objects on the OST (see 
https://jira.hpdd.intel.com/browse/LU-4825 for details).  In Lustre 2.9 and 
later it is possible to set on the MDS:

   mds# lctl set_param osp..create_count=0

to stop MDS allocation of new objects on that OST. On older versions it is 
possible to set on the OSS:

  oss# lctl set_param obdfilter..degraded=1

so that it tells the MDS to avoid it if possible, but this isn't a hard 
exclusion.

It is also possible to use a testing hack to mark an OST as out of inodes, but 
that only works for one OST per OSS and it sounds like that won't be useful in 
this case.

Cheers, Andreas
You're making me want Lustre 2.9 more :) but for now I'm still stuck on 2.8 and 
because this is very much production these days I'm more careful with the 
update (hoping to finally get hw allocated for a test env soon to test the 
update).
Thanks,
Eli
Another related question:
The migration has been running for several hours now on one OST but I am yet to 
see 1 block being freed from the OSS point of view, is this not mv but cp as 
far as the original OST is concerned?
(also man lfs has no lfs migrate that was added in 2.9?)
Thanks,
Eli

On Mar 20, 2017, at 13:11, Brett Lee 
> wrote:

Hi Eli,

I believe that is still the case with lfs_migrate.  If otherwise, we'll 
probably hear soon.

You should be able to disable those OSTs while keeping the file system active - 
via a command on the MDS(s) as well as the clients.  My notes have the command 
as shown below, but please confirm via the appropriate Lustre manual:

lctl set_param osc.--*.active=0

Brett
--
Protect Yourself Against Cybercrime
PDS Software Solutions LLC
https://www.TrustPDS.com

On Mon, Mar 20, 2017 at 10:43 AM, E.S. Rosenberg 
> wrote:
In the man page it says the following:

Because  lfs_migrate  is  not yet closely integrated with the MDS, it cannot 
determine whether a file is currently open and/or in-use by other applications 
or nodes.  That makes it UNSAFE for use on files that might be modified by 
other applications, since the migrated file is only a copy of the current file, 
and this will result in the old file becoming an open-unlinked file and any  
modifications to that file will be lost.

Is this still the case?
Is there a better way to disable OSTs while keeping the filesystem live?

Background:
We need to take a OSS enclosure that hosts multiple OSTs offline for hardware 
maintenance, I'd like to do this without bringing Lustre as a whole down. I 
made sure there is enough space on the other OSTs to house the contents of the 
machine going offline and am now about to move things.

Thanks,
Eli

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lfs_migrate

2017-03-20 Thread E.S. Rosenberg 
On Mon, Mar 20, 2017 at 10:19 PM, Dilger, Andreas 
wrote:

> The underlying "lfs migrate" command (not the "lfs_migrate" script) in
> newer Lustre versions (2.9) is capable of migrating files that are in use
> by using the "--block" option, which prevents other processes from
> accessing or modifying the file during migration.
>
> Unfortunately, "lfs_migrate" doesn't pass that argument on, though it
> wouldn't be hard to change the script. Ideally, the "lfs_migrate" script
> would pass all unknown options to "lfs migrate".
>
>
> The other item of note is that setting the OST inactive on the MDS will
> prevent the MDS from deleting objects on the OST (see
> https://jira.hpdd.intel.com/browse/LU-4825 for details).  In Lustre 2.9
> and later it is possible to set on the MDS:
>
>mds# lctl set_param osp..create_count=0
>
> to stop MDS allocation of new objects on that OST. On older versions it is
> possible to set on the OSS:
>
>   oss# lctl set_param obdfilter..degraded=1
>
> so that it tells the MDS to avoid it if possible, but this isn't a hard
> exclusion.
>
> It is also possible to use a testing hack to mark an OST as out of inodes,
> but that only works for one OST per OSS and it sounds like that won't be
> useful in this case.
>
> Cheers, Andreas
>
You're making me want Lustre 2.9 more :) but for now I'm still stuck on 2.8
and because this is very much production these days I'm more careful with
the update (hoping to finally get hw allocated for a test env soon to test
the update).
Thanks,
Eli

>
> On Mar 20, 2017, at 13:11, Brett Lee  wrote:
>
> Hi Eli,
>
> I believe that is still the case with lfs_migrate.  If otherwise, we'll
> probably hear soon.
>
> You should be able to disable those OSTs while keeping the file system
> active - via a command on the MDS(s) as well as the clients.  My notes have
> the command as shown below, but please confirm via the appropriate Lustre
> manual:
>
> lctl set_param osc.--*.active=0
>
> Brett
> --
> Protect Yourself Against Cybercrime
> PDS Software Solutions LLC
> https://www.TrustPDS.com 
>
> On Mon, Mar 20, 2017 at 10:43 AM, E.S. Rosenberg <
> esr+lus...@mail.hebrew.edu> wrote:
>
>> In the man page it says the following:
>>
>> Because  lfs_migrate  is  not yet closely integrated with the MDS, it
>> cannot determine whether a file is currently open and/or in-use by other
>> applications or nodes.  That makes it UNSAFE for use on files that might be
>> modified by other applications, since the migrated file is only a copy of
>> the current file, and this will result in the old file becoming an
>> open-unlinked file and any  modifications to that file will be lost.
>>
>> Is this still the case?
>> Is there a better way to disable OSTs while keeping the filesystem live?
>>
>> Background:
>> We need to take a OSS enclosure that hosts multiple OSTs offline for
>> hardware maintenance, I'd like to do this without bringing Lustre as a
>> whole down. I made sure there is enough space on the other OSTs to house
>> the contents of the machine going offline and am now about to move things.
>>
>> Thanks,
>> Eli
>>
>> ___
>> lustre-discuss mailing list
>> lustre-discuss@lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>
>>
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
>
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lfs_migrate

2017-03-20 Thread Dilger, Andreas 
The underlying "lfs migrate" command (not the "lfs_migrate" script) in newer 
Lustre versions (2.9) is capable of migrating files that are in use by using 
the "--block" option, which prevents other processes from accessing or 
modifying the file during migration.

Unfortunately, "lfs_migrate" doesn't pass that argument on, though it wouldn't 
be hard to change the script. Ideally, the "lfs_migrate" script would pass all 
unknown options to "lfs migrate".


The other item of note is that setting the OST inactive on the MDS will prevent 
the MDS from deleting objects on the OST (see 
https://jira.hpdd.intel.com/browse/LU-4825 for details).  In Lustre 2.9 and 
later it is possible to set on the MDS:

   mds# lctl set_param osp..create_count=0

to stop MDS allocation of new objects on that OST. On older versions it is 
possible to set on the OSS:

  oss# lctl set_param obdfilter..degraded=1

so that it tells the MDS to avoid it if possible, but this isn't a hard 
exclusion.

It is also possible to use a testing hack to mark an OST as out of inodes, but 
that only works for one OST per OSS and it sounds like that won't be useful in 
this case.

Cheers, Andreas

On Mar 20, 2017, at 13:11, Brett Lee 
> wrote:

Hi Eli,

I believe that is still the case with lfs_migrate.  If otherwise, we'll 
probably hear soon.

You should be able to disable those OSTs while keeping the file system active - 
via a command on the MDS(s) as well as the clients.  My notes have the command 
as shown below, but please confirm via the appropriate Lustre manual:

lctl set_param osc.--*.active=0

Brett
--
Protect Yourself Against Cybercrime
PDS Software Solutions LLC
https://www.TrustPDS.com

On Mon, Mar 20, 2017 at 10:43 AM, E.S. Rosenberg 
> wrote:
In the man page it says the following:

Because  lfs_migrate  is  not yet closely integrated with the MDS, it cannot 
determine whether a file is currently open and/or in-use by other applications 
or nodes.  That makes it UNSAFE for use on files that might be modified by 
other applications, since the migrated file is only a copy of the current file, 
and this will result in the old file becoming an open-unlinked file and any  
modifications to that file will be lost.

Is this still the case?
Is there a better way to disable OSTs while keeping the filesystem live?

Background:
We need to take a OSS enclosure that hosts multiple OSTs offline for hardware 
maintenance, I'd like to do this without bringing Lustre as a whole down. I 
made sure there is enough space on the other OSTs to house the contents of the 
machine going offline and am now about to move things.

Thanks,
Eli

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] how homogenous should a lustre cluster be?

2017-03-20 Thread White, Cliff 
Comments inlne.

From: lustre-discuss 
>
 on behalf of "E.S. Rosenberg" 
>
Date: Monday, March 20, 2017 at 10:19 AM
To: "lustre-discuss@lists.lustre.org" 
>
Subject: [lustre-discuss] how homogenous should a lustre cluster be?

Dear all,

>How homogenous/not homogenous are your clusters?

>At the moment all our OSS are identical and running lustrefs (lustre 2.8.0), 
>now that I am taking one OSS offline for hardware maintenance I started 
>wondering if I can bring it back as a ZFS OSS or would that make my lustre 
>blow up?

Two things - 1) If you are re-making or replacing an existing OSS you need to 
do it properly, or there will be problems. See the Lustre Manual for the 
process.

2) We test with mixed clusters all the time. No issues at all with making one 
OSS ZFS and other OSS ldiskfs, no issues with multiple hardware types in a 
cluster.
Depending on the mix of hardware, you may wish to implement striping policies 
to avoid performance and space utilization issues. For example, if you stripe 
across multiple OST, and one OST is older, slower hardware the overall IO speed 
will be limited by your slowest OST. OST pools are very useful for this.


>Also in a more general sense do other people have lustre clusters with 
>OSS/OSTs that are different hardware generations etc.?
>
There are many customers with multiple hardware generations in a cluster. Uses 
tend to vary with the site/use. Typically when people add a big pile of new 
OSS, they put the new gear up as a separate filesystem or pool to simplify the 
performance/space situation.


>Thanks,
>Eli

Hope this helps
Cliffw


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] how homogenous should a lustre cluster be?

2017-03-20 Thread E.S. Rosenberg 
Dear all,

How homogenous/not homogenous are your clusters?

At the moment all our OSS are identical and running lustrefs (lustre
2.8.0), now that I am taking one OSS offline for hardware maintenance I
started wondering if I can bring it back as a ZFS OSS or would that make my
lustre blow up?

Also in a more general sense do other people have lustre clusters with
OSS/OSTs that are different hardware generations etc.?

Thanks,
Eli
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lfs_migrate

2017-03-20 Thread Brett Lee 
Hi Eli,

I believe that is still the case with lfs_migrate.  If otherwise, we'll
probably hear soon.

You should be able to disable those OSTs while keeping the file system
active - via a command on the MDS(s) as well as the clients.  My notes have
the command as shown below, but please confirm via the appropriate Lustre
manual:

lctl set_param osc.--*.active=0

Brett
--
Protect Yourself Against Cybercrime
PDS Software Solutions LLC
https://www.TrustPDS.com 

On Mon, Mar 20, 2017 at 10:43 AM, E.S. Rosenberg  wrote:

> In the man page it says the following:
>
> Because  lfs_migrate  is  not yet closely integrated with the MDS, it
> cannot determine whether a file is currently open and/or in-use by other
> applications or nodes.  That makes it UNSAFE for use on files that might be
> modified by other applications, since the migrated file is only a copy of
> the current file, and this will result in the old file becoming an
> open-unlinked file and any  modifications to that file will be lost.
>
> Is this still the case?
> Is there a better way to disable OSTs while keeping the filesystem live?
>
> Background:
> We need to take a OSS enclosure that hosts multiple OSTs offline for
> hardware maintenance, I'd like to do this without bringing Lustre as a
> whole down. I made sure there is enough space on the other OSTs to house
> the contents of the machine going offline and am now about to move things.
>
> Thanks,
> Eli
>
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
>
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] lfs_migrate

2017-03-20 Thread E.S. Rosenberg 
In the man page it says the following:

Because  lfs_migrate  is  not yet closely integrated with the MDS, it
cannot determine whether a file is currently open and/or in-use by other
applications or nodes.  That makes it UNSAFE for use on files that might be
modified by other applications, since the migrated file is only a copy of
the current file, and this will result in the old file becoming an
open-unlinked file and any  modifications to that file will be lost.

Is this still the case?
Is there a better way to disable OSTs while keeping the filesystem live?

Background:
We need to take a OSS enclosure that hosts multiple OSTs offline for
hardware maintenance, I'd like to do this without bringing Lustre as a
whole down. I made sure there is enough space on the other OSTs to house
the contents of the machine going offline and am now about to move things.

Thanks,
Eli
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Backup software for Lustre

2017-03-20 Thread Brett Lee 
Ah, yes.  Thanks for correcting my statement, Andreas.  I'd like to retract
my last email - was probably too happy seeing the striping patterns return
after extracting the tar file. ;)

Andreas, Related to your statement about backing up but not restoring the
lustre.lov for directories, I'm curious what accounts for (in my tests,
anyway) the directories being restored with their non-default striping
patterns in-tact?  I'm not aware of all the Lustre xattrs and what they
provide. Thanks.

Brett
--
Protect Yourself Against Cybercrime
PDS Software Solutions LLC
https://www.TrustPDS.com 

On Mon, Mar 20, 2017 at 1:51 AM, Dilger, Andreas 
wrote:

> On Mar 19, 2017, at 20:57, Brett Lee  wrote:
> >
> > Help from down under! :)  Thanks Malcolm!!
> >
> > Using the correct extract syntax, both directories and files restored
> with the expected stripe size and stripe count.  This means that Zmanda
> should be able to work as a backup solution for some Lustre file systems,
> especially if able to using multiple per-directory "agents" to perform
> backups in parallel.
>
> I tested this on RHEL7, and it seems tar needs a bit of encouragement to
> restore Lustre xattrs:
>
> tar xvf  --xattrs-include="lustre.*"
>
> This will restore the important "lustre.lov" xattr with the file layout
> (Lustre itself discards the other "lustre.*" xattrs silently) for regular
> files via mknod+setxattr.  It still doesn't restore the default lustre.lov
> xattr onto the directory, even though it backed those xattrs up.
>
> Unfortunately, I didn't see the tar process even trying to access any
> config file like /etc/xattr.conf or /etc/tar.conf whatever, so I don't
> think there is any way to make this the default without recompiling tar.
> Sadly, I thought we'd gotten away from this once RHEL6 had patched their
> tar and GNU tar had added xattr support natively in 1.27, but it seems that
> we'll need to patch it again.  One option would be to add "lustre.*" to the
> xattrs-include list if extracting to a Lustre filesystem (e.g. checked via
> statfs() at the start).
>
> Cheers, Andreas
>
> > Go Lustre!
> >
> > Folks, my bad for only checking one source - the 3rd hit via Google for
> "backup restore xattrs tar":
> > https://www.cyberciti.biz/faq/linux-tar-rsync-preserving-acl
> s-selinux-contexts/
> > I wasn't aware that "tar" was so dynamic in this respect.  Alas, this
> again justifyies the adage "trust, but verify" - and especially with
> backups.
> >
> > Brett
> > --
> > Protect Yourself Against Cybercrime
> > PDS Software Solutions LLC
> > https://www.TrustPDS.com
> >
> > On Sun, Mar 19, 2017 at 3:34 PM, Cowe, Malcolm J <
> malcolm.j.c...@intel.com> wrote:
> > The version of tar included in RHEL 7 doesn’t restore the lustre xattrs
> by default – you can use the following to extract files with the requisite
> xattrs:
> >
> >
> >
> > tar --xattrs-include=lustre.* -xf .tar
> >
> >
> >
> > This assumes the files were backed up with the --xattrs flag:
> >
> >
> >
> > tar --xattrs -cf .tar 
> >
> >
> >
> > Note, that you don’t appear to need to whitelist the Lustre xattrs when
> backing up, only when restoring.
> >
> >
> >
> > Malcolm.
> >
> >
> >
> >
> >
> > From: lustre-discuss  on
> behalf of "Dilger, Andreas" 
> > Date: Monday, 20 March 2017 at 8:11 am
> > To: Brett Lee 
> > Cc: Lustre discussion 
> >
> >
> > Subject: Re: [lustre-discuss] Backup software for Lustre
> >
> >
> >
> > The use of openat() could be problematic since this precludes storing
> the xattr before the file is opened. That said, I don't see anywhere in
> your strace log that (f)setxattr() is called to restore the xattrs, for
> either the regular files or directories, even after the file is opened or
> written?
> >
> >
> >
> > Does the RHEL tar have a whitelist of xattrs to be restored?  The fact
> that there are Lustre xattrs after the restore appears to just be normal
> behavior for creating a file, not anything related to tar restoring xattrs.
> >
> > Cheers, Andreas
> >
> >
> >
> > On Mar 19, 2017, at 10:45, Brett Lee  wrote:
> >
> > Sure, happy to help.  I did not see mknod+setxattr in the strace
> output.  Included is a trimmed version of the strace output, along with a
> few more bits of information.  Thanks!
> >
> > # cat /proc/fs/lustre/version
> > lustre: 2.7.19.8
> > # cat /etc/redhat-release
> > CentOS Linux release 7.3.1611 (Core)
> > # uname -r
> > 3.10.0-514.2.2.el7_lustre.x86_64
> > # rpm -qa|grep tar
> > tar-1.26-31.el7.x86_64
> > # sha1sum `which tar` `which gtar`
> > ea17ec98894212b2e2285eb2dd99aad76185ea7d  /usr/bin/tar
> > ea17ec98894212b2e2285eb2dd99aad76185ea7d  /usr/bin/gtar
> >
> > Striping was set on the four directories before creating the files.
> > mkdir -p /scratch/1; lfs setstripe -c 1 --stripe-size

[lustre-discuss] Lustre 2.8.0 and 2.9.0 failed to build for kernel 3.10.0-514.el7.x86_64

2017-03-20 Thread Thome, Pedro (R Center - Brazil) 
Hi

I tried to install lustre 2.8.0 and 2.9.0 on a Centos 7.3 system and got this 
error:

Error! Bad return status for module build on kernel: 3.10.0-514.el7.x86_64 
(x86_64)
Consult /var/lib/dkms/lustre-client/2.9.0/build/make.log for more information.

I attached the installation output and the make.log as well.

I switched from 2.8.0 to 2.9.0 since 2.9.0 is the one that support this kernel.

Thanks,
Pedro Thomé
Installing Lustre Client
Loaded plugins: fastestmirror, langpacks
No plugin match for: refresh-packagekit
Loading mirror speeds from cached hostfile
Resolving Dependencies
--> Running transaction check
---> Package lustre-client-dkms.noarch 0:2.9.0-1.el7 will be installed
--> Processing Dependency: /usr/bin/expect for package: 
lustre-client-dkms-2.9.0-1.el7.noarch
--> Running transaction check
---> Package expect.x86_64 0:5.45-14.el7_1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved


 Package  Arch Version   RepositorySize

Installing:
 lustre-client-dkms   noarch   2.9.0-1.el7   CST   12 M
Installing for dependencies:
 expect   x86_64   5.45-14.el7_1 OS_DVD   262 k

Transaction Summary

Install  1 Package (+1 Dependent package)

Total download size: 12 M
Installed size: 34 M
Downloading packages:

Total  102 MB/s |  12 MB  00:00 
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : expect-5.45-14.el7_1.x86_64  1/2 
  Installing : lustre-client-dkms-2.9.0-1.el7.noarch2/2 
Loading new lustre-client-2.9.0 DKMS files...
Building for 3.10.0-514.el7.x86_64
Building initial module for 3.10.0-514.el7.x86_64
Error! Bad return status for module build on kernel: 3.10.0-514.el7.x86_64 
(x86_64)
Consult /var/lib/dkms/lustre-client/2.9.0/build/make.log for more information.
warning: %post(lustre-client-dkms-2.9.0-1.el7.noarch) scriptlet failed, exit 
status 10
Non-fatal POSTIN scriptlet failure in rpm package 
lustre-client-dkms-2.9.0-1.el7.noarch
  Verifying  : lustre-client-dkms-2.9.0-1.el7.noarch1/2 
  Verifying  : expect-5.45-14.el7_1.x86_64  2/2 

Installed:
  lustre-client-dkms.noarch 0:2.9.0-1.el7   

Dependency Installed:
  expect.x86_64 0:5.45-14.el7_1 

Complete!

Rebuilding Lustre kernel modules using MOFED's ofa_kernel...

Removing old lustre-client-2.9.0 DKMS files...

--
Deleting module version: 2.9.0
completely from the DKMS tree.
--
Done.
Loading new lustre-client-2.9.0 DKMS files...
Building for 3.10.0-514.el7.x86_64
Building initial module for 3.10.0-514.el7.x86_64
Error! Bad return status for module build on kernel: 3.10.0-514.el7.x86_64 
(x86_64)
Consult /var/lib/dkms/lustre-client/2.9.0/build/make.log for more information.

make.log
Description: make.log
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Backup software for Lustre

2017-03-20 Thread Dilger, Andreas 
On Mar 19, 2017, at 20:57, Brett Lee  wrote:
> 
> Help from down under! :)  Thanks Malcolm!!
> 
> Using the correct extract syntax, both directories and files restored with 
> the expected stripe size and stripe count.  This means that Zmanda should be 
> able to work as a backup solution for some Lustre file systems, especially if 
> able to using multiple per-directory "agents" to perform backups in parallel.

I tested this on RHEL7, and it seems tar needs a bit of encouragement to 
restore Lustre xattrs:

tar xvf  --xattrs-include="lustre.*"

This will restore the important "lustre.lov" xattr with the file layout (Lustre 
itself discards the other "lustre.*" xattrs silently) for regular files via 
mknod+setxattr.  It still doesn't restore the default lustre.lov xattr onto the 
directory, even though it backed those xattrs up.

Unfortunately, I didn't see the tar process even trying to access any config 
file like /etc/xattr.conf or /etc/tar.conf whatever, so I don't think there is 
any way to make this the default without recompiling tar.  Sadly, I thought 
we'd gotten away from this once RHEL6 had patched their tar and GNU tar had 
added xattr support natively in 1.27, but it seems that we'll need to patch it 
again.  One option would be to add "lustre.*" to the xattrs-include list if 
extracting to a Lustre filesystem (e.g. checked via statfs() at the start).

Cheers, Andreas

> Go Lustre!
> 
> Folks, my bad for only checking one source - the 3rd hit via Google for 
> "backup restore xattrs tar":
> https://www.cyberciti.biz/faq/linux-tar-rsync-preserving-acls-selinux-contexts/
> I wasn't aware that "tar" was so dynamic in this respect.  Alas, this again 
> justifyies the adage "trust, but verify" - and especially with backups.
> 
> Brett
> --
> Protect Yourself Against Cybercrime
> PDS Software Solutions LLC
> https://www.TrustPDS.com
> 
> On Sun, Mar 19, 2017 at 3:34 PM, Cowe, Malcolm J  
> wrote:
> The version of tar included in RHEL 7 doesn’t restore the lustre xattrs by 
> default – you can use the following to extract files with the requisite 
> xattrs:
> 
>  
> 
> tar --xattrs-include=lustre.* -xf .tar
> 
>  
> 
> This assumes the files were backed up with the --xattrs flag:
> 
>  
> 
> tar --xattrs -cf .tar 
> 
>  
> 
> Note, that you don’t appear to need to whitelist the Lustre xattrs when 
> backing up, only when restoring.
> 
>  
> 
> Malcolm.
> 
>  
> 
>  
> 
> From: lustre-discuss  on behalf of 
> "Dilger, Andreas" 
> Date: Monday, 20 March 2017 at 8:11 am
> To: Brett Lee 
> Cc: Lustre discussion 
> 
> 
> Subject: Re: [lustre-discuss] Backup software for Lustre
> 
>  
> 
> The use of openat() could be problematic since this precludes storing the 
> xattr before the file is opened. That said, I don't see anywhere in your 
> strace log that (f)setxattr() is called to restore the xattrs, for either the 
> regular files or directories, even after the file is opened or written?  
> 
>  
> 
> Does the RHEL tar have a whitelist of xattrs to be restored?  The fact that 
> there are Lustre xattrs after the restore appears to just be normal behavior 
> for creating a file, not anything related to tar restoring xattrs. 
> 
> Cheers, Andreas
> 
>  
> 
> On Mar 19, 2017, at 10:45, Brett Lee  wrote:
> 
> Sure, happy to help.  I did not see mknod+setxattr in the strace output.  
> Included is a trimmed version of the strace output, along with a few more 
> bits of information.  Thanks!
> 
> # cat /proc/fs/lustre/version 
> lustre: 2.7.19.8
> # cat /etc/redhat-release 
> CentOS Linux release 7.3.1611 (Core) 
> # uname -r
> 3.10.0-514.2.2.el7_lustre.x86_64
> # rpm -qa|grep tar
> tar-1.26-31.el7.x86_64
> # sha1sum `which tar` `which gtar`
> ea17ec98894212b2e2285eb2dd99aad76185ea7d  /usr/bin/tar
> ea17ec98894212b2e2285eb2dd99aad76185ea7d  /usr/bin/gtar
> 
> Striping was set on the four directories before creating the files.
> mkdir -p /scratch/1; lfs setstripe -c 1 --stripe-size 128K /scratch/1; lfs 
> getstripe /scratch/1
> mkdir -p /scratch/2; lfs setstripe -c 2 --stripe-size 256K /scratch/2; lfs 
> getstripe /scratch/2
> mkdir -p /scratch/3; lfs setstripe -c 3 --stripe-size 768K /scratch/3; lfs 
> getstripe /scratch/3
> mkdir -p /scratch/4; lfs setstripe -c 4 --stripe-size 1M/scratch/4; lfs 
> getstripe /scratch/4
> After tar, all files and directories all had the default Lustre striping.
> 
> # tar ztvf /scratch.tgz 
> drwxr-xr-x root/root 0 2017-03-19 10:54 scratch/
> drwxr-xr-x root/root 0 2017-03-19 10:57 scratch/4/
> -rw-r--r-- root/root   4194304 2017-03-19 10:57 scratch/4/4.dd
> drwxr-xr-x root/root 0 2017-03-19 10:57 scratch/3/
> -rw-r--r-- root/root   4194304 2017-03-19 10:57 scratch/3/3.dd
> drwxr-xr-x root/root 0 2017-03-19 10:57 scratch/1/
>