Re: [lustre-discuss] lfs_migrate issue

[Jason Williams]

Hello Ahmed,


I'm rather new to the lfs_migrate command as well, but one thing to double 
check is after you run the migrate, have you done an


lfs getstripe


for a couple of the files it thinks it migrated to make sure they moved off of 
the OST?  Also, did you properly disable the OST in the MDS to make sure new 
files were not written to it?  Here is the document I was following: 
http://wiki.lustre.org/Handling_Full_OSTs



--
Jason Williams
Assistant Director
Systems and Data Center Operations.
Maryland Advanced Research Computing Center (MARCC)
Johns Hopkins University
jas...@jhu.edu



From: lustre-discuss  on behalf of 
Ahmed Fahmy 
Sent: Tuesday, January 15, 2019 6:33:24 AM
To: lustre-discuss@lists.lustre.org
Cc: Supercomputer
Subject: [lustre-discuss] lfs_migrate issue


Good day,

I have been trying to migrate the data one of the OSTs in my lustre file system 
before removing the OST.

I have used the following command:

lfs find --obd lustre-OST0001_UUID /lustre | lfs_migrate -sy

I believe the data has been copied correctly.


However, when I check the size of the directories that has been participating 
in the migration process, I notice that the size has increased with a 3 or 4 GB 
increase, even after removing the OST.

I am not sure what is the reason for this issue and how I can return those 
directories to their original sizes.

I am using lustre 2.10.1.

Any help would be appreciated.

Regards,

Ahmed Fahmy
Bibliotheca Alexandrina

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lfs_migrate rsync vs. lfs migrate and layout swap

[Dilger, Andreas]

You should be able to push using SSH, which I'd imagine would not be blocked?  
It is possible to also fetch patches via http and git protocol, but I don't 
think we allow unauthenticated pushes. 

Cheers, Andreas

> On Nov 25, 2017, at 15:01, Daniel Kobras  wrote:
> 
> Hi!
> 
> 
>> Am 20.11.2017 um 00:01 schrieb Dilger, Andreas :
>> 
>> It would be interesting to strace your rsync vs. "lfs migrate" read/write 
>> patterns so that the copy method of "lfs migrate" can be improved to match 
>> rsync. Since they are both userspace copy actions they should be about the 
>> same performance. It may be that "lfs migrate" is using O_DIRECT to minimize 
>> client cache pollution (I don't have the code handy to check right now).  In 
>> the future we could use "copyfile()" to avoid this as well. 
> 
> lfs migrate indeed uses O_DIRECT for reading the source file. A few tests on 
> a system running 2.10.1 yielded a 10x higher throughput with a modified lfs 
> migrate that simply dropped the O_DIRECT flag. I’ve filed 
> https://jira.hpdd.intel.com/browse/LU-10278 about it. (A simple patch to make 
> O_DIRECT optional is ready, but I still need to charm the gods of the 
> firewall to let me push to Gerrit.)
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lfs_migrate rsync vs. lfs migrate and layout swap

[Daniel Kobras]

Hi!


> Am 20.11.2017 um 00:01 schrieb Dilger, Andreas :
> 
> It would be interesting to strace your rsync vs. "lfs migrate" read/write 
> patterns so that the copy method of "lfs migrate" can be improved to match 
> rsync. Since they are both userspace copy actions they should be about the 
> same performance. It may be that "lfs migrate" is using O_DIRECT to minimize 
> client cache pollution (I don't have the code handy to check right now).  In 
> the future we could use "copyfile()" to avoid this as well. 

lfs migrate indeed uses O_DIRECT for reading the source file. A few tests on a 
system running 2.10.1 yielded a 10x higher throughput with a modified lfs 
migrate that simply dropped the O_DIRECT flag. I’ve filed 
https://jira.hpdd.intel.com/browse/LU-10278 about it. (A simple patch to make 
O_DIRECT optional is ready, but I still need to charm the gods of the firewall 
to let me push to Gerrit.)

Kind regards,

Daniel
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lfs_migrate rsync vs. lfs migrate and layout swap

[Dilger, Andreas]

On Nov 24, 2017, at 06:55, Dauchy, Nathan (ARC-TNC)[CSRA, LLC] 
<nathan.dau...@nasa.gov> wrote:
> 
> For those following along and interested in another difference between the 
> two migrate methods...
> 
> lfs migrate layout swap is (apparently) able to handle files with multiple 
> links, whereas the rsync method bails out.

There is a patch to allow rsync to handle hard-linked files, but I'm not sure 
if it has landed yet.  The layout swap method is of course preferred.


> 
> From: Dilger, Andreas [andreas.dil...@intel.com]
> Sent: Sunday, November 19, 2017 4:01 PM
> To: Dauchy, Nathan (ARC-TNC)[CSRA, LLC]
> Cc: lustre-discuss@lists.lustre.org
> Subject: Re: [lustre-discuss] lfs_migrate rsync vs. lfs migrate and layout 
> swap
> 
> It would be interesting to strace your rsync vs. "lfs migrate" read/write 
> patterns so that the copy method of "lfs migrate" can be improved to match 
> rsync. Since they are both userspace copy actions they should be about the 
> same performance. It may be that "lfs migrate" is using O_DIRECT to minimize 
> client cache pollution (I don't have the code handy to check right now).  In 
> the future we could use "copyfile()" to avoid this as well.
> 
> The main benefit of migrate is that it keeps the open file handles and inode 
> number on the MDS. Using rsync is just a copy+rename, which is why it is not 
> safe for in-use files.
> 
> There is no need to clean up volatile files, they are essentially 
> open-unlinked files, so they clean up automatically if the program or client 
> crash.
> 
> Cheers, Andreas
> 
>> On Nov 19, 2017, at 11:31, Dauchy, Nathan (ARC-TNC)[CSRA, LLC] 
>> <nathan.dau...@nasa.gov> wrote:
>> 
>> Greetings,
>> 
>> I'm trying to clarify and confirm the differences between lfs_migrate's use 
>> of rsync vs. "lfs migrate".  This is in regards to performance, 
>> checksumming, and interrupts.  Relevant code changes that introduced the two 
>> methods are here:
>> https://jira.hpdd.intel.com/browse/LU-2445
>> https://review.whamcloud.com/#/c/5620/
>> 
>> The quick testing I have done is with a 8GB file with stripe count of 4, and 
>> included the patch to lfs_migrate from:
>> https://review.whamcloud.com/#/c/20621/
>> (and client cache was dropped between each test)
>> 
>> $ time ./lfs_migrate -y bigfile
>> real1m13.643s
>> 
>> $ time ./lfs_migrate -y -s bigfile
>> real1m13.194s
>> 
>> $ time ./lfs_migrate -y -f bigfile
>> real0m31.791s
>> 
>> $ time ./lfs_migrate -y -f -s bigfile
>> real0m28.020s
>> 
>> * Performance:  The migrate runs faster when forcing rsync (assuming 
>> multiple stripes).  There is also minimal performance benefit to skipping 
>> the checksum with the rsync method.  Interestingly, performance with "lfs 
>> migrate" as the backend is barely effected (and within the noise when I ran 
>> multiple tests) by the choice of checksumming or not.  So, my question is 
>> whether there is some serialization going on with the layout swap method 
>> which causes it to be slower?
>> 
>> * Checksums:  In reading the migrate code in lfs.c, it is not obvious to me 
>> that there is any checksumming done at all for "lfs migrate".  That would 
>> explain why there is minimal performance difference.  How is data integrity 
>> ensured with this method?  Does the file data version somehow capture the 
>> checksum too?
>> 
>> * Interrupts:  If the rsync method is interrupted (kill -9, or client 
>> reboot) then a ".tmp.XX" file is left.  This is reasonably easy to 
>> search for and clean up.  With the lfs migrate layout swap method, what 
>> happens to the "volatile file" and it's objects?  Is an lfsck required in 
>> order to clean up the objects?
>> 
>> At this point, the "old" method seems preferable.  Are there other benefits 
>> to using the lfs migrate layout swap method that I'm missing?
>> 
>> Thanks for any clarifications or other suggestions!
>> 
>> -Nathan
>> ___
>> lustre-discuss mailing list
>> lustre-discuss@lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Intel Corporation







___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lfs_migrate rsync vs. lfs migrate and layout swap

[Dauchy, Nathan (ARC-TNC)[CSRA, LLC]]

For those following along and interested in another difference between the two 
migrate methods...

lfs migrate layout swap is (apparently) able to handle files with multiple 
links, whereas the rsync method bails out.

-Nathan


From: Dilger, Andreas [andreas.dil...@intel.com]
Sent: Sunday, November 19, 2017 4:01 PM
To: Dauchy, Nathan (ARC-TNC)[CSRA, LLC]
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] lfs_migrate rsync vs. lfs migrate and layout swap

It would be interesting to strace your rsync vs. "lfs migrate" read/write 
patterns so that the copy method of "lfs migrate" can be improved to match 
rsync. Since they are both userspace copy actions they should be about the same 
performance. It may be that "lfs migrate" is using O_DIRECT to minimize client 
cache pollution (I don't have the code handy to check right now).  In the 
future we could use "copyfile()" to avoid this as well.

The main benefit of migrate is that it keeps the open file handles and inode 
number on the MDS. Using rsync is just a copy+rename, which is why it is not 
safe for in-use files.

There is no need to clean up volatile files, they are essentially open-unlinked 
files, so they clean up automatically if the program or client crash.

Cheers, Andreas

> On Nov 19, 2017, at 11:31, Dauchy, Nathan (ARC-TNC)[CSRA, LLC] 
> <nathan.dau...@nasa.gov> wrote:
>
> Greetings,
>
> I'm trying to clarify and confirm the differences between lfs_migrate's use 
> of rsync vs. "lfs migrate".  This is in regards to performance, checksumming, 
> and interrupts.  Relevant code changes that introduced the two methods are 
> here:
> https://jira.hpdd.intel.com/browse/LU-2445
> https://review.whamcloud.com/#/c/5620/
>
> The quick testing I have done is with a 8GB file with stripe count of 4, and 
> included the patch to lfs_migrate from:
> https://review.whamcloud.com/#/c/20621/
> (and client cache was dropped between each test)
>
> $ time ./lfs_migrate -y bigfile
> real1m13.643s
>
> $ time ./lfs_migrate -y -s bigfile
> real1m13.194s
>
> $ time ./lfs_migrate -y -f bigfile
> real0m31.791s
>
> $ time ./lfs_migrate -y -f -s bigfile
> real0m28.020s
>
> * Performance:  The migrate runs faster when forcing rsync (assuming multiple 
> stripes).  There is also minimal performance benefit to skipping the checksum 
> with the rsync method.  Interestingly, performance with "lfs migrate" as the 
> backend is barely effected (and within the noise when I ran multiple tests) 
> by the choice of checksumming or not.  So, my question is whether there is 
> some serialization going on with the layout swap method which causes it to be 
> slower?
>
> * Checksums:  In reading the migrate code in lfs.c, it is not obvious to me 
> that there is any checksumming done at all for "lfs migrate".  That would 
> explain why there is minimal performance difference.  How is data integrity 
> ensured with this method?  Does the file data version somehow capture the 
> checksum too?
>
> * Interrupts:  If the rsync method is interrupted (kill -9, or client reboot) 
> then a ".tmp.XX" file is left.  This is reasonably easy to search for and 
> clean up.  With the lfs migrate layout swap method, what happens to the 
> "volatile file" and it's objects?  Is an lfsck required in order to clean up 
> the objects?
>
> At this point, the "old" method seems preferable.  Are there other benefits 
> to using the lfs migrate layout swap method that I'm missing?
>
> Thanks for any clarifications or other suggestions!
>
> -Nathan
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lfs_migrate rsync vs. lfs migrate and layout swap

[Dauchy, Nathan (ARC-TNC)[CSRA, LLC]]

Brian, Andreas,

Thanks for the info.  The benefit of keeping open filehandles and inode number 
is good to understand, but for my immediate case the OSTs have been deactivated 
and then set max_create_count=0 for several weeks, so I'm not too concerned 
about any remaining open files. 

If I get anything interesting back from strace, I'll report here.

What about the checksum issue?  It still looks to me like that is only done 
with the rsync method.

Thanks,
Nathan


From: Dilger, Andreas [andreas.dil...@intel.com]
Sent: Sunday, November 19, 2017 4:01 PM
To: Dauchy, Nathan (ARC-TNC)[CSRA, LLC]
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] lfs_migrate rsync vs. lfs migrate and layout swap

It would be interesting to strace your rsync vs. "lfs migrate" read/write 
patterns so that the copy method of "lfs migrate" can be improved to match 
rsync. Since they are both userspace copy actions they should be about the same 
performance. It may be that "lfs migrate" is using O_DIRECT to minimize client 
cache pollution (I don't have the code handy to check right now).  In the 
future we could use "copyfile()" to avoid this as well.

The main benefit of migrate is that it keeps the open file handles and inode 
number on the MDS. Using rsync is just a copy+rename, which is why it is not 
safe for in-use files.

There is no need to clean up volatile files, they are essentially open-unlinked 
files, so they clean up automatically if the program or client crash.

Cheers, Andreas

> On Nov 19, 2017, at 11:31, Dauchy, Nathan (ARC-TNC)[CSRA, LLC] 
> <nathan.dau...@nasa.gov> wrote:
>
> Greetings,
>
> I'm trying to clarify and confirm the differences between lfs_migrate's use 
> of rsync vs. "lfs migrate".  This is in regards to performance, checksumming, 
> and interrupts.  Relevant code changes that introduced the two methods are 
> here:
> https://jira.hpdd.intel.com/browse/LU-2445
> https://review.whamcloud.com/#/c/5620/
>
> The quick testing I have done is with a 8GB file with stripe count of 4, and 
> included the patch to lfs_migrate from:
> https://review.whamcloud.com/#/c/20621/
> (and client cache was dropped between each test)
>
> $ time ./lfs_migrate -y bigfile
> real1m13.643s
>
> $ time ./lfs_migrate -y -s bigfile
> real1m13.194s
>
> $ time ./lfs_migrate -y -f bigfile
> real0m31.791s
>
> $ time ./lfs_migrate -y -f -s bigfile
> real0m28.020s
>
> * Performance:  The migrate runs faster when forcing rsync (assuming multiple 
> stripes).  There is also minimal performance benefit to skipping the checksum 
> with the rsync method.  Interestingly, performance with "lfs migrate" as the 
> backend is barely effected (and within the noise when I ran multiple tests) 
> by the choice of checksumming or not.  So, my question is whether there is 
> some serialization going on with the layout swap method which causes it to be 
> slower?
>
> * Checksums:  In reading the migrate code in lfs.c, it is not obvious to me 
> that there is any checksumming done at all for "lfs migrate".  That would 
> explain why there is minimal performance difference.  How is data integrity 
> ensured with this method?  Does the file data version somehow capture the 
> checksum too?
>
> * Interrupts:  If the rsync method is interrupted (kill -9, or client reboot) 
> then a ".tmp.XX" file is left.  This is reasonably easy to search for and 
> clean up.  With the lfs migrate layout swap method, what happens to the 
> "volatile file" and it's objects?  Is an lfsck required in order to clean up 
> the objects?
>
> At this point, the "old" method seems preferable.  Are there other benefits 
> to using the lfs migrate layout swap method that I'm missing?
>
> Thanks for any clarifications or other suggestions!
>
> -Nathan
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lfs_migrate rsync vs. lfs migrate and layout swap

[Dilger, Andreas]

It would be interesting to strace your rsync vs. "lfs migrate" read/write 
patterns so that the copy method of "lfs migrate" can be improved to match 
rsync. Since they are both userspace copy actions they should be about the same 
performance. It may be that "lfs migrate" is using O_DIRECT to minimize client 
cache pollution (I don't have the code handy to check right now).  In the 
future we could use "copyfile()" to avoid this as well. 

The main benefit of migrate is that it keeps the open file handles and inode 
number on the MDS. Using rsync is just a copy+rename, which is why it is not 
safe for in-use files. 

There is no need to clean up volatile files, they are essentially open-unlinked 
files, so they clean up automatically if the program or client crash. 

Cheers, Andreas

> On Nov 19, 2017, at 11:31, Dauchy, Nathan (ARC-TNC)[CSRA, LLC] 
>  wrote:
> 
> Greetings,
> 
> I'm trying to clarify and confirm the differences between lfs_migrate's use 
> of rsync vs. "lfs migrate".  This is in regards to performance, checksumming, 
> and interrupts.  Relevant code changes that introduced the two methods are 
> here:
> https://jira.hpdd.intel.com/browse/LU-2445
> https://review.whamcloud.com/#/c/5620/
> 
> The quick testing I have done is with a 8GB file with stripe count of 4, and 
> included the patch to lfs_migrate from:
> https://review.whamcloud.com/#/c/20621/
> (and client cache was dropped between each test)
> 
> $ time ./lfs_migrate -y bigfile
> real1m13.643s
> 
> $ time ./lfs_migrate -y -s bigfile
> real1m13.194s
> 
> $ time ./lfs_migrate -y -f bigfile
> real0m31.791s
> 
> $ time ./lfs_migrate -y -f -s bigfile
> real0m28.020s
> 
> * Performance:  The migrate runs faster when forcing rsync (assuming multiple 
> stripes).  There is also minimal performance benefit to skipping the checksum 
> with the rsync method.  Interestingly, performance with "lfs migrate" as the 
> backend is barely effected (and within the noise when I ran multiple tests) 
> by the choice of checksumming or not.  So, my question is whether there is 
> some serialization going on with the layout swap method which causes it to be 
> slower?
> 
> * Checksums:  In reading the migrate code in lfs.c, it is not obvious to me 
> that there is any checksumming done at all for "lfs migrate".  That would 
> explain why there is minimal performance difference.  How is data integrity 
> ensured with this method?  Does the file data version somehow capture the 
> checksum too?
> 
> * Interrupts:  If the rsync method is interrupted (kill -9, or client reboot) 
> then a ".tmp.XX" file is left.  This is reasonably easy to search for and 
> clean up.  With the lfs migrate layout swap method, what happens to the 
> "volatile file" and it's objects?  Is an lfsck required in order to clean up 
> the objects?
> 
> At this point, the "old" method seems preferable.  Are there other benefits 
> to using the lfs migrate layout swap method that I'm missing?
> 
> Thanks for any clarifications or other suggestions!
> 
> -Nathan
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lfs_migrate rsync vs. lfs migrate and layout swap

[Brian Andrus]

I may be off, but I think the big gain with lfs_migrate is with live files.
Using rsync, you may end up with a file that is not the same if changes 
occurred during the sync, whereas lfs_migrate is taking those changes 
into account so there is no deviation.

Such "belt and suspenders" will incur overhead, but be safer.

Brian Andrus


On 11/19/2017 10:31 AM, Dauchy, Nathan (ARC-TNC)[CSRA, LLC] wrote:

Greetings,

I'm trying to clarify and confirm the differences between lfs_migrate's use of rsync vs. 
"lfs migrate".  This is in regards to performance, checksumming, and 
interrupts.  Relevant code changes that introduced the two methods are here:
https://jira.hpdd.intel.com/browse/LU-2445
https://review.whamcloud.com/#/c/5620/

The quick testing I have done is with a 8GB file with stripe count of 4, and 
included the patch to lfs_migrate from:
https://review.whamcloud.com/#/c/20621/
(and client cache was dropped between each test)

$ time ./lfs_migrate -y bigfile
real1m13.643s

$ time ./lfs_migrate -y -s bigfile
real1m13.194s

$ time ./lfs_migrate -y -f bigfile
real0m31.791s

$ time ./lfs_migrate -y -f -s bigfile
real0m28.020s

* Performance:  The migrate runs faster when forcing rsync (assuming multiple stripes).  
There is also minimal performance benefit to skipping the checksum with the rsync method. 
 Interestingly, performance with "lfs migrate" as the backend is barely 
effected (and within the noise when I ran multiple tests) by the choice of checksumming 
or not.  So, my question is whether there is some serialization going on with the layout 
swap method which causes it to be slower?

* Checksums:  In reading the migrate code in lfs.c, it is not obvious to me that there is 
any checksumming done at all for "lfs migrate".  That would explain why there 
is minimal performance difference.  How is data integrity ensured with this method?  Does 
the file data version somehow capture the checksum too?

* Interrupts:  If the rsync method is interrupted (kill -9, or client reboot) then a 
".tmp.XX" file is left.  This is reasonably easy to search for and clean up.  With 
the lfs migrate layout swap method, what happens to the "volatile file" and it's objects? 
 Is an lfsck required in order to clean up the objects?

At this point, the "old" method seems preferable.  Are there other benefits to 
using the lfs migrate layout swap method that I'm missing?

Thanks for any clarifications or other suggestions!

-Nathan
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lfs_migrate

[Dilger, Andreas]

On May 7, 2017, at 14:27, E.S. Rosenberg <esr+lus...@mail.hebrew.edu> wrote:
> 
> Since we were pressured for time (the migrate was to empty a disk enclosure 
> that was functioning too close to failure for comfort) I mailed the affected 
> user a list of affected files and how they could take care of things.  If I 
> do have time in the future I may write this script.

It looks like there is already a patch that is handling the hard links in 
lfs_migrate
(https://review.whamcloud.com/25851 that I even worked on, but apparently 
forgot about).

This needs to be updated and get some test cases in order to land, but at least 
provides a good starting point for this work.

> Now that we again have a working enclosure, should I be taking action myself 
> to re-balance the files (with migrate) or should I just let time & lustre do 
> its' thing?

If you aren't close to running out of space on the older OSTs, and your data 
has a "limited lifetime" (e.g. created and removed over time) then there is no 
requirement to manually rebalance the storage, it will prefer allocating to the 
empty OST(s) and will even out over time.

The main reason you might want to manually rebalance the OSTs is because the 
MDS _does_ prefer the empty OSTs for allocations, and if you have large amounts 
of parallel IO it can cause increased load on that OST and potentially slow 
down the application(s) as it is doing more work than other OSTs.

Cheers, Andreas

> On Tue, May 2, 2017 at 3:51 AM, Dilger, Andreas <andreas.dil...@intel.com> 
> wrote:
>> If your filesystem was created with Lustre 2.1 or later then you can use:
>> 
>>FID=$(lfs path2fid "/path/to/file")
>>lfs fid2path "$FID"
>> 
>> to find all the pathnames that are hard links to that file. There is a patch 
>> to add a "lfs path2links" option that does this in a single step, but it is 
>> not in any release yet. 
>> 
>> The number of pathnames should match the hard link count returned by "stat 
>> -c%h" if the files don't have too many hard links (i.e. below 140 or so) and 
>> then you can manually migrate the file and re-link the other pathnames to 
>> the new file with "ln -f".
>> 
>> That is something that has been on the todo list for lfs_migrate for a 
>> while, so if you wanted to implement that in the script and submit a patch 
>> to Gerrit it would be appreciated. 
>> 
>> Cheers, Andreas
>> 
>> On May 1, 2017, at 06:59, E.S. Rosenberg <esr+lus...@mail.hebrew.edu> wrote:
>> 
>>> Now that we have reached close to the end of the migration process we have 
>>> a lot of files that are being skipped due to "multiple hard links", I am 
>>> not sure what my strategy should be concerning such files.  Is there any 
>>> migration automation possible on these? Or is my only route contacting the 
>>> owners (who may just not have known how to use 'ln -s')?
>>> 
>>> Any advice would be very welcome.
>>> Thanks,
>>> Eliyahu - אליהו
>>> 
>>> On Wed, Apr 12, 2017 at 6:55 PM, Todd, Allen <allen.t...@sig.com> wrote:
>>> Thanks Andreas -- good to know there is yet another reason to upgrade.  We 
>>> are on 2.7.0.  I was trying to hold out for progressive file layout to land.
>>> 
>>> Allen
>>> 
>>> -Original Message-
>>> From: lustre-discuss [mailto:lustre-discuss-boun...@lists.lustre.org] On 
>>> Behalf Of Dilger, Andreas
>>> Sent: Wednesday, April 12, 2017 8:19 AM
>>> To: Todd, Allen <allen.t...@msx.bala.susq.com>
>>> Cc: E.S. Rosenberg <e...@cs.huji.ac.il>; lustre-discuss@lists.lustre.org
>>> Subject: Re: [lustre-discuss] lfs_migrate
>>> 
>>> On Apr 10, 2017, at 14:53, Todd, Allen <allen.t...@sig.com> wrote:
>>> >
>>> > While everyone is talking about lfs migrate, I would like to point out 
>>> > that it appears
>>> > to be missing an option to preserve file modification and access times, 
>>> > which makes it
>>> > less useful for behind the scenes data management tasks.
>>> 
>>> This should actually be the default, though there was a bug in older 
>>> versions of Lustre that didn't preserve the timestamps.  That was fixed in 
>>> Lustre 2.8.
>>> 
>>> Cheers, Andreas
>>> 
>>> > Allen
>>> >
>>> > -Original Message-
>>> > From: lustre-discuss [mailto:lustre-discuss-boun...@lists.lustre.org]
>>> > On Behalf Of Henri Doreau
>>> > Sent: Tuesday, April 04, 201

Re: [lustre-discuss] lfs_migrate

[E.S. Rosenberg]

Since we were pressured for time (the migrate was to empty a disk enclosure
that was functioning too close to failure for comfort) I mailed the
affected user a list of affected files and how they could take care of
things.
If I do have time in the future I may write this script.

Now that we again have a working enclosure, should I be taking action
myself to re-balance the files (with migrate) or should I just let time &
lustre do its' thing?

Thanks,
Eli

On Tue, May 2, 2017 at 3:51 AM, Dilger, Andreas <andreas.dil...@intel.com>
wrote:

> If your filesystem was created with Lustre 2.1 or later then you can use:
>
>FID=$(lfs path2fid "/path/to/file")
>lfs fid2path "$FID"
>
> to find all the pathnames that are hard links to that file. There is a
> patch to add a "lfs path2links" option that does this in a single step, but
> it is not in any release yet.
>
> The number of pathnames should match the hard link count returned by "stat
> -c%h" if the files don't have too many hard links (i.e. below 140 or so)
> and then you can manually migrate the file and re-link the other pathnames
> to the new file with "ln -f".
>
> That is something that has been on the todo list for lfs_migrate for a
> while, so if you wanted to implement that in the script and submit a patch
> to Gerrit it would be appreciated.
>
> Cheers, Andreas
>
> On May 1, 2017, at 06:59, E.S. Rosenberg <esr+lus...@mail.hebrew.edu>
> wrote:
>
> Now that we have reached close to the end of the migration process we have
> a lot of files that are being skipped due to "multiple hard links", I am
> not sure what my strategy should be concerning such files.
> Is there any migration automation possible on these? Or is my only route
> contacting the owners (who may just not have known how to use 'ln -s')?
>
> Any advice would be very welcome.
> Thanks,
> Eliyahu - אליהו
>
> On Wed, Apr 12, 2017 at 6:55 PM, Todd, Allen <allen.t...@sig.com> wrote:
>
>> Thanks Andreas -- good to know there is yet another reason to upgrade.
>> We are on 2.7.0.  I was trying to hold out for progressive file layout to
>> land.
>>
>> Allen
>>
>> -Original Message-
>> From: lustre-discuss [mailto:lustre-discuss-boun...@lists.lustre.org] On
>> Behalf Of Dilger, Andreas
>> Sent: Wednesday, April 12, 2017 8:19 AM
>> To: Todd, Allen <allen.t...@msx.bala.susq.com>
>> Cc: E.S. Rosenberg <e...@cs.huji.ac.il>; lustre-discuss@lists.lustre.org
>> Subject: Re: [lustre-discuss] lfs_migrate
>>
>> On Apr 10, 2017, at 14:53, Todd, Allen <allen.t...@sig.com> wrote:
>> >
>> > While everyone is talking about lfs migrate, I would like to point out
>> that it appears to be missing an option to preserve file modification and
>> access times, which makes it less useful for behind the scenes data
>> management tasks.
>>
>> This should actually be the default, though there was a bug in older
>> versions of Lustre that didn't preserve the timestamps.  That was fixed in
>> Lustre 2.8.
>>
>> Cheers, Andreas
>>
>> > Allen
>> >
>> > -Original Message-
>> > From: lustre-discuss [mailto:lustre-discuss-boun...@lists.lustre.org]
>> > On Behalf Of Henri Doreau
>> > Sent: Tuesday, April 04, 2017 3:18 AM
>> > To: E.S. Rosenberg <e...@cs.huji.ac.il>
>> > Cc: lustre-discuss@lists.lustre.org
>> > Subject: Re: [lustre-discuss] lfs_migrate
>> >
>> > Hello,
>> >
>> > the manpage for lfs(1) lists the available options in 2.8:
>> > """
>> > lfs migrate -m  directory
>> > lfs migrate [-c | --stripe-count ]
>> >   [-i | --stripe-index ]
>> >   [-S | --stripe-size ]
>> >   [-p | --pool ]
>> >   [-o | --ost-list ]
>> >   [-b | --block]
>> >   [-n | --non-block] file|directory """
>> >
>> > Although this is certainly terse, I guess that most parameters are
>> intuitive.
>> >
>> > The command will open the file to restripe (blocking concurrent
>> accesses or not, depending on -b/-n), create a special "volatile"
>> (=unlinked) one with the requested striping parameters and copy the source
>> into the destination.
>> >
>> > If the copy succeeds, the two files are atomically swapped and
>> concurrent access protection is released.
>> >
>> > In non-blocking mode, the process will detect if the source file was
>&g

Re: [lustre-discuss] lfs_migrate

[Dilger, Andreas]

If your filesystem was created with Lustre 2.1 or later then you can use:

   FID=$(lfs path2fid "/path/to/file")
   lfs fid2path "$FID"

to find all the pathnames that are hard links to that file. There is a patch to 
add a "lfs path2links" option that does this in a single step, but it is not in 
any release yet.

The number of pathnames should match the hard link count returned by "stat 
-c%h" if the files don't have too many hard links (i.e. below 140 or so) and 
then you can manually migrate the file and re-link the other pathnames to the 
new file with "ln -f".

That is something that has been on the todo list for lfs_migrate for a while, 
so if you wanted to implement that in the script and submit a patch to Gerrit 
it would be appreciated.

Cheers, Andreas

On May 1, 2017, at 06:59, E.S. Rosenberg 
<esr+lus...@mail.hebrew.edu<mailto:esr+lus...@mail.hebrew.edu>> wrote:

Now that we have reached close to the end of the migration process we have a 
lot of files that are being skipped due to "multiple hard links", I am not sure 
what my strategy should be concerning such files.
Is there any migration automation possible on these? Or is my only route 
contacting the owners (who may just not have known how to use 'ln -s')?

Any advice would be very welcome.
Thanks,
Eliyahu - אליהו

On Wed, Apr 12, 2017 at 6:55 PM, Todd, Allen 
<allen.t...@sig.com<mailto:allen.t...@sig.com>> wrote:
Thanks Andreas -- good to know there is yet another reason to upgrade.  We are 
on 2.7.0.  I was trying to hold out for progressive file layout to land.

Allen

-Original Message-
From: lustre-discuss 
[mailto:lustre-discuss-boun...@lists.lustre.org<mailto:lustre-discuss-boun...@lists.lustre.org>]
 On Behalf Of Dilger, Andreas
Sent: Wednesday, April 12, 2017 8:19 AM
To: Todd, Allen 
<allen.t...@msx.bala.susq.com<mailto:allen.t...@msx.bala.susq.com>>
Cc: E.S. Rosenberg <e...@cs.huji.ac.il<mailto:e...@cs.huji.ac.il>>; 
lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org>
Subject: Re: [lustre-discuss] lfs_migrate

On Apr 10, 2017, at 14:53, Todd, Allen 
<allen.t...@sig.com<mailto:allen.t...@sig.com>> wrote:
>
> While everyone is talking about lfs migrate, I would like to point out that 
> it appears to be missing an option to preserve file modification and access 
> times, which makes it less useful for behind the scenes data management tasks.

This should actually be the default, though there was a bug in older versions 
of Lustre that didn't preserve the timestamps.  That was fixed in Lustre 2.8.

Cheers, Andreas

> Allen
>
> -Original Message-
> From: lustre-discuss 
> [mailto:lustre-discuss-boun...@lists.lustre.org<mailto:lustre-discuss-boun...@lists.lustre.org>]
> On Behalf Of Henri Doreau
> Sent: Tuesday, April 04, 2017 3:18 AM
> To: E.S. Rosenberg <e...@cs.huji.ac.il<mailto:e...@cs.huji.ac.il>>
> Cc: lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org>
> Subject: Re: [lustre-discuss] lfs_migrate
>
> Hello,
>
> the manpage for lfs(1) lists the available options in 2.8:
> """
> lfs migrate -m  directory
> lfs migrate [-c | --stripe-count ]
>   [-i | --stripe-index ]
>   [-S | --stripe-size ]
>   [-p | --pool ]
>   [-o | --ost-list ]
>   [-b | --block]
>   [-n | --non-block] file|directory """
>
> Although this is certainly terse, I guess that most parameters are intuitive.
>
> The command will open the file to restripe (blocking concurrent accesses or 
> not, depending on -b/-n), create a special "volatile" (=unlinked) one with 
> the requested striping parameters and copy the source into the destination.
>
> If the copy succeeds, the two files are atomically swapped and concurrent 
> access protection is released.
>
> In non-blocking mode, the process will detect if the source file was already 
> opened or if there's an open during the copy process and abort safely. It is 
> then up to the admin to reschedule the migration later, maybe with -b.
>
> HTH
>
> Henri
>
> On 02/avril - 14:43 E.S. Rosenberg wrote:
>> Thanks for all the great replies!
>>
>> I may be wrong on this but 'lfs migrate' does not seem to be
>> documented in the manpage (my local one is 2.8 so I expect that but
>> even manpages that I find online).
>>
>> Any pointers would be very welcome.
>>
>> On Thu, Mar 23, 2017 at 12:31 PM, Henri Doreau 
>> <henri.dor...@cea.fr<mailto:henri.dor...@cea.fr>> wrote:
>>
>>> On 20/mars - 22:50 E.S. Rosenberg wrote:
>>>> On Mon, Mar 20, 2017 at 10:19 PM, Dilger, Andrea

Re: [lustre-discuss] lfs_migrate

[E.S. Rosenberg]

Now that we have reached close to the end of the migration process we have
a lot of files that are being skipped due to "multiple hard links", I am
not sure what my strategy should be concerning such files.
Is there any migration automation possible on these? Or is my only route
contacting the owners (who may just not have known how to use 'ln -s')?

Any advice would be very welcome.
Thanks,
Eliyahu - אליהו

On Wed, Apr 12, 2017 at 6:55 PM, Todd, Allen <allen.t...@sig.com> wrote:

> Thanks Andreas -- good to know there is yet another reason to upgrade.  We
> are on 2.7.0.  I was trying to hold out for progressive file layout to land.
>
> Allen
>
> -Original Message-
> From: lustre-discuss [mailto:lustre-discuss-boun...@lists.lustre.org] On
> Behalf Of Dilger, Andreas
> Sent: Wednesday, April 12, 2017 8:19 AM
> To: Todd, Allen <allen.t...@msx.bala.susq.com>
> Cc: E.S. Rosenberg <e...@cs.huji.ac.il>; lustre-discuss@lists.lustre.org
> Subject: Re: [lustre-discuss] lfs_migrate
>
> On Apr 10, 2017, at 14:53, Todd, Allen <allen.t...@sig.com> wrote:
> >
> > While everyone is talking about lfs migrate, I would like to point out
> that it appears to be missing an option to preserve file modification and
> access times, which makes it less useful for behind the scenes data
> management tasks.
>
> This should actually be the default, though there was a bug in older
> versions of Lustre that didn't preserve the timestamps.  That was fixed in
> Lustre 2.8.
>
> Cheers, Andreas
>
> > Allen
> >
> > -Original Message-
> > From: lustre-discuss [mailto:lustre-discuss-boun...@lists.lustre.org]
> > On Behalf Of Henri Doreau
> > Sent: Tuesday, April 04, 2017 3:18 AM
> > To: E.S. Rosenberg <e...@cs.huji.ac.il>
> > Cc: lustre-discuss@lists.lustre.org
> > Subject: Re: [lustre-discuss] lfs_migrate
> >
> > Hello,
> >
> > the manpage for lfs(1) lists the available options in 2.8:
> > """
> > lfs migrate -m  directory
> > lfs migrate [-c | --stripe-count ]
> >   [-i | --stripe-index ]
> >   [-S | --stripe-size ]
> >   [-p | --pool ]
> >   [-o | --ost-list ]
> >   [-b | --block]
> >   [-n | --non-block] file|directory """
> >
> > Although this is certainly terse, I guess that most parameters are
> intuitive.
> >
> > The command will open the file to restripe (blocking concurrent accesses
> or not, depending on -b/-n), create a special "volatile" (=unlinked) one
> with the requested striping parameters and copy the source into the
> destination.
> >
> > If the copy succeeds, the two files are atomically swapped and
> concurrent access protection is released.
> >
> > In non-blocking mode, the process will detect if the source file was
> already opened or if there's an open during the copy process and abort
> safely. It is then up to the admin to reschedule the migration later, maybe
> with -b.
> >
> > HTH
> >
> > Henri
> >
> > On 02/avril - 14:43 E.S. Rosenberg wrote:
> >> Thanks for all the great replies!
> >>
> >> I may be wrong on this but 'lfs migrate' does not seem to be
> >> documented in the manpage (my local one is 2.8 so I expect that but
> >> even manpages that I find online).
> >>
> >> Any pointers would be very welcome.
> >>
> >> On Thu, Mar 23, 2017 at 12:31 PM, Henri Doreau <henri.dor...@cea.fr>
> wrote:
> >>
> >>> On 20/mars - 22:50 E.S. Rosenberg wrote:
> >>>> On Mon, Mar 20, 2017 at 10:19 PM, Dilger, Andreas <
> >>> andreas.dil...@intel.com>
> >>>> wrote:
> >>>>
> >>>>> The underlying "lfs migrate" command (not the "lfs_migrate"
> >>>>> script) in newer Lustre versions (2.9) is capable of migrating
> >>>>> files that are in
> >>> use
> >>>>> by using the "--block" option, which prevents other processes from
> >>>>> accessing or modifying the file during migration.
> >>>>>
> >>>>> Unfortunately, "lfs_migrate" doesn't pass that argument on, though
> >>>>> it wouldn't be hard to change the script. Ideally, the "lfs_migrate"
> >>> script
> >>>>> would pass all unknown options to "lfs migrate".
> >>>>>
> >>>>>
> >>>>> The other item of note is that setting the OST inactive 

Re: [lustre-discuss] lfs_migrate

[Todd, Allen]

Thanks Andreas -- good to know there is yet another reason to upgrade.  We are 
on 2.7.0.  I was trying to hold out for progressive file layout to land.

Allen

-Original Message-
From: lustre-discuss [mailto:lustre-discuss-boun...@lists.lustre.org] On Behalf 
Of Dilger, Andreas
Sent: Wednesday, April 12, 2017 8:19 AM
To: Todd, Allen <allen.t...@msx.bala.susq.com>
Cc: E.S. Rosenberg <e...@cs.huji.ac.il>; lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] lfs_migrate

On Apr 10, 2017, at 14:53, Todd, Allen <allen.t...@sig.com> wrote:
>
> While everyone is talking about lfs migrate, I would like to point out that 
> it appears to be missing an option to preserve file modification and access 
> times, which makes it less useful for behind the scenes data management tasks.

This should actually be the default, though there was a bug in older versions 
of Lustre that didn't preserve the timestamps.  That was fixed in Lustre 2.8.

Cheers, Andreas

> Allen
>
> -Original Message-
> From: lustre-discuss [mailto:lustre-discuss-boun...@lists.lustre.org]
> On Behalf Of Henri Doreau
> Sent: Tuesday, April 04, 2017 3:18 AM
> To: E.S. Rosenberg <e...@cs.huji.ac.il>
> Cc: lustre-discuss@lists.lustre.org
> Subject: Re: [lustre-discuss] lfs_migrate
>
> Hello,
>
> the manpage for lfs(1) lists the available options in 2.8:
> """
> lfs migrate -m  directory
> lfs migrate [-c | --stripe-count ]
>   [-i | --stripe-index ]
>   [-S | --stripe-size ]
>   [-p | --pool ]
>   [-o | --ost-list ]
>   [-b | --block]
>   [-n | --non-block] file|directory """
>
> Although this is certainly terse, I guess that most parameters are intuitive.
>
> The command will open the file to restripe (blocking concurrent accesses or 
> not, depending on -b/-n), create a special "volatile" (=unlinked) one with 
> the requested striping parameters and copy the source into the destination.
>
> If the copy succeeds, the two files are atomically swapped and concurrent 
> access protection is released.
>
> In non-blocking mode, the process will detect if the source file was already 
> opened or if there's an open during the copy process and abort safely. It is 
> then up to the admin to reschedule the migration later, maybe with -b.
>
> HTH
>
> Henri
>
> On 02/avril - 14:43 E.S. Rosenberg wrote:
>> Thanks for all the great replies!
>>
>> I may be wrong on this but 'lfs migrate' does not seem to be
>> documented in the manpage (my local one is 2.8 so I expect that but
>> even manpages that I find online).
>>
>> Any pointers would be very welcome.
>>
>> On Thu, Mar 23, 2017 at 12:31 PM, Henri Doreau <henri.dor...@cea.fr> wrote:
>>
>>> On 20/mars - 22:50 E.S. Rosenberg wrote:
>>>> On Mon, Mar 20, 2017 at 10:19 PM, Dilger, Andreas <
>>> andreas.dil...@intel.com>
>>>> wrote:
>>>>
>>>>> The underlying "lfs migrate" command (not the "lfs_migrate"
>>>>> script) in newer Lustre versions (2.9) is capable of migrating
>>>>> files that are in
>>> use
>>>>> by using the "--block" option, which prevents other processes from
>>>>> accessing or modifying the file during migration.
>>>>>
>>>>> Unfortunately, "lfs_migrate" doesn't pass that argument on, though
>>>>> it wouldn't be hard to change the script. Ideally, the "lfs_migrate"
>>> script
>>>>> would pass all unknown options to "lfs migrate".
>>>>>
>>>>>
>>>>> The other item of note is that setting the OST inactive on the MDS
>>>>> will prevent the MDS from deleting objects on the OST (see
>>>>> https://jira.hpdd.intel.com/browse/LU-4825 for details).  In
>>>>> Lustre
>>> 2.9
>>>>> and later it is possible to set on the MDS:
>>>>>
>>>>>   mds# lctl set_param osp..create_count=0
>>>>>
>>>>> to stop MDS allocation of new objects on that OST. On older
>>>>> versions
>>> it is
>>>>> possible to set on the OSS:
>>>>>
>>>>>  oss# lctl set_param obdfilter..degraded=1
>>>>>
>>>>> so that it tells the MDS to avoid it if possible, but this isn't a
>>>>> hard exclusion.
>>>>>
>>>>> It is also possible to use a testing hack to mark an OST as out of
>>> inodes,

Re: [lustre-discuss] lfs_migrate

[Dilger, Andreas]

On Apr 10, 2017, at 14:53, Todd, Allen <allen.t...@sig.com> wrote:
> 
> While everyone is talking about lfs migrate, I would like to point out that 
> it appears to be missing an option to preserve file modification and access 
> times, which makes it less useful for behind the scenes data management tasks.

This should actually be the default, though there was a bug in older versions 
of Lustre that didn't preserve the timestamps.  That was fixed in Lustre 2.8.

Cheers, Andreas

> Allen
> 
> -Original Message-
> From: lustre-discuss [mailto:lustre-discuss-boun...@lists.lustre.org] On 
> Behalf Of Henri Doreau
> Sent: Tuesday, April 04, 2017 3:18 AM
> To: E.S. Rosenberg <e...@cs.huji.ac.il>
> Cc: lustre-discuss@lists.lustre.org
> Subject: Re: [lustre-discuss] lfs_migrate
> 
> Hello,
> 
> the manpage for lfs(1) lists the available options in 2.8:
> """
> lfs migrate -m  directory
> lfs migrate [-c | --stripe-count ]
>   [-i | --stripe-index ]
>   [-S | --stripe-size ]
>   [-p | --pool ]
>   [-o | --ost-list ]
>   [-b | --block]
>   [-n | --non-block] file|directory """
> 
> Although this is certainly terse, I guess that most parameters are intuitive.
> 
> The command will open the file to restripe (blocking concurrent accesses or 
> not, depending on -b/-n), create a special "volatile" (=unlinked) one with 
> the requested striping parameters and copy the source into the destination.
> 
> If the copy succeeds, the two files are atomically swapped and concurrent 
> access protection is released.
> 
> In non-blocking mode, the process will detect if the source file was already 
> opened or if there's an open during the copy process and abort safely. It is 
> then up to the admin to reschedule the migration later, maybe with -b.
> 
> HTH
> 
> Henri
> 
> On 02/avril - 14:43 E.S. Rosenberg wrote:
>> Thanks for all the great replies!
>> 
>> I may be wrong on this but 'lfs migrate' does not seem to be
>> documented in the manpage (my local one is 2.8 so I expect that but
>> even manpages that I find online).
>> 
>> Any pointers would be very welcome.
>> 
>> On Thu, Mar 23, 2017 at 12:31 PM, Henri Doreau <henri.dor...@cea.fr> wrote:
>> 
>>> On 20/mars - 22:50 E.S. Rosenberg wrote:
>>>> On Mon, Mar 20, 2017 at 10:19 PM, Dilger, Andreas <
>>> andreas.dil...@intel.com>
>>>> wrote:
>>>> 
>>>>> The underlying "lfs migrate" command (not the "lfs_migrate"
>>>>> script) in newer Lustre versions (2.9) is capable of migrating
>>>>> files that are in
>>> use
>>>>> by using the "--block" option, which prevents other processes
>>>>> from accessing or modifying the file during migration.
>>>>> 
>>>>> Unfortunately, "lfs_migrate" doesn't pass that argument on,
>>>>> though it wouldn't be hard to change the script. Ideally, the 
>>>>> "lfs_migrate"
>>> script
>>>>> would pass all unknown options to "lfs migrate".
>>>>> 
>>>>> 
>>>>> The other item of note is that setting the OST inactive on the
>>>>> MDS will prevent the MDS from deleting objects on the OST (see
>>>>> https://jira.hpdd.intel.com/browse/LU-4825 for details).  In
>>>>> Lustre
>>> 2.9
>>>>> and later it is possible to set on the MDS:
>>>>> 
>>>>>   mds# lctl set_param osp..create_count=0
>>>>> 
>>>>> to stop MDS allocation of new objects on that OST. On older
>>>>> versions
>>> it is
>>>>> possible to set on the OSS:
>>>>> 
>>>>>  oss# lctl set_param obdfilter..degraded=1
>>>>> 
>>>>> so that it tells the MDS to avoid it if possible, but this isn't
>>>>> a hard exclusion.
>>>>> 
>>>>> It is also possible to use a testing hack to mark an OST as out
>>>>> of
>>> inodes,
>>>>> but that only works for one OST per OSS and it sounds like that
>>>>> won't
>>> be
>>>>> useful in this case.
>>>>> 
>>>>> Cheers, Andreas
>>>>> 
>>>> You're making me want Lustre 2.9 more :) but for now I'm still
>>>> stuck on
>>> 2.8
>>>> and because this is very much production these days I'm mo

Re: [lustre-discuss] lfs_migrate

[Todd, Allen]

While everyone is talking about lfs migrate, I would like to point out that it 
appears to be missing an option to preserve file modification and access times, 
which makes it less useful for behind the scenes data management tasks.

Allen

-Original Message-
From: lustre-discuss [mailto:lustre-discuss-boun...@lists.lustre.org] On Behalf 
Of Henri Doreau
Sent: Tuesday, April 04, 2017 3:18 AM
To: E.S. Rosenberg <e...@cs.huji.ac.il>
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] lfs_migrate

Hello,

the manpage for lfs(1) lists the available options in 2.8:
"""
lfs migrate -m  directory
lfs migrate [-c | --stripe-count ]
   [-i | --stripe-index ]
   [-S | --stripe-size ]
   [-p | --pool ]
   [-o | --ost-list ]
   [-b | --block]
   [-n | --non-block] file|directory """

Although this is certainly terse, I guess that most parameters are intuitive.

The command will open the file to restripe (blocking concurrent accesses or 
not, depending on -b/-n), create a special "volatile" (=unlinked) one with the 
requested striping parameters and copy the source into the destination.

If the copy succeeds, the two files are atomically swapped and concurrent 
access protection is released.

In non-blocking mode, the process will detect if the source file was already 
opened or if there's an open during the copy process and abort safely. It is 
then up to the admin to reschedule the migration later, maybe with -b.

HTH

Henri

On 02/avril - 14:43 E.S. Rosenberg wrote:
> Thanks for all the great replies!
>
> I may be wrong on this but 'lfs migrate' does not seem to be
> documented in the manpage (my local one is 2.8 so I expect that but
> even manpages that I find online).
>
> Any pointers would be very welcome.
>
> On Thu, Mar 23, 2017 at 12:31 PM, Henri Doreau <henri.dor...@cea.fr> wrote:
>
> > On 20/mars - 22:50 E.S. Rosenberg wrote:
> > > On Mon, Mar 20, 2017 at 10:19 PM, Dilger, Andreas <
> > andreas.dil...@intel.com>
> > > wrote:
> > >
> > > > The underlying "lfs migrate" command (not the "lfs_migrate"
> > > > script) in newer Lustre versions (2.9) is capable of migrating
> > > > files that are in
> > use
> > > > by using the "--block" option, which prevents other processes
> > > > from accessing or modifying the file during migration.
> > > >
> > > > Unfortunately, "lfs_migrate" doesn't pass that argument on,
> > > > though it wouldn't be hard to change the script. Ideally, the 
> > > > "lfs_migrate"
> > script
> > > > would pass all unknown options to "lfs migrate".
> > > >
> > > >
> > > > The other item of note is that setting the OST inactive on the
> > > > MDS will prevent the MDS from deleting objects on the OST (see
> > > > https://jira.hpdd.intel.com/browse/LU-4825 for details).  In
> > > > Lustre
> > 2.9
> > > > and later it is possible to set on the MDS:
> > > >
> > > >mds# lctl set_param osp..create_count=0
> > > >
> > > > to stop MDS allocation of new objects on that OST. On older
> > > > versions
> > it is
> > > > possible to set on the OSS:
> > > >
> > > >   oss# lctl set_param obdfilter..degraded=1
> > > >
> > > > so that it tells the MDS to avoid it if possible, but this isn't
> > > > a hard exclusion.
> > > >
> > > > It is also possible to use a testing hack to mark an OST as out
> > > > of
> > inodes,
> > > > but that only works for one OST per OSS and it sounds like that
> > > > won't
> > be
> > > > useful in this case.
> > > >
> > > > Cheers, Andreas
> > > >
> > > You're making me want Lustre 2.9 more :) but for now I'm still
> > > stuck on
> > 2.8
> > > and because this is very much production these days I'm more
> > > careful with the update (hoping to finally get hw allocated for a
> > > test env soon to
> > test
> > > the update).
> > > Thanks,
> > > Eli
> > >
> >
> > Hello,
> >
> > this safer version of `lfs migrate' (LU-4840) is actually available
> > in 2.8.
> >
> > When used with --non-block flag, a concurrent open of the file being
> > migrated will cause the migration to fail. With --block (or nothing,
> > it's the default behavior) and as Andreas said, concurrent opens
> > will

Re: [lustre-discuss] lfs_migrate

[Henri Doreau]

Hi,

before people look for the man page in vain: although there is actually
a lfs-migrate(1) man page in 2.8 sources it doesn't seem to be
installed...

Page was added in [1] (included in 2.8) but only added to the
Makefile.am in [2] (which will be part of 2.10).

Regards

Henri

[1] https://review.whamcloud.com/#/c/17392
[2] https://review.whamcloud.com/#/c/24371

On 04/avril - 22:20 Dilger, Andreas wrote:
> There is the "lfs-migrate.1" man page that has a few more details, though it 
> would be great if someone had the time to add more details on each of the 
> options.
> 
> On Apr 4, 2017, at 01:18, Henri Doreau  wrote:
> > 
> > Hello,
> > 
> > the manpage for lfs(1) lists the available options in 2.8:
> > """
> > lfs migrate -m  directory
> > lfs migrate [-c | --stripe-count ]
> >   [-i | --stripe-index ]
> >   [-S | --stripe-size ]
> >   [-p | --pool ]
> >   [-o | --ost-list ]
> >   [-b | --block]
> >   [-n | --non-block] file|directory
> > """
> > 
> > Although this is certainly terse, I guess that most parameters are 
> > intuitive.
> > 
> > The command will open the file to restripe (blocking concurrent accesses
> > or not, depending on -b/-n), create a special "volatile" (=unlinked) one
> > with the requested striping parameters and copy the source into the
> > destination.
> > 
> > If the copy succeeds, the two files are atomically swapped and
> > concurrent access protection is released.
> > 
> > In non-blocking mode, the process will detect if the source file was
> > already opened or if there's an open during the copy process and abort
> > safely. It is then up to the admin to reschedule the migration later,
> > maybe with -b.
> > 
> > HTH
> > 
> > Henri
> > 
> > On 02/avril - 14:43 E.S. Rosenberg wrote:
> >> Thanks for all the great replies!
> >> 
> >> I may be wrong on this but 'lfs migrate' does not seem to be documented in
> >> the manpage (my local one is 2.8 so I expect that but even manpages that I
> >> find online).
> >> 
> >> Any pointers would be very welcome.
> >> 
> >> On Thu, Mar 23, 2017 at 12:31 PM, Henri Doreau  wrote:
> >> 
> >>> On 20/mars - 22:50 E.S. Rosenberg wrote:
>  On Mon, Mar 20, 2017 at 10:19 PM, Dilger, Andreas 
>  
>  wrote:
>  
> > The underlying "lfs migrate" command (not the "lfs_migrate" script) in
> > newer Lustre versions (2.9) is capable of migrating files that are in 
> > use
> > by using the "--block" option, which prevents other processes from
> > accessing or modifying the file during migration.
> > 
> > Unfortunately, "lfs_migrate" doesn't pass that argument on, though it
> > wouldn't be hard to change the script. Ideally, the "lfs_migrate" script
> > would pass all unknown options to "lfs migrate".
> > 
> > 
> > The other item of note is that setting the OST inactive on the MDS will
> > prevent the MDS from deleting objects on the OST (see
> > https://jira.hpdd.intel.com/browse/LU-4825 for details).  In Lustre 2.9
> > and later it is possible to set on the MDS:
> > 
> >   mds# lctl set_param osp..create_count=0
> > 
> > to stop MDS allocation of new objects on that OST. On older versions
> > it is possible to set on the OSS:
> > 
> >  oss# lctl set_param obdfilter..degraded=1
> > 
> > so that it tells the MDS to avoid it if possible, but this isn't a hard
> > exclusion.
> > 
> > It is also possible to use a testing hack to mark an OST as out of 
> > inodes,
> > but that only works for one OST per OSS and it sounds like that won't be
> > useful in this case.
> > 
> > Cheers, Andreas
> > 
>  You're making me want Lustre 2.9 more :) but for now I'm still stuck on
>  2.8 and because this is very much production these days I'm more careful 
>  with
>  the update (hoping to finally get hw allocated for a test env soon to 
>  test
>  the update).
>  Thanks,
>  Eli
>  
> >>> 
> >>> Hello,
> >>> 
> >>> this safer version of `lfs migrate' (LU-4840) is actually available in
> >>> 2.8.
> >>> 
> >>> When used with --non-block flag, a concurrent open of the file being
> >>> migrated will cause the migration to fail. With --block (or nothing,
> >>> it's the default behavior) and as Andreas said, concurrent opens will
> >>> block until the migration completes.
> >>> 
> >>> Regards
> >>> 
> >>> --
> >>> Henri Doreau
> >>> 
> 
> Cheers, Andreas
> --
> Andreas Dilger
> Lustre Principal Architect
> Intel Corporation
> 
> 
> 
> 
> 
> 
> 

-- 
Henri Doreau
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lfs_migrate

[Dilger, Andreas]

There is the "lfs-migrate.1" man page that has a few more details, though it 
would be great if someone had the time to add more details on each of the 
options.

On Apr 4, 2017, at 01:18, Henri Doreau  wrote:
> 
> Hello,
> 
> the manpage for lfs(1) lists the available options in 2.8:
> """
> lfs migrate -m  directory
> lfs migrate [-c | --stripe-count ]
>   [-i | --stripe-index ]
>   [-S | --stripe-size ]
>   [-p | --pool ]
>   [-o | --ost-list ]
>   [-b | --block]
>   [-n | --non-block] file|directory
> """
> 
> Although this is certainly terse, I guess that most parameters are intuitive.
> 
> The command will open the file to restripe (blocking concurrent accesses
> or not, depending on -b/-n), create a special "volatile" (=unlinked) one
> with the requested striping parameters and copy the source into the
> destination.
> 
> If the copy succeeds, the two files are atomically swapped and
> concurrent access protection is released.
> 
> In non-blocking mode, the process will detect if the source file was
> already opened or if there's an open during the copy process and abort
> safely. It is then up to the admin to reschedule the migration later,
> maybe with -b.
> 
> HTH
> 
> Henri
> 
> On 02/avril - 14:43 E.S. Rosenberg wrote:
>> Thanks for all the great replies!
>> 
>> I may be wrong on this but 'lfs migrate' does not seem to be documented in
>> the manpage (my local one is 2.8 so I expect that but even manpages that I
>> find online).
>> 
>> Any pointers would be very welcome.
>> 
>> On Thu, Mar 23, 2017 at 12:31 PM, Henri Doreau  wrote:
>> 
>>> On 20/mars - 22:50 E.S. Rosenberg wrote:
 On Mon, Mar 20, 2017 at 10:19 PM, Dilger, Andreas 
 
 wrote:
 
> The underlying "lfs migrate" command (not the "lfs_migrate" script) in
> newer Lustre versions (2.9) is capable of migrating files that are in use
> by using the "--block" option, which prevents other processes from
> accessing or modifying the file during migration.
> 
> Unfortunately, "lfs_migrate" doesn't pass that argument on, though it
> wouldn't be hard to change the script. Ideally, the "lfs_migrate" script
> would pass all unknown options to "lfs migrate".
> 
> 
> The other item of note is that setting the OST inactive on the MDS will
> prevent the MDS from deleting objects on the OST (see
> https://jira.hpdd.intel.com/browse/LU-4825 for details).  In Lustre 2.9
> and later it is possible to set on the MDS:
> 
>   mds# lctl set_param osp..create_count=0
> 
> to stop MDS allocation of new objects on that OST. On older versions
> it is possible to set on the OSS:
> 
>  oss# lctl set_param obdfilter..degraded=1
> 
> so that it tells the MDS to avoid it if possible, but this isn't a hard
> exclusion.
> 
> It is also possible to use a testing hack to mark an OST as out of inodes,
> but that only works for one OST per OSS and it sounds like that won't be
> useful in this case.
> 
> Cheers, Andreas
> 
 You're making me want Lustre 2.9 more :) but for now I'm still stuck on
 2.8 and because this is very much production these days I'm more careful 
 with
 the update (hoping to finally get hw allocated for a test env soon to test
 the update).
 Thanks,
 Eli
 
>>> 
>>> Hello,
>>> 
>>> this safer version of `lfs migrate' (LU-4840) is actually available in
>>> 2.8.
>>> 
>>> When used with --non-block flag, a concurrent open of the file being
>>> migrated will cause the migration to fail. With --block (or nothing,
>>> it's the default behavior) and as Andreas said, concurrent opens will
>>> block until the migration completes.
>>> 
>>> Regards
>>> 
>>> --
>>> Henri Doreau
>>> 

Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Intel Corporation







___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lfs_migrate

[Henri Doreau]

Hello,

the manpage for lfs(1) lists the available options in 2.8:
"""
lfs migrate -m  directory
lfs migrate [-c | --stripe-count ]
   [-i | --stripe-index ]
   [-S | --stripe-size ]
   [-p | --pool ]
   [-o | --ost-list ]
   [-b | --block]
   [-n | --non-block] file|directory
"""

Although this is certainly terse, I guess that most parameters are intuitive.

The command will open the file to restripe (blocking concurrent accesses
or not, depending on -b/-n), create a special "volatile" (=unlinked) one
with the requested striping parameters and copy the source into the
destination.

If the copy succeeds, the two files are atomically swapped and
concurrent access protection is released.

In non-blocking mode, the process will detect if the source file was
already opened or if there's an open during the copy process and abort
safely. It is then up to the admin to reschedule the migration later,
maybe with -b.

HTH

Henri

On 02/avril - 14:43 E.S. Rosenberg wrote:
> Thanks for all the great replies!
> 
> I may be wrong on this but 'lfs migrate' does not seem to be documented in
> the manpage (my local one is 2.8 so I expect that but even manpages that I
> find online).
> 
> Any pointers would be very welcome.
> 
> On Thu, Mar 23, 2017 at 12:31 PM, Henri Doreau  wrote:
> 
> > On 20/mars - 22:50 E.S. Rosenberg wrote:
> > > On Mon, Mar 20, 2017 at 10:19 PM, Dilger, Andreas <
> > andreas.dil...@intel.com>
> > > wrote:
> > >
> > > > The underlying "lfs migrate" command (not the "lfs_migrate" script) in
> > > > newer Lustre versions (2.9) is capable of migrating files that are in
> > use
> > > > by using the "--block" option, which prevents other processes from
> > > > accessing or modifying the file during migration.
> > > >
> > > > Unfortunately, "lfs_migrate" doesn't pass that argument on, though it
> > > > wouldn't be hard to change the script. Ideally, the "lfs_migrate"
> > script
> > > > would pass all unknown options to "lfs migrate".
> > > >
> > > >
> > > > The other item of note is that setting the OST inactive on the MDS will
> > > > prevent the MDS from deleting objects on the OST (see
> > > > https://jira.hpdd.intel.com/browse/LU-4825 for details).  In Lustre
> > 2.9
> > > > and later it is possible to set on the MDS:
> > > >
> > > >mds# lctl set_param osp..create_count=0
> > > >
> > > > to stop MDS allocation of new objects on that OST. On older versions
> > it is
> > > > possible to set on the OSS:
> > > >
> > > >   oss# lctl set_param obdfilter..degraded=1
> > > >
> > > > so that it tells the MDS to avoid it if possible, but this isn't a hard
> > > > exclusion.
> > > >
> > > > It is also possible to use a testing hack to mark an OST as out of
> > inodes,
> > > > but that only works for one OST per OSS and it sounds like that won't
> > be
> > > > useful in this case.
> > > >
> > > > Cheers, Andreas
> > > >
> > > You're making me want Lustre 2.9 more :) but for now I'm still stuck on
> > 2.8
> > > and because this is very much production these days I'm more careful with
> > > the update (hoping to finally get hw allocated for a test env soon to
> > test
> > > the update).
> > > Thanks,
> > > Eli
> > >
> >
> > Hello,
> >
> > this safer version of `lfs migrate' (LU-4840) is actually available in
> > 2.8.
> >
> > When used with --non-block flag, a concurrent open of the file being
> > migrated will cause the migration to fail. With --block (or nothing,
> > it's the default behavior) and as Andreas said, concurrent opens will
> > block until the migration completes.
> >
> > Regards
> >
> > --
> > Henri Doreau
> >

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lfs_migrate

[Henri Doreau]

On 20/mars - 22:50 E.S. Rosenberg wrote:
> On Mon, Mar 20, 2017 at 10:19 PM, Dilger, Andreas 
> wrote:
> 
> > The underlying "lfs migrate" command (not the "lfs_migrate" script) in
> > newer Lustre versions (2.9) is capable of migrating files that are in use
> > by using the "--block" option, which prevents other processes from
> > accessing or modifying the file during migration.
> >
> > Unfortunately, "lfs_migrate" doesn't pass that argument on, though it
> > wouldn't be hard to change the script. Ideally, the "lfs_migrate" script
> > would pass all unknown options to "lfs migrate".
> >
> >
> > The other item of note is that setting the OST inactive on the MDS will
> > prevent the MDS from deleting objects on the OST (see
> > https://jira.hpdd.intel.com/browse/LU-4825 for details).  In Lustre 2.9
> > and later it is possible to set on the MDS:
> >
> >mds# lctl set_param osp..create_count=0
> >
> > to stop MDS allocation of new objects on that OST. On older versions it is
> > possible to set on the OSS:
> >
> >   oss# lctl set_param obdfilter..degraded=1
> >
> > so that it tells the MDS to avoid it if possible, but this isn't a hard
> > exclusion.
> >
> > It is also possible to use a testing hack to mark an OST as out of inodes,
> > but that only works for one OST per OSS and it sounds like that won't be
> > useful in this case.
> >
> > Cheers, Andreas
> >
> You're making me want Lustre 2.9 more :) but for now I'm still stuck on 2.8
> and because this is very much production these days I'm more careful with
> the update (hoping to finally get hw allocated for a test env soon to test
> the update).
> Thanks,
> Eli
> 

Hello,

this safer version of `lfs migrate' (LU-4840) is actually available in
2.8.

When used with --non-block flag, a concurrent open of the file being
migrated will cause the migration to fail. With --block (or nothing,
it's the default behavior) and as Andreas said, concurrent opens will
block until the migration completes.

Regards

-- 
Henri Doreau
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lfs_migrate

[Dilger, Andreas]

If you mark the OSTs degraded and the MDS will still avoid them for new 
allocations, though you should make a second scanning pass to verify.

Deactivating the OSTs on the MDS will allow it to delete the (now unused) OST 
objects.

Cheers, Andreas

On Mar 20, 2017, at 17:03, E.S. Rosenberg 
> wrote:



On Mon, Mar 20, 2017 at 10:59 PM, Dilger, Andreas 
> wrote:
If you've marked the OST inactive on the MDS then that is not surprising. See 
https://jira.hpdd.intel.com/browse/LU-4825 and the comments in my previous 
email.
Ah OK.
But if I re-activate the OST will lfs_migrate still move them away from the 
device?

Cheers, Andreas

On Mar 20, 2017, at 16:56, E.S. Rosenberg 
> wrote:



On Mon, Mar 20, 2017 at 10:50 PM, E.S. Rosenberg 
> wrote:


On Mon, Mar 20, 2017 at 10:19 PM, Dilger, Andreas 
> wrote:
The underlying "lfs migrate" command (not the "lfs_migrate" script) in newer 
Lustre versions (2.9) is capable of migrating files that are in use by using 
the "--block" option, which prevents other processes from accessing or 
modifying the file during migration.

Unfortunately, "lfs_migrate" doesn't pass that argument on, though it wouldn't 
be hard to change the script. Ideally, the "lfs_migrate" script would pass all 
unknown options to "lfs migrate".


The other item of note is that setting the OST inactive on the MDS will prevent 
the MDS from deleting objects on the OST (see 
https://jira.hpdd.intel.com/browse/LU-4825 for details).  In Lustre 2.9 and 
later it is possible to set on the MDS:

   mds# lctl set_param osp..create_count=0

to stop MDS allocation of new objects on that OST. On older versions it is 
possible to set on the OSS:

  oss# lctl set_param obdfilter..degraded=1

so that it tells the MDS to avoid it if possible, but this isn't a hard 
exclusion.

It is also possible to use a testing hack to mark an OST as out of inodes, but 
that only works for one OST per OSS and it sounds like that won't be useful in 
this case.

Cheers, Andreas
You're making me want Lustre 2.9 more :) but for now I'm still stuck on 2.8 and 
because this is very much production these days I'm more careful with the 
update (hoping to finally get hw allocated for a test env soon to test the 
update).
Thanks,
Eli
Another related question:
The migration has been running for several hours now on one OST but I am yet to 
see 1 block being freed from the OSS point of view, is this not mv but cp as 
far as the original OST is concerned?
(also man lfs has no lfs migrate that was added in 2.9?)
Thanks,
Eli

On Mar 20, 2017, at 13:11, Brett Lee 
> wrote:

Hi Eli,

I believe that is still the case with lfs_migrate.  If otherwise, we'll 
probably hear soon.

You should be able to disable those OSTs while keeping the file system active - 
via a command on the MDS(s) as well as the clients.  My notes have the command 
as shown below, but please confirm via the appropriate Lustre manual:

lctl set_param osc.--*.active=0

Brett
--
Protect Yourself Against Cybercrime
PDS Software Solutions LLC
https://www.TrustPDS.com

On Mon, Mar 20, 2017 at 10:43 AM, E.S. Rosenberg 
> wrote:
In the man page it says the following:

Because  lfs_migrate  is  not yet closely integrated with the MDS, it cannot 
determine whether a file is currently open and/or in-use by other applications 
or nodes.  That makes it UNSAFE for use on files that might be modified by 
other applications, since the migrated file is only a copy of the current file, 
and this will result in the old file becoming an open-unlinked file and any  
modifications to that file will be lost.

Is this still the case?
Is there a better way to disable OSTs while keeping the filesystem live?

Background:
We need to take a OSS enclosure that hosts multiple OSTs offline for hardware 
maintenance, I'd like to do this without bringing Lustre as a whole down. I 
made sure there is enough space on the other OSTs to house the contents of the 
machine going offline and am now about to move things.

Thanks,
Eli

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org



___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org

Re: [lustre-discuss] lfs_migrate

[Dilger, Andreas]

If you've marked the OST inactive on the MDS then that is not surprising. See 
https://jira.hpdd.intel.com/browse/LU-4825 and the comments in my previous 
email.

Cheers, Andreas

On Mar 20, 2017, at 16:56, E.S. Rosenberg 
> wrote:



On Mon, Mar 20, 2017 at 10:50 PM, E.S. Rosenberg 
> wrote:


On Mon, Mar 20, 2017 at 10:19 PM, Dilger, Andreas 
> wrote:
The underlying "lfs migrate" command (not the "lfs_migrate" script) in newer 
Lustre versions (2.9) is capable of migrating files that are in use by using 
the "--block" option, which prevents other processes from accessing or 
modifying the file during migration.

Unfortunately, "lfs_migrate" doesn't pass that argument on, though it wouldn't 
be hard to change the script. Ideally, the "lfs_migrate" script would pass all 
unknown options to "lfs migrate".


The other item of note is that setting the OST inactive on the MDS will prevent 
the MDS from deleting objects on the OST (see 
https://jira.hpdd.intel.com/browse/LU-4825 for details).  In Lustre 2.9 and 
later it is possible to set on the MDS:

   mds# lctl set_param osp..create_count=0

to stop MDS allocation of new objects on that OST. On older versions it is 
possible to set on the OSS:

  oss# lctl set_param obdfilter..degraded=1

so that it tells the MDS to avoid it if possible, but this isn't a hard 
exclusion.

It is also possible to use a testing hack to mark an OST as out of inodes, but 
that only works for one OST per OSS and it sounds like that won't be useful in 
this case.

Cheers, Andreas
You're making me want Lustre 2.9 more :) but for now I'm still stuck on 2.8 and 
because this is very much production these days I'm more careful with the 
update (hoping to finally get hw allocated for a test env soon to test the 
update).
Thanks,
Eli
Another related question:
The migration has been running for several hours now on one OST but I am yet to 
see 1 block being freed from the OSS point of view, is this not mv but cp as 
far as the original OST is concerned?
(also man lfs has no lfs migrate that was added in 2.9?)
Thanks,
Eli

On Mar 20, 2017, at 13:11, Brett Lee 
> wrote:

Hi Eli,

I believe that is still the case with lfs_migrate.  If otherwise, we'll 
probably hear soon.

You should be able to disable those OSTs while keeping the file system active - 
via a command on the MDS(s) as well as the clients.  My notes have the command 
as shown below, but please confirm via the appropriate Lustre manual:

lctl set_param osc.--*.active=0

Brett
--
Protect Yourself Against Cybercrime
PDS Software Solutions LLC
https://www.TrustPDS.com

On Mon, Mar 20, 2017 at 10:43 AM, E.S. Rosenberg 
> wrote:
In the man page it says the following:

Because  lfs_migrate  is  not yet closely integrated with the MDS, it cannot 
determine whether a file is currently open and/or in-use by other applications 
or nodes.  That makes it UNSAFE for use on files that might be modified by 
other applications, since the migrated file is only a copy of the current file, 
and this will result in the old file becoming an open-unlinked file and any  
modifications to that file will be lost.

Is this still the case?
Is there a better way to disable OSTs while keeping the filesystem live?

Background:
We need to take a OSS enclosure that hosts multiple OSTs offline for hardware 
maintenance, I'd like to do this without bringing Lustre as a whole down. I 
made sure there is enough space on the other OSTs to house the contents of the 
machine going offline and am now about to move things.

Thanks,
Eli

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lfs_migrate

[E.S. Rosenberg]

On Mon, Mar 20, 2017 at 10:19 PM, Dilger, Andreas 
wrote:

> The underlying "lfs migrate" command (not the "lfs_migrate" script) in
> newer Lustre versions (2.9) is capable of migrating files that are in use
> by using the "--block" option, which prevents other processes from
> accessing or modifying the file during migration.
>
> Unfortunately, "lfs_migrate" doesn't pass that argument on, though it
> wouldn't be hard to change the script. Ideally, the "lfs_migrate" script
> would pass all unknown options to "lfs migrate".
>
>
> The other item of note is that setting the OST inactive on the MDS will
> prevent the MDS from deleting objects on the OST (see
> https://jira.hpdd.intel.com/browse/LU-4825 for details).  In Lustre 2.9
> and later it is possible to set on the MDS:
>
>mds# lctl set_param osp..create_count=0
>
> to stop MDS allocation of new objects on that OST. On older versions it is
> possible to set on the OSS:
>
>   oss# lctl set_param obdfilter..degraded=1
>
> so that it tells the MDS to avoid it if possible, but this isn't a hard
> exclusion.
>
> It is also possible to use a testing hack to mark an OST as out of inodes,
> but that only works for one OST per OSS and it sounds like that won't be
> useful in this case.
>
> Cheers, Andreas
>
You're making me want Lustre 2.9 more :) but for now I'm still stuck on 2.8
and because this is very much production these days I'm more careful with
the update (hoping to finally get hw allocated for a test env soon to test
the update).
Thanks,
Eli

>
> On Mar 20, 2017, at 13:11, Brett Lee  wrote:
>
> Hi Eli,
>
> I believe that is still the case with lfs_migrate.  If otherwise, we'll
> probably hear soon.
>
> You should be able to disable those OSTs while keeping the file system
> active - via a command on the MDS(s) as well as the clients.  My notes have
> the command as shown below, but please confirm via the appropriate Lustre
> manual:
>
> lctl set_param osc.--*.active=0
>
> Brett
> --
> Protect Yourself Against Cybercrime
> PDS Software Solutions LLC
> https://www.TrustPDS.com 
>
> On Mon, Mar 20, 2017 at 10:43 AM, E.S. Rosenberg <
> esr+lus...@mail.hebrew.edu> wrote:
>
>> In the man page it says the following:
>>
>> Because  lfs_migrate  is  not yet closely integrated with the MDS, it
>> cannot determine whether a file is currently open and/or in-use by other
>> applications or nodes.  That makes it UNSAFE for use on files that might be
>> modified by other applications, since the migrated file is only a copy of
>> the current file, and this will result in the old file becoming an
>> open-unlinked file and any  modifications to that file will be lost.
>>
>> Is this still the case?
>> Is there a better way to disable OSTs while keeping the filesystem live?
>>
>> Background:
>> We need to take a OSS enclosure that hosts multiple OSTs offline for
>> hardware maintenance, I'd like to do this without bringing Lustre as a
>> whole down. I made sure there is enough space on the other OSTs to house
>> the contents of the machine going offline and am now about to move things.
>>
>> Thanks,
>> Eli
>>
>> ___
>> lustre-discuss mailing list
>> lustre-discuss@lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>
>>
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
>
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lfs_migrate

[Dilger, Andreas]

The underlying "lfs migrate" command (not the "lfs_migrate" script) in newer 
Lustre versions (2.9) is capable of migrating files that are in use by using 
the "--block" option, which prevents other processes from accessing or 
modifying the file during migration.

Unfortunately, "lfs_migrate" doesn't pass that argument on, though it wouldn't 
be hard to change the script. Ideally, the "lfs_migrate" script would pass all 
unknown options to "lfs migrate".


The other item of note is that setting the OST inactive on the MDS will prevent 
the MDS from deleting objects on the OST (see 
https://jira.hpdd.intel.com/browse/LU-4825 for details).  In Lustre 2.9 and 
later it is possible to set on the MDS:

   mds# lctl set_param osp..create_count=0

to stop MDS allocation of new objects on that OST. On older versions it is 
possible to set on the OSS:

  oss# lctl set_param obdfilter..degraded=1

so that it tells the MDS to avoid it if possible, but this isn't a hard 
exclusion.

It is also possible to use a testing hack to mark an OST as out of inodes, but 
that only works for one OST per OSS and it sounds like that won't be useful in 
this case.

Cheers, Andreas

On Mar 20, 2017, at 13:11, Brett Lee 
> wrote:

Hi Eli,

I believe that is still the case with lfs_migrate.  If otherwise, we'll 
probably hear soon.

You should be able to disable those OSTs while keeping the file system active - 
via a command on the MDS(s) as well as the clients.  My notes have the command 
as shown below, but please confirm via the appropriate Lustre manual:

lctl set_param osc.--*.active=0

Brett
--
Protect Yourself Against Cybercrime
PDS Software Solutions LLC
https://www.TrustPDS.com

On Mon, Mar 20, 2017 at 10:43 AM, E.S. Rosenberg 
> wrote:
In the man page it says the following:

Because  lfs_migrate  is  not yet closely integrated with the MDS, it cannot 
determine whether a file is currently open and/or in-use by other applications 
or nodes.  That makes it UNSAFE for use on files that might be modified by 
other applications, since the migrated file is only a copy of the current file, 
and this will result in the old file becoming an open-unlinked file and any  
modifications to that file will be lost.

Is this still the case?
Is there a better way to disable OSTs while keeping the filesystem live?

Background:
We need to take a OSS enclosure that hosts multiple OSTs offline for hardware 
maintenance, I'd like to do this without bringing Lustre as a whole down. I 
made sure there is enough space on the other OSTs to house the contents of the 
machine going offline and am now about to move things.

Thanks,
Eli

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lfs_migrate

[Brett Lee]

Hi Eli,

I believe that is still the case with lfs_migrate.  If otherwise, we'll
probably hear soon.

You should be able to disable those OSTs while keeping the file system
active - via a command on the MDS(s) as well as the clients.  My notes have
the command as shown below, but please confirm via the appropriate Lustre
manual:

lctl set_param osc.--*.active=0

Brett
--
Protect Yourself Against Cybercrime
PDS Software Solutions LLC
https://www.TrustPDS.com 

On Mon, Mar 20, 2017 at 10:43 AM, E.S. Rosenberg  wrote:

> In the man page it says the following:
>
> Because  lfs_migrate  is  not yet closely integrated with the MDS, it
> cannot determine whether a file is currently open and/or in-use by other
> applications or nodes.  That makes it UNSAFE for use on files that might be
> modified by other applications, since the migrated file is only a copy of
> the current file, and this will result in the old file becoming an
> open-unlinked file and any  modifications to that file will be lost.
>
> Is this still the case?
> Is there a better way to disable OSTs while keeping the filesystem live?
>
> Background:
> We need to take a OSS enclosure that hosts multiple OSTs offline for
> hardware maintenance, I'd like to do this without bringing Lustre as a
> whole down. I made sure there is enough space on the other OSTs to house
> the contents of the machine going offline and am now about to move things.
>
> Thanks,
> Eli
>
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
>
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lfs_migrate transfer question

[Bob Ball]

Pre-make your list of files, split it into 5 or so parts, give each of 5 
clients one piece, and let them migrate in parallel.


bob

On 9/26/2016 9:00 AM, Jérôme BECOT wrote:

Hello,

As we had a inode usage issue, we did as Andreas advised :

- add a new ost with much more inodes
- disable the first ost
- use lfs_migrate to move files to the new ost

We have about 16M files on this OST (7.5TB). Our OSS are connected via 
10Gbps Ethernet, though the lfs_migrate command has been launched on a 
1Gbps connected client.


It's running for 5 days and is at about 50% now. It's pretty slow.

Since the command has been launched on a client, does the data transit 
through it ? Is there any way to improve performances ? (we will have 
to move data from the 2nd ost back once the first one will get more 
inodes, then reformat it, then rebalance)


Thank you



___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lfs_migrate question

[Dilger, Andreas]

On 2016/02/17, 12:01, "lustre-discuss on behalf of Ms. Megan Larko" 

 on behalf of dobsonu...@gmail.com> wrote:

Greetings to One and All!

I am looking at the lfs_migrate command to move files among OST's in a file 
system.
In "Lustre Software Release 2.x Operations Manual" Section 33.2.2 Description 
of lfs_migrate it indicates that "Because lfs_migrate is not closely integrated 
with the MDS, it cannot determine whether a file is currently open and/or 
in-use by other applications or nodes.  This makes it UNSAFE (capitalized in 
Manual) for use on files that might be modified by other applications, since 
the migrated file is only a copy of the current file.  This results in the old 
file becoming an open-unlinked file and any modifications to that file are 
lost."

This entry is somewhat out-of-date.  IIRC, with Lustre 2.5+ the migrate command 
with the "--block" option will prevent other threads from accessing the file 
during migration.  Without the "--block" option, if the file is modified before 
migration completes then the migration will be aborted.  Patches to update the 
manual would be welcome.

All of the lfs_migrate examples show the command being run on an active/mounted 
Lustre file system.  Is there any way in which one knows whether a 
rebalanced/migrated file was in-use at the time of migration (or that it was 
not in-use at the time of migration)?  On a mounted Lustre FS, is it necessary 
to make the file system or directories therein read-only for the migration 
activity?  Would this trait of lfs_migrate being unable to determine whether 
the file scheduled to be migrated is or is not in-use pose an issue if new 
OST's are added to the file system and lfs_migrate command is issued (rather 
than wait for Lustre to re-balance the load over new OSTs by attrition, as it 
were)?

If "lfs_migrate" reports "falling back to rsync-based migration" then the 
client/server do not support the atomic layout swap needed to handle migration 
of open files, and is essentially just doing a copy+rename.  It will still 
check (unless you disable checksums) whether the file was modified during 
migration, but this will only detect active writers, and cannot handle open 
file handles (which may later write to the file).

In most cases this is fine, especially if you limit file migration to older 
files that are no longer in use, unless you have a workload that opens and 
modifies existing files.

Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Intel High Performance Data Division
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [Lustre-discuss] lfs_migrate errors

[Peter Mistich]

Bruno,

Yes I am root and have permissions.


Thanks,
Pete

On 01/10/2014 05:51 AM, Faccini, Bruno wrote:
 Hello Peter,
 Does the concerned file-system have been migrated and concerned files have 
 been created from 1.8.x ? If yes, you certainly trigger the issue we are 
 already tracking in LU-4293 where IGIF FIDs are not allowed to swap layouts.  
 A patch should come up soon to avoid this limitation.
 Hope this helps.
 Bye.
 Bruno.

 Le Dec 18, 2013 à 2:42 AM, Peter Mistich peter.mist...@rackspace.com
   a écrit :

 Hello,
 today I added a ost and trying to balance them and when I run
 lfs_migrate I get cannot swap layouts between filename and a volatile
 file (Operation not permitted)

 I am running lustre-2.5.52

 any help would be great.

 Thanks,
 Pete
 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss
 -
 Intel Corporation SAS (French simplified joint stock company)
 Registered headquarters: Les Montalets- 2, rue de Paris,
 92196 Meudon Cedex, France
 Registration Number:  302 456 199 R.C.S. NANTERRE
 Capital: 4,572,000 Euros

 This e-mail and any attachments may contain confidential material for
 the sole use of the intended recipient(s). Any review or distribution
 by others is strictly prohibited. If you are not the intended
 recipient, please contact the sender and delete all copies.


___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] lfs_migrate errors

[Peter Mistich]

Yes I have root and write permissions.

Still have the issue not sure how to fix.

Thanks,
Pete

On 01/10/2014 01:50 PM, Dilger, Andreas wrote:
 On 2013/12/19, 6:16 AM, Peter Mistich peter.mist...@rackspace.com
 wrote:

 hope this helps

 /opt/zenoss/perf//Daemons/bb01lr02s06.dfw1/zenperfsnmp-thresholds.pickle:
 cannot
 swap layouts between
 /opt/zenoss/perf//Daemons/bb01lr02s06.dfw1/zenperfsnmp-thresholds.pickle
 and a volatile file (Operation not permitted)
 error: migrate: migrate stripe file
 '/opt/zenoss/perf/Daemons/bb01lr02s06.dfw1/zenperfsnmp-thresholds.pickle'
 failed

 lfs getstripe -v
 /opt/zenoss/perf/Daemons/bb01lr02s06.dfw1/zenperfsnmp-thresholds.pickle
 /opt/zenoss/perf/Daemons/bb01lr02s06.dfw1/zenperfsnmp-thresholds.pickle
 lmm_magic:  0x0BD10BD0
 lmm_seq:0x20423
 lmm_object_id:  0x324d
 This looks like a new-style FID (i.e. file allocated on 2.x server)
 instead of an upgraded Lustre 1.x IGIF FID (would have lmm_seq 
 0x1).

 This is different than the problem that I was investigating in LU-4293.

 Are you running this as root, or a regular user?  Presumably you have
 write permission on the file?


 lmm_stripe_count:   1
 lmm_stripe_size:1048576
 lmm_pattern:1
 lmm_layout_gen: 0
 lmm_stripe_offset:  1
  obdidx   objid   objid   group
   1 1347728   0x1490900


 Thanks,
 Pete



 On 12/19/2013 12:07 AM, Dilger, Andreas wrote:
 If you have only run with Lustre 2.5, can you please run lfs getstripe
 -v /path/to/file to print out the FID and other layout information for
 the file.

 Cheers, Andreas

 On Dec 17, 2013, at 19:22, Peter Mistich
 peter.mist...@rackspace.com wrote:

 Thank Amdreas, I will try the new lfs_migrate and let you know.

 However I have only run lustre 2.5

 Thanks,
 Pete

 On 12/17/2013 07:49 PM, Dilger, Andreas wrote:
 I saw this same problem on my system - it happens when trying to
 migrate a file created with Lustre 1.8.

 See https://jira.hpdd.intel.com/browse/LU-4293 for details.

 I have an updated version of lfs_migrate that works around this
 problem that I should push to Gerrit. The patch will be linked to the
 above bug when ready.

 Cheers, Andreas

 On Dec 17, 2013, at 18:43, Peter Mistich
 peter.mist...@rackspace.commailto:peter.mist...@rackspace.com
 wrote:

 Hello,
 today I added a ost and trying to balance them and when I run
 lfs_migrate I get cannot swap layouts between filename and a
 volatile
 file (Operation not permitted)

 I am running lustre-2.5.52

 any help would be great.

 Thanks,
 Pete
 ___
 Lustre-discuss mailing list

 Lustre-discuss@lists.lustre.orgmailto:Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss
 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss


 Cheers, Andreas

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] lfs_migrate errors

[Dilger, Andreas]

On 2014/01/13, 11:29 AM, Peter Mistich peter.mist...@rackspace.com
wrote:

Yes I have root and write permissions.

Still have the issue not sure how to fix.

Can you please do the following on the MDS as root (hopefully when the
system is relatively lightly loaded:

  DEBUG_SAVE=$(lctl get_param debug)
  lctl set_param debug=-1
  lctl clear

Then immediately run on the client:

  lctl set_param debug=-1
  lctl clear
  lfs_migrate /path/to/file/with/problems
  lctl dk /tmp/debug.client

and then on the MDS:

  lctl dk /tmp/debug.mds

and attach the resulting debug.mds and debug.client files to
http://jira.hpdd.intel.com/browse/LU-4293.

Cheers, Andreas

On 01/10/2014 01:50 PM, Dilger, Andreas wrote:
 On 2013/12/19, 6:16 AM, Peter Mistich peter.mist...@rackspace.com
 wrote:

 hope this helps

 
/opt/zenoss/perf//Daemons/bb01lr02s06.dfw1/zenperfsnmp-thresholds.pickle
:
 cannot
 swap layouts between
 
/opt/zenoss/perf//Daemons/bb01lr02s06.dfw1/zenperfsnmp-thresholds.pickle
 and a volatile file (Operation not permitted)
 error: migrate: migrate stripe file
 
'/opt/zenoss/perf/Daemons/bb01lr02s06.dfw1/zenperfsnmp-thresholds.pickle
'
 failed

 lfs getstripe -v
 /opt/zenoss/perf/Daemons/bb01lr02s06.dfw1/zenperfsnmp-thresholds.pickle
 /opt/zenoss/perf/Daemons/bb01lr02s06.dfw1/zenperfsnmp-thresholds.pickle
 lmm_magic:  0x0BD10BD0
 lmm_seq:0x20423
 lmm_object_id:  0x324d
 This looks like a new-style FID (i.e. file allocated on 2.x server)
 instead of an upgraded Lustre 1.x IGIF FID (would have lmm_seq 
 0x1).

 This is different than the problem that I was investigating in LU-4293.

 Are you running this as root, or a regular user?  Presumably you have
 write permission on the file?


 lmm_stripe_count:   1
 lmm_stripe_size:1048576
 lmm_pattern:1
 lmm_layout_gen: 0
 lmm_stripe_offset:  1
  obdidx   objid   objid   group
   1 1347728   0x1490900


 Thanks,
 Pete



 On 12/19/2013 12:07 AM, Dilger, Andreas wrote:
 If you have only run with Lustre 2.5, can you please run lfs
getstripe
 -v /path/to/file to print out the FID and other layout information
for
 the file.

 Cheers, Andreas

 On Dec 17, 2013, at 19:22, Peter Mistich
 peter.mist...@rackspace.com wrote:

 Thank Amdreas, I will try the new lfs_migrate and let you know.

 However I have only run lustre 2.5

 Thanks,
 Pete

 On 12/17/2013 07:49 PM, Dilger, Andreas wrote:
 I saw this same problem on my system - it happens when trying to
 migrate a file created with Lustre 1.8.

 See https://jira.hpdd.intel.com/browse/LU-4293 for details.

 I have an updated version of lfs_migrate that works around this
 problem that I should push to Gerrit. The patch will be linked to
the
 above bug when ready.

 Cheers, Andreas

 On Dec 17, 2013, at 18:43, Peter Mistich
 peter.mist...@rackspace.commailto:peter.mist...@rackspace.com
 wrote:

 Hello,
 today I added a ost and trying to balance them and when I run
 lfs_migrate I get cannot swap layouts between filename and a
 volatile
 file (Operation not permitted)

 I am running lustre-2.5.52

 any help would be great.

 Thanks,
 Pete



Cheers, Andreas
-- 
Andreas Dilger

Lustre Software Architect
Intel High Performance Data Division


___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] lfs_migrate errors

[Faccini, Bruno]

Hello Peter,
Does the concerned file-system have been migrated and concerned files have been 
created from 1.8.x ? If yes, you certainly trigger the issue we are already 
tracking in LU-4293 where IGIF FIDs are not allowed to swap layouts.  A patch 
should come up soon to avoid this limitation.
Hope this helps.
Bye.
Bruno.

Le Dec 18, 2013 à 2:42 AM, Peter Mistich peter.mist...@rackspace.com
 a écrit :

 Hello,
 today I added a ost and trying to balance them and when I run 
 lfs_migrate I get cannot swap layouts between filename and a volatile 
 file (Operation not permitted)
 
 I am running lustre-2.5.52
 
 any help would be great.
 
 Thanks,
 Pete
 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss

-
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: Les Montalets- 2, rue de Paris, 
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] lfs_migrate errors

[Dilger, Andreas]

On 2013/12/19, 6:16 AM, Peter Mistich peter.mist...@rackspace.com
wrote:

hope this helps

/opt/zenoss/perf//Daemons/bb01lr02s06.dfw1/zenperfsnmp-thresholds.pickle:
cannot 
swap layouts between
/opt/zenoss/perf//Daemons/bb01lr02s06.dfw1/zenperfsnmp-thresholds.pickle
and a volatile file (Operation not permitted)
error: migrate: migrate stripe file
'/opt/zenoss/perf/Daemons/bb01lr02s06.dfw1/zenperfsnmp-thresholds.pickle'
failed

lfs getstripe -v 
/opt/zenoss/perf/Daemons/bb01lr02s06.dfw1/zenperfsnmp-thresholds.pickle
/opt/zenoss/perf/Daemons/bb01lr02s06.dfw1/zenperfsnmp-thresholds.pickle
lmm_magic:  0x0BD10BD0
lmm_seq:0x20423
lmm_object_id:  0x324d

This looks like a new-style FID (i.e. file allocated on 2.x server)
instead of an upgraded Lustre 1.x IGIF FID (would have lmm_seq 
0x1).

This is different than the problem that I was investigating in LU-4293.

Are you running this as root, or a regular user?  Presumably you have
write permission on the file?


lmm_stripe_count:   1
lmm_stripe_size:1048576
lmm_pattern:1
lmm_layout_gen: 0
lmm_stripe_offset:  1
 obdidx   objid   objid   group
  1 1347728   0x1490900


Thanks,
Pete



On 12/19/2013 12:07 AM, Dilger, Andreas wrote:
 If you have only run with Lustre 2.5, can you please run lfs getstripe
-v /path/to/file to print out the FID and other layout information for
the file.

 Cheers, Andreas

 On Dec 17, 2013, at 19:22, Peter Mistich
peter.mist...@rackspace.com wrote:

 Thank Amdreas, I will try the new lfs_migrate and let you know.

 However I have only run lustre 2.5

 Thanks,
 Pete

 On 12/17/2013 07:49 PM, Dilger, Andreas wrote:
 I saw this same problem on my system - it happens when trying to
migrate a file created with Lustre 1.8.

 See https://jira.hpdd.intel.com/browse/LU-4293 for details.

 I have an updated version of lfs_migrate that works around this
problem that I should push to Gerrit. The patch will be linked to the
above bug when ready.

 Cheers, Andreas

 On Dec 17, 2013, at 18:43, Peter Mistich
peter.mist...@rackspace.commailto:peter.mist...@rackspace.com
wrote:

 Hello,
 today I added a ost and trying to balance them and when I run
 lfs_migrate I get cannot swap layouts between filename and a
volatile
 file (Operation not permitted)

 I am running lustre-2.5.52

 any help would be great.

 Thanks,
 Pete
 ___
 Lustre-discuss mailing list
 
Lustre-discuss@lists.lustre.orgmailto:Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss



Cheers, Andreas
-- 
Andreas Dilger

Lustre Software Architect
Intel High Performance Data Division


___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] lfs_migrate errors

[Peter Mistich]

hope this helps

/opt/zenoss/perf//Daemons/bb01lr02s06.dfw1/zenperfsnmp-thresholds.pickle: 
cannot 
swap layouts between 
/opt/zenoss/perf//Daemons/bb01lr02s06.dfw1/zenperfsnmp-thresholds.pickle 
and a volatile file (Operation not permitted)
error: migrate: migrate stripe file 
'/opt/zenoss/perf/Daemons/bb01lr02s06.dfw1/zenperfsnmp-thresholds.pickle' failed

lfs getstripe -v 
/opt/zenoss/perf/Daemons/bb01lr02s06.dfw1/zenperfsnmp-thresholds.pickle
/opt/zenoss/perf/Daemons/bb01lr02s06.dfw1/zenperfsnmp-thresholds.pickle
lmm_magic:  0x0BD10BD0
lmm_seq:0x20423
lmm_object_id:  0x324d
lmm_stripe_count:   1
lmm_stripe_size:1048576
lmm_pattern:1
lmm_layout_gen: 0
lmm_stripe_offset:  1
 obdidx   objid   objid   group
  1 1347728   0x1490900


Thanks,
Pete



On 12/19/2013 12:07 AM, Dilger, Andreas wrote:
 If you have only run with Lustre 2.5, can you please run lfs getstripe -v 
 /path/to/file to print out the FID and other layout information for the file.

 Cheers, Andreas

 On Dec 17, 2013, at 19:22, Peter Mistich peter.mist...@rackspace.com 
 wrote:

 Thank Amdreas, I will try the new lfs_migrate and let you know.

 However I have only run lustre 2.5

 Thanks,
 Pete

 On 12/17/2013 07:49 PM, Dilger, Andreas wrote:
 I saw this same problem on my system - it happens when trying to migrate a 
 file created with Lustre 1.8.

 See https://jira.hpdd.intel.com/browse/LU-4293 for details.

 I have an updated version of lfs_migrate that works around this problem 
 that I should push to Gerrit. The patch will be linked to the above bug 
 when ready.

 Cheers, Andreas

 On Dec 17, 2013, at 18:43, Peter Mistich 
 peter.mist...@rackspace.commailto:peter.mist...@rackspace.com wrote:

 Hello,
 today I added a ost and trying to balance them and when I run
 lfs_migrate I get cannot swap layouts between filename and a volatile
 file (Operation not permitted)

 I am running lustre-2.5.52

 any help would be great.

 Thanks,
 Pete
 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.orgmailto:Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] lfs_migrate errors

[Dilger, Andreas]

On 2013/12/17 7:29 PM, Peter Mistich peter.mist...@rackspace.com wrote:
Could you send me the updated version of lfs_migrate.

This is a no production system I could test it out.

You can get it from the patch in Gerrit: http://review.whamcloud.com/8616
via the gitweb link on that page:

http://git.whamcloud.com/?p=fs/lustre-release.git;a=blob;f=lustre/scripts/l
fs_migrate;h=e9c8aaeddec4a9f7efac364ad6740accb4cb3293;hb=a99a1172147b33b5fa
118779399267bc6e4490bd

On 12/17/2013 07:49 PM, Dilger, Andreas wrote:
 I saw this same problem on my system - it happens when trying to
migrate a file created with Lustre 1.8.

 See https://jira.hpdd.intel.com/browse/LU-4293 for details.

 I have an updated version of lfs_migrate that works around this problem
that I should push to Gerrit. The patch will be linked to the above bug
when ready.

 Cheers, Andreas

 On Dec 17, 2013, at 18:43, Peter Mistich
peter.mist...@rackspace.commailto:peter.mist...@rackspace.com wrote:

 Hello,
 today I added a ost and trying to balance them and when I run
 lfs_migrate I get cannot swap layouts between filename and a volatile
 file (Operation not permitted)

 I am running lustre-2.5.52

 any help would be great.

 Thanks,
 Pete
 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.orgmailto:Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss




Cheers, Andreas
-- 
Andreas Dilger

Lustre Software Architect
Intel High Performance Data Division


___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] lfs_migrate errors

[Dilger, Andreas]

If you have only run with Lustre 2.5, can you please run lfs getstripe -v 
/path/to/file to print out the FID and other layout information for the file. 

Cheers, Andreas

 On Dec 17, 2013, at 19:22, Peter Mistich peter.mist...@rackspace.com 
 wrote:
 
 Thank Amdreas, I will try the new lfs_migrate and let you know.
 
 However I have only run lustre 2.5
 
 Thanks,
 Pete
 
 On 12/17/2013 07:49 PM, Dilger, Andreas wrote:
 I saw this same problem on my system - it happens when trying to migrate a 
 file created with Lustre 1.8.
 
 See https://jira.hpdd.intel.com/browse/LU-4293 for details.
 
 I have an updated version of lfs_migrate that works around this problem that 
 I should push to Gerrit. The patch will be linked to the above bug when 
 ready.
 
 Cheers, Andreas
 
 On Dec 17, 2013, at 18:43, Peter Mistich 
 peter.mist...@rackspace.commailto:peter.mist...@rackspace.com wrote:
 
 Hello,
 today I added a ost and trying to balance them and when I run
 lfs_migrate I get cannot swap layouts between filename and a volatile
 file (Operation not permitted)
 
 I am running lustre-2.5.52
 
 any help would be great.
 
 Thanks,
 Pete
 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.orgmailto:Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss
 
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] lfs_migrate errors

[Dilger, Andreas]

I saw this same problem on my system - it happens when trying to migrate a file 
created with Lustre 1.8.

See https://jira.hpdd.intel.com/browse/LU-4293 for details.

I have an updated version of lfs_migrate that works around this problem that I 
should push to Gerrit. The patch will be linked to the above bug when ready.

Cheers, Andreas

On Dec 17, 2013, at 18:43, Peter Mistich 
peter.mist...@rackspace.commailto:peter.mist...@rackspace.com wrote:

Hello,
today I added a ost and trying to balance them and when I run
lfs_migrate I get cannot swap layouts between filename and a volatile
file (Operation not permitted)

I am running lustre-2.5.52

any help would be great.

Thanks,
Pete
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.orgmailto:Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] lfs_migrate errors

[Peter Mistich]

Thank Amdreas, I will try the new lfs_migrate and let you know.

However I have only run lustre 2.5

Thanks,
Pete

On 12/17/2013 07:49 PM, Dilger, Andreas wrote:
 I saw this same problem on my system - it happens when trying to migrate a 
 file created with Lustre 1.8.

 See https://jira.hpdd.intel.com/browse/LU-4293 for details.

 I have an updated version of lfs_migrate that works around this problem that 
 I should push to Gerrit. The patch will be linked to the above bug when ready.

 Cheers, Andreas

 On Dec 17, 2013, at 18:43, Peter Mistich 
 peter.mist...@rackspace.commailto:peter.mist...@rackspace.com wrote:

 Hello,
 today I added a ost and trying to balance them and when I run
 lfs_migrate I get cannot swap layouts between filename and a volatile
 file (Operation not permitted)

 I am running lustre-2.5.52

 any help would be great.

 Thanks,
 Pete
 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.orgmailto:Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] lfs_migrate errors

[Peter Mistich]

Andreas,

Could you send me the updated version of lfs_migrate.

This is a no production system I could test it out.

Thanks,
Pete

On 12/17/2013 07:49 PM, Dilger, Andreas wrote:
 I saw this same problem on my system - it happens when trying to migrate a 
 file created with Lustre 1.8.

 See https://jira.hpdd.intel.com/browse/LU-4293 for details.

 I have an updated version of lfs_migrate that works around this problem that 
 I should push to Gerrit. The patch will be linked to the above bug when ready.

 Cheers, Andreas

 On Dec 17, 2013, at 18:43, Peter Mistich 
 peter.mist...@rackspace.commailto:peter.mist...@rackspace.com wrote:

 Hello,
 today I added a ost and trying to balance them and when I run
 lfs_migrate I get cannot swap layouts between filename and a volatile
 file (Operation not permitted)

 I am running lustre-2.5.52

 any help would be great.

 Thanks,
 Pete
 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.orgmailto:Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] lfs_migrate errors

[Peter Mistich]

Andreas,

Could you send me the updated version of lfs_migrate.

This is a no production system I could test it out.

Thanks,
Pete

On 12/17/2013 07:49 PM, Dilger, Andreas wrote:
 I saw this same problem on my system - it happens when trying to migrate a 
 file created with Lustre 1.8.

 See https://jira.hpdd.intel.com/browse/LU-4293 for details.

 I have an updated version of lfs_migrate that works around this problem that 
 I should push to Gerrit. The patch will be linked to the above bug when ready.

 Cheers, Andreas

 On Dec 17, 2013, at 18:43, Peter Mistich 
 peter.mist...@rackspace.commailto:peter.mist...@rackspace.com wrote:

 Hello,
 today I added a ost and trying to balance them and when I run
 lfs_migrate I get cannot swap layouts between filename and a volatile
 file (Operation not permitted)

 I am running lustre-2.5.52

 any help would be great.

 Thanks,
 Pete
 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.orgmailto:Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] lfs_migrate question

[Dilger, Andreas]

On 2012-10-18, at 16:11, Jason Brooks 
brook...@ohsu.edumailto:brook...@ohsu.edu wrote:

I suffered an oss crash where my oss server had a cpu fault.  I have it running 
again, but I am trying to decommission it.  I am migrating the data off of it 
onto other ost's using the lfs find command with lfs_migrate.

It's been nearly 36 hours and about 2 terabytes have been moved.  This means I 
am about halfway.  Is this a decent rate?

Depends on how large your files are, and how fast the network is, but I 
wouldn't call it outstanding...

Here are the particulars, which basically are snags.  I know they affect 
things, I just am not certain to what degree:

  1.  I am running lfs_migrate on two systems, migrating different 
subdirectories of the same mount point.

This increases contention on the MDS, but two clients shouldn't be overloading 
the server.  Presumably you are only finding and migrating files which are 
striped over the affected server?

  1.  All systems are running using ip over infiniband.

IPoIB is far slower than native IB, both for data and metadata, but in the 
middle if migration is probably not the time to be messing with your network 
configuration.

  1.  None of my client-only systems have lfs or lfs_migrate.  I think this is 
because they are ubuntu and only the lustre kernel modules are installed.  Thus 
I can't run it there.

This is just a shell script, so you could have copied it from another mode.

  1.   Oh, and that also means that the lustre filesytem is mounted on the 
oss's too.

This is not an ideal situation, since the memory usage on the client is 
competing with the memory of the OSS.

  1.  lfs_migrate and lfs did not seem to operate correctly on the oss's that 
are 1.8.6.  Works ok on 1.8.8 though.

Can't really comment based on this limited information.

  1.  AND the two systems I am running lfs_migrate on are probably the very 
systems with free ost space on them.  In other words, file blocks are being 
written to the very systems that lfs_migrate is being run on and/or there is a 
lot of block write traffic between the two.


Lustre versions:
Mds/mgs: 1.8.6
5 of 7 OSS's: 1.8.6
2 of 7 oss's: 1.8.8

Clients: 1.8.6, ubuntu.


___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.orgmailto:Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] lfs_migrate for rebalancing

[Sebastien Piechurski]

Hi,

lfs_migrate will rebalance space usage across the OSTs of your filesystem by 
copying files and deleting the original ones on the most empty OSTs. It will 
not change stripe count of files. You have to migrate lots of files to see the 
new balance. This is why you don't see the usefulness with only one file.
lfs_migrate also accepts options to specify which OSTs you want to redistribute.

Sebastien.


From: lustre-discuss-boun...@lists.lustre.org 
[mailto:lustre-discuss-boun...@lists.lustre.org] On Behalf Of Andrus, Brian 
Contractor
Sent: samedi 16 juillet 2011 18:31
To: lustre-discuss@lists.lustre.org
Subject: [Lustre-discuss] lfs_migrate for rebalancing

All,

Am I incorrect in the way I read that lfs_migrate can be used to rebalance  a 
filesystem after adding OSTs?

Here is some output on a 22M file that was all on one OST and we now have 9:

[root@compute-3-1 ps2]# lfs getstripe PS01-071.xml
PS01-071.xml
lmm_stripe_count:   1
lmm_stripe_size:1048576
lmm_stripe_offset:  6
obdidx   objid  objidgroup
 6 4411098   0x434eda0

[root@compute-3-1 ps2]# lfs_migrate -y PS01-071.xml
PS01-071.xml: done
[root@compute-3-1 ps2]# lfs getstripe PS01-071.xml
PS01-071.xml
lmm_stripe_count:   1
lmm_stripe_size:1048576
lmm_stripe_offset:  4
obdidx   objid  objidgroup
 450856625  0x30802b10

[root@compute-3-1 ps2]# cp PS01-071.xml PS01-071.xml.tmp
[root@compute-3-1 ps2]# lfs getstripe PS01-071.xml.tmp
PS01-071.xml.tmp
lmm_stripe_count:   9
lmm_stripe_size:1048576
lmm_stripe_offset:  2
obdidx   objid  objidgroup
 273524719  0x461e5ef0
 5 4389362   0x42f9f20
 353566843  0x3315d7b0
 047139754  0x2cf4baa0
 6 4411100   0x434edc0
 156746451  0x361e1d30
 8 4416089   0x4362590
 7 4411127   0x434ef70
 450856626  0x30802b20

[root@compute-3-1 ps2]#

So it moved the file to another OST, but left the strip count as 1.
But when I cp'ed the file, it properly striped it across all the OSTs.

This doesn't seem all that useful for rebalancing.

Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss