Re: [lustre-discuss] Full OST, any way of avoiding it without hanging?

2019-01-07 Thread Harr, Cameron 
In my brief attempts to use lfs migrate, I found performance pretty slow 
(it's serial). I also got some ASSERTs which should be fixed in 2.10 per 
LU-8807; note that I was using 2.8. On a more trivial level, I found the 
-v|--verbose option to the command doesn't work.


On 01/07/2019 12:26 PM, Mohr Jr, Richard Frank (Rick Mohr) wrote:
>> On Jan 7, 2019, at 2:09 PM, Jason Williams  wrote:
>>
>> One last question, How safe is lfs_migrate?  The man page on the 
>> installation says it's UNSAFE for possibly in-use files.  The lustre manual 
>> doesn't have the same warning and says something about it being a bit more 
>> integrated with the MDS.
>>
>> http://doc.lustre.org/lustre_manual.xhtml#dbdoclet.lfs_migrate
> I have not used lfs_migrate, so I don’t have any direct experience.  From 
> what I have read online, the lfs_migrate script used to just do a copy/rename 
> of the file.  But now it looks like the “lfs” command has a built-in “lfs 
> migrate” subcommand which supports a “—block” option to prevent file access 
> while the migration is happening.  So it might be safe to use.
>
> Perhaps someone else on the list with more experience using this command 
> could chime in.
>
> --
> Rick Mohr
> Senior HPC System Administrator
> National Institute for Computational Sciences
> http://www.nics.tennessee.edu
>
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Full OST, any way of avoiding it without hanging?

2019-01-07 Thread Mohr Jr, Richard Frank (Rick Mohr) 


> On Jan 7, 2019, at 2:09 PM, Jason Williams  wrote:
> 
> One last question, How safe is lfs_migrate?  The man page on the installation 
> says it's UNSAFE for possibly in-use files.  The lustre manual doesn't have 
> the same warning and says something about it being a bit more integrated with 
> the MDS.
> 
> http://doc.lustre.org/lustre_manual.xhtml#dbdoclet.lfs_migrate

I have not used lfs_migrate, so I don’t have any direct experience.  From what 
I have read online, the lfs_migrate script used to just do a copy/rename of the 
file.  But now it looks like the “lfs” command has a built-in “lfs migrate” 
subcommand which supports a “—block” option to prevent file access while the 
migration is happening.  So it might be safe to use.

Perhaps someone else on the list with more experience using this command could 
chime in.

--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Full OST, any way of avoiding it without hanging?

2019-01-07 Thread Jason Williams 
Thank you again Rick.

One last question, How safe is lfs_migrate?  The man page on the installation 
says it's UNSAFE for possibly in-use files.  The lustre manual doesn't have the 
same warning and says something about it being a bit more integrated with the 
MDS.


http://doc.lustre.org/lustre_manual.xhtml#dbdoclet.lfs_migrate


How safe do you think this would be to run on some files on the OST with it 
disabled on the MDS and active jobs running on the cluster?  I could do this by 
group, possibly to mitigate concerns of open files, if need be.


--
Jason Williams
Assistant Director
Systems and Data Center Operations.
Maryland Advanced Research Computing Center (MARCC)
Johns Hopkins University
jas...@jhu.edu<mailto:jas...@jhu.edu>




From: Mohr Jr, Richard Frank (Rick Mohr) 
Sent: Monday, January 7, 2019 1:07 PM
To: Jason Williams
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] Full OST, any way of avoiding it without hanging?



> On Jan 7, 2019, at 12:53 PM, Jason Williams  wrote:
>
> As I have gone through the testing, I think you may be right.  I think I 
> disabled the OST in a slightly different way and that caused issues.
>
> Do you happen to know where I could find out a bit more about what the "lctl 
> set_param osp..max_create_count=0” command would do?

The Lustre manual has a section on removing MDTs/OSTs:

http://doc.lustre.org/lustre_manual.xhtml#dbdoclet.deactivating_mdt_ost

--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Full OST, any way of avoiding it without hanging?

2019-01-07 Thread Mohr Jr, Richard Frank (Rick Mohr) 


> On Jan 7, 2019, at 12:53 PM, Jason Williams  wrote:
> 
> As I have gone through the testing, I think you may be right.  I think I 
> disabled the OST in a slightly different way and that caused issues.
> 
> Do you happen to know where I could find out a bit more about what the "lctl 
> set_param osp..max_create_count=0” command would do?

The Lustre manual has a section on removing MDTs/OSTs:

http://doc.lustre.org/lustre_manual.xhtml#dbdoclet.deactivating_mdt_ost

--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Full OST, any way of avoiding it without hanging?

2019-01-07 Thread Jason Williams 
As I have gone through the testing, I think you may be right.  I think I 
disabled the OST in a slightly different way and that caused issues.


Do you happen to know where I could find out a bit more about what the "lctl 
set_param osp..max_create_count=0” command would do?




--
Jason Williams
Assistant Director
Systems and Data Center Operations.
Maryland Advanced Research Computing Center (MARCC)
Johns Hopkins University
jas...@jhu.edu<mailto:jas...@jhu.edu>




From: Mohr Jr, Richard Frank (Rick Mohr) 
Sent: Monday, January 7, 2019 12:35 PM
To: Jason Williams
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] Full OST, any way of avoiding it without hanging?



Jason,

The results you described sound like the correct behavior when you deactivate 
an OST on the MDS.  When you run “lctl —device  deactivate”, you are 
essentially telling the MDS to ignore that OST when it assigns stripes to a new 
file.  The OST will still be visible to all clients and the MDS, which allows 
the clients to keep reading files from that OST and allows you to delete files 
from the OST.  The only down side is that any file that already exists on that 
OST can still be written to.  Deactivating an OST is intended to stop the flow 
of new data to that OST while you work on removing some of the existing data, 
but it doesn’t actually make the OST read-only.  I think you can get the same 
effect from Lustre 2.9 (or newer) by using "lctl set_param 
osp..max_create_count=0”.

I suspect that what you originally did was to deactivate the OST using 
something like "lctl conf_param .osc.active=0”.  This will notify all 
Lustre clients to deactivate the OST, which I believe causes the hangs you were 
seeing when any client tries to remove or stat a file on that OST.

--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu


> On Jan 7, 2019, at 11:56 AM, Jason Williams  wrote:
>
> Sorry for the spam, but here are a few more interesting results:
>
> If I create a file that stripes only on the full OST, and then disable the 
> OST, I get the following:
>
>• I can over write the file within it's original size and it takes up 
> space on the "disabled" OST.
>• I can zero the file.
>• I can write more data to the file than it originally had, ie. the 
> original file before disabling the OST was 1G, I can overwrite the file with 
> > 1G with the OST disabled.
>• If I create a new file asking for that OST with the OST disabled, I 
> get a different OST
>
>
> # 4 and #2 are the only expected behavior.  I'm not sure what the behavior 
> should be in the case of #1 and #3.
>
>
> --
> Jason Williams
> Assistant Director
> Systems and Data Center Operations.
> Maryland Advanced Research Computing Center (MARCC)
> Johns Hopkins University
> jas...@jhu.edu
>
>
> From: lustre-discuss  on behalf of 
> Jason Williams 
> Sent: Monday, January 7, 2019 11:47:09 AM
> To: Mohr Jr, Richard Frank (Rick Mohr)
> Cc: lustre-discuss@lists.lustre.org
> Subject: Re: [lustre-discuss] Full OST, any way of avoiding it without 
> hanging?
>
> So I found this: http://wiki.lustre.org/Handling_Full_OSTs which is what I 
> thought I had followed before but ran into hang issues.  I did some quick 
> testing with this and found that:
>
> 1. if I deactivate the OST in the MDS, no new files appear to be created on 
> that OST (expected behavior) and no hangs.
> 2. If I first create a file on the OST with it activated, then deactivate the 
> OST, and OVERWRITE a file what was spanned on that OST, the indexes stay the 
> same and the file successfully overwrites (the file spanned 4 OSTs, so 
> perhaps a little more testing with a single OST in the index is necessary)
> 3. Deactivating the OST shows it as inactive in the MDS but UP in the Client. 
> (not expected.)
> 4. I am able to delete a file that spans that OST with the OST deactivated, 
> no hang.
>
> I think the only thing here that concerns me a bit is #2.
>
> --
> Jason Williams
> Assistant Director
> Systems and Data Center Operations.
> Maryland Advanced Research Computing Center (MARCC)
> Johns Hopkins University
> jas...@jhu.edu
>
>
> From: lustre-discuss  on behalf of 
> Jason Williams 
> Sent: Sunday, January 6, 2019 5:22:16 PM
> To: Mohr Jr, Richard Frank (Rick Mohr)
> Cc: lustre-discuss@lists.lustre.org
> Subject: Re: [lustre-discuss] Full OST, any way of avoiding it without 
> hanging?
>
> Hi Rick,
> I thought what I had done was disable it on the MDS, but perhaps I was 
> following the wrong instructions. Do you know where the best instructions for 
> what you are describing can be found? I would be willing

Re: [lustre-discuss] Full OST, any way of avoiding it without hanging?

2019-01-07 Thread Mohr Jr, Richard Frank (Rick Mohr) 


Jason,

The results you described sound like the correct behavior when you deactivate 
an OST on the MDS.  When you run “lctl —device  deactivate”, you are 
essentially telling the MDS to ignore that OST when it assigns stripes to a new 
file.  The OST will still be visible to all clients and the MDS, which allows 
the clients to keep reading files from that OST and allows you to delete files 
from the OST.  The only down side is that any file that already exists on that 
OST can still be written to.  Deactivating an OST is intended to stop the flow 
of new data to that OST while you work on removing some of the existing data, 
but it doesn’t actually make the OST read-only.  I think you can get the same 
effect from Lustre 2.9 (or newer) by using "lctl set_param 
osp..max_create_count=0”.

I suspect that what you originally did was to deactivate the OST using 
something like "lctl conf_param .osc.active=0”.  This will notify all 
Lustre clients to deactivate the OST, which I believe causes the hangs you were 
seeing when any client tries to remove or stat a file on that OST.

--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu


> On Jan 7, 2019, at 11:56 AM, Jason Williams  wrote:
> 
> Sorry for the spam, but here are a few more interesting results:
> 
> If I create a file that stripes only on the full OST, and then disable the 
> OST, I get the following:
> 
>   • I can over write the file within it's original size and it takes up 
> space on the "disabled" OST.
>   • I can zero the file.
>   • I can write more data to the file than it originally had, ie. the 
> original file before disabling the OST was 1G, I can overwrite the file with 
> > 1G with the OST disabled.
>   • If I create a new file asking for that OST with the OST disabled, I 
> get a different OST 
> 
> 
> # 4 and #2 are the only expected behavior.  I'm not sure what the behavior 
> should be in the case of #1 and #3.
> 
> 
> --
> Jason Williams
> Assistant Director
> Systems and Data Center Operations.
> Maryland Advanced Research Computing Center (MARCC)
> Johns Hopkins University
> jas...@jhu.edu
> 
> 
> From: lustre-discuss  on behalf of 
> Jason Williams 
> Sent: Monday, January 7, 2019 11:47:09 AM
> To: Mohr Jr, Richard Frank (Rick Mohr)
> Cc: lustre-discuss@lists.lustre.org
> Subject: Re: [lustre-discuss] Full OST, any way of avoiding it without 
> hanging?
>  
> So I found this: http://wiki.lustre.org/Handling_Full_OSTs which is what I 
> thought I had followed before but ran into hang issues.  I did some quick 
> testing with this and found that:
> 
> 1. if I deactivate the OST in the MDS, no new files appear to be created on 
> that OST (expected behavior) and no hangs.
> 2. If I first create a file on the OST with it activated, then deactivate the 
> OST, and OVERWRITE a file what was spanned on that OST, the indexes stay the 
> same and the file successfully overwrites (the file spanned 4 OSTs, so 
> perhaps a little more testing with a single OST in the index is necessary) 
> 3. Deactivating the OST shows it as inactive in the MDS but UP in the Client. 
> (not expected.)
> 4. I am able to delete a file that spans that OST with the OST deactivated, 
> no hang.
> 
> I think the only thing here that concerns me a bit is #2.
> 
> --
> Jason Williams
> Assistant Director
> Systems and Data Center Operations.
> Maryland Advanced Research Computing Center (MARCC)
> Johns Hopkins University
> jas...@jhu.edu
> 
> 
> From: lustre-discuss  on behalf of 
> Jason Williams 
> Sent: Sunday, January 6, 2019 5:22:16 PM
> To: Mohr Jr, Richard Frank (Rick Mohr)
> Cc: lustre-discuss@lists.lustre.org
> Subject: Re: [lustre-discuss] Full OST, any way of avoiding it without 
> hanging?
>  
> Hi Rick, 
> I thought what I had done was disable it on the MDS, but perhaps I was 
> following the wrong instructions. Do you know where the best instructions for 
> what you are describing can be found? I would be willing to try again.
> 
> —
> Sent you tersely from my phone
> Jason Williams
> From: Mohr Jr, Richard Frank (Rick Mohr) 
> Sent: Sunday, January 6, 2019 4:56 PM
> To: Jason Williams
> Cc: lustre-discuss@lists.lustre.org
> Subject: Re: [lustre-discuss] Full OST, any way of avoiding it without 
> hanging?
>  
> 
> 
> > On Jan 5, 2019, at 9:49 PM, Jason Williams  wrote:
> > 
> > I have looked around the internet and found you can disable an OST, but 
> > when I have tried that, any writes (including deletes) to the OST hang the 
> > clients indefinitely. Does anyone know a way to make an OST basically 
> > "read-only" with the ex

Re: [lustre-discuss] Full OST, any way of avoiding it without hanging?

2019-01-07 Thread Jason Williams 
Sorry for the spam, but here are a few more interesting results:


If I create a file that stripes only on the full OST, and then disable the OST, 
I get the following:


  1.  I can over write the file within it's original size and it takes up space 
on the "disabled" OST.
  2.  I can zero the file.
  3.  I can write more data to the file than it originally had, ie. the 
original file before disabling the OST was 1G, I can overwrite the file with > 
1G with the OST disabled.
  4.  If I create a new file asking for that OST with the OST disabled, I get a 
different OST


# 4 and #2 are the only expected behavior.  I'm not sure what the behavior 
should be in the case of #1 and #3.



--
Jason Williams
Assistant Director
Systems and Data Center Operations.
Maryland Advanced Research Computing Center (MARCC)
Johns Hopkins University
jas...@jhu.edu<mailto:jas...@jhu.edu>



From: lustre-discuss  on behalf of 
Jason Williams 
Sent: Monday, January 7, 2019 11:47:09 AM
To: Mohr Jr, Richard Frank (Rick Mohr)
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] Full OST, any way of avoiding it without hanging?


So I found this: http://wiki.lustre.org/Handling_Full_OSTs which is what I 
thought I had followed before but ran into hang issues.  I did some quick 
testing with this and found that:


1. if I deactivate the OST in the MDS, no new files appear to be created on 
that OST (expected behavior) and no hangs.

2. If I first create a file on the OST with it activated, then deactivate the 
OST, and OVERWRITE a file what was spanned on that OST, the indexes stay the 
same and the file successfully overwrites (the file spanned 4 OSTs, so perhaps 
a little more testing with a single OST in the index is necessary)

3. Deactivating the OST shows it as inactive in the MDS but UP in the Client. 
(not expected.)

4. I am able to delete a file that spans that OST with the OST deactivated, no 
hang.

I think the only thing here that concerns me a bit is #2.


--
Jason Williams
Assistant Director
Systems and Data Center Operations.
Maryland Advanced Research Computing Center (MARCC)
Johns Hopkins University
jas...@jhu.edu<mailto:jas...@jhu.edu>



From: lustre-discuss  on behalf of 
Jason Williams 
Sent: Sunday, January 6, 2019 5:22:16 PM
To: Mohr Jr, Richard Frank (Rick Mohr)
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] Full OST, any way of avoiding it without hanging?

Hi Rick,
I thought what I had done was disable it on the MDS, but perhaps I was 
following the wrong instructions. Do you know where the best instructions for 
what you are describing can be found? I would be willing to try again.

—
Sent you tersely from my phone
Jason Williams

From: Mohr Jr, Richard Frank (Rick Mohr) 
Sent: Sunday, January 6, 2019 4:56 PM
To: Jason Williams
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] Full OST, any way of avoiding it without hanging?



> On Jan 5, 2019, at 9:49 PM, Jason Williams  wrote:
>
> I have looked around the internet and found you can disable an OST, but when 
> I have tried that, any writes (including deletes) to the OST hang the clients 
> indefinitely. Does anyone know a way to make an OST basically "read-only" 
> with the exception of deletes so we can work to clear out the OST?

What command did you use to disable the OST?

There is a way to disable the OST on all the clients, but there is also a way 
to deactivate it on the MDS. The latter method should prevent the MDS from 
allocating any new files to the OST, but still allow clients to read and delete 
files on that OST.

--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Full OST, any way of avoiding it without hanging?

2019-01-07 Thread Jason Williams 
So I found this: http://wiki.lustre.org/Handling_Full_OSTs which is what I 
thought I had followed before but ran into hang issues.  I did some quick 
testing with this and found that:


1. if I deactivate the OST in the MDS, no new files appear to be created on 
that OST (expected behavior) and no hangs.

2. If I first create a file on the OST with it activated, then deactivate the 
OST, and OVERWRITE a file what was spanned on that OST, the indexes stay the 
same and the file successfully overwrites (the file spanned 4 OSTs, so perhaps 
a little more testing with a single OST in the index is necessary)

3. Deactivating the OST shows it as inactive in the MDS but UP in the Client. 
(not expected.)

4. I am able to delete a file that spans that OST with the OST deactivated, no 
hang.

I think the only thing here that concerns me a bit is #2.


--
Jason Williams
Assistant Director
Systems and Data Center Operations.
Maryland Advanced Research Computing Center (MARCC)
Johns Hopkins University
jas...@jhu.edu<mailto:jas...@jhu.edu>



From: lustre-discuss  on behalf of 
Jason Williams 
Sent: Sunday, January 6, 2019 5:22:16 PM
To: Mohr Jr, Richard Frank (Rick Mohr)
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] Full OST, any way of avoiding it without hanging?

Hi Rick,
I thought what I had done was disable it on the MDS, but perhaps I was 
following the wrong instructions. Do you know where the best instructions for 
what you are describing can be found? I would be willing to try again.

—
Sent you tersely from my phone
Jason Williams

From: Mohr Jr, Richard Frank (Rick Mohr) 
Sent: Sunday, January 6, 2019 4:56 PM
To: Jason Williams
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] Full OST, any way of avoiding it without hanging?



> On Jan 5, 2019, at 9:49 PM, Jason Williams  wrote:
>
> I have looked around the internet and found you can disable an OST, but when 
> I have tried that, any writes (including deletes) to the OST hang the clients 
> indefinitely. Does anyone know a way to make an OST basically "read-only" 
> with the exception of deletes so we can work to clear out the OST?

What command did you use to disable the OST?

There is a way to disable the OST on all the clients, but there is also a way 
to deactivate it on the MDS. The latter method should prevent the MDS from 
allocating any new files to the OST, but still allow clients to read and delete 
files on that OST.

--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Full OST, any way of avoiding it without hanging?

2019-01-06 Thread Jason Williams 
Hi Rick,
I thought what I had done was disable it on the MDS, but perhaps I was 
following the wrong instructions. Do you know where the best instructions for 
what you are describing can be found? I would be willing to try again.

—
Sent you tersely from my phone
Jason Williams

From: Mohr Jr, Richard Frank (Rick Mohr) 
Sent: Sunday, January 6, 2019 4:56 PM
To: Jason Williams
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] Full OST, any way of avoiding it without hanging?



> On Jan 5, 2019, at 9:49 PM, Jason Williams  wrote:
>
> I have looked around the internet and found you can disable an OST, but when 
> I have tried that, any writes (including deletes) to the OST hang the clients 
> indefinitely. Does anyone know a way to make an OST basically "read-only" 
> with the exception of deletes so we can work to clear out the OST?

What command did you use to disable the OST?

There is a way to disable the OST on all the clients, but there is also a way 
to deactivate it on the MDS. The latter method should prevent the MDS from 
allocating any new files to the OST, but still allow clients to read and delete 
files on that OST.

--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Full OST, any way of avoiding it without hanging?

2019-01-06 Thread Mohr Jr, Richard Frank (Rick Mohr) 


> On Jan 5, 2019, at 9:49 PM, Jason Williams  wrote:
> 
> I have looked around the internet and found you can disable an OST, but when 
> I have tried that, any writes (including deletes) to the OST hang the clients 
> indefinitely.  Does anyone know a way to make an OST basically "read-only" 
> with the exception of deletes so we can work to clear out the OST?  

What command did you use to disable the OST?

There is a way to disable the OST on all the clients, but there is also a way 
to deactivate it on the MDS.  The latter method should prevent the MDS from 
allocating any new files to the OST, but still allow clients to read and delete 
files on that OST.

--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Full OST, any way of avoiding it without hanging?

2019-01-05 Thread Jeff Johnson 
Jason,

If there are files located on the full OST that you can delete then delete
them without trying to deactivate the OST. There is a process where you can
find files solely located on the full OST using lfs then deactivate the
full OST, copy the files to new files with some suffix (.new perhaps) so
they end up on other OSTs, delete the original files, rename the new files
to the old file names (thereby removing the suffix used previously) and
then reactivating the deactivated OST. You might be lucky enough to find a
few very large single stripe files located on that OST where moving the
file gets you easy gains.

You *should* be able to delete a file that resides on an inactive OST but
I’m not over a keyboard where I can do it before making the assertion it is
possible.

—Jeff


On Sat, Jan 5, 2019 at 18:49 Jason Williams  wrote:

> Hello,
>
>
> We have a lustre system (version 2.10.4) that has unfortunately fallen
> victim to a 100% full OST... Every time we clear some space on it, the
> system fills it right back up again.
>
>
> I have looked around the internet and found you can disable an OST, but
> when I have tried that, any writes (including deletes) to the OST hang the
> clients indefinitely.  Does anyone know a way to make an OST basically
> "read-only" with the exception of deletes so we can work to clear out the
> OST?  Or better yet, a way to "drain" or move files off an OST with a
> script (keeping in mind it might not be known if the files are in use at
> the time).  Or even a way to tell lustre "Hey don't write any new data
> here, but reading and removing data is OK."
>
>
>
>
> --
> Jason Williams
> Assistant Director
> Systems and Data Center Operations.
> Maryland Advanced Research Computing Center (MARCC)
> Johns Hopkins University
> jas...@jhu.edu
>
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
-- 
--
Jeff Johnson
Co-Founder
Aeon Computing

jeff.john...@aeoncomputing.com
www.aeoncomputing.com
t: 858-412-3810 x1001   f: 858-412-3845
m: 619-204-9061

4170 Morena Boulevard, Suite C - San Diego, CA 92117

High-Performance Computing / Lustre Filesystems / Scale-out Storage
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] Full OST, any way of avoiding it without hanging?

2019-01-05 Thread Jason Williams 
Hello,


We have a lustre system (version 2.10.4) that has unfortunately fallen victim 
to a 100% full OST... Every time we clear some space on it, the system fills it 
right back up again.


I have looked around the internet and found you can disable an OST, but when I 
have tried that, any writes (including deletes) to the OST hang the clients 
indefinitely.  Does anyone know a way to make an OST basically "read-only" with 
the exception of deletes so we can work to clear out the OST?  Or better yet, a 
way to "drain" or move files off an OST with a script (keeping in mind it might 
not be known if the files are in use at the time).  Or even a way to tell 
lustre "Hey don't write any new data here, but reading and removing data is OK."




--
Jason Williams
Assistant Director
Systems and Data Center Operations.
Maryland Advanced Research Computing Center (MARCC)
Johns Hopkins University
jas...@jhu.edu

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org