Re: [gpfsug-discuss] policy ilm features?

[Jan-Frode Myklebust]

We just discussed this a bit internally, and I found "something* that might
help... There's a mmrestripefs --inode-criteria command that can be used to
identify files with these unknown-to-ILM flags set. Something like:

# echo illreplicated > criteria
# mmrestripefs gpfs01 -p --inode-criteria criteria -o result
Scanning file system metadata, phase 1 ...
 100 % complete on Thu Feb 18 15:30:10 2021
Scan completed successfully.
Scanning file system metadata, phase 2 ...
 100 % complete on Thu Feb 18 15:30:10 2021
Scan completed successfully.
Scanning file system metadata, phase 3 ...
Scan completed successfully.
Scanning file system metadata, phase 4 ...
 100 % complete on Thu Feb 18 15:30:10 2021
Scan completed successfully.
Scanning file system metadata, phase 5 ...
 100 % complete on Thu Feb 18 15:30:10 2021
Scan completed successfully.
Scanning user file metadata ...
 100.00 % complete on Thu Feb 18 15:30:14 2021  (835584 inodes
with total   4271 MB data processed)
Scan completed successfully.
Check file '/mnt/gpfs01/result' on scale-dev-01 for inodes with broken
disk addresses or failures.
# cat /mnt/gpfs01/result
This inode list was generated in the Parallel Inode Traverse on Thu
Feb 18 15:30:10 2021
INODE_NUMBER DUMMY_INFO SNAPSHOT_ID ISGLOBAL_SNAPSHOT
INDEPENDENT_FSETID MEMO(INODE_FLAGS FILE_TYPE [ERROR])
 90:00   1 0
   illreplicated unbalanced REGULAR_FILE

Unclear to me if "-p" is too much work for this task though, or maybe
it can be limited with --metadata-only or other options..


HTH




On Sat, Jan 20, 2018 at 12:10 AM Edward Wahl  wrote:

>
> This one has been on my list a long time so I figured I'd ask here first
> before
> I open an apar or request an enhancement (most likely).
>
>  Is there a way using the policy engine to determine the following?
>
> -metadata replication total/current
> -unbalanced file
>
> Looking to catch things like this that stand out on my filesystem without
> having to run several hundred million 'mmlsattr's.
>
> metadata replication: 1 max 2
> flags:unbalanced
>
> Ed
>
>
>
> --
>
> Ed Wahl
> Ohio Supercomputer Center
> 614-292-9302
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] policy ilm features?

[Frederick Stock]

Hello Ed.  Jordan contacted me about the question you are posing so I am responding to your message.  Could you please provide clarification as to why the existence of the unbalanced flag is of a concern, or why you would want to know all the files that have this flag set?  The flag would be cleared once the file was rebalanced either through normal access or through the execution of the mmrestripefs/mmrestripefile commands.
Fred__Fred Stock | IBM Pittsburgh Lab | 720-430-8821sto...@us.ibm.com
 
 
- Original message -From: "Wahl, Edward" Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: gpfsug main discussion list Cc:Subject: [EXTERNAL] Re: [gpfsug-discuss] policy ilm features?Date: Tue, Feb 2, 2021 11:52 AM 
Replying to a 3 year old message I sent, hoping that in the last couple of years that Scale has added some ILM extensions into the policy engine that I have missed, or somehow didn't notice?
Just ran into a file with an 'unbalanced' flag and I REALLY don't want to have to mmlsattr everything. AGAIN. /facepalm  
 
IBM?  Bueller? Bueller?  
 
When everyone answers: "No", I'm guessing this needs to be a request for improvement/enhancement?  
 
Ed Wahl
Ohio Supercomputer Center
 
 
From: gpfsug-discuss-boun...@spectrumscale.org  on behalf of Edward Wahl Sent: Friday, February 2, 2018 3:23 PMTo: John Hearns Cc: gpfsug main discussion list Subject: Re: [gpfsug-discuss] policy ilm features?
 
Thanks John, this was the path I was HOPING to go down as I do similar thingsalready, but there appears to be no extended attribute in ILM for what I want.Data block replication flag exists in the ILM, but not MetaData, or balance.Yet these states ARE reported by mmlsattr, so there must be a flag somewhere.bad MD replication & balance example: mmlsattr -L /fs/scratch/sysp/ed/180days.polfile name:    /fs/scratch/sysp/ed/180days.polmetadata replication: 1 max 2data replication: 1 max 2flags:    illreplicated,unbalancedEncrypted:    yesFile next to it for comparison. note proper MD replication and balance. mmlsattr -L  /fs/scratch/sysp/ed/120days.polfile name:    /fs/scratch/sysp/ed/120days.polmetadata replication: 2 max 2data replication: 1 max 2flags:   Encrypted:    yesmisc_attributes flags from a policy run showing no difference in status:FJAEu -- /fs/scratch/sysp/ed/180days.polFJAEu -- /fs/scratch/sysp/ed/120days.polFile system has MD replication enabled, but not Data, so ALL files show "J" ilmflagmmlsfs scratch -mflag    value    description---  --- -m 2    Default number of metadata replicasmmlsfs scratch -rflag    value    description---  --- -r 1    Default number of data replicasI poked around a little trying to find out if perhaps using GetXattr wouldwork and show me what I wanted, it does not. All I sem to be able to get is theFile Encryption Key.I was hoping perhaps someone had found a cheaper way for this to work ratherthan hundreds of millions of 'mmlsattr' execs.  :-(On the plus side, I've only run across a few of these and all appear to befrom before we did the MD replication and re-striping.  On the minus, I have NOidea where they are, and they appears to be on both of our filesystems.  Soseveral hundred million files to check.EdOn Mon, 22 Jan 2018 08:29:42 +John Hearns  wrote:> Ed,> This is not a perfect answer. You need to look at policies for this. I have> been doing something similar recently.>> Something like:>> RULE 'list_file' EXTERNAL LIST 'all-files' EXEC> '/var/mmfs/etc/mmpolicyExec-list' RULE 'listall' list 'all-files'> SHOW( varchar(kb_allocated) || '  ' || varchar(file_size) || ' ' ||> varchar(misc_attributes) || ' ' || name || ' ' || fileset_name  ) WHERE> REGEX(misc_attributes,'[J]')>>> So this policy shows the kbytes allocates, file size, the miscellaneous> attributes, name and fileset name For all files with  miscellaneous> attributes of 'J'   which means 'Some data blocks might be ill replicated'>>>>> -Original Message-> From: gpfsug-discuss-boun...@spectrumscale.org> [mailto:gpfsug-discuss-boun...@spectrumscale.org] On Behalf Of Edward Wahl> Sent: Friday, January 19, 2018 10:38 PM To: gpfsug-discuss@spectrumscale.org> Subject: [gpfsug-discuss] policy ilm features?>>> This one has been on my list a long time so I figured I'd ask here first> before I open an apar or request an enhancement (most likely).>>  Is there a way using the policy engine to determine the following?>> -metadata replication total/current> -unbalanced file>> Looking to catc

Re: [gpfsug-discuss] policy ilm features?

[Wahl, Edward]

Replying to a 3 year old message I sent, hoping that in the last couple of 
years that Scale has added some ILM extensions into the policy engine that I 
have missed, or somehow didn't notice?
Just ran into a file with an 'unbalanced' flag and I REALLY don't want to have 
to mmlsattr everything. AGAIN. /facepalm

IBM?  Bueller? Bueller?

When everyone answers: "No", I'm guessing this needs to be a request for 
improvement/enhancement?

Ed Wahl
Ohio Supercomputer Center


From: gpfsug-discuss-boun...@spectrumscale.org 
 on behalf of Edward Wahl 

Sent: Friday, February 2, 2018 3:23 PM
To: John Hearns 
Cc: gpfsug main discussion list 
Subject: Re: [gpfsug-discuss] policy ilm features?


Thanks John, this was the path I was HOPING to go down as I do similar things
already, but there appears to be no extended attribute in ILM for what I want.
Data block replication flag exists in the ILM, but not MetaData, or balance.

Yet these states ARE reported by mmlsattr, so there must be a flag somewhere.


bad MD replication & balance example:

 mmlsattr -L /fs/scratch/sysp/ed/180days.pol
file name:/fs/scratch/sysp/ed/180days.pol
metadata replication: 1 max 2
data replication: 1 max 2
flags:illreplicated,unbalanced
Encrypted:yes

File next to it for comparison. note proper MD replication and balance.

 mmlsattr -L  /fs/scratch/sysp/ed/120days.pol
file name:/fs/scratch/sysp/ed/120days.pol
metadata replication: 2 max 2
data replication: 1 max 2
flags:
Encrypted:yes

misc_attributes flags from a policy run showing no difference in status:
FJAEu -- /fs/scratch/sysp/ed/180days.pol
FJAEu -- /fs/scratch/sysp/ed/120days.pol


File system has MD replication enabled, but not Data, so ALL files show "J" ilm
flag

mmlsfs scratch -m
flagvaluedescription
---  ---
 -m 2Default number of metadata replicas
mmlsfs scratch -r
flagvaluedescription
---  ---
 -r 1Default number of data replicas


I poked around a little trying to find out if perhaps using GetXattr would
work and show me what I wanted, it does not. All I sem to be able to get is the
File Encryption Key.


I was hoping perhaps someone had found a cheaper way for this to work rather
than hundreds of millions of 'mmlsattr' execs.  :-(

On the plus side, I've only run across a few of these and all appear to be
from before we did the MD replication and re-striping.  On the minus, I have NO
idea where they are, and they appears to be on both of our filesystems.  So
several hundred million files to check.

Ed


On Mon, 22 Jan 2018 08:29:42 +
John Hearns  wrote:

> Ed,
> This is not a perfect answer. You need to look at policies for this. I have
> been doing something similar recently.
>
> Something like:
>
> RULE 'list_file' EXTERNAL LIST 'all-files' EXEC
> '/var/mmfs/etc/mmpolicyExec-list' RULE 'listall' list 'all-files'
> SHOW( varchar(kb_allocated) || '  ' || varchar(file_size) || ' ' ||
> varchar(misc_attributes) || ' ' || name || ' ' || fileset_name  ) WHERE
> REGEX(misc_attributes,'[J]')
>
>
> So this policy shows the kbytes allocates, file size, the miscellaneous
> attributes, name and fileset name For all files with  miscellaneous
> attributes of 'J'   which means 'Some data blocks might be ill replicated'
>
>
>
>
> -Original Message-
> From: gpfsug-discuss-boun...@spectrumscale.org
> [mailto:gpfsug-discuss-boun...@spectrumscale.org] On Behalf Of Edward Wahl
> Sent: Friday, January 19, 2018 10:38 PM To: gpfsug-discuss@spectrumscale.org
> Subject: [gpfsug-discuss] policy ilm features?
>
>
> This one has been on my list a long time so I figured I'd ask here first
> before I open an apar or request an enhancement (most likely).
>
>  Is there a way using the policy engine to determine the following?
>
> -metadata replication total/current
> -unbalanced file
>
> Looking to catch things like this that stand out on my filesystem without
> having to run several hundred million 'mmlsattr's.
>
> metadata replication: 1 max 2
> flags:unbalanced
>
> Ed
>
>
>
> --
>
> Ed Wahl
> Ohio Supercomputer Center
> 614-292-9302
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss=01%7C01%7Cjohn.hearns%40asml.com%7C056e34c5a8df4d8f10fd08d55f91e73c%7Caf73baa8f5944eb2a39d93e96cad61fc%7C1=dnt7vV4TCd68l7fSJnY35eyNM%2B8p

Re: [gpfsug-discuss] policy ilm features?

[Edward Wahl]

Thanks John, this was the path I was HOPING to go down as I do similar things
already, but there appears to be no extended attribute in ILM for what I want.
Data block replication flag exists in the ILM, but not MetaData, or balance.

Yet these states ARE reported by mmlsattr, so there must be a flag somewhere.


bad MD replication & balance example:

 mmlsattr -L /fs/scratch/sysp/ed/180days.pol 
file name:/fs/scratch/sysp/ed/180days.pol
metadata replication: 1 max 2
data replication: 1 max 2
flags:illreplicated,unbalanced
Encrypted:yes

File next to it for comparison. note proper MD replication and balance.

 mmlsattr -L  /fs/scratch/sysp/ed/120days.pol 
file name:/fs/scratch/sysp/ed/120days.pol
metadata replication: 2 max 2
data replication: 1 max 2
flags:
Encrypted:yes

misc_attributes flags from a policy run showing no difference in status:
FJAEu -- /fs/scratch/sysp/ed/180days.pol
FJAEu -- /fs/scratch/sysp/ed/120days.pol


File system has MD replication enabled, but not Data, so ALL files show "J" ilm
flag

mmlsfs scratch -m
flagvaluedescription
---  ---
 -m 2Default number of metadata replicas
mmlsfs scratch -r
flagvaluedescription
---  ---
 -r 1Default number of data replicas


I poked around a little trying to find out if perhaps using GetXattr would
work and show me what I wanted, it does not. All I sem to be able to get is the
File Encryption Key.


I was hoping perhaps someone had found a cheaper way for this to work rather
than hundreds of millions of 'mmlsattr' execs.  :-(

On the plus side, I've only run across a few of these and all appear to be
from before we did the MD replication and re-striping.  On the minus, I have NO
idea where they are, and they appears to be on both of our filesystems.  So
several hundred million files to check. 

Ed


On Mon, 22 Jan 2018 08:29:42 +
John Hearns  wrote:

> Ed,
> This is not a perfect answer. You need to look at policies for this. I have
> been doing something similar recently.
> 
> Something like:
> 
> RULE 'list_file' EXTERNAL LIST 'all-files' EXEC
> '/var/mmfs/etc/mmpolicyExec-list' RULE 'listall' list 'all-files'
> SHOW( varchar(kb_allocated) || '  ' || varchar(file_size) || ' ' ||
> varchar(misc_attributes) || ' ' || name || ' ' || fileset_name  ) WHERE
> REGEX(misc_attributes,'[J]')
> 
> 
> So this policy shows the kbytes allocates, file size, the miscellaneous
> attributes, name and fileset name For all files with  miscellaneous
> attributes of 'J'   which means 'Some data blocks might be ill replicated'
> 
> 
> 
> 
> -Original Message-
> From: gpfsug-discuss-boun...@spectrumscale.org
> [mailto:gpfsug-discuss-boun...@spectrumscale.org] On Behalf Of Edward Wahl
> Sent: Friday, January 19, 2018 10:38 PM To: gpfsug-discuss@spectrumscale.org
> Subject: [gpfsug-discuss] policy ilm features?
> 
> 
> This one has been on my list a long time so I figured I'd ask here first
> before I open an apar or request an enhancement (most likely).
> 
>  Is there a way using the policy engine to determine the following?
> 
> -metadata replication total/current
> -unbalanced file
> 
> Looking to catch things like this that stand out on my filesystem without
> having to run several hundred million 'mmlsattr's.
> 
> metadata replication: 1 max 2
> flags:unbalanced
> 
> Ed
> 
> 
> 
> --
> 
> Ed Wahl
> Ohio Supercomputer Center
> 614-292-9302
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss=01%7C01%7Cjohn.hearns%40asml.com%7C056e34c5a8df4d8f10fd08d55f91e73c%7Caf73baa8f5944eb2a39d93e96cad61fc%7C1=dnt7vV4TCd68l7fSJnY35eyNM%2B8pNrZElImSZeZbit8%3D=0
> -- The information contained in this communication and any attachments is
> confidential and may be privileged, and is for the sole use of the intended
> recipient(s). Any unauthorized review, use, disclosure or distribution is
> prohibited. Unless explicitly stated otherwise in the body of this
> communication or the attachment thereto (if any), the information is provided
> on an AS-IS basis without any express or implied warranties or liabilities.
> To the extent you are relying on this information, you are doing so at your
> own risk. If you are not the intended recipient, please notify the sender
> immediately by replying to this message and destroy all copies of this
> message and any attachments. Neither the sender nor the company/group of
> companies he or she represents shall be liable for the proper and 

Re: [gpfsug-discuss] policy ilm features?

[John Hearns]

Ed,
This is not a perfect answer. You need to look at policies for this. I have 
been doing something similar recently.

Something like:

RULE 'list_file' EXTERNAL LIST 'all-files' EXEC 
'/var/mmfs/etc/mmpolicyExec-list'
RULE 'listall' list 'all-files'
SHOW( varchar(kb_allocated) || '  ' || varchar(file_size) || ' ' || 
varchar(misc_attributes) || ' ' || name || ' ' || fileset_name  )
WHERE REGEX(misc_attributes,'[J]')


So this policy shows the kbytes allocates, file size, the miscellaneous 
attributes, name and fileset name
For all files with  miscellaneous attributes of 'J'   which means 'Some data 
blocks might be ill replicated'




-Original Message-
From: gpfsug-discuss-boun...@spectrumscale.org 
[mailto:gpfsug-discuss-boun...@spectrumscale.org] On Behalf Of Edward Wahl
Sent: Friday, January 19, 2018 10:38 PM
To: gpfsug-discuss@spectrumscale.org
Subject: [gpfsug-discuss] policy ilm features?


This one has been on my list a long time so I figured I'd ask here first before 
I open an apar or request an enhancement (most likely).

 Is there a way using the policy engine to determine the following?

-metadata replication total/current
-unbalanced file

Looking to catch things like this that stand out on my filesystem without 
having to run several hundred million 'mmlsattr's.

metadata replication: 1 max 2
flags:unbalanced

Ed



--

Ed Wahl
Ohio Supercomputer Center
614-292-9302
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss=01%7C01%7Cjohn.hearns%40asml.com%7C056e34c5a8df4d8f10fd08d55f91e73c%7Caf73baa8f5944eb2a39d93e96cad61fc%7C1=dnt7vV4TCd68l7fSJnY35eyNM%2B8pNrZElImSZeZbit8%3D=0
-- The information contained in this communication and any attachments is 
confidential and may be privileged, and is for the sole use of the intended 
recipient(s). Any unauthorized review, use, disclosure or distribution is 
prohibited. Unless explicitly stated otherwise in the body of this 
communication or the attachment thereto (if any), the information is provided 
on an AS-IS basis without any express or implied warranties or liabilities. To 
the extent you are relying on this information, you are doing so at your own 
risk. If you are not the intended recipient, please notify the sender 
immediately by replying to this message and destroy all copies of this message 
and any attachments. Neither the sender nor the company/group of companies he 
or she represents shall be liable for the proper and complete transmission of 
the information contained in this communication, or for any delay in its 
receipt.
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] policy ilm features?

[Marc A Kaplan]

Hint. RTFineManual, particularly the Admin guide, and look for 
MISC_ATTRIBUTES, 

Regarding metadata replication, one first has to ask, which metadata is 
replicated when and what if anything the mmchattr -m does or changes...

https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General+Parallel+File+System+(GPFS)/page/Configuring+GPFS+for+Reliability


--marc K.

___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss