Re: [gpfsug-discuss] mmfind -ls and so forth

[Buterbaugh, Kevin L]

Hi Marc,

I test in production … just kidding.  But - not kidding - I did read the entire 
mmfind.README, compiled the binary as described therein, and read the output of 
“mmfind -h”.

But what I forgot was that when you run a bash shell script with “bash -x” it 
doesn’t show you the redirection you did to a file … and since the mmfind ran 
for ~5 days, including over a weekend, and including Monday which I took off 
from work to have our 16 1/2 year old Siberian Husky put to sleep, I simply 
forgot that in the script itself I had redirected the output to a file.  Stupid 
of me, I know, but unlike Delusional Donald, I’ll admit my stupid mistakes.

Thanks, and sorry.  I will try the mmfind as you suggested in your previous 
response the next time I need to run one to see if that significantly improves 
the performance…

Kevin

On Mar 7, 2018, at 1:15 PM, Marc A Kaplan 
mailto:makap...@us.ibm.com>> wrote:

As always when dealing with computers and potentially long running jobs, run a 
test on a handful of files first, so you can rapidly debug.
Did you read the mmfind.README ?

It mentions...that this sample utility "some user assembly required..."  ...

mmfindUtil_processOutputFile.c
  A utility to parse the "list file" produced by mmapplypolicy and to print
it in a find-compatible format
  mmfind invokes it once mmapplypolicy begins to populate the "list file"

mmfindUtil_processOutputFile.sampleMakefile
  copy to 'makefile', modify as needed, and run 'make' to compile 
mmfindUtil_processOutputFile.c
  This should produce a binary called mmfindUtil_processOutputFile
  mmfind will not be able to run until this utility has been compiled
  on the node from which you launch mmfind.

Works for me...
[root@n2 ilm]# ./mmfind /goo/zdbig -ls
2463649  256 drwxr-xr-x   2 root root   262144 Feb  9 11:41 /goo/zdbig
68044970 -rw-r--r--   1 root root0 Feb  9 11:41 
/goo/zdbig/xy

___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C7c170869f3294124be3608d5845fdecc%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636560469687764985&sdata=yNvpm34DY0AtEm2Y4OIMll5IW1v5kP3X3vHx3sQ%2B8Rs%3D&reserved=0

___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] mmfind -ls and so forth

[Marc A Kaplan]

(John Hearns, et. al.)

Some minor script hacking would be the easiest way add test(s) for other 
MISC_ATTRIBUTES

Notice mmfind concentrates on providing the most popular classic(POSIX) 
and Linux predicates, BUT also adds a few gpfs specific predicates
(mmfind --help show you these)  -ea -eaWithValue -gpfsImmut -gpfsAppOnly

Look at the implementation of  -gpfsImmut in  tr_findToPol.pl ...

sub tr_gpfsImmut{
  return "( /* -gpfsImmut */ MISC_ATTRIBUTES LIKE '%X%')";
}


So easy to extend this for any or all the others  True it's perl, but 
you don't have to be a perl expert to cut-paste-hack another predicate 
into the script.

Let us know how you make out with this...
Perhaps we shall add  a general predicate -gpfsMiscAttrLike '...'   to the 
next version...

-- Marc K of GPFS



From:   John Hearns 
To: gpfsug main discussion list 
Date:   03/08/2018 04:59 AM
Subject:Re: [gpfsug-discuss] mmfind -ls and so forth
Sent by:gpfsug-discuss-boun...@spectrumscale.org



On the subject of mmfind,
I would like to find files which have the misc attributes relevant to AFM.
For instance files which have the attribute ‘v’ The file is newly created, 
not yet copied to home
I can write a policy to do this, and I have a relevant policy written.
 
However I would like to do this using mmfind, which seems a nice general 
utility.
This syntax does not work:
 
mmfind /hpc -eaWithValue MISC_ATTRIBUTES===v
 
Before anyone says it, I am mixing up MISC_ATTRIBUTES and extended 
attributes!
My question really is – has anyone done this sort of serch using mmfind?
 
Thankyou
 
 
 
From: gpfsug-discuss-boun...@spectrumscale.org [
mailto:gpfsug-discuss-boun...@spectrumscale.org] On Behalf Of Marc A 
Kaplan
Sent: Wednesday, March 07, 2018 8:16 PM
To: gpfsug main discussion list 
Subject: Re: [gpfsug-discuss] mmfind -ls and so forth
 
As always when dealing with computers and potentially long running jobs, 
run a test on a handful of files first, so you can rapidly debug.
Did you read the mmfind.README ?

It mentions...that this sample utility "some user assembly required..." 
... 

mmfindUtil_processOutputFile.c
  A utility to parse the "list file" produced by mmapplypolicy and to 
print
it in a find-compatible format
  mmfind invokes it once mmapplypolicy begins to populate the "list file"

mmfindUtil_processOutputFile.sampleMakefile
  copy to 'makefile', modify as needed, and run 'make' to compile 
mmfindUtil_processOutputFile.c
  This should produce a binary called mmfindUtil_processOutputFile
  mmfind will not be able to run until this utility has been compiled
  on the node from which you launch mmfind.

Works for me...
[root@n2 ilm]# ./mmfind /goo/zdbig -ls
2463649  256 drwxr-xr-x   2 root root   262144 Feb  9 11:41 
/goo/zdbig
68044970 -rw-r--r--   1 root root0 Feb  9 11:41 
/goo/zdbig/xy
-- The information contained in this communication and any attachments is 
confidential and may be privileged, and is for the sole use of the 
intended recipient(s). Any unauthorized review, use, disclosure or 
distribution is prohibited. Unless explicitly stated otherwise in the body 
of this communication or the attachment thereto (if any), the information 
is provided on an AS-IS basis without any express or implied warranties or 
liabilities. To the extent you are relying on this information, you are 
doing so at your own risk. If you are not the intended recipient, please 
notify the sender immediately by replying to this message and destroy all 
copies of this message and any attachments. Neither the sender nor the 
company/group of companies he or she represents shall be liable for the 
proper and complete transmission of the information contained in this 
communication, or for any delay in its receipt. 
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=LDC-t-w-jkuH2fJZ1lME_JUjzABDz3y90ptTlYWM3rc&s=xrFd1LD5dWq9GogfeOGs9ZCtqoptErjmGfJzD3eXhz4&e=





___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] mmfind -ls and so forth

[John Hearns]

On the subject of mmfind,
I would like to find files which have the misc attributes relevant to AFM.
For instance files which have the attribute 'v' The file is newly created, not 
yet copied to home
I can write a policy to do this, and I have a relevant policy written.

However I would like to do this using mmfind, which seems a nice general 
utility.
This syntax does not work:

mmfind /hpc -eaWithValue MISC_ATTRIBUTES===v

Before anyone says it, I am mixing up MISC_ATTRIBUTES and extended attributes!
My question really is - has anyone done this sort of serch using mmfind?

Thankyou



From: gpfsug-discuss-boun...@spectrumscale.org 
[mailto:gpfsug-discuss-boun...@spectrumscale.org] On Behalf Of Marc A Kaplan
Sent: Wednesday, March 07, 2018 8:16 PM
To: gpfsug main discussion list 
Subject: Re: [gpfsug-discuss] mmfind -ls and so forth

As always when dealing with computers and potentially long running jobs, run a 
test on a handful of files first, so you can rapidly debug.
Did you read the mmfind.README ?

It mentions...that this sample utility "some user assembly required..."  ...

mmfindUtil_processOutputFile.c
  A utility to parse the "list file" produced by mmapplypolicy and to print
it in a find-compatible format
  mmfind invokes it once mmapplypolicy begins to populate the "list file"

mmfindUtil_processOutputFile.sampleMakefile
  copy to 'makefile', modify as needed, and run 'make' to compile 
mmfindUtil_processOutputFile.c
  This should produce a binary called mmfindUtil_processOutputFile
  mmfind will not be able to run until this utility has been compiled
  on the node from which you launch mmfind.

Works for me...
[root@n2 ilm]# ./mmfind /goo/zdbig -ls
2463649  256 drwxr-xr-x   2 root root   262144 Feb  9 11:41 /goo/zdbig
68044970 -rw-r--r--   1 root root0 Feb  9 11:41 
/goo/zdbig/xy
-- The information contained in this communication and any attachments is 
confidential and may be privileged, and is for the sole use of the intended 
recipient(s). Any unauthorized review, use, disclosure or distribution is 
prohibited. Unless explicitly stated otherwise in the body of this 
communication or the attachment thereto (if any), the information is provided 
on an AS-IS basis without any express or implied warranties or liabilities. To 
the extent you are relying on this information, you are doing so at your own 
risk. If you are not the intended recipient, please notify the sender 
immediately by replying to this message and destroy all copies of this message 
and any attachments. Neither the sender nor the company/group of companies he 
or she represents shall be liable for the proper and complete transmission of 
the information contained in this communication, or for any delay in its 
receipt.
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] mmfind -ls and so forth

[Marc A Kaplan]

As always when dealing with computers and potentially long running jobs, 
run a test on a handful of files first, so you can rapidly debug.
Did you read the mmfind.README ?

It mentions...that this sample utility "some user assembly required..." 
... 

mmfindUtil_processOutputFile.c
  A utility to parse the "list file" produced by mmapplypolicy and to 
print
it in a find-compatible format
  mmfind invokes it once mmapplypolicy begins to populate the "list file"

mmfindUtil_processOutputFile.sampleMakefile
  copy to 'makefile', modify as needed, and run 'make' to compile 
mmfindUtil_processOutputFile.c
  This should produce a binary called mmfindUtil_processOutputFile
  mmfind will not be able to run until this utility has been compiled
  on the node from which you launch mmfind.

Works for me...
[root@n2 ilm]# ./mmfind /goo/zdbig -ls
2463649  256 drwxr-xr-x   2 root root   262144 Feb  9 11:41 
/goo/zdbig
68044970 -rw-r--r--   1 root root0 Feb  9 11:41 
/goo/zdbig/xy


___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] mmfind -ls, -exec but use -xargs wherever you can.

[Marc A Kaplan]

So much the more reasons to use 

   mmfind ... -xargs  ...

Which, for large number of files, gives you a very much more performant 
and parallelized execution of the classic

  find ... | xargs ...

The difference is exec is run in line with the evaluation of the other 
find conditionals (like -type f) but spawns a new command shell for each 
evaluation of exec...

Whereas -xargs is run after the pathnames of all of the (matching) files 
are discovered ... 
Like classic xargs, if your command can take a list of files, you save 
overhead there
BUT -xargs also runs multiple instances of your command in  multiple 
parallel processes on multiple nodes.


___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] mmfind -ls

[John Hearns]

Hi. I hope this reply comes through. I often get bounced when replying here.

In fact the reason is because I am not running ls.  This was just an example.
I am running mmgetlocation  to get the chunks allocation on each NSD of a file.

Secondly my problem is that a space is needed:

mmfind  /mountpoint -type f -exec mmgetlocation -D myproblemnsd -f {} \;
Note the space before the \

TO my shame this is the same as in the normal find command


From: gpfsug-discuss-boun...@spectrumscale.org 
[mailto:gpfsug-discuss-boun...@spectrumscale.org] On Behalf Of Marc A Kaplan
Sent: Thursday, February 22, 2018 3:21 PM
To: gpfsug main discussion list 
Cc: gpfsug-discuss-boun...@spectrumscale.org
Subject: Re: [gpfsug-discuss] mmfind -ls

Leaving aside the -exec option, and whether you choose classic find or mmfind,

why not just use the -ls option - same output, less overhead...

mmfind pathname -type f -ls




From:John Hearns mailto:john.hea...@asml.com>>
To:gpfsug main discussion list 
mailto:gpfsug-discuss@spectrumscale.org>>
Cc:
"gpfsug-discuss-boun...@spectrumscale.org<mailto:gpfsug-discuss-boun...@spectrumscale.org>"
 
mailto:gpfsug-discuss-boun...@spectrumscale.org>>
Date:02/22/2018 04:03 AM
Subject:Re: [gpfsug-discuss] mmfind will not exec
Sent by:
gpfsug-discuss-boun...@spectrumscale.org<mailto:gpfsug-discuss-boun...@spectrumscale.org>




Stupid me. The space between the {}  and \; is significant.
/usr/local/bin/mmfind /hpc/bscratch -type f -exec /bin/ls {} \;

Still would be nice to have the documentation clarified please.





From: 
gpfsug-discuss-boun...@spectrumscale.org<mailto:gpfsug-discuss-boun...@spectrumscale.org>
 [mailto:gpfsug-discuss-boun...@spectrumscale.org] On Behalf Of IBM Spectrum 
Scale
Sent: Thursday, February 22, 2018 2:26 AM
To: gpfsug main discussion list 
mailto:gpfsug-discuss@spectrumscale.org>>
Cc: 
gpfsug-discuss-boun...@spectrumscale.org<mailto:gpfsug-discuss-boun...@spectrumscale.org>
Subject: Re: [gpfsug-discuss] mmfind will not exec

Looking at the mmfind.README it indicates that it only supports the format you 
used with the semi-colon.  Did you capture any output of the problem?

Regards, The Spectrum Scale (GPFS) team

--
If you feel that your question can benefit other users of  Spectrum Scale 
(GPFS), then please post it to the public IBM developerWroks Forum at 
https://www.ibm.com/developerworks/community/forums/html/forum?id=----0479<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__emea01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fwww.ibm.com-252Fdeveloperworks-252Fcommunity-252Fforums-252Fhtml-252Fforum-253Fid-253D-2D-2D-2D-2D0479-26data-3D01-257C01-257Cjohn.hearns-2540asml.com-257Ce691619b05ef45106cd008d579934e0f-257Caf73baa8f5944eb2a39d93e96cad61fc-257C1-26sdata-3Dbd0PN6hi0nMTKeXMcsvUVNOEtRF4PGaJ-252BIm97cQiNjc-253D-26reserved-3D0%26d%3DDwMGaQ%26c%3Djf_iaSHvJObTbx-siA1ZOg%26r%3DcvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8%26m%3D77Whh54a5VWNFaaczlMhEzn7B802MGX9m-C2xj4sP1k%26s%3Dz7FmlsZ2aoUL9VjH82dy2HVT0bY9FERqHXQW5a3NHgQ%26e%3D&data=01%7C01%7Cjohn.hearns%40asml.com%7C376ff95e5cc845d8c85e08d579ff753d%7Caf73baa8f5944eb2a39d93e96cad61fc%7C1&sdata=tkrVw99IxTZUDHj6WzXIEI0HxrFmci8LNdSF3t0OPpE%3D&reserved=0>.

If your query concerns a potential software error in Spectrum Scale (GPFS) and 
you have an IBM software maintenance contract please contact  1-800-237-5511 in 
the United States or your local IBM Service Center in other countries.

The forum is informally monitored as time permits and should not be used for 
priority messages to the Spectrum Scale (GPFS) team.



From:John Hearns mailto:john.hea...@asml.com>>
To:gpfsug main discussion list 
mailto:gpfsug-discuss@spectrumscale.org>>
Date:02/21/2018 06:45 PM
Subject:[gpfsug-discuss] mmfind will not exec
Sent by:
gpfsug-discuss-boun...@spectrumscale.org<mailto:gpfsug-discuss-boun...@spectrumscale.org>





I would dearly like to use mmfind in a project I am working on  (version 
4.2.3.4 at the moment)

mmfind /hpc/bscratch  -type f  work fine

mmfind /hpc/bscratch  -type f -exec /bin/ls {}\ ;  crashes and burns

I know there are supposed to be problems with exec and mmfind, and this is 
sample software shipped without warranty etc.
But why let me waste hours on this when it won’t work?
There is even an example in the README for mmfind

./mmfind /encFS -type f -exec /bin/readMyFile {} \;
But in the help for mmfind:
-exec COMMANDs are terminated by a standalone ';' or by the string '

Re: [gpfsug-discuss] mmfind -ls

[Marc A Kaplan]

Leaving aside the -exec option, and whether you choose classic find or 
mmfind,

why not just use the -ls option - same output, less overhead...

mmfind pathname -type f -ls 




From:   John Hearns 
To: gpfsug main discussion list 
Cc: "gpfsug-discuss-boun...@spectrumscale.org" 

Date:   02/22/2018 04:03 AM
Subject:Re: [gpfsug-discuss] mmfind will not exec
Sent by:gpfsug-discuss-boun...@spectrumscale.org



Stupid me. The space between the {}  and \; is significant.
/usr/local/bin/mmfind /hpc/bscratch -type f -exec /bin/ls {} \;
 
Still would be nice to have the documentation clarified please.
  
 
 
 
 
From: gpfsug-discuss-boun...@spectrumscale.org [
mailto:gpfsug-discuss-boun...@spectrumscale.org] On Behalf Of IBM Spectrum 
Scale
Sent: Thursday, February 22, 2018 2:26 AM
To: gpfsug main discussion list 
Cc: gpfsug-discuss-boun...@spectrumscale.org
Subject: Re: [gpfsug-discuss] mmfind will not exec
 
Looking at the mmfind.README it indicates that it only supports the format 
you used with the semi-colon.  Did you capture any output of the problem?

Regards, The Spectrum Scale (GPFS) team

--
If you feel that your question can benefit other users of  Spectrum Scale 
(GPFS), then please post it to the public IBM developerWroks Forum at 
https://www.ibm.com/developerworks/community/forums/html/forum?id=----0479
. 

If your query concerns a potential software error in Spectrum Scale (GPFS) 
and you have an IBM software maintenance contract please contact 
1-800-237-5511 in the United States or your local IBM Service Center in 
other countries. 

The forum is informally monitored as time permits and should not be used 
for priority messages to the Spectrum Scale (GPFS) team.



From:John Hearns 
To:gpfsug main discussion list 
Date:02/21/2018 06:45 PM
Subject:[gpfsug-discuss] mmfind will not exec
Sent by:gpfsug-discuss-boun...@spectrumscale.org




I would dearly like to use mmfind in a project I am working on  (version 
4.2.3.4 at the moment)
 
mmfind /hpc/bscratch  -type f  work fine
 
mmfind /hpc/bscratch  -type f -exec /bin/ls {}\ ;  crashes and burns
 
I know there are supposed to be problems with exec and mmfind, and this is 
sample software shipped without warranty etc.
But why let me waste hours on this when it won’t work?
There is even an example in the README for mmfind 
 
./mmfind /encFS -type f -exec /bin/readMyFile {} \;
But in the help for mmfind:
-exec COMMANDs are terminated by a standalone ';' or by the string '{} +’
 
So which is it? The normal find version {} \;or {} +
 
-- The information contained in this communication and any attachments is 
confidential and may be privileged, and is for the sole use of the 
intended recipient(s). Any unauthorized review, use, disclosure or 
distribution is prohibited. Unless explicitly stated otherwise in the body 
of this communication or the attachment thereto (if any), the information 
is provided on an AS-IS basis without any express or implied warranties or 
liabilities. To the extent you are relying on this information, you are 
doing so at your own risk. If you are not the intended recipient, please 
notify the sender immediately by replying to this message and destroy all 
copies of this message and any attachments. Neither the sender nor the 
company/group of companies he or she represents shall be liable for the 
proper and complete transmission of the information contained in this 
communication, or for any delay in its receipt. 
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=OC7XNZeulP0vmS8Fq-RJuun5wOqFPootm0QHxBXUfKg&s=LUvpk53AaNcHSGQgDgH8FAiOOsH1H0OPOV9MFGMIi9E&e=
 
-- The information contained in this communication and any attachments is 
confidential and may be privileged, and is for the sole use of the 
intended recipient(s). Any unauthorized review, use, disclosure or 
distribution is prohibited. Unless explicitly stated otherwise in the body 
of this communication or the attachment thereto (if any), the information 
is provided on an AS-IS basis without any express or implied warranties or 
liabilities. To the extent you are relying on this information, you are 
doing so at your own risk. If you are not the intended recipient, please 
notify the sender immediately by replying to this message and destroy all 
copies of this message and any attachments. Neither the sender nor the 
company/group of companies he or she represents shall be liable for the 
proper and complete transmission of the information contained in this 
communication, or for any delay in its receipt. 
___