Re: [gpfsug-discuss] Change uidNumber and gidNumber for billions of files

2020-06-12 Thread Laurence Horrocks-Barlow
I seem to remember Marc Kaplan discussing using the ILM and mmfind for this. There is a presentation from 2018 which skims on an example http://files.gpfsug.org/presentations/2018/USA/SpectrumScalePolicyBP.pdf -- Lauz On 10/06/2020 23:40, Kerner, Chad A wrote: You can do a policy scan though

Re: [gpfsug-discuss] Change uidNumber and gidNumber for billions of files

2020-06-10 Thread Kerner, Chad A
You can do a policy scan though and get a list of files that have ACLs applied to them. Then you would not have to check every file with a shell utility or C, just process that list. Likewise, you can get the uid/gid as well and process that list with the new mapping(split it into multiple

Re: [gpfsug-discuss] Change uidNumber and gidNumber for billions of files

2020-06-10 Thread Jonathan Buzzard
On 10/06/2020 16:31, Lohit Valleru wrote: [SNIP] > I might mostly start small with a single lab, and only change files > without ACLs. May I know if anyone has a method/tool to find out which > files/dirs have NFS4 ACLs set? As far as we know - it is just one > fileset/lab, but it would be

Re: [gpfsug-discuss] Change uidNumber and gidNumber for billions of files

2020-06-10 Thread Lohit Valleru
Thank you everyone for the Inputs. The answers to some of the questions are as follows: > From Jez: I've done this a few times in the past in a previous life.  In many > respects it is easier (and faster!) to remap the AD side to the uids already > on the filesystem. - Yes we had

Re: [gpfsug-discuss] Change uidNumber and gidNumber for billions of files

2020-06-10 Thread Jonathan Buzzard
On 10/06/2020 02:15, Aaron Knister wrote: Lohit, I did this while working @ NASA. I had two tools I used, one affectionately known as "luke file walker" (to modify traditional unix permissions) and the other known as the "milleniumfacl" (to modify posix ACLs). Stupid jokes aside, there were

Re: [gpfsug-discuss] Change uidNumber and gidNumber for billions of files

2020-06-10 Thread Simon Thompson
Quota … I thought there was a work around for this. I think it went along the lines of. Set the soft quota to what you want. Set the hard quota 150% more. Set the grace period to 1 second. I think the issue is that when you are over soft quota, each operation has to queisce each time until you

Re: [gpfsug-discuss] Change uidNumber and gidNumber for billions of files

2020-06-09 Thread Aaron Knister
Lohit, I did this while working @ NASA. I had two tools I used, one affectionately known as "luke file walker" (to modify traditional unix permissions) and the other known as the "milleniumfacl" (to modify posix ACLs). Stupid jokes aside, there were some real technical challenges here. I don't

Re: [gpfsug-discuss] Change uidNumber and gidNumber for billions of files

2020-06-09 Thread Jonathan Buzzard
On 09/06/2020 14:57, Jonathan Buzzard wrote: [SNIP] I actually thinking on it more thought a generic C random UID/GID to UID/GID mapping program is a really simple piece of code and should be nearly as fast as chown -R. It will be very slightly slower as you have to look the mapping up for

Re: [gpfsug-discuss] Change uidNumber and gidNumber for billions of files

2020-06-09 Thread Stephen Ulmer
Jonathan brings up a good point that you’ll only get one shot at this — if you’re using the file system as your record of who owns what. You might want to use the policy engine to record the existing file names and ownership (and then provide updates using the same policy engine for the things

Re: [gpfsug-discuss] Change uidNumber and gidNumber for billions of files

2020-06-09 Thread Jonathan Buzzard
On 09/06/2020 14:07, Stephen Ulmer wrote: Jonathan brings up a good point that you’ll only get one shot at this — if you’re using the file system as your record of who owns what. Not strictly true if my existing UID's are in the range 1-1 and my target UID's are in the range

Re: [gpfsug-discuss] Change uidNumber and gidNumber for billions of files

2020-06-09 Thread Jonathan Buzzard
On 08/06/2020 18:44, Lohit Valleru wrote: Hello Everyone, We are planning to migrate from LDAP to AD, and one of the best solution was to change the uidNumber and gidNumber to what SSSD or Centrify would resolve. May I know, if anyone has come across a tool/tools that can change the

Re: [gpfsug-discuss] Change uidNumber and gidNumber for billions of files

2020-06-09 Thread Simon Thompson
> I presume there is a large technical blocker which is why you are looking at > remapping the filesystem? Like anytime there is a corporate AD with mandated attributes?  Though isn’t there an AD thing now for doing schema view type things now which allow you to inherit certain attributes and

Re: [gpfsug-discuss] Change uidNumber and gidNumber for billions of files

2020-06-09 Thread Jez Tucker
Hi Lohit (hey Jim & Christof),   Whilst you _could_ trawl your entire filesystem, flip uids and work out how to successfully replace ACL ids without actually pushing ACLs (which could break defined inheritance options somewhere in your file tree if you had not first audited your filesystem) the

Re: [gpfsug-discuss] Change uidNumber and gidNumber for billions of files

2020-06-08 Thread Jim Doherty
You will need to do this with chown from the  c library functions  (could do this from perl or python).   If you try to change this from a shell script  you will hit the Linux command  which will have a lot more overhead. I had a customer attempt this using the shell and it ended up taking

Re: [gpfsug-discuss] Change uidNumber and gidNumber for billions of files

2020-06-08 Thread Christof Schmitt
If there are ACLs, then you also need to update all ACLs  (gpfs_getacl(), update uids and gids in all entries, gpfs_putacl()), in addition to the chown() call.   Regards,   Christof Schmitt || IBM || Spectrum Scale Development || Tucson, AZchristof.schm...@us.ibm.com  ||  +1-520-799-2469    (T/L:

[gpfsug-discuss] Change uidNumber and gidNumber for billions of files

2020-06-08 Thread Lohit Valleru
Hello Everyone, We are planning to migrate from LDAP to AD, and one of the best solution was to change the uidNumber and gidNumber to what SSSD or Centrify would resolve. May I know, if anyone has come across a tool/tools that can change the uidNumbers and gidNumbers of billions of files