Re: [gpfsug-discuss] Self service creation of filesets

[Robert Horton]

On Wed, 2021-03-03 at 12:14 -0500, Russell Nordquist wrote:
CAUTION: This email originated from outside of the ICR. Do not click links or 
open attachments unless you recognize the sender's email address and know the 
content is safe.

Sounds like I am not the only one that needs this. The REST API has everything 
needed to do this, but the problem is we can’t restrict the GUI role account to 
just the commands they need. They need “storage administrator” access which 
means the could also make/delete filesystems. I guess you could use sudo and 
wrap the CLI, but I am told that’s old fashioned :)  Too bad we can’t make a 
API role with specific POST commands tied to it. I am surprised there is no RFE 
for that yet. The closest I see is
http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe_ID=148244 am 
I missing something.


That reminds me... We use a Python wrapper around the REST API to monitor usage 
against fileset quotas etc. In principle this will also set quotas (and create 
filesets) but it means giving it storage administrator access. It would be nice 
if the GUI had sufficiently fine grained permissions that you could set quotas 
without being able to delete the filesystem.

Rob

--

Robert Horton | Research Data Storage Lead
The Institute of Cancer Research | 237 Fulham Road | London | SW3 6JB
T +44 (0)20 7153 5350 | E 
robert.hor...@icr.ac.uk | W www.icr.ac.uk | 
Twitter @ICR_London
Facebook: www.facebook.com/theinstituteofcancerresearch

The Institute of Cancer Research: Royal Cancer Hospital, a charitable Company 
Limited by Guarantee, Registered in England under Company No. 534147 with its 
Registered Office at 123 Old Brompton Road, London SW7 3RP.

This e-mail message is confidential and for use by the addressee only.  If the 
message is received by anyone other than the addressee, please return the 
message to the sender by replying to it and then delete the message from your 
computer and network.
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] Self service creation of filesets

[Russell Nordquist]

Sounds like I am not the only one that needs this. The REST API has everything 
needed to do this, but the problem is we can’t restrict the GUI role account to 
just the commands they need. They need “storage administrator” access which 
means the could also make/delete filesystems. I guess you could use sudo and 
wrap the CLI, but I am told that’s old fashioned :)  Too bad we can’t make a 
API role with specific POST commands tied to it. I am surprised there is no RFE 
for that yet. The closest I see is
http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe_ID=148244 
 
am I missing something. 

What I would want is to be able to grant the the following calls + maybe a few 
more. 

The related REST API calls.
https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.1/com.ibm.spectrum.scale.v5r01.doc/bl1adm_apiv2postfilesystemfilesets.htm
 

https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.1/com.ibm.spectrum.scale.v5r01.doc/bl1adm_apiv2postfilesystemfilesetlink.htm
 


Russell


> On Mar 3, 2021, at 3:06 AM, Loic Tortay  wrote:
> 
> On 02/03/2021 20:31, Russell Nordquist wrote:
>> Hi all
>> We are trying to use filesets quite a bit, but it’s a hassle that only the 
>> admins can create them. To the users it’s just a directory so it slows 
>> things down. Has anyone deployed a self service model for creating filesets? 
>> Maybe using the API? This feels like shared pain that someone has already 
>> worked on….
> Hello,
> We have a quota management delegation (CLI) tool that allows "power-users" 
> (PI and such) to create and remove filesets and manage users quotas for the 
> groups/projects they're heading.
> 
> Like someone else said, from their point of view they're just directories, so 
> they create a "directory with quotas".
> In our experience, "directories with quotas" are the most convenient way for 
> end-users to understand and use quotas.
> 
> This is a tool written in C, about 13 years ago, using the GPFS API (and a 
> few calls to GPFS commands where there is no API or it's lacking).
> 
> Delegation authorization (identifying "power-users") is external to the tool.
> 
> Permissions & ACLs are also set on the junction when a fileset is created so 
> that it's both immediately usable ("instant processing") and accessible to 
> "power-users" (for space management purposes).
> 
> There are extra features for staff to allow higher-level operations (e.g. 
> create an independent fileset for a group/project, change the group/project 
> quotas, etc.)
> 
> The dated looking user documentation is 
> https://ccspsmon.in2p3.fr/spsquota.html
> 
> Both the tool and the documentation have a few site-specific things, so it's 
> not open-source (and it has become a "legacy" tool in need of a 
> rewrite/refactoring).
> 
> 
> Loïc.
> -- 
> |   Loïc Tortay   - IN2P3 Computing Centre |
> 

___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] Self service creation of filesets

[Loic Tortay]

On 02/03/2021 20:31, Russell Nordquist wrote:

Hi all

We are trying to use filesets quite a bit, but it’s a hassle that only the 
admins can create them. To the users it’s just a directory so it slows things 
down. Has anyone deployed a self service model for creating filesets? Maybe 
using the API? This feels like shared pain that someone has already worked on….


Hello,
We have a quota management delegation (CLI) tool that allows 
"power-users" (PI and such) to create and remove filesets and manage 
users quotas for the groups/projects they're heading.


Like someone else said, from their point of view they're just 
directories, so they create a "directory with quotas".
In our experience, "directories with quotas" are the most convenient way 
for end-users to understand and use quotas.


This is a tool written in C, about 13 years ago, using the GPFS API (and 
a few calls to GPFS commands where there is no API or it's lacking).


Delegation authorization (identifying "power-users") is external to the 
tool.


Permissions & ACLs are also set on the junction when a fileset is 
created so that it's both immediately usable ("instant processing") and 
accessible to "power-users" (for space management purposes).


There are extra features for staff to allow higher-level operations 
(e.g. create an independent fileset for a group/project, change the 
group/project quotas, etc.)


The dated looking user documentation is 
https://ccspsmon.in2p3.fr/spsquota.html


Both the tool and the documentation have a few site-specific things, so 
it's not open-source (and it has become a "legacy" tool in need of a 
rewrite/refactoring).



Loïc.
--
|   Loïc Tortay   - IN2P3 Computing Centre |
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] Self service creation of filesets

[Kerner, Chad A]

We have a similar process. One of our customers has a web app that their 
managers use to provision spaces. That web app drops a json file into a 
specific location and a cron job kicks off a python script every so often to 
process the files and provision the space(fileset creation, link, quota, owner, 
group, perms, etc). Failures are queued and a jira ticket opened. Successes 
update the database for the web app. They are not requiring instant processing, 
so we process hourly on the back end side of things.

Chad
--
Chad Kerner, Senior Storage Engineer
Storage Enabling Technologies
National Center for Supercomputing Applications
University of Illinois, Urbana-Champaign

On 3/2/21, 4:38 PM, "gpfsug-discuss-boun...@spectrumscale.org on behalf of 
Simon Thompson"  wrote:

Not quite user self-service 

But we have some web tooling for project registration that pushes sanitised 
messages onto a redis (rq) backed message bus which then does "stuff". For 
example create and populate groups in AD and LDAP. Create and link a fileset, 
set quota etc ... Our consumer code is all built to be tolerant to running it a 
second time safely and has quite a bit of internal locking to prevent multiple 
instances running at the same time (though we have multiple consumer entities 
to handle fail-over). The fault tolerant thing is quite important as create a 
fileset can fail for a number of reasons (e.g. restripefs running), so we can 
always just requeue the requests again.

Simon

On 02/03/2021, 19:40, "gpfsug-discuss-boun...@spectrumscale.org on behalf 
of russ...@nordquist.info"  wrote:

Hi all

We are trying to use filesets quite a bit, but it’s a hassle that only 
the admins can create them. To the users it’s just a directory so it slows 
things down. Has anyone deployed a self service model for creating filesets? 
Maybe using the API? This feels like shared pain that someone has already 
worked on….

thanks
Russell


___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org

https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss__;!!DZ3fjg!uQVokpQk0pPyjpae7a_Aui1wGk3k7xJzIxzX1DBNfOyNOfzZeJFUjVOqN3OVEyVqdw$
 

___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org

https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss__;!!DZ3fjg!uQVokpQk0pPyjpae7a_Aui1wGk3k7xJzIxzX1DBNfOyNOfzZeJFUjVOqN3OVEyVqdw$
 


___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] Self service creation of filesets

[Simon Thompson]

Not quite user self-service 

But we have some web tooling for project registration that pushes sanitised 
messages onto a redis (rq) backed message bus which then does "stuff". For 
example create and populate groups in AD and LDAP. Create and link a fileset, 
set quota etc ... Our consumer code is all built to be tolerant to running it a 
second time safely and has quite a bit of internal locking to prevent multiple 
instances running at the same time (though we have multiple consumer entities 
to handle fail-over). The fault tolerant thing is quite important as create a 
fileset can fail for a number of reasons (e.g. restripefs running), so we can 
always just requeue the requests again.

Simon

On 02/03/2021, 19:40, "gpfsug-discuss-boun...@spectrumscale.org on behalf of 
russ...@nordquist.info"  wrote:

Hi all

We are trying to use filesets quite a bit, but it’s a hassle that only the 
admins can create them. To the users it’s just a directory so it slows things 
down. Has anyone deployed a self service model for creating filesets? Maybe 
using the API? This feels like shared pain that someone has already worked on….

thanks
Russell


___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] Self service creation of filesets

[Alec]

This does feel like another situation where I may use a custom attribute
and a periodic script to do the fileset creation.  Honestly I would want
the change management around fileset creation.

But I could see a few custom attributes on a newly created user dir... Like
maybe just setting user.quota=10TB...  Then have a policy that discovers
these does the work of creating the fileset, setting the quotas, migrating
data to the fileset, and then mounting the fileset over the original
directory.  Honestly that sounds so nice I may have to implement this...
Lol.

Like I could see doing something like discovering directories that have
user.archive=true and automatically gzipping large files within. Would be
nice if GPFS policy engine could have a IF_ANCESTOR_ATTRIBUTE=.

Alec

On Tue, Mar 2, 2021, 11:40 AM Russell Nordquist 
wrote:

> Hi all
>
> We are trying to use filesets quite a bit, but it’s a hassle that only the
> admins can create them. To the users it’s just a directory so it slows
> things down. Has anyone deployed a self service model for creating
> filesets? Maybe using the API? This feels like shared pain that someone has
> already worked on….
>
> thanks
> Russell
>
>
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss