Responses below: > On Aug 5, 2020, at 3:20 AM, Bandaru, Vivek Shresta <[email protected]> wrote: > > Adding restrictions to output files would mean the user will not be able to > access the output files even if he was able to create a new experiment with > input files. That would mean, the user waits for the experiment’s output only > to find out he doesn’t have enough storage space. Reason why I did not add > any validation for output data. Is this a valid use case? Also, is there any > way to know how much space an experiment’s output will need before launching > the experiment?
I agree, we wouldn't want to restrict storing the output files just because the user hit their quota limit. This does raise the question, what should happen? Probably there should be a soft and hard quota limit. When the user hits the soft limit they can't add new files manually (uploading files in the portal), and the user is informed of the situation and advised to free up some space or request a larger storage quota. If they hit the hard limit then no further additions are allowed, even output files. > After the previous discussion, I assumed once GroupResourceProfile and > UserStoragePreference (apparently it’s about to be deprecated now) are in > play, Airavata would be able to choose which StoragePreference should be > chosen, reason why I went with mechanism where the API server validates the > storage limit. With the current architecture as is, if Airavata tracking the > user storage(MFT in the future) is not necessary, then maybe the gateway > worrying about the storage quota makes more sense. I think Airavata validating the storage limit can make sense. But if Airavata is managing it, then it could just query the storage resource for the amount of storage space used by a user. Airavata already has SSH access to get and put files on the storage resources, so it can certainly run the equivalent of 'du'. That way we don't need to add an API for the portal to report storage space to the API server. Please do raise the PR, it would be good to see how you've implemented your approach. Thanks, Marcus
smime.p7s
Description: S/MIME cryptographic signature
