There are two entities, storage resource and storage path. Both need to be defined. Currently, Storage resource is in
GATEWAY_DATA_STORE_RESOURCE_ID and the path is in GATEWAY_DATA_STORE_DIR Also see GATEWAY_DATA_STORE_HOSTNAME and FILE_UPLOAD_TEMP_DIR Which may be used in certain operations as well. Thanks, Sudhakar. From: "Bandaru, Vivek Shresta" <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Monday, July 27, 2020 at 4:30 AM To: "[email protected]" <[email protected]> Subject: Re: User storage My current understanding is that, by default, all the experiments are stored on the same server where Airavata-Django-Portal is deployed, in the Gateway User Directory specified in settings_local.py. Came to this conclusion from this<https://github.com/apache/airavata-django-portal/blob/master/django_airavata/apps/api/datastore.py#L195> code which is used to create the experimentDataDir path. From my prior discussions, I understood that the ‘StoragePreferences’ are used when the current/default experimentDataDir is full and a user wants to store his/her experiment data elsewhere. Now, according to my current understanding from Friday’s meeting, this still needs to be implemented by integrating MFT with AiravataDjangoPortal and adding a new dropdown in the experiment creation phase, where the user can choose out of the default experiment directory and his storage preferences, where he wants to create and store the new experiment. Is my current understanding right? Thanks for reading. Regards, Vivek. From: "Bandaru, Vivek Shresta" <[email protected]> Date: Friday, July 24, 2020 at 3:12 AM To: "[email protected]" <[email protected]> Subject: Re: User storage Hi Marcus, Thanks for the reply. Earlier, I did observe that all the experiments are being created in ‘/tmp’ directory using this<https://github.com/apache/airavata-django-portal/blob/5ca39e986fe2fe53020ed45cff9c0a8a62262976/django_airavata/apps/api/datastore.py#L195> code since, the GATEWAY_DATA_STORE_DIR in the settings_local.py.sample reflects ‘/tmp’ in my local. What I’m confused about is, when does the location in ‘Storage Preferences’ come into picture? I was under the impression that all the user data is stored at the place defined by an admin through the Storage Preferences in the Gateway Resource Profile page. Debugging the code proved otherwise. Now, according to my current understanding, a user experiment’s input and output files are stored in the ‘GATEWAY_DATA_STORE_DIR/{username}/{ProjectId}/{ExperimentId}/’ directory. But what is the role of the File system location mentioned in a Storage Preferences? If in my local, I was able to run any experiment which either generated an output file or needed an input file, I would have probably understood the this better. But the only experiment I’m able to run in my local is ‘Echo’, which doesn’t need any input file and sadly for some reason is not generating any output and is always in ‘Launched’ state. How can I create other experiments which are available on https://testdrive.airavata.org/ in my local? Regards, Vivek. From: "Christie, Marcus Aaron" <[email protected]> Reply-To: <[email protected]> Date: Thursday, July 23, 2020 at 6:35 PM To: "[email protected]" <[email protected]> Subject: Re: User storage Hi Vivek, The File System Location is used by Airavata to know the location of the gateway storage directory on the user portal so it can stage output files back to the portal. In practice however, the File System Location is rarely used because when an experiment is created, the portal sets an experimentDataDir property on it to specify exactly where output files should be put. The Django portal has settings that determine the directory where the gateway storage. [0] [0] https://github.com/apache/airavata-django-portal/blob/master/django_airavata/settings_local.py.sample#L84-L86 On Jul 21, 2020, at 2:45 AM, Bandaru, Vivek Shresta <[email protected]<mailto:[email protected]>> wrote: Hi All, I’m confused on when the File System Location specified in StoragePreferences in ‘Gateway Resource Profile’ come into picture. Upon login, a storage directory (ex: /tmp/default-admin for the user default-admin) is created for every user where all his experiments and the respective input and output data is stored. Every experiment has its own directory with the same name as that of the experiment created under the above mentioned directory. <image001.png> Not sure where does the File System Location – ‘/var/www/portals/gateway-user-data’ mentioned in the Storage Preferences come into picture. Is this specific to Airavata and earlier mentioned file system location specific to Django-portal gateway? <image002.png> During one of my very earlier interactions regarding the GSoC project, I was told about File System location in the Storage preferences which has all the gateways data. Please refer to the below mentioned screenshot shared (Note: This screenshot isn’t from my local). <image003.png> But in my local, the directory var, doesn’t have any directory called ‘www’. Please refer to the below mentioned screenshot. <image004.png> I would like to know a little more on how file system storage is used in Django-Portal and Airavata and any pointers are appreciated. Thanks for reading. Regards, Vivek.
