[
https://issues.apache.org/jira/browse/AIRAVATA-3346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Christie updated AIRAVATA-3346:
--------------------------------------
Description:
Running the Airavata Django portal locally, a developer experiences the portal
just like a user of a deployed portal, *except* when it comes to gateway user
storage, which is only available on the deployed portal. So for example, on
seagrid.org, I can log in, view my experiments, and download my output files.
If I run the Airavata Django portal locally and have configured it to connect
to the same Keycloak and API server, I can also login and view my experiments,
but I can't download my output files (or upload files, or clone experiments,
etc.).
However, there exists a REST API to get and put files into gateway user storage
so in theory it should be possible to create a filesystem abstraction, locally,
that uses the REST API underneath to make it appear that the user's files are
all available.
The benefit of this is that it opens up local development to be able to test a
broader range of use cases. I know when I'm developing the Django portal code,
there are some things that are just very hard to test because I don't actually
have access to the files locally. Also, for extension developers, this will be
helpful too. For example, currently creating a custom output view provider
requires using a fake output file to test locally because the data files aren't
available.
This seems like a good option: PyFilesystem2 https://pypi.org/project/fs/
One thing, we need token based authentication in the REST API, but I've already
started work on that in a separate branch.
Another thing, it could be good to put in this work now since it might ease
transitioning to Airavata MFT for the data storage backend. MFT would just be
another PyFilesystem2 filesystem implementation.
h5. TODO
- [x] Need support for Bearer token authentication so that the REST API can be
used directly
- [x] Update API view function so that they are APIViews and use the Django
REST Framework authentication/permission support
- [x] Implement remote API support in ADP SDK user_storage functions
-- to get the user storage view working remotely:
--- [x] user_storage.dir_exists
--- [x] user_storage.listdir
--- [x] user_storage.get_file
--- [x] user_storage.delete_dir
--- [x] user_storage.create_user_dir
--- [x] user_storage.save
-- to get input file upload working remotely:
--- [x] user_storage.save_input_file
-- to get experiment data directory creation and launch working remotely:
--- [x] not on user_storage but ExperimentViewSet.launch
-- to get experiment cloning working remotely:
--- [x] not on user_storage but ExperimentViewSet.clone
- [ ] support tus uploads to the remote data store
- [ ] Bug: infinite loop error when token is invalid
was:
Running the Airavata Django portal locally, a developer experiences the portal
just like a user of a deployed portal, *except* when it comes to gateway user
storage, which is only available on the deployed portal. So for example, on
seagrid.org, I can log in, view my experiments, and download my output files.
If I run the Airavata Django portal locally and have configured it to connect
to the same Keycloak and API server, I can also login and view my experiments,
but I can't download my output files (or upload files, or clone experiments,
etc.).
However, there exists a REST API to get and put files into gateway user storage
so in theory it should be possible to create a filesystem abstraction, locally,
that uses the REST API underneath to make it appear that the user's files are
all available.
The benefit of this is that it opens up local development to be able to test a
broader range of use cases. I know when I'm developing the Django portal code,
there are some things that are just very hard to test because I don't actually
have access to the files locally. Also, for extension developers, this will be
helpful too. For example, currently creating a custom output view provider
requires using a fake output file to test locally because the data files aren't
available.
This seems like a good option: PyFilesystem2 https://pypi.org/project/fs/
One thing, we need token based authentication in the REST API, but I've already
started work on that in a separate branch.
Another thing, it could be good to put in this work now since it might ease
transitioning to Airavata MFT for the data storage backend. MFT would just be
another PyFilesystem2 filesystem implementation.
h5. TODO
- [x] Need support for Bearer token authentication so that the REST API can be
used directly
- [x] Update API view function so that they are APIViews and use the Django
REST Framework authentication/permission support
- [x] Implement remote API support in ADP SDK user_storage functions
-- to get the user storage view working remotely:
--- [x] user_storage.dir_exists
--- [x] user_storage.listdir
--- [x] user_storage.get_file
--- [x] user_storage.delete_dir
--- [x] user_storage.create_user_dir
--- [x] user_storage.save
-- to get input file upload working remotely:
--- [x] user_storage.save_input_file
-- to get experiment data directory creation and launch working remotely:
--- [ ] not on user_storage but ExperimentViewSet.launch
-- to get experiment cloning working remotely:
- [ ] Bug: infinite loop error when token is invalid
> Implement remote FS abstraction of user storage
> -----------------------------------------------
>
> Key: AIRAVATA-3346
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3346
> Project: Airavata
> Issue Type: Bug
> Components: Django Portal
> Reporter: Marcus Christie
> Assignee: Marcus Christie
> Priority: Major
>
> Running the Airavata Django portal locally, a developer experiences the
> portal just like a user of a deployed portal, *except* when it comes to
> gateway user storage, which is only available on the deployed portal. So for
> example, on seagrid.org, I can log in, view my experiments, and download my
> output files. If I run the Airavata Django portal locally and have configured
> it to connect to the same Keycloak and API server, I can also login and view
> my experiments, but I can't download my output files (or upload files, or
> clone experiments, etc.).
> However, there exists a REST API to get and put files into gateway user
> storage so in theory it should be possible to create a filesystem
> abstraction, locally, that uses the REST API underneath to make it appear
> that the user's files are all available.
> The benefit of this is that it opens up local development to be able to test
> a broader range of use cases. I know when I'm developing the Django portal
> code, there are some things that are just very hard to test because I don't
> actually have access to the files locally. Also, for extension developers,
> this will be helpful too. For example, currently creating a custom output
> view provider requires using a fake output file to test locally because the
> data files aren't available.
> This seems like a good option: PyFilesystem2 https://pypi.org/project/fs/
> One thing, we need token based authentication in the REST API, but I've
> already started work on that in a separate branch.
> Another thing, it could be good to put in this work now since it might ease
> transitioning to Airavata MFT for the data storage backend. MFT would just be
> another PyFilesystem2 filesystem implementation.
> h5. TODO
> - [x] Need support for Bearer token authentication so that the REST API can
> be used directly
> - [x] Update API view function so that they are APIViews and use the Django
> REST Framework authentication/permission support
> - [x] Implement remote API support in ADP SDK user_storage functions
> -- to get the user storage view working remotely:
> --- [x] user_storage.dir_exists
> --- [x] user_storage.listdir
> --- [x] user_storage.get_file
> --- [x] user_storage.delete_dir
> --- [x] user_storage.create_user_dir
> --- [x] user_storage.save
> -- to get input file upload working remotely:
> --- [x] user_storage.save_input_file
> -- to get experiment data directory creation and launch working remotely:
> --- [x] not on user_storage but ExperimentViewSet.launch
> -- to get experiment cloning working remotely:
> --- [x] not on user_storage but ExperimentViewSet.clone
> - [ ] support tus uploads to the remote data store
> - [ ] Bug: infinite loop error when token is invalid
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
