[jira] [Commented] (AIRAVATA-3649) Automatically calculate queue settings with custom code
[
https://issues.apache.org/jira/browse/AIRAVATA-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17769750#comment-17769750
]
ASF subversion and git services commented on AIRAVATA-3649:
---
Commit 6339f7047f954af9cc269f8771c031695291e411 in
airavata-cookiecutter-django-app's branch refs/heads/main from Marcus Christie
[
https://gitbox.apache.org/repos/asf?p=airavata-cookiecutter-django-app.git;h=6339f70
]
AIRAVATA-3649 Adding documentation link
> Automatically calculate queue settings with custom code
> ---
>
> Key: AIRAVATA-3649
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3649
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> For some applications the ideal queue settings for an experiment can be
> straightforwardly calculated from the input values for the experiment. An
> administrator may want to define some custom code to calculate the ideal
> queue settings and then register it for use with one or more applications.
> This will make streamline the use of the gateway for users and also reduce
> errors in over- or under-allocating resources.
> h3. Design
> - custom code to be implemented as a module function that takes
> experiment_model and returns a dictionary of possibly different queue setting
> values
> - decorator {{@queue_settings_calculator}}
> - decorator attributes: id (allows renaming function), name (display to user)
> - decorator adds each decorated function to a registry
> - new REST API returns information about all registered
> {{@queue_settings_calculator}} functions
> - new application_settings field for queue_settings_calculator id. UI
> provides drop-down to allow admin to select a queue_settings_calculator
> - Experiment Editor will call calculate_queue_settings REST API if the
> application interface has a queue_settings_calculator configured. Will pass
> the entire Experiment Model. Updates the queue settings with the returned
> values.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3698) Add support for Python 3.11
[ https://issues.apache.org/jira/browse/AIRAVATA-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17769229#comment-17769229 ] ASF subversion and git services commented on AIRAVATA-3698: --- Commit e88ae808bcc8684a408900cc55f6eb5de3b063af in airavata-django-portal's branch refs/heads/AIRAVATA-3698--Add-support-for-Python-3.11 from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=e88ae808 ] AIRAVATA-3698 Fix new default theme page revisions > Add support for Python 3.11 > --- > > Key: AIRAVATA-3698 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3698 > Project: Airavata > Issue Type: Improvement > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > Upgrade Wagtail and Django versions to get Python 3.11 support -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3698) Add support for Python 3.11
[ https://issues.apache.org/jira/browse/AIRAVATA-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17769228#comment-17769228 ] ASF subversion and git services commented on AIRAVATA-3698: --- Commit 13e2d6f6be0115a88b532bfc580ad9243ecb6f9f in airavata-django-portal's branch refs/heads/AIRAVATA-3698--Add-support-for-Python-3.11 from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=13e2d6f6 ] Merge branch 'develop' into AIRAVATA-3698--Add-support-for-Python-3.11 > Add support for Python 3.11 > --- > > Key: AIRAVATA-3698 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3698 > Project: Airavata > Issue Type: Improvement > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > Upgrade Wagtail and Django versions to get Python 3.11 support -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3698) Add support for Python 3.11
[ https://issues.apache.org/jira/browse/AIRAVATA-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17769227#comment-17769227 ] ASF subversion and git services commented on AIRAVATA-3698: --- Commit 75ff2835a078a515991dc48082fc19fdd9a72ae5 in airavata-django-portal's branch refs/heads/AIRAVATA-3698--Add-support-for-Python-3.11 from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=75ff2835 ] Merge pull request #186 from apache/AIRAVATA-3698-fix-page-revision-content-type AIRAVATA-3698 management command to fix up the content type id in page revision json > Add support for Python 3.11 > --- > > Key: AIRAVATA-3698 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3698 > Project: Airavata > Issue Type: Improvement > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > Upgrade Wagtail and Django versions to get Python 3.11 support -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3698) Add support for Python 3.11
[ https://issues.apache.org/jira/browse/AIRAVATA-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17769226#comment-17769226 ] ASF subversion and git services commented on AIRAVATA-3698: --- Commit dd41406b112c7a6d5d2af74901f6de291b733ade in airavata-django-portal's branch refs/heads/AIRAVATA-3698--Add-support-for-Python-3.11 from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=dd41406b ] AIRAVATA-3698 management command to fix up the content type id in page revision json > Add support for Python 3.11 > --- > > Key: AIRAVATA-3698 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3698 > Project: Airavata > Issue Type: Improvement > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > Upgrade Wagtail and Django versions to get Python 3.11 support -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3698) Add support for Python 3.11
[ https://issues.apache.org/jira/browse/AIRAVATA-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17769200#comment-17769200 ] ASF subversion and git services commented on AIRAVATA-3698: --- Commit dd41406b112c7a6d5d2af74901f6de291b733ade in airavata-django-portal's branch refs/heads/develop from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=dd41406b ] AIRAVATA-3698 management command to fix up the content type id in page revision json > Add support for Python 3.11 > --- > > Key: AIRAVATA-3698 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3698 > Project: Airavata > Issue Type: Improvement > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > Upgrade Wagtail and Django versions to get Python 3.11 support -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3698) Add support for Python 3.11
[ https://issues.apache.org/jira/browse/AIRAVATA-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17769201#comment-17769201 ] ASF subversion and git services commented on AIRAVATA-3698: --- Commit 75ff2835a078a515991dc48082fc19fdd9a72ae5 in airavata-django-portal's branch refs/heads/develop from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=75ff2835 ] Merge pull request #186 from apache/AIRAVATA-3698-fix-page-revision-content-type AIRAVATA-3698 management command to fix up the content type id in page revision json > Add support for Python 3.11 > --- > > Key: AIRAVATA-3698 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3698 > Project: Airavata > Issue Type: Improvement > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > Upgrade Wagtail and Django versions to get Python 3.11 support -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3698) Add support for Python 3.11
[ https://issues.apache.org/jira/browse/AIRAVATA-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17768898#comment-17768898 ] ASF subversion and git services commented on AIRAVATA-3698: --- Commit dd41406b112c7a6d5d2af74901f6de291b733ade in airavata-django-portal's branch refs/heads/master from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=dd41406b ] AIRAVATA-3698 management command to fix up the content type id in page revision json > Add support for Python 3.11 > --- > > Key: AIRAVATA-3698 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3698 > Project: Airavata > Issue Type: Improvement > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > Upgrade Wagtail and Django versions to get Python 3.11 support -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3698) Add support for Python 3.11
[ https://issues.apache.org/jira/browse/AIRAVATA-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17768899#comment-17768899 ] ASF subversion and git services commented on AIRAVATA-3698: --- Commit 75ff2835a078a515991dc48082fc19fdd9a72ae5 in airavata-django-portal's branch refs/heads/master from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=75ff2835 ] Merge pull request #186 from apache/AIRAVATA-3698-fix-page-revision-content-type AIRAVATA-3698 management command to fix up the content type id in page revision json > Add support for Python 3.11 > --- > > Key: AIRAVATA-3698 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3698 > Project: Airavata > Issue Type: Improvement > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > Upgrade Wagtail and Django versions to get Python 3.11 support -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3698) Add support for Python 3.11
[ https://issues.apache.org/jira/browse/AIRAVATA-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17768811#comment-17768811 ] ASF subversion and git services commented on AIRAVATA-3698: --- Commit dd41406b112c7a6d5d2af74901f6de291b733ade in airavata-django-portal's branch refs/heads/AIRAVATA-3698-fix-page-revision-content-type from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=dd41406b ] AIRAVATA-3698 management command to fix up the content type id in page revision json > Add support for Python 3.11 > --- > > Key: AIRAVATA-3698 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3698 > Project: Airavata > Issue Type: Improvement > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > Upgrade Wagtail and Django versions to get Python 3.11 support -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767584#comment-17767584
]
ASF subversion and git services commented on AIRAVATA-3697:
---
Commit 71bc7c3d30e34daef0b308de15092f00cfe821cb in airavata's branch
refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=71bc7c3d30 ]
AIRAVATA-3697 Set lang and locale on WSGIDaemonProcess
Copying this change over from the template to the per gateway override files.
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
> Attachments: Photophysical properties and photodegradation mechanism
> of 2-(2′-furanyl)-1H-benzimidazole (Fuberidazole) - ScienceDirect.html
>
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
>
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767585#comment-17767585
]
ASF subversion and git services commented on AIRAVATA-3697:
---
Commit b3d68364c323bd03af0f435c54b6039a3743f148 in airavata's branch
refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=b3d68364c3 ]
Merge branch 'AIRAVATA-3697'
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
> Attachments: Photophysical properties and photodegradation mechanism
> of 2-(2′-furanyl)-1H-benzimidazole (Fuberidazole) - ScienceDirect.html
>
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/generic/base.py",
> line 70, in view
>return self.dispatch(request, *args, **kwargs)
> File
>
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767583#comment-17767583
]
ASF subversion and git services commented on AIRAVATA-3697:
---
Commit 6fcefb2c476fd73cc5070d378f0730713f5ac65f in airavata's branch
refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=6fcefb2c47 ]
AIRAVATA-3697 Switching to column level utf8 character set
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
> Attachments: Photophysical properties and photodegradation mechanism
> of 2-(2′-furanyl)-1H-benzimidazole (Fuberidazole) - ScienceDirect.html
>
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/generic/base.py",
> line 70, in view
>return
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17766801#comment-17766801
]
ASF subversion and git services commented on AIRAVATA-3697:
---
Commit 71bc7c3d30e34daef0b308de15092f00cfe821cb in airavata's branch
refs/heads/AIRAVATA-3697 from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=71bc7c3d30 ]
AIRAVATA-3697 Set lang and locale on WSGIDaemonProcess
Copying this change over from the template to the per gateway override files.
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
> Attachments: Photophysical properties and photodegradation mechanism
> of 2-(2′-furanyl)-1H-benzimidazole (Fuberidazole) - ScienceDirect.html
>
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
>
[jira] [Commented] (AIRAVATA-3702) User input files renamed with random extension when experiment launched
[ https://issues.apache.org/jira/browse/AIRAVATA-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743902#comment-17743902 ] ASF subversion and git services commented on AIRAVATA-3702: --- Commit 335ee8a488b7d64ebd85bbd02019df8d4916d587 in airavata-django-portal's branch refs/heads/master from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=335ee8a4 ] AIRAVATA-3702 Update SDK for bug fix > User input files renamed with random extension when experiment launched > --- > > Key: AIRAVATA-3702 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3702 > Project: Airavata > Issue Type: Bug > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > When a user uploads an input file, that files goes into a staging directory > (the user's "tmp" directory). If a user uploads two input files with the same > name, the second one will be renamed with some random characters added to the > end of the name (see > https://docs.djangoproject.com/en/4.2/ref/files/storage/#django.core.files.storage.Storage.get_alternative_name). > However, the original name of the file should be stored in the > replica_catalog and when the file is moved out of the staging directory and > into the experiment data directory, it should be renamed back to the > original, user-supplied filename. That's failing to happen. > Actually, it looks like the root of the problem is that the randomized > filename is getting stored in replica_catalog. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3698) Add support for Python 3.11
[ https://issues.apache.org/jira/browse/AIRAVATA-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743844#comment-17743844 ] ASF subversion and git services commented on AIRAVATA-3698: --- Commit f1fb9f198e675cf6513de22a7b18757d8518a0ee in airavata-django-portal's branch refs/heads/AIRAVATA-3698--Add-support-for-Python-3.11 from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=f1fb9f19 ] Merge branch 'develop' into AIRAVATA-3698--Add-support-for-Python-3.11 > Add support for Python 3.11 > --- > > Key: AIRAVATA-3698 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3698 > Project: Airavata > Issue Type: Improvement > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > Upgrade Wagtail and Django versions to get Python 3.11 support -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3702) User input files renamed with random extension when experiment launched
[ https://issues.apache.org/jira/browse/AIRAVATA-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743843#comment-17743843 ] ASF subversion and git services commented on AIRAVATA-3702: --- Commit 17d744ddb8b0987e4124dbc37ea1638f35f9f957 in airavata-django-portal's branch refs/heads/AIRAVATA-3698--Add-support-for-Python-3.11 from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=17d744dd ] Merge branch 'AIRAVATA-3702--User-input-files-renamed-with-random-extension-when-experiment-launched' into develop > User input files renamed with random extension when experiment launched > --- > > Key: AIRAVATA-3702 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3702 > Project: Airavata > Issue Type: Bug > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > When a user uploads an input file, that files goes into a staging directory > (the user's "tmp" directory). If a user uploads two input files with the same > name, the second one will be renamed with some random characters added to the > end of the name (see > https://docs.djangoproject.com/en/4.2/ref/files/storage/#django.core.files.storage.Storage.get_alternative_name). > However, the original name of the file should be stored in the > replica_catalog and when the file is moved out of the staging directory and > into the experiment data directory, it should be renamed back to the > original, user-supplied filename. That's failing to happen. > Actually, it looks like the root of the problem is that the randomized > filename is getting stored in replica_catalog. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3702) User input files renamed with random extension when experiment launched
[ https://issues.apache.org/jira/browse/AIRAVATA-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743842#comment-17743842 ] ASF subversion and git services commented on AIRAVATA-3702: --- Commit 335ee8a488b7d64ebd85bbd02019df8d4916d587 in airavata-django-portal's branch refs/heads/AIRAVATA-3698--Add-support-for-Python-3.11 from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=335ee8a4 ] AIRAVATA-3702 Update SDK for bug fix > User input files renamed with random extension when experiment launched > --- > > Key: AIRAVATA-3702 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3702 > Project: Airavata > Issue Type: Bug > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > When a user uploads an input file, that files goes into a staging directory > (the user's "tmp" directory). If a user uploads two input files with the same > name, the second one will be renamed with some random characters added to the > end of the name (see > https://docs.djangoproject.com/en/4.2/ref/files/storage/#django.core.files.storage.Storage.get_alternative_name). > However, the original name of the file should be stored in the > replica_catalog and when the file is moved out of the staging directory and > into the experiment data directory, it should be renamed back to the > original, user-supplied filename. That's failing to happen. > Actually, it looks like the root of the problem is that the randomized > filename is getting stored in replica_catalog. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3702) User input files renamed with random extension when experiment launched
[ https://issues.apache.org/jira/browse/AIRAVATA-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743837#comment-17743837 ] ASF subversion and git services commented on AIRAVATA-3702: --- Commit 335ee8a488b7d64ebd85bbd02019df8d4916d587 in airavata-django-portal's branch refs/heads/develop from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=335ee8a4 ] AIRAVATA-3702 Update SDK for bug fix > User input files renamed with random extension when experiment launched > --- > > Key: AIRAVATA-3702 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3702 > Project: Airavata > Issue Type: Bug > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > When a user uploads an input file, that files goes into a staging directory > (the user's "tmp" directory). If a user uploads two input files with the same > name, the second one will be renamed with some random characters added to the > end of the name (see > https://docs.djangoproject.com/en/4.2/ref/files/storage/#django.core.files.storage.Storage.get_alternative_name). > However, the original name of the file should be stored in the > replica_catalog and when the file is moved out of the staging directory and > into the experiment data directory, it should be renamed back to the > original, user-supplied filename. That's failing to happen. > Actually, it looks like the root of the problem is that the randomized > filename is getting stored in replica_catalog. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3702) User input files renamed with random extension when experiment launched
[ https://issues.apache.org/jira/browse/AIRAVATA-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743838#comment-17743838 ] ASF subversion and git services commented on AIRAVATA-3702: --- Commit 17d744ddb8b0987e4124dbc37ea1638f35f9f957 in airavata-django-portal's branch refs/heads/develop from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=17d744dd ] Merge branch 'AIRAVATA-3702--User-input-files-renamed-with-random-extension-when-experiment-launched' into develop > User input files renamed with random extension when experiment launched > --- > > Key: AIRAVATA-3702 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3702 > Project: Airavata > Issue Type: Bug > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > When a user uploads an input file, that files goes into a staging directory > (the user's "tmp" directory). If a user uploads two input files with the same > name, the second one will be renamed with some random characters added to the > end of the name (see > https://docs.djangoproject.com/en/4.2/ref/files/storage/#django.core.files.storage.Storage.get_alternative_name). > However, the original name of the file should be stored in the > replica_catalog and when the file is moved out of the staging directory and > into the experiment data directory, it should be renamed back to the > original, user-supplied filename. That's failing to happen. > Actually, it looks like the root of the problem is that the randomized > filename is getting stored in replica_catalog. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3702) User input files renamed with random extension when experiment launched
[ https://issues.apache.org/jira/browse/AIRAVATA-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743836#comment-17743836 ] ASF subversion and git services commented on AIRAVATA-3702: --- Commit 3a6edf04d58244e5092315dd26afaccc9b60a39e in airavata-django-portal-sdk's branch refs/heads/master from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata-django-portal-sdk.git;h=3a6edf0 ] AIRAVATA-3702 Preserve original filename of input file when it has the same name > User input files renamed with random extension when experiment launched > --- > > Key: AIRAVATA-3702 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3702 > Project: Airavata > Issue Type: Bug > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > When a user uploads an input file, that files goes into a staging directory > (the user's "tmp" directory). If a user uploads two input files with the same > name, the second one will be renamed with some random characters added to the > end of the name (see > https://docs.djangoproject.com/en/4.2/ref/files/storage/#django.core.files.storage.Storage.get_alternative_name). > However, the original name of the file should be stored in the > replica_catalog and when the file is moved out of the staging directory and > into the experiment data directory, it should be renamed back to the > original, user-supplied filename. That's failing to happen. > Actually, it looks like the root of the problem is that the randomized > filename is getting stored in replica_catalog. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739718#comment-17739718
]
ASF subversion and git services commented on AIRAVATA-3697:
---
Commit 6fcefb2c476fd73cc5070d378f0730713f5ac65f in airavata's branch
refs/heads/develop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=6fcefb2c47 ]
AIRAVATA-3697 Switching to column level utf8 character set
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
> Attachments: Photophysical properties and photodegradation mechanism
> of 2-(2′-furanyl)-1H-benzimidazole (Fuberidazole) - ScienceDirect.html
>
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/generic/base.py",
> line 70, in view
>return
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739719#comment-17739719
]
ASF subversion and git services commented on AIRAVATA-3697:
---
Commit 3a5b8358cf295ae33d151d59ceacc71279580aa4 in airavata's branch
refs/heads/develop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=3a5b8358cf ]
Merge branch 'AIRAVATA-3697' into develop
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
> Attachments: Photophysical properties and photodegradation mechanism
> of 2-(2′-furanyl)-1H-benzimidazole (Fuberidazole) - ScienceDirect.html
>
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/generic/base.py",
> line 70, in view
>return self.dispatch(request, *args,
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739717#comment-17739717
]
ASF subversion and git services commented on AIRAVATA-3697:
---
Commit 6fcefb2c476fd73cc5070d378f0730713f5ac65f in airavata's branch
refs/heads/AIRAVATA-3697 from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=6fcefb2c47 ]
AIRAVATA-3697 Switching to column level utf8 character set
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
> Attachments: Photophysical properties and photodegradation mechanism
> of 2-(2′-furanyl)-1H-benzimidazole (Fuberidazole) - ScienceDirect.html
>
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/generic/base.py",
> line 70, in view
>return
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739716#comment-17739716
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 161dd0a21b3853446f9d1b35755bdc25af6e5d72 in airavata's branch
refs/heads/AIRAVATA-3697 from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=161dd0a21b ]
AIRAVATA-3694 Fix comparison when var is a string
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739714#comment-17739714
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 09e6aaf4350cf35ed92555a81e27aeb33d952843 in airavata's branch
refs/heads/AIRAVATA-3697 from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=09e6aaf435 ]
AIRAVATA-3694 Ansible: configure data archive max ages for scigap hosted
gateways
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739715#comment-17739715
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit ca9d5155baa101fa57fe9c3ed9a13a9d5673ab2f in airavata's branch
refs/heads/AIRAVATA-3697 from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=ca9d5155ba ]
AIRAVATA-3694 Fix comparison when var is a string
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3698) Add support for Python 3.11
[ https://issues.apache.org/jira/browse/AIRAVATA-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738192#comment-17738192 ] ASF subversion and git services commented on AIRAVATA-3698: --- Commit e036cf70a0ce36f0683363047b239b91385e9b10 in airavata's branch refs/heads/develop from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata.git;h=e036cf70a0 ] AIRAVATA-3698 Switch Django deploy to Python 3.9 > Add support for Python 3.11 > --- > > Key: AIRAVATA-3698 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3698 > Project: Airavata > Issue Type: Improvement > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > Upgrade Wagtail and Django versions to get Python 3.11 support -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3698) Add support for Python 3.11
[ https://issues.apache.org/jira/browse/AIRAVATA-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738193#comment-17738193 ] ASF subversion and git services commented on AIRAVATA-3698: --- Commit 54c7adf1ab2d33b062eb77bf0741363529e4717f in airavata's branch refs/heads/develop from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata.git;h=54c7adf1ab ] AIRAVATA-3698 Update mod_wsgi to work with Py 3.9 > Add support for Python 3.11 > --- > > Key: AIRAVATA-3698 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3698 > Project: Airavata > Issue Type: Improvement > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > Upgrade Wagtail and Django versions to get Python 3.11 support -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3698) Add support for Python 3.11
[ https://issues.apache.org/jira/browse/AIRAVATA-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738194#comment-17738194 ] ASF subversion and git services commented on AIRAVATA-3698: --- Commit e8a711a2fcd673f03d80428965b6824a4cdec08a in airavata's branch refs/heads/develop from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata.git;h=e8a711a2fc ] AIRAVATA-3698 Referenc Django virtual env directory in vhost conf > Add support for Python 3.11 > --- > > Key: AIRAVATA-3698 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3698 > Project: Airavata > Issue Type: Improvement > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > Upgrade Wagtail and Django versions to get Python 3.11 support -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3698) Add support for Python 3.11
[ https://issues.apache.org/jira/browse/AIRAVATA-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738195#comment-17738195 ] ASF subversion and git services commented on AIRAVATA-3698: --- Commit 61841473880e7b765f400c253161233681057366 in airavata's branch refs/heads/develop from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata.git;h=6184147388 ] Merge branch 'AIRAVATA-3698--Add-support-for-Python-3.11' into develop > Add support for Python 3.11 > --- > > Key: AIRAVATA-3698 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3698 > Project: Airavata > Issue Type: Improvement > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > Upgrade Wagtail and Django versions to get Python 3.11 support -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3700) Creating new project bug: Cannot read properties of undefined (reading 'preventDefault')
[
https://issues.apache.org/jira/browse/AIRAVATA-3700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17737839#comment-17737839
]
ASF subversion and git services commented on AIRAVATA-3700:
---
Commit 3912deccf734595537cf657ec5c04f6827c8ae67 in airavata-django-portal's
branch refs/heads/AIRAVATA-3698--Add-support-for-Python-3.11 from Marcus
Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=3912decc ]
AIRAVATA-3700 Handle ENTER key submission of new project form
> Creating new project bug: Cannot read properties of undefined (reading
> 'preventDefault')
>
>
> Key: AIRAVATA-3700
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3700
> Project: Airavata
> Issue Type: Bug
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> {code}
> Frontend error: Cannot read properties of undefined (reading
> 'preventDefault'): "\"v-on handler\""
> stacktrace:
> onCreateProject
> (webpack://django-airavata-workspace-views/static/django_airavata_workspace/js/components/project/ProjectButtonNew.vue:45:)
> invokeWithErrorHandling
> (webpack://django../static/common/node_modules/vue/dist/vue.runtime.esm.js:1854:25)
> fns.slice
> (webpack://django../static/common/node_modules/vue/dist/vue.runtime.esm.js:2179:13)
> invokeWithErrorHandling
> (webpack://django../static/common/node_modules/vue/dist/vue.runtime.esm.js:1854:25)
> toArray
> (webpack://django../static/common/node_modules/vue/dist/vue.runtime.esm.js:3882:8)
> onSubmit
> (webpack://django-airavata-workspace-views/static/django_airavata_workspace/js/components/project/ProjectEditor.vue:97:)
> invokeWithErrorHandling
> (webpack://django../static/common/node_modules/vue/dist/vue.runtime.esm.js:1854:25)
> fns.slice
> (webpack://django../static/common/node_modules/vue/dist/vue.runtime.esm.js:2179:13)
> _wrapper
> (webpack://django../static/common/node_modules/vue/dist/vue.runtime.esm.js:6911:24)
> {code}
> h3. Steps to reproduce:
> 1. Click on the project sidebar in the Workspace.
> 2. Click on the *New Project +* button.
> 3. Enter a name for a new project.
> 4. Press the ENTER key.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3698) Add support for Python 3.11
[ https://issues.apache.org/jira/browse/AIRAVATA-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17737840#comment-17737840 ] ASF subversion and git services commented on AIRAVATA-3698: --- Commit d253b505f52e8562af2d6f0fe26927b0498762bb in airavata-django-portal's branch refs/heads/AIRAVATA-3698--Add-support-for-Python-3.11 from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=d253b505 ] Merge branch 'develop' into AIRAVATA-3698--Add-support-for-Python-3.11 > Add support for Python 3.11 > --- > > Key: AIRAVATA-3698 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3698 > Project: Airavata > Issue Type: Improvement > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > Upgrade Wagtail and Django versions to get Python 3.11 support -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3700) Creating new project bug: Cannot read properties of undefined (reading 'preventDefault')
[
https://issues.apache.org/jira/browse/AIRAVATA-3700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17737838#comment-17737838
]
ASF subversion and git services commented on AIRAVATA-3700:
---
Commit 3912deccf734595537cf657ec5c04f6827c8ae67 in airavata-django-portal's
branch refs/heads/develop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=3912decc ]
AIRAVATA-3700 Handle ENTER key submission of new project form
> Creating new project bug: Cannot read properties of undefined (reading
> 'preventDefault')
>
>
> Key: AIRAVATA-3700
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3700
> Project: Airavata
> Issue Type: Bug
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> {code}
> Frontend error: Cannot read properties of undefined (reading
> 'preventDefault'): "\"v-on handler\""
> stacktrace:
> onCreateProject
> (webpack://django-airavata-workspace-views/static/django_airavata_workspace/js/components/project/ProjectButtonNew.vue:45:)
> invokeWithErrorHandling
> (webpack://django../static/common/node_modules/vue/dist/vue.runtime.esm.js:1854:25)
> fns.slice
> (webpack://django../static/common/node_modules/vue/dist/vue.runtime.esm.js:2179:13)
> invokeWithErrorHandling
> (webpack://django../static/common/node_modules/vue/dist/vue.runtime.esm.js:1854:25)
> toArray
> (webpack://django../static/common/node_modules/vue/dist/vue.runtime.esm.js:3882:8)
> onSubmit
> (webpack://django-airavata-workspace-views/static/django_airavata_workspace/js/components/project/ProjectEditor.vue:97:)
> invokeWithErrorHandling
> (webpack://django../static/common/node_modules/vue/dist/vue.runtime.esm.js:1854:25)
> fns.slice
> (webpack://django../static/common/node_modules/vue/dist/vue.runtime.esm.js:2179:13)
> _wrapper
> (webpack://django../static/common/node_modules/vue/dist/vue.runtime.esm.js:6911:24)
> {code}
> h3. Steps to reproduce:
> 1. Click on the project sidebar in the Workspace.
> 2. Click on the *New Project +* button.
> 3. Enter a name for a new project.
> 4. Press the ENTER key.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3698) Add support for Python 3.11
[ https://issues.apache.org/jira/browse/AIRAVATA-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17737714#comment-17737714 ] ASF subversion and git services commented on AIRAVATA-3698: --- Commit e8a711a2fcd673f03d80428965b6824a4cdec08a in airavata's branch refs/heads/AIRAVATA-3698--Add-support-for-Python-3.11 from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata.git;h=e8a711a2fc ] AIRAVATA-3698 Referenc Django virtual env directory in vhost conf > Add support for Python 3.11 > --- > > Key: AIRAVATA-3698 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3698 > Project: Airavata > Issue Type: Improvement > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > Upgrade Wagtail and Django versions to get Python 3.11 support -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3698) Add support for Python 3.11
[ https://issues.apache.org/jira/browse/AIRAVATA-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17737325#comment-17737325 ] ASF subversion and git services commented on AIRAVATA-3698: --- Commit 54c7adf1ab2d33b062eb77bf0741363529e4717f in airavata's branch refs/heads/AIRAVATA-3698--Add-support-for-Python-3.11 from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata.git;h=54c7adf1ab ] AIRAVATA-3698 Update mod_wsgi to work with Py 3.9 > Add support for Python 3.11 > --- > > Key: AIRAVATA-3698 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3698 > Project: Airavata > Issue Type: Improvement > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > Upgrade Wagtail and Django versions to get Python 3.11 support -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3698) Add support for Python 3.11
[ https://issues.apache.org/jira/browse/AIRAVATA-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17737324#comment-17737324 ] ASF subversion and git services commented on AIRAVATA-3698: --- Commit e036cf70a0ce36f0683363047b239b91385e9b10 in airavata's branch refs/heads/AIRAVATA-3698--Add-support-for-Python-3.11 from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata.git;h=e036cf70a0 ] AIRAVATA-3698 Switch Django deploy to Python 3.9 > Add support for Python 3.11 > --- > > Key: AIRAVATA-3698 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3698 > Project: Airavata > Issue Type: Improvement > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > Upgrade Wagtail and Django versions to get Python 3.11 support -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3698) Add support for Python 3.11
[ https://issues.apache.org/jira/browse/AIRAVATA-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17737253#comment-17737253 ] ASF subversion and git services commented on AIRAVATA-3698: --- Commit 7dfe250751bc4be6f644cbc1a45acfdbae53e96c in airavata-django-portal's branch refs/heads/AIRAVATA-3698--Add-support-for-Python-3.11 from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=7dfe2507 ] AIRAVATA-3698 Pull in fixes for EntryPoint properties > Add support for Python 3.11 > --- > > Key: AIRAVATA-3698 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3698 > Project: Airavata > Issue Type: Improvement > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > Upgrade Wagtail and Django versions to get Python 3.11 support -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3698) Add support for Python 3.11
[ https://issues.apache.org/jira/browse/AIRAVATA-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17737246#comment-17737246 ] ASF subversion and git services commented on AIRAVATA-3698: --- Commit 1bb6b2ab5f4273188e91cadc28c236a931042086 in airavata-django-portal-commons's branch refs/heads/main from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata-django-portal-commons.git;h=1bb6b2a ] AIRAVATA-3698 Fix renames of EntryPoint properties > Add support for Python 3.11 > --- > > Key: AIRAVATA-3698 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3698 > Project: Airavata > Issue Type: Improvement > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > Upgrade Wagtail and Django versions to get Python 3.11 support -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3698) Add support for Python 3.11
[ https://issues.apache.org/jira/browse/AIRAVATA-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736230#comment-17736230 ] ASF subversion and git services commented on AIRAVATA-3698: --- Commit 53585786f4493b4765f778c4b617b7edd6e681f3 in airavata-django-portal's branch refs/heads/AIRAVATA-3698--Add-support-for-Python-3.11 from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=53585786 ] AIRAVATA-3698 Changing supported Python version to 3.9 - 3.11 > Add support for Python 3.11 > --- > > Key: AIRAVATA-3698 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3698 > Project: Airavata > Issue Type: Improvement > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > Upgrade Wagtail and Django versions to get Python 3.11 support -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3698) Add support for Python 3.11
[ https://issues.apache.org/jira/browse/AIRAVATA-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736209#comment-17736209 ] ASF subversion and git services commented on AIRAVATA-3698: --- Commit 5432028312d2c9130d1bfb99788dec87004213b7 in airavata-django-portal's branch refs/heads/AIRAVATA-3698--Add-support-for-Python-3.11 from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=54320283 ] AIRAVATA-3698 Fixing deprecation warnings > Add support for Python 3.11 > --- > > Key: AIRAVATA-3698 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3698 > Project: Airavata > Issue Type: Improvement > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > Upgrade Wagtail and Django versions to get Python 3.11 support -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3698) Add support for Python 3.11
[ https://issues.apache.org/jira/browse/AIRAVATA-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736210#comment-17736210 ] ASF subversion and git services commented on AIRAVATA-3698: --- Commit b47c4e4adb69ea5678ab0b5b45fec9775419d15e in airavata-django-portal's branch refs/heads/AIRAVATA-3698--Add-support-for-Python-3.11 from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=b47c4e4a ] AIRAVATA-3698 Fixing deprecation warnings in commons package > Add support for Python 3.11 > --- > > Key: AIRAVATA-3698 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3698 > Project: Airavata > Issue Type: Improvement > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > Upgrade Wagtail and Django versions to get Python 3.11 support -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3698) Add support for Python 3.11
[ https://issues.apache.org/jira/browse/AIRAVATA-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736207#comment-17736207 ] ASF subversion and git services commented on AIRAVATA-3698: --- Commit 703a37a6c399172c36cdda9f99d4dba6c92a334e in airavata-django-portal's branch refs/heads/AIRAVATA-3698--Add-support-for-Python-3.11 from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=703a37a6 ] AIRAVATA-3698 Upgrade to Wagtail 4.2 > Add support for Python 3.11 > --- > > Key: AIRAVATA-3698 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3698 > Project: Airavata > Issue Type: Improvement > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > Upgrade Wagtail and Django versions to get Python 3.11 support -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3698) Add support for Python 3.11
[ https://issues.apache.org/jira/browse/AIRAVATA-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736211#comment-17736211 ] ASF subversion and git services commented on AIRAVATA-3698: --- Commit 119c88e307e75acb86ec700a0cfe62264eadce55 in airavata-django-portal's branch refs/heads/AIRAVATA-3698--Add-support-for-Python-3.11 from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=119c88e3 ] AIRAVATA-3698 Upgrading to Django 4.2 > Add support for Python 3.11 > --- > > Key: AIRAVATA-3698 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3698 > Project: Airavata > Issue Type: Improvement > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > Upgrade Wagtail and Django versions to get Python 3.11 support -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3698) Add support for Python 3.11
[ https://issues.apache.org/jira/browse/AIRAVATA-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736205#comment-17736205 ] ASF subversion and git services commented on AIRAVATA-3698: --- Commit 28f1d838456ef0c3dfd227b350fc6a5b9c5a049d in airavata-django-portal's branch refs/heads/AIRAVATA-3698--Add-support-for-Python-3.11 from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=28f1d838 ] AIRAVATA-3698 Upgrade to Wagtail 3.0, WAGTAILADMIN_BASE_URL setting added > Add support for Python 3.11 > --- > > Key: AIRAVATA-3698 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3698 > Project: Airavata > Issue Type: Improvement > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > Upgrade Wagtail and Django versions to get Python 3.11 support -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3698) Add support for Python 3.11
[ https://issues.apache.org/jira/browse/AIRAVATA-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736208#comment-17736208 ] ASF subversion and git services commented on AIRAVATA-3698: --- Commit 47f09aecae5a42c5d20921b6b65d712734bc019a in airavata-django-portal's branch refs/heads/AIRAVATA-3698--Add-support-for-Python-3.11 from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=47f09aec ] AIRAVATA-3698 Upgrade to Wagtail 5.0, remove wagtailfontawesome > Add support for Python 3.11 > --- > > Key: AIRAVATA-3698 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3698 > Project: Airavata > Issue Type: Improvement > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > Upgrade Wagtail and Django versions to get Python 3.11 support -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3698) Add support for Python 3.11
[ https://issues.apache.org/jira/browse/AIRAVATA-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736206#comment-17736206 ] ASF subversion and git services commented on AIRAVATA-3698: --- Commit 839e35148d9bb4b8911e8f67c2aa000d580004a1 in airavata-django-portal's branch refs/heads/AIRAVATA-3698--Add-support-for-Python-3.11 from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=839e3514 ] AIRAVATA-3698 Upgrade to Wagtail 4.0 > Add support for Python 3.11 > --- > > Key: AIRAVATA-3698 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3698 > Project: Airavata > Issue Type: Improvement > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > Upgrade Wagtail and Django versions to get Python 3.11 support -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3698) Add support for Python 3.11
[ https://issues.apache.org/jira/browse/AIRAVATA-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736169#comment-17736169 ] ASF subversion and git services commented on AIRAVATA-3698: --- Commit f2c981106d9baea9cab620a6356e181a8222549a in airavata-django-portal-commons's branch refs/heads/main from Marcus Christie [ https://gitbox.apache.org/repos/asf?p=airavata-django-portal-commons.git;h=f2c9811 ] AIRAVATA-3698 Fixing deprecation warnings > Add support for Python 3.11 > --- > > Key: AIRAVATA-3698 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3698 > Project: Airavata > Issue Type: Improvement > Components: Django Portal >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > Upgrade Wagtail and Django versions to get Python 3.11 support -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729819#comment-17729819
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 161dd0a21b3853446f9d1b35755bdc25af6e5d72 in airavata's branch
refs/heads/develop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=161dd0a21b ]
AIRAVATA-3694 Fix comparison when var is a string
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729818#comment-17729818
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit ca9d5155baa101fa57fe9c3ed9a13a9d5673ab2f in airavata's branch
refs/heads/develop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=ca9d5155ba ]
AIRAVATA-3694 Fix comparison when var is a string
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729817#comment-17729817
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 161dd0a21b3853446f9d1b35755bdc25af6e5d72 in airavata's branch
refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=161dd0a21b ]
AIRAVATA-3694 Fix comparison when var is a string
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729816#comment-17729816
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit ca9d5155baa101fa57fe9c3ed9a13a9d5673ab2f in airavata's branch
refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=ca9d5155ba ]
AIRAVATA-3694 Fix comparison when var is a string
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729808#comment-17729808
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 09e6aaf4350cf35ed92555a81e27aeb33d952843 in airavata's branch
refs/heads/develop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=09e6aaf435 ]
AIRAVATA-3694 Ansible: configure data archive max ages for scigap hosted
gateways
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729502#comment-17729502
]
ASF subversion and git services commented on AIRAVATA-3697:
---
Commit 538745201bd07a88ae3696aa59fa6d8ae4e81ceb in airavata's branch
refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=538745201b ]
Merge branch 'AIRAVATA-3697' into develop
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
> Attachments: Photophysical properties and photodegradation mechanism
> of 2-(2′-furanyl)-1H-benzimidazole (Fuberidazole) - ScienceDirect.html
>
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/generic/base.py",
> line 70, in view
>return self.dispatch(request, *args,
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729501#comment-17729501
]
ASF subversion and git services commented on AIRAVATA-3697:
---
Commit f4cc483734c6f9c674c26e5f6b9627c4d7e9fc96 in airavata's branch
refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=f4cc483734 ]
AIRAVATA-3697 Set lang and locale on WSGIDaemonProcess
This is so that the Django portal code can handle filenames with UTF8
characters.
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
> Attachments: Photophysical properties and photodegradation mechanism
> of 2-(2′-furanyl)-1H-benzimidazole (Fuberidazole) - ScienceDirect.html
>
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
>
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729504#comment-17729504
]
ASF subversion and git services commented on AIRAVATA-3697:
---
Commit 8bf8ee00691c016d7357aba094fd43b093aebfe0 in airavata's branch
refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=8bf8ee0069 ]
Merge branch 'AIRAVATA-3697' into develop
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
> Attachments: Photophysical properties and photodegradation mechanism
> of 2-(2′-furanyl)-1H-benzimidazole (Fuberidazole) - ScienceDirect.html
>
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/generic/base.py",
> line 70, in view
>return self.dispatch(request, *args,
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729503#comment-17729503
]
ASF subversion and git services commented on AIRAVATA-3697:
---
Commit 33300327fc98d34ba5a62dbe75aea2678120a065 in airavata's branch
refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=33300327fc ]
AIRAVATA-3697 Db migrations and changing PRODUCT_NAME, REPLICA_NAME to clobs
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
> Attachments: Photophysical properties and photodegradation mechanism
> of 2-(2′-furanyl)-1H-benzimidazole (Fuberidazole) - ScienceDirect.html
>
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/generic/base.py",
> line 70, in view
>return
[jira] [Commented] (AIRAVATA-3696) Defer instantiating output view providers until after Django apps are ready
[
https://issues.apache.org/jira/browse/AIRAVATA-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729494#comment-17729494
]
ASF subversion and git services commented on AIRAVATA-3696:
---
Commit eef9d7e5bbc781291e5dc21ce5f683b17517b436 in airavata-django-portal's
branch refs/heads/staging from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=eef9d7e5 ]
AIRAVATA-3696 removing unused import
> Defer instantiating output view providers until after Django apps are ready
> ---
>
> Key: AIRAVATA-3696
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3696
> Project: Airavata
> Issue Type: Bug
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Currently in settings.py all output view providers are instantiated. However,
> they might import code that transitively imports Django models which is not
> allowed in Django until all apps have been loaded (see
> https://docs.djangoproject.com/en/3.2/ref/applications/#initialization-process).
> A better approach would be to instantiate the output view providers in the
> ApiConfig.ready().
> {code}
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/settings.py",
> line 644, in
> OUTPUT_VIEW_PROVIDERS[entry_point.name] = entry_point.load()()
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/pkg_resources/__init__.py",
> line 2517, in load
> return self.resolve()
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/pkg_resources/__init__.py",
> line 2523, in resolve
> module = __import__(self.module_name, fromlist=['__name__'], level=0)
> File
> "/Users/machrist/Airavata/django/regsnp-django-app/regsnp_django_app/output_views.py",
> line 32, in
> from airavata_django_portal_sdk import urls
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/airavata_django_portal_sdk/urls.py",
> line 6, in
> from . import views
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/airavata_django_portal_sdk/views.py",
> line 16, in
> from rest_framework.decorators import api_view
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/decorators.py",
> line 13, in
> from rest_framework.views import APIView
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/views.py",
> line 17, in
> from rest_framework.schemas import DefaultSchema
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/schemas/__init__.py",
> line 33, in
> authentication_classes=api_settings.DEFAULT_AUTHENTICATION_CLASSES,
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 225, in __getattr__
> val = perform_import(val, attr)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 168, in perform_import
> return [import_from_string(item, setting_name) for item in val]
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 168, in
> return [import_from_string(item, setting_name) for item in val]
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 177, in import_from_string
> return import_string(val)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/django/utils/module_loading.py",
> line 17, in import_string
> module = import_module(module_path)
> File
> "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/__init__.py",
> line 126, in import_module
> return _bootstrap._gcd_import(name[level:], package, level)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/authentication.py",
> line 6, in
> from django_airavata.apps.auth import utils
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/auth/utils.py",
> line 14, in
> from . import models
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/auth/models.py",
> line 17, in
> class EmailVerification(models.Model):
> File
>
[jira] [Commented] (AIRAVATA-3696) Defer instantiating output view providers until after Django apps are ready
[
https://issues.apache.org/jira/browse/AIRAVATA-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729493#comment-17729493
]
ASF subversion and git services commented on AIRAVATA-3696:
---
Commit 1bebb7eb5253de3a1cedb57372380ceb98eb8d1b in airavata-django-portal's
branch refs/heads/staging from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=1bebb7eb ]
AIRAVATA-3696 instantiate output view providers in ready()
> Defer instantiating output view providers until after Django apps are ready
> ---
>
> Key: AIRAVATA-3696
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3696
> Project: Airavata
> Issue Type: Bug
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Currently in settings.py all output view providers are instantiated. However,
> they might import code that transitively imports Django models which is not
> allowed in Django until all apps have been loaded (see
> https://docs.djangoproject.com/en/3.2/ref/applications/#initialization-process).
> A better approach would be to instantiate the output view providers in the
> ApiConfig.ready().
> {code}
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/settings.py",
> line 644, in
> OUTPUT_VIEW_PROVIDERS[entry_point.name] = entry_point.load()()
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/pkg_resources/__init__.py",
> line 2517, in load
> return self.resolve()
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/pkg_resources/__init__.py",
> line 2523, in resolve
> module = __import__(self.module_name, fromlist=['__name__'], level=0)
> File
> "/Users/machrist/Airavata/django/regsnp-django-app/regsnp_django_app/output_views.py",
> line 32, in
> from airavata_django_portal_sdk import urls
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/airavata_django_portal_sdk/urls.py",
> line 6, in
> from . import views
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/airavata_django_portal_sdk/views.py",
> line 16, in
> from rest_framework.decorators import api_view
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/decorators.py",
> line 13, in
> from rest_framework.views import APIView
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/views.py",
> line 17, in
> from rest_framework.schemas import DefaultSchema
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/schemas/__init__.py",
> line 33, in
> authentication_classes=api_settings.DEFAULT_AUTHENTICATION_CLASSES,
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 225, in __getattr__
> val = perform_import(val, attr)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 168, in perform_import
> return [import_from_string(item, setting_name) for item in val]
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 168, in
> return [import_from_string(item, setting_name) for item in val]
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 177, in import_from_string
> return import_string(val)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/django/utils/module_loading.py",
> line 17, in import_string
> module = import_module(module_path)
> File
> "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/__init__.py",
> line 126, in import_module
> return _bootstrap._gcd_import(name[level:], package, level)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/authentication.py",
> line 6, in
> from django_airavata.apps.auth import utils
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/auth/utils.py",
> line 14, in
> from . import models
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/auth/models.py",
> line 17, in
> class EmailVerification(models.Model):
> File
>
[jira] [Commented] (AIRAVATA-3696) Defer instantiating output view providers until after Django apps are ready
[
https://issues.apache.org/jira/browse/AIRAVATA-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729498#comment-17729498
]
ASF subversion and git services commented on AIRAVATA-3696:
---
Commit eef9d7e5bbc781291e5dc21ce5f683b17517b436 in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=eef9d7e5 ]
AIRAVATA-3696 removing unused import
> Defer instantiating output view providers until after Django apps are ready
> ---
>
> Key: AIRAVATA-3696
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3696
> Project: Airavata
> Issue Type: Bug
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Currently in settings.py all output view providers are instantiated. However,
> they might import code that transitively imports Django models which is not
> allowed in Django until all apps have been loaded (see
> https://docs.djangoproject.com/en/3.2/ref/applications/#initialization-process).
> A better approach would be to instantiate the output view providers in the
> ApiConfig.ready().
> {code}
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/settings.py",
> line 644, in
> OUTPUT_VIEW_PROVIDERS[entry_point.name] = entry_point.load()()
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/pkg_resources/__init__.py",
> line 2517, in load
> return self.resolve()
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/pkg_resources/__init__.py",
> line 2523, in resolve
> module = __import__(self.module_name, fromlist=['__name__'], level=0)
> File
> "/Users/machrist/Airavata/django/regsnp-django-app/regsnp_django_app/output_views.py",
> line 32, in
> from airavata_django_portal_sdk import urls
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/airavata_django_portal_sdk/urls.py",
> line 6, in
> from . import views
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/airavata_django_portal_sdk/views.py",
> line 16, in
> from rest_framework.decorators import api_view
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/decorators.py",
> line 13, in
> from rest_framework.views import APIView
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/views.py",
> line 17, in
> from rest_framework.schemas import DefaultSchema
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/schemas/__init__.py",
> line 33, in
> authentication_classes=api_settings.DEFAULT_AUTHENTICATION_CLASSES,
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 225, in __getattr__
> val = perform_import(val, attr)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 168, in perform_import
> return [import_from_string(item, setting_name) for item in val]
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 168, in
> return [import_from_string(item, setting_name) for item in val]
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 177, in import_from_string
> return import_string(val)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/django/utils/module_loading.py",
> line 17, in import_string
> module = import_module(module_path)
> File
> "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/__init__.py",
> line 126, in import_module
> return _bootstrap._gcd_import(name[level:], package, level)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/authentication.py",
> line 6, in
> from django_airavata.apps.auth import utils
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/auth/utils.py",
> line 14, in
> from . import models
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/auth/models.py",
> line 17, in
> class EmailVerification(models.Model):
> File
>
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729500#comment-17729500
]
ASF subversion and git services commented on AIRAVATA-3697:
---
Commit a64264c10391df4bddeaa8663e18c3e508382dfc in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=a64264c1 ]
AIRAVATA-3697 fix download of utf8 filename
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
> Attachments: Photophysical properties and photodegradation mechanism
> of 2-(2′-furanyl)-1H-benzimidazole (Fuberidazole) - ScienceDirect.html
>
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/generic/base.py",
> line 70, in view
>return
[jira] [Commented] (AIRAVATA-3696) Defer instantiating output view providers until after Django apps are ready
[
https://issues.apache.org/jira/browse/AIRAVATA-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729497#comment-17729497
]
ASF subversion and git services commented on AIRAVATA-3696:
---
Commit 1bebb7eb5253de3a1cedb57372380ceb98eb8d1b in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=1bebb7eb ]
AIRAVATA-3696 instantiate output view providers in ready()
> Defer instantiating output view providers until after Django apps are ready
> ---
>
> Key: AIRAVATA-3696
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3696
> Project: Airavata
> Issue Type: Bug
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Currently in settings.py all output view providers are instantiated. However,
> they might import code that transitively imports Django models which is not
> allowed in Django until all apps have been loaded (see
> https://docs.djangoproject.com/en/3.2/ref/applications/#initialization-process).
> A better approach would be to instantiate the output view providers in the
> ApiConfig.ready().
> {code}
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/settings.py",
> line 644, in
> OUTPUT_VIEW_PROVIDERS[entry_point.name] = entry_point.load()()
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/pkg_resources/__init__.py",
> line 2517, in load
> return self.resolve()
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/pkg_resources/__init__.py",
> line 2523, in resolve
> module = __import__(self.module_name, fromlist=['__name__'], level=0)
> File
> "/Users/machrist/Airavata/django/regsnp-django-app/regsnp_django_app/output_views.py",
> line 32, in
> from airavata_django_portal_sdk import urls
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/airavata_django_portal_sdk/urls.py",
> line 6, in
> from . import views
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/airavata_django_portal_sdk/views.py",
> line 16, in
> from rest_framework.decorators import api_view
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/decorators.py",
> line 13, in
> from rest_framework.views import APIView
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/views.py",
> line 17, in
> from rest_framework.schemas import DefaultSchema
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/schemas/__init__.py",
> line 33, in
> authentication_classes=api_settings.DEFAULT_AUTHENTICATION_CLASSES,
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 225, in __getattr__
> val = perform_import(val, attr)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 168, in perform_import
> return [import_from_string(item, setting_name) for item in val]
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 168, in
> return [import_from_string(item, setting_name) for item in val]
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 177, in import_from_string
> return import_string(val)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/django/utils/module_loading.py",
> line 17, in import_string
> module = import_module(module_path)
> File
> "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/__init__.py",
> line 126, in import_module
> return _bootstrap._gcd_import(name[level:], package, level)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/authentication.py",
> line 6, in
> from django_airavata.apps.auth import utils
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/auth/utils.py",
> line 14, in
> from . import models
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/auth/models.py",
> line 17, in
> class EmailVerification(models.Model):
> File
>
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729496#comment-17729496
]
ASF subversion and git services commented on AIRAVATA-3697:
---
Commit a64264c10391df4bddeaa8663e18c3e508382dfc in airavata-django-portal's
branch refs/heads/staging from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=a64264c1 ]
AIRAVATA-3697 fix download of utf8 filename
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
> Attachments: Photophysical properties and photodegradation mechanism
> of 2-(2′-furanyl)-1H-benzimidazole (Fuberidazole) - ScienceDirect.html
>
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/generic/base.py",
> line 70, in view
>return
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729495#comment-17729495
]
ASF subversion and git services commented on AIRAVATA-3697:
---
Commit f29a74da5994561f6bec89df39db6a72f54128ab in airavata-django-portal's
branch refs/heads/staging from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=f29a74da ]
AIRAVATA-3697 log error if filesystemencoding isn't 'utf-8'
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
> Attachments: Photophysical properties and photodegradation mechanism
> of 2-(2′-furanyl)-1H-benzimidazole (Fuberidazole) - ScienceDirect.html
>
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/generic/base.py",
> line 70, in view
>
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729499#comment-17729499
]
ASF subversion and git services commented on AIRAVATA-3697:
---
Commit f29a74da5994561f6bec89df39db6a72f54128ab in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=f29a74da ]
AIRAVATA-3697 log error if filesystemencoding isn't 'utf-8'
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
> Attachments: Photophysical properties and photodegradation mechanism
> of 2-(2′-furanyl)-1H-benzimidazole (Fuberidazole) - ScienceDirect.html
>
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/generic/base.py",
> line 70, in view
>
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17728863#comment-17728863
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 09e6aaf4350cf35ed92555a81e27aeb33d952843 in airavata's branch
refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=09e6aaf435 ]
AIRAVATA-3694 Ansible: configure data archive max ages for scigap hosted
gateways
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17724005#comment-17724005
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 8db03aa7458118819801a3371866cd08a0a172ef in airavata-django-portal's
branch refs/heads/develop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=8db03aa7 ]
AIRAVATA-3694 handle case where file was already deleted
This is to handle the case where a user directory is symlinked as an alias
and the files to delete end up listed more than once. Code just emits a warning
when this happens.
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17724006#comment-17724006
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 984b2614d671b0199f85d446b6ecf038e9e6ad48 in airavata-django-portal's
branch refs/heads/develop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=984b2614 ]
AIRAVATA-3694 Remove data retention message for now
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17724004#comment-17724004
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 984b2614d671b0199f85d446b6ecf038e9e6ad48 in airavata-django-portal's
branch refs/heads/staging from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=984b2614 ]
AIRAVATA-3694 Remove data retention message for now
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17724000#comment-17724000
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 984b2614d671b0199f85d446b6ecf038e9e6ad48 in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=984b2614 ]
AIRAVATA-3694 Remove data retention message for now
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717817#comment-17717817
]
ASF subversion and git services commented on AIRAVATA-3697:
---
Commit a64264c10391df4bddeaa8663e18c3e508382dfc in airavata-django-portal's
branch refs/heads/develop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=a64264c1 ]
AIRAVATA-3697 fix download of utf8 filename
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/generic/base.py",
> line 70, in view
>return self.dispatch(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 509, in
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717816#comment-17717816
]
ASF subversion and git services commented on AIRAVATA-3697:
---
Commit d0de482e134b2aac883d8c67da3a7481613ed97d in airavata-django-portal-sdk's
branch refs/heads/master from Marcus Christie
[
https://gitbox.apache.org/repos/asf?p=airavata-django-portal-sdk.git;h=d0de482 ]
AIRAVATA-3697 Set filename appropriately when downloading file with utf8
characters
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/generic/base.py",
> line 70, in view
>return self.dispatch(request, *args, **kwargs)
> File
>
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717794#comment-17717794
]
ASF subversion and git services commented on AIRAVATA-3697:
---
Commit f29a74da5994561f6bec89df39db6a72f54128ab in airavata-django-portal's
branch refs/heads/develop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=f29a74da ]
AIRAVATA-3697 log error if filesystemencoding isn't 'utf-8'
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/generic/base.py",
> line 70, in view
>return self.dispatch(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
>
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717370#comment-17717370
]
ASF subversion and git services commented on AIRAVATA-3697:
---
Commit 538745201bd07a88ae3696aa59fa6d8ae4e81ceb in airavata's branch
refs/heads/develop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=538745201b ]
Merge branch 'AIRAVATA-3697' into develop
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/generic/base.py",
> line 70, in view
>return self.dispatch(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 509, in dispatch
>response =
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717368#comment-17717368
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 7876cea94fcc8bccffdf9a89138a4b8fd34bf454 in airavata's branch
refs/heads/develop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=7876cea94f ]
AIRAVATA-3694 configured testdrive for user data archives
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717372#comment-17717372
]
ASF subversion and git services commented on AIRAVATA-3697:
---
Commit 8bf8ee00691c016d7357aba094fd43b093aebfe0 in airavata's branch
refs/heads/develop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=8bf8ee0069 ]
Merge branch 'AIRAVATA-3697' into develop
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/generic/base.py",
> line 70, in view
>return self.dispatch(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 509, in dispatch
>response =
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717369#comment-17717369
]
ASF subversion and git services commented on AIRAVATA-3697:
---
Commit f4cc483734c6f9c674c26e5f6b9627c4d7e9fc96 in airavata's branch
refs/heads/develop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=f4cc483734 ]
AIRAVATA-3697 Set lang and locale on WSGIDaemonProcess
This is so that the Django portal code can handle filenames with UTF8
characters.
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/generic/base.py",
> line 70, in view
>return self.dispatch(request, *args, **kwargs)
> File
>
[jira] [Commented] (AIRAVATA-3690) Add Citation to Airavata repos
[ https://issues.apache.org/jira/browse/AIRAVATA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717367#comment-17717367 ] ASF subversion and git services commented on AIRAVATA-3690: --- Commit 03175bd669150b83a6869327dcfd3d7afadddeaa in airavata's branch refs/heads/develop from Suresh Marru [ https://gitbox.apache.org/repos/asf?p=airavata.git;h=03175bd669 ] Adding Citation Info Addressing issue #AIRAVATA-3690 > Add Citation to Airavata repos > -- > > Key: AIRAVATA-3690 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3690 > Project: Airavata > Issue Type: Task >Reporter: Suresh Marru >Assignee: Suresh Marru >Priority: Major > > GitHub now integrates citation capability - > [https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-citation-files] > > We can include a preferred citation - > [https://github.com/citation-file-format/ruby-cff/issues/55] > More details are at - [https://citation-file-format.github.io/] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717371#comment-17717371
]
ASF subversion and git services commented on AIRAVATA-3697:
---
Commit 33300327fc98d34ba5a62dbe75aea2678120a065 in airavata's branch
refs/heads/develop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=33300327fc ]
AIRAVATA-3697 Db migrations and changing PRODUCT_NAME, REPLICA_NAME to clobs
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/generic/base.py",
> line 70, in view
>return self.dispatch(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line
[jira] [Commented] (AIRAVATA-3696) Defer instantiating output view providers until after Django apps are ready
[
https://issues.apache.org/jira/browse/AIRAVATA-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717259#comment-17717259
]
ASF subversion and git services commented on AIRAVATA-3696:
---
Commit eef9d7e5bbc781291e5dc21ce5f683b17517b436 in airavata-django-portal's
branch refs/heads/develop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=eef9d7e5 ]
AIRAVATA-3696 removing unused import
> Defer instantiating output view providers until after Django apps are ready
> ---
>
> Key: AIRAVATA-3696
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3696
> Project: Airavata
> Issue Type: Bug
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Currently in settings.py all output view providers are instantiated. However,
> they might import code that transitively imports Django models which is not
> allowed in Django until all apps have been loaded (see
> https://docs.djangoproject.com/en/3.2/ref/applications/#initialization-process).
> A better approach would be to instantiate the output view providers in the
> ApiConfig.ready().
> {code}
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/settings.py",
> line 644, in
> OUTPUT_VIEW_PROVIDERS[entry_point.name] = entry_point.load()()
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/pkg_resources/__init__.py",
> line 2517, in load
> return self.resolve()
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/pkg_resources/__init__.py",
> line 2523, in resolve
> module = __import__(self.module_name, fromlist=['__name__'], level=0)
> File
> "/Users/machrist/Airavata/django/regsnp-django-app/regsnp_django_app/output_views.py",
> line 32, in
> from airavata_django_portal_sdk import urls
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/airavata_django_portal_sdk/urls.py",
> line 6, in
> from . import views
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/airavata_django_portal_sdk/views.py",
> line 16, in
> from rest_framework.decorators import api_view
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/decorators.py",
> line 13, in
> from rest_framework.views import APIView
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/views.py",
> line 17, in
> from rest_framework.schemas import DefaultSchema
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/schemas/__init__.py",
> line 33, in
> authentication_classes=api_settings.DEFAULT_AUTHENTICATION_CLASSES,
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 225, in __getattr__
> val = perform_import(val, attr)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 168, in perform_import
> return [import_from_string(item, setting_name) for item in val]
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 168, in
> return [import_from_string(item, setting_name) for item in val]
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 177, in import_from_string
> return import_string(val)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/django/utils/module_loading.py",
> line 17, in import_string
> module = import_module(module_path)
> File
> "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/__init__.py",
> line 126, in import_module
> return _bootstrap._gcd_import(name[level:], package, level)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/authentication.py",
> line 6, in
> from django_airavata.apps.auth import utils
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/auth/utils.py",
> line 14, in
> from . import models
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/auth/models.py",
> line 17, in
> class EmailVerification(models.Model):
> File
>
[jira] [Commented] (AIRAVATA-3696) Defer instantiating output view providers until after Django apps are ready
[
https://issues.apache.org/jira/browse/AIRAVATA-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717244#comment-17717244
]
ASF subversion and git services commented on AIRAVATA-3696:
---
Commit 1bebb7eb5253de3a1cedb57372380ceb98eb8d1b in airavata-django-portal's
branch refs/heads/develop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=1bebb7eb ]
AIRAVATA-3696 instantiate output view providers in ready()
> Defer instantiating output view providers until after Django apps are ready
> ---
>
> Key: AIRAVATA-3696
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3696
> Project: Airavata
> Issue Type: Bug
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Currently in settings.py all output view providers are instantiated. However,
> they might import code that transitively imports Django models which is not
> allowed in Django until all apps have been loaded (see
> https://docs.djangoproject.com/en/3.2/ref/applications/#initialization-process).
> A better approach would be to instantiate the output view providers in the
> ApiConfig.ready().
> {code}
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/settings.py",
> line 644, in
> OUTPUT_VIEW_PROVIDERS[entry_point.name] = entry_point.load()()
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/pkg_resources/__init__.py",
> line 2517, in load
> return self.resolve()
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/pkg_resources/__init__.py",
> line 2523, in resolve
> module = __import__(self.module_name, fromlist=['__name__'], level=0)
> File
> "/Users/machrist/Airavata/django/regsnp-django-app/regsnp_django_app/output_views.py",
> line 32, in
> from airavata_django_portal_sdk import urls
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/airavata_django_portal_sdk/urls.py",
> line 6, in
> from . import views
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/airavata_django_portal_sdk/views.py",
> line 16, in
> from rest_framework.decorators import api_view
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/decorators.py",
> line 13, in
> from rest_framework.views import APIView
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/views.py",
> line 17, in
> from rest_framework.schemas import DefaultSchema
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/schemas/__init__.py",
> line 33, in
> authentication_classes=api_settings.DEFAULT_AUTHENTICATION_CLASSES,
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 225, in __getattr__
> val = perform_import(val, attr)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 168, in perform_import
> return [import_from_string(item, setting_name) for item in val]
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 168, in
> return [import_from_string(item, setting_name) for item in val]
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 177, in import_from_string
> return import_string(val)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/django/utils/module_loading.py",
> line 17, in import_string
> module = import_module(module_path)
> File
> "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/__init__.py",
> line 126, in import_module
> return _bootstrap._gcd_import(name[level:], package, level)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/authentication.py",
> line 6, in
> from django_airavata.apps.auth import utils
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/auth/utils.py",
> line 14, in
> from . import models
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/auth/models.py",
> line 17, in
> class EmailVerification(models.Model):
> File
>
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717218#comment-17717218
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 75a4e563dda44e84abae93d8d1838238a34b7a1d in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=75a4e563 ]
AIRAVATA-3694 --max-age option added to archive_user_data
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717212#comment-17717212
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 6a52ac3c2b968f46ecd9c2cd08317a97da8c5bef in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=6a52ac3c ]
AIRAVATA-3694 handle case where there is nothing to archive
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717192#comment-17717192
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 9e5ccfacff38ae70169103c0268368d898fbafa1 in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=9e5ccfac ]
AIRAVATA-3694 REST API and admin view to see when experiment data dir has been
archived
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3695) new optional list Thrift property causes error in thrift_utils if not present in request
[
https://issues.apache.org/jira/browse/AIRAVATA-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717201#comment-17717201
]
ASF subversion and git services commented on AIRAVATA-3695:
---
Commit 4442cde5e9f9a46ba6f4f3af8103980b810eeb45 in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=4442cde5 ]
AIRAVATA-3695 Bug fix for null reference
> new optional list Thrift property causes error in thrift_utils if not present
> in request
>
>
> Key: AIRAVATA-3695
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3695
> Project: Airavata
> Issue Type: Bug
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> {code}
> [2023-04-19 20:59:43,639 django_airavata.apps.api.exceptions:63 ERROR] API
> exception
> Traceback (most recent call last):
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/views.py",
> line 506, in dispatch
> response = handler(request, *args, **kwargs)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/mixins.py",
> line 68, in update
> self.perform_update(serializer)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/views.py",
> line 225, in perform_update
> experiment = serializer.save(
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/serializers.py",
> line 200, in save
> self.instance = self.update(self.instance, validated_data)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 148, in update
> return self.create(validated_data)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 144, in create
> params = self.process_nested_fields(validated_data)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 139, in process_nested_fields
> params[field_name] = serializer.create(
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 144, in create
> params = self.process_nested_fields(validated_data)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 129, in process_nested_fields
> if (params[field_name] is not None or
> KeyError: 'autoScheduledCompResourceSchedulingList'
> {code}
> thrift_utils.py assumes that every field will be present even when the field
> is optional/nullable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717213#comment-17717213
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 9e5ccfacff38ae70169103c0268368d898fbafa1 in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=9e5ccfac ]
AIRAVATA-3694 REST API and admin view to see when experiment data dir has been
archived
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3695) new optional list Thrift property causes error in thrift_utils if not present in request
[
https://issues.apache.org/jira/browse/AIRAVATA-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717222#comment-17717222
]
ASF subversion and git services commented on AIRAVATA-3695:
---
Commit 4442cde5e9f9a46ba6f4f3af8103980b810eeb45 in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=4442cde5 ]
AIRAVATA-3695 Bug fix for null reference
> new optional list Thrift property causes error in thrift_utils if not present
> in request
>
>
> Key: AIRAVATA-3695
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3695
> Project: Airavata
> Issue Type: Bug
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> {code}
> [2023-04-19 20:59:43,639 django_airavata.apps.api.exceptions:63 ERROR] API
> exception
> Traceback (most recent call last):
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/views.py",
> line 506, in dispatch
> response = handler(request, *args, **kwargs)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/mixins.py",
> line 68, in update
> self.perform_update(serializer)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/views.py",
> line 225, in perform_update
> experiment = serializer.save(
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/serializers.py",
> line 200, in save
> self.instance = self.update(self.instance, validated_data)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 148, in update
> return self.create(validated_data)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 144, in create
> params = self.process_nested_fields(validated_data)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 139, in process_nested_fields
> params[field_name] = serializer.create(
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 144, in create
> params = self.process_nested_fields(validated_data)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 129, in process_nested_fields
> if (params[field_name] is not None or
> KeyError: 'autoScheduledCompResourceSchedulingList'
> {code}
> thrift_utils.py assumes that every field will be present even when the field
> is optional/nullable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717215#comment-17717215
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit d0768e84bfd60814cf8f292afe380c1ef8bb9996 in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=d0768e84 ]
AIRAVATA-3694 Attempt to rollback the archive if something fails when deleting
archived data
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717219#comment-17717219
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 28b9ee7885d7aebeb60b2abfbbe1b1cba7712804 in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=28b9ee78 ]
AIRAVATA-3694 make the archive directory, including any missing parents
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3693) Windows support: use run-s to build frontend code and library
[
https://issues.apache.org/jira/browse/AIRAVATA-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717186#comment-17717186
]
ASF subversion and git services commented on AIRAVATA-3693:
---
Commit 2b5ba36b3cfc61fc1aa2130785c4e77a51bf11cf in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=2b5ba36b ]
AIRAVATA-3693 add npm-run-all to package.json
> Windows support: use run-s to build frontend code and library
> -
>
> Key: AIRAVATA-3693
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3693
> Project: Airavata
> Issue Type: Improvement
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Currently the common frontend code in the Django Portal builds the JS/CSS and
> the library by using {{&&}} to run the two build steps one after the other.
> But this only works in Bash shell environments. To work on Windows,
> [run-s|https://github.com/mysticatea/npm-run-all] is a better alternative.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717214#comment-17717214
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 2a63b583725ca2d28d5936c55080a155e6c0f713 in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=2a63b583 ]
AIRAVATA-3694 Documented new user data archive settings
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3693) Windows support: use run-s to build frontend code and library
[
https://issues.apache.org/jira/browse/AIRAVATA-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717205#comment-17717205
]
ASF subversion and git services commented on AIRAVATA-3693:
---
Commit 5ab69a4a0ece3ccdffc478e4301c9c975c6e6508 in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=5ab69a4a ]
AIRAVATA-3693 Add windows JS build to action
> Windows support: use run-s to build frontend code and library
> -
>
> Key: AIRAVATA-3693
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3693
> Project: Airavata
> Issue Type: Improvement
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Currently the common frontend code in the Django Portal builds the JS/CSS and
> the library by using {{&&}} to run the two build steps one after the other.
> But this only works in Bash shell environments. To work on Windows,
> [run-s|https://github.com/mysticatea/npm-run-all] is a better alternative.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717189#comment-17717189
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit e0abdebb508d867fd104f3e09c712c9c46cf1eb5 in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=e0abdebb ]
AIRAVATA-3694 minimum archive size setting
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717217#comment-17717217
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 086fb53fc8d816c76f5df69d5a33c4cb0e47bb6a in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=086fb53f ]
AIRAVATA-3694 Notice to user when experiment data is archived and also the
general archive policy
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3695) new optional list Thrift property causes error in thrift_utils if not present in request
[
https://issues.apache.org/jira/browse/AIRAVATA-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717221#comment-17717221
]
ASF subversion and git services commented on AIRAVATA-3695:
---
Commit 668357b0dc6660bd946cf943d6afe6b8966b8551 in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=668357b0 ]
AIRAVATA-3695 Bug fix for handling missing optional fields
> new optional list Thrift property causes error in thrift_utils if not present
> in request
>
>
> Key: AIRAVATA-3695
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3695
> Project: Airavata
> Issue Type: Bug
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> {code}
> [2023-04-19 20:59:43,639 django_airavata.apps.api.exceptions:63 ERROR] API
> exception
> Traceback (most recent call last):
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/views.py",
> line 506, in dispatch
> response = handler(request, *args, **kwargs)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/mixins.py",
> line 68, in update
> self.perform_update(serializer)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/views.py",
> line 225, in perform_update
> experiment = serializer.save(
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/serializers.py",
> line 200, in save
> self.instance = self.update(self.instance, validated_data)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 148, in update
> return self.create(validated_data)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 144, in create
> params = self.process_nested_fields(validated_data)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 139, in process_nested_fields
> params[field_name] = serializer.create(
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 144, in create
> params = self.process_nested_fields(validated_data)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 129, in process_nested_fields
> if (params[field_name] is not None or
> KeyError: 'autoScheduledCompResourceSchedulingList'
> {code}
> thrift_utils.py assumes that every field will be present even when the field
> is optional/nullable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3693) Windows support: use run-s to build frontend code and library
[
https://issues.apache.org/jira/browse/AIRAVATA-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717206#comment-17717206
]
ASF subversion and git services commented on AIRAVATA-3693:
---
Commit e56c3306f5f1a799d1d5fd1dc03951a10b62674a in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=e56c3306 ]
AIRAVATA-3693 windows JS build action set to use cmd shell
> Windows support: use run-s to build frontend code and library
> -
>
> Key: AIRAVATA-3693
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3693
> Project: Airavata
> Issue Type: Improvement
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Currently the common frontend code in the Django Portal builds the JS/CSS and
> the library by using {{&&}} to run the two build steps one after the other.
> But this only works in Bash shell environments. To work on Windows,
> [run-s|https://github.com/mysticatea/npm-run-all] is a better alternative.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717209#comment-17717209
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit b9dc12e4e27e4f7ef199b36590736ffb399b8f81 in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=b9dc12e4 ]
AIRAVATA-3694 Basic unarchive_user_data command
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717216#comment-17717216
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit e4aa3d4d99138634556d46628d747b1a40e077be in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=e4aa3d4d ]
AIRAVATA-3694 Reset modification time when unarchiving
i.e., 'touch' all of the unarchived files
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3693) Windows support: use run-s to build frontend code and library
[
https://issues.apache.org/jira/browse/AIRAVATA-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717185#comment-17717185
]
ASF subversion and git services commented on AIRAVATA-3693:
---
Commit e56c3306f5f1a799d1d5fd1dc03951a10b62674a in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=e56c3306 ]
AIRAVATA-3693 windows JS build action set to use cmd shell
> Windows support: use run-s to build frontend code and library
> -
>
> Key: AIRAVATA-3693
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3693
> Project: Airavata
> Issue Type: Improvement
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Currently the common frontend code in the Django Portal builds the JS/CSS and
> the library by using {{&&}} to run the two build steps one after the other.
> But this only works in Bash shell environments. To work on Windows,
> [run-s|https://github.com/mysticatea/npm-run-all] is a better alternative.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717223#comment-17717223
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 15f91433b519f096f0c29869d9b3e9ba2c33c505 in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=15f91433 ]
AIRAVATA-3694 log exception when archive fails so it generates an error email
in production
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
