[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739718#comment-17739718
]
ASF subversion and git services commented on AIRAVATA-3697:
---
Commit 6fcefb2c476fd73cc5070d378f0730713f5ac65f in airavata's branch
refs/heads/develop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=6fcefb2c47 ]
AIRAVATA-3697 Switching to column level utf8 character set
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
> Attachments: Photophysical properties and photodegradation mechanism
> of 2-(2′-furanyl)-1H-benzimidazole (Fuberidazole) - ScienceDirect.html
>
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/generic/base.py",
> line 70, in view
>return
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739719#comment-17739719
]
ASF subversion and git services commented on AIRAVATA-3697:
---
Commit 3a5b8358cf295ae33d151d59ceacc71279580aa4 in airavata's branch
refs/heads/develop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=3a5b8358cf ]
Merge branch 'AIRAVATA-3697' into develop
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
> Attachments: Photophysical properties and photodegradation mechanism
> of 2-(2′-furanyl)-1H-benzimidazole (Fuberidazole) - ScienceDirect.html
>
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/generic/base.py",
> line 70, in view
>return self.dispatch(request, *args,
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739717#comment-17739717
]
ASF subversion and git services commented on AIRAVATA-3697:
---
Commit 6fcefb2c476fd73cc5070d378f0730713f5ac65f in airavata's branch
refs/heads/AIRAVATA-3697 from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=6fcefb2c47 ]
AIRAVATA-3697 Switching to column level utf8 character set
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
> Attachments: Photophysical properties and photodegradation mechanism
> of 2-(2′-furanyl)-1H-benzimidazole (Fuberidazole) - ScienceDirect.html
>
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/generic/base.py",
> line 70, in view
>return
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739716#comment-17739716
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 161dd0a21b3853446f9d1b35755bdc25af6e5d72 in airavata's branch
refs/heads/AIRAVATA-3697 from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=161dd0a21b ]
AIRAVATA-3694 Fix comparison when var is a string
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739714#comment-17739714
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 09e6aaf4350cf35ed92555a81e27aeb33d952843 in airavata's branch
refs/heads/AIRAVATA-3697 from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=09e6aaf435 ]
AIRAVATA-3694 Ansible: configure data archive max ages for scigap hosted
gateways
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739715#comment-17739715
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit ca9d5155baa101fa57fe9c3ed9a13a9d5673ab2f in airavata's branch
refs/heads/AIRAVATA-3697 from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=ca9d5155ba ]
AIRAVATA-3694 Fix comparison when var is a string
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739683#comment-17739683
]
Marcus Christie commented on AIRAVATA-3697:
---
Migration script for the changes mentioned in the previous comment:
{code:sql}
alter table DATA_PRODUCT modify PRODUCT_NAME text character set utf8mb4 collate
utf8mb4_bin;
alter table DATA_PRODUCT modify PRODUCT_DESCRIPTION varchar(255) character set
utf8mb4 collate utf8mb4_bin;
alter table DATA_REPLICA_LOCATION modify REPLICA_NAME text character set
utf8mb4 collate utf8mb4_bin;
alter table DATA_REPLICA_LOCATION modify REPLICA_DESCRIPTION varchar(255)
character set utf8mb4 collate utf8mb4_bin;
alter table DATA_REPLICA_LOCATION modify FILE_PATH varchar(1024) character set
utf8mb4 collate utf8mb4_bin;
{code}
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
> Attachments: Photophysical properties and photodegradation mechanism
> of 2-(2′-furanyl)-1H-benzimidazole (Fuberidazole) - ScienceDirect.html
>
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args,
