[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717370#comment-17717370
]
ASF subversion and git services commented on AIRAVATA-3697:
---
Commit 538745201bd07a88ae3696aa59fa6d8ae4e81ceb in airavata's branch
refs/heads/develop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=538745201b ]
Merge branch 'AIRAVATA-3697' into develop
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/generic/base.py",
> line 70, in view
>return self.dispatch(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 509, in dispatch
>response =
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717368#comment-17717368
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 7876cea94fcc8bccffdf9a89138a4b8fd34bf454 in airavata's branch
refs/heads/develop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=7876cea94f ]
AIRAVATA-3694 configured testdrive for user data archives
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717372#comment-17717372
]
ASF subversion and git services commented on AIRAVATA-3697:
---
Commit 8bf8ee00691c016d7357aba094fd43b093aebfe0 in airavata's branch
refs/heads/develop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=8bf8ee0069 ]
Merge branch 'AIRAVATA-3697' into develop
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/generic/base.py",
> line 70, in view
>return self.dispatch(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 509, in dispatch
>response =
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717369#comment-17717369
]
ASF subversion and git services commented on AIRAVATA-3697:
---
Commit f4cc483734c6f9c674c26e5f6b9627c4d7e9fc96 in airavata's branch
refs/heads/develop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=f4cc483734 ]
AIRAVATA-3697 Set lang and locale on WSGIDaemonProcess
This is so that the Django portal code can handle filenames with UTF8
characters.
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/generic/base.py",
> line 70, in view
>return self.dispatch(request, *args, **kwargs)
> File
>
[jira] [Commented] (AIRAVATA-3690) Add Citation to Airavata repos
[ https://issues.apache.org/jira/browse/AIRAVATA-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717367#comment-17717367 ] ASF subversion and git services commented on AIRAVATA-3690: --- Commit 03175bd669150b83a6869327dcfd3d7afadddeaa in airavata's branch refs/heads/develop from Suresh Marru [ https://gitbox.apache.org/repos/asf?p=airavata.git;h=03175bd669 ] Adding Citation Info Addressing issue #AIRAVATA-3690 > Add Citation to Airavata repos > -- > > Key: AIRAVATA-3690 > URL: https://issues.apache.org/jira/browse/AIRAVATA-3690 > Project: Airavata > Issue Type: Task >Reporter: Suresh Marru >Assignee: Suresh Marru >Priority: Major > > GitHub now integrates citation capability - > [https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-citation-files] > > We can include a preferred citation - > [https://github.com/citation-file-format/ruby-cff/issues/55] > More details are at - [https://citation-file-format.github.io/] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717371#comment-17717371
]
ASF subversion and git services commented on AIRAVATA-3697:
---
Commit 33300327fc98d34ba5a62dbe75aea2678120a065 in airavata's branch
refs/heads/develop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=33300327fc ]
AIRAVATA-3697 Db migrations and changing PRODUCT_NAME, REPLICA_NAME to clobs
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/generic/base.py",
> line 70, in view
>return self.dispatch(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717363#comment-17717363
]
Marcus Christie commented on AIRAVATA-3697:
---
Fixed the previous error by making PRODUCT_NAME and REPLICA_NAME CLOBs:
{code}
alter table DATA_REPLICA_LOCATION modify REPLICA_NAME text;
alter table DATA_PRODUCT modify PRODUCT_NAME text;
{code}
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/generic/base.py",
> line 70, in view
>return self.dispatch(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 509, in dispatch
>response = self.handle_exception(exc)
> File
>
[jira] [Comment Edited] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717360#comment-17717360
]
Marcus Christie edited comment on AIRAVATA-3697 at 4/27/23 7:44 PM:
I'm now getting this error
{code}
Caused by: org.apache.openjpa.persistence.PersistenceException: (conn:12787)
Data too long for column 'REPLICA_NAME' at row 1 {prepstmnt 1166842203
INSERT INTO DATA_REPLICA_LOCATION (REPLICA_ID, CREATION_TIME,
FILE_PATH, LAST_MODIFIED_TIME, PRODUCT_URI,
REPLICA_DESCRIPTION, REPLICA_LOCATION_CATEGORY, REPLICA_NAME,
REPLICA_PERSISTENT_TYPE, STORAGE_RESOURCE_ID, VALID_UNTIL_TIME)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
{code}
I'll see if I can make the REPLICA_NAME column a CLOB.
was (Author: marcuschristie):
I'm now getting this error
```
Caused by: org.apache.openjpa.persistence.PersistenceException: (conn:12787)
Data too long for column 'REPLICA_NAME' at row 1 {prepstmnt 1166842203
INSERT INTO DATA_REPLICA_LOCATION (REPLICA_ID, CREATION_TIME,
FILE_PATH, LAST_MODIFIED_TIME, PRODUCT_URI,
REPLICA_DESCRIPTION, REPLICA_LOCATION_CATEGORY, REPLICA_NAME,
REPLICA_PERSISTENT_TYPE, STORAGE_RESOURCE_ID, VALID_UNTIL_TIME)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
```
I'll see if I can make the REPLICA_NAME column a CLOB.
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed),
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717360#comment-17717360
]
Marcus Christie commented on AIRAVATA-3697:
---
I'm now getting this error
```
Caused by: org.apache.openjpa.persistence.PersistenceException: (conn:12787)
Data too long for column 'REPLICA_NAME' at row 1 {prepstmnt 1166842203
INSERT INTO DATA_REPLICA_LOCATION (REPLICA_ID, CREATION_TIME,
FILE_PATH, LAST_MODIFIED_TIME, PRODUCT_URI,
REPLICA_DESCRIPTION, REPLICA_LOCATION_CATEGORY, REPLICA_NAME,
REPLICA_PERSISTENT_TYPE, STORAGE_RESOURCE_ID, VALID_UNTIL_TIME)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
```
I'll see if I can make the REPLICA_NAME column a CLOB.
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
>return view_func(*args, **kwargs)
> File
>
[jira] [Comment Edited] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717358#comment-17717358
]
Marcus Christie edited comment on AIRAVATA-3697 at 4/27/23 7:40 PM:
Ran the following in a development environment to test out converting the
replica_catalog to utf8:
{code}
SET FOREIGN_KEY_CHECKS=0;
alter database replica_catalog CHARACTER SET = utf8 COLLATE = utf8_bin;
alter table DATA_PRODUCT convert to character set utf8 collate utf8_bin;
alter table DATA_REPLICA_LOCATION convert to character set utf8 collate
utf8_bin;
alter table DATA_PRODUCT_METADATA convert to character set utf8 collate
utf8_bin;
alter table DATA_REPLICA_METADATA convert to character set utf8 collate
utf8_bin;
alter table CONFIGURATION convert to character set utf8 collate utf8_bin;
SET FOREIGN_KEY_CHECKS=1;
{code}
was (Author: marcuschristie):
Ran the following in a development environment to test out converting the
replica_catalog to utf8:
```
SET FOREIGN_KEY_CHECKS=0;
alter database replica_catalog CHARACTER SET = utf8 COLLATE = utf8_bin;
alter table DATA_PRODUCT convert to character set utf8 collate utf8_bin;
alter table DATA_REPLICA_LOCATION convert to character set utf8 collate
utf8_bin;
alter table DATA_PRODUCT_METADATA convert to character set utf8 collate
utf8_bin;
alter table DATA_REPLICA_METADATA convert to character set utf8 collate
utf8_bin;
alter table CONFIGURATION convert to character set utf8 collate utf8_bin;
SET FOREIGN_KEY_CHECKS=1;
```
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s =
[jira] [Commented] (AIRAVATA-3697) Handle unicode characters in file names
[
https://issues.apache.org/jira/browse/AIRAVATA-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717358#comment-17717358
]
Marcus Christie commented on AIRAVATA-3697:
---
Ran the following in a development environment to test out converting the
replica_catalog to utf8:
```
SET FOREIGN_KEY_CHECKS=0;
alter database replica_catalog CHARACTER SET = utf8 COLLATE = utf8_bin;
alter table DATA_PRODUCT convert to character set utf8 collate utf8_bin;
alter table DATA_REPLICA_LOCATION convert to character set utf8 collate
utf8_bin;
alter table DATA_PRODUCT_METADATA convert to character set utf8 collate
utf8_bin;
alter table DATA_REPLICA_METADATA convert to character set utf8 collate
utf8_bin;
alter table CONFIGURATION convert to character set utf8 collate utf8_bin;
SET FOREIGN_KEY_CHECKS=1;
```
> Handle unicode characters in file names
> ---
>
> Key: AIRAVATA-3697
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API, Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Getting the following Django error when browsing experiment output files with
> unicode characters in the file names:
> {code}
> Traceback (most recent call last):
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
> line 506, in dispatch
>response = handler(request, *args, **kwargs)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1467, in get
>return self._create_response(request, experiment_id, path)
> File
> "/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
> line 1471, in _create_response
>directories, files = user_storage.list_experiment_dir(request,
> experiment_id, path)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 629, in list_experiment_dir
>backend=backend, owner=experiment.userName)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
> line 760, in _get_data_product_uri
>if user_file.exists():
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
> line 808, in exists
>return self.query.has_results(using=self.db)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
> line 561, in has_results
>return compiler.has_results()
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1145, in has_results
>return bool(self.execute_sql(SINGLE))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
> line 1175, in execute_sql
>cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 66, in execute
>return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 75, in _execute_with_wrappers
>return executor(sql, params, many, context)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>return self.cursor.execute(sql, params)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
> line 73, in execute
>return self.cursor.execute(query, args)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
> line 199, in execute
>args = tuple(map(db.literal, args))
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
> line 280, in literal
>s = self.string_literal(o.encode(self.encoding))
> During handling of the above exception ('utf-8' codec can't encode characters
> in position 197-199: surrogates not allowed), another exception occurred:
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
> line 47, in inner
>response = get_response(request)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
> line 181, in _get_response
>response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File
> "/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
> line 54, in
[jira] [Created] (AIRAVATA-3697) Handle unicode characters in file names
Marcus Christie created AIRAVATA-3697:
-
Summary: Handle unicode characters in file names
Key: AIRAVATA-3697
URL: https://issues.apache.org/jira/browse/AIRAVATA-3697
Project: Airavata
Issue Type: Bug
Components: Airavata API, Django Portal
Reporter: Marcus Christie
Assignee: Marcus Christie
Getting the following Django error when browsing experiment output files with
unicode characters in the file names:
{code}
Traceback (most recent call last):
File
"/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
line 506, in dispatch
response = handler(request, *args, **kwargs)
File
"/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
line 1467, in get
return self._create_response(request, experiment_id, path)
File
"/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/views.py",
line 1471, in _create_response
directories, files = user_storage.list_experiment_dir(request,
experiment_id, path)
File
"/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
line 629, in list_experiment_dir
backend=backend, owner=experiment.userName)
File
"/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/airavata_django_portal_sdk/user_storage/api.py",
line 760, in _get_data_product_uri
if user_file.exists():
File
"/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/query.py",
line 808, in exists
return self.query.has_results(using=self.db)
File
"/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/query.py",
line 561, in has_results
return compiler.has_results()
File
"/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
line 1145, in has_results
return bool(self.execute_sql(SINGLE))
File
"/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py",
line 1175, in execute_sql
cursor.execute(sql, params)
File
"/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
line 66, in execute
return self._execute_with_wrappers(sql, params, many=False,
executor=self._execute)
File
"/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
line 75, in _execute_with_wrappers
return executor(sql, params, many, context)
File
"/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/utils.py",
line 84, in _execute
return self.cursor.execute(sql, params)
File
"/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/db/backends/mysql/base.py",
line 73, in execute
return self.cursor.execute(query, args)
File
"/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/cursors.py",
line 199, in execute
args = tuple(map(db.literal, args))
File
"/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/MySQLdb/connections.py",
line 280, in literal
s = self.string_literal(o.encode(self.encoding))
During handling of the above exception ('utf-8' codec can't encode characters
in position 197-199: surrogates not allowed), another exception occurred:
File
"/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py",
line 47, in inner
response = get_response(request)
File
"/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/core/handlers/base.py",
line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File
"/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/decorators/csrf.py",
line 54, in wrapped_view
return view_func(*args, **kwargs)
File
"/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/django/views/generic/base.py",
line 70, in view
return self.dispatch(request, *args, **kwargs)
File
"/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
line 509, in dispatch
response = self.handle_exception(exc)
File
"/var/www/portals/django-seagrid/venv/lib64/python3.6/site-packages/rest_framework/views.py",
line 466, in handle_exception
response = exception_handler(exc, context)
File
"/var/www/portals/django-seagrid/airavata-django-portal/django_airavata/apps/api/exceptions.py",
line 54, in custom_exception_handler
log.error("API exception", exc_info=exc, extra={'request':
context['request']})
File "/usr/lib64/python3.6/logging/__init__.py", line 1337, in error
self._log(ERROR, msg, args, **kwargs)
File "/usr/lib64/python3.6/logging/__init__.py", line 1444, in _log
self.handle(record)
File
[jira] [Commented] (AIRAVATA-3696) Defer instantiating output view providers until after Django apps are ready
[
https://issues.apache.org/jira/browse/AIRAVATA-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717259#comment-17717259
]
ASF subversion and git services commented on AIRAVATA-3696:
---
Commit eef9d7e5bbc781291e5dc21ce5f683b17517b436 in airavata-django-portal's
branch refs/heads/develop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=eef9d7e5 ]
AIRAVATA-3696 removing unused import
> Defer instantiating output view providers until after Django apps are ready
> ---
>
> Key: AIRAVATA-3696
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3696
> Project: Airavata
> Issue Type: Bug
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Currently in settings.py all output view providers are instantiated. However,
> they might import code that transitively imports Django models which is not
> allowed in Django until all apps have been loaded (see
> https://docs.djangoproject.com/en/3.2/ref/applications/#initialization-process).
> A better approach would be to instantiate the output view providers in the
> ApiConfig.ready().
> {code}
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/settings.py",
> line 644, in
> OUTPUT_VIEW_PROVIDERS[entry_point.name] = entry_point.load()()
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/pkg_resources/__init__.py",
> line 2517, in load
> return self.resolve()
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/pkg_resources/__init__.py",
> line 2523, in resolve
> module = __import__(self.module_name, fromlist=['__name__'], level=0)
> File
> "/Users/machrist/Airavata/django/regsnp-django-app/regsnp_django_app/output_views.py",
> line 32, in
> from airavata_django_portal_sdk import urls
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/airavata_django_portal_sdk/urls.py",
> line 6, in
> from . import views
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/airavata_django_portal_sdk/views.py",
> line 16, in
> from rest_framework.decorators import api_view
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/decorators.py",
> line 13, in
> from rest_framework.views import APIView
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/views.py",
> line 17, in
> from rest_framework.schemas import DefaultSchema
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/schemas/__init__.py",
> line 33, in
> authentication_classes=api_settings.DEFAULT_AUTHENTICATION_CLASSES,
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 225, in __getattr__
> val = perform_import(val, attr)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 168, in perform_import
> return [import_from_string(item, setting_name) for item in val]
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 168, in
> return [import_from_string(item, setting_name) for item in val]
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 177, in import_from_string
> return import_string(val)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/django/utils/module_loading.py",
> line 17, in import_string
> module = import_module(module_path)
> File
> "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/__init__.py",
> line 126, in import_module
> return _bootstrap._gcd_import(name[level:], package, level)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/authentication.py",
> line 6, in
> from django_airavata.apps.auth import utils
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/auth/utils.py",
> line 14, in
> from . import models
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/auth/models.py",
> line 17, in
> class EmailVerification(models.Model):
> File
>
[jira] [Resolved] (AIRAVATA-3696) Defer instantiating output view providers until after Django apps are ready
[
https://issues.apache.org/jira/browse/AIRAVATA-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Christie resolved AIRAVATA-3696.
---
Resolution: Fixed
> Defer instantiating output view providers until after Django apps are ready
> ---
>
> Key: AIRAVATA-3696
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3696
> Project: Airavata
> Issue Type: Bug
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Currently in settings.py all output view providers are instantiated. However,
> they might import code that transitively imports Django models which is not
> allowed in Django until all apps have been loaded (see
> https://docs.djangoproject.com/en/3.2/ref/applications/#initialization-process).
> A better approach would be to instantiate the output view providers in the
> ApiConfig.ready().
> {code}
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/settings.py",
> line 644, in
> OUTPUT_VIEW_PROVIDERS[entry_point.name] = entry_point.load()()
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/pkg_resources/__init__.py",
> line 2517, in load
> return self.resolve()
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/pkg_resources/__init__.py",
> line 2523, in resolve
> module = __import__(self.module_name, fromlist=['__name__'], level=0)
> File
> "/Users/machrist/Airavata/django/regsnp-django-app/regsnp_django_app/output_views.py",
> line 32, in
> from airavata_django_portal_sdk import urls
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/airavata_django_portal_sdk/urls.py",
> line 6, in
> from . import views
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/airavata_django_portal_sdk/views.py",
> line 16, in
> from rest_framework.decorators import api_view
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/decorators.py",
> line 13, in
> from rest_framework.views import APIView
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/views.py",
> line 17, in
> from rest_framework.schemas import DefaultSchema
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/schemas/__init__.py",
> line 33, in
> authentication_classes=api_settings.DEFAULT_AUTHENTICATION_CLASSES,
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 225, in __getattr__
> val = perform_import(val, attr)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 168, in perform_import
> return [import_from_string(item, setting_name) for item in val]
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 168, in
> return [import_from_string(item, setting_name) for item in val]
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 177, in import_from_string
> return import_string(val)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/django/utils/module_loading.py",
> line 17, in import_string
> module = import_module(module_path)
> File
> "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/__init__.py",
> line 126, in import_module
> return _bootstrap._gcd_import(name[level:], package, level)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/authentication.py",
> line 6, in
> from django_airavata.apps.auth import utils
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/auth/utils.py",
> line 14, in
> from . import models
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/auth/models.py",
> line 17, in
> class EmailVerification(models.Model):
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/django/db/models/base.py",
> line 108, in __new__
> app_config = apps.get_containing_app_config(module)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/django/apps/registry.py",
>
[jira] [Commented] (AIRAVATA-3696) Defer instantiating output view providers until after Django apps are ready
[
https://issues.apache.org/jira/browse/AIRAVATA-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717244#comment-17717244
]
ASF subversion and git services commented on AIRAVATA-3696:
---
Commit 1bebb7eb5253de3a1cedb57372380ceb98eb8d1b in airavata-django-portal's
branch refs/heads/develop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=1bebb7eb ]
AIRAVATA-3696 instantiate output view providers in ready()
> Defer instantiating output view providers until after Django apps are ready
> ---
>
> Key: AIRAVATA-3696
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3696
> Project: Airavata
> Issue Type: Bug
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Currently in settings.py all output view providers are instantiated. However,
> they might import code that transitively imports Django models which is not
> allowed in Django until all apps have been loaded (see
> https://docs.djangoproject.com/en/3.2/ref/applications/#initialization-process).
> A better approach would be to instantiate the output view providers in the
> ApiConfig.ready().
> {code}
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/settings.py",
> line 644, in
> OUTPUT_VIEW_PROVIDERS[entry_point.name] = entry_point.load()()
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/pkg_resources/__init__.py",
> line 2517, in load
> return self.resolve()
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/pkg_resources/__init__.py",
> line 2523, in resolve
> module = __import__(self.module_name, fromlist=['__name__'], level=0)
> File
> "/Users/machrist/Airavata/django/regsnp-django-app/regsnp_django_app/output_views.py",
> line 32, in
> from airavata_django_portal_sdk import urls
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/airavata_django_portal_sdk/urls.py",
> line 6, in
> from . import views
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/airavata_django_portal_sdk/views.py",
> line 16, in
> from rest_framework.decorators import api_view
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/decorators.py",
> line 13, in
> from rest_framework.views import APIView
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/views.py",
> line 17, in
> from rest_framework.schemas import DefaultSchema
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/schemas/__init__.py",
> line 33, in
> authentication_classes=api_settings.DEFAULT_AUTHENTICATION_CLASSES,
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 225, in __getattr__
> val = perform_import(val, attr)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 168, in perform_import
> return [import_from_string(item, setting_name) for item in val]
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 168, in
> return [import_from_string(item, setting_name) for item in val]
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
> line 177, in import_from_string
> return import_string(val)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/django/utils/module_loading.py",
> line 17, in import_string
> module = import_module(module_path)
> File
> "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/__init__.py",
> line 126, in import_module
> return _bootstrap._gcd_import(name[level:], package, level)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/authentication.py",
> line 6, in
> from django_airavata.apps.auth import utils
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/auth/utils.py",
> line 14, in
> from . import models
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/auth/models.py",
> line 17, in
> class EmailVerification(models.Model):
> File
>
[jira] [Created] (AIRAVATA-3696) Defer instantiating output view providers until after Django apps are ready
Marcus Christie created AIRAVATA-3696:
-
Summary: Defer instantiating output view providers until after
Django apps are ready
Key: AIRAVATA-3696
URL: https://issues.apache.org/jira/browse/AIRAVATA-3696
Project: Airavata
Issue Type: Bug
Components: Django Portal
Reporter: Marcus Christie
Assignee: Marcus Christie
Currently in settings.py all output view providers are instantiated. However,
they might import code that transitively imports Django models which is not
allowed in Django until all apps have been loaded (see
https://docs.djangoproject.com/en/3.2/ref/applications/#initialization-process).
A better approach would be to instantiate the output view providers in the
ApiConfig.ready().
{code}
File
"/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/settings.py",
line 644, in
OUTPUT_VIEW_PROVIDERS[entry_point.name] = entry_point.load()()
File
"/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/pkg_resources/__init__.py",
line 2517, in load
return self.resolve()
File
"/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/pkg_resources/__init__.py",
line 2523, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File
"/Users/machrist/Airavata/django/regsnp-django-app/regsnp_django_app/output_views.py",
line 32, in
from airavata_django_portal_sdk import urls
File
"/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/airavata_django_portal_sdk/urls.py",
line 6, in
from . import views
File
"/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/airavata_django_portal_sdk/views.py",
line 16, in
from rest_framework.decorators import api_view
File
"/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/decorators.py",
line 13, in
from rest_framework.views import APIView
File
"/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/views.py",
line 17, in
from rest_framework.schemas import DefaultSchema
File
"/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/schemas/__init__.py",
line 33, in
authentication_classes=api_settings.DEFAULT_AUTHENTICATION_CLASSES,
File
"/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
line 225, in __getattr__
val = perform_import(val, attr)
File
"/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
line 168, in perform_import
return [import_from_string(item, setting_name) for item in val]
File
"/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
line 168, in
return [import_from_string(item, setting_name) for item in val]
File
"/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/settings.py",
line 177, in import_from_string
return import_string(val)
File
"/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/django/utils/module_loading.py",
line 17, in import_string
module = import_module(module_path)
File
"/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/__init__.py",
line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File
"/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/authentication.py",
line 6, in
from django_airavata.apps.auth import utils
File
"/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/auth/utils.py",
line 14, in
from . import models
File
"/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/auth/models.py",
line 17, in
class EmailVerification(models.Model):
File
"/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/django/db/models/base.py",
line 108, in __new__
app_config = apps.get_containing_app_config(module)
File
"/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/django/apps/registry.py",
line 253, in get_containing_app_config
self.check_apps_ready()
File
"/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/django/apps/registry.py",
line 136, in check_apps_ready
raise AppRegistryNotReady("Apps aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717218#comment-17717218
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 75a4e563dda44e84abae93d8d1838238a34b7a1d in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=75a4e563 ]
AIRAVATA-3694 --max-age option added to archive_user_data
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717212#comment-17717212
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 6a52ac3c2b968f46ecd9c2cd08317a97da8c5bef in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=6a52ac3c ]
AIRAVATA-3694 handle case where there is nothing to archive
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717192#comment-17717192
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 9e5ccfacff38ae70169103c0268368d898fbafa1 in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=9e5ccfac ]
AIRAVATA-3694 REST API and admin view to see when experiment data dir has been
archived
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3695) new optional list Thrift property causes error in thrift_utils if not present in request
[
https://issues.apache.org/jira/browse/AIRAVATA-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717201#comment-17717201
]
ASF subversion and git services commented on AIRAVATA-3695:
---
Commit 4442cde5e9f9a46ba6f4f3af8103980b810eeb45 in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=4442cde5 ]
AIRAVATA-3695 Bug fix for null reference
> new optional list Thrift property causes error in thrift_utils if not present
> in request
>
>
> Key: AIRAVATA-3695
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3695
> Project: Airavata
> Issue Type: Bug
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> {code}
> [2023-04-19 20:59:43,639 django_airavata.apps.api.exceptions:63 ERROR] API
> exception
> Traceback (most recent call last):
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/views.py",
> line 506, in dispatch
> response = handler(request, *args, **kwargs)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/mixins.py",
> line 68, in update
> self.perform_update(serializer)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/views.py",
> line 225, in perform_update
> experiment = serializer.save(
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/serializers.py",
> line 200, in save
> self.instance = self.update(self.instance, validated_data)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 148, in update
> return self.create(validated_data)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 144, in create
> params = self.process_nested_fields(validated_data)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 139, in process_nested_fields
> params[field_name] = serializer.create(
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 144, in create
> params = self.process_nested_fields(validated_data)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 129, in process_nested_fields
> if (params[field_name] is not None or
> KeyError: 'autoScheduledCompResourceSchedulingList'
> {code}
> thrift_utils.py assumes that every field will be present even when the field
> is optional/nullable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717213#comment-17717213
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 9e5ccfacff38ae70169103c0268368d898fbafa1 in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=9e5ccfac ]
AIRAVATA-3694 REST API and admin view to see when experiment data dir has been
archived
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3695) new optional list Thrift property causes error in thrift_utils if not present in request
[
https://issues.apache.org/jira/browse/AIRAVATA-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717222#comment-17717222
]
ASF subversion and git services commented on AIRAVATA-3695:
---
Commit 4442cde5e9f9a46ba6f4f3af8103980b810eeb45 in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=4442cde5 ]
AIRAVATA-3695 Bug fix for null reference
> new optional list Thrift property causes error in thrift_utils if not present
> in request
>
>
> Key: AIRAVATA-3695
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3695
> Project: Airavata
> Issue Type: Bug
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> {code}
> [2023-04-19 20:59:43,639 django_airavata.apps.api.exceptions:63 ERROR] API
> exception
> Traceback (most recent call last):
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/views.py",
> line 506, in dispatch
> response = handler(request, *args, **kwargs)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/mixins.py",
> line 68, in update
> self.perform_update(serializer)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/views.py",
> line 225, in perform_update
> experiment = serializer.save(
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/serializers.py",
> line 200, in save
> self.instance = self.update(self.instance, validated_data)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 148, in update
> return self.create(validated_data)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 144, in create
> params = self.process_nested_fields(validated_data)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 139, in process_nested_fields
> params[field_name] = serializer.create(
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 144, in create
> params = self.process_nested_fields(validated_data)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 129, in process_nested_fields
> if (params[field_name] is not None or
> KeyError: 'autoScheduledCompResourceSchedulingList'
> {code}
> thrift_utils.py assumes that every field will be present even when the field
> is optional/nullable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717215#comment-17717215
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit d0768e84bfd60814cf8f292afe380c1ef8bb9996 in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=d0768e84 ]
AIRAVATA-3694 Attempt to rollback the archive if something fails when deleting
archived data
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717219#comment-17717219
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 28b9ee7885d7aebeb60b2abfbbe1b1cba7712804 in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=28b9ee78 ]
AIRAVATA-3694 make the archive directory, including any missing parents
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3693) Windows support: use run-s to build frontend code and library
[
https://issues.apache.org/jira/browse/AIRAVATA-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717186#comment-17717186
]
ASF subversion and git services commented on AIRAVATA-3693:
---
Commit 2b5ba36b3cfc61fc1aa2130785c4e77a51bf11cf in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=2b5ba36b ]
AIRAVATA-3693 add npm-run-all to package.json
> Windows support: use run-s to build frontend code and library
> -
>
> Key: AIRAVATA-3693
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3693
> Project: Airavata
> Issue Type: Improvement
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Currently the common frontend code in the Django Portal builds the JS/CSS and
> the library by using {{&&}} to run the two build steps one after the other.
> But this only works in Bash shell environments. To work on Windows,
> [run-s|https://github.com/mysticatea/npm-run-all] is a better alternative.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717214#comment-17717214
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 2a63b583725ca2d28d5936c55080a155e6c0f713 in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=2a63b583 ]
AIRAVATA-3694 Documented new user data archive settings
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3693) Windows support: use run-s to build frontend code and library
[
https://issues.apache.org/jira/browse/AIRAVATA-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717205#comment-17717205
]
ASF subversion and git services commented on AIRAVATA-3693:
---
Commit 5ab69a4a0ece3ccdffc478e4301c9c975c6e6508 in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=5ab69a4a ]
AIRAVATA-3693 Add windows JS build to action
> Windows support: use run-s to build frontend code and library
> -
>
> Key: AIRAVATA-3693
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3693
> Project: Airavata
> Issue Type: Improvement
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Currently the common frontend code in the Django Portal builds the JS/CSS and
> the library by using {{&&}} to run the two build steps one after the other.
> But this only works in Bash shell environments. To work on Windows,
> [run-s|https://github.com/mysticatea/npm-run-all] is a better alternative.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717189#comment-17717189
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit e0abdebb508d867fd104f3e09c712c9c46cf1eb5 in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=e0abdebb ]
AIRAVATA-3694 minimum archive size setting
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717217#comment-17717217
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 086fb53fc8d816c76f5df69d5a33c4cb0e47bb6a in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=086fb53f ]
AIRAVATA-3694 Notice to user when experiment data is archived and also the
general archive policy
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3695) new optional list Thrift property causes error in thrift_utils if not present in request
[
https://issues.apache.org/jira/browse/AIRAVATA-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717221#comment-17717221
]
ASF subversion and git services commented on AIRAVATA-3695:
---
Commit 668357b0dc6660bd946cf943d6afe6b8966b8551 in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=668357b0 ]
AIRAVATA-3695 Bug fix for handling missing optional fields
> new optional list Thrift property causes error in thrift_utils if not present
> in request
>
>
> Key: AIRAVATA-3695
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3695
> Project: Airavata
> Issue Type: Bug
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> {code}
> [2023-04-19 20:59:43,639 django_airavata.apps.api.exceptions:63 ERROR] API
> exception
> Traceback (most recent call last):
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/views.py",
> line 506, in dispatch
> response = handler(request, *args, **kwargs)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/mixins.py",
> line 68, in update
> self.perform_update(serializer)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/views.py",
> line 225, in perform_update
> experiment = serializer.save(
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/serializers.py",
> line 200, in save
> self.instance = self.update(self.instance, validated_data)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 148, in update
> return self.create(validated_data)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 144, in create
> params = self.process_nested_fields(validated_data)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 139, in process_nested_fields
> params[field_name] = serializer.create(
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 144, in create
> params = self.process_nested_fields(validated_data)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 129, in process_nested_fields
> if (params[field_name] is not None or
> KeyError: 'autoScheduledCompResourceSchedulingList'
> {code}
> thrift_utils.py assumes that every field will be present even when the field
> is optional/nullable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3693) Windows support: use run-s to build frontend code and library
[
https://issues.apache.org/jira/browse/AIRAVATA-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717206#comment-17717206
]
ASF subversion and git services commented on AIRAVATA-3693:
---
Commit e56c3306f5f1a799d1d5fd1dc03951a10b62674a in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=e56c3306 ]
AIRAVATA-3693 windows JS build action set to use cmd shell
> Windows support: use run-s to build frontend code and library
> -
>
> Key: AIRAVATA-3693
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3693
> Project: Airavata
> Issue Type: Improvement
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Currently the common frontend code in the Django Portal builds the JS/CSS and
> the library by using {{&&}} to run the two build steps one after the other.
> But this only works in Bash shell environments. To work on Windows,
> [run-s|https://github.com/mysticatea/npm-run-all] is a better alternative.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717209#comment-17717209
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit b9dc12e4e27e4f7ef199b36590736ffb399b8f81 in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=b9dc12e4 ]
AIRAVATA-3694 Basic unarchive_user_data command
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717216#comment-17717216
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit e4aa3d4d99138634556d46628d747b1a40e077be in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=e4aa3d4d ]
AIRAVATA-3694 Reset modification time when unarchiving
i.e., 'touch' all of the unarchived files
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3693) Windows support: use run-s to build frontend code and library
[
https://issues.apache.org/jira/browse/AIRAVATA-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717185#comment-17717185
]
ASF subversion and git services commented on AIRAVATA-3693:
---
Commit e56c3306f5f1a799d1d5fd1dc03951a10b62674a in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=e56c3306 ]
AIRAVATA-3693 windows JS build action set to use cmd shell
> Windows support: use run-s to build frontend code and library
> -
>
> Key: AIRAVATA-3693
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3693
> Project: Airavata
> Issue Type: Improvement
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Currently the common frontend code in the Django Portal builds the JS/CSS and
> the library by using {{&&}} to run the two build steps one after the other.
> But this only works in Bash shell environments. To work on Windows,
> [run-s|https://github.com/mysticatea/npm-run-all] is a better alternative.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717223#comment-17717223
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 15f91433b519f096f0c29869d9b3e9ba2c33c505 in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=15f91433 ]
AIRAVATA-3694 log exception when archive fails so it generates an error email
in production
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3693) Windows support: use run-s to build frontend code and library
[
https://issues.apache.org/jira/browse/AIRAVATA-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717183#comment-17717183
]
ASF subversion and git services commented on AIRAVATA-3693:
---
Commit 3dfcac881b7af93fdffe929cb3a1e8cabe9f522f in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=3dfcac88 ]
AIRAVATA-3693 Switch to run-s so it works on Windows too
> Windows support: use run-s to build frontend code and library
> -
>
> Key: AIRAVATA-3693
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3693
> Project: Airavata
> Issue Type: Improvement
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Currently the common frontend code in the Django Portal builds the JS/CSS and
> the library by using {{&&}} to run the two build steps one after the other.
> But this only works in Bash shell environments. To work on Windows,
> [run-s|https://github.com/mysticatea/npm-run-all] is a better alternative.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717220#comment-17717220
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit e27ac668017f076b22e06156199714cad5aa8783 in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=e27ac668 ]
AIRAVATA-3694 adding missing mock for test
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717224#comment-17717224
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 8db03aa7458118819801a3371866cd08a0a172ef in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=8db03aa7 ]
AIRAVATA-3694 handle case where file was already deleted
This is to handle the case where a user directory is symlinked as an alias
and the files to delete end up listed more than once. Code just emits a warning
when this happens.
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3693) Windows support: use run-s to build frontend code and library
[
https://issues.apache.org/jira/browse/AIRAVATA-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717204#comment-17717204
]
ASF subversion and git services commented on AIRAVATA-3693:
---
Commit 3dfcac881b7af93fdffe929cb3a1e8cabe9f522f in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=3dfcac88 ]
AIRAVATA-3693 Switch to run-s so it works on Windows too
> Windows support: use run-s to build frontend code and library
> -
>
> Key: AIRAVATA-3693
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3693
> Project: Airavata
> Issue Type: Improvement
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Currently the common frontend code in the Django Portal builds the JS/CSS and
> the library by using {{&&}} to run the two build steps one after the other.
> But this only works in Bash shell environments. To work on Windows,
> [run-s|https://github.com/mysticatea/npm-run-all] is a better alternative.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717199#comment-17717199
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit e27ac668017f076b22e06156199714cad5aa8783 in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=e27ac668 ]
AIRAVATA-3694 adding missing mock for test
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717210#comment-17717210
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit e0abdebb508d867fd104f3e09c712c9c46cf1eb5 in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=e0abdebb ]
AIRAVATA-3694 minimum archive size setting
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717195#comment-17717195
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit e4aa3d4d99138634556d46628d747b1a40e077be in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=e4aa3d4d ]
AIRAVATA-3694 Reset modification time when unarchiving
i.e., 'touch' all of the unarchived files
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717198#comment-17717198
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 28b9ee7885d7aebeb60b2abfbbe1b1cba7712804 in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=28b9ee78 ]
AIRAVATA-3694 make the archive directory, including any missing parents
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717191#comment-17717191
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 6a52ac3c2b968f46ecd9c2cd08317a97da8c5bef in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=6a52ac3c ]
AIRAVATA-3694 handle case where there is nothing to archive
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3693) Windows support: use run-s to build frontend code and library
[
https://issues.apache.org/jira/browse/AIRAVATA-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717207#comment-17717207
]
ASF subversion and git services commented on AIRAVATA-3693:
---
Commit 2b5ba36b3cfc61fc1aa2130785c4e77a51bf11cf in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=2b5ba36b ]
AIRAVATA-3693 add npm-run-all to package.json
> Windows support: use run-s to build frontend code and library
> -
>
> Key: AIRAVATA-3693
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3693
> Project: Airavata
> Issue Type: Improvement
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Currently the common frontend code in the Django Portal builds the JS/CSS and
> the library by using {{&&}} to run the two build steps one after the other.
> But this only works in Bash shell environments. To work on Windows,
> [run-s|https://github.com/mysticatea/npm-run-all] is a better alternative.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3695) new optional list Thrift property causes error in thrift_utils if not present in request
[
https://issues.apache.org/jira/browse/AIRAVATA-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717200#comment-17717200
]
ASF subversion and git services commented on AIRAVATA-3695:
---
Commit 668357b0dc6660bd946cf943d6afe6b8966b8551 in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=668357b0 ]
AIRAVATA-3695 Bug fix for handling missing optional fields
> new optional list Thrift property causes error in thrift_utils if not present
> in request
>
>
> Key: AIRAVATA-3695
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3695
> Project: Airavata
> Issue Type: Bug
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> {code}
> [2023-04-19 20:59:43,639 django_airavata.apps.api.exceptions:63 ERROR] API
> exception
> Traceback (most recent call last):
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/views.py",
> line 506, in dispatch
> response = handler(request, *args, **kwargs)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/mixins.py",
> line 68, in update
> self.perform_update(serializer)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/views.py",
> line 225, in perform_update
> experiment = serializer.save(
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/venv/lib/python3.10/site-packages/rest_framework/serializers.py",
> line 200, in save
> self.instance = self.update(self.instance, validated_data)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 148, in update
> return self.create(validated_data)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 144, in create
> params = self.process_nested_fields(validated_data)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 139, in process_nested_fields
> params[field_name] = serializer.create(
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 144, in create
> params = self.process_nested_fields(validated_data)
> File
> "/Users/machrist/Airavata/django/django_airavata_gateway/django_airavata/apps/api/thrift_utils.py",
> line 129, in process_nested_fields
> if (params[field_name] is not None or
> KeyError: 'autoScheduledCompResourceSchedulingList'
> {code}
> thrift_utils.py assumes that every field will be present even when the field
> is optional/nullable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717211#comment-17717211
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 1e130acbe407cce15ddde58244b3f410b949b81b in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=1e130acb ]
AIRAVATA-3694 log error if some error happens trying to create a user archive
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717203#comment-17717203
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 8db03aa7458118819801a3371866cd08a0a172ef in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=8db03aa7 ]
AIRAVATA-3694 handle case where file was already deleted
This is to handle the case where a user directory is symlinked as an alias
and the files to delete end up listed more than once. Code just emits a warning
when this happens.
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717193#comment-17717193
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 2a63b583725ca2d28d5936c55080a155e6c0f713 in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=2a63b583 ]
AIRAVATA-3694 Documented new user data archive settings
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717197#comment-17717197
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 75a4e563dda44e84abae93d8d1838238a34b7a1d in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=75a4e563 ]
AIRAVATA-3694 --max-age option added to archive_user_data
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717194#comment-17717194
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit d0768e84bfd60814cf8f292afe380c1ef8bb9996 in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=d0768e84 ]
AIRAVATA-3694 Attempt to rollback the archive if something fails when deleting
archived data
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717208#comment-17717208
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit a248cbd7d37730547f5827c43bf371861ca2a8e3 in airavata-django-portal's
branch refs/heads/delta-topology-workshop from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=a248cbd7 ]
AIRAVATA-3694 Basic archive_user_data management command
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3693) Windows support: use run-s to build frontend code and library
[
https://issues.apache.org/jira/browse/AIRAVATA-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717184#comment-17717184
]
ASF subversion and git services commented on AIRAVATA-3693:
---
Commit 5ab69a4a0ece3ccdffc478e4301c9c975c6e6508 in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=5ab69a4a ]
AIRAVATA-3693 Add windows JS build to action
> Windows support: use run-s to build frontend code and library
> -
>
> Key: AIRAVATA-3693
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3693
> Project: Airavata
> Issue Type: Improvement
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Currently the common frontend code in the Django Portal builds the JS/CSS and
> the library by using {{&&}} to run the two build steps one after the other.
> But this only works in Bash shell environments. To work on Windows,
> [run-s|https://github.com/mysticatea/npm-run-all] is a better alternative.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717202#comment-17717202
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 15f91433b519f096f0c29869d9b3e9ba2c33c505 in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=15f91433 ]
AIRAVATA-3694 log exception when archive fails so it generates an error email
in production
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717196#comment-17717196
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 086fb53fc8d816c76f5df69d5a33c4cb0e47bb6a in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=086fb53f ]
AIRAVATA-3694 Notice to user when experiment data is archived and also the
general archive policy
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717188#comment-17717188
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit b9dc12e4e27e4f7ef199b36590736ffb399b8f81 in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=b9dc12e4 ]
AIRAVATA-3694 Basic unarchive_user_data command
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717190#comment-17717190
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit 1e130acbe407cce15ddde58244b3f410b949b81b in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=1e130acb ]
AIRAVATA-3694 log error if some error happens trying to create a user archive
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AIRAVATA-3694) User data archive management commands
[
https://issues.apache.org/jira/browse/AIRAVATA-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717187#comment-17717187
]
ASF subversion and git services commented on AIRAVATA-3694:
---
Commit a248cbd7d37730547f5827c43bf371861ca2a8e3 in airavata-django-portal's
branch refs/heads/master from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata-django-portal.git;h=a248cbd7 ]
AIRAVATA-3694 Basic archive_user_data management command
> User data archive management commands
> -
>
> Key: AIRAVATA-3694
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3694
> Project: Airavata
> Issue Type: New Feature
> Components: Django Portal
>Reporter: Marcus Christie
>Assignee: Marcus Christie
>Priority: Major
>
> Create management commands to manage archiving user data. The use case is the
> gateway admin wants to archive older data and then delete that user data to
> free up disk space.
> The management commands will handle creating archives (as tarballs) and
> deleting the data from the user data archive directory. There will also be an
> unarchive command. There are settings for the max age of files to be archived
> and for the directory in which archives should be copied.
> How the archive file are archived. It's expected that the gateway admin would
> periodically (perhaps by cron) copy the archive files from the web server to
> some other file server.
> h3. Description
> archive_user_data creates a tarball archive of user data for all files and
> directories that are older than some configured amount of days. In addition
> to the tarball is a text file that lists all of the files and directories
> that are archived. The tarball and text file can be periodically pushed to
> tape backup or any other backup location.
> The configuration settings are
> - GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS
> - GATEWAY_USER_DATA_ARCHIVE_DIRECTORY
> -- this is the directory in which to place the archive files and is also the
> place where temporary files are generated. Since the archive files can be
> large, it's important that there be enough free disk space on the partition
> where this directory lives
> - GATEWAY_USER_DATA_ARCHIVE_MINIMUM_ARCHIVE_SIZE_GB
> -- defaults to 1 GB. This can be used to prevent creating a lot of small
> archives since tape archives often want a few large files instead of many
> small files.
> h4. Running archive_user_data
> All commands should be run as the gateway server user (pga).
> {code}
> python manage.py archive_user_data --dry-run
> {code}
> This just prints the files and directories that would be archived and exits.
> Good for checking that configuration is correct, etc.
> {code}
> python manage.py archive_user_data
> {code}
> This will actually create an archive and then delete from user data the files
> that were archived.
> {code}
> python manage.py archive_user_data --max-age MAX_AGE
> {code}
> The --max-age flag allows overriding the
> GATEWAY_USER_DATA_ARCHIVE_MAX_AGE_DAYS setting. This can be a good option to
> create the first few archives when introducing the user data archive to an
> existing gateway.
> h4. Running unarchive_user_data
> unarchive_user_data requires an archive tarball as input. The main use case
> for this command is that the gateway administrator wants to restore some
> particular user data. First, the right archive must be found. The experiment
> details view in Experiment Statistics will display the name of the archive
> file for an experiment data directory that has been archived. Use this to
> then retrieve the tarball from backup. Then run unarchive_user_data on the
> file.
> {code}
> python manage.py unarchive_user_data
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
> The timestamps will be restored from the archive, including the last modified
> timestamps. This means that the next time archive_user_data runs, all files
> unarchived will be re-archived. Sometimes that is desired, but if you want to
> reset the last modified times, use the {{--reset-modification}} option:
> {code}
> python manage.py unarchive_user_data --reset-modification
> /path/to/archive_seagrid_older_than_2023-04-17-22-15-34.tgz
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
