** Description changed:
Environment: Ubuntu Bionic, OpenStack Train, Volume and Backup Driver
Ceph
This bug was also reproduced on Queens.
Steps to reproduce:
1) Create a volume v1 as user demo
2) Create a full backup of v1 as user admin
3) Create an incremental backup of v1 as user demo
4) Result: Backup is created successfully, but error in logs. Logs do not
show INFO message and parent_backup.num_dependent_backups is not incremented
[0].
Expectation: Either no error successfully finishing method
create_backup, or not creating backup at all, with backup entity set as
Error (see below).
Using POSIX driver, the backup is not created successfully and backup
entity remains in error state. The POSIX driver tries to read
backup.parent_id before creating the backup therefore raises the error
earlier.
Trace:
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server
[req-19fccda2-b765-4b68-9663-aeae5cf3002e 3bee8c02ecce4744af287b3b8dd40ca5
9370e1e898214d88bd9eda4f88eccba9 - f883c6315a004471b2ab1b04d8a111d9
f883c6315a004471b2ab1b04d8a111d9] Exception during message handling:
cinder.exception.BackupNotFound: Backup 7a15b9ec-7648-483f-ad32-6074567eca2a
could not be found.
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server Traceback (most
recent call last):
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in
_process_incoming
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server res =
self.dispatcher.dispatch(message)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 274, in
dispatch
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return
self._do_dispatch(endpoint, method, ctxt, args)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in
_do_dispatch
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server result =
func(ctxt, **new_args)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/cinder/backup/manager.py", line 447, in
create_backup
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server
backup.parent_id)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/cinder/objects/base.py", line 352, in get_by_id
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server orm_obj =
db.get_by_id(context, cls.model, id, *args, **kwargs)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/cinder/db/api.py", line 1801, in get_by_id
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return
IMPL.get_by_id(context, model, id, *args, **kwargs)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 189, in
wrapper
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return
f(*args, **kwargs)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 7125, in
get_by_id
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return
_GET_METHODS[model](context, id, *args, **kwargs)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 189, in
wrapper
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return
f(*args, **kwargs)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 5154, in
backup_get
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server
project_only=project_only)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 5165, in
_backup_get
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server raise
exception.BackupNotFound(backup_id=backup_id)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server
cinder.exception.BackupNotFound: Backup 7a15b9ec-7648-483f-ad32-6074567eca2a
could not be found.
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server
[0]
https://github.com/openstack/cinder/blob/e79b98367bd72f0323258dc0506de28b31c4aa48/cinder/backup/manager.py#L448
+
+ ====================================================================
+
+ [Impact]
+
+ Whenever an admin creates a backup for a tenant volume, all subsequent
+ incremental backups will have a silent error during creation when
+ linking the dependent backup to its parent because the code detects it
+ has a parent (by listing backups from all projects) but is unable to
+ find the DB entity while scoped to only its project. Upon deleting the
+ child backup, it will hit another silent error (because of lack of link)
+ and it will skip cleaning up the quota for the backup. The quota will
+ then misrepresent the number and size of backups, and the discrepancy
+ will increase as this error is hit for several backups. Ultimately, the
+ user will hit a quota error with a lower number/size of backups than the
+ actual quota limit.
+
+ The fix addressed the problem by changing the code that searches for the
+ parent backup. The code will now only search for backups under the same
+ project, so it will prevent the error from happening.
+
+ [Test case]
+
+ 1) Reproducing the bug:
+
+ 1a) Create a volume v1 as user demo
+ 1b) Create a full backup b1 of v1 as user admin
+ 1c) Create an incremental backup b2 of v1 as user demo
+ 1d) Confirm backup b2 is created successfully
+ 1e) Check log file "/var/log/cinder/cinder-backup.log" for error message
below with b2's ID.
+
+ cinder.exception.BackupNotFound: Backup <b2_id> could not be found
+
+ 2) Cleanup not necessary
+
+ 3) Install package that contains fixed code
+
+ 4) Confirm bug is fixed
+
+ 4a) Repeat steps 1a to 1d, make sure to create a new volume and new backups
from it.
+ 4b) Check log file "/var/log/cinder/cinder-backup.log" for error messages
related to new b2 ID. There shouldn't be any.
+
+ [Regression Potential]
+
+ The fix has already been validated in upstream CI with new functional
+ tests that perform that specific workflow (however, checks API backup
+ field instead of logs), as far back as Rocky release. The code changes
+ affect the creation step of backups, and both the new and previously
+ existing functional tests thoroughly test the creation step in different
+ ways, therefore regressions with this patch are not expected.
+
+ [Other Info]
+
+ For releases older than Train, the fix for bug 1809323 needs to be
+ included as a dependency.
** Also affects: cinder (Ubuntu)
Importance: Undecided
Status: New
** Also affects: cloud-archive
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1869746
Title:
Cinder throws error creating incremental backup from parent in another
project
To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1869746/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
