This bug that I found, please give me advice, I come to fix the bug,
thank you.
** Description changed:
We have recently encountered a problem, that is, when the creation of
volume will be an accidental failure.
The failure log is: Volume with id d6241355-a7e9-4a07-9d40-beaa5b4861ec
is already being cleaned up or another host has taken over it
View the create volume code:
- @objects.Volume.set_workers
- def create_volume(self, context, volume, request_spec=None,
- filter_properties=None, allow_reschedule=True):
- """Creates the volume."""
- # Log about unsupported drivers
- utils.log_unsupported_driver_warning(self.driver)
+ @objects.Volume.set_workers
+ def create_volume(self, context, volume, request_spec=None,
+ filter_properties=None, allow_reschedule=True):
+ """Creates the volume."""
+ # Log about unsupported drivers
+ utils.log_unsupported_driver_warning(self.driver)
I find that the use decorate "set_workers",
The next code is:
if (worker.service_id != service_id or worker.status != self.status):
- try:
- db.worker_update(
- self._context, worker.id,
- filters={'service_id': worker.service_id,
- 'status': worker.status,
- 'race_preventer': worker.race_preventer,
- 'updated_at': worker.updated_at},
- service_id=service_id,
- status=self.status,
- orm_worker=worker)
- except exception.WorkerNotFound:
- self.worker = None
- raise exception.CleanableInUse(type=self.__class__.__name__,
- id=self.id)
+ try:
+ db.worker_update(
+ self._context, worker.id,
+ filters={'service_id': worker.service_id,
+ 'status': worker.status,
+ 'race_preventer': worker.race_preventer,
+ 'updated_at': worker.updated_at},
+ service_id=service_id,
+ status=self.status,
+ orm_worker=worker)
+ except exception.WorkerNotFound:
+ self.worker = None
+ raise exception.CleanableInUse(type=self.__class__.__name__,
+ id=self.id)
Finally, see the last error is abnormal, that is raise
exception.CleanableInUse
Why is this happen and can not find a worker?
Finally, by looking at sqlalchemy and pymysql code, and breakpoint
debugging
Can be seen by breakpoint debug:
(Pdb) worker = db.worker_get(self._context, resource_type=resource_type,
resource_id=self.id) #Query the data according to the conditions
(Pdb) worker.updated_at
datetime.datetime(2017, 8, 21, 4, 2, 5, 513630)
#Query the updated_at time datetime.datetime(year, month, day[, hour[,
minute[, second[, microsecond[,tzinfo]]]]])
(Pdb) p (resource_type, self.id)
('Volume', '7bcbfa37-37f9-42b4-962a-84fb2688f14a')
-
-
+
View through the database:
MariaDB [cinder]> SELECT workers.created_at AS workers_created_at,
workers.deleted_at AS workers_deleted_at, workers.deleted AS workers_deleted,
workers.updated_at AS workers_updated_at, workers.id AS workers_id,
workers.resource_type AS workers_resource_type, workers.resource_id AS
workers_resource_id, workers.status AS workers_status, workers.service_id AS
workers_service_id, workers.race_preventer AS workers_race_preventer FROM
workers WHERE workers.deleted = false AND workers.resource_type != 'SENTINEL'
AND workers.resource_type = 'Volume' AND workers.resource_id =
'7bcbfa37-37f9-42b4-962a-84fb2688f14a' LIMIT 1;
+---------------------+--------------------+-----------------+----------------------------+------------+-----------------------+--------------------------------------+----------------+--------------------+------------------------+
| workers_created_at | workers_deleted_at | workers_deleted |
workers_updated_at | workers_id | workers_resource_type |
workers_resource_id | workers_status | workers_service_id |
workers_race_preventer |
+---------------------+--------------------+-----------------+----------------------------+------------+-----------------------+--------------------------------------+----------------+--------------------+------------------------+
| 2017-08-21 04:02:05 | NULL | 0 | 2017-08-21
04:02:05.513631 | 2105 | Volume |
7bcbfa37-37f9-42b4-962a-84fb2688f14a | deleting | NULL |
0 |
+---------------------+--------------------+-----------------+----------------------------+------------+-----------------------+--------------------------------------+----------------+--------------------+------------------------+
-
- You can see the time in the database workers_updated_at = 2017-08-21
04:02:05.513631
- Query time by oslo_db.sqlalchemy worker.updated_at =
datetime.datetime(2017, 8, 21, 4, 2, 5, 513630)
+ You can see the time in the database workers_updated_at = 2017-08-21
04:02:05.513631
+ Query time by oslo_db.sqlalchemy worker.updated_at =
datetime.datetime(2017, 8, 21, 4, 2, 5, 513630)
From the above, you can see two different time only the last
microseconds. It's this microsecond different. it is easy to lead the
other the query data failed that based on update_at field
-
So look at the worker_update code:
def worker_update(context, id, filters=None, orm_worker=None, **values):
- """Update a worker with given values."""
- filters = filters or {}
- query = _worker_query(context, id=id, **filters) #will query worker
by filters, the filters contain update_at, so query failed
-
+ """Update a worker with given values."""
+ filters = filters or {}
+ query = _worker_query(context, id=id, **filters) #will query worker
by filters, the filters contain update_at, so query failed
Solution:
method 1: Modify the pymysql code, this is not currently possible.
method 2: Adjust the query conditions filters, remove the update_at, through
the worker_id, service_id, status, race_preventer these conditions is
sufficient to query the current worker
The second method approach looks more viable, but the first method
fundamentally solves the problem
I recommend the second method
+
+ This bug that I found, please give me advice, I come to fix the bug,
+ thank you.
** Description changed:
- We have recently encountered a problem, that is, when the creation of
- volume will be an accidental failure.
-
- The failure log is: Volume with id d6241355-a7e9-4a07-9d40-beaa5b4861ec
- is already being cleaned up or another host has taken over it
-
- View the create volume code:
-
- @objects.Volume.set_workers
- def create_volume(self, context, volume, request_spec=None,
- filter_properties=None, allow_reschedule=True):
- """Creates the volume."""
- # Log about unsupported drivers
- utils.log_unsupported_driver_warning(self.driver)
-
- I find that the use decorate "set_workers",
-
- The next code is:
-
- if (worker.service_id != service_id or worker.status != self.status):
- try:
- db.worker_update(
- self._context, worker.id,
- filters={'service_id': worker.service_id,
- 'status': worker.status,
- 'race_preventer': worker.race_preventer,
- 'updated_at': worker.updated_at},
- service_id=service_id,
- status=self.status,
- orm_worker=worker)
- except exception.WorkerNotFound:
- self.worker = None
- raise exception.CleanableInUse(type=self.__class__.__name__,
- id=self.id)
-
- Finally, see the last error is abnormal, that is raise
exception.CleanableInUse
- Why is this happen and can not find a worker?
-
- Finally, by looking at sqlalchemy and pymysql code, and breakpoint
- debugging
-
- Can be seen by breakpoint debug:
- (Pdb) worker = db.worker_get(self._context, resource_type=resource_type,
resource_id=self.id) #Query the data according to the conditions
- (Pdb) worker.updated_at
- datetime.datetime(2017, 8, 21, 4, 2, 5, 513630)
#Query the updated_at time datetime.datetime(year, month, day[, hour[,
minute[, second[, microsecond[,tzinfo]]]]])
- (Pdb) p (resource_type, self.id)
- ('Volume', '7bcbfa37-37f9-42b4-962a-84fb2688f14a')
-
- View through the database:
- MariaDB [cinder]> SELECT workers.created_at AS workers_created_at,
workers.deleted_at AS workers_deleted_at, workers.deleted AS workers_deleted,
workers.updated_at AS workers_updated_at, workers.id AS workers_id,
workers.resource_type AS workers_resource_type, workers.resource_id AS
workers_resource_id, workers.status AS workers_status, workers.service_id AS
workers_service_id, workers.race_preventer AS workers_race_preventer FROM
workers WHERE workers.deleted = false AND workers.resource_type != 'SENTINEL'
AND workers.resource_type = 'Volume' AND workers.resource_id =
'7bcbfa37-37f9-42b4-962a-84fb2688f14a' LIMIT 1;
-
+---------------------+--------------------+-----------------+----------------------------+------------+-----------------------+--------------------------------------+----------------+--------------------+------------------------+
- | workers_created_at | workers_deleted_at | workers_deleted |
workers_updated_at | workers_id | workers_resource_type |
workers_resource_id | workers_status | workers_service_id |
workers_race_preventer |
-
+---------------------+--------------------+-----------------+----------------------------+------------+-----------------------+--------------------------------------+----------------+--------------------+------------------------+
- | 2017-08-21 04:02:05 | NULL | 0 | 2017-08-21
04:02:05.513631 | 2105 | Volume |
7bcbfa37-37f9-42b4-962a-84fb2688f14a | deleting | NULL |
0 |
-
+---------------------+--------------------+-----------------+----------------------------+------------+-----------------------+--------------------------------------+----------------+--------------------+------------------------+
-
- You can see the time in the database workers_updated_at = 2017-08-21
04:02:05.513631
- Query time by oslo_db.sqlalchemy worker.updated_at =
datetime.datetime(2017, 8, 21, 4, 2, 5, 513630)
-
- From the above, you can see two different time only the last
- microseconds. It's this microsecond different. it is easy to lead the
- other the query data failed that based on update_at field
-
- So look at the worker_update code:
- def worker_update(context, id, filters=None, orm_worker=None, **values):
- """Update a worker with given values."""
- filters = filters or {}
- query = _worker_query(context, id=id, **filters) #will query worker
by filters, the filters contain update_at, so query failed
-
- Solution:
- method 1: Modify the pymysql code, this is not currently possible.
- method 2: Adjust the query conditions filters, remove the update_at, through
the worker_id, service_id, status, race_preventer these conditions is
sufficient to query the current worker
-
- The second method approach looks more viable, but the first method
- fundamentally solves the problem
-
- I recommend the second method
-
- This bug that I found, please give me advice, I come to fix the bug,
- thank you.
+ The wrong version, sorry
** Changed in: ubiquity (Ubuntu)
Status: New => Invalid
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1712270
Title:
cinder get worker failed
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1712270/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
