Public bug reported: Currently, we have two APIs to update server's metadata: 1. update: only update/add the key=value user provided in this call 2. update_all: replace all previous added key=value pair with the key=value pair user provided in this call they are using the same _update_instance_metadata method, differed only with one boolean key: http://git.openstack.org/cgit/openstack/nova/tree/nova/api/openstack/compute/server_metadata.py#n78 http://git.openstack.org/cgit/openstack/nova/tree/nova/api/openstack/compute/server_metadata.py#n95
Then it will be handled in the below work flow: I. get the server object http://git.openstack.org/cgit/openstack/nova/tree/nova/api/openstack/compute/server_metadata.py#n108 II. handled in compute/api.py update_instance_metadata: http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/api.py#n3689 the difference of update_all and update is handled here: i) if we are using update all, the target meatadata will be set as the metadata passed in, say new_meta ii) if we are using update, the target meatadata will be set to old_meta + new_meta III. then we just set instance.metadata to the target metadata, and call instance.save() to do the job. IV. it is finally handled in DB: http://git.openstack.org/cgit/openstack/nova/tree/nova/db/sqlalchemy/api.py#n2694 here we compared the metadata read from DB and the target metadata we passed in: i) if we call update_all, the target metadata only contains new_meta, so key=value pairs that are in old_meta but not in new_meta will be deleted. ii) if we call update, the target metadata will contains new_meta + old_meta so old_meta will not be deleted. The above mentioned process worked pretty well for general uses, but when we come to concurrently usage, problem may occour: For example, we have two concurrent meta_data update calls, say meta_A and meta_B, they are called at the same time since the target meta is generated at the API level(in previous step I and II), it the two calls came in at the same time, the instance.metadata got in these two calls will be the same (old_meta), here comes the problem, the target meta for A call will be old_meta + A, target meta for B call will be old_meta + B, and they will then go the rest of the process; When it comes to the DB layer, step IV, as we only have one DB so it is first time come first serve style, lets say call A has successfully handled, the metadata in DB is now old_meta + A, then we will handle call B, as the target meta is old_meta + B hence A will be removed. ** Affects: nova Importance: Undecided Assignee: Zhenyu Zheng (zhengzhenyu) Status: New ** Changed in: nova Assignee: (unassigned) => Zhenyu Zheng (zhengzhenyu) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1650188 Title: Concurrently update server's metadata are handled badly Status in OpenStack Compute (nova): New Bug description: Currently, we have two APIs to update server's metadata: 1. update: only update/add the key=value user provided in this call 2. update_all: replace all previous added key=value pair with the key=value pair user provided in this call they are using the same _update_instance_metadata method, differed only with one boolean key: http://git.openstack.org/cgit/openstack/nova/tree/nova/api/openstack/compute/server_metadata.py#n78 http://git.openstack.org/cgit/openstack/nova/tree/nova/api/openstack/compute/server_metadata.py#n95 Then it will be handled in the below work flow: I. get the server object http://git.openstack.org/cgit/openstack/nova/tree/nova/api/openstack/compute/server_metadata.py#n108 II. handled in compute/api.py update_instance_metadata: http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/api.py#n3689 the difference of update_all and update is handled here: i) if we are using update all, the target meatadata will be set as the metadata passed in, say new_meta ii) if we are using update, the target meatadata will be set to old_meta + new_meta III. then we just set instance.metadata to the target metadata, and call instance.save() to do the job. IV. it is finally handled in DB: http://git.openstack.org/cgit/openstack/nova/tree/nova/db/sqlalchemy/api.py#n2694 here we compared the metadata read from DB and the target metadata we passed in: i) if we call update_all, the target metadata only contains new_meta, so key=value pairs that are in old_meta but not in new_meta will be deleted. ii) if we call update, the target metadata will contains new_meta + old_meta so old_meta will not be deleted. The above mentioned process worked pretty well for general uses, but when we come to concurrently usage, problem may occour: For example, we have two concurrent meta_data update calls, say meta_A and meta_B, they are called at the same time since the target meta is generated at the API level(in previous step I and II), it the two calls came in at the same time, the instance.metadata got in these two calls will be the same (old_meta), here comes the problem, the target meta for A call will be old_meta + A, target meta for B call will be old_meta + B, and they will then go the rest of the process; When it comes to the DB layer, step IV, as we only have one DB so it is first time come first serve style, lets say call A has successfully handled, the metadata in DB is now old_meta + A, then we will handle call B, as the target meta is old_meta + B hence A will be removed. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1650188/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
