Reviewed: https://review.opendev.org/736842 Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7019c5cf50bc4ad97b302220664a26efaf81a7fd Submitter: Zuul Branch: master
commit 7019c5cf50bc4ad97b302220664a26efaf81a7fd Author: Slawek Kaplonski <[email protected]> Date: Thu Jun 18 23:18:07 2020 +0200 Make _ensure_default_security_group method atomic Method _ensure_default_security_group wasn't atomic as it first tries to get default SG and if that not exists in DB, it tries to create it. It may happend, like e.g. in Calico plugin that between get_default_sg_id method and create_security_group method, this default SG will be created by other neutron worker. And in such case there will be Duplicate entry exception raised. So this patch is adding handling of such exception. Change-Id: I515c310f221e7d9ae3be59a26260538d1bc591c2 Closes-Bug: #1883730 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1883730 Title: Failed to create a duplicate DefaultSecurityGroup (with Calico plugin) Status in neutron: Fix Released Bug description: With Ussuri I'm hitting this in the neutron server: 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers [req-6160aa90-b69e-451b-9774-07b6d840f41f 088d04a2122548c5acb628348db93c40 11447be9beda4bf78dab27cdb75058e2 - default default] Mechanism driver 'calico' failed in update_port_postcommit: neutron_lib.objects.exceptions.NeutronDbObjectDuplicateEntry: Failed to create a duplicate DefaultSecurityGroup: for attribute(s) ['PRIMARY'] with value(s) 11447be9beda4bf78dab27cdb75058e2 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers Traceback (most recent call last): 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers cursor, statement, parameters, context 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 581, in do_execute 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers cursor.execute(statement, parameters) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 165, in execute 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers result = self._query(query) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 321, in _query 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers conn.query(q) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 860, in query 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers self._affected_rows = self._read_query_result(unbuffered=unbuffered) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1061, in _read_query_result 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers result.read() 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1349, in read 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers first_packet = self.connection._read_packet() 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1018, in _read_packet 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers packet.check_error() 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 384, in check_error 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers err.raise_mysql_exception(self._data) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers raise errorclass(errno, errval) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers pymysql.err.IntegrityError: (1062, "Duplicate entry '11447be9beda4bf78dab27cdb75058e2' for key 'PRIMARY'") 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers The above exception was the direct cause of the following exception: 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers Traceback (most recent call last): 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/neutron/objects/base.py", line 867, in create 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers self, self.obj_context, self.modify_fields_to_db(fields)) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/neutron/objects/db/api.py", line 69, in create_object 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers context.session.add(db_obj) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers next(self.gen) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1064, in _transaction_scope 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers yield resource 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers next(self.gen) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 680, in _session 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers self.session.flush() 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/orm/session.py", line 2479, in flush 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers self._flush(objects) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/orm/session.py", line 2617, in _flush 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers transaction.rollback(_capture_exception=True) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__ 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers compat.reraise(exc_type, exc_value, exc_tb) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 153, in reraise 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers raise value 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/orm/session.py", line 2577, in _flush 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers flush_context.execute() 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers rec.execute(self) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/orm/unitofwork.py", line 589, in execute 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers uow, 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers insert, 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/orm/persistence.py", line 1084, in _emit_insert_statements 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers c = cached_connections[connection].execute(statement, multiparams) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 982, in execute 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers return meth(self, multiparams, params) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers return connection._execute_clauseelement(self, multiparams, params) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1101, in _execute_clauseelement 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers distilled_params, 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1250, in _execute_context 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers e, statement, parameters, cursor, context 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1474, in _handle_dbapi_exception 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers util.raise_from_cause(newraise, exc_info) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers reraise(type(exception), exception, tb=exc_tb, cause=cause) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 152, in reraise 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers raise value.with_traceback(tb) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers cursor, statement, parameters, context 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 581, in do_execute 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers cursor.execute(statement, parameters) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 165, in execute 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers result = self._query(query) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 321, in _query 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers conn.query(q) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 860, in query 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers self._affected_rows = self._read_query_result(unbuffered=unbuffered) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1061, in _read_query_result 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers result.read() 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1349, in read 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers first_packet = self.connection._read_packet() 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1018, in _read_packet 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers packet.check_error() 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 384, in check_error 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers err.raise_mysql_exception(self._data) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers raise errorclass(errno, errval) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry '11447be9beda4bf78dab27cdb75058e2' for key 'PRIMARY'") 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers [SQL: INSERT INTO default_security_group (project_id, security_group_id) VALUES (%(project_id)s, %(security_group_id)s)] 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers [parameters: {'project_id': '11447be9beda4bf78dab27cdb75058e2', 'security_group_id': '9f3a473c-b08a-4cf2-8327-10ecc8b87301'}] 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers (Background on this error at: http://sqlalche.me/e/gkpj) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers During handling of the above exception, another exception occurred: 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers Traceback (most recent call last): 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/managers.py", line 477, in _call_on_drivers 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers getattr(driver.obj, method_name)(context) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/dist-packages/networking_calico/plugins/ml2/drivers/calico/mech_calico.py", line 173, in wrapper 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers return f(self, *args, **kwargs) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/dist-packages/networking_calico/plugins/ml2/drivers/calico/mech_calico.py", line 789, in update_port_postcommit 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers self._icehouse_migration_step(context, port, original) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/dist-packages/networking_calico/plugins/ml2/drivers/calico/mech_calico.py", line 913, in _icehouse_migration_step 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers self._port_bound_update(context, port) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/dist-packages/networking_calico/plugins/ml2/drivers/calico/mech_calico.py", line 896, in _port_bound_update 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers self.endpoint_syncer.write_endpoint(port, context._plugin_context) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/dist-packages/networking_calico/plugins/ml2/drivers/calico/endpoints.py", line 153, in write_endpoint 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers self.policy_syncer.write_sgs_to_etcd(port['security_groups'], context) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/dist-packages/networking_calico/plugins/ml2/drivers/calico/policy.py", line 99, in write_sgs_to_etcd 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers context, filters={'security_group_id': sgids} 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/neutron_lib/db/api.py", line 233, in wrapped 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers return method(*args, **kwargs) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/neutron/db/securitygroups_db.py", line 749, in get_security_group_rules 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers self._ensure_default_security_group(context, project_id) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/neutron/db/securitygroups_db.py", line 871, in _ensure_default_security_group 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers default_sg=True)['id'] 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/neutron_lib/db/api.py", line 233, in wrapped 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers return method(*args, **kwargs) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/neutron/db/securitygroups_db.py", line 114, in create_security_group 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers sg.create() 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/neutron/objects/base.py", line 337, in decorator 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers return func(self, *args, **kwargs) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/neutron/objects/rbac_db.py", line 365, in func 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers return new_method(self, orig_method) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/neutron/objects/rbac_db.py", line 301, in _create_hook 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers orig_create(self) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/neutron/objects/securitygroup.py", line 79, in create 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers default_group.create() 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/neutron/objects/base.py", line 337, in decorator 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers return func(self, *args, **kwargs) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3/dist-packages/neutron/objects/base.py", line 870, in create 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers object_class=self.__class__, db_exception=db_exc) 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers neutron_lib.objects.exceptions.NeutronDbObjectDuplicateEntry: Failed to create a duplicate DefaultSecurityGroup: for attribute(s) ['PRIMARY'] with value(s) 11447be9beda4bf78dab27cdb75058e2 2020-06-15 14:41:44.209 21931 ERROR neutron.plugins.ml2.managers 2020-06-15 14:41:44.227 21931 ERROR neutron.plugins.ml2.plugin [req-6160aa90-b69e-451b-9774-07b6d840f41f 088d04a2122548c5acb628348db93c40 11447be9beda4bf78dab27cdb75058e2 - default default] mechanism_manager.update_port_postcommit failed for port 7dcd8de4-c74a-4b4a-b68b-5701cabfefc5: neutron.plugins.ml2.common.exceptions.MechanismDriverError IIUC, this is triggered by my Neutron driver calling rules = self.db.get_security_group_rules( context, filters={'security_group_id': sgids} ) where the context has project_id 11447be9beda4bf78dab27cdb75058e2. Deep down inside that call, Neutron tries to ensure that there is a default security group for that project, and somehow that hits the reported exception. Here's the code in securitygroups_db.py: def _ensure_default_security_group(self, context, tenant_id): """Create a default security group if one doesn't exist. :returns: the default security group id for given tenant. """ default_group_id = self._get_default_sg_id(context, tenant_id) if default_group_id: return default_group_id security_group = { 'security_group': {'name': 'default', 'tenant_id': tenant_id, 'description': _('Default security group')} } return self.create_security_group(context, security_group, default_sg=True)['id'] Obviously it checks first if the default SG already exists for the project, before creating it if not. So why would that code hit the duplicate exception as shown above? If it helps, the Calico driver code can be found at https://github.com/projectcalico/networking-calico/. The specific get_security_group_rules call, in the traceback above, is here: https://github.com/projectcalico/networking- calico/blob/master/networking_calico/plugins/ml2/drivers/calico/policy.py#L98 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1883730/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
