Re: [Netstack] Database common blueprint

Thu, 03 May 2012 13:31:01 -0700 

On 05/03/2012 08:13 PM, Dan Wendlandt wrote:




On Thu, May 3, 2012 at 10:01 AM, Gary Kotton <gkot...@redhat.com 
<mailto:gkot...@redhat.com>> wrote:


    On 05/03/2012 07:15 PM, Dan Wendlandt wrote:

    Hi Gary,

    Great to see all the effort on improving usability + consistency.

    On Thu, May 3, 2012 at 7:36 AM, Gary Kotton <gkot...@redhat.com
    <mailto:gkot...@redhat.com>> wrote:
    >
    >
    > Hi,
    > I have taken a more thorough look at the database
    implementations. On the plugin side all of the plugins make use
    of the common database code (../quantum/quantum/db/*).


    Yes, or at least, any plugin that uses a database uses the common
    database code.  Some plugins don't have a local database at all,
    as they rely on an external system (NVP is the only example in
    the repo, but I've heard of other people developing plugins that
    don't use a local DB).

    >
    > Today I tested a number of scenarios with connectivity to the
    server. This works well with the linuxbridge and the openvswitch
    plugins. Taking into account that we working on the scalability
    of the agents
    (https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms),
    I am pretty convinced that a large part of the blueprint is
    redundant. Nonetheless I still think we need to address
    > 1. Common configuration for the database strings (similarly to
    those in OpenStack)


    Agreed on this one.

    >
    > 2. If the Quantum service starts and the database is down.
    Quantum service terminates. Output:
    >    [f17-dev@localhost ~]$ quantum list_nets default
    >    Traceback (most recent call last):
    >      File
    "/usr/lib/python2.7/site-packages/quantum/client/cli_lib.py",
    line 237, in list_nets_v11
    >        res = client.list_networks(**filters)
    >      File
    "/usr/lib/python2.7/site-packages/quantum/client/__init__.py",
    line 145, in with_params
    >        ret = self.function(instance, *args)
    >      File
    "/usr/lib/python2.7/site-packages/quantum/client/__init__.py",
    line 522, in list_networks
    >        return self.get(self.networks_path, params=filters)
    >      File
    "/usr/lib/python2.7/site-packages/quantum/client/__init__.py",
    line 379, in get
    >        headers=headers, params=params)
    >      File
    "/usr/lib/python2.7/site-packages/quantum/client/__init__.py",
    line 371, in retry_request
    >        raise exceptions.ConnectionFailed(reason=_("Maximum
    attempts reached"))
    >    ConnectionFailed: Connection to quantum failed: Maximum
    attempts reached


    Does the service terminate?  Or is it just that each request
    fails?  The former would definitely be a problem, but the stack
    trace at least suggests that the service actually catches the error.

    The service terminates. I'll use the fixes below and get back to
    you if it resolved.



Perhaps our setup is different, as I am not seeing the service (i.e., 
quantum-server) terminate if the database is down.  Below I list 
networks, stop mysql, confirm that mysql is down, list networks with 
quantum client (get client error), use screen to confirm that service 
continues to run despite error, restart mysql, list networks with 
quantum client again demonstrating that quantum-server is again 
accessing the restarted database.


Are you seeing something else for the same pattern of events?


I did the following:
1. rebooted the applaince
2. killed the mysql service
3. killed the quantum service
4. started the quantime service
5. when the service starts it exists with an exception at db.create_engine

I should have been more specific. I provided the message from the 
quantum CLI and not the quantum log. My bad.
Yes, this is an edge case. When we first encountered it we resolved it 
by starting the mysql service prior to the quantum. It would be best if 
the service tried to reconnect.


Thanks
Gary


Dan


root@devstack-multi1:~# quantum list_nets default
Virtual Networks for Tenant default
Network ID: f6513015-2ba5-443b-8055-a2809cd34c20
root@devstack-multi1:~# /etc/init.d/mysql stop
root@devstack-multi1:~# mysql -uroot -pnova

ERROR 2002 (HY000): Can't connect to local MySQL server through socket 
'/var/run/mysqld/mysqld.sock' (2)

root@devstack-multi1:~# su stack
stack@devstack-multi1:/root$ screen -x stack

[1]+  Stopped                 screen -x stack
stack@devstack-multi1:/root$ quantum list_nets default
Traceback (most recent call last):

  File "/opt/stack/python-quantumclient/quantum/client/cli_lib.py", 
line 237, in list_nets_v11

    res = client.list_networks(**filters)

  File "/opt/stack/python-quantumclient/quantum/client/__init__.py", 
line 145, in with_params

    ret = self.function(instance, *args)

  File "/opt/stack/python-quantumclient/quantum/client/__init__.py", 
line 522, in list_networks

    return self.get(self.networks_path, params=filters)

  File "/opt/stack/python-quantumclient/quantum/client/__init__.py", 
line 379, in get

    headers=headers, params=params)

  File "/opt/stack/python-quantumclient/quantum/client/__init__.py", 
line 364, in retry_request

    headers=headers, params=params)

  File "/opt/stack/python-quantumclient/quantum/client/__init__.py", 
line 305, in do_request

    self._handle_fault_response(status_code, data)

  File "/opt/stack/python-quantumclient/quantum/client/__init__.py", 
line 228, in _handle_fault_response

    EXCEPTION_HANDLERS[self.version](status_code, des_error_body)

  File "/opt/stack/python-quantumclient/quantum/client/__init__.py", 
line 119, in exception_handler_v11

    raise exceptions.QuantumClientException(message=msg)

QuantumClientException: 500-{'message': 'Traceback (most recent call 
last):\n  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", 
line 336, in handle_one_response\n    result = 
self.application(self.environ, start_response)\n  File 
"/usr/lib/python2.7/dist-packages/paste/urlmap.py", line 203, in 
__call__\n    return app(environ, start_response)\n  File 
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in 
__call__\n    return resp(environ, start_response)\n  File 
"/usr/lib/pymodules/python2.7/routes/middleware.py", line 131, in 
__call__\n    response = self.app(environ, start_response)\n  File 
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in 
__call__\n    return resp(environ, start_response)\n  File 
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in 
__call__\n    return resp(environ, start_response)\n  File 
"/usr/lib/pymodules/python2.7/routes/middleware.py", line 131, in 
__call__\n    response = self.app(environ, start_response)\n  File 
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in 
__call__\n    return resp(environ, start_response)\n  File 
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in 
__call__\n    resp = self.call_func(req, *args, **self.kwargs)\n  File 
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 208, in 
call_func\n    return self.func(req, *args, **kwargs)\n  File 
"/opt/stack/quantum/quantum/wsgi.py", line 748, in __call__\n   
 action_result = self.dispatch(request, action, args)\n  File 
"/opt/stack/quantum/quantum/wsgi.py", line 780, in dispatch\n   
 return controller_method(request=request, **action_args)\n  File 
"/opt/stack/quantum/quantum/api/api_common.py", line 108, in 
the_func\n    return func(*args, **kwargs)\n  File 
"/opt/stack/quantum/quantum/api/networks.py", line 101, in index\n   
 return self._items(request, tenant_id)\n  File 
"/opt/stack/quantum/quantum/api/networks.py", line 84, in _items\n   
 filter_opts=filter_opts)\n  File 
"/opt/stack/quantum/quantum/plugins/openvswitch/ovs_quantum_plugin.py", line 
122, in get_all_networks\n    for x in db.network_list(tenant_id):\n 
 File "/opt/stack/quantum/quantum/db/api.py", line 131, in 
network_list\n    filter_by(tenant_id=tenant_id).\\\n  File 
"/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py", 
line 2066, in all\n    return list(self)\n  File 
"/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py", 
line 2176, in __iter__\n    return 
self._execute_and_instances(context)\n  File 
"/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py", 
line 2189, in _execute_and_instances\n    close_with_result=True)\n 
 File 
"/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py", 
line 2180, in _connection_from_session\n    **kw)\n  File 
"/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py", 
line 729, in connection\n    close_with_result=close_with_result)\n 
 File 
"/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py", 
line 735, in _connection_for_bind\n    return 
engine.contextual_connect(**kwargs)\n  File 
"/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py", 
line 2476, in contextual_connect\n    self.pool.connect(),\n  File 
"/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/pool.py", 
line 224, in connect\n    return _ConnectionFairy(self).checkout()\n 
 File 
"/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/pool.py", 
line 466, in checkout\n    self.connection = 
self._connection_record.get_connection()\n  File 
"/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/pool.py", 
line 300, in get_connection\n    self.connection = self.__connect()\n 
 File 
"/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/pool.py", 
line 330, in __connect\n    connection = self.__pool._creator()\n 
 File 
"/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/engine/strategies.py", 
line 80, in connect\n    return dialect.connect(*cargs, **cparams)\n 
 File 
"/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/engine/default.py", 
line 285, in connect\n    return self.dbapi.connect(*cargs, 
**cparams)\n  File "/usr/lib/pymodules/python2.7/MySQLdb/__init__.py", 
line 81, in Connect\n    return Connection(*args, **kwargs)\n  File 
"/usr/lib/pymodules/python2.7/MySQLdb/connections.py", line 187, in 
__init__\n    super(Connection, self).__init__(*args, 
**kwargs2)\nOperationalError: (OperationalError) (2002, "Can\'t 
connect to local MySQL server through socket 
\'/var/run/mysqld/mysqld.sock\' (2)") None None\n'}

stack@devstack-multi1:/root$ screen -x stack

[2]+  Stopped                 screen -x stack
stack@devstack-multi1:/root$ sudo service mysql start
mysql start/running, process 27677
stack@devstack-multi1:/root$ quantum list_nets default
Virtual Networks for Tenant default
Network ID: f6513015-2ba5-443b-8055-a2809cd34c20










    One improvement we definitely need is to hide such errors from
    the tenant (who should just get generic errors).  We had a bug
    filed on that already, and Guilherme proposed a patch for it
    yesterday: https://review.openstack.org/#/c/7019/ .  Perhaps that
    is what you're suggesting below?

    Brilliant - this addresses part of it (saved a few -1's for me :))



    Thanks!

    dan

    >
    > 3. Error handling - for example when using the CLI and the DB
    is down we get the attached output. It would be better if the
    trace was looged and the user would receive the specific error
    message.
    >    [f17-dev@localhost ~]$ quantum list_nets default
    >    Traceback (most recent call last):
    >      File
    "/usr/lib/python2.7/site-packages/quantum/client/cli_lib.py",
    line 237, in list_nets_v11
    >        res = client.list_networks(**filters)
    >      File
    "/usr/lib/python2.7/site-packages/quantum/client/__init__.py",
    line 145, in with_params
    >        ret = self.function(instance, *args)
    >      File
    "/usr/lib/python2.7/site-packages/quantum/client/__init__.py",
    line 522, in list_networks
    >    ...
    >        raise exceptions.QuantumClientException(message=msg)
    >    QuantumClientException: 500-{'message': 'Traceback (most
    recent call last):\n  File
    "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 336, in
        handle_one_response\n    result =
    self.application(self.environ, start_response)\n  File
    "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 203, in
    __call__\n    return app(environ, start_response)\n  File
    "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in
    __call__\n    return resp(environ, start_response)\n  File
    "/usr/lib/python2.7/site-packages/routes/middleware.py", line
    131, in __call__\n    response = self.app(environ,
    start_response)\n  File
    "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in
    __call__\n    return resp(environ, start_response)\n  File
    "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in
    __call__\n    return resp(environ, start_response)\n  File
    "/usr/lib/python2.7/site-packages/routes/middleware.py", line
    131, in __call__\n    response = self.app(environ,
    start_response)\n  File
    "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in
    __call__\n    return resp(environ, start_response)\n  File
    "/usr/lib/python2.7/site-packages/webob/dec.py", line 147, in
    __call__\n    resp = self.call_func(req, *args, **self.kwargs)\n
     File "/usr/lib/python2.7/site-packages/webob/dec.py", line 210,
    in call_func\n    return self.func(req, *args, **kwargs)\n  File
    "/usr/lib/python2.7/site-packages/quantum/wsgi.py", line 748, in
    __call__\n    action_result = self.dispatch(request, action,
    args)\n  File "/usr/lib/python2.7/site-packages/quantum/wsgi.py",
    line 780, in dispatch\n    return
    controller_method(request=request, **action_args)\n  File
    "/usr/lib/python2.7/site-packages/quantum/api/api_common.py",
    line 108, in the_func\n    return func(*args, **kwargs)\n  File
    "/usr/lib/python2.7/site-packages/quantum/api/networks.py", line
    101, in index\n    return self._items(request, tenant_id)\n  File
    "/usr/lib/python2.7/site-packages/quantum/api/networks.py", line
    84, in _items\n    filter_opts=filter_opts)\n  File
    
"/usr/lib/python2.7/site-packages/quantum/plugins/linuxbridge/LinuxBridgePlugin.py",
    line 71, in get_all_networks\n    networks_list =
    db.network_list(tenant_id)\n  File
    "/usr/lib/python2.7/site-packages/quantum/db/api.py", line 131,
    in network_list\n    filter_by(tenant_id=tenant_id).\\\n  File
    "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py",
    line 2066, in all\n    return list(self)\n  File
    "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py",
    line 2176, in __iter__\n    return
    self._execute_and_instances(context)\n  File
    "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py",

    line 2189, in _execute_and_instances\n  
     close_with_result=True)\n  File

    "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py",
    line 2180, in _connection_from_session\n    **kw)\n  File
    "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py",

    line 729, in connection\n  
     close_with_result=close_with_result)\n  File

    "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py",
    line 735, in _connection_for_bind\n    return
    engine.contextual_connect(**kwargs)\n  File
    "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py",
    line 2476, in contextual_connect\n    self.pool.connect(),\n
     File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py",
    line 224, in connect\n    return
    _ConnectionFairy(self).checkout()\n  File
    "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line
    466, in checkout\n    self.connection =
    self._connection_record.get_connection()\n  File
    "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line
    300, in get_connection\n    self.connection = self.__connect()\n
     File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py",
    line 330, in __connect\n    connection = self.__pool._creator()\n
     File
    "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py",
    line 80, in connect\n    return dialect.connect(*cargs,
    **cparams)\n  File
    "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line
    285, in connect\n    return self.dbapi.connect(*cargs,
    **cparams)\n  File
    "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line
    81, in Connect\n    return Connection(*args, **kwargs)\n  File
    "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line
    187, in __init__\n    super(Connection, self).__init__(*args,
    **kwargs2)\nOperationalError: (OperationalError) (2003, "Can\'t
    connect to MySQL server on \'localhost.localdomain\' (111)") None
    None\n'}
    >
    > Any thought and or comments. Any objections if I update the
    blueprint to deal with the the above items.
    > Thanks
    > Gary
    >
    >
    > --
    > Mailing list: https://launchpad.net/~netstack
    <https://launchpad.net/%7Enetstack>
    > Post to     : netstack@lists.launchpad.net
    <mailto:netstack@lists.launchpad.net>
    > Unsubscribe : https://launchpad.net/~netstack
    <https://launchpad.net/%7Enetstack>
    > More help   : https://help.launchpad.net/ListHelp




    --
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~
    Dan Wendlandt
    Nicira, Inc: www.nicira.com <http://www.nicira.com>
    twitter: danwendlandt
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~





--
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dan Wendlandt
Nicira, Inc: www.nicira.com <http://www.nicira.com>
twitter: danwendlandt
~~~~~~~~~~~~~~~~~~~~~~~~~~~




-- 
Mailing list: https://launchpad.net/~netstack
Post to     : netstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~netstack
More help   : https://help.launchpad.net/ListHelp






















					Reply via email to