Well the python driver you reference is a third party driver, because the project doesn’t ship official drivers. You may have better luck looking for a datastax driver support forum, or wait until after the holiday for more people to be checking email.
-- Jeff Jirsa > On Jan 1, 2018, at 12:14 PM, Alan Hamlett <alan.haml...@gmail.com> wrote: > > Still getting the cassandra.cluster.NoHostAvailable error periodically from > uWSGI hosts. Setting up the connection with postfork: > https://github.com/alanhamlett/flask-cqlalchemy/blob/653ed3298af7dd617a972e9f87437f6e53f741b9/flask_cqlalchemy/__init__.py#L56 > > Lazy connection is False, Retry connection is True. Could this be a bug in > cassandra-driver's connection pooling? > > P.S. Blocking a web app when connection isn't available (default non-lazy > connect) is really bad. With a web app you want requests that don't depend on > Cassandra to complete, but cassandra-driver blocks all requests when there's > no Cassandra connection even if it's not needed for the current web app's > request. This design decision gives me very low confidence in the Python > cassandra-driver. > >> On Sun, Dec 31, 2017 at 2:34 PM, Alan Hamlett <alan.haml...@gmail.com> wrote: >> Thanks for the reply, I think it's related. However, after using a fork of >> Flask-CQLAlchemy with postfork I'm still getting the NoHostAvailable error >> once per 4k requests. One strange thing is the error rate doesn't increase >> with the number of requests, since some uWSGI clients with ~20k requests >> over the same time period have an error rate of once per 20k requests. Both >> uWSGI hosts have the same number of worker processes. >> >> Flask-CQLAlchemy Fork with Patch: >> >> https://github.com/alanhamlett/flask-cqlalchemy/tree/a7e5c7c7cf0c51a19be98791dd4c47b72b97d9be >> >> Error Traceback seen after patch applied: >> >> Failed to create connection pool for new host 10.1.2.3: >> Traceback (most recent call last): >> File "cassandra/cluster.py", line 2452, in >> cassandra.cluster.Session.add_or_renew_pool.run_add_or_renew_pool >> File "cassandra/pool.py", line 332, in >> cassandra.pool.HostConnection.__init__ >> File "cassandra/cluster.py", line 1195, in >> cassandra.cluster.Cluster.connection_factory >> File "cassandra/connection.py", line 341, in >> cassandra.connection.Connection.factory >> cassandra.OperationTimedOut: errors=Timed out creating connection (5 >> seconds), last_host=None >> Traceback (most recent call last): >> File "./venv/lib/python3.4/site-packages/flask/app.py", line 1982, in >> wsgi_app >> response = self.full_dispatch_request() >> File "./venv/lib/python3.4/site-packages/flask/app.py", line 1614, in >> full_dispatch_request >> rv = self.handle_user_exception(e) >> File "./venv/lib/python3.4/site-packages/flask/app.py", line 1517, in >> handle_user_exception >> reraise(exc_type, exc_value, tb) >> File "./venv/lib/python3.4/site-packages/flask/_compat.py", line 33, in >> reraise >> raise value >> File "./venv/lib/python3.4/site-packages/flask/app.py", line 1612, in >> full_dispatch_request >> rv = self.dispatch_request() >> File "./venv/lib/python3.4/site-packages/flask/app.py", line 1598, in >> dispatch_request >> return self.view_functions[rule.endpoint](**req.view_args) >> File "./app/api_utils.py", line 876, in get_durations >> use_cassandra=use_cassandra, >> File "./venv/lib/python3.4/site-packages/datadog/dogstatsd/context.py", >> line 53, in wrapped >> return func(*args, **kwargs) >> File "./app/api_utils.py", line 1339, in heartbeats_to_durations >> for heartbeat in heartbeats: >> File "./venv/lib/python3.4/site-packages/cassandra/cqlengine/query.py", >> line 512, in __iter__ >> self._execute_query() >> File "./venv/lib/python3.4/site-packages/cassandra/cqlengine/query.py", >> line 469, in _execute_query >> self._result_generator = (i for i in self._execute(self._select_query())) >> File "./venv/lib/python3.4/site-packages/cassandra/cqlengine/query.py", >> line 401, in _execute >> result = _execute_statement(self.model, statement, self._consistency, >> self._timeout, connection=connection) >> File "./venv/lib/python3.4/site-packages/cassandra/cqlengine/query.py", >> line 1505, in _execute_statement >> return conn.execute(s, params, timeout=timeout, connection=connection) >> File >> "./venv/lib/python3.4/site-packages/cassandra/cqlengine/connection.py", line >> 341, in execute >> result = conn.session.execute(query, params, timeout=timeout) >> File "cassandra/cluster.py", line 2122, in >> cassandra.cluster.Session.execute >> File "cassandra/cluster.py", line 3982, in >> cassandra.cluster.ResponseFuture.result >> cassandra.cluster.NoHostAvailable: ('Unable to complete the operation >> against any hosts', {}) >> >>> On Sun, Dec 31, 2017 at 9:04 AM, Jeff Jirsa <jji...@gmail.com> wrote: >>> uWSGI forks and the driver / cqlalchemy may need to reconnect or otherwise >>> fix the state after each fork - you could try to prove this is the cause by >>> checking uWSGI logs or ps for indication that a worker process has >>> exited/been recycled. If you think it may be related to this, check out >>> @postfork decorator >>> >>> >>> -- >>> Jeff Jirsa >>> >>> >>>> On Dec 31, 2017, at 8:52 AM, Alan Hamlett <alan.haml...@gmail.com> wrote: >>>> >>>> More info: The NoHostAvailable error is happening at random times on each >>>> client host, so it's probably a client error. If the Cassandra cluster was >>>> really offline then all client hosts would report the error at the same >>>> time instead of different random times. The NoHostAvailable error occurs >>>> about once every 30 minutes, so most request call Model.create() without >>>> the error. >>>> >>>>> On Sun, Dec 31, 2017 at 1:07 AM, Alan Hamlett <alan.haml...@gmail.com> >>>>> wrote: >>>>> I'm seeing tracebacks in my Python Flask app when creating rows: >>>>> >>>>> Traceback (most recent call last): >>>>> File "/opt/app/current/app/api.py", line 1174, in consume_heartbeat >>>>> Heartbeat.create(**form_data) >>>>> File >>>>> "/opt/app/current/venv/lib/python3.4/site-packages/cassandra/cqlengine/models.py", >>>>> line 672, in create >>>>> return cls.objects.create(**kwargs) >>>>> File >>>>> "/opt/app/current/venv/lib/python3.4/site-packages/cassandra/cqlengine/query.py", >>>>> line 977, in create >>>>> .using(connection=self._connection) \ >>>>> File >>>>> "/opt/app/current/venv/lib/python3.4/site-packages/cassandra/cqlengine/models.py", >>>>> line 738, in save >>>>> if_exists=self._if_exists).save() >>>>> File >>>>> "/opt/app/current/venv/lib/python3.4/site-packages/cassandra/cqlengine/query.py", >>>>> line 1476, in save >>>>> self._execute(insert) >>>>> File >>>>> "/opt/app/current/venv/lib/python3.4/site-packages/cassandra/cqlengine/query.py", >>>>> line 1351, in _execute >>>>> results = _execute_statement(self.model, statement, >>>>> self._consistency, self._timeout, connection=connection) >>>>> File >>>>> "/opt/app/current/venv/lib/python3.4/site-packages/cassandra/cqlengine/query.py", >>>>> line 1505, in _execute_statement >>>>> return conn.execute(s, params, timeout=timeout, connection=connection) >>>>> File >>>>> "/opt/app/current/venv/lib/python3.4/site-packages/cassandra/cqlengine/connection.py", >>>>> line 341, in execute >>>>> result = conn.session.execute(query, params, timeout=timeout) >>>>> File "cassandra/cluster.py", line 2122, in >>>>> cassandra.cluster.Session.execute >>>>> File "cassandra/cluster.py", line 3982, in >>>>> cassandra.cluster.ResponseFuture.result >>>>> cassandra.cluster.NoHostAvailable: ('Unable to complete the operation >>>>> against any hosts', {}) >>>>> >>>>> I'm using the cassandra-driver client library 3.12.0 via Flask-CQLAlchemy >>>>> 1.2.0 (https://github.com/thegeorgeous/flask-cqlalchemy) with uWSGI >>>>> (https://github.com/unbit/uwsgi). >>>>> >>>>> cassandra.cqlengine.connection.setup is being passed lazy_connect=True >>>>> and retry_connect=Truecassandra.cqlengine because lazy_connect=False >>>>> causes requests to timeout to the Flask app for some reason. >>>>> >>>>> Also seeing these errors in my uWSGI log file: >>>>> >>>>> [control connection] Error connecting to 10.1.2.3: Traceback (most recent >>>>> call last): File "cassandra/cluster.py", line 2781, in >>>>> cassandra.cluster.ControlConnection._reconnect_internal File >>>>> "cassandra/cluster.py", line 2803, in >>>>> cassandra.cluster.ControlConnection._try_connect File >>>>> "cassandra/cluster.py", line 1195, in >>>>> cassandra.cluster.Cluster.connection_factory File >>>>> "cassandra/connection.py", line 341, in >>>>> cassandra.connection.Connection.factory cassandra.OperationTimedOut: >>>>> errors=Timed out creating connection (5 seconds), last_host=None >>>>> >>>>> What's causing these connection and timeout errors? Something related to >>>>> Flask-CQLAlchemy? >>>> >>>> >> >> > > > > -- > Alan Hamlett > ahamlett.com
