Re: [python] [flask] [CQLAlchemy] NoHostAvailable on create

2018-01-05 Thread Jeff Jirsa
The warn is a hint you’ve got tombstones, maybe not a big deal, but a hint at your data model. It’s not causing this The log at INFO is Cassandra connection to your app getting severed, Cassandra is saying the reset is on the other side (app side, maybe firewall or something in the middle

Re: [python] [flask] [CQLAlchemy] NoHostAvailable on create

2018-01-05 Thread Alan Hamlett
Update: Still getting the NoHostAvailable periodically in client logs. Also seeing these INFO and WARN messages in /var/log/cassandra/system.log INFO [epollEventLoopGroup-2-5] 2018-01-06 01:39:02,412 Message.java:623 - Unexpected exception during request; channel = [id: 0xae99b597,

Re: [python] [flask] [CQLAlchemy] NoHostAvailable on create

2018-01-02 Thread Alan Hamlett
Still getting the NoHostAvailable with more hosts, just occurring less frequently. Created a JIRA issue on the Python cassandra-driver tracker: https://datastax-oss.atlassian.net/browse/PYTHON-891 On Mon, Jan 1, 2018 at 8:43 PM, Alan Hamlett wrote: > Adding more nodes to

Re: [python] [flask] [CQLAlchemy] NoHostAvailable on create

2018-01-01 Thread Alan Hamlett
Adding more nodes to the cluster fixed the error. Looks like a bug in python-driver connection pool: 1. The connection pool only has one host 2. A query times out, causing that connection to be removed from the pool 3. Another query executes, but there are no hosts in the pool On Mon, Jan 1,

Re: [python] [flask] [CQLAlchemy] NoHostAvailable on create

2018-01-01 Thread Jeff Jirsa
Well the python driver you reference is a third party driver, because the project doesn’t ship official drivers. You may have better luck looking for a datastax driver support forum, or wait until after the holiday for more people to be checking email. -- Jeff Jirsa > On Jan 1, 2018, at

Re: [python] [flask] [CQLAlchemy] NoHostAvailable on create

2018-01-01 Thread Alan Hamlett
Still getting the cassandra.cluster.NoHostAvailable error periodically from uWSGI hosts. Setting up the connection with postfork: https://github.com/alanhamlett/flask-cqlalchemy/blob/653ed3298af7dd617a972e9f87437f6e53f741b9/flask_cqlalchemy/__init__.py#L56 Lazy connection is False, Retry

Re: [python] [flask] [CQLAlchemy] NoHostAvailable on create

2017-12-31 Thread Alan Hamlett
Thanks for the reply, I think it's related. However, after using a fork of Flask-CQLAlchemy with postfork I'm still getting the NoHostAvailable error once per 4k requests. One strange thing is the error rate doesn't increase with the number of requests, since some uWSGI clients with ~20k requests

Re: [python] [flask] [CQLAlchemy] NoHostAvailable on create

2017-12-31 Thread Jeff Jirsa
uWSGI forks and the driver / cqlalchemy may need to reconnect or otherwise fix the state after each fork - you could try to prove this is the cause by checking uWSGI logs or ps for indication that a worker process has exited/been recycled. If you think it may be related to this, check out

Re: [python] [flask] [CQLAlchemy] NoHostAvailable on create

2017-12-31 Thread Alan Hamlett
More info: The NoHostAvailable error is happening at random times on each client host, so it's probably a client error. If the Cassandra cluster was really offline then all client hosts would report the error at the same time instead of different random times. The NoHostAvailable error occurs

[python] [flask] [CQLAlchemy] NoHostAvailable on create

2017-12-31 Thread Alan Hamlett
I'm seeing tracebacks in my Python Flask app when creating rows: Traceback (most recent call last): File "/opt/app/current/app/api.py", line 1174, in consume_heartbeat Heartbeat.create(**form_data) File "/opt/app/current/venv/lib/python3.4/site-packages/cassandra/cqlengine/models.py",