Question upon gracefully restarting c* node(s)

[Jing Meng]

Hi all.

Recently we made a change to our production env c* cluster (2.1.18) -
placing the commit log to the same SSD where data is stored, which needs
restarting all nodes.

Before restarting a cassandra node, we ran the following nodetool utils:
$ nodetool disablethrift && sleep 5
$ nodetool disablebinary && sleep 5
$ nodetool disable gossip && sleep 5
$ nodetool drain && sleep 5

It was "graceful" as expected (no significant errors found), but the
process is still a myth to us: are those commands used above "sufficient",
and/or why? The offical doc (docs.datastax.com) did not help with this
operation detail, though "nodetool drain" is apparently essential.

Re: [python] [flask] [CQLAlchemy] NoHostAvailable on create

[Alan Hamlett]

Adding more nodes to the cluster fixed the error. Looks like a bug in
python-driver connection pool:

1. The connection pool only has one host
2. A query times out, causing that connection to be removed from the pool
3. Another query executes, but there are no hosts in the pool

On Mon, Jan 1, 2018 at 12:21 PM, Jeff Jirsa  wrote:

> Well the python driver you reference is a third party driver, because the
> project doesn’t ship official drivers. You may have better luck looking for
> a datastax driver support forum, or wait until after the holiday for more
> people to be checking email.
>
>
> --
> Jeff Jirsa
>
>
> On Jan 1, 2018, at 12:14 PM, Alan Hamlett  wrote:
>
> Still getting the cassandra.cluster.NoHostAvailable error periodically
> from uWSGI hosts. Setting up the connection with postfork:
> https://github.com/alanhamlett/flask-cqlalchemy/blob/
> 653ed3298af7dd617a972e9f87437f6e53f741b9/flask_cqlalchemy/__init__.py#L56
>
> Lazy connection is False, Retry connection is True. Could this be a bug in
> cassandra-driver's connection pooling?
>
> P.S. Blocking a web app when connection isn't available (default non-lazy
> connect) is really bad. With a web app you want requests that don't depend
> on Cassandra to complete, but cassandra-driver blocks all requests when
> there's no Cassandra connection even if it's not needed for the current web
> app's request. This design decision gives me very low confidence in the
> Python cassandra-driver.
>
> On Sun, Dec 31, 2017 at 2:34 PM, Alan Hamlett 
> wrote:
>
>> Thanks for the reply, I think it's related. However, after using a fork
>> of Flask-CQLAlchemy with postfork I'm still getting the NoHostAvailable
>> error once per 4k requests. One strange thing is the error rate doesn't
>> increase with the number of requests, since some uWSGI clients with ~20k
>> requests over the same time period have an error rate of once per 20k
>> requests. Both uWSGI hosts have the same number of worker processes.
>>
>> *Flask-CQLAlchemy Fork with Patch:*
>>
>> https://github.com/alanhamlett/flask-cqlalchemy/tree/a7e5c7c
>> 7cf0c51a19be98791dd4c47b72b97d9be
>>
>> *Error Traceback seen after patch applied:*
>>
>> Failed to create connection pool for new host 10.1.2.3:
>> Traceback (most recent call last):
>>   File "cassandra/cluster.py", line 2452, in
>> cassandra.cluster.Session.add_or_renew_pool.run_add_or_renew_pool
>>   File "cassandra/pool.py", line 332, in cassandra.pool.HostConnection.
>> __init__
>>   File "cassandra/cluster.py", line 1195, in
>> cassandra.cluster.Cluster.connection_factory
>>   File "cassandra/connection.py", line 341, in
>> cassandra.connection.Connection.factory
>> cassandra.OperationTimedOut: errors=Timed out creating connection (5
>> seconds), last_host=None
>> Traceback (most recent call last):
>>   File "./venv/lib/python3.4/site-packages/flask/app.py", line 1982, in
>> wsgi_app
>> response = self.full_dispatch_request()
>>   File "./venv/lib/python3.4/site-packages/flask/app.py", line 1614, in
>> full_dispatch_request
>> rv = self.handle_user_exception(e)
>>   File "./venv/lib/python3.4/site-packages/flask/app.py", line 1517, in
>> handle_user_exception
>> reraise(exc_type, exc_value, tb)
>>   File "./venv/lib/python3.4/site-packages/flask/_compat.py", line 33,
>> in reraise
>> raise value
>>   File "./venv/lib/python3.4/site-packages/flask/app.py", line 1612, in
>> full_dispatch_request
>> rv = self.dispatch_request()
>>   File "./venv/lib/python3.4/site-packages/flask/app.py", line 1598, in
>> dispatch_request
>> return self.view_functions[rule.endpoint](**req.view_args)
>>   File "./app/api_utils.py", line 876, in get_durations
>> use_cassandra=use_cassandra,
>>   File "./venv/lib/python3.4/site-packages/datadog/dogstatsd/context.py",
>> line 53, in wrapped
>> return func(*args, **kwargs)
>>   File "./app/api_utils.py", line 1339, in heartbeats_to_durations
>> for heartbeat in heartbeats:
>>   File "./venv/lib/python3.4/site-packages/cassandra/cqlengine/query.py",
>> line 512, in __iter__
>> self._execute_query()
>>   File "./venv/lib/python3.4/site-packages/cassandra/cqlengine/query.py",
>> line 469, in _execute_query
>> self._result_generator = (i for i in self._execute(self._select_que
>> ry()))
>>   File "./venv/lib/python3.4/site-packages/cassandra/cqlengine/query.py",
>> line 401, in _execute
>> result = _execute_statement(self.model, statement, self._consistency,
>> self._timeout, connection=connection)
>>   File "./venv/lib/python3.4/site-packages/cassandra/cqlengine/query.py",
>> line 1505, in _execute_statement
>> return conn.execute(s, params, timeout=timeout, connection=connection)
>>   File 
>> "./venv/lib/python3.4/site-packages/cassandra/cqlengine/connection.py",
>> line 341, in execute
>> result = conn.session.execute(query, params, timeout=timeout)
>>   File "cassandra/cluster.py", line 2122, in
>> 

Reg:- limitation as PROS and CONS of Using Collections in Data modeling

[@Nandan@]

Hi All,
I want to know, what will be a limitation in case of using Collections such
as SET, LIST, MAP?
Like in my condition, which inserting Video details, I have to insert
language based such as
Language:- English
Title:- Video Name
Language:- Hindi
Title:- Video_name in Hindi
Language:- Chinese
Title:- Video_name in Chinese.
The condition for using this, like my website is having multi-language
support condition, so If User will open English Page, Video title will show
in English and When User open Chinese page, then Video_title in Chinese
will be displayed.

Please suggest me.

Thanks and Best Regards,

Re: How to get page id without transmitting data to client

[Eunsu Kim]

Thank you for your response. happy new year


> On 30 Dec 2017, at 5:33 AM, Andy Tolbert  wrote:
> 
> Hi Eunsu,
> 
> Unfortunately there is not really a way to do this that I'm aware of.  The 
> page id contains data indicating where to start reading the next set of rows 
> (such as partition and clustering information), and in order to get to that 
> position you have to actually read the data.
> 
> The driver does have an API for manually specifying the page id to use and 
> we've documented some strategies 
> 
>  for storing and reusing the page id later, but not sure if that helps for 
> your particular use case.
> 
> Thanks,
> Andy
> 
> On Thu, Dec 28, 2017 at 9:11 PM, Eunsu Kim  > wrote:
> Hello everybody,
> 
> I am using the datastax Java driver (3.3.0).
> 
> When query large amounts of data, we set the fetch size (1) and transmit 
> the data to the browser on a page-by-page basis.
> 
> I am wondering if I can get the page id without receiving the real rows from 
> the cassandra to my server.
> 
> I only need 100 in front of 100,000. But I want the next page to be 11th.
> 
> If you have a good idea, please share it.
> 
> Thank you.
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
> 
> For additional commands, e-mail: user-h...@cassandra.apache.org 
> 
> 
> 

Re: [python] [flask] [CQLAlchemy] NoHostAvailable on create

[Jeff Jirsa]

Well the python driver you reference is a third party driver, because the 
project doesn’t ship official drivers. You may have better luck looking for a 
datastax driver support forum, or wait until after the holiday for more people 
to be checking email.


-- 
Jeff Jirsa


> On Jan 1, 2018, at 12:14 PM, Alan Hamlett  wrote:
> 
> Still getting the cassandra.cluster.NoHostAvailable error periodically from 
> uWSGI hosts. Setting up the connection with postfork:
> https://github.com/alanhamlett/flask-cqlalchemy/blob/653ed3298af7dd617a972e9f87437f6e53f741b9/flask_cqlalchemy/__init__.py#L56
> 
> Lazy connection is False, Retry connection is True. Could this be a bug in 
> cassandra-driver's connection pooling?
> 
> P.S. Blocking a web app when connection isn't available (default non-lazy 
> connect) is really bad. With a web app you want requests that don't depend on 
> Cassandra to complete, but cassandra-driver blocks all requests when there's 
> no Cassandra connection even if it's not needed for the current web app's 
> request. This design decision gives me very low confidence in the Python 
> cassandra-driver.
> 
>> On Sun, Dec 31, 2017 at 2:34 PM, Alan Hamlett  wrote:
>> Thanks for the reply, I think it's related. However, after using a fork of 
>> Flask-CQLAlchemy with postfork I'm still getting the NoHostAvailable error 
>> once per 4k requests. One strange thing is the error rate doesn't increase 
>> with the number of requests, since some uWSGI clients with ~20k requests 
>> over the same time period have an error rate of once per 20k requests. Both 
>> uWSGI hosts have the same number of worker processes.
>> 
>> Flask-CQLAlchemy Fork with Patch:
>> 
>> https://github.com/alanhamlett/flask-cqlalchemy/tree/a7e5c7c7cf0c51a19be98791dd4c47b72b97d9be
>> 
>> Error Traceback seen after patch applied:
>> 
>> Failed to create connection pool for new host 10.1.2.3:
>> Traceback (most recent call last):
>>   File "cassandra/cluster.py", line 2452, in 
>> cassandra.cluster.Session.add_or_renew_pool.run_add_or_renew_pool
>>   File "cassandra/pool.py", line 332, in 
>> cassandra.pool.HostConnection.__init__
>>   File "cassandra/cluster.py", line 1195, in 
>> cassandra.cluster.Cluster.connection_factory
>>   File "cassandra/connection.py", line 341, in 
>> cassandra.connection.Connection.factory
>> cassandra.OperationTimedOut: errors=Timed out creating connection (5 
>> seconds), last_host=None
>> Traceback (most recent call last):
>>   File "./venv/lib/python3.4/site-packages/flask/app.py", line 1982, in 
>> wsgi_app
>> response = self.full_dispatch_request()
>>   File "./venv/lib/python3.4/site-packages/flask/app.py", line 1614, in 
>> full_dispatch_request
>> rv = self.handle_user_exception(e)
>>   File "./venv/lib/python3.4/site-packages/flask/app.py", line 1517, in 
>> handle_user_exception
>> reraise(exc_type, exc_value, tb)
>>   File "./venv/lib/python3.4/site-packages/flask/_compat.py", line 33, in 
>> reraise
>> raise value
>>   File "./venv/lib/python3.4/site-packages/flask/app.py", line 1612, in 
>> full_dispatch_request
>> rv = self.dispatch_request()
>>   File "./venv/lib/python3.4/site-packages/flask/app.py", line 1598, in 
>> dispatch_request
>> return self.view_functions[rule.endpoint](**req.view_args)
>>   File "./app/api_utils.py", line 876, in get_durations
>> use_cassandra=use_cassandra,
>>   File "./venv/lib/python3.4/site-packages/datadog/dogstatsd/context.py", 
>> line 53, in wrapped
>> return func(*args, **kwargs)
>>   File "./app/api_utils.py", line 1339, in heartbeats_to_durations
>> for heartbeat in heartbeats:
>>   File "./venv/lib/python3.4/site-packages/cassandra/cqlengine/query.py", 
>> line 512, in __iter__
>> self._execute_query()
>>   File "./venv/lib/python3.4/site-packages/cassandra/cqlengine/query.py", 
>> line 469, in _execute_query
>> self._result_generator = (i for i in self._execute(self._select_query()))
>>   File "./venv/lib/python3.4/site-packages/cassandra/cqlengine/query.py", 
>> line 401, in _execute
>> result = _execute_statement(self.model, statement, self._consistency, 
>> self._timeout, connection=connection)
>>   File "./venv/lib/python3.4/site-packages/cassandra/cqlengine/query.py", 
>> line 1505, in _execute_statement
>> return conn.execute(s, params, timeout=timeout, connection=connection)
>>   File 
>> "./venv/lib/python3.4/site-packages/cassandra/cqlengine/connection.py", line 
>> 341, in execute
>> result = conn.session.execute(query, params, timeout=timeout)
>>   File "cassandra/cluster.py", line 2122, in 
>> cassandra.cluster.Session.execute
>>   File "cassandra/cluster.py", line 3982, in 
>> cassandra.cluster.ResponseFuture.result
>> cassandra.cluster.NoHostAvailable: ('Unable to complete the operation 
>> against any hosts', {})
>> 
>>> On Sun, Dec 31, 2017 at 9:04 AM, Jeff Jirsa  wrote:
>>> uWSGI forks and the driver / cqlalchemy 

Re: [python] [flask] [CQLAlchemy] NoHostAvailable on create

[Alan Hamlett]

Still getting the cassandra.cluster.NoHostAvailable error periodically from
uWSGI hosts. Setting up the connection with postfork:
https://github.com/alanhamlett/flask-cqlalchemy/blob/653ed3298af7dd617a972e9f87437f6e53f741b9/flask_cqlalchemy/__init__.py#L56

Lazy connection is False, Retry connection is True. Could this be a bug in
cassandra-driver's connection pooling?

P.S. Blocking a web app when connection isn't available (default non-lazy
connect) is really bad. With a web app you want requests that don't depend
on Cassandra to complete, but cassandra-driver blocks all requests when
there's no Cassandra connection even if it's not needed for the current web
app's request. This design decision gives me very low confidence in the
Python cassandra-driver.

On Sun, Dec 31, 2017 at 2:34 PM, Alan Hamlett 
wrote:

> Thanks for the reply, I think it's related. However, after using a fork of
> Flask-CQLAlchemy with postfork I'm still getting the NoHostAvailable error
> once per 4k requests. One strange thing is the error rate doesn't increase
> with the number of requests, since some uWSGI clients with ~20k requests
> over the same time period have an error rate of once per 20k requests. Both
> uWSGI hosts have the same number of worker processes.
>
> *Flask-CQLAlchemy Fork with Patch:*
>
> https://github.com/alanhamlett/flask-cqlalchemy/tree/
> a7e5c7c7cf0c51a19be98791dd4c47b72b97d9be
>
> *Error Traceback seen after patch applied:*
>
> Failed to create connection pool for new host 10.1.2.3:
> Traceback (most recent call last):
>   File "cassandra/cluster.py", line 2452, in cassandra.cluster.Session.add_
> or_renew_pool.run_add_or_renew_pool
>   File "cassandra/pool.py", line 332, in cassandra.pool.HostConnection.
> __init__
>   File "cassandra/cluster.py", line 1195, in cassandra.cluster.Cluster.
> connection_factory
>   File "cassandra/connection.py", line 341, in cassandra.connection.
> Connection.factory
> cassandra.OperationTimedOut: errors=Timed out creating connection (5
> seconds), last_host=None
> Traceback (most recent call last):
>   File "./venv/lib/python3.4/site-packages/flask/app.py", line 1982, in
> wsgi_app
> response = self.full_dispatch_request()
>   File "./venv/lib/python3.4/site-packages/flask/app.py", line 1614, in
> full_dispatch_request
> rv = self.handle_user_exception(e)
>   File "./venv/lib/python3.4/site-packages/flask/app.py", line 1517, in
> handle_user_exception
> reraise(exc_type, exc_value, tb)
>   File "./venv/lib/python3.4/site-packages/flask/_compat.py", line 33, in
> reraise
> raise value
>   File "./venv/lib/python3.4/site-packages/flask/app.py", line 1612, in
> full_dispatch_request
> rv = self.dispatch_request()
>   File "./venv/lib/python3.4/site-packages/flask/app.py", line 1598, in
> dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
>   File "./app/api_utils.py", line 876, in get_durations
> use_cassandra=use_cassandra,
>   File "./venv/lib/python3.4/site-packages/datadog/dogstatsd/context.py",
> line 53, in wrapped
> return func(*args, **kwargs)
>   File "./app/api_utils.py", line 1339, in heartbeats_to_durations
> for heartbeat in heartbeats:
>   File "./venv/lib/python3.4/site-packages/cassandra/cqlengine/query.py",
> line 512, in __iter__
> self._execute_query()
>   File "./venv/lib/python3.4/site-packages/cassandra/cqlengine/query.py",
> line 469, in _execute_query
> self._result_generator = (i for i in self._execute(self._select_
> query()))
>   File "./venv/lib/python3.4/site-packages/cassandra/cqlengine/query.py",
> line 401, in _execute
> result = _execute_statement(self.model, statement, self._consistency,
> self._timeout, connection=connection)
>   File "./venv/lib/python3.4/site-packages/cassandra/cqlengine/query.py",
> line 1505, in _execute_statement
> return conn.execute(s, params, timeout=timeout, connection=connection)
>   File "./venv/lib/python3.4/site-packages/cassandra/cqlengine/connection.py",
> line 341, in execute
> result = conn.session.execute(query, params, timeout=timeout)
>   File "cassandra/cluster.py", line 2122, in cassandra.cluster.Session.
> execute
>   File "cassandra/cluster.py", line 3982, in cassandra.cluster.
> ResponseFuture.result
> cassandra.cluster.NoHostAvailable: ('Unable to complete the operation
> against any hosts', {})
>
> On Sun, Dec 31, 2017 at 9:04 AM, Jeff Jirsa  wrote:
>
>> uWSGI forks and the driver / cqlalchemy may need to reconnect or
>> otherwise fix the state after each fork - you could try to prove this is
>> the cause by checking uWSGI logs or ps for indication that a worker process
>> has exited/been recycled. If you think it may be related to this, check out
>> @postfork decorator
>>
>>
>> --
>> Jeff Jirsa
>>
>>
>> On Dec 31, 2017, at 8:52 AM, Alan Hamlett  wrote:
>>
>> More info: The NoHostAvailable error is happening at random times on each
>> client host, 

Re: Upgrade from 1.2.x to 2.0.x, upgradesstables has doubled the size on disk?

[Matija Gobec]

Dan,

What partitioner are you using and did you just swap out the binary?
Going from 70GB to 200GB+ is extremely odd in any scenario.
Maybe Carlos Rolo has an idea about this issue. He did a ton of 1.2 cluster
upgrades.

As for the tombstones, its the stat for the last five minutes. You could
have had it on the old cluster but didn't catch it with cfstats.

Side note:
Looking at the outputs looks like you have something wrong with the data
model too. 2GB partition size is kinda big.

Best,
Matija

On Sun, Dec 31, 2017 at 3:47 AM, Dan Washusen  wrote:

> Nope, just ran out of disk space... So on 1.2.x I had 70GB used on a 200GB
> disk and everything was great, with 2.0.x I'm now at 99% used and getting
> exception while compacting about insufficient disk space. FML...
>
> Dan Washusen
>
>
> On Sun, Dec 31, 2017 at 6:47 AM, Dan Washusen  wrote:
>
>> Thanks for the response Jeff. It wasn't snapshots but after running
>> upgradesstables on all nodes I started a repair and it seems like the file
>> sizes are reducing:
>>
>>  INFO [CompactionExecutor:1626] 2017-12-30 19:42:36,065
>> CompactionTask.java (line 299) Compacted 2 sstables to
>> [/appdata/lib/cassandra/data/dp/s_evt/dp-s_evt-jb-302,].  9,663,696,752
>> bytes to 4,834,895,601 (~50% of original) in 3,899,888ms = 1.182320MB/s.
>> 90,533 total partitions merged to 45,278.  Partition merge counts were
>> {1:23, 2:45255, }
>>
>> Dan Washusen
>>
>>
>> On Sun, Dec 31, 2017 at 1:51 AM, Jeff Jirsa  wrote:
>>
>>> 1.2 to 2.0 was a long time ago for many of us, but I don’t recall
>>> anything that should have doubled size other than perhaps temporarily
>>> during the sstable rewrite or snapshots (which may? Be automatic on
>>> upgrade).
>>>
>>> The bloom filters, sstable count, compression ratio in cfstats all look
>>> similar, only the size is double, so that sorta hints st maybe a snapshot
>>>
>>> You have few sstables, looks like STCS, so it’d be possible that if the
>>> upgrade is still running, maybe one sstable of the old version still
>>> (temporarily) exists on disk causing it to be double counted.
>>>
>>>
>>>
>>> --
>>> Jeff Jirsa
>>>
>>>
>>> On Dec 29, 2017, at 4:33 PM, Dan Washusen  wrote:
>>>
>>> Hi All,
>>> We're taking advantage of the lull in traffic to go through a production
>>> cluster upgrade from 1.2.x (latest) to 2.0.x (latest). We have three nodes
>>> with a replication factor of three. I've noticed that the 'space used' has
>>> almost doubled as a result of running 'nodetool upgradesstables'.
>>>
>>> Anyone have any ideas? Is that to be expected?
>>>
>>> For comparison, on a node (pre-upgrade):
>>>
 nodetool cfstats dp.s_evt
 Keyspace: dp
 Read Count: 190570567
 Read Latency: 2.6280611004164145 ms.
 Write Count: 46213651
 Write Latency: 0.08166790944519835 ms.
 Pending Tasks: 0
 Column Family: s_evt
 SSTable count: 8
 Space used (live): 36269415929
 Space used (total): 36274282945
 SSTable Compression Ratio: 0.2345030140572
 Number of Keys (estimate): 3213696
 Memtable Columns Count: 2934
 Memtable Data Size: 9561951
 Memtable Switch Count: 1974
 Read Count: 190570567
 Read Latency: 2.628 ms.
 Write Count: 46213651
 Write Latency: 0.082 ms.
 Pending Tasks: 0
 Bloom Filter False Positives: 1162636
 Bloom Filter False Ratio: 0.73869
 Bloom Filter Space Used: 4492256
 Compacted row minimum size: 373
 Compacted row maximum size: 1996099046
 Compacted row mean size: 63595
 Average live cells per slice (last five minutes): 11.0
 Average tombstones per slice (last five minutes): 0.0
>>>
>>>
>>> And after upgrading and running 'upgradesstables' (different node):
>>>
 nodetool cfstats dp.s_evt
 Keyspace: dp
 Read Count: 1461617
 Read Latency: 4.9734411921864625 ms.
 Write Count: 359250
 Write Latency: 0.11328054279749478 ms.
 Pending Tasks: 0
 Table: s_evt
 SSTable count: 6
 Space used (live), bytes: 71266932602
 Space used (total), bytes: 71266932602
 Off heap memory used (total), bytes: 44853104
 SSTable Compression Ratio: 0.2387480210082192
 Number of keys (estimate): 3307776
 Memtable cell count: 603223
 Memtable data size, bytes: 121913569
 Memtable switch count: 9
 Local read count: