Hi Javier,

Glad to hear it is solved now. Cassandra 3.11.1 should be a more stable
version and 3.11 a better series.

Excuse my misunderstanding, your table seems to be better designed than

Welcome to the Apache Cassandra community!

C*heers ;-)
Alain Rodriguez - @arodream - al...@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting

2018-02-19 9:31 GMT+00:00 Javier Pareja <pareja.jav...@gmail.com>:

> Hi,
> Thank you for your reply.
> As I was bothered by this problem, last night I upgraded the cluster to
> version 3.11.1 and everything is working now. As far as I can tell the
> counter table can be read now. I will be doing more testing today with this
> version but it is looking good.
> To answer your questions:
> - I might not have explained the table definition very well but the table
> does not have 6 partitions, but 6 partition keys. There are thousands of
> partitions in that table, a combination of all those partition keys. I also
> made sure that the partitions remained small when designing the table.
> - I also enabled tracing in the CQLSH but it showed nothing when querying
> this row. It however did when querying other tables...
> Thanks again for your reply!! I am very excited to be part of the
> Cassandra user base.
> Javier
> F Javier Pareja
> On Mon, Feb 19, 2018 at 8:08 AM, Alain RODRIGUEZ <arodr...@gmail.com>
> wrote:
>> Hello,
>> This table has 6 partition keys, 4 primary keys and 5 counters.
>> I think the root issue is this ^. There might be some inefficiency or
>> issues with counter, but this design, makes Cassandra relatively
>> inefficient in most cases and using standard columns or counters
>> indifferently.
>> Cassandra data is supposed to be well distributed for a maximal
>> efficiency. With only 6 partitions, if you have 6+ nodes, there is 100%
>> chances that the load is fairly imbalanced. If you have less nodes, it's
>> still probably poorly balanced. Also reading from a small number of
>> sstables and in parallel within many nodes ideally to split the work and
>> make queries efficient, but in this case cassandra is reading huge
>> partitions from one node most probably. When the size of the request is too
>> big it can timeout. I am not sure how pagination works with counters, but I
>> believe even if pagination is working, at some point, you are just reading
>> too much (or too inefficiently) and the timeout is reached.
>> I imagined it worked well for a while as counters are very small columns
>> / tables compared to any event data but at some point you might have
>> reached 'physical' limit, because you are pulling *all* the information
>> you need from one partition (and probably many SSTables)
>> Is there really no other way to design this use case?
>> When data starts to be inserted, I can query the counters correctly from
>>> that particular row but after a few minutes updating the table with
>>> thousands of events, I get a read timeout every time
>> Troubleshot:
>> - Use tracing to understand what takes so long with your queries
>> - Check for warns / error in the logs. Cassandra use to complain when it
>> is unhappy with the configurations. There a lot of interesting and it's
>> been a while I last had a failure with no relevant informations in the logs.
>> - Check SSTable per read and other read performances for this counter
>> table. Using some monitoring could make the reason of this timeout obvious.
>> If you use Datadog for example, I guess that a quick look at the "Read
>> Path" Dashboard would help. If you are using any other tool, look for
>> SSTable per reads, Tombstone scanned (if any), keycache hitrate, resources
>> (as maybe fast insert rate compactions and implicit 'read-before-writes'
>> are making the machine less responsive.
>> Fix:
>> - Improve design to improve the findings you made above ^
>> - Improve compaction strategy or read operations depending on the
>> findings above ^
>> I am not saying there is no bug in counters and in your version, but I
>> would say it is to early to state this, given the data model, some other
>> reasons could explain this slowness.
>> If you don't have any monitoring in place, tracing and logs are a nice
>> place to start digging. If you want to share those here, we can help
>> interpreting outputs you will share if needed :).
>> C*heers,
>> Alain
>> 2018-02-17 11:40 GMT+00:00 Javier Pareja <pareja.jav...@gmail.com>:
>>> Hello everyone,
>>> I get a timeout error when reading a particular row from a large
>>> counters table.
>>> I have a storm topology that inserts data into a Cassandra counter
>>> table. This table has 6 partition keys, 4 primary keys and 5 counters.
>>> When data starts to be inserted, I can query the counters correctly from
>>> that particular row but after a few minutes updating the table with
>>> thousands of events, I get a readtimeout every time I try to read a
>>> particular row from the table (the most frequently updated). Other rows I
>>> can read quick and fine. Also if I run "select *", the top few hundreds are
>>> returned quick and fine as expected. The storm topology is stopped but the
>>> error is still there.
>>> I am using Cassandra 3.6.
>>> More information here:
>>> https://stackoverflow.com/q/48833146
>>> Are counters in this version broken? I run the query from CQLSH and get
>>> the same error every time. I tried running it with trace enabled and get
>>> nothing but the error:
>>> ReadTimeout: Error from server: code=1200 [Coordinator node timed out 
>>> waiting for replica nodes' responses] message="Operation timed out - 
>>> received only 0 responses." info={'received_responses': 0, 
>>> 'required_responses': 1, 'consistency': 'ONE'}
>>> Any ideas?

Reply via email to