Re: Very odd & inconsistent results from SASI query

2017-03-20 Thread Voytek Jarnot
Apologies for the stream-of-consciousness replies, but are the dropped message stats output by tpstats an accumulation since the node came up, or are there processes which clear and/or time-out the info? On Mon, Mar 20, 2017 at 3:18 PM, Voytek Jarnot wrote: > No dropped

Re: Very odd & inconsistent results from SASI query

2017-03-20 Thread Voytek Jarnot
No dropped messages in tpstats on any of the nodes. On Mon, Mar 20, 2017 at 3:11 PM, Voytek Jarnot wrote: > Appreciate the reply, Kurt. > > I sanitized it out of the traces, but all trace outputs listed the same > node for all three queries (1 working, 2 not working).

Re: Very odd & inconsistent results from SASI query

2017-03-20 Thread Voytek Jarnot
Appreciate the reply, Kurt. I sanitized it out of the traces, but all trace outputs listed the same node for all three queries (1 working, 2 not working). Read repair chance set to 0.0 as recommended when using TWCS. I'll check tpstats - in this environment, load is not an issue, but network

Re: Very odd & inconsistent results from SASI query

2017-03-20 Thread kurt greaves
As secondary indexes are stored individually on each node what you're suggesting sounds exactly like a consistency issue. the fact that you read 0 cells on one query implies the node that got the query did not have any data for the row. The reason you would sometimes see different behaviours is

Re: Very odd & inconsistent results from SASI query

2017-03-17 Thread Voytek Jarnot
A wrinkle further confounds the issue: running a repair on the node which was servicing the queries has cleared things up and all the queries now work. That doesn't make a whole lot of sense to me - my assumption was that a repair shouldn't have fixed it. On Fri, Mar 17, 2017 at 12:03 PM, Voytek

Very odd & inconsistent results from SASI query

2017-03-17 Thread Voytek Jarnot
Cassandra 3.9, 4 nodes, rf=3 Hi folks, we're see 0 results returned from queries that (a) should return results, and (b) will return results with minor tweaks. I've attached the sanitized trace outputs for the following 3 queries (pk1 and pk2 are partition keys, ck1 is clustering key, val1 is