Apologies for the stream-of-consciousness replies, but are the dropped message stats output by tpstats an accumulation since the node came up, or are there processes which clear and/or time-out the info?
On Mon, Mar 20, 2017 at 3:18 PM, Voytek Jarnot <voytek.jar...@gmail.com> wrote: > No dropped messages in tpstats on any of the nodes. > > On Mon, Mar 20, 2017 at 3:11 PM, Voytek Jarnot <voytek.jar...@gmail.com> > wrote: > >> Appreciate the reply, Kurt. >> >> I sanitized it out of the traces, but all trace outputs listed the same >> node for all three queries (1 working, 2 not working). Read repair chance >> set to 0.0 as recommended when using TWCS. >> >> I'll check tpstats - in this environment, load is not an issue, but >> network issues may be. >> >> On Mon, Mar 20, 2017 at 2:42 PM, kurt greaves <k...@instaclustr.com> >> wrote: >> >>> As secondary indexes are stored individually on each node what you're >>> suggesting sounds exactly like a consistency issue. the fact that you read >>> 0 cells on one query implies the node that got the query did not have any >>> data for the row. The reason you would sometimes see different behaviours >>> is likely because of read repairs. The fact that the repair guides the >>> issue pretty much guarantees it's a consistency issue. >>> >>> You should check for dropped mutations in tpstats/logs and if they are >>> occurring try and stop that from happening (probably load related). You >>> could also try performing reads and writes at LOCAL_QUORUM for stronger >>> consistency, however note this has a performance/latency impact. >>> >>> >>> >> >
