Thanks ZAIDI,
Using C++ driver doesn't have tracing with driver so executing those from
cqlsh. when i am tracing i am getting below error, i increased
--request-timeout to 3600 in cqlsh.
> ReadTimeout: code=1200 [Coordinator node timed out waiting for replica
> nodes' responses] message="Operation timed out - received only 0
> responses." info={'received_responses': 0, 'required_responses': 1,
> 'consistency': 'ONE'}
> Statement trace did not complete within 10 seconds
The below are cfstats and cfhistograms, i can see read latency, cell count
and Maximum live cells per slice (last five minutes) are high. is there any
way to get around this with out changing data model.
Percentile SSTables Write Latency Read Latency Partition
> Size Cell Count
> (micros) (micros)
> (bytes)
> 50% 1.00 20.00 NaN
> 1331 20
> 75% 2.00 29.00 NaN
> 6866 86
> 95% 8.00 60.00 NaN
> 126934 1331
> 98% 10.00 103.00 NaN
> 315852 3973
> 99% 12.00 149.00 NaN
> 545791 8239
> Min 0.00 0.00 0.00
> 104 0
> Max 20.00 12730764.00 9773372036884776000.00
> 74975550 83457
Read Count: 44514407
> Read Latency: 82.92876612928933 ms.
> Write Count: 3007585812
> Write Latency: 0.07094456590853208 ms.
> Pending Flushes: 0
> SSTable count: 9
> Space used (live): 66946214374
> Space used (total): 66946214374
> Space used by snapshots (total): 0
> Off heap memory used (total): 33706492
> SSTable Compression Ratio: 0.5598380206656697
> Number of keys (estimate): 2483819
> Memtable cell count: 15008
> Memtable data size: 330597
> Memtable off heap memory used: 518502
> Memtable switch count: 39915
> Local read count: 44514407
> Local read latency: 82.929 ms
> Local write count: 3007585849
> Local write latency: 0.071 ms
> Pending flushes: 0
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.00000
> Bloom filter space used: 12623632
> Bloom filter off heap memory used: 12623560
> Index summary off heap memory used: 3285614
> Compression metadata off heap memory used: 17278816
> Compacted partition minimum bytes: 104
> Compacted partition maximum bytes: 74975550
> Compacted partition mean bytes: 27111
> Average live cells per slice (last five minutes): 388.7486606077893
> Maximum live cells per slice (last five minutes): 28983.0
> Average tombstones per slice (last five minutes): 0.0
> Maximum tombstones per slice (last five minutes): 0.0
Thanks
Pranay.
On Fri, Jul 7, 2017 at 11:16 AM, Thakrar, Jayesh <
jthak...@conversantmedia.com> wrote:
> Can you provide more details.
>
> E.g. table structure, the app used for the query, the query itself and the
> error message.
>
>
>
> Also get the output of the following commands from your cluster nodes
> (note that one command uses "." and the other "space" between keyspace and
> tablename)
>
>
>
> nodetool -h <hostname> tablestats <keyspace>.<tablename>
>
> nodetool -h <hostname> tablehistograms <keyspace> <tablename>
>
>
>
> Timeouts can happen at the client/application level (which can be tuned)
> and at the coordinator node level (which too can be tuned).
>
> But again those timeouts are a symptom of something.
>
> It can happen at the client side because of connection pool queue too full
> (which is likely due to response time from the cluster/coordinate nodes).
>
> And the issues at the cluster side could be due to several reasons.
>
> E.g. your query has to scan through too many tombstones, causing the delay
> or your query (if using filter).
>
>
>
> *From: *"ZAIDI, ASAD A" <az1...@att.com>
> *Date: *Friday, July 7, 2017 at 9:45 AM
> *To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
> *Subject: *RE: READ Queries timing out.
>
>
>
> >> I analysed the GC logs not having any issues with major GC's
>
> If you don’t have issues on GC , than why do you want to
> [tune] GC parameters ?
>
> Instead focus on why select queries are taking time.. may be take a look
> on their trace?
>
>
>
>
>
> *From:* Pranay akula [mailto:pranay.akula2...@gmail.com]
> *Sent:* Friday, July 07, 2017 9:27 AM
> *To:* user@cassandra.apache.org
> *Subject:* READ Queries timing out.
>
>
>
> Lately i am seeing some select queries timing out, data modelling to blame
> for but not in a situation to redo it.
>
>
>
> Does increasing heap will help ??
>
>
>
> currently using 1GB new_heap, I analysed the GC logs not having any issues
> with major GC's .
>
>
>
> Using G1GC , does increasing new_heap will help ??
>
>
>
> currently using JVM_OPTS="$JVM_OPTS -XX:MaxGCPauseMillis=500", even if i
> increase heap to lets say 2GB is that effective b/c young GC's will kick in
> more frequently to complete in 500ms right ??
>
>
>
>
>
> Thanks
>
> Pranay.
>
