Re: COUNTER timeout

2021-09-15 Thread Joe Obernberger
Thank you! Clocks were out of sync; chronyd wasn't chrony'ding. Going so much faster now!  Cheers. -Joe On 9/15/2021 4:07 PM, Bowen Song wrote: Well, the log says cross node timeout, latency a bit over 44 seconds. Here's a few most likely causes: 1. The clocks are not in sync - please

Re: COUNTER timeout

2021-09-15 Thread Bowen Song
Well, the log says cross node timeout, latency a bit over 44 seconds. Here's a few most likely causes: 1. The clocks are not in sync - please check the time on each server, and ensure NTP client is running on all Cassandra servers 2. Long stop the world GC pauses - please check the GC logs

Re: COUNTER timeout

2021-09-15 Thread Joe Obernberger
Thank you Erick - looking through all the logs on the nodes I found this: INFO  [CompactionExecutor:17551] 2021-09-15 15:13:20,524 CompactionTask.java:245 - Compacted (fb0cdca0-1658-11ec-9098-dd70c3a3487a) 4 sstables to

Re: TWCS on Non TTL Data

2021-09-15 Thread Jim Shaw
You may try roll up the data, i.e. a table only 1 month data, old data roll up to a table keep a year data. Thanks, Jim On Wed, Sep 15, 2021 at 1:26 AM Isaeed Mohanna wrote: > My cluster column is the time series timestamp, so basically sourceId, > metric type for partition key and timestamp

Re: COUNTER timeout

2021-09-15 Thread Bowen Song
Check the logs on the Cassandra servers first. Many different things can cause the same result, and you will have to dig in deeper to discover the true cause. On 14/09/2021 23:55, Joe Obernberger wrote: I'm getting a lot of the following errors during ingest of data: