Re: Bursts of Thrift threads make cluster unresponsive

cribed happened, or is the > order with which you presented them the order you notice things going > wrong? > > On Thu, Jun 27, 2019 at 1:29 PM Dmitry Simonov > wrote: > >> Thanks for your reply! >> >> > Have you tried increasing concurrent reads until you see mor

Re: Bursts of Thrift threads make cluster unresponsive

=transport.scope=Native-Transport-Requests.name=TotalBlockedTasks.Count > > > > On Thu, Jun 27, 2019 at 1:52 AM Dmitry Simonov > wrote: > >> Hello! >> >> We've met several times the following problem. >> >> Cassandra cluster (5 nodes) becomes unresponsive

Bursts of Thrift threads make cluster unresponsive

. I cannot tie this problem with increased load from clients ("read rate" does't increase during the problem). Also looks like there is no problem with disks (I/O latencies are OK). Could anybody please give some advice in further troubleshooting? -- Best Regards, Dmitry Simonov

cqlsh COPY ... TO ... doesn't work if one node down

:21:40,064 Error attempting to reconnect to 10.0.0.47, scheduling retry in 512.0 seconds: [Errno None] Tried connecting to [('10.0.0.47', 9042)]. Last error: timed out :1:(4, 'Interrupted system call') IOError: IOError: IOError: IOError: IOError: -- Best Regards, Dmitry Simonov

Re: Network problems during repair make it hang on "Wait for validation to complete"

In the previous message, I have pasted source code from cassandra 2.2.8 by mistake. Re-checked for 2.2.11 source. These lines are the same. 2018-06-21 2:49 GMT+05:00 Dmitry Simonov : > Hello! > > Using Cassandra 2.2.11, I observe behaviour, that is very similar to > https://issue

Network problems during repair make it hang on "Wait for validation to complete"

issues) during repair may also lead to the same problem. Is it a known issue? -- Best Regards, Dmitry Simonov

Re: Many SSTables only on one node

; On 5 Apr 2018, at 8:35 pm, Dmitry Simonov <dimmobor...@gmail.com> wrote: > > Thank you! > I'll check this out. > > 2018-04-05 15:00 GMT+05:00 Alexander Dejanovski <a...@thelastpickle.com>: > >> 40 pending compactions is pretty high and you should have w

Re: Many SSTables only on one node

e number of pending compactions and see if it prevents the issue > from happening again. > > Cheers, > > > On Thu, Apr 5, 2018 at 11:33 AM Dmitry Simonov <dimmobor...@gmail.com> > wrote: > >> Hi, Alexander! >> >> SizeTieredCompactionStrategy i

Re: Many SSTables only on one node

.@thelastpickle.com>: > Hi Dmitry, > > could you tell us which compaction strategy that table is currently using ? > Also, what is the compaction max throughput and is auto-compaction > correctly enabled on that node ? > > Did you recently run repair ? > > Thanks, &

Many SSTables only on one node

, on other nodes: 3 sstable. Data size in this keyspace was not very big ~100-200Mb per node. There is no such problem with other CFs of that keyspace. nodetool compact solved the issue as a quick-fix. But I'm wondering, what was the cause? How prevent it from repeating? -- Best Regards, Dmitry

Re: "READ messages were dropped ... for internal timeout" after big amount of writes

nconsistency (huge I/O wait at some > point, blocked proc etc) > > > > > On 16 March 2018 at 07:33, Dmitry Simonov <dimmobor...@gmail.com> wrote: > >> Hello! >> >> We are experiencing problems with Cassandra 2.2.8. >> There is a cluster with 3 node

"READ messages were dropped ... for internal timeout" after big amount of writes

;batch_mutate" queries at all, only reads! There are no GC warnings about long pauses Could you please help troubleshooting the issue? -- Best Regards, Dmitry Simonov