Re: Drop tables takes too long

Hi We were trying to overcome OOM crashes. Fair enough :-). We changed settings to default on one node. GC times became about two times > smaller on that node. > That's encouraging! Looks like even if the number of tables is really high, there is still space for optimization. Have you made

Re: Drop tables takes too long

Hi, > Why did you move from defaults that much? We were trying to overcome OOM crashes. > Would you consider giving defaults a try on a canary node and monitor / compare GC times to other nodes? We changed settings to default on one node. GC times became about two times smaller on that node. >

Re: Drop tables takes too long

Hi, A few comments: Long GC Pauses take about one minute > This is huge. About JVM config, I haven't played much with G1GC, but the following seems to be a bad idea according to comments: ## Main G1GC tunable: lowering the pause target will lower throughput and vise versa. ## 200ms is the JVM

Re: Drop tables takes too long

Thanks Alain, > Or is it on happening during drop table actions? Some other schema changes (e.g. adding columns to tables) also takes too much time. Link to complete set of GC options: https://pastebin.com/4qyENeyu > Have you had a look at logs, mainly errors and warnings? In logs I found

Re: Drop tables takes too long

Hi > Long GC Pauses take about one minute. But why it takes so much time and > how that can be fixed? This is very long. Looks like you are having a major issue, and it is not just about dropping tables... Or is it on happening during drop table actions? Knowing the complete set of GC options

Re: Drop tables takes too long

Thanks Zhao Yang, > Could you try some jvm tool to find out which thread are allocating memory or gc? maybe the migration stage thread.. I use Cassandra Cluster Manager to locally reproduce the issue. I tried to use VisualVM to find out which threads are allocating memory, but VisualVM does not

Re: Drop tables takes too long

Hi Bohdan, Carlos, Could you try some jvm tool to find out which thread are allocating memory or gc? maybe the migration stage thread.. BTW, is your cluster under high load while dropping table? As far as I remember, in older c* version, it applies the schema mutation in memory, ie. DROP, then

Re: Drop tables takes too long

Hi, Problem is still not solved. Does anybody have any idea what to do with it? Thanks 2017-04-20 15:05 GMT+03:00 Bohdan Tantsiura : > Thanks Carlos, > > In each keyspace we also have 11 MVs. > > It is impossible to reduce number of tables now. Long GC Pauses take about >

Re: Drop tables takes too long

Thanks Carlos, In each keyspace we also have 11 MVs. It is impossible to reduce number of tables now. Long GC Pauses take about one minute. But why it takes so much time and how that can be fixed? Each node in cluster has 128GB RAM, so resources are not constrained now Thanks 2017-04-20 13:18

Re: Drop tables takes too long

You have 4800 Tables in total? That is a lot of tables, plus MVs? or MVs are already considered in the 60*80 account? I would recommend to reduce the table number. Other thing is that you need to check your log file for GC Pauses, and how long those pauses take. You also might need to increase

Drop tables takes too long

Hi, We are using cassandra 3.10 in a 10 nodes cluster with replication = 3. MAX_HEAP_SIZE=64GB on all nodes, G1 GC is used. We have about 60 keyspaces with about 80 tables in each keyspace. We had to delete three tables and two materialized views from each keyspace. It began to take more and more