Re: OOM after a while during compacting
Yeah, they are pretty much unique but it's only a few requests per day so hitting all the nodes would be fine for now. 2018-04-05 15:43 GMT+02:00 Evelyn Smith <u5015...@gmail.com>: > Not sure if it differs for SASI Secondary Indexes but my understanding is > it’s a bad idea to use high cardinality columns for Secondary Indexes. > Not sure what your data model looks like but I’d assume UUID would have > very high cardinality. > > If that’s the case it pretty much guarantees any query on the secondary > index will hit all the nodes, which is what you want to avoid. > > Also Secondary Indexes are generally bad for Cassandra, if you don’t need > them or there's a way around using them I’d go with that. > > Regards, > Eevee. > > > On 5 Apr 2018, at 11:27 pm, Zsolt Pálmai <zpal...@gmail.com> wrote: > > Tried both (although with the biggest table) and the result is the same. > > I stumbled upon this jira issue: https://issues.apache.o > rg/jira/browse/CASSANDRA-12662 > Since the sasi indexes I use are only helping in debugging (for now) I > dropped them and it seems the tables get compacted now (at least it made it > further then before and the jvm metrics look healthy). > > Still this is not ideal as it would be nice to have those secondary > indexes :/ . > > The columns I indexed are basically uuids (so I can match the rows from > other systems but this is usually triggered manually so performance loss is > acceptable). > Is there a recommended index to use here? Or setting > the max_compaction_flush_memory_in_mb value? I saw that it can cause > different kind of problems... Or the default secondary index? > > Thanks > > > > 2018-04-05 15:14 GMT+02:00 Evelyn Smith <u5015...@gmail.com>: > >> Probably a dumb question but it’s good to clarify. >> >> Are you compacting the whole keyspace or are you compacting tables one at >> a time? >> >> >> On 5 Apr 2018, at 9:47 pm, Zsolt Pálmai <zpal...@gmail.com> wrote: >> >> Hi! >> >> I have a setup with 4 AWS nodes (m4xlarge - 4 cpu, 16gb ram, 1TB ssd >> each) and when running the nodetool compact command on any of the servers I >> get out of memory exception after a while. >> >> - Before calling the compact first I did a repair and before that there >> was a bigger update on a lot of entries so I guess a lot of sstables were >> created. The reapir created around ~250 pending compaction tasks, 2 of the >> nodes I managed to finish with upgrading to a 2xlarge machine and twice the >> heap (but running the compact on them manually also killed one :/ so this >> isn't an ideal solution) >> >> Some more info: >> - Version is the newest 3.11.2 with java8u116 >> - Using LeveledCompactionStrategy (we have mostly reads) >> - Heap size is set to 8GB >> - Using G1GC >> - I tried moving the memtable out of the heap. It helped but I still got >> an OOM last night >> - Concurrent compactors is set to 1 but it still happens and also tried >> setting throughput between 16 and 128, no changes. >> - Storage load is 127Gb/140Gb/151Gb/155Gb >> - 1 keyspace, 16 tables but there are a few SASI indexes on big tables. >> - The biggest partition I found was 90Mb but that table has only 2 >> sstables attached and compacts in seconds. The rest is mostly 1 line >> partition with a few 10KB of data. >> - Worst SSTable case: SSTables in each level: [1, 20/10, 106/100, 15, 0, >> 0, 0, 0, 0] >> >> In the metrics it looks something like this before dying: >> https://ibb.co/kLhdXH >> >> What the heap dump looks like of the top objects: https://ibb.co/ctkyXH >> >> The load is usually pretty low, the nodes are almost idling (avg 500 >> reads/sec, 30-40 writes/sec with occasional few second spikes with >100 >> writes) and the pending tasks is also around 0 usually. >> >> Any ideas? I'm starting to run out of ideas. Maybe the secondary indexes >> cause problems? I could finish some bigger compactions where there was no >> index attached but I'm not sure 100% if this is the cause. >> >> Thanks, >> Zsolt >> >> >> >> >> > >
Re: OOM after a while during compacting
Tried both (although with the biggest table) and the result is the same. I stumbled upon this jira issue: https://issues.apache. org/jira/browse/CASSANDRA-12662 Since the sasi indexes I use are only helping in debugging (for now) I dropped them and it seems the tables get compacted now (at least it made it further then before and the jvm metrics look healthy). Still this is not ideal as it would be nice to have those secondary indexes :/ . The columns I indexed are basically uuids (so I can match the rows from other systems but this is usually triggered manually so performance loss is acceptable). Is there a recommended index to use here? Or setting the max_compaction_flush_memory_in_mb value? I saw that it can cause different kind of problems... Or the default secondary index? Thanks 2018-04-05 15:14 GMT+02:00 Evelyn Smith <u5015...@gmail.com>: > Probably a dumb question but it’s good to clarify. > > Are you compacting the whole keyspace or are you compacting tables one at > a time? > > > On 5 Apr 2018, at 9:47 pm, Zsolt Pálmai <zpal...@gmail.com> wrote: > > Hi! > > I have a setup with 4 AWS nodes (m4xlarge - 4 cpu, 16gb ram, 1TB ssd each) > and when running the nodetool compact command on any of the servers I get > out of memory exception after a while. > > - Before calling the compact first I did a repair and before that there > was a bigger update on a lot of entries so I guess a lot of sstables were > created. The reapir created around ~250 pending compaction tasks, 2 of the > nodes I managed to finish with upgrading to a 2xlarge machine and twice the > heap (but running the compact on them manually also killed one :/ so this > isn't an ideal solution) > > Some more info: > - Version is the newest 3.11.2 with java8u116 > - Using LeveledCompactionStrategy (we have mostly reads) > - Heap size is set to 8GB > - Using G1GC > - I tried moving the memtable out of the heap. It helped but I still got > an OOM last night > - Concurrent compactors is set to 1 but it still happens and also tried > setting throughput between 16 and 128, no changes. > - Storage load is 127Gb/140Gb/151Gb/155Gb > - 1 keyspace, 16 tables but there are a few SASI indexes on big tables. > - The biggest partition I found was 90Mb but that table has only 2 > sstables attached and compacts in seconds. The rest is mostly 1 line > partition with a few 10KB of data. > - Worst SSTable case: SSTables in each level: [1, 20/10, 106/100, 15, 0, > 0, 0, 0, 0] > > In the metrics it looks something like this before dying: > https://ibb.co/kLhdXH > > What the heap dump looks like of the top objects: https://ibb.co/ctkyXH > > The load is usually pretty low, the nodes are almost idling (avg 500 > reads/sec, 30-40 writes/sec with occasional few second spikes with >100 > writes) and the pending tasks is also around 0 usually. > > Any ideas? I'm starting to run out of ideas. Maybe the secondary indexes > cause problems? I could finish some bigger compactions where there was no > index attached but I'm not sure 100% if this is the cause. > > Thanks, > Zsolt > > > > >
OOM after a while during compacting
Hi! I have a setup with 4 AWS nodes (m4xlarge - 4 cpu, 16gb ram, 1TB ssd each) and when running the nodetool compact command on any of the servers I get out of memory exception after a while. - Before calling the compact first I did a repair and before that there was a bigger update on a lot of entries so I guess a lot of sstables were created. The reapir created around ~250 pending compaction tasks, 2 of the nodes I managed to finish with upgrading to a 2xlarge machine and twice the heap (but running the compact on them manually also killed one :/ so this isn't an ideal solution) Some more info: - Version is the newest 3.11.2 with java8u116 - Using LeveledCompactionStrategy (we have mostly reads) - Heap size is set to 8GB - Using G1GC - I tried moving the memtable out of the heap. It helped but I still got an OOM last night - Concurrent compactors is set to 1 but it still happens and also tried setting throughput between 16 and 128, no changes. - Storage load is 127Gb/140Gb/151Gb/155Gb - 1 keyspace, 16 tables but there are a few SASI indexes on big tables. - The biggest partition I found was 90Mb but that table has only 2 sstables attached and compacts in seconds. The rest is mostly 1 line partition with a few 10KB of data. - Worst SSTable case: SSTables in each level: [1, 20/10, 106/100, 15, 0, 0, 0, 0, 0] In the metrics it looks something like this before dying: https://ibb.co/kLhdXH What the heap dump looks like of the top objects: https://ibb.co/ctkyXH The load is usually pretty low, the nodes are almost idling (avg 500 reads/sec, 30-40 writes/sec with occasional few second spikes with >100 writes) and the pending tasks is also around 0 usually. Any ideas? I'm starting to run out of ideas. Maybe the secondary indexes cause problems? I could finish some bigger compactions where there was no index attached but I'm not sure 100% if this is the cause. Thanks, Zsolt
