[jira] [Commented] (CASSANDRA-9291) Too many tombstones in schema_columns from creating too many CFs
[ https://issues.apache.org/jira/browse/CASSANDRA-9291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15062258#comment-15062258 ] Thierry Schellenbach commented on CASSANDRA-9291: - Thierry from getstream.io here. We're getting this error message: WARN [ReadStage:447081] 2015-12-17 16:18:16,908 SliceQueryFilter.java (line 231) Read 3981 live and 12251 tombstone cells in system.schema_columns (see tombstone_warn_threshold). 2147483357 columns was requested, slices=[-] CPU on 3 of our Cassandra machines is at 100%. Our infrastructure is down as a result. nodetool resetlocalschema doesn't fix the issue. > Too many tombstones in schema_columns from creating too many CFs > > > Key: CASSANDRA-9291 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9291 > Project: Cassandra > Issue Type: Bug > Environment: Production Cluster with 2 DCs of 3 nodes each and 1 DC > of 7 nodes, running on dedicated Xeon hexacore, 96GB ram, RAID for Data and > SSF for commitlog, running Debian 7 (with Java 1.7.0_76-b13 64-Bit, 8GB and > 16GB of heap tested). > Dev Cluster with 1 DC with 3 nodes and 1 DC with 1 node, running on > virtualized env., Ubuntu 12.04.5 (with Java 1.7.0_72-b14 64-Bit 1GB, 4GB > heap) >Reporter: Luis Correia >Priority: Blocker > Attachments: after_schema.txt, before_schema.txt, schemas500.cql > > > When creating lots of columnfamilies (about 200) the system.schema_columns > gets filled with tombstones and therefore prevents clients using the binary > protocol of connecting. > Clients already connected continue normal operation (reading and inserting). > Log messages are: > For the first tries (sorry for the lack of precision): > bq. ERROR [main] 2015-04-22 00:01:38,527 SliceQueryFilter.java (line 200) > Scanned over 10 tombstones in system.schema_columns; query aborted (see > tombstone_failure_threshold) > For each client that tries to connect but fails with timeout: > bq. WARN [ReadStage:35] 2015-04-27 15:40:10,600 SliceQueryFilter.java (line > 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see > tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] > bq. WARN [ReadStage:40] 2015-04-27 15:40:10,609 SliceQueryFilter.java (line > 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see > tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] > bq. WARN [ReadStage:61] 2015-04-27 15:40:10,670 SliceQueryFilter.java (line > 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see > tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] > bq. WARN [ReadStage:51] 2015-04-27 15:40:10,670 SliceQueryFilter.java (line > 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see > tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] > bq. WARN [ReadStage:55] 2015-04-27 15:40:10,675 SliceQueryFilter.java (line > 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see > tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] > bq. WARN [ReadStage:35] 2015-04-27 15:40:10,707 SliceQueryFilter.java (line > 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see > tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] > bq. WARN [ReadStage:40] 2015-04-27 15:40:10,708 SliceQueryFilter.java (line > 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see > tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] > bq. WARN [ReadStage:43] 2015-04-27 15:40:10,715 SliceQueryFilter.java (line > 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see > tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] > bq. WARN [ReadStage:51] 2015-04-27 15:40:10,736 SliceQueryFilter.java (line > 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see > tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] > bq. WARN [ReadStage:61] 2015-04-27 15:40:10,736 SliceQueryFilter.java (line > 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see > tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] > bq. WARN [ReadStage:35] 2015-04-27 15:40:10,750 SliceQueryFilter.java (line > 231) Read 864 live and 2664 tombstoned cells in system.schema_columns (see > tombstone_warn_threshold). 2147281748 columns was requested, slices=[-] > bq. WARN [ReadStage:40] 2015-04-27 15:40:10,751 SliceQueryFilter.java (line > 231) Read 864 live and 2664 tombstoned cells in system.schema_columns (see > tombstone_warn_threshold). 2147281748 columns was requested, slices=[-] > bq. WARN [ReadStage:55] 2015-04-27 15:40:10,759
[jira] [Commented] (CASSANDRA-9291) Too many tombstones in schema_columns from creating too many CFs
[ https://issues.apache.org/jira/browse/CASSANDRA-9291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660844#comment-14660844 ] Ben Bromhead commented on CASSANDRA-9291: - It worked for us but took a little bit of doing it a few times on the impacted nodes until we got them to stream from the right nodes. You need to have more good nodes than broken ones in this case (or at least one). resetLocalSchema() in the MigrationManager picks the first node from Gossiper.instance.getLiveMembers(), so if you have enough (or at least one) good node in your cluster you can eventually get to a state of normality. I think you can get the liveMembers list via JMX (correct me if I'm wrong here) to help figure out which node it will use, otherwise you can roll through the whole cluster excluding the known good ones. Too many tombstones in schema_columns from creating too many CFs Key: CASSANDRA-9291 URL: https://issues.apache.org/jira/browse/CASSANDRA-9291 Project: Cassandra Issue Type: Bug Components: Core Environment: Production Cluster with 2 DCs of 3 nodes each and 1 DC of 7 nodes, running on dedicated Xeon hexacore, 96GB ram, RAID for Data and SSF for commitlog, running Debian 7 (with Java 1.7.0_76-b13 64-Bit, 8GB and 16GB of heap tested). Dev Cluster with 1 DC with 3 nodes and 1 DC with 1 node, running on virtualized env., Ubuntu 12.04.5 (with Java 1.7.0_72-b14 64-Bit 1GB, 4GB heap) Reporter: Luis Correia Priority: Blocker Attachments: after_schema.txt, before_schema.txt, schemas500.cql When creating lots of columnfamilies (about 200) the system.schema_columns gets filled with tombstones and therefore prevents clients using the binary protocol of connecting. Clients already connected continue normal operation (reading and inserting). Log messages are: For the first tries (sorry for the lack of precision): bq. ERROR [main] 2015-04-22 00:01:38,527 SliceQueryFilter.java (line 200) Scanned over 10 tombstones in system.schema_columns; query aborted (see tombstone_failure_threshold) For each client that tries to connect but fails with timeout: bq. WARN [ReadStage:35] 2015-04-27 15:40:10,600 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:40] 2015-04-27 15:40:10,609 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:61] 2015-04-27 15:40:10,670 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:51] 2015-04-27 15:40:10,670 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:55] 2015-04-27 15:40:10,675 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:35] 2015-04-27 15:40:10,707 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] bq. WARN [ReadStage:40] 2015-04-27 15:40:10,708 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] bq. WARN [ReadStage:43] 2015-04-27 15:40:10,715 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:51] 2015-04-27 15:40:10,736 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] bq. WARN [ReadStage:61] 2015-04-27 15:40:10,736 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] bq. WARN [ReadStage:35] 2015-04-27 15:40:10,750 SliceQueryFilter.java (line 231) Read 864 live and 2664 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147281748 columns was requested, slices=[-] bq. WARN [ReadStage:40] 2015-04-27 15:40:10,751 SliceQueryFilter.java (line 231) Read 864 live and 2664
[jira] [Commented] (CASSANDRA-9291) Too many tombstones in schema_columns from creating too many CFs
[ https://issues.apache.org/jira/browse/CASSANDRA-9291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660654#comment-14660654 ] Sebastian Estevez commented on CASSANDRA-9291: -- Did this work for you? From experience I think `resetlocalschema` picks up new schema info from other nodes that are also full of tombstones after deleting the local info. So you're still stuck with tombstones. It's not a workaround unless I'm missing something. Too many tombstones in schema_columns from creating too many CFs Key: CASSANDRA-9291 URL: https://issues.apache.org/jira/browse/CASSANDRA-9291 Project: Cassandra Issue Type: Bug Components: Core Environment: Production Cluster with 2 DCs of 3 nodes each and 1 DC of 7 nodes, running on dedicated Xeon hexacore, 96GB ram, RAID for Data and SSF for commitlog, running Debian 7 (with Java 1.7.0_76-b13 64-Bit, 8GB and 16GB of heap tested). Dev Cluster with 1 DC with 3 nodes and 1 DC with 1 node, running on virtualized env., Ubuntu 12.04.5 (with Java 1.7.0_72-b14 64-Bit 1GB, 4GB heap) Reporter: Luis Correia Priority: Blocker Attachments: after_schema.txt, before_schema.txt, schemas500.cql When creating lots of columnfamilies (about 200) the system.schema_columns gets filled with tombstones and therefore prevents clients using the binary protocol of connecting. Clients already connected continue normal operation (reading and inserting). Log messages are: For the first tries (sorry for the lack of precision): bq. ERROR [main] 2015-04-22 00:01:38,527 SliceQueryFilter.java (line 200) Scanned over 10 tombstones in system.schema_columns; query aborted (see tombstone_failure_threshold) For each client that tries to connect but fails with timeout: bq. WARN [ReadStage:35] 2015-04-27 15:40:10,600 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:40] 2015-04-27 15:40:10,609 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:61] 2015-04-27 15:40:10,670 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:51] 2015-04-27 15:40:10,670 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:55] 2015-04-27 15:40:10,675 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:35] 2015-04-27 15:40:10,707 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] bq. WARN [ReadStage:40] 2015-04-27 15:40:10,708 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] bq. WARN [ReadStage:43] 2015-04-27 15:40:10,715 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:51] 2015-04-27 15:40:10,736 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] bq. WARN [ReadStage:61] 2015-04-27 15:40:10,736 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] bq. WARN [ReadStage:35] 2015-04-27 15:40:10,750 SliceQueryFilter.java (line 231) Read 864 live and 2664 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147281748 columns was requested, slices=[-] bq. WARN [ReadStage:40] 2015-04-27 15:40:10,751 SliceQueryFilter.java (line 231) Read 864 live and 2664 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147281748 columns was requested, slices=[-] bq. WARN [ReadStage:55] 2015-04-27 15:40:10,759 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] bq. WARN [ReadStage:51]
[jira] [Commented] (CASSANDRA-9291) Too many tombstones in schema_columns from creating too many CFs
[ https://issues.apache.org/jira/browse/CASSANDRA-9291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660628#comment-14660628 ] Ben Bromhead commented on CASSANDRA-9291: - FYI for those stumbling across this ticket experiencing similar issues `nodetool resetlocalschema` will drop schema info from the node and resync with other nodes, clearing out existing data (and tombstones). So this would be the work around until 3.0 Too many tombstones in schema_columns from creating too many CFs Key: CASSANDRA-9291 URL: https://issues.apache.org/jira/browse/CASSANDRA-9291 Project: Cassandra Issue Type: Bug Components: Core Environment: Production Cluster with 2 DCs of 3 nodes each and 1 DC of 7 nodes, running on dedicated Xeon hexacore, 96GB ram, RAID for Data and SSF for commitlog, running Debian 7 (with Java 1.7.0_76-b13 64-Bit, 8GB and 16GB of heap tested). Dev Cluster with 1 DC with 3 nodes and 1 DC with 1 node, running on virtualized env., Ubuntu 12.04.5 (with Java 1.7.0_72-b14 64-Bit 1GB, 4GB heap) Reporter: Luis Correia Priority: Blocker Attachments: after_schema.txt, before_schema.txt, schemas500.cql When creating lots of columnfamilies (about 200) the system.schema_columns gets filled with tombstones and therefore prevents clients using the binary protocol of connecting. Clients already connected continue normal operation (reading and inserting). Log messages are: For the first tries (sorry for the lack of precision): bq. ERROR [main] 2015-04-22 00:01:38,527 SliceQueryFilter.java (line 200) Scanned over 10 tombstones in system.schema_columns; query aborted (see tombstone_failure_threshold) For each client that tries to connect but fails with timeout: bq. WARN [ReadStage:35] 2015-04-27 15:40:10,600 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:40] 2015-04-27 15:40:10,609 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:61] 2015-04-27 15:40:10,670 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:51] 2015-04-27 15:40:10,670 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:55] 2015-04-27 15:40:10,675 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:35] 2015-04-27 15:40:10,707 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] bq. WARN [ReadStage:40] 2015-04-27 15:40:10,708 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] bq. WARN [ReadStage:43] 2015-04-27 15:40:10,715 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:51] 2015-04-27 15:40:10,736 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] bq. WARN [ReadStage:61] 2015-04-27 15:40:10,736 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] bq. WARN [ReadStage:35] 2015-04-27 15:40:10,750 SliceQueryFilter.java (line 231) Read 864 live and 2664 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147281748 columns was requested, slices=[-] bq. WARN [ReadStage:40] 2015-04-27 15:40:10,751 SliceQueryFilter.java (line 231) Read 864 live and 2664 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147281748 columns was requested, slices=[-] bq. WARN [ReadStage:55] 2015-04-27 15:40:10,759 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] bq. WARN [ReadStage:51] 2015-04-27 15:40:10,821
[jira] [Commented] (CASSANDRA-9291) Too many tombstones in schema_columns from creating too many CFs
[ https://issues.apache.org/jira/browse/CASSANDRA-9291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14581829#comment-14581829 ] Aleksey Yeschenko commented on CASSANDRA-9291: -- It's most definitely not a minor issue, no. It's just very non-trivial to fix in the 2.x line (I don't know a way). Too many tombstones in schema_columns from creating too many CFs Key: CASSANDRA-9291 URL: https://issues.apache.org/jira/browse/CASSANDRA-9291 Project: Cassandra Issue Type: Bug Components: Core Environment: Production Cluster with 2 DCs of 3 nodes each and 1 DC of 7 nodes, running on dedicated Xeon hexacore, 96GB ram, RAID for Data and SSF for commitlog, running Debian 7 (with Java 1.7.0_76-b13 64-Bit, 8GB and 16GB of heap tested). Dev Cluster with 1 DC with 3 nodes and 1 DC with 1 node, running on virtualized env., Ubuntu 12.04.5 (with Java 1.7.0_72-b14 64-Bit 1GB, 4GB heap) Reporter: Luis Correia Priority: Blocker Attachments: after_schema.txt, before_schema.txt, schemas500.cql When creating lots of columnfamilies (about 200) the system.schema_columns gets filled with tombstones and therefore prevents clients using the binary protocol of connecting. Clients already connected continue normal operation (reading and inserting). Log messages are: For the first tries (sorry for the lack of precision): bq. ERROR [main] 2015-04-22 00:01:38,527 SliceQueryFilter.java (line 200) Scanned over 10 tombstones in system.schema_columns; query aborted (see tombstone_failure_threshold) For each client that tries to connect but fails with timeout: bq. WARN [ReadStage:35] 2015-04-27 15:40:10,600 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:40] 2015-04-27 15:40:10,609 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:61] 2015-04-27 15:40:10,670 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:51] 2015-04-27 15:40:10,670 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:55] 2015-04-27 15:40:10,675 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:35] 2015-04-27 15:40:10,707 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] bq. WARN [ReadStage:40] 2015-04-27 15:40:10,708 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] bq. WARN [ReadStage:43] 2015-04-27 15:40:10,715 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:51] 2015-04-27 15:40:10,736 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] bq. WARN [ReadStage:61] 2015-04-27 15:40:10,736 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] bq. WARN [ReadStage:35] 2015-04-27 15:40:10,750 SliceQueryFilter.java (line 231) Read 864 live and 2664 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147281748 columns was requested, slices=[-] bq. WARN [ReadStage:40] 2015-04-27 15:40:10,751 SliceQueryFilter.java (line 231) Read 864 live and 2664 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147281748 columns was requested, slices=[-] bq. WARN [ReadStage:55] 2015-04-27 15:40:10,759 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] bq. WARN [ReadStage:51] 2015-04-27 15:40:10,821 SliceQueryFilter.java (line 231) Read 864 live and 2664 tombstoned cells in system.schema_columns (see
[jira] [Commented] (CASSANDRA-9291) Too many tombstones in schema_columns from creating too many CFs
[ https://issues.apache.org/jira/browse/CASSANDRA-9291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526593#comment-14526593 ] Aleksey Yeschenko commented on CASSANDRA-9291: -- Thanks for the report. It's a known issue - a duplicate of CASSANDRA-8189 - and will be resolved soon in 3.0. Too many tombstones in schema_columns from creating too many CFs Key: CASSANDRA-9291 URL: https://issues.apache.org/jira/browse/CASSANDRA-9291 Project: Cassandra Issue Type: Bug Components: Core Environment: Production Cluster with 2 DCs of 3 nodes each and 1 DC of 7 nodes, running on dedicated Xeon hexacore, 96GB ram, RAID for Data and SSF for commitlog, running Debian 7 (with Java 1.7.0_76-b13 64-Bit, 8GB and 16GB of heap tested). Dev Cluster with 1 DC with 3 nodes and 1 DC with 1 node, running on virtualized env., Ubuntu 12.04.5 (with Java 1.7.0_72-b14 64-Bit 1GB, 4GB heap) Reporter: Luis Correia Priority: Blocker Attachments: after_schema.txt, before_schema.txt, schemas500.cql When creating lots of columnfamilies (about 200) the system.schema_columns gets filled with tombstones and therefore prevents clients using the binary protocol of connecting. Clients already connected continue normal operation (reading and inserting). Log messages are: For the first tries (sorry for the lack of precision): bq. ERROR [main] 2015-04-22 00:01:38,527 SliceQueryFilter.java (line 200) Scanned over 10 tombstones in system.schema_columns; query aborted (see tombstone_failure_threshold) For each client that tries to connect but fails with timeout: bq. WARN [ReadStage:35] 2015-04-27 15:40:10,600 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:40] 2015-04-27 15:40:10,609 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:61] 2015-04-27 15:40:10,670 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:51] 2015-04-27 15:40:10,670 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:55] 2015-04-27 15:40:10,675 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:35] 2015-04-27 15:40:10,707 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] bq. WARN [ReadStage:40] 2015-04-27 15:40:10,708 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] bq. WARN [ReadStage:43] 2015-04-27 15:40:10,715 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:51] 2015-04-27 15:40:10,736 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] bq. WARN [ReadStage:61] 2015-04-27 15:40:10,736 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] bq. WARN [ReadStage:35] 2015-04-27 15:40:10,750 SliceQueryFilter.java (line 231) Read 864 live and 2664 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147281748 columns was requested, slices=[-] bq. WARN [ReadStage:40] 2015-04-27 15:40:10,751 SliceQueryFilter.java (line 231) Read 864 live and 2664 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147281748 columns was requested, slices=[-] bq. WARN [ReadStage:55] 2015-04-27 15:40:10,759 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] bq. WARN [ReadStage:51] 2015-04-27 15:40:10,821 SliceQueryFilter.java (line 231) Read 864 live and 2664 tombstoned cells in system.schema_columns (see tombstone_warn_threshold).
[jira] [Commented] (CASSANDRA-9291) Too many tombstones in schema_columns from creating too many CFs
[ https://issues.apache.org/jira/browse/CASSANDRA-9291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526656#comment-14526656 ] Luis Correia commented on CASSANDRA-9291: - I cannot see this as a Minor issue as it can bring down an healthy Cluster just be creating few more new CFs. Please mind that in a new Cluster the problem can be mitigating by remodeling your data. In an existing Cluster (mind Production!) you can _prevent clients from connecting just by creating new CFs_. And the solution will be to wait for the gc_grace_period hoping the right amount of tombstones will be removed by compaction (that's at least 7 days). I've tried to import the stables to a new cluster and shift the time (skip to the day compaction should clear the tombstones), various schema import/export gymnastics, etc. Nothing worked. I had to delete system.schema_columns and re-create only the right amount of CF's (fixing the token range for each node) in order to get my clients connecting again. Too many tombstones in schema_columns from creating too many CFs Key: CASSANDRA-9291 URL: https://issues.apache.org/jira/browse/CASSANDRA-9291 Project: Cassandra Issue Type: Bug Components: Core Environment: Production Cluster with 2 DCs of 3 nodes each and 1 DC of 7 nodes, running on dedicated Xeon hexacore, 96GB ram, RAID for Data and SSF for commitlog, running Debian 7 (with Java 1.7.0_76-b13 64-Bit, 8GB and 16GB of heap tested). Dev Cluster with 1 DC with 3 nodes and 1 DC with 1 node, running on virtualized env., Ubuntu 12.04.5 (with Java 1.7.0_72-b14 64-Bit 1GB, 4GB heap) Reporter: Luis Correia Priority: Blocker Attachments: after_schema.txt, before_schema.txt, schemas500.cql When creating lots of columnfamilies (about 200) the system.schema_columns gets filled with tombstones and therefore prevents clients using the binary protocol of connecting. Clients already connected continue normal operation (reading and inserting). Log messages are: For the first tries (sorry for the lack of precision): bq. ERROR [main] 2015-04-22 00:01:38,527 SliceQueryFilter.java (line 200) Scanned over 10 tombstones in system.schema_columns; query aborted (see tombstone_failure_threshold) For each client that tries to connect but fails with timeout: bq. WARN [ReadStage:35] 2015-04-27 15:40:10,600 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:40] 2015-04-27 15:40:10,609 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:61] 2015-04-27 15:40:10,670 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:51] 2015-04-27 15:40:10,670 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:55] 2015-04-27 15:40:10,675 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:35] 2015-04-27 15:40:10,707 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] bq. WARN [ReadStage:40] 2015-04-27 15:40:10,708 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] bq. WARN [ReadStage:43] 2015-04-27 15:40:10,715 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-] bq. WARN [ReadStage:51] 2015-04-27 15:40:10,736 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] bq. WARN [ReadStage:61] 2015-04-27 15:40:10,736 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-] bq. WARN [ReadStage:35] 2015-04-27 15:40:10,750 SliceQueryFilter.java (line 231) Read 864 live and 2664 tombstoned cells in system.schema_columns (see
