I have definitely seen corruption, especially in system tables, when there are multiple instances of Cassandra running/trying to start. We had an internal tool that was supposed to restart processes (like Cassandra) if they were down, but it often re-checked before Cassandra was fully up and started another Cassandra process. Unwinding it could be very ugly.
Sean Durity From: kurt greaves <k...@instaclustr.com> Sent: Monday, August 13, 2018 7:24 AM To: User <user@cassandra.apache.org> Subject: [EXTERNAL] Re: Data Corruption due to multiple Cassandra 2.1 processes? Yeah that's not ideal and could lead to problems. I think corruption is only likely if compactions occur, but seems like data loss is a potential not to mention all sorts of other possible nasties that could occur running two C*'s at once. Seems to me that 11540 should have gone to 2.1 in the first place, but it just got missed. Very simple patch so I think a backport should be accepted. On 7 August 2018 at 15:57, Steinmaurer, Thomas <thomas.steinmau...@dynatrace.com<mailto:thomas.steinmau...@dynatrace.com>> wrote: Hello, with 2.1, in case a second Cassandra process/instance is started on a host (by accident), may this result in some sort of corruption, although Cassandra will exit at some point in time due to not being able to bind TCP ports already in use? What we have seen in this scenario is something like that: ERROR [main] 2018-08-05 21:10:24,046 CassandraDaemon.java:120 - Error starting local jmx server: java.rmi.server.ExportException: Port already in use: 7199; nested exception is: java.net.BindException: Address already in use (Bind failed) … But then continuing with stuff like opening system and even user tables: INFO [main] 2018-08-05 21:10:24,060 CacheService.java:110 - Initializing key cache with capacity of 100 MBs. INFO [main] 2018-08-05 21:10:24,067 CacheService.java:132 - Initializing row cache with capacity of 0 MBs INFO [main] 2018-08-05 21:10:24,073 CacheService.java:149 - Initializing counter cache with capacity of 50 MBs INFO [main] 2018-08-05 21:10:24,074 CacheService.java:160 - Scheduling counter cache save to every 7200 seconds (going to save all keys). INFO [main] 2018-08-05 21:10:24,161 ColumnFamilyStore.java:365 - Initializing system.sstable_activity INFO [SSTableBatchOpen:2] 2018-08-05 21:10:24,692 SSTableReader.java:475 - Opening /var/opt/xxx-managed/cassandra/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-165 (2023 bytes) INFO [SSTableBatchOpen:3] 2018-08-05 21:10:24,692 SSTableReader.java:475 - Opening /var/opt/xxx-managed/cassandra/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-167 (2336 bytes) INFO [SSTableBatchOpen:1] 2018-08-05 21:10:24,692 SSTableReader.java:475 - Opening /var/opt/xxx-managed/cassandra/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-166 (2686 bytes) INFO [main] 2018-08-05 21:10:24,755 ColumnFamilyStore.java:365 - Initializing system.hints INFO [SSTableBatchOpen:1] 2018-08-05 21:10:24,758 SSTableReader.java:475 - Opening /var/opt/xxx-managed/cassandra/system/hints-2666e20573ef38b390fefecf96e8f0c7/system-hints-ka-377 (46210621 bytes) INFO [main] 2018-08-05 21:10:24,766 ColumnFamilyStore.java:365 - Initializing system.compaction_history INFO [SSTableBatchOpen:1] 2018-08-05 21:10:24,768 SSTableReader.java:475 - Opening /var/opt/xxx-managed/cassandra/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/system-compaction_history-ka-129 (91269 bytes) … Replaying commit logs: … INFO [main] 2018-08-05 21:10:25,896 CommitLogReplayer.java:267 - Replaying /var/opt/dynatrace-managed/cassandra/commitlog/CommitLog-4-1533133668366.log INFO [main] 2018-08-05 21:10:25,896 CommitLogReplayer.java:270 - Replaying /var/opt/dynatrace-managed/cassandra/commitlog/CommitLog-4-1533133668366.log (CL version 4, messaging version 8) … Even writing memtables already (below just pasted system tables, but also user tables): … INFO [MemtableFlushWriter:4] 2018-08-05 21:11:52,524 Memtable.java:347 - Writing Memtable-size_estimates@1941663179(2.655MiB<mailto:Memtable-size_estimates@1941663179(2.655MiB> serialized bytes, 325710 ops, 2%/0% of on/off-heap limit) INFO [MemtableFlushWriter:3] 2018-08-05 21:11:52,552 Memtable.java:347 - Writing Memtable-peer_events@1474667699(0.199KiB<mailto:Memtable-peer_events@1474667699(0.199KiB> serialized bytes, 4 ops, 0%/0% of on/off-heap limit) … Until it comes to a point where it can’t bind ports like the storage port 7000: ERROR [main] 2018-08-05 21:11:54,350 CassandraDaemon.java:395 - Fatal configuration error org.apache.cassandra.exceptions.ConfigurationException: /XXX:7000 is in use by another process. Change listen_address:storage_port in cassandra.yaml to values that do not conflict with other services at org.apache.cassandra.net<https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.cassandra.net&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=dT37H4lDhtCITNUYYOYimbR_t3oZ1llnqmnl6U_IG5I&s=4clNj_DUyt33zyqhP38rfg0z5222c24YibywZ9okhKw&e=>.MessagingService.getServerSockets(MessagingService.java:495) ~[apache-cassandra-2.1.18.jar:2.1.18] … Until Cassandra stops: … INFO [StorageServiceShutdownHook] 2018-08-05 21:11:54,361 Gossiper.java:1454 - Announcing shutdown … So, we have around 2 minutes where Cassandra is mangling with existing data, although it shouldn’t. Sounds like a potential candidate for data corruption, right? E.g. later on we then see things like (still while being in progress to shutdown?): WARN [SharedPool-Worker-1] 2018-08-05 21:11:58,181 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {} java.lang.RuntimeException: java.io.FileNotFoundException: /var/opt/xxx-managed/cassandra/xxx/xxx-fdc68b70950611e8ad7179f2d5bfa3cf/xxx-xxx-ka-15-Data.db (No such file or directory) at org.apache.cassandra.io<https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.cassandra.io&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=dT37H4lDhtCITNUYYOYimbR_t3oZ1llnqmnl6U_IG5I&s=QbL2c8hMPFEDG4q9lLbimSf5FCE4RIfmfYWbLM9395w&e=>.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:52) ~[apache-cassandra-2.1.18.jar:2.1.18] at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile.createPooledReader(CompressedPoolingSegmentedFile.java:95) ~[apache-cassandra-2.1.18.jar:2.1.18] at org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:62) ~[apache-cassandra-2.1.18.jar:2.1.18] … I found this one here: https://issues.apache.org/jira/browse/CASSANDRA-11540<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_CASSANDRA-2D11540&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=dT37H4lDhtCITNUYYOYimbR_t3oZ1llnqmnl6U_IG5I&s=GBvrnBmKbFQqFfGp2-UHglmGaQkGPGEDjdK9oQJBYF4&e=> So, if this all leads to corruption, might this be a candidate for a backport for a 2.1 bugfix release? Thanks a lot! Thomas The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freistädterstraße 313 ________________________________ The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
