Re: Data Corruption due to multiple Cassandra 2.1 processes?

[kurt greaves]

New ticket for backporting, referencing the existing.

On Mon., 13 Aug. 2018, 22:50 Steinmaurer, Thomas, <
thomas.steinmau...@dynatrace.com> wrote:

> Thanks Kurt.
>
>
>
> What is the proper workflow here to get this accepted? Create a new ticket
> dedicated for the backport referencing 11540 or re-open 11540?
>
>
>
> Thanks for your help.
>
>
>
> Thomas
>
>
>
> *From:* kurt greaves 
> *Sent:* Montag, 13. August 2018 13:24
> *To:* User 
> *Subject:* Re: Data Corruption due to multiple Cassandra 2.1 processes?
>
>
>
> Yeah that's not ideal and could lead to problems. I think corruption is
> only likely if compactions occur, but seems like data loss is a potential
> not to mention all sorts of other possible nasties that could occur running
> two C*'s at once. Seems to me that 11540 should have gone to 2.1 in the
> first place, but it just got missed. Very simple patch so I think a
> backport should be accepted.
>
>
>
> On 7 August 2018 at 15:57, Steinmaurer, Thomas <
> thomas.steinmau...@dynatrace.com> wrote:
>
> Hello,
>
>
>
> with 2.1, in case a second Cassandra process/instance is started on a host
> (by accident), may this result in some sort of corruption, although
> Cassandra will exit at some point in time due to not being able to bind TCP
> ports already in use?
>
>
>
> What we have seen in this scenario is something like that:
>
>
>
> ERROR [main] 2018-08-05 21:10:24,046 CassandraDaemon.java:120 - Error
> starting local jmx server:
>
> java.rmi.server.ExportException: Port already in use: 7199; nested
> exception is:
>
> java.net.BindException: Address already in use (Bind
> failed)
>
> …
>
>
>
> But then continuing with stuff like opening system and even user tables:
>
>
>
> INFO  [main] 2018-08-05 21:10:24,060 CacheService.java:110 - Initializing
> key cache with capacity of 100 MBs.
>
> INFO  [main] 2018-08-05 21:10:24,067 CacheService.java:132 - Initializing
> row cache with capacity of 0 MBs
>
> INFO  [main] 2018-08-05 21:10:24,073 CacheService.java:149 - Initializing
> counter cache with capacity of 50 MBs
>
> INFO  [main] 2018-08-05 21:10:24,074 CacheService.java:160 - Scheduling
> counter cache save to every 7200 seconds (going to save all keys).
>
> INFO  [main] 2018-08-05 21:10:24,161 ColumnFamilyStore.java:365 -
> Initializing system.sstable_activity
>
> INFO  [SSTableBatchOpen:2] 2018-08-05 21:10:24,692 SSTableReader.java:475
> - Opening
> /var/opt/xxx-managed/cassandra/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-165
> (2023 bytes)
>
> INFO  [SSTableBatchOpen:3] 2018-08-05 21:10:24,692 SSTableReader.java:475
> - Opening
> /var/opt/xxx-managed/cassandra/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-167
> (2336 bytes)
>
> INFO  [SSTableBatchOpen:1] 2018-08-05 21:10:24,692 SSTableReader.java:475
> - Opening
> /var/opt/xxx-managed/cassandra/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-166
> (2686 bytes)
>
> INFO  [main] 2018-08-05 21:10:24,755 ColumnFamilyStore.java:365 -
> Initializing system.hints
>
> INFO  [SSTableBatchOpen:1] 2018-08-05 21:10:24,758 SSTableReader.java:475
> - Opening
> /var/opt/xxx-managed/cassandra/system/hints-2666e20573ef38b390fefecf96e8f0c7/system-hints-ka-377
> (46210621 bytes)
>
> INFO  [main] 2018-08-05 21:10:24,766 ColumnFamilyStore.java:365 -
> Initializing system.compaction_history
>
> INFO  [SSTableBatchOpen:1] 2018-08-05 21:10:24,768 SSTableReader.java:475
> - Opening
> /var/opt/xxx-managed/cassandra/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/system-compaction_history-ka-129
> (91269 bytes)
>
> …
>
>
>
> Replaying commit logs:
>
>
>
> …
>
> INFO  [main] 2018-08-05 21:10:25,896 CommitLogReplayer.java:267 -
> Replaying
> /var/opt/dynatrace-managed/cassandra/commitlog/CommitLog-4-1533133668366.log
>
> INFO  [main] 2018-08-05 21:10:25,896 CommitLogReplayer.java:270 -
> Replaying
> /var/opt/dynatrace-managed/cassandra/commitlog/CommitLog-4-1533133668366.log
> (CL version 4, messaging version 8)
>
> …
>
>
>
> Even writing memtables already (below just pasted system tables, but also
> user tables):
>
>
>
> …
>
> INFO  [MemtableFlushWriter:4] 2018-08-05 21:11:52,524 Memtable.java:347 -
> Writing Memtable-size_estimates@1941663179(2.655MiB serialized bytes,
> 325710 ops, 2%/0% of on/off-heap limit)
>
> INFO  [MemtableFlushWriter:3] 2018-08-05 21:11:52,552 Memtable.java:347 -
> Writing Memtable-peer_events@1474667699(0.199KiB serialized bytes, 4 ops,
> 0%/0% of on/off-heap limit)
>
> …
>
>
>
> Until it comes to a point where it can’t bind ports like the storage port
> 7000:
>
>
>
> ERROR [main] 2018-08-05 21:11:54,350 CassandraDaemon.java:395 - Fatal
> configuration error
>
> org.apache.cassandra.exceptions.ConfigurationException: /XXX:7000 is in
> use by another process.  Change listen_address:storage_port in
> cassandra.yaml to values that do not conflict with other services
>
> at 

Fwd: Removing Extra Spaces and Row counts while using Capture Command

[kumar bharath]

Hi All,

I am using Cassandra Capture Command to perform a select query operation to
write data from a column family into JSON format file for further
processing. I am able to do that successfully, but  I am seeing extra
spaces and row count values after every few records.

please suggest a to get rid of these unusual extra spaces and row count
values.

Regards,
Bharath Kumar B

Re: upgrade 2.1 to 3.0

[kooljava2]

 We did run "nodetool upgradesstables" on all nodes. The C* version is 3.0.15
Thank you. 

   On Saturday, 11 August 2018, 21:47:52 GMT-7, Elliott Sims 
 wrote:  
 
 Might be a silly question, but did you run "nodetool upgradesstables" and 
convert to the 3.0 format?  Also, which 3.0?  Newest, or an earlier 3.0.x?

On Fri, Aug 10, 2018 at 3:05 PM, kooljava2  wrote:

Hello,
We recently upgrade C* from 2.1 to 3.0. After the upgrade we are seeing 
increase in the  total read bytes and read ops on the EBS volumes. It almost 
doubled on all the nodes.  The number of writes are same. 


Thank you. 


  

Adding new datacenter to the cluster

[Vitali Dyachuk]

Hello,
I'm going to follow this documentation to add a new datacenter to the C*
cluster
https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsAddDCToCluster.html

The main step is to run nodetool rebuild which will sync data to the new
datacenter,
this will load cluster badly since the main keyspace size is 2TB.
1) What are the best practicies to add a new datacenter with a lot of data?
2) How is it possible to stop rebuild?
3) What are the throttling possibilities
I'm using C* 3.0.17 with RF3

Vitali.

RE: [EXTERNAL] Re: Data Corruption due to multiple Cassandra 2.1 processes?

[Durity, Sean R]

I have definitely seen corruption, especially in system tables, when there are 
multiple instances of Cassandra running/trying to start. We had an internal 
tool that was supposed to restart processes (like Cassandra) if they were down, 
but it often re-checked before Cassandra was fully up and started another 
Cassandra process. Unwinding it could be very ugly.


Sean Durity

From: kurt greaves 
Sent: Monday, August 13, 2018 7:24 AM
To: User 
Subject: [EXTERNAL] Re: Data Corruption due to multiple Cassandra 2.1 processes?

Yeah that's not ideal and could lead to problems. I think corruption is only 
likely if compactions occur, but seems like data loss is a potential not to 
mention all sorts of other possible nasties that could occur running two C*'s 
at once. Seems to me that 11540 should have gone to 2.1 in the first place, but 
it just got missed. Very simple patch so I think a backport should be accepted.

On 7 August 2018 at 15:57, Steinmaurer, Thomas 
mailto:thomas.steinmau...@dynatrace.com>> 
wrote:
Hello,

with 2.1, in case a second Cassandra process/instance is started on a host (by 
accident), may this result in some sort of corruption, although Cassandra will 
exit at some point in time due to not being able to bind TCP ports already in 
use?

What we have seen in this scenario is something like that:

ERROR [main] 2018-08-05 21:10:24,046 CassandraDaemon.java:120 - Error starting 
local jmx server:
java.rmi.server.ExportException: Port already in use: 7199; nested exception is:
java.net.BindException: Address already in use (Bind failed)
…

But then continuing with stuff like opening system and even user tables:

INFO  [main] 2018-08-05 21:10:24,060 CacheService.java:110 - Initializing key 
cache with capacity of 100 MBs.
INFO  [main] 2018-08-05 21:10:24,067 CacheService.java:132 - Initializing row 
cache with capacity of 0 MBs
INFO  [main] 2018-08-05 21:10:24,073 CacheService.java:149 - Initializing 
counter cache with capacity of 50 MBs
INFO  [main] 2018-08-05 21:10:24,074 CacheService.java:160 - Scheduling counter 
cache save to every 7200 seconds (going to save all keys).
INFO  [main] 2018-08-05 21:10:24,161 ColumnFamilyStore.java:365 - Initializing 
system.sstable_activity
INFO  [SSTableBatchOpen:2] 2018-08-05 21:10:24,692 SSTableReader.java:475 - 
Opening 
/var/opt/xxx-managed/cassandra/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-165
 (2023 bytes)
INFO  [SSTableBatchOpen:3] 2018-08-05 21:10:24,692 SSTableReader.java:475 - 
Opening 
/var/opt/xxx-managed/cassandra/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-167
 (2336 bytes)
INFO  [SSTableBatchOpen:1] 2018-08-05 21:10:24,692 SSTableReader.java:475 - 
Opening 
/var/opt/xxx-managed/cassandra/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-166
 (2686 bytes)
INFO  [main] 2018-08-05 21:10:24,755 ColumnFamilyStore.java:365 - Initializing 
system.hints
INFO  [SSTableBatchOpen:1] 2018-08-05 21:10:24,758 SSTableReader.java:475 - 
Opening 
/var/opt/xxx-managed/cassandra/system/hints-2666e20573ef38b390fefecf96e8f0c7/system-hints-ka-377
 (46210621 bytes)
INFO  [main] 2018-08-05 21:10:24,766 ColumnFamilyStore.java:365 - Initializing 
system.compaction_history
INFO  [SSTableBatchOpen:1] 2018-08-05 21:10:24,768 SSTableReader.java:475 - 
Opening 
/var/opt/xxx-managed/cassandra/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/system-compaction_history-ka-129
 (91269 bytes)
…

Replaying commit logs:

…
INFO  [main] 2018-08-05 21:10:25,896 CommitLogReplayer.java:267 - Replaying 
/var/opt/dynatrace-managed/cassandra/commitlog/CommitLog-4-1533133668366.log
INFO  [main] 2018-08-05 21:10:25,896 CommitLogReplayer.java:270 - Replaying 
/var/opt/dynatrace-managed/cassandra/commitlog/CommitLog-4-1533133668366.log 
(CL version 4, messaging version 8)
…

Even writing memtables already (below just pasted system tables, but also user 
tables):

…
INFO  [MemtableFlushWriter:4] 2018-08-05 21:11:52,524 Memtable.java:347 - 
Writing 
Memtable-size_estimates@1941663179(2.655MiB
 serialized bytes, 325710 ops, 2%/0% of on/off-heap limit)
INFO  [MemtableFlushWriter:3] 2018-08-05 21:11:52,552 Memtable.java:347 - 
Writing 
Memtable-peer_events@1474667699(0.199KiB
 serialized bytes, 4 ops, 0%/0% of on/off-heap limit)
…

Until it comes to a point where it can’t bind ports like the storage port 7000:

ERROR [main] 2018-08-05 21:11:54,350 CassandraDaemon.java:395 - Fatal 
configuration error
org.apache.cassandra.exceptions.ConfigurationException: /XXX:7000 is in use by 
another process.  Change listen_address:storage_port in cassandra.yaml to 
values that do not conflict with other services
at 

RE: Data Corruption due to multiple Cassandra 2.1 processes?

[Steinmaurer, Thomas]

Thanks Kurt.

What is the proper workflow here to get this accepted? Create a new ticket 
dedicated for the backport referencing 11540 or re-open 11540?

Thanks for your help.

Thomas

From: kurt greaves 
Sent: Montag, 13. August 2018 13:24
To: User 
Subject: Re: Data Corruption due to multiple Cassandra 2.1 processes?

Yeah that's not ideal and could lead to problems. I think corruption is only 
likely if compactions occur, but seems like data loss is a potential not to 
mention all sorts of other possible nasties that could occur running two C*'s 
at once. Seems to me that 11540 should have gone to 2.1 in the first place, but 
it just got missed. Very simple patch so I think a backport should be accepted.

On 7 August 2018 at 15:57, Steinmaurer, Thomas 
mailto:thomas.steinmau...@dynatrace.com>> 
wrote:
Hello,

with 2.1, in case a second Cassandra process/instance is started on a host (by 
accident), may this result in some sort of corruption, although Cassandra will 
exit at some point in time due to not being able to bind TCP ports already in 
use?

What we have seen in this scenario is something like that:

ERROR [main] 2018-08-05 21:10:24,046 CassandraDaemon.java:120 - Error starting 
local jmx server:
java.rmi.server.ExportException: Port already in use: 7199; nested exception is:
java.net.BindException: Address already in use (Bind failed)
…

But then continuing with stuff like opening system and even user tables:

INFO  [main] 2018-08-05 21:10:24,060 CacheService.java:110 - Initializing key 
cache with capacity of 100 MBs.
INFO  [main] 2018-08-05 21:10:24,067 CacheService.java:132 - Initializing row 
cache with capacity of 0 MBs
INFO  [main] 2018-08-05 21:10:24,073 CacheService.java:149 - Initializing 
counter cache with capacity of 50 MBs
INFO  [main] 2018-08-05 21:10:24,074 CacheService.java:160 - Scheduling counter 
cache save to every 7200 seconds (going to save all keys).
INFO  [main] 2018-08-05 21:10:24,161 ColumnFamilyStore.java:365 - Initializing 
system.sstable_activity
INFO  [SSTableBatchOpen:2] 2018-08-05 21:10:24,692 SSTableReader.java:475 - 
Opening 
/var/opt/xxx-managed/cassandra/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-165
 (2023 bytes)
INFO  [SSTableBatchOpen:3] 2018-08-05 21:10:24,692 SSTableReader.java:475 - 
Opening 
/var/opt/xxx-managed/cassandra/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-167
 (2336 bytes)
INFO  [SSTableBatchOpen:1] 2018-08-05 21:10:24,692 SSTableReader.java:475 - 
Opening 
/var/opt/xxx-managed/cassandra/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-166
 (2686 bytes)
INFO  [main] 2018-08-05 21:10:24,755 ColumnFamilyStore.java:365 - Initializing 
system.hints
INFO  [SSTableBatchOpen:1] 2018-08-05 21:10:24,758 SSTableReader.java:475 - 
Opening 
/var/opt/xxx-managed/cassandra/system/hints-2666e20573ef38b390fefecf96e8f0c7/system-hints-ka-377
 (46210621 bytes)
INFO  [main] 2018-08-05 21:10:24,766 ColumnFamilyStore.java:365 - Initializing 
system.compaction_history
INFO  [SSTableBatchOpen:1] 2018-08-05 21:10:24,768 SSTableReader.java:475 - 
Opening 
/var/opt/xxx-managed/cassandra/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/system-compaction_history-ka-129
 (91269 bytes)
…

Replaying commit logs:

…
INFO  [main] 2018-08-05 21:10:25,896 CommitLogReplayer.java:267 - Replaying 
/var/opt/dynatrace-managed/cassandra/commitlog/CommitLog-4-1533133668366.log
INFO  [main] 2018-08-05 21:10:25,896 CommitLogReplayer.java:270 - Replaying 
/var/opt/dynatrace-managed/cassandra/commitlog/CommitLog-4-1533133668366.log 
(CL version 4, messaging version 8)
…

Even writing memtables already (below just pasted system tables, but also user 
tables):

…
INFO  [MemtableFlushWriter:4] 2018-08-05 21:11:52,524 Memtable.java:347 - 
Writing 
Memtable-size_estimates@1941663179(2.655MiB
 serialized bytes, 325710 ops, 2%/0% of on/off-heap limit)
INFO  [MemtableFlushWriter:3] 2018-08-05 21:11:52,552 Memtable.java:347 - 
Writing 
Memtable-peer_events@1474667699(0.199KiB
 serialized bytes, 4 ops, 0%/0% of on/off-heap limit)
…

Until it comes to a point where it can’t bind ports like the storage port 7000:

ERROR [main] 2018-08-05 21:11:54,350 CassandraDaemon.java:395 - Fatal 
configuration error
org.apache.cassandra.exceptions.ConfigurationException: /XXX:7000 is in use by 
another process.  Change listen_address:storage_port in cassandra.yaml to 
values that do not conflict with other services
at 

How to rename the column name in Cassandra tables

[Irtiza Ali]

Hello everyone,

*Issue*
Currently, we are facing an issue of renaming the Cassandra table's column
name. According to the documentation, one can change the name of only those
columns that are part of primary or clustering columns(keys).


*Question*
Is there any way to rename the name of *non-primary or clustering
columns(keys)?*

Thank you

IA

Re: Configuration parameter to reject incremental repair?

[kurt greaves]

No flag currently exists. Probably a good idea considering the serious
issues with incremental repairs since forever, and the change of defaults
since 3.0.

On 7 August 2018 at 16:44, Steinmaurer, Thomas <
thomas.steinmau...@dynatrace.com> wrote:

> Hello,
>
>
>
> we are running Cassandra in AWS and On-Premise at customer sites,
> currently 2.1 in production with 3.11 in loadtest.
>
>
>
> In a migration path from 2.1 to 3.11.x, I’m afraid that at some point in
> time we end up in incremental repairs being enabled / ran a first time
> unintentionally, cause:
>
> a) A lot of online resources / examples do not use the -full command-line
> option
>
> b) Our internal (support) tickets of course also state nodetool repair
> command without the -full option, as these are for 2.1
>
>
>
> Especially for On-Premise customers (with less control than with our AWS
> deployments), this asks a bit for getting out-of-control once we have 3.11
> out and nodetool repair being run without the -full command-line option.
>
>
>
> So, what do you think about a JVM system property, cassandra.yaml … to
> basically let the operator chose if incremental repairs are allowed or not?
> I know, such a flag still can be flipped then (by the customer), but as a
> first safety stage possibly sufficient enough.
>
>
>
> Or perhaps something like that is already available (vaguely remember
> something like that for MV).
>
>
>
> Thanks a lot,
>
> Thomas
>
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
>

Re: Data Corruption due to multiple Cassandra 2.1 processes?

[kurt greaves]

Yeah that's not ideal and could lead to problems. I think corruption is
only likely if compactions occur, but seems like data loss is a potential
not to mention all sorts of other possible nasties that could occur running
two C*'s at once. Seems to me that 11540 should have gone to 2.1 in the
first place, but it just got missed. Very simple patch so I think a
backport should be accepted.

On 7 August 2018 at 15:57, Steinmaurer, Thomas <
thomas.steinmau...@dynatrace.com> wrote:

> Hello,
>
>
>
> with 2.1, in case a second Cassandra process/instance is started on a host
> (by accident), may this result in some sort of corruption, although
> Cassandra will exit at some point in time due to not being able to bind TCP
> ports already in use?
>
>
>
> What we have seen in this scenario is something like that:
>
>
>
> ERROR [main] 2018-08-05 21:10:24,046 CassandraDaemon.java:120 - Error
> starting local jmx server:
>
> java.rmi.server.ExportException: Port already in use: 7199; nested
> exception is:
>
> java.net.BindException: Address already in use (Bind
> failed)
>
> …
>
>
>
> But then continuing with stuff like opening system and even user tables:
>
>
>
> INFO  [main] 2018-08-05 21:10:24,060 CacheService.java:110 - Initializing
> key cache with capacity of 100 MBs.
>
> INFO  [main] 2018-08-05 21:10:24,067 CacheService.java:132 - Initializing
> row cache with capacity of 0 MBs
>
> INFO  [main] 2018-08-05 21:10:24,073 CacheService.java:149 - Initializing
> counter cache with capacity of 50 MBs
>
> INFO  [main] 2018-08-05 21:10:24,074 CacheService.java:160 - Scheduling
> counter cache save to every 7200 seconds (going to save all keys).
>
> INFO  [main] 2018-08-05 21:10:24,161 ColumnFamilyStore.java:365 -
> Initializing system.sstable_activity
>
> INFO  [SSTableBatchOpen:2] 2018-08-05 21:10:24,692 SSTableReader.java:475
> - Opening /var/opt/xxx-managed/cassandra/system/sstable_activity-
> 5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-165 (2023
> bytes)
>
> INFO  [SSTableBatchOpen:3] 2018-08-05 21:10:24,692 SSTableReader.java:475
> - Opening /var/opt/xxx-managed/cassandra/system/sstable_activity-
> 5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-167 (2336
> bytes)
>
> INFO  [SSTableBatchOpen:1] 2018-08-05 21:10:24,692 SSTableReader.java:475
> - Opening /var/opt/xxx-managed/cassandra/system/sstable_activity-
> 5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-166 (2686
> bytes)
>
> INFO  [main] 2018-08-05 21:10:24,755 ColumnFamilyStore.java:365 -
> Initializing system.hints
>
> INFO  [SSTableBatchOpen:1] 2018-08-05 21:10:24,758 SSTableReader.java:475
> - Opening /var/opt/xxx-managed/cassandra/system/hints-
> 2666e20573ef38b390fefecf96e8f0c7/system-hints-ka-377 (46210621 bytes)
>
> INFO  [main] 2018-08-05 21:10:24,766 ColumnFamilyStore.java:365 -
> Initializing system.compaction_history
>
> INFO  [SSTableBatchOpen:1] 2018-08-05 21:10:24,768 SSTableReader.java:475
> - Opening /var/opt/xxx-managed/cassandra/system/compaction_history-
> b4dbb7b4dc493fb5b3bfce6e434832ca/system-compaction_history-ka-129 (91269
> bytes)
>
> …
>
>
>
> Replaying commit logs:
>
>
>
> …
>
> INFO  [main] 2018-08-05 21:10:25,896 CommitLogReplayer.java:267 -
> Replaying /var/opt/dynatrace-managed/cassandra/commitlog/CommitLog-
> 4-1533133668366.log
>
> INFO  [main] 2018-08-05 21:10:25,896 CommitLogReplayer.java:270 -
> Replaying 
> /var/opt/dynatrace-managed/cassandra/commitlog/CommitLog-4-1533133668366.log
> (CL version 4, messaging version 8)
>
> …
>
>
>
> Even writing memtables already (below just pasted system tables, but also
> user tables):
>
>
>
> …
>
> INFO  [MemtableFlushWriter:4] 2018-08-05 21:11:52,524 Memtable.java:347 -
> Writing Memtable-size_estimates@1941663179(2.655MiB serialized bytes,
> 325710 ops, 2%/0% of on/off-heap limit)
>
> INFO  [MemtableFlushWriter:3] 2018-08-05 21:11:52,552 Memtable.java:347 -
> Writing Memtable-peer_events@1474667699(0.199KiB serialized bytes, 4 ops,
> 0%/0% of on/off-heap limit)
>
> …
>
>
>
> Until it comes to a point where it can’t bind ports like the storage port
> 7000:
>
>
>
> ERROR [main] 2018-08-05 21:11:54,350 CassandraDaemon.java:395 - Fatal
> configuration error
>
> org.apache.cassandra.exceptions.ConfigurationException: /XXX:7000 is in
> use by another process.  Change listen_address:storage_port in
> cassandra.yaml to values that do not conflict with other services
>
> at org.apache.cassandra.net.MessagingService.
> getServerSockets(MessagingService.java:495) ~[apache-cassandra-2.1.18.jar:
> 2.1.18]
>
> …
>
>
>
> Until Cassandra stops:
>
>
>
> …
>
> INFO  [StorageServiceShutdownHook] 2018-08-05 21:11:54,361
> Gossiper.java:1454 - Announcing shutdown
>
> …
>
>
>
>
>
> So, we have around 2 minutes where Cassandra is mangling with existing
> data, although it shouldn’t.
>
>
>
> Sounds like a potential candidate for data corruption, right? E.g. later
> on we then see things like (still while being in progress to shutdown?):

Re: Extending Cassandra on AWS from single Region to Multi-Region

[Kyrylo Lebediev]

Hi,


There is an instruction how to switch to a different snitch in datastax docs: 
https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsSwitchSnitch.html

Personally I haven't tried to change Ec2 to Ec2Multi, but my understanding is 
that as long as 'dc' and 'rack' values are left unchanged for all nodes after 
changing snitch, the shouldn't be any issues. But it's just my guess, you have 
to test this procedure in non-prod env before making this change in prod.


Also, consider using GossipingPropertyFileSnitch which  requires some 
additional configuration in AWS (editing cassandra-racdc.properties file) 
comparing to native AWS snitches, gives more flexibility.


Regards,

Kyrill


From: srinivasarao daruna 
Sent: Friday, August 10, 2018 5:14:30 PM
To: user@cassandra.apache.org
Subject: Re: Extending Cassandra on AWS from single Region to Multi-Region

Hey All,

Any info on this topic.?

Thank You,
Regards,
Srini

On Wed, Aug 8, 2018, 9:46 PM srinivasarao daruna 
mailto:sree.srin...@gmail.com>> wrote:
Hi All,

We have built Cassandra on AWS EC2 instances. Initially when creating cluster 
we have not considered multi-region deployment and we have used AWS EC2Snitch.

We have used EBS Volumes to save our data and each of those disks were filled 
around 350G.
We want to extend it to Multi Region and wanted to know the better approach and 
recommendations to achieve this process.

I agree that we have made a mistake by not using EC2MultiRegionSnitch, but its 
past now and if anyone faced or implemented similar thing i would like to get 
some guidance.

Any help would be very much appreciated.

Thank You,
Regards,
Srini

Re: Compression Tuning Tutorial

[Kyrylo Lebediev]

Thank you, Jon!


From: Jonathan Haddad 
Sent: Thursday, August 9, 2018 7:29:24 PM
To: user
Subject: Re: Compression Tuning Tutorial

There's a discussion about direct I/O here you might find interesting: 
https://issues.apache.org/jira/browse/CASSANDRA-14466

I suspect the main reason is that O_DIRECT wasn't added till Java 10, and while 
it could be used with some workarounds, there's a lot of entropy around 
changing something like this.  It's not a trivial task to do it right, and 
mixing has some really nasty issues.

At least it means there's lots of room for improvement though :)


On Thu, Aug 9, 2018 at 5:36 AM Kyrylo Lebediev 
 wrote:

Thank you Jon, great article as usually!


One topic that was discussed in the article is filesystem cache which is 
traditionally leveraged for data caching in Cassandra (with row-caching 
disabled by default).

IIRC mmap() is used.

Some RDBMS and NoSQL DB's as well use direct I/O + async I/O + maintain own, 
not kernel-managed, DB Cache thus improving overall performance.

As Cassandra is designed to be a DB with low response time, this approach with 
DIO/AIO/DB Cache seems to be a really useful feature.

Just out of curiosity, are there reasons why this advanced IO stack wasn't 
implemented, except lack of resources to do this?


Regards,

Kyrill


From: Eric Plowe mailto:eric.pl...@gmail.com>>
Sent: Wednesday, August 8, 2018 9:39:44 PM
To: user@cassandra.apache.org
Subject: Re: Compression Tuning Tutorial

Great post, Jonathan! Thank you very much.

~Eric

On Wed, Aug 8, 2018 at 2:34 PM Jonathan Haddad 
mailto:j...@jonhaddad.com>> wrote:
Hey folks,

We've noticed a lot over the years that people create tables usually leaving 
the default compression parameters, and have spent a lot of time helping teams 
figure out the right settings for their cluster based on their workload.  I 
finally managed to write some thoughts down along with a high level breakdown 
of how the internals function that should help people pick better settings for 
their cluster.

This post focuses on a mixed 50:50 read:write workload, but the same 
conclusions are drawn from a read heavy workload.  Hopefully this helps some 
folks get better performance / save some money on hardware!

http://thelastpickle.com/blog/2018/08/08/compression_performance.html


--
Jon Haddad
Principal Consultant, The Last Pickle


--
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade