Re: Requesting Feedback for Cassandra as a backup solution.

[CPC]

hi,

We implemented same strategy in one of our customers. Since 2016 we had one
downtime in one DC because of high temperature(whole physical DC shutdown).

With that approach I assume you will use Cassandra as a queue. You have to
be careful about modeling and should use multiple partitions may be based
on hour or fixed size partitions.

Another thing is that Kafka has really high throughput so you should plan
how many Cassandra node you need to meet same throughput.

Another approach would be to use another Kafka cluster or queue technology
as backup.



On Sat, Feb 17, 2024, 11:45 AM Gowtham S  wrote:

> Dear Cassandra Community,
>
> I am reaching out to seek your valuable feedback and insights on a
> proposed solution we are considering for managing Kafka outages using
> Cassandra.
>
> At our organization, we heavily rely on Kafka for real-time data
> processing and messaging. However, like any technology, Kafka is
> susceptible to occasional outages which can disrupt our operations and
> impact our services. To mitigate the impact of such outages and ensure
> continuity, we are exploring the possibility of leveraging Cassandra as a
> backup solution.
>
> Our proposed approach involves storing messages in Cassandra during Kafka
> outages. Subsequently, we plan to implement a scheduler that will read from
> Cassandra and attempt to write these messages back into Kafka once it is
> operational again.
>
> We believe that by adopting this strategy, we can achieve the following
> benefits:
>
>1.
>
>Improved Fault Tolerance: By having a backup mechanism in place, we
>can reduce the risk of data loss and ensure continuity of operations during
>Kafka outages.
>2.
>
>Enhanced Reliability: Cassandra's distributed architecture and
>built-in replication features make it well-suited for storing data
>reliably, even in the face of failures.
>3.
>
>Scalability: Both Cassandra and Kafka are designed to scale
>horizontally, allowing us to handle increased loads seamlessly.
>
> Before proceeding further with this approach, we would greatly appreciate
> any feedback, suggestions, or concerns from the community. Specifically, we
> are interested in hearing about:
>
>- Potential challenges or drawbacks of using Cassandra as a backup
>solution for Kafka outages.
>- Best practices or recommendations for implementing such a backup
>mechanism effectively.
>- Any alternative approaches or technologies that we should consider?
>
> Your expertise and insights are invaluable to us, and we are eager to
> learn from your experiences and perspectives. Please feel free to share
> your thoughts or reach out to us with any questions or clarifications.
>
> Thank you for taking the time to consider our proposal, and we look
> forward to hearing from you soon.
> Thanks and regards,
> Gowtham S
>

Re: Cassandra and Vertica

[CPC]

Data distribution(vertica segements cassandra partition keys) are similar.
Both dbms hold data as immutable data files but this is it. Aside from that
nothing is similar. Cassandra was designed for OLTP loads but vertica
designed for analytical loads.

On Sun, 15 Nov 2020 at 23:37, Manu Chadha  wrote:

> Has anyone here used Vertica? Out of curiosity, are Cassandra and Vertica
> databases similar? I recently spent time reading about Vertica. It has
> features like columnar data structures called projections which optimise
> query execution. It also has no single point of failure. But it comes out
> as an RDBMS database which supports join. The fact that in Cassandra as
> well a table is created to serve a query, it made me wonder if/how
> Cassandra and Vertica are similar and different?
>
>
>
> Sent from Mail  for
> Windows 10
>
>
>

Re: high write latency on a single table

[CPC]

In the mean time we opened traces and found that normal insert operations
does not have high latency. Delete operations have high latency. And all
deletes are range deletes. Is there any performance regression about range
deletes? Mean time of delete operations are 45 times higher than insert
operations.

On Thu, Jul 25, 2019, 11:46 AM mehmet bursali 
wrote:

> awesome! so we can make a further investigation by using cassandra
> exporter on this link.  https://github.com/criteo/cassandra_exporter
> This exporter gives detailed information for read/write operations on each
> column  by using metrics below..
>
>  org:apache:cassandra:metrics:columnfamily:.*  ( reads from table metrics in 
> cassandra 
> https://cassandra.apache.org/doc/latest/operating/metrics.html#table-metrics )
>
>
>
>
>
>
>
>
>
>
>
> On Wednesday, July 24, 2019, 11:51:28 PM GMT+3, CPC 
> wrote:
>
>
> Hi Mehmet,
>
> Yes prometheus and opscenter
>
> On Wed, 24 Jul 2019 at 17:09, mehmet bursali 
> wrote:
>
> hi,
> do you use any perfomance monitoring tool like prometheus?
>
>
>
>
> On Monday, July 22, 2019, 1:16:58 PM GMT+3, CPC 
> wrote:
>
>
> Hi everybody,
>
> State column contains "R" or "D" values. Just a single character. As Rajsekhar
> said, only difference is the table can contain high number of cell count.
> In the mean time we made a major compaction and data per node was 5-6 gb.
>
> On Mon, Jul 22, 2019, 10:56 AM Rajsekhar Mallick 
> wrote:
>
> Hello Team,
>
> The difference in write latencies between both the tables though
> significant,but the higher latency being 11.353 ms is still acceptable.
>
> Overall Writes not being an issue, but write latency for this particular
> table on the higher side does point towards data being written to the table.
> Few things which I noticed, is the data in cell count column in nodetool
> tablehistogram o/p for message_history_state table is scattered.
> The partition size histogram for the tables is consistent, but the column
> count histogram for the impacted table isn't uniform.
> May be we can start thinking on these lines.
>
> I would also wait for some expert advice here.
>
> Thanks
>
>
> On Mon,the 22 Jul, 2019, 12:31 PM Ben Slater, 
> wrote:
>
> Is the size of the data in your “state” column variable? The higher write
> latencies at the 95%+ could line up with large volumes of data for
> particular rows in that column (the one column not in both tables)?
>
> Cheers
> Ben
>
> ---
>
>
> *Ben Slater**Chief Product Officer*
>
> <https://www.instaclustr.com/platform/>
>
> <https://www.facebook.com/instaclustr>   <https://twitter.com/instaclustr>
><https://www.linkedin.com/company/instaclustr>
>
> Read our latest technical blog posts here
> <https://www.instaclustr.com/blog/>.
>
> This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
> and Instaclustr Inc (USA).
>
> This email and any attachments may contain confidential and legally
> privileged information.  If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
>
>
> On Mon, 22 Jul 2019 at 16:46, CPC  wrote:
>
> Hi guys,
>
> Any idea? I thought it might be a bug but could not find anything related
> on jira.
>
> On Fri, Jul 19, 2019, 12:45 PM CPC  wrote:
>
> Hi Rajsekhar,
>
> Here the details:
>
> 1)
>
> [cassadm@bipcas00 ~]$ nodetool tablestats tims.MESSAGE_HISTORY
> Total number of tables: 259
> 
> Keyspace : tims
> Read Count: 208256144
> Read Latency: 7.655146714749506 ms
> Write Count: 2218205275
> Write Latency: 1.7826005103175133 ms
> Pending Flushes: 0
> Table: MESSAGE_HISTORY
> SSTable count: 41
> Space used (live): 976964101899
> Space used (total): 976964101899
> Space used by snapshots (total): 3070598526780
> Off heap memory used (total): 185828820
> SSTable Compression Ratio: 0.8219217809913125
> Number of partitions (estimate): 8175715
> Memtable cell count: 73124
> Memtable data size: 26543733
> Memtable off heap memory used: 27829672
> Memtable switch count: 1607
> Local read count: 7871917
> Local read latency: 1.187 ms
> Local write count: 172220954
> Local write latency: 0.021 ms
>  

Re: high write latency on a single table

[CPC]

Hi Mehmet,

Yes prometheus and opscenter

On Wed, 24 Jul 2019 at 17:09, mehmet bursali 
wrote:

> hi,
> do you use any perfomance monitoring tool like prometheus?
>
>
>
>
> On Monday, July 22, 2019, 1:16:58 PM GMT+3, CPC 
> wrote:
>
>
> Hi everybody,
>
> State column contains "R" or "D" values. Just a single character. As Rajsekhar
> said, only difference is the table can contain high number of cell count.
> In the mean time we made a major compaction and data per node was 5-6 gb.
>
> On Mon, Jul 22, 2019, 10:56 AM Rajsekhar Mallick 
> wrote:
>
> Hello Team,
>
> The difference in write latencies between both the tables though
> significant,but the higher latency being 11.353 ms is still acceptable.
>
> Overall Writes not being an issue, but write latency for this particular
> table on the higher side does point towards data being written to the table.
> Few things which I noticed, is the data in cell count column in nodetool
> tablehistogram o/p for message_history_state table is scattered.
> The partition size histogram for the tables is consistent, but the column
> count histogram for the impacted table isn't uniform.
> May be we can start thinking on these lines.
>
> I would also wait for some expert advice here.
>
> Thanks
>
>
> On Mon,the 22 Jul, 2019, 12:31 PM Ben Slater, 
> wrote:
>
> Is the size of the data in your “state” column variable? The higher write
> latencies at the 95%+ could line up with large volumes of data for
> particular rows in that column (the one column not in both tables)?
>
> Cheers
> Ben
>
> ---
>
>
> *Ben Slater**Chief Product Officer*
>
> <https://www.instaclustr.com/platform/>
>
> <https://www.facebook.com/instaclustr>   <https://twitter.com/instaclustr>
><https://www.linkedin.com/company/instaclustr>
>
> Read our latest technical blog posts here
> <https://www.instaclustr.com/blog/>.
>
> This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
> and Instaclustr Inc (USA).
>
> This email and any attachments may contain confidential and legally
> privileged information.  If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
>
>
> On Mon, 22 Jul 2019 at 16:46, CPC  wrote:
>
> Hi guys,
>
> Any idea? I thought it might be a bug but could not find anything related
> on jira.
>
> On Fri, Jul 19, 2019, 12:45 PM CPC  wrote:
>
> Hi Rajsekhar,
>
> Here the details:
>
> 1)
>
> [cassadm@bipcas00 ~]$ nodetool tablestats tims.MESSAGE_HISTORY
> Total number of tables: 259
> 
> Keyspace : tims
> Read Count: 208256144
> Read Latency: 7.655146714749506 ms
> Write Count: 2218205275
> Write Latency: 1.7826005103175133 ms
> Pending Flushes: 0
> Table: MESSAGE_HISTORY
> SSTable count: 41
> Space used (live): 976964101899
> Space used (total): 976964101899
> Space used by snapshots (total): 3070598526780
> Off heap memory used (total): 185828820
> SSTable Compression Ratio: 0.8219217809913125
> Number of partitions (estimate): 8175715
> Memtable cell count: 73124
> Memtable data size: 26543733
> Memtable off heap memory used: 27829672
> Memtable switch count: 1607
> Local read count: 7871917
> Local read latency: 1.187 ms
> Local write count: 172220954
> Local write latency: 0.021 ms
> Pending flushes: 0
> Percent repaired: 0.0
> Bloom filter false positives: 130
> Bloom filter false ratio: 0.0
> Bloom filter space used: 10898488
> Bloom filter off heap memory used: 10898160
> Index summary off heap memory used: 2480140
> Compression metadata off heap memory used: 144620848
> Compacted partition minimum bytes: 36
> Compacted partition maximum bytes: 557074610
> Compacted partition mean bytes: 155311
> Average live cells per slice (last five minutes): 
> 25.56639344262295
> Maximum live cells per slice (last five minutes): 5722
> Average tombstones per slice (last five minutes): 
> 1.8681948424068768
> Maximum tombstones per slice (last five minutes): 770
>   

Re: high write latency on a single table

[CPC]

Hi everybody,

State column contains "R" or "D" values. Just a single character. As Rajsekhar
said, only difference is the table can contain high number of cell count.
In the mean time we made a major compaction and data per node was 5-6 gb.

On Mon, Jul 22, 2019, 10:56 AM Rajsekhar Mallick 
wrote:

> Hello Team,
>
> The difference in write latencies between both the tables though
> significant,but the higher latency being 11.353 ms is still acceptable.
>
> Overall Writes not being an issue, but write latency for this particular
> table on the higher side does point towards data being written to the table.
> Few things which I noticed, is the data in cell count column in nodetool
> tablehistogram o/p for message_history_state table is scattered.
> The partition size histogram for the tables is consistent, but the column
> count histogram for the impacted table isn't uniform.
> May be we can start thinking on these lines.
>
> I would also wait for some expert advice here.
>
> Thanks
>
>
> On Mon,the 22 Jul, 2019, 12:31 PM Ben Slater, 
> wrote:
>
>> Is the size of the data in your “state” column variable? The higher write
>> latencies at the 95%+ could line up with large volumes of data for
>> particular rows in that column (the one column not in both tables)?
>>
>> Cheers
>> Ben
>>
>> ---
>>
>>
>> *Ben Slater**Chief Product Officer*
>>
>> <https://www.instaclustr.com/platform/>
>>
>> <https://www.facebook.com/instaclustr>
>> <https://twitter.com/instaclustr>
>> <https://www.linkedin.com/company/instaclustr>
>>
>> Read our latest technical blog posts here
>> <https://www.instaclustr.com/blog/>.
>>
>> This email has been sent on behalf of Instaclustr Pty. Limited
>> (Australia) and Instaclustr Inc (USA).
>>
>> This email and any attachments may contain confidential and legally
>> privileged information.  If you are not the intended recipient, do not copy
>> or disclose its content, but please reply to this email immediately and
>> highlight the error to the sender and then immediately delete the message.
>>
>>
>> On Mon, 22 Jul 2019 at 16:46, CPC  wrote:
>>
>>> Hi guys,
>>>
>>> Any idea? I thought it might be a bug but could not find anything
>>> related on jira.
>>>
>>> On Fri, Jul 19, 2019, 12:45 PM CPC  wrote:
>>>
>>>> Hi Rajsekhar,
>>>>
>>>> Here the details:
>>>>
>>>> 1)
>>>>
>>>> [cassadm@bipcas00 ~]$ nodetool tablestats tims.MESSAGE_HISTORY
>>>> Total number of tables: 259
>>>> 
>>>> Keyspace : tims
>>>> Read Count: 208256144
>>>> Read Latency: 7.655146714749506 ms
>>>> Write Count: 2218205275
>>>> Write Latency: 1.7826005103175133 ms
>>>> Pending Flushes: 0
>>>> Table: MESSAGE_HISTORY
>>>> SSTable count: 41
>>>> Space used (live): 976964101899
>>>> Space used (total): 976964101899
>>>> Space used by snapshots (total): 3070598526780
>>>> Off heap memory used (total): 185828820
>>>> SSTable Compression Ratio: 0.8219217809913125
>>>> Number of partitions (estimate): 8175715
>>>> Memtable cell count: 73124
>>>> Memtable data size: 26543733
>>>> Memtable off heap memory used: 27829672
>>>> Memtable switch count: 1607
>>>> Local read count: 7871917
>>>> Local read latency: 1.187 ms
>>>> Local write count: 172220954
>>>> Local write latency: 0.021 ms
>>>> Pending flushes: 0
>>>> Percent repaired: 0.0
>>>> Bloom filter false positives: 130
>>>> Bloom filter false ratio: 0.0
>>>> Bloom filter space used: 10898488
>>>> Bloom filter off heap memory used: 10898160
>>>> Index summary off heap memory used: 2480140
>>>> Compression metadata off heap memory used: 144620848
>>>> Compacted partition minimum bytes: 36
>>>> Compacted partition maximum bytes: 557074610
>>>>

Re: high write latency on a single table

[CPC]

Hi guys,

Any idea? I thought it might be a bug but could not find anything related
on jira.

On Fri, Jul 19, 2019, 12:45 PM CPC  wrote:

> Hi Rajsekhar,
>
> Here the details:
>
> 1)
>
> [cassadm@bipcas00 ~]$ nodetool tablestats tims.MESSAGE_HISTORY
> Total number of tables: 259
> 
> Keyspace : tims
> Read Count: 208256144
> Read Latency: 7.655146714749506 ms
> Write Count: 2218205275
> Write Latency: 1.7826005103175133 ms
> Pending Flushes: 0
> Table: MESSAGE_HISTORY
> SSTable count: 41
> Space used (live): 976964101899
> Space used (total): 976964101899
> Space used by snapshots (total): 3070598526780
> Off heap memory used (total): 185828820
> SSTable Compression Ratio: 0.8219217809913125
> Number of partitions (estimate): 8175715
> Memtable cell count: 73124
> Memtable data size: 26543733
> Memtable off heap memory used: 27829672
> Memtable switch count: 1607
> Local read count: 7871917
> Local read latency: 1.187 ms
> Local write count: 172220954
> Local write latency: 0.021 ms
> Pending flushes: 0
> Percent repaired: 0.0
> Bloom filter false positives: 130
> Bloom filter false ratio: 0.0
> Bloom filter space used: 10898488
> Bloom filter off heap memory used: 10898160
> Index summary off heap memory used: 2480140
> Compression metadata off heap memory used: 144620848
> Compacted partition minimum bytes: 36
> Compacted partition maximum bytes: 557074610
> Compacted partition mean bytes: 155311
> Average live cells per slice (last five minutes): 
> 25.56639344262295
> Maximum live cells per slice (last five minutes): 5722
> Average tombstones per slice (last five minutes): 
> 1.8681948424068768
> Maximum tombstones per slice (last five minutes): 770
> Dropped Mutations: 97812
>
> 
> [cassadm@bipcas00 ~]$ nodetool tablestats tims.MESSAGE_HISTORY_STATE
> Total number of tables: 259
> 
> Keyspace : tims
> Read Count: 208257486
> Read Latency: 7.655137315414438 ms
> Write Count: 2218218966
> Write Latency: 1.7825896304427324 ms
> Pending Flushes: 0
> Table: MESSAGE_HISTORY_STATE
> SSTable count: 5
> Space used (live): 6403033568
> Space used (total): 6403033568
> Space used by snapshots (total): 19086872706
> Off heap memory used (total): 6727565
> SSTable Compression Ratio: 0.271857664111622
> Number of partitions (estimate): 1396462
> Memtable cell count: 77450
> Memtable data size: 620776
> Memtable off heap memory used: 1338914
> Memtable switch count: 1616
> Local read count: 988278
> Local read latency: 0.518 ms
> Local write count: 109292691
> Local write latency: 11.353 ms
> Pending flushes: 0
> Percent repaired: 0.0
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 1876208
> Bloom filter off heap memory used: 1876168
> Index summary off heap memory used: 410747
> Compression metadata off heap memory used: 3101736
> Compacted partition minimum bytes: 36
> Compacted partition maximum bytes: 129557750
> Compacted partition mean bytes: 17937
> Average live cells per slice (last five minutes): 
> 4.692893401015229
> Maximum live cells per slice (last five minutes): 258
> Average tombstones per slice (last five minutes): 1.0
> Maximum tombstones per slice (last five minutes): 1
> Dropped Mutations: 1344158
>
> 2)
>
> [cassadm@bipcas00 conf]$ nodetool tablehistograms tims MESSAGE_HISTORY
> tims/MESSAGE_HISTORY histograms
> Percentile  SSTables Write Latency  Read LatencyPartition Size
> Cell Count
>   (micros)  (micros)   (bytes)
> 50% 3.00  

Re: high write latency on a single table

[CPC]

 histograms
Percentile  SSTables Write Latency  Read LatencyPartition
SizeCell Count
  (micros)  (micros)
(bytes)
50% 5.00 20.50315.85
924 1
75% 6.00 35.43379.02
5722 7
95%10.00   4055.27785.94
61214   310
98%10.00  74975.55   3379.39
182785   924
99%10.00 107964.79  10090.81
315852  1916
Min 0.00  3.31 42.51
 36 0
Max10.00 322381.14  25109.16
129557750   1629722

3) RF=3

4)CL  QUORUM

5) Single insert prepared statement. no LOGGED/UNLOGGED batch or LWT


On Thu, 18 Jul 2019 at 20:51, Rajsekhar Mallick 
wrote:

> Hello,
>
> Kindly post below details
>
> 1. Nodetool cfstats for both the tables.
> 2. Nodetool cfhistograms for both the tables.
> 3. Replication factor of the tables.
> 4. Consistency with which write requests are sent
> 5. Also the type of write queries for the table  if handy would also help
> (Light weight transactions or Batch writes or Prepared statements)
>
> Thanks
>
> On 2019/07/18 15:48:09, CPC  wrote:
> > Hi all,>
> >
> > Our cassandra cluster consist of two dc and every dc we have 10 nodes.
> We>
> > are using DSE 5.1.12 (cassandra 3.11).We have a high local write latency
> on>
> > a single table. All other tables in our keyspace  have normal latencies>
> > like 0.02 msec,even tables that have more write tps and more data.
> Below>
> > you can find two table descriptions and their latencies.>
> > message_history_state have high local write latency. This is not node>
> > specific every node have this high local write latency for>
> > message_history_state. Have you ever see such a behavior or any clue
> why>
> > this could happen?>
> >
> >  CREATE TABLE tims."MESSAGE_HISTORY" (>
> > > username text,>
> > > date_partition text,>
> > > jid text,>
> > > sent_time timestamp,>
> > > message_id text,>
> > > stanza text,>
> > > PRIMARY KEY ((username, date_partition), jid, sent_time,
> message_id)>
> > > ) WITH CLUSTERING ORDER BY (jid ASC, sent_time DESC, message_id ASC)>
> > > AND bloom_filter_fp_chance = 0.01>
> > > AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}>
> > > AND comment = ''>
> > > AND compaction = {'bucket_high': '1.5', 'bucket_low': '0.5',
> 'class':>
> > > 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',>
> > > 'enabled': 'true', 'max_threshold': '32', 'min_sstable_size': '50',>
> > > 'min_threshold': '4', 'tombstone_compaction_interval': '86400',>
> > > 'tombstone_threshold': '0.2', 'unchecked_tombstone_compaction':
> 'false'}>
> > > AND compression = {'chunk_length_in_kb': '64', 'class':>
> > > 'org.apache.cassandra.io.compress.LZ4Compressor'}>
> > > AND crc_check_chance = 1.0>
> > > AND dclocal_read_repair_chance = 0.0>
> > > AND default_time_to_live = 0>
> > > AND gc_grace_seconds = 86400>
> > > AND max_index_interval = 2048>
> > > AND memtable_flush_period_in_ms = 0>
> > > AND min_index_interval = 128>
> > > AND read_repair_chance = 0.0>
> > > AND speculative_retry = '99PERCENTILE';>
> > >>
> > > CREATE TABLE tims."MESSAGE_HISTORY_STATE" (>
> > > username text,>
> > > date_partition text,>
> > > message_id text,>
> > > jid text,>
> > > state text,>
> > > sent_time timestamp,>
> > > PRIMARY KEY ((username, date_partition), message_id, jid, state)>
> > > ) WITH CLUSTERING ORDER BY (message_id ASC, jid ASC, state ASC)>
> > > AND bloom_filter_fp_chance = 0.01>
> > > AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}>
> > > AND comment = ''>
> > > AND compaction = {'class':>
> > > 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',>
> > > 'max_threshold': '32', 'min_threshold': '4'}>
> > > AND compression = {'chunk_length_in_kb': '64', 'class':>
> > > 'org.apache.cassandra.io.compress.LZ4Compressor'}>
> > > AND crc_check_chance = 1.0>
> > > AND dclocal_read_repair_chance = 0.1>
> > > AND default_time_to_live = 0>
> > > AND gc_grace_seconds = 864000>
> > > AND max_index_interval = 2048>
> > > AND memtable_flush_period_in_ms = 0>
> > > AND min_index_interval = 128>
> > > AND read_repair_chance = 0.0>
> > > AND speculative_retry = '99PERCENTILE';>
> > >>
> >
> > message_history Local write latency: 0.021 ms>
> > message_history_state Local write latency: 11.353 ms>
> >
> > Thanks in advance.>
> >
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

high write latency on a single table

[CPC]

Hi all,

Our cassandra cluster consist of two dc and every dc we have 10 nodes. We
are using DSE 5.1.12 (cassandra 3.11).We have a high local write latency on
a single table. All other tables in our keyspace  have normal latencies
like 0.02 msec,even tables that have more write tps and more data. Below
you can find two table descriptions and their latencies.
message_history_state have high local write latency. This is not node
specific every node have this high local write latency for
message_history_state. Have you ever see such a behavior or any clue why
this could happen?

 CREATE TABLE tims."MESSAGE_HISTORY" (
> username text,
> date_partition text,
> jid text,
> sent_time timestamp,
> message_id text,
> stanza text,
> PRIMARY KEY ((username, date_partition), jid, sent_time, message_id)
> ) WITH CLUSTERING ORDER BY (jid ASC, sent_time DESC, message_id ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'bucket_high': '1.5', 'bucket_low': '0.5', 'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'enabled': 'true', 'max_threshold': '32', 'min_sstable_size': '50',
> 'min_threshold': '4', 'tombstone_compaction_interval': '86400',
> 'tombstone_threshold': '0.2', 'unchecked_tombstone_compaction': 'false'}
> AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 86400
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
>
> CREATE TABLE tims."MESSAGE_HISTORY_STATE" (
> username text,
> date_partition text,
> message_id text,
> jid text,
> state text,
> sent_time timestamp,
> PRIMARY KEY ((username, date_partition), message_id, jid, state)
> ) WITH CLUSTERING ORDER BY (message_id ASC, jid ASC, state ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
>

message_history Local write latency: 0.021 ms
message_history_state Local write latency: 11.353 ms

Thanks in advance.

range repairs multiple dc

[CPC]

Hi All,

I searched over documentation but could not find enough reference regarding
-pr option. In some documentation it says you have to cover all ring in
some places it says you have to run it on every node regardless of you have
multiple dc.

In our case we have three dc (DC1,DC2,DC3) with every DC having 4 nodes and
12 nodes cluster in total. If i run "nodetool repair -pr --full" on very
node in DC1, does it means DC1 is consistent but DC2 and DC3 is not or  DC1
is not consistent at all? Because in our case we added DC3 to our cluster
and will remove DC2 from cluster so i dont care whether DC2 have consistent
data. I dont want to repair DC2.

Also can i run "nodetool repair -pr -full" in parallel? I mean run it at
the same time in each DC or run it more than one node in same DC? Does
-dcpar option making the same thing?

Best Regards...

Re: High IO and poor read performance on 3.11.2 cassandra cluster

[CPC]

Could you decrease chunk_length_in_kb to 16 or 8 and repeat the test.

On Wed, Sep 5, 2018, 5:51 AM wxn...@zjqunshuo.com 
wrote:

> How large is your row? You may meet reading wide row problem.
>
> -Simon
>
> *From:* Laxmikant Upadhyay 
> *Date:* 2018-09-05 01:01
> *To:* user 
> *Subject:* High IO and poor read performance on 3.11.2 cassandra cluster
> We have 3 node cassandra cluster (3.11.2) in single dc.
>
> We have written 450 million records on the table with LCS. The write
> latency is fine.  After write we perform read and update operations.
>
> When we run read+update operations on newly inserted 1 million records (on
> top of 450 m records) then the read latency and io usage is under control.
> However when we perform read+update on old 1 million records which are part
> of 450 million records we observe high read latency (The performance goes
> down by 4 times in comparison 1st case ).  We have not observed major gc
> pauses.
>
> *system information:*
> *cpu core :*  24
> *disc type : *ssd . we are using raid with deadline schedular
> *disk space:*
> df -h :
> Filesystem  Size  Used Avail Use% Mounted on
> /dev/sdb11.9T  393G  1.5T  22% /var/lib/cassandra
> *memory:*
> free -g
>   totalusedfree  shared  buff/cache
>  available
> Mem: 62  30   0   0  32
>   31
> Swap: 8   0   8
>
> ==
>
> *schema*
>
> desc table ks.xyz;
>
> CREATE TABLE ks.xyz (
> key text,
> column1 text,
> value text,
> PRIMARY KEY (key, column1)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (column1 ASC)
> AND bloom_filter_fp_chance = 0.1
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
>
> ==
> Below is some system stats snippet when read operations was running:
>
> *iotop -o * : Observation : the disk read goes up to 5.5 G/s
>
> Total DISK READ :   *3.86 G/s* | Total DISK WRITE :1252.88 K/s
> Actual DISK READ:  * 3.92 G/s* | Actual DISK WRITE:   0.00 B/s
>   TID  PRIO  USER DISK READ  DISK WRITE  SWAPIN IO>COMMAND
> 10715 be/4 cassandr  375.89 M/s   99.79 K/s  0.00 % 29.15 % java
> -Dorg.xerial.snappy.tempdir=/var/tmp
> -Dja~etrics-core-3.1.0.jar:/usr/share/cassandra/lib/
> 10714 be/4 cassandr  358.56 M/s  107.18 K/s  0.00 % 27.06 % java
> -Dorg.xerial.snappy.tempdir=/var/tmp
> -Dja~etrics-core-3.1.0.jar:/usr/share/cassandra/lib/
> 10712 be/4 cassandr  351.86 M/s  147.83 K/s  0.00 % 25.02 % java
> -Dorg.xerial.snappy.tempdir=/var/tmp
> -Dja~etrics-core-3.1.0.jar:/usr/share/cassandra/lib/
> 10718 be/4 cassandr  359.82 M/s  110.87 K/s  0.00 % 24.49 % java
> -Dorg.xerial.snappy.tempdir=/var/tmp
> -Dja~etrics-core-3.1.0.jar:/usr/share/cassandra/lib/
> 10711 be/4 cassandr  333.03 M/s  125.66 K/s  0.00 % 23.37 % java
> -Dorg.xerial.snappy.tempdir=/var/tmp
> -Dja~etrics-core-3.1.0.jar:/usr/share/cassandra/lib/
> 10716 be/4 cassandr  330.80 M/s  103.48 K/s  0.00 % 23.02 % java
> -Dorg.xerial.snappy.tempdir=/var/tmp
> -Dja~etrics-core-3.1.0.jar:/usr/share/cassandra/lib/
> 10717 be/4 cassandr  319.49 M/s  118.27 K/s  0.00 % 22.11 % java
> -Dorg.xerial.snappy.tempdir=/var/tmp
> -Dja~etrics-core-3.1.0.jar:/usr/share/cassandra/lib/
> 10713 be/4 cassandr  300.62 M/s  118.27 K/s  0.00 % 21.65 % java
> -Dorg.xerial.snappy.tempdir=/var/tmp
> -Dja~etrics-core-3.1.0.jar:/usr/share/cassandra/lib/
> 10719 be/4 cassandr  294.98 M/s   81.31 K/s  0.00 % 21.60 % java
> -Dorg.xerial.snappy.tempdir=/var/tmp
> -Dja~etrics-core-3.1.0.jar:/usr/share/cassandra/lib/
> 10720 be/4 cassandr  289.00 M/s   73.92 K/s  0.00 % 21.45 % java
> -Dorg.xerial.snappy.tempdir=/var/tmp
> -Dja~etrics-core-3.1.0.jar:/usr/share/cassandra/lib/
> 10742 be/4 cassandr  240.98 M/s   81.31 K/s  0.00 % 17.68 % java
> -Dorg.xerial.snappy.tempdir=/var/tmp
> -Dja~etrics-core-3.1.0.jar:/usr/share/cassandra/lib/
> 10743 be/4 cassandr  224.43 M/s   36.96 K/s  0.00 % 17.57 % java
> -Dorg.xerial.snappy.tempdir=/var/tmp
> -Dja~etrics-core-3.1.0.jar:/usr/share/cassandra/lib/
> 10744 be/4 cassandr  113.29 M/s   14.78 K/s  0.00 % 10.22 % java
> -Dorg.xerial.snappy.tempdir=/var/tmp
> -Dja~etrics-core-3.1.0.jar:/usr/share/cassandra/lib/
> 10745 be/4 cassandr   61.63 M/s   33.26 K/s  0.00 %  4.20 % 

Re: cassandra getendpoints do not match with tracing

[CPC]

Any ideas? below is getendpoint result for a specific pk

172.16.5.235

172.16.5.229

172.16.5.228

172.16.5.223

172.16.5.234

172.16.5.241
and below is a trace with same pk

 Preparing statement [Native-Transport-Requests-2] | 2018-06-22
16:33:21.118000 | 172.16.5.242 |   6757 | 10.201.165.77



   reading
data from /172.16.5.243 [Native-Transport-Requests-2] | 2018-06-22
16:33:21.119000 | 172.16.5.242 |   7143 | 10.201.165.77


  Sending
READ message to /172.16.5.243 message size 185 bytes
[MessagingService-Outgoing-/172.16.5.243-Small] | 2018-06-22
16:33:21.119000 | 172.16.5.242 |   7511 | 10.201.165.77


speculating read retry on /172.16.5.236 [Native-Transport-Requests-2] |
2018-06-22 16:33:21.126000 | 172.16.5.242 |  14149 | 10.201.165.77


Sending READ message to /172.16.5.236 message size 185 bytes
[MessagingService-Outgoing-/172.16.5.236-Small] | 2018-06-22
16:33:21.126000 | 172.16.5.242 |  14286 | 10.201.165.77
at the end of the tracing

 Submitting range requests on 1 ranges with a concurrency of 1 (5.859375E-4
rows per range expected) [Native-Transport-Requests-1] | 2018-06-22
20:53:20.594000 | 172.16.5.242 |   2764 | 10.201.165.77




  Enqueuing
request to /172.16.5.234 [Native-Transport-Requests-1] | 2018-06-22
20:53:20.594000 | 172.16.5.242 |   2871 | 10.201.165.77


Submitted 1 concurrent range requests [Native-Transport-Requests-1] |
2018-06-22 20:53:20.594000 | 172.16.5.242 |   2939 | 10.201.165.77


Sending RANGE_SLICE message to /172.16.5.234 message size 252 bytes
[MessagingService-Outgoing-/172.16.5.234-Small] | 2018-06-22
20:53:20.594000 | 172.16.5.242 |   2967 | 10.201.165.77


RANGE_SLICE message received from /172.16.5.242 [MessagingService-Incoming-/
172.16.5.242] | 2018-06-22 20:53:20.60 | 172.16.5.234 | 28
| 10.201.165.77


   Executing read on
tims.MESSAGE_HISTORY using index msgididx [ReadStage-1] | 2018-06-22
20:53:20.601000 | 172.16.5.234 |360 | 10.201.165.77



  Executing
single-partition query on MESSAGE_HISTORY.msgididx [ReadStage-1] |
2018-06-22 20:53:20.601000 | 172.16.5.234 |468 | 10.201.165.77



 REQUEST_RESPONSE
message received from /172.16.5.234 [MessagingService-Incoming-/172.16.5.234]
| 2018-06-22 20:53:20.612000 | 172.16.5.242 |  21179 | 10.201.165.77


Processing response from /172.16.5.234 [RequestResponseStage-6] |
2018-06-22 20:53:20.612000 | 172.16.5.242 |  21238 | 10.201.165.77


Request complete | 2018-06-22 20:53:20.612342 | 172.16.5.242 |
21342 | 10.201.165.77

I understand that 234 is an endpoint so it should communicate with it. But
I dont understand why tracing includes 243 and 236 ips?  They are not
endpoints and we are submitting this query over 172.16.5.242.



On Fri, 22 Jun 2018 at 21:43, CPC  wrote:

> Hi all,
>
> Recently  we added some nodes to our cluster. After adding nodes we
> noticed that when we nodetool getendpoints tims "MESSAGE_HISTORY"
> partitionkey1 it reports three nodes per dc with 6 nodes in total which is
> expected since RF is 3. But when we run a query with local_one and tracing
> on , we see some nodes in tracing which is not reported by getendpoints.
> And tracing says that this node which is not reported by getendpoints is
> reading some sstables. Is this node coordinator or there is something wrong?
>

cassandra getendpoints do not match with tracing

[CPC]

Hi all,

Recently  we added some nodes to our cluster. After adding nodes we noticed
that when we nodetool getendpoints tims "MESSAGE_HISTORY"  partitionkey1 it
reports three nodes per dc with 6 nodes in total which is expected since RF
is 3. But when we run a query with local_one and tracing on , we see some
nodes in tracing which is not reported by getendpoints. And tracing says
that this node which is not reported by getendpoints is reading some
sstables. Is this node coordinator or there is something wrong?

Re: DC aware failover

[CPC]

Hi Alex,

Is lost-token-range detection impl finished? Since this feature is more
appealing I want to test it.

Thank you for your help

On Nov 16, 2017 10:35 AM, "Alexander Dejanovski" <a...@thelastpickle.com>
wrote:

Hi,

The policy is used in production at least in my former company.

I can help if you have issues using it.

Cheers,

Le jeu. 16 nov. 2017 à 08:32, CPC <acha...@gmail.com> a écrit :

> Hi,
>
> We want to implement DC aware failover policy. For example if application
> could not reach some part of the ring or if we loose 50% of local DC then
> we want our application automatically to switch other DC. We found this
> project on GitHub  https://github.com/adejanovski/cassandra-dcaware-
> failover but we don't know whether it is stable and used in production.
> Do you know about this project or do you know other projects that provide
> same kind of functionality.
>
> Thanks...
>
-- 
-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

DC aware failover

[CPC]

Hi,

We want to implement DC aware failover policy. For example if application
could not reach some part of the ring or if we loose 50% of local DC then
we want our application automatically to switch other DC. We found this
project on GitHub  https://github.com/adejanovski/cassandra-dcaware-failover
but we don't know whether it is stable and used in production. Do you know
about this project or do you know other projects that provide same kind of
functionality.

Thanks...

Ip restriction for username

[CPC]

Hi,

Is there some method to restrict a user to specific ip range/mask (MySQL
and postgre has this kind of functionality)? I know dse have more advanced
authentication like Kerberos and ldap but I don't know whether those can
provide this functionality.

Thanks

Re: Do not use Cassandra 3.11.0+ or Cassandra 3.0.12+

[CPC]

Hi,

Is this bug fixed in dse 5.1.3? As I understand calling jmx getTombStoneRatio
trigers that bug. We are using opscenter as well and do you have any idea
whether opscenter using/calling this method?

Thanks

On Aug 29, 2017 6:35 AM, "Jeff Jirsa"  wrote:

> I shouldn't actually say I don't think it can happen on 3.0 - I haven't
> seen this happen on 3.0 without some other code change to enable it, but
> like I said, we're still investigating.
>
> --
> Jeff Jirsa
>
>
> > On Aug 28, 2017, at 8:30 PM, Jeff Jirsa  wrote:
> >
> > For what it's worth, I don't think this impacts 3.0 without adding some
> other code change (the reporter of the bug on 3.0 had added custom metrics
> that exposed a concurrency issue).
> >
> > We're looking at it on 3.11. I think 13038 made it far more likely to
> occur, but I think it could have happened pre-13038 as well (would take
> some serious luck with your deletion time distribution though - the
> rounding in 13038 does make it more likely, but the race was already there).
> >
> > --
> > Jeff Jirsa
> >
> >
> >> On Aug 28, 2017, at 8:24 PM, Jay Zhuang 
> wrote:
> >>
> >> We're using 3.0.12+ for a few months and haven't seen the issue like
> >> that. Do we know what could trigger the problem? Or is 3.0.x really
> >> impacted?
> >>
> >> Thanks,
> >> Jay
> >>
> >>> On 8/28/17 6:02 AM, Hannu Kröger wrote:
> >>> Hello,
> >>>
> >>> Current latest Cassandra version (3.11.0, possibly also 3.0.12+) has a
> race
> >>> condition that causes Cassandra to create broken sstables (stats file
> in
> >>> sstables to be precise).
> >>>
> >>> Bug described here:
> >>> https://issues.apache.org/jira/browse/CASSANDRA-13752
> >>>
> >>> This change might be causing it (but not sure):
> >>> https://issues.apache.org/jira/browse/CASSANDRA-13038
> >>>
> >>> Other related issues:
> >>> https://issues.apache.org/jira/browse/CASSANDRA-13718
> >>> https://issues.apache.org/jira/browse/CASSANDRA-13756
> >>>
> >>> I would not recommend using 3.11.0 nor upgrading to 3.0.12 or higher
> before
> >>> this is fixed.
> >>>
> >>> Cheers,
> >>> Hannu
> >>>
> >>
> >> -
> >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> >> For additional commands, e-mail: user-h...@cassandra.apache.org
> >>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: Maximum and recommended storage per node

[CPC]

Hi Kiran,

Is this raw storage size per node or allowed data per node? Can you provide
links or articles about those recommendations?

On 28 July 2017 at 12:45, Kiran mk <coolkiran2...@gmail.com> wrote:

> Recommended is 4TB per node
>
> Best regards,
> Koran.M.K
>
> On 28-Jul-2017 1:57 PM, "CPC" <acha...@gmail.com> wrote:
>
>> Hi all,
>>
>> Is there any recommended and maximum storage per node? In old articles
>>  1tb per node was maximum but is it still apply. Or is it just depends on
>> our latency requirements? Can you share your production experiences?
>>
>> Thank you...
>>
>

Maximum and recommended storage per node

[CPC]

Hi all,

Is there any recommended and maximum storage per node? In old articles  1tb
per node was maximum but is it still apply. Or is it just depends on our
latency requirements? Can you share your production experiences?

Thank you...

Re: C* data modeling for time series

[CPC]

If all of your queries like this(i mean get all devices given a  a time
range) Hadoop would be more appropriate since those are analytical queries.

Anyway, to query such data with spark Cassandra connector  your partition
key could include day and hash of your deviceid as pseudo partition key
column (could be abs(murmur(deviceid)%500) we add this column to distribute
data more evenly) . When you want query a time range you should generate a
rdd of tuple2 with all days that intersect with that date and for each day
your rdd should include 0..500 range. Like:

(20170726,0)
(20170726,1)
.
.
.
(20170726,499)

Then you should join this rdd with your table using joinwithcassandratable
method.

On Jul 26, 2017 4:41 PM, "Junaid Nasir" <jna...@an10.io> wrote:

all devices.
After selecting the data I group them and perform other actions i.e sum,
avg on fields and then display those to compare how devices are doing
compared to each other.

On Wed, Jul 26, 2017 at 5:32 PM, CPC <acha...@gmail.com> wrote:

> Hi Junaid,
>
> Given a time range do you want to take all devices or a specific device?
>
>
> On Jul 26, 2017 3:15 PM, "Junaid Nasir" <jna...@an10.io> wrote:
>
> I have a C* cluster (3 nodes) with some 60gb data (replication factor 2).
> when I started using C* coming from SQL background didn't give much thought
> about modeling the data correctly. so what I did was
>
> CREATE TABLE data ( deviceId int,
> time timestamp,
> field1 text,
> filed2 text,
> field3 text,
> PRIMARY KEY(deviceId, time)) WITH CLUSTERING ORDER BY 
> (time ASC);
>
> but most of the queries I run (using spark and datastax connector)
> compares data of different devices for some time period. for example
>
> SELECT * FROM data WHERE time > '2017-07-01 12:00:00';
>
> from my understanding this runs a full table scan. as shown in spark UI
> (from DAG visualization "Scan org.apache.spark.sql.cassandra
> .CassandraSourceRelation@32bb7d65") meaning C* will read all the data and
> then filter for time. Spark jobs runs for hours even for smaller time
> frames.
>
> what is the right approach for data modeling for such queries?. I want to
> get a general idea of things to look for when modeling such data.
> really appreciate all the help from this community :). if you need any
> extra details please ask me here.
>
> Regards,
> Junaid
>
>
>

Re: C* data modeling for time series

[CPC]

Hi Junaid,

Given a time range do you want to take all devices or a specific device?

On Jul 26, 2017 3:15 PM, "Junaid Nasir"  wrote:

I have a C* cluster (3 nodes) with some 60gb data (replication factor 2).
when I started using C* coming from SQL background didn't give much thought
about modeling the data correctly. so what I did was

CREATE TABLE data ( deviceId int,
time timestamp,
field1 text,
filed2 text,
field3 text,
PRIMARY KEY(deviceId, time)) WITH CLUSTERING ORDER
BY (time ASC);

but most of the queries I run (using spark and datastax connector) compares
data of different devices for some time period. for example

SELECT * FROM data WHERE time > '2017-07-01 12:00:00';

from my understanding this runs a full table scan. as shown in spark UI
(from DAG visualization "Scan org.apache.spark.sql.cassandra.
CassandraSourceRelation@32bb7d65") meaning C* will read all the data and
then filter for time. Spark jobs runs for hours even for smaller time
frames.

what is the right approach for data modeling for such queries?. I want to
get a general idea of things to look for when modeling such data.
really appreciate all the help from this community :). if you need any
extra details please ask me here.

Regards,
Junaid

Re: private interface for interdc messaging

[CPC]

Hi,

setting broadcast adress does not resolve problem. We still saw interdc
traffic like below:
dc1 public => dc2 private

traffic. We we want is:

dc1 private => dc2 private

Any idea? Also could it be related to
https://issues.apache.org/jira/browse/CASSANDRA-12673. Any idea?

On 7 July 2017 at 21:51, CPC <acha...@gmail.com> wrote:

> Thank you Nitan.
>
>
> On Jul 7, 2017 8:59 PM, "Nitan Kainth" <ni...@bamlabs.com> wrote:
>
> Yes. Because that's the ip used for internode communication
>
> Sent from my iPhone
>
> On Jul 7, 2017, at 10:52 AM, CPC <acha...@gmail.com> wrote:
>
> Hi Nitan,
>
> Do you mean setting broadcast_address to private network would suffice?
>
> On 7 July 2017 at 20:45, Nitan Kainth <ni...@bamlabs.com> wrote:
>
>> You can setup broadcast address for the IP on which Nodes will
>> communicate with each other. You network team can setup routing table from
>> source to target.
>> We had similar setup done in one of my previous project where we
>> segregated network between application and C* nodes communication.
>>
>> > On Jul 7, 2017, at 10:28 AM, CPC <acha...@gmail.com> wrote:
>> >
>> > Hi,
>> >
>> > We are building 2 datacenters with each machine have one public(for
>> native client connections) and one for private(internode communication).
>> What we noticed that nodes in one datacenters trying to communicate with
>> other nodes in other dc over their public interfaces.
>> > I mean:
>> > DC1 Node1 public interface -> DC2 Node1 private interface
>> > But what we perefer is:
>> > DC1 Node1 private interface -> DC2 Node1 private interface
>> >
>> > Is there any configuration so a node make interdc connection over its
>> private network?
>> >
>> > Thank you...
>>
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>
>
>

Re: private interface for interdc messaging

[CPC]

Thank you Nitan.

On Jul 7, 2017 8:59 PM, "Nitan Kainth" <ni...@bamlabs.com> wrote:

Yes. Because that's the ip used for internode communication

Sent from my iPhone

On Jul 7, 2017, at 10:52 AM, CPC <acha...@gmail.com> wrote:

Hi Nitan,

Do you mean setting broadcast_address to private network would suffice?

On 7 July 2017 at 20:45, Nitan Kainth <ni...@bamlabs.com> wrote:

> You can setup broadcast address for the IP on which Nodes will communicate
> with each other. You network team can setup routing table from source to
> target.
> We had similar setup done in one of my previous project where we
> segregated network between application and C* nodes communication.
>
> > On Jul 7, 2017, at 10:28 AM, CPC <acha...@gmail.com> wrote:
> >
> > Hi,
> >
> > We are building 2 datacenters with each machine have one public(for
> native client connections) and one for private(internode communication).
> What we noticed that nodes in one datacenters trying to communicate with
> other nodes in other dc over their public interfaces.
> > I mean:
> > DC1 Node1 public interface -> DC2 Node1 private interface
> > But what we perefer is:
> > DC1 Node1 private interface -> DC2 Node1 private interface
> >
> > Is there any configuration so a node make interdc connection over its
> private network?
> >
> > Thank you...
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: private interface for interdc messaging

[CPC]

Hi Nitan,

Do you mean setting broadcast_address to private network would suffice?

On 7 July 2017 at 20:45, Nitan Kainth <ni...@bamlabs.com> wrote:

> You can setup broadcast address for the IP on which Nodes will communicate
> with each other. You network team can setup routing table from source to
> target.
> We had similar setup done in one of my previous project where we
> segregated network between application and C* nodes communication.
>
> > On Jul 7, 2017, at 10:28 AM, CPC <acha...@gmail.com> wrote:
> >
> > Hi,
> >
> > We are building 2 datacenters with each machine have one public(for
> native client connections) and one for private(internode communication).
> What we noticed that nodes in one datacenters trying to communicate with
> other nodes in other dc over their public interfaces.
> > I mean:
> > DC1 Node1 public interface -> DC2 Node1 private interface
> > But what we perefer is:
> > DC1 Node1 private interface -> DC2 Node1 private interface
> >
> > Is there any configuration so a node make interdc connection over its
> private network?
> >
> > Thank you...
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

private interface for interdc messaging

[CPC]

Hi,

We are building 2 datacenters with each machine have one public(for native
client connections) and one for private(internode communication). What we
noticed that nodes in one datacenters trying to communicate with other
nodes in other dc over their public interfaces.
I mean:
DC1 Node1 public interface -> DC2 Node1 private interface
But what we perefer is:
DC1 Node1 private interface -> DC2 Node1 private interface

Is there any configuration so a node make interdc connection over its
private network?

Thank you...