Re: ssl certificate hot reloading test - cassandra 4.1

[Dinesh Joshi]

In addition to what Andy mentioned, I want to point out that for the vast
majority of use-cases, we would like to _avoid_ interruptions when a
certificate is updated so it is by design. If you're dealing with a
situation where you want to ensure that the connections are cycled, you can
follow Andy's advice. It will require automation outside the database that
you might already have. If there is demand, we can consider adding a
feature to slowly cycle the connections so the old SSL context is not used
anymore.

One more thing you should bear in mind is that Cassandra will not load the
new SSL context if it cannot successfully initialize it. This is again by
design to prevent an outage when the updated truststore is corrupted or
could not be read in some way.

thanks,
Dinesh

On Mon, Apr 15, 2024 at 9:45 AM Tolbert, Andy  wrote:

> I should mention, when toggling disablebinary/enablebinary between
> instances, you will probably want to give some time between doing this
> so connections can reestablish, and you will want to verify that the
> connections can actually reestablish.  You also need to be mindful of
> this being disruptive to inflight queries (if your client is
> configured for retries it will probably be fine).  Semantically to
> your applications it should look a lot like a rolling cluster bounce.
>
> Thanks,
> Andy
>
> On Mon, Apr 15, 2024 at 11:39 AM pabbireddy avinash
>  wrote:
> >
> > Thanks Andy for your reply . We will test the scenario you mentioned.
> >
> > Regards
> > Avinash
> >
> > On Mon, Apr 15, 2024 at 11:28 AM, Tolbert, Andy 
> wrote:
> >>
> >> Hi Avinash,
> >>
> >> As far as I understand it, if the underlying keystore/trustore(s)
> >> Cassandra is configured for is updated, this *will not* provoke
> >> Cassandra to interrupt existing connections, it's just that the new
> >> stores will be used for future TLS initialization.
> >>
> >> Via:
> https://cassandra.apache.org/doc/4.1/cassandra/operating/security.html#ssl-certificate-hot-reloading
> >>
> >> > When the files are updated, Cassandra will reload them and use them
> for subsequent connections
> >>
> >> I suppose one could do a rolling disablebinary/enablebinary (if it's
> >> only client connections) after you roll out a keystore/truststore
> >> change as a way of enforcing the existing connections to reestablish.
> >>
> >> Thanks,
> >> Andy
> >>
> >>
> >> On Mon, Apr 15, 2024 at 11:11 AM pabbireddy avinash
> >>  wrote:
> >> >
> >> > Dear Community,
> >> >
> >> > I hope this email finds you well. I am currently testing SSL
> certificate hot reloading on a Cassandra cluster running version 4.1 and
> encountered a situation that requires your expertise.
> >> >
> >> > Here's a summary of the process and issue:
> >> >
> >> > Reloading Process: We reloaded certificates signed by our in-house
> certificate authority into the cluster, which was initially running with
> self-signed certificates. The reload was done node by node.
> >> >
> >> > Truststore and Keystore: The truststore and keystore passwords are
> the same across the cluster.
> >> >
> >> > Unexpected Behavior: Despite the different truststore configurations
> for the self-signed and new CA certificates, we observed no breakdown in
> server-to-server communication during the reload. We did not upload the new
> CA cert into the old truststore.We anticipated interruptions due to the
> differing truststore configurations but did not encounter any.
> >> >
> >> > Post-Reload Changes: After reloading, we updated the cqlshrc file
> with the new CA certificate and key to connect to cqlsh.
> >> >
> >> > server_encryption_options:
> >> >
> >> > internode_encryption: all
> >> >
> >> > keystore: ~/conf/server-keystore.jks
> >> >
> >> > keystore_password: 
> >> >
> >> > truststore: ~/conf/server-truststore.jks
> >> >
> >> > truststore_password: 
> >> >
> >> > protocol: TLS
> >> >
> >> > algorithm: SunX509
> >> >
> >> > store_type: JKS
> >> >
> >> > cipher_suites: [TLS_RSA_WITH_AES_256_CBC_SHA]
> >> >
> >> > require_client_auth: true
> >> >
> >> > client_encryption_options:
> >> >
> >> > enabled: true
> >> >
> >> > keystore: ~/conf/server-keystore.jks
> >> >
> >> > keystore_password: 
> >> >
> >> > require_client_auth: true
> >> >
> >> > truststore: ~/conf/server-truststore.jks
> >> >
> >> > truststore_password: 
> >> >
> >> > protocol: TLS
> >> >
> >> > algorithm: SunX509
> >> >
> >> > store_type: JKS
> >> >
> >> > cipher_suites: [TLS_RSA_WITH_AES_256_CBC_SHA]
> >> >
> >> > Given this situation, I have the following questions:
> >> >
> >> > Could there be a reason for the continuity of server-to-server
> communication despite the different truststores?
> >> > Is there a possibility that the old truststore remains cached even
> after reloading the certificates on a node?
> >> > Have others encountered similar issues, and if so, what were your
> solutions?
> >> >
> >> > Any insights or 

Re: Big Data Question

[Dinesh Joshi]

Minor correction, zero copy streaming aka faster streaming also works for STCS.DineshOn Aug 21, 2023, at 8:01 AM, Jeff Jirsa  wrote:There's a lot of questionable advice scattered in this thread. Set aside most of the guidance like 2TB/node, it's old and super nuanced.If you're bare metal, do what your organization is good at. If you have millions of dollars in SAN equipment and you know how SANs work and fail and get backed up, run on a SAN if your organization knows how to properly operate a SAN. Just make sure you understand it's a single point of failure.If you're in the cloud, EBS is basically the same concept. You can lose EBS in an AZ, just like you can lose SAN in a DC. Persist outside of that. Have backups. Know how to restore them. The reason the "2TB/node" limit was a thing was around time to recover from failure more than anything else. I described this in detail here, in 2015, before faster-streaming in 4.0 was a thing : https://stackoverflow.com/questions/31563447/cassandra-cluster-data-density-data-size-per-node-looking-for-feedback-and/31690279 . With faster streaming, IF you use LCS (so faster streaming works), you can probably go at least 4-5x more dense than before, if you understand how likely your disks are to fail and you can ensure you dont have correlated failures when they age out (that means if you're on bare metal, measuring flash life, and ideally mixing vendors to avoid firmware bugs). You'll still see risks of huge clusters, largely in gossip and schema propagation. Upcoming CEPs address those. 4.0 is better there (with schema, especially) than 3.0 was, but for "max nodes in a cluster", what you're really comparing is "how many gossip speakers and tokens are in the cluster" (which means your vnode settings matter, for things like pending range calculators). Looking at the roadmap, your real question comes down to : - If you expect to use the transactional features in Accord/5.0 to transact across rows/keys, you probably want to keep one cluster- If you dont ever expect to use multi-key transactions, just de-risk by sharding your cluster into many smaller clusters now, with consistent hashing to map keys to clusters, and have 4 clusters of the same smaller size, with whatever node density you think you can do based on your compaction strategy and streaming rate (and disk type). If you have time and budget, create a 3 node cluster with whatever disks you have, fill them, start working on them - expand to 4, treat one as failed and replace it - simulate the operations you'll do at that size. It's expensive to mimic a 500 host cluster, but if you've got budget, try it in AWS and see what happens when you apply your real schema, and then do a schema change.On Mon, Aug 21, 2023 at 7:31 AM Joe Obernberger  wrote:

  

  
  
For our scenario, the goal is to minimize down-time for a single
  (at least initially) data center system.  Data-loss is basically
  unacceptable.  I wouldn't say we have a "rusty slow data center" -
  we can certainly use SSDs and have servers connected via 10G
  copper to a fast back-plane.  For our specific use case with
  Cassandra (lots of writes, small number of reads), the network
  load is usually pretty low.  I suspect that would change if we
  used Kubernetes + central persistent storage.  
  Good discussion.
-Joe

On 8/17/2023 7:37 PM, daemeon reiydelle
  wrote:


  
  
I
  started to respond, then realized I and the other OP posters
  are not thinking the same: What is the business case for
  availability, data los/reload/recoverability? You all argue
  for higher availability and damn the cost. But noone asked
  "can you lose access, for 20 minutes, to a portion of the
  data, 10 times a year, on a 250 node cluster in AWS, if it is
  not lost"? Can you lose access 1-2 times a year for the cost
  of a 500 node cluster holding the same data?


Then
  we can discuss 32/64g JVM and SSD's.

  

  .
  Arthur C. Clarke famously said that "technology
sufficiently advanced is indistinguishable from magic."
Magic is coming, and it's coming for all of us
  

  

  
Daemeon
Reiydelle

  email: daeme...@gmail.com
LI: https://www.linkedin.com/in/daemeonreiydelle/

San Francisco
1.415.501.0198/Skype daemeon.c.m.reiydelle
  

  

  


  
  
  
On Thu, Aug 17, 2023 at
  1:53 PM Joe Obernberger 
 

Re: Reactive DSE Java Driver seeing high response times from Cassandra

[Dinesh Joshi]

Hi Shaurya,

I'm sorry but we don't maintain the DSE driver.

Dinesh

> On Jun 2, 2023, at 1:07 PM, Shaurya Gupta  wrote:
> 
> Hi
> 
> We are seeing high response times on the application side from the Java 
> Reactive DSE driver.
> Cassandra server side metrics show response times of 2 ms but the DSE logs 
> show a response time (95th percentile) of 40ms.
> Following 
> https://docs.datastax.com/en/developer/java-driver/4.2/manual/core/pooling/ 
> it appears that the connections are open for sometime during a load run and 
> they go to 0, then again open-connections goes to the configured value (4) + 
> 1 and then again to 0.
> Why is that ? Load run is at 30K RPM.
> This happens even when I reduce the heartbeat interval to as low as 250 
> milliseconds.
> The CPU consumed by the container is just 35% of available.
> 
> Can someone help with this ?
> 
> Thanks
> -- 
> Shaurya Gupta
> 
> 

Re: Help determining pending compactions

[Dinesh Joshi]

It would be helpful if you could tell us what version of Cassandra you’re using?

Dinesh

> On Oct 30, 2022, at 10:07 AM, Richard Hesse  wrote:
> 
> 
> Hi, I'm hoping to get some help with a vexing issue with one of our 
> keyspaces. During Reaper repair sessions, one keyspace will end up with 
> hanging, non-started compactions. That is, the number of compactions as 
> reported by nodetool compactionstats stays flat and there are no running 
> compactions. Is there a way to determine which tables Cassandra is stuck on 
> here?
> 
> Looking at graphs of pending compactions during Reaper sessions, the number 
> of compactions will shoot up (as expected). The number will work its way 
> down, and sometimes it will stop and plateau at a fixed level. A full 
> compaction will get things going again, but we prefer to avoid those (even 
> with the -s option).
> 
> I realize there are various compaction tuning parameters regarding minimum 
> age, tombstone percentage, etc but I need to know which sstables to look at 
> first before blindly changing values. These are leveled compaction strategy 
> tables FWIW.
> 
> TIA!
> 
> -richard

Re: Using zstd compression on Cassandra 3.x

[Dinesh Joshi]

Is there something preventing you from upgrading to 4.0? It is backward 
compatible with 3.0 so clients don’t need to change.

If you don’t want to absolutely upgrade you can extract the implementation from 
4.0 and use it. I would advise against this path though as zstd implementation 
is nuanced.

Dinesh

> On Sep 12, 2022, at 7:09 PM, Eunsu Kim  wrote:
> 
> Hi all,
> 
> Since zstd compression is a very good compression algorithm, it is available 
> in Cassandra 4.0. Because the overall performance and ratio are excellent
> 
> There is open source available for Cassandra 3.x.
> https://github.com/MatejTymes/cassandra-zstd
> 
> Do you have any experience applying this to production?
> 
> I want to improve performance and disk usage by applying it to a running 
> Cassandra cluster.
> 
> Thanks.

Re: Bootstrap data streaming order

[Dinesh Joshi]

The data is requested asynchronously from peers. There is some logic to select 
the peers however there isn’t a set order for data delivery. Why do you ask?

> 
> On Sep 8, 2022, at 11:35 PM, Marc Hoppins  wrote:
> 
> Hulloa all,
> 
> Can anyone shed light on the order which nodes will deliver data to a new 
> node?  Or point me toward a suitable chart/document?
> 
> Does the new node accept data from each node in turn or simultaneously from 
> multiple nodes?
> 
> Thanks
> 
> Marc

Re: Apache Cassandra Marketing Meeting

[Dinesh Joshi]

This sounds great. Thanks for taking time to do this! Please post back notes of 
the meeting on the dev list so those who couldn't attend are in the loop.

> On Apr 29, 2022, at 3:16 PM, Patrick McFadin  wrote:
> 
> Hi everyone,
> 
> There are several community members working on various marketing efforts to 
> get out the good word on Cassandra. I've been asked by a few to help 
> facilitate a meeting where we can share and coordinate. 
> 
> I've taken the first step by creating a calendar item with a zoom link.
> 
> When: May 5th, 0900 PST - 1700 GMT
> 
> The basic agenda for the first meeting I've collected so far:
> 4.1 release press events
> Cassandra World Party planning
> ApacheCon - Getting the word out
> Here's a handy(but huge) calendar link:
> https://calendar.google.com/event?action=TEMPLATE=NTNsaTZnZ3FoNTZpdjJiY21lbGltamFyc3Ega2w5cHVoZ2s3cXRkdXFhdHRlOHRmZDVtcHNAZw=kl9puhgk7qtduqatte8tfd5mps%40group.calendar.google.com
>  
> 
> 
> Direct link to the zoom: https://datastax.zoom.us/j/390839037 
> 
> 
> Hope to see you there!
> 
> Patrick

Re: sstables changing in snapshots

[Dinesh Joshi]

Cassandra creates hardlinks[1] first and then writes the manifest[2]. But that 
is not the last thing it writes either[3]. This should definitely be 
documented. Could you please open a jira?

[1] 
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1956
[2] 
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1977
[3] 
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1981

> On Mar 22, 2022, at 4:53 PM, James Brown  wrote:
> 
> There are not overlapping snapshots, so I don't think it's a second snapshot. 
> There are overlapping repairs.
> 
>> How does the backup process ensure the snapshot is taken before starting to 
>> upload it ? 
> 
> It just runs nice nodetool ${jmx_args[@]} snapshot -t "$TAG" ${keyspaces[@]}
> 
>> A snapshot is only safe to use after the "manifest.json" file is written.
> 
> Is this true? I don't see this anywhere in the documentation for Cassandra (I 
> would expect it on the Backups page, for example) or in the help of nodetool 
> snapshot. It was my understanding that when the nodetool snapshot process 
> finished, the snapshot was done. If that's wrong, it definitely could be that 
> we're just jumping the gun.
> 
> James Brown
> Infrastructure Architect @ easypost.com
> 
> 
> On 2022-03-22 at 10:38:56, Paul Chandler  wrote:
>> Hi Yifan,
>> 
>> It looks like you are right, I can reproduce this, when creating the second 
>> snapshot the ctime does get updated to the time of the second snapshot.
>> 
>> I guess this is what is causing tar to produce the error.
>> 
>> Paul 
>> 
>>> On 22 Mar 2022, at 17:12, Yifan Cai  wrote:
>>> 
>>> I am wondering if the cause is tarring when creating hardlinks, i.e. 
>>> creating a new snapshot. 
>>> 
>>> A quick experiment on my Mac indicates the file status (ctime) is updated 
>>> when creating hardlink. 
>>> 
>>> ➜ stat -f "Access (atime): %Sa%nModify (mtime): %Sm%nChange (ctime): %Sc" a
>>> Access (atime): Mar 22 10:03:43 2022
>>> Modify (mtime): Mar 22 10:03:43 2022
>>> Change (ctime): Mar 22 10:05:43 2022
>>> 
>>> On Tue, Mar 22, 2022 at 10:01 AM Jeff Jirsa  wrote:
>>> The most useful thing that folks can provide is an indication of what was 
>>> writing to those data files when you were doing backups.
>>> 
>>> It's almost certainly one of:
>>> - Memtable flush 
>>> - Compaction
>>> - Streaming from repair/move/bootstrap
>>> 
>>> If you have logs that indicate compaction starting/finishing with those 
>>> sstables, or memtable flushing those sstables, or if the .log file is 
>>> included in your backup, pasting the contents of that .log file into a 
>>> ticket will make this much easier to debug.
>>> 
>>> 
>>> 
>>> On Tue, Mar 22, 2022 at 9:49 AM Yifan Cai  wrote:
>>> I do not think there is a ticket already. Feel free to create one. 
>>> https://issues.apache.org/jira/projects/CASSANDRA/issues/
>>> 
>>> It would be helpful to provide
>>> 1. The version of the cassandra
>>> 2. The options used for snapshotting 
>>> 
>>> - Yifan
>>> 
>>> On Tue, Mar 22, 2022 at 9:41 AM Paul Chandler  wrote:
>>> Hi all,
>>> 
>>> Was there any further progress made on this? Did a Jira get created?
>>> 
>>> I have been debugging our backup scripts and seem to have found the same 
>>> problem. 
>>> 
>>> As far as I can work out so far, it seems that this happens when a new 
>>> snapshot is created and the old snapshot is being tarred.
>>> 
>>> I get a similar message:
>>> 
>>> /bin/tar: 
>>> var/lib/cassandra/backup/keyspacename/tablename-4eec3b01aba811e896342351775ccc66/snapshots/csbackup_2022-03-22T14\\:04\\:05/nb-523601-big-Data.db:
>>>  file changed as we read it
>>> 
>>> Thanks 
>>> 
>>> Paul 
>>> 
>>> 
>>> 
>>>> On 19 Mar 2022, at 02:41, Dinesh Joshi  wrote:
>>>> 
>>>> Do you have a repro that you can share with us? If so, please file a jira 
>>>> and we'll take a look.
>>>> 
>>>>> On Mar 18, 2022, at 12:15 PM, James Brown  wrote:
>>>>> 
>>>>> This in 4.0.3 after running nodetool snapshot that we're seeing sstables 
>>>>> change, yes.
>>>>> 
>

Re: sstables changing in snapshots

[Dinesh Joshi]

Do you have a repro that you can share with us? If so, please file a jira and 
we'll take a look.

> On Mar 18, 2022, at 12:15 PM, James Brown  wrote:
> 
> This in 4.0.3 after running nodetool snapshot that we're seeing sstables 
> change, yes.
> 
> James Brown
> Infrastructure Architect @ easypost.com 
> 
> On 2022-03-18 at 12:06:00, Jeff Jirsa  > wrote:
>> This is nodetool snapshot yes? 3.11 or 4.0?
>> 
>> In versions prior to 3.0, sstables would be written with -tmp- in the name, 
>> then renamed when complete, so an sstable definitely never changed once it 
>> had the final file name. With the new transaction log mechanism, we use one 
>> name and a transaction log to note what's in flight and what's not, so if 
>> the snapshot system is including sstables being written (from flush, from 
>> compaction, or from streaming), those aren't final and should be skipped.
>> 
>> 
>> 
>> 
>> On Fri, Mar 18, 2022 at 11:46 AM James Brown > > wrote:
>> We use the boring combo of cassandra snapshots + tar to backup our cassandra 
>> nodes; every once in a while, we'll notice tar failing with the following:
>> 
>> tar: 
>> data/addresses/addresses-eb0196100b7d11ec852b1541747d640a/snapshots/backup20220318183708/nb-167-big-Data.db:
>>  file changed as we read it
>> 
>> I find this a bit perplexing; what would cause an sstable inside a snapshot 
>> to change? The only thing I can think of is an incremental repair changing 
>> the "repaired_at" flag on the sstable, but it seems like that should 
>> "un-share" the hardlinked sstable rather than running the risk of mutating a 
>> snapshot.
>> 
>> 
>> James Brown
>> Cassandra admin @ easypost.com 

Re: TLS/SSL overhead

[Dinesh Joshi]

I wish there was an easy answer to this question. Like you pointed out it is 
hardware dependent but software stack plays a big part. For instance, the JVM 
you're running makes a difference too. Cassandra comes with netty and IIRC we 
include tcnative which accelerates TLS. You could also slip Amazon's Corretto 
Crypto Provider into your runtime. I am not suggesting using everything all at 
once but a combination of libraries, runtimes, JVM, OS, cipher suites can make 
a big difference. Therefore it is best to try it out on your stack.

Typically modern hardware has accelerators for common encryption algorithms. If 
the software stack enables you to optimally take advantage of the hardware then 
you could see very little to no impact on latencies.

Cassandra maintains persistent connections therefore the visible impact is on 
connection establishment time (TLS handshake is expensive). Encryption will 
make thundering herd problems worse. You should watch out for those two issues.

Dinesh


> On Feb 5, 2022, at 3:53 AM, onmstester onmstester  wrote:
> 
> Hi, 
> 
> Anyone measured impact of wire encryption using TLS 
> (client_encryption/server_encryption) on cluster latency/throughput? 
> It may be dependent on Hardware or even data model but I already did some 
> sort of measurements and got to 2% for client encryption and 3-5% for client 
> + server encryption and wanted to validate that with community.
> 
> Best Regards
> 
> Sent using Zoho Mail 
> 
> 

Re: Incremental repairs getting stuck a lot

[Dinesh Joshi]

Could you file a jira with the details?

Dinesh

> On Nov 26, 2021, at 2:40 PM, James Brown  wrote:
> 
> 
> We're on 4.0.1 and switched to incremental repairs a couple of months ago. 
> They work fine about 95% of the time, but once in a while a session will get 
> stuck and will have to be cancelled (with `nodetool repair_admin cancel -s 
> `). Typically the session will be in REPAIRING but nothing will 
> actually be happening.
> 
> Output of nodetool repair_admin:
> 
> $ nodetool repair_admin
> id   | state | last activity | 
> coordinator  | participants   
>   
>   
>   
>   
>   
>| participants_wp
> 3a059b10-4ef6-11ec-925f-8f7bcf0ba035 | REPAIRING | 6771 (s)  | 
> /[fd00:ea51:d057:200:1:0:0:8e]:25472 | 
> fd00:ea51:d057:200:1:0:0:8e,fd00:ea51:d057:200:1:0:0:8f,fd00:ea51:d057:200:1:0:0:92,fd00:ea51:d057:100:1:0:0:571,fd00:ea51:d057:100:1:0:0:570,fd00:ea51:d057:200:1:0:0:93,fd00:ea51:d057:100:1:0:0:573,fd00:ea51:d057:200:1:0:0:90,fd00:ea51:d057:200:1:0:0:91,fd00:ea51:d057:100:1:0:0:572,fd00:ea51:d057:100:1:0:0:575,fd00:ea51:d057:100:1:0:0:574,fd00:ea51:d057:200:1:0:0:94,fd00:ea51:d057:100:1:0:0:577,fd00:ea51:d057:200:1:0:0:95,fd00:ea51:d057:100:1:0:0:576
>  | 
> [fd00:ea51:d057:200:1:0:0:8e]:25472,[fd00:ea51:d057:200:1:0:0:8f]:25472,[fd00:ea51:d057:200:1:0:0:92]:25472,[fd00:ea51:d057:100:1:0:0:571]:25472,[fd00:ea51:d057:100:1:0:0:570]:25472,[fd00:ea51:d057:200:1:0:0:93]:25472,[fd00:ea51:d057:100:1:0:0:573]:25472,[fd00:ea51:d057:200:1:0:0:90]:25472,[fd00:ea51:d057:200:1:0:0:91]:25472,[fd00:ea51:d057:100:1:0:0:572]:25472,[fd00:ea51:d057:100:1:0:0:575]:25472,[fd00:ea51:d057:100:1:0:0:574]:25472,[fd00:ea51:d057:200:1:0:0:94]:25472,[fd00:ea51:d057:100:1:0:0:577]:25472,[fd00:ea51:d057:200:1:0:0:95]:25472,[fd00:ea51:d057:100:1:0:0:576]:25472
> 
> Running `jstack` on the coordinator shows two repair threads, both idle:
> 
> "Repair#167:1" #602177 daemon prio=5 os_prio=0 cpu=9.60ms elapsed=57359.81s 
> tid=0x7fa6d1741800 nid=0x18e6c waiting on condition  [0x7fc529f9a000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at jdk.internal.misc.Unsafe.park(java.base@11.0.11/Native Method)
> - parking to wait for  <0x00045ba93a18> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at 
> java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.11/LockSupport.java:234)
> at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.11/AbstractQueuedSynchronizer.java:2123)
> at 
> java.util.concurrent.LinkedBlockingQueue.poll(java.base@11.0.11/LinkedBlockingQueue.java:458)
> at 
> java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.11/ThreadPoolExecutor.java:1053)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.11/ThreadPoolExecutor.java:1114)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.11/ThreadPoolExecutor.java:628)
> at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.lang.Thread.run(java.base@11.0.11/Thread.java:829)
> 
> "Repair#170:1" #654814 daemon prio=5 os_prio=0 cpu=9.62ms elapsed=7369.98s 
> tid=0x7fa6aec09000 nid=0x1a96f waiting on condition  [0x7fc535aae000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at jdk.internal.misc.Unsafe.park(java.base@11.0.11/Native Method)
> - parking to wait for  <0x0004c45bf7d8> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at 
> java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.11/LockSupport.java:234)
> at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.11/AbstractQueuedSynchronizer.java:2123)
> at 
> java.util.concurrent.LinkedBlockingQueue.poll(java.base@11.0.11/LinkedBlockingQueue.java:458)
> at 
> java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.11/ThreadPoolExecutor.java:1053)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.11/ThreadPoolExecutor.java:1114)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.11/ThreadPoolExecutor.java:628)
> at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.lang.Thread.run(java.base@11.0.11/Thread.java:829)
> 
> nodetool netstats says there is nothing happening:
> 
> $ nodetool netstats | head -n 2
> Mode: NORMAL
> Not sending any streams.
> 
> There's nothing interesting in the logs for this repair; the last 

Re: Does Open-Source Cassandra 4 support mutual-TLS ?

[Dinesh Joshi]

> On Sep 22, 2021, at 8:03 AM, Bowen Song  wrote:
> 
> Interesting. The better approach is to match the certificate's common name or 
> SAN to the CQL username. This way the authorisation part, which depends on 
> the CQL username, will continue to work as usual.
> 
Yes, that is one way of deriving an identity. The other "standard" mechanism is 
to use spiffe ids[1] to encode an identity and to derive your 
username/principal from it.

Dinesh

[1] 
https://github.com/spiffe/spiffe/blob/main/standards/X509-SVID.md#2-spiffe-id

> On 22/09/2021 15:40, Dinesh Joshi wrote:
>>> On Sep 22, 2021, at 2:02 AM, Bowen Song >> <mailto:bo...@bso.ng>> wrote:
>>> Out of curiosity, I have two further questions.
>>> 
>>> 1. I know the client can optionally provider a certificate for the TLS 
>>> handshake, but is it possible to require the client to provide a 
>>> certificate?
>>> 
>> 
>> Yep, you can turn on require_client_auth under the client_encyrption_options 
>> in Cassandra YAML to 
>> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L1160 
>> <https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L1160> 
>> This requires the CQL client to provide client certificates.
>> 
>>> 2. Does Cassandra check that the username matches the client certificate? 
>>> E.g. TLS client certificate is issued to "bob", but logging in CQL using 
>>> the username "charol".
>>> 
>> no. The client certificates that you provide should be trusted by the 
>> server's truststore for the connection to succeed. Thats it. AFAIK there 
>> isn't an accepted secure standard to encode a "username" inside a TLS client 
>> certificate that we can use to derive a principal. 
>>> If the answer to Q2 is "no", it would mean that the TLS client certificate 
>>> is used for primary authentication (but not authorisation), and the CQL 
>>> username & password will be used for secondary authentication and 
>>> authorisation. Anyone who has a valid client certificate and private key 
>>> pair can impersonate any CQL user if they know the username and password. 
>>> Depending on the threat model, this may or may not pose a security risk.
>>> 
>> The client certificate can be used to ensure that it is a trusted client. If 
>> a bad actor is able to not only obtain a valid client certificate but also 
>> the private key pair then you have bigger issues :)
>> 
>> In 4.0 the IAuthenticator interface has changed slightly to pass down the 
>> client certificates that the server receives. In theory one can create an 
>> implementation of this interface that can derive an identity from the client 
>> certificates and then use that to perform authentication. This would give 
>> you what you're looking for. If you'd like, you can go as far as ignoring 
>> the username/password passed down from the cilent and only rely on the 
>> identity encoded in the certificate as your source of truth.
>> 
>> Dinesh
>>> 
>>> On 21/09/2021 23:16, Dinesh Joshi wrote:
>>>> It sort of supports it. You still need to send in the username/password 
>>>> credentials along with the client certificate to authenticate. Cassandra 
>>>> will not derive the identity purely from the client certificate.
>>>> 
>>>> Dinesh
>>>> 
>>>>> On Sep 21, 2021, at 11:59 AM, S G  
>>>>> <mailto:sg.online.em...@gmail.com> wrote:
>>>>> 
>>>>> Hello,
>>>>> 
>>>>> Does anyone know if opensource Cassandra support mutual-TLS ?
>>>>> The documentation doesn't conclusively deny or accept the support for the 
>>>>> same.
>>>>> 
>>>>> Thanks !
>> 

Re: Does Open-Source Cassandra 4 support mutual-TLS ?

[Dinesh Joshi]

> On Sep 22, 2021, at 2:02 AM, Bowen Song  wrote:
> Out of curiosity, I have two further questions.
> 
> 1. I know the client can optionally provider a certificate for the TLS 
> handshake, but is it possible to require the client to provide a certificate?
> 

Yep, you can turn on require_client_auth under the client_encyrption_options in 
Cassandra YAML to 
https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L1160 
<https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L1160> This 
requires the CQL client to provide client certificates.

> 2. Does Cassandra check that the username matches the client certificate? 
> E.g. TLS client certificate is issued to "bob", but logging in CQL using the 
> username "charol".
> 
no. The client certificates that you provide should be trusted by the server's 
truststore for the connection to succeed. Thats it. AFAIK there isn't an 
accepted secure standard to encode a "username" inside a TLS client certificate 
that we can use to derive a principal. 
> If the answer to Q2 is "no", it would mean that the TLS client certificate is 
> used for primary authentication (but not authorisation), and the CQL username 
> & password will be used for secondary authentication and authorisation. 
> Anyone who has a valid client certificate and private key pair can 
> impersonate any CQL user if they know the username and password. Depending on 
> the threat model, this may or may not pose a security risk.
> 
The client certificate can be used to ensure that it is a trusted client. If a 
bad actor is able to not only obtain a valid client certificate but also the 
private key pair then you have bigger issues :)

In 4.0 the IAuthenticator interface has changed slightly to pass down the 
client certificates that the server receives. In theory one can create an 
implementation of this interface that can derive an identity from the client 
certificates and then use that to perform authentication. This would give you 
what you're looking for. If you'd like, you can go as far as ignoring the 
username/password passed down from the cilent and only rely on the identity 
encoded in the certificate as your source of truth.

Dinesh
> 
> On 21/09/2021 23:16, Dinesh Joshi wrote:
>> It sort of supports it. You still need to send in the username/password 
>> credentials along with the client certificate to authenticate. Cassandra 
>> will not derive the identity purely from the client certificate.
>> 
>> Dinesh
>> 
>>> On Sep 21, 2021, at 11:59 AM, S G  
>>> <mailto:sg.online.em...@gmail.com> wrote:
>>> 
>>> Hello,
>>> 
>>> Does anyone know if opensource Cassandra support mutual-TLS ?
>>> The documentation doesn't conclusively deny or accept the support for the 
>>> same.
>>> 
>>> Thanks !

Re: Does Open-Source Cassandra 4 support mutual-TLS ?

[Dinesh Joshi]

It sort of supports it. You still need to send in the username/password 
credentials along with the client certificate to authenticate. Cassandra will 
not derive the identity purely from the client certificate.

Dinesh

> On Sep 21, 2021, at 11:59 AM, S G  wrote:
> 
> Hello,
> 
> Does anyone know if opensource Cassandra support mutual-TLS ?
> The documentation doesn't conclusively deny or accept the support for the 
> same.
> 
> Thanks !

Re: 4.0 best feature/fix?

[Dinesh Joshi]

Those of y’all who have difficulty with Java 11, have you asked about it on the 
mailing list or filed a Jira?

Dinesh

> On May 7, 2021, at 8:14 AM, Joe Obernberger  
> wrote:
> 
> 
> I can retry Java 11.
> 
> I am seeing this error a lot - still debugging, but I'll throw it out there - 
> using 4.11.1 driver and a 4 node RC1 cluster.  I'm seeing warning in the 
> cassandra logs about slow queries, but no errors.  This error is client side.
> 
> Caused by: com.datastax.oss.driver.api.core.AllNodesFailedException: All 4 
> node(s) tried for the query failed (showing first 3 nodes, use getAllErrors() 
> for more): Node(endPoint=/172.16.100.39:9042, 
> hostId=93f9cb0f-ea71-4e3d-b62a-f0ea0e888c47, hashCode=345a8431): 
> [java.lang.NullPointerException], Node(endPoint=/172.16.100.36:9042, 
> hostId=d9702f96-256e-45ae-8e12-69a42712be50, hashCode=4c7ac5bb): 
> [java.lang.NullPointerException], Node(endPoint=chaos/172.16.100.37:9042, 
> hostId=08a19658-40be-4e55-8709-812b3d4ac750, hashCode=7ba07f0e): 
> [java.lang.NullPointerException]
> at 
> com.datastax.oss.driver.api.core.AllNodesFailedException.copy(AllNodesFailedException.java:141)
> at 
> com.datastax.oss.driver.internal.core.util.concurrent.CompletableFutures.getUninterruptibly(CompletableFutures.java:149)
> at 
> com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:53)
> at 
> com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:30)
> at 
> com.datastax.oss.driver.internal.core.session.DefaultSession.execute(DefaultSession.java:230)
> at 
> com.datastax.oss.driver.api.core.cql.SyncCqlSession.execute(SyncCqlSession.java:54)
> at 
> com.ngc.helios.heliosingestservice.IngestService.splitOrigData(IngestService.java:55)
> at 
> com.ngc.helios.heliosingestservice.IngestService_ClientProxy.splitOrigData(IngestService_ClientProxy.zig:157)
> at 
> com.ngc.helios.heliosingestservice.IngestResource.splitOrigData(IngestResource.java:27)
> at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at 
> org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:170)
> at 
> org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:130)
> at 
> org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:643)
> at 
> org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:507)
> at 
> org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:457)
> at 
> org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364)
> at 
> org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:459)
> at 
> org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:419)
> at 
> org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:393)
> at 
> org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:68)
> at 
> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:492)
> ... 15 more
> Suppressed: java.lang.NullPointerException
> at 
> com.datastax.oss.protocol.internal.PrimitiveSizes.sizeOfShortBytes(PrimitiveSizes.java:59)
> at 
> com.datastax.oss.protocol.internal.request.Execute$Codec.encodedSize(Execute.java:78)
> at 
> com.datastax.oss.protocol.internal.FrameCodec.encodedBodySize(FrameCodec.java:272)
> at 
> com.datastax.oss.protocol.internal.SegmentBuilder.addFrame(SegmentBuilder.java:75)
> at 
> com.datastax.oss.driver.internal.core.protocol.FrameToSegmentEncoder.write(FrameToSegmentEncoder.java:56)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)
> at 
> io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:304)
> at 
> 

Re: Repairs/compactions on tables with solr indexes

[Dinesh Joshi]

Hi Ayub,

DSE is a DataStax product and this is the Apache Cassandra mailing list. Could 
you reach out to DataStax?

Dinesh

> On Aug 7, 2019, at 11:17 PM, Ayub M  wrote:
> 
> Hello, we are using DSE Search workload with Search and Cass running on same 
> nodes/jvm. 
> 
> 1. When repairs are run, does it initiate rebuilds of solr indexes? Does it 
> rebuild only when any data is repaired?
> 2. How about the compactions, does it trigger any search indexes rebuilds? I 
> guess not, since data is not getting changed, but not sure. Or maybe when it 
> cleans tombstones, how does solr handles deleted data?
> 4. Is it generally a good idea to run both Cass and Search on same node/JVM? 
> Any potential issues which could arise from such a setup or thats a good way 
> to setup since data is colocated on the same nodes.
> 
> Regards,
> Ayub


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: RMI TCP Connection threads

[Dinesh Joshi]

Try obtaining a thread dump. It will help debug. Anything that goes via JMX 
such as nodetool could be responsible for it.

Dinesh

> On Jul 28, 2019, at 10:57 PM, Vlad  > wrote:
> 
> Hi,
> 
> suddenly I noticed that one of three nodes started consume CPU in RMI TCP 
> Connection threads.
> 
> What it could be?
> 
> Thanks.

Re: ApacheCon Europe 2019

[Dinesh Joshi]

Hi Anuj,

I don't think there are any plans to have a Cassandra track at AC EU. I 
encourage you to submit to it regardless.

Thanks,

Dinesh

> On May 13, 2019, at 10:06 AM, Anuj Wadehra  
> wrote:
> 
> Hi,
> 
> Do we have any plans for dedicated Apache Cassandra track or sessions at 
> ApacheCon Berlin in Oct 2019?
> 
> CFP closes 26 May, 2019.
> 
> Thanks
> Anuj Wadehra
> 


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException:

[Dinesh Joshi]

More info with detailed explanation: 
https://www.instaclustr.com/apache-cassandra-scalability-allow-filtering-partition-keys/

Dinesh

> On Apr 16, 2019, at 11:24 PM, Mahesh Daksha  wrote:
> 
> Hi, 
> 
> how much data you are trying to read in the single query? is it large in size 
> or normal text data.
> Looking at the exception it seems the node is unable to deliver data within 
> stipulated time. I have faced similar issue with the response data in huge in 
> size (some binary data). But it was solved as we spread the data across 
> multiple rows.
> 
> Thanks,
> Mahesh Daksha
> 
> On Wed, Apr 17, 2019 at 11:42 AM Krishnanand Khambadkone 
>  wrote:
> Hi,  I have a single instance cassandra server.  I am trying to execute a 
> query with ALLOW FILTERING option.  When I run this same query from cqlsh it 
> runs fine but when I try to execute the query through the java driver it 
> throws this exception.  I have increased all the timeouts in cassandra.yaml 
> file and also included the read timeout option in the SimpleStetement query I 
> am running.  Any idea how I can fix this issue.
> Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: 
> Cassandra timeout during read query at consistency LOCAL_ONE (1 responses 
> were required but only 0 replica responded)
> 
> 


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: 3PP: C++14/17 standard support

[Dinesh Joshi]

The driver is actually maintained by DataStax and not the Cassandra community. 
Please look at the documentation here - https://github.com/datastax/cpp-driver 
 Hopefully someone from their driver 
team can confirm.

Dinesh

> On Mar 29, 2019, at 2:54 AM, Deepti Sharma S  
> wrote:
> 
> Hi Team,
>  
> We are planning to migrate the code-base with C++14/17 standard (GCC C++ 
> compiler), and we are linking Cassandra Version 2.6 c++ client libraries in 
> our module.
> Would you please confirm the support of these libraries with C++14/17 
> standard?
>  
>  
>  
> DEEPTI SHARMA 
> Specialist 
> ITIL 2011 Foundation Certified 
> BDGS, R
> 
> Ericsson
> 3rd Floor, ASF Insignia - Block B Kings Canyon,
> Gwal Pahari, Gurgaon, Haryana 122 003, India
> Phone 0124-6243000 
> deepti.s.sha...@ericsson.com 
> www.ericsson.com 

Re: Disable Truststore CA check for internode_encryption

[Dinesh Joshi]

> On Feb 27, 2019, at 4:20 PM, Jai Bheemsen Rao Dhanwada 
>  wrote:
> 
> I am trying to setup 1-way SSL, basically I am trying to use the SSL options 
> only for the encrypt the data on the wire and trust everyone who is 
> connecting to me.
> 

It *might* be possible with a custom truststore but to the best of my knowledge 
Cassandra doesn't provide what you want out of the box.

Dinesh


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re:

[Dinesh Joshi]

You're sending the unsubscribe request to the wrong email alias. Please send 
email to user-unsubscr...@cassandra.apache.org 


Dinesh

> On Jan 13, 2019, at 9:12 AM, Irtiza Ali  wrote:
> 
> Unsubscribe 
> 
> On Sun, 13 Jan 2019, 22:11 Osman YOZGATLIOĞLU 
> mailto:osman.yozgatlio...@krontech.com> 
> wrote:
> Thank you for clarification.
> 
> Regards
> 
> Osman
> 
> 
> 
> On 13.01.2019 11:24, Jürgen Albersdorfer wrote:
>> Just turn it off. There is no persistent change to the cluster until the 
>> node has finished bootstrap and in Status UN.
>> 
>> Von meinem iPhone gesendet
>> 
>> Am 12.01.2019 um 22:36 schrieb Osman YOZGATLIOĞLU 
>> mailto:osman.yozgatlio...@krontech.com>>:
>> 
>>> Hello,
>>> 
>>> I have one joining node. I decided to change cluster topology and I need to 
>>> move this node to another cluster.
>>> 
>>> How can I decommission joining node? I can't find exact case at google.
>>> 
>>> 
>>> 
>>> Regards,
>>> 
>>> Osman

Re: [ANNOUNCE] StratIO's Lucene plugin fork

[Dinesh Joshi]

Pretty cool!

Dinesh

> On Oct 30, 2018, at 6:31 PM, Jonathan Haddad  wrote:
> 
> Very cool Ben, thanks for sharing!  
> 
>> On Tue, Oct 30, 2018 at 6:14 PM Ben Slater  
>> wrote:
>> For anyone who is interested, we’ve published a blog with some more 
>> background on this and some more detail of our ongoing plans:
>> https://www.instaclustr.com/instaclustr-support-cassandra-lucene-index/
>> 
>> Cheers
>> Ben
>> 
>>> On Fri, 19 Oct 2018 at 09:42 kurt greaves  wrote:
>>> Hi all,
>>> 
>>> We've had confirmation from Stratio that they are no longer maintaining 
>>> their Lucene plugin for Apache Cassandra. We've thus decided to fork the 
>>> plugin to continue maintaining it. At this stage we won't be making any 
>>> additions to the plugin in the short term unless absolutely necessary, and 
>>> as 4.0 nears we'll begin making it compatible with the new major release. 
>>> We plan on taking the existing PR's and issues from the Stratio repository 
>>> and getting them merged/resolved, however this likely won't happen until 
>>> early next year. Having said that, we welcome all contributions and will 
>>> dedicate time to reviewing bugs in the current versions if people lodge 
>>> them and can help.
>>> 
>>> I'll note that this is new ground for us, we don't have much existing 
>>> knowledge of the plugin but are determined to learn. If anyone out there 
>>> has established knowledge about the plugin we'd be grateful for any 
>>> assistance!
>>> 
>>> You can find our fork here: 
>>> https://github.com/instaclustr/cassandra-lucene-index
>>> At the moment, the only difference is that there is a 3.11.3 branch which 
>>> just has some minor changes to dependencies to better support 3.11.3.
>>> 
>>> Cheers,
>>> Kurt
>> -- 
>> Ben Slater
>> Chief Product Officer
>> 
>> 
>> Read our latest technical blog posts here.
>> This email has been sent on behalf of Instaclustr Pty. Limited (Australia) 
>> and Instaclustr Inc (USA).
>> This email and any attachments may contain confidential and legally 
>> privileged information.  If you are not the intended recipient, do not copy 
>> or disclose its content, but please reply to this email immediately and 
>> highlight the error to the sender and then immediately delete the message.
> 
> 
> -- 
> Jon Haddad
> http://www.rustyrazorblade.com
> twitter: rustyrazorblade

Re: nodetool rebuild

[Dinesh Joshi]

It would be helpful to describe your setup - specifically are you using vnodes? 
How is the keyspace setup? One option would be to copy SSTables from the 
replicas and running clean up. That might actually be faster. Since the 
SSTables are compressed you should use a tool that copies without compressing 
the data stream in transit. 

Dinesh

> On Sep 16, 2018, at 2:07 AM, Vitali Dyachuk  wrote:
> 
> Both stream throughput settings are set to 0, meaning that there is no stream 
> throttling on the C* side. Yes, i see high cpu used by STREAM-IN thread, 
> sstables are compressed up to 80%
> What about copying sstables with rsync and then running repair? Probably its 
> not that simple, vut If the data is RF3 so one node should have all the key 
> ranges and repair will not recalculate all the hashes?
> 
> Vitali
> 
>> On Sun, Sep 16, 2018, 02:33 dinesh.jo...@yahoo.com.INVALID 
>>  wrote:
>> Its a long shot but do you have stream_throughput_outbound_megabits_per_sec 
>> or inter_dc_stream_throughput_outbound_megabits_per_sec set to a low value?
>> 
>> You're right in that 3.0 streaming uses 1 thread for incoming and outgoing 
>> connection each per peer. It not only reads the bytes off of the channel but 
>> also deserializes the partitions on that same thread. If you see high CPU 
>> use by STREAM-IN thread then your streaming is CPU bound. In this situation 
>> a powerful CPU will definitely help. Dropping internode compression and 
>> encryption will also help. Are your SSTables compressed?
>> 
>> Dinesh
>> 
>> 
>> On Friday, September 14, 2018, 4:15:28 AM PDT, Vitali Dyachuk 
>>  wrote:
>> 
>> 
>> None of these throttling are helpful for streaming if you have even a 
>> 150-200 Mbit/s bandwidth which is affordable in any cloud. Tweaking network 
>> tcp memory, window size etc does not help, the bottleneck is not the network.
>> These are my findings on how streaming is limited in C* 3.0.*
>> 
>> 1)  Streaming of the particular range which needs to be steamed to the new 
>> node is limited with one 1 thread and no tweaking of cpu affinity etc helps, 
>> probably the powerfull computing VM will help
>> 2) Disabling compression internode_compression and disabling compression per 
>> table in our case helps a bit
>> 3) When streaming has been dropped there is no resume available for the 
>> streaming range so it will start from the beginning 
>> 
>> One of the options could be to create snapshots of sstables on the source 
>> node and just copy all sstable snapshots to new node and then run repair, 
>> data is ~5TB, RF3 ?
>> How is it possible at all to stream data fast to a new node/nodes ? 
>> 
>> Vitali.
>> 
>> On Wed, Sep 12, 2018 at 5:02 PM Surbhi Gupta  
>> wrote:
>> Increase 3 throughput 
>> Compaction throughput 
>> Stream throughput 
>> Interdcstream throughput (if rebuilding from another DC)
>> 
>> Make all of the above to 0 and see if there is any improvement and later set 
>> the value if u can’t leave these values to 0.
>> 
>> On Wed, Sep 12, 2018 at 5:42 AM Vitali Dyachuk  wrote:
>> Hi,
>> I'm currently streaming data with nodetool rebuild on 2 nodes, each node is 
>> streaming from different location. The problem is that it takes ~7 days to 
>> stream 4Tb of data to 1 node, the speed on each side is ~150Mbit/s  so it 
>> should take around 
>> ~2,5 days . Although there are resources on the destnodes and in the source 
>> regions.
>> I've increased stream throughput, but its only affects outbound connections. 
>>  
>> Tested with iperf the bandwidth is 600Mibt/s from both sides. Last week i've 
>> changed the CS from ST to LC because of huge sstables and compaction of them 
>> is still ongoing.
>> How does rebuild command works ? Does it calculate the range then request 
>> the needed sstables from that node and start streaming ? How is it possible 
>> to speed up the streaming ?
>> 
>> Vitali.

Re: Using CDC Feature to Stream C* to Kafka (Design Proposal)

[Dinesh Joshi]

> On Sep 9, 2018, at 6:08 AM, Jonathan Haddad  > wrote:
> 
> There may be some use cases for it.. but I'm not sure what they are.  It 
> might help if you shared the use cases where the extra complexity is 
> required?  When does writing to Cassandra which then dedupes and writes to 
> Kafka a preferred design then using Kafka and simply writing to Cassandra?

From the reading of the proposal, it seems bring functionality similar to 
MySQL's binlog to Kafka connector. This is useful for many applications that 
want to be notified when certain (or any) rows change in the database primarily 
for a event driven application architecture.

Implementing this in the database layer means there is a standard approach to 
getting a change notification stream. Downstream subscribers can then decide 
which notifications to act on.

LinkedIn's databus is similar in functionality - 
https://github.com/linkedin/databus  
However it is for heterogenous datastores.

> On Thu, Sep 6, 2018 at 1:53 PM Joy Gao  > wrote:
> 
> 
> We have a WIP design doc 
>  that goes over 
> this idea in details.
> 
> We haven't sort out all the edge cases yet, but would love to get some 
> feedback from the community on the general feasibility of this approach. Any 
> ideas/concerns/questions would be helpful to us. Thanks!
> 

Interesting idea. I did go over the proposal briefly. I concur with Jon about 
adding more use-cases to clarify this feature's potential use-cases.

Dinesh

Re: bigger data density with Cassandra 4.0?

[Dinesh Joshi]

Although the extent of benefits depend on the specific use case, the cluster 
size is definitely not a limiting factor.

Dinesh

> On Aug 27, 2018, at 5:05 AM, kurt greaves  wrote:
> 
> I believe there are caveats that it will only really help if you're not using 
> vnodes, or you have a very small cluster, and also internode encryption is 
> not enabled. Alternatively if you're using JBOD vnodes will be marginally 
> better, but JBOD is not a great idea (and doesn't guarantee a massive 
> improvement).
> 
>> On 27 August 2018 at 15:46, dinesh.jo...@yahoo.com.INVALID 
>>  wrote:
>> Yes, this feature will help with operating nodes with higher data density.
>> 
>> Dinesh
>> 
>> 
>> On Saturday, August 25, 2018, 9:01:27 PM PDT, onmstester onmstester 
>>  wrote:
>> 
>> 
>> I've noticed this new feature of 4.0:
>> Streaming optimizations 
>> (https://cassandra.apache.org/blog/2018/08/07/faster_streaming_in_cassandra.html)
>> Is this mean that we could have much more data density with Cassandra 4.0 
>> (less problems than 3.X)? I mean > 10 TB of data on each node without 
>> worrying about node join/remove?
>> This is something needed for Write-Heavy applications that do not read a 
>> lot. When you have like 2 TB of data per day and need to keep it for 6 
>> month, it would be waste of money to purchase 180 servers (even Commodity or 
>> Cloud). 
>> IMHO, even if 4.0 fix problem with streaming/joining a new node, still 
>> Compaction is another evil for a big node, but we could tolerate that somehow
>> 
>> Sent using Zoho Mail
>> 
>> 
>> 
> 

Re: How to register cassandra custom codecs in spark? (https://github.com/datastax/spark-cassandra-connector/issues/1173)

[Dinesh Joshi]

Hi Revanth,
I took a quick look and don't think you can override existing mappings. You 
should see a warning like this -
18/04/15 19:50:27 WARN CodecRegistry: Ignoring codec CustomTSTypeCodec [date 
<-> java.lang.Long] because it collides with previously registered codec 
CustomTSTypeCodec [date <-> java.lang.Long]
The issue is that CodecRegistry does not allow you to override existing type 
mappings[1][2]. You can however register a codec that converts between a 
CqlType and your custom class.
Dinesh
[1] 
https://docs.datastax.com/en/developer/java-driver/3.1/manual/custom_codecs/[2] 
https://github.com/datastax/java-driver/blob/3.x/driver-core/src/main/java/com/datastax/driver/core/CodecRegistry.java#L357




 

On Sunday, April 15, 2018, 1:06:53 AM PDT, Revanth Reddy 
<revanthkuma...@gmail.com> wrote:  
 
 Hi Dinesh,
Thanks a lot for your response . I had tried with the existing codec 
(SimpleTimestampCodec) ,but it says the codec has been registered but it is not 
converting the timestamp column to long . also i don't observe any error.
Below are the steps that i had followed and also attached the code that i had 
used:
I had used this property to set the codec in spark 
config("spark.cassandra.connection.factory", 
"com.rasa.devops.cassandra.export.CustomConnectionFactory")

Since there is no direct way to refer the custom codec in spark , i had 
followed the above approach.
I had used the following dependencies :
               com.datastax.spark 
spark-cassandra-connector_2.11 
2.0.1   
com.datastax.cassandra 
cassandra-driver-extras 3.1.4 
  joda-time 
joda-time 2.9.1 
Cassandra table used for testing :

CREATE TABLE dev.my_table (

    partition_key text PRIMARY KEY,

    some_timestamp timestamp

);






cqlsh:dev> select * from my_table;




 partition_key | some_timestamp

---+--

           key | 2015-12-25 18:30:00+

           foo | 2018-04-11 06:22:29+









Thanks & RegardsS.Revanth kumar reddyhttps://github.com/Re1tReddy/
http://34.230.100.5:8080/crazyusers/login










    










                                                                                
           

On Sun, Apr 15, 2018 at 11:36 AM, Dinesh Joshi <dinesh.jo...@yahoo.com> wrote:

Hi Revanth,
How do you register the custom codec? Do you get any errors? Have you tried 
using a pre-existing codec? It would be helpful if you can give more 
information.
DineshOn Saturday, April 14, 2018, 7:29:30 PM PDT, Revanth Reddy 
<revanthkuma...@gmail.com> wrote:  
 
 

Hi Team ,
I want to write a custom cassandra codec and i want to use that codec in my 
spark application while reading the data from cassandra table .
Basically the custom codecs are used to convert one column type to another 
while reading from cassandra. for example i have a timestamp column in 
cassandra table ,but i want to read it as long and save as parquet file to HDFS 
, then this cutom codecs will be usefull instead of doing an additional 
transformation in the spark code.
I want to know how to set/register this custom cassandra codecs in spark ?
 I had tried with solution provided in the below link but it does not work , 
any ideas around this will be helpfull.
https://stackoverflow.com/ questions/40363611/adding- custom-codec-to- 
cassandraconnector/49750791? noredirect=1#comment86515847_ 49750791



Thanks & RegardsS.Revanth kumar reddyhttps://github.com/Re1tReddy/
http://34.230.100.5:8080/ crazyusers/login










    










                                                                                
           
  

  

Re: How to register cassandra custom codecs in spark? (https://github.com/datastax/spark-cassandra-connector/issues/1173)

[Dinesh Joshi]

Hi Revanth,
How do you register the custom codec? Do you get any errors? Have you tried 
using a pre-existing codec? It would be helpful if you can give more 
information.
DineshOn Saturday, April 14, 2018, 7:29:30 PM PDT, Revanth Reddy 
 wrote:  
 
 

Hi Team ,
I want to write a custom cassandra codec and i want to use that codec in my 
spark application while reading the data from cassandra table .
Basically the custom codecs are used to convert one column type to another 
while reading from cassandra. for example i have a timestamp column in 
cassandra table ,but i want to read it as long and save as parquet file to HDFS 
, then this cutom codecs will be usefull instead of doing an additional 
transformation in the spark code.
I want to know how to set/register this custom cassandra codecs in spark ?
 I had tried with solution provided in the below link but it does not work , 
any ideas around this will be helpfull.
https://stackoverflow.com/questions/40363611/adding-custom-codec-to-cassandraconnector/49750791?noredirect=1#comment86515847_49750791



Thanks & RegardsS.Revanth kumar reddyhttps://github.com/Re1tReddy/
http://34.230.100.5:8080/crazyusers/login










    










                                                                                
           
  

Re: RE: RE: What versions should the documentation support now?

[Dinesh Joshi]

trunk is the next release which is 4.0. You won't find a branch named 4.0 yet.
Dinesh 

On Tuesday, March 13, 2018, 11:39:44 PM PDT, Kenneth Brotman 
<kenbrot...@yahoo.com> wrote:  
 
 #yiv3841634821 #yiv3841634821 -- _filtered #yiv3841634821 
{font-family:Helvetica;panose-1:2 11 6 4 2 2 2 2 2 4;} _filtered #yiv3841634821 
{font-family:Helvetica;panose-1:2 11 6 4 2 2 2 2 2 4;} _filtered #yiv3841634821 
{font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;} _filtered #yiv3841634821 
{font-family:Tahoma;panose-1:2 11 6 4 3 5 4 4 2 4;} _filtered #yiv3841634821 
{panose-1:2 11 6 9 4 5 4 2 2 4;}#yiv3841634821 #yiv3841634821 
p.yiv3841634821MsoNormal, #yiv3841634821 li.yiv3841634821MsoNormal, 
#yiv3841634821 div.yiv3841634821MsoNormal 
{margin:0in;margin-bottom:.0001pt;font-size:12.0pt;}#yiv3841634821 a:link, 
#yiv3841634821 span.yiv3841634821MsoHyperlink 
{color:blue;text-decoration:underline;}#yiv3841634821 a:visited, #yiv3841634821 
span.yiv3841634821MsoHyperlinkFollowed 
{color:purple;text-decoration:underline;}#yiv3841634821 
p.yiv3841634821MsoAcetate, #yiv3841634821 li.yiv3841634821MsoAcetate, 
#yiv3841634821 div.yiv3841634821MsoAcetate 
{margin:0in;margin-bottom:.0001pt;font-size:8.0pt;}#yiv3841634821 
p.yiv3841634821msonormal, #yiv3841634821 li.yiv3841634821msonormal, 
#yiv3841634821 div.yiv3841634821msonormal 
{margin-right:0in;margin-left:0in;font-size:12.0pt;}#yiv3841634821 
span.yiv3841634821msohyperlink {}#yiv3841634821 
span.yiv3841634821msohyperlinkfollowed {}#yiv3841634821 
span.yiv3841634821emailstyle17 {}#yiv3841634821 p.yiv3841634821msonormal1, 
#yiv3841634821 li.yiv3841634821msonormal1, #yiv3841634821 
div.yiv3841634821msonormal1 
{margin:0in;margin-bottom:.0001pt;font-size:12.0pt;}#yiv3841634821 
span.yiv3841634821msohyperlink1 
{color:blue;text-decoration:underline;}#yiv3841634821 
span.yiv3841634821msohyperlinkfollowed1 
{color:purple;text-decoration:underline;}#yiv3841634821 
span.yiv3841634821emailstyle171 {color:#1F497D;}#yiv3841634821 
span.yiv3841634821BalloonTextChar {}#yiv3841634821 
span.yiv3841634821EmailStyle27 {color:#1F497D;}#yiv3841634821 
.yiv3841634821MsoChpDefault {font-size:10.0pt;} _filtered #yiv3841634821 
{margin:1.0in 1.0in 1.0in 1.0in;}#yiv3841634821 div.yiv3841634821WordSection1 
{}#yiv3841634821 
I show a 3.0 and a 3.11 branch but no 4.0.  I’m at 
https://github.com/apache/cassandra .    

  

  

From: Dinesh Joshi [mailto:dinesh.jo...@yahoo.com.INVALID] 
Sent: Tuesday, March 13, 2018 11:30 PM
To: user@cassandra.apache.org
Subject: Re: RE: What versions should the documentation support now?

  

Kenneth,

  

The only 4.x docs should go in trunk. If you would like to contribute docs to 
the 2.x and/or 3.x releases, please make pull requests against branches for 
those versions.

  

During normal development process, the docs should be updated in trunk. When a 
release is cut from trunk, any further fixes to the docs pertaining to that 
release should go into that branch. This is in principle the same process that 
the code follows. So the docs will live with their respective branches. You 
should not put the documentation for older releases in trunk because it will 
end up confusing the user.

  

It looks like the in-tree docs were introduced in 4.x. They seem to also be 
present in the 3.11 branch. If you're inclined, you might back port them to the 
older 3.x & 2.x releases and update them.

  

Personally, I think focusing on making the 4.x docs awesome is a better use of 
your time.

  

Thanks,

  

Dinesh

  

  

On Tuesday, March 13, 2018, 11:03:04 PM PDT, Kenneth Brotman 
<kenbrot...@yahoo.com.INVALID> wrote: 

  

  

I made sub directories “2_x” and “3_x” under docs and put a copy of the doc in 
each.  No links were changed yet.  We can work on the files first and discuss 
how we want to change the template and links.  I did the pull request already.

 

Kenneth Brotman

 

From: Jonathan Haddad [mailto:j...@jonhaddad.com] 
Sent: Tuesday, March 13, 2018 6:19 PM
To: user@cassandra.apache.org
Subject: Re: What versions should the documentation support now?

 

Yes, I agree, we should host versioned docs.  I don't think anyone is against 
it, it's a matter of someone having the time to do it.

 

On Tue, Mar 13, 2018 at 6:14 PM kurt greaves <k...@instaclustr.com> wrote:



I’ve never heard of anyone shipping docs for multiple versions, I don’t know 
why we’d do that.  You can get the docs for any version you need by downloading 
C*, the docs are included.  I’m a firm -1 on changing that process.


We should still host versioned docs on the website however. Either that or we 
specify "since version x" for each component in the docs with notes on 
behaviour.

​

  

Re: RE: What versions should the documentation support now?

[Dinesh Joshi]

Kenneth,
The only 4.x docs should go in trunk. If you would like to contribute docs to 
the 2.x and/or 3.x releases, please make pull requests against branches for 
those versions.
During normal development process, the docs should be updated in trunk. When a 
release is cut from trunk, any further fixes to the docs pertaining to that 
release should go into that branch. This is in principle the same process that 
the code follows. So the docs will live with their respective branches. You 
should not put the documentation for older releases in trunk because it will 
end up confusing the user.
It looks like the in-tree docs were introduced in 4.x. They seem to also be 
present in the 3.11 branch. If you're inclined, you might back port them to the 
older 3.x & 2.x releases and update them.
Personally, I think focusing on making the 4.x docs awesome is a better use of 
your time.
Thanks,
Dinesh 

On Tuesday, March 13, 2018, 11:03:04 PM PDT, Kenneth Brotman 
 wrote:  
 
 #yiv9726083586 #yiv9726083586 -- _filtered #yiv9726083586 
{font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;} _filtered #yiv9726083586 
{font-family:Tahoma;panose-1:2 11 6 4 3 5 4 4 2 4;}#yiv9726083586 
#yiv9726083586 p.yiv9726083586MsoNormal, #yiv9726083586 
li.yiv9726083586MsoNormal, #yiv9726083586 div.yiv9726083586MsoNormal 
{margin:0in;margin-bottom:.0001pt;font-size:12.0pt;}#yiv9726083586 a:link, 
#yiv9726083586 span.yiv9726083586MsoHyperlink 
{color:blue;text-decoration:underline;}#yiv9726083586 a:visited, #yiv9726083586 
span.yiv9726083586MsoHyperlinkFollowed 
{color:purple;text-decoration:underline;}#yiv9726083586 
span.yiv9726083586EmailStyle17 {color:#1F497D;}#yiv9726083586 
.yiv9726083586MsoChpDefault {} _filtered #yiv9726083586 {margin:1.0in 1.0in 
1.0in 1.0in;}#yiv9726083586 div.yiv9726083586WordSection1 {}#yiv9726083586 
I made sub directories “2_x” and “3_x” under docs and put a copy of the doc in 
each.  No links were changed yet.  We can work on the files first and discuss 
how we want to change the template and links.  I did the pull request already.

  

Kenneth Brotman

  

From: Jonathan Haddad [mailto:j...@jonhaddad.com] 
Sent: Tuesday, March 13, 2018 6:19 PM
To: user@cassandra.apache.org
Subject: Re: What versions should the documentation support now?

  

Yes, I agree, we should host versioned docs.  I don't think anyone is against 
it, it's a matter of someone having the time to do it.

  

On Tue, Mar 13, 2018 at 6:14 PM kurt greaves  wrote:



I’ve never heard of anyone shipping docs for multiple versions, I don’t know 
why we’d do that.  You can get the docs for any version you need by downloading 
C*, the docs are included.  I’m a firm -1 on changing that process.


We should still host versioned docs on the website however. Either that or we 
specify "since version x" for each component in the docs with notes on 
behaviour.

​

  

Re: Fast Writes to Cassandra Failing Through Python Script

[Dinesh Joshi]

What does your schema look like? Are you seeing any warnings or errors in the 
server log?
Dinesh 

On Tuesday, March 13, 2018, 3:43:33 PM PDT, Faraz Mateen  
wrote:  
 
 Hi everyone,
I seem to have hit a problem in which writing to cassandra through a python 
script fails and also occasionally causes cassandra node to crash. Here are the 
details of my problem.
I have a python based streaming application that reads data from kafka at a 
high rate and pushes it to cassandra through datastax's cassandra driver for 
python. My cassandra setup consists of 3 nodes and a replication factor of 2. 
Problem is that my python application crashes after writing ~12000 records with 
the following error:

Exception: Error from server: code=1100 [Coordinator node timed out waiting for 
replica nodes' responses] message="Operation timed out - received only 0 
responses." info={'received_responses':
 0, 'consistency': 'LOCAL_ONE', 'required_responses': 1}Sometimes the  python 
application crashes with this traceback:

cassandra.OperationTimedOut: errors={'10.128.1.1': 'Client request timeout. See 
Session.execute[_async](timeou t)'}, last_host=10.128.1.1With the error above, 
one of the cassandra node crashes as well. When I look at cassandra system logs 
(/var/log/cassandra/system.log ), I see the following exception:

https://gist.github.com/ farazmateen/ e7aa5749f963ad2293f8be0ca1ccdc 22/ 
e3fd274af32c20eb9f534849a31734 dcd33745b4

According to the suggestion in post linked below, I have set my JVM Heap size 
to 8GB but the problem still persists.:
https://dzone.com/articles/dia gnosing-and-fixing-cassandra- timeouts
Cluster:
   
   - Cassandra version 3.9
   - 3 nodes, with 8 cores and 30GB of RAM each.
   - Keyspace has a replication factor of 2.
   - Write consistency is LOCAL_ONE
   - MAX HEAP SIZE is set to 8GB.
Any help will be greatly appreciated.
--
Faraz
  

Re: overhead of empty tables

[Dinesh Joshi]

Each table in a keyspace is stored as a separate directory in the data 
directory. If you many tables you'll have a lot of files. Some file systems 
have issues dealing with a lot of files in a single directory. Other than that, 
there will likely be some book keeping overhead within the Cassandra process. 
How many tables are we talking about here?

Here's more information about it: 
https://docs.datastax.com/en/cassandra/3.0/cassandra/dml/dmlHowDataWritten.html
Dinesh 

On Thursday, February 15, 2018, 3:34:49 PM PST, Alaa Zubaidi (PDF) 
 wrote:  
 
 Is there any overhead if my keyspace contains many empty tables?ThanksAlaa
This message may contain confidential and privileged information. If it has 
been sent to you in error, please reply to advise the sender of the error and 
then immediately permanently delete it and all attachments to it from your 
systems. If you are not the intended recipient, do not read, copy, disclose or 
otherwise use this message or any attachments to it. The sender disclaims any 
liability for such unauthorized use. PLEASE NOTE that all incoming e-mails sent 
to PDF e-mail accounts will be archived and may be scanned by us and/or by 
external service providers to detect and prevent threats to our systems, 
investigate illegal or inappropriate behavior, and/or eliminate unsolicited 
promotional e-mails (“spam”). If you have any concerns about this process, 
please contact us at legal.departm...@pdf.com.