Re: Restricting data access at column and/or row level

[Nitan Kainth]

Thanks Stefan.What is tentative release time?Regards,NitanCell: 510 449 9629On Oct 3, 2023, at 7:26 PM, Miklosovic, Stefan  wrote:Hi,columns can be restricted per user by Dynamic Data Masking (will be in 5.0).https://cassandra.apache.org/doc/trunk/cassandra/developing/cql/dynamic_data_masking.htmlI am not sure about specific rows. To my knowledge I do not think that is possible.From: Nitan Kainth Sent: Tuesday, October 3, 2023 23:31To: cassandraSubject: Restricting data access  at column and/or row levelYou don't often get email from nitankai...@gmail.com. Learn why this is importantNetApp Security WARNING: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe.Hi Team,I have a requirement to grant select privileges on a table to some user restricting few columns and few rows. Something like this:c1 | c2 | c3 | c4+++  5 |  1 |  5 |  110 |  1 |  1 |  1  1 |  1 |  1 |  1  8 |  1 |  1 |  1  2 |  1 |  1 |  1  4 |  1 |  1 |  1  7 |  5 |  1 |  1  6 |  1 |  1 |  1  9 |  1 |  1 |  1  3 |  4 |  1 |  1User1 should be able to see only c1 and c2:c1 | c2+  5 |  110 |  1  1 |  1  8 |  1  2 |  1  4 |  1  7 |  5  6 |  1  9 |  1  3 |  4User2 should be able to see only c1=5:c1 | c2+  5 |  1I created Materialized view but unfortunately, it can’t work without granting permission on base table.Regards,NitanCell: 510 449 9629

Restricting data access at column and/or row level

[Nitan Kainth]

Hi Team,

I have a requirement to grant select privileges on a table to some user 
restricting few columns and few rows. Something like this:

c1 | c2 | c3 | c4
+++
  5 |  1 |  5 |  1
10 |  1 |  1 |  1
  1 |  1 |  1 |  1
  8 |  1 |  1 |  1
  2 |  1 |  1 |  1
  4 |  1 |  1 |  1
  7 |  5 |  1 |  1
  6 |  1 |  1 |  1
  9 |  1 |  1 |  1
  3 |  4 |  1 |  1

User1 should be able to see only c1 and c2:
c1 | c2
+
  5 |  1
10 |  1
  1 |  1
  8 |  1
  2 |  1
  4 |  1
  7 |  5
  6 |  1
  9 |  1
  3 |  4

User2 should be able to see only c1=5:
c1 | c2
+
  5 |  1

I created Materialized view but unfortunately, it can’t work without granting 
permission on base table.


Regards,
Nitan
Cell: 510 449 9629

Re: Export as csv

[Nitan Kainth]

Use copy command or dsbulk


Regards,
Nitan K.
510-449-9629


From: Arvinder Dhillon 
Date: Thursday, July 21, 2022 at 12:52 PM
To: user@cassandra.apache.org 
Subject: Export as csv
What tool do I use to dump a columns of a table having 40K partitions into a 
csv file?
I tried caqsh with CAPTURE, it stuck after 100 rows and needs "enter" to dump 
next 100, even with allow filtering.

Thanks,
Arvi

Re: Corrupt SSTable

[Nitan Kainth]

If you are not deleting or updating data then it should be safe to use 2nd 
approach. 

Regards,
Nitan
Cell: 510 449 9629

> On Aug 13, 2020, at 11:48 AM, Pushpendra Rajpoot 
>  wrote:
> 
> 
> Hi,
> 
> I have a cluster of 2 DC, each DC has 5 nodes in production. This cluster is 
> based on active-passive model i.e. application is writing data on one DC 
> (Active) & it's replicated to other DC (Passive).
> 
> My Passive DC has corrupt sstables (3 nodes out of 5 nodes) whereas there are 
> no corrupt sstables on any node in Active DC.
> 
> Now, I am planning to do the following activity on affected nodes in Passive 
> DC:
> 1. delete keyspace having corrupt sstables
> 2. run 'nodetool rebuild'
> 
> DO you see any problem in the above approach ?
> 
> Another approach which I am considering as 'Second Option' is as given below:
> 1. Stop the affected node
> 2. Remove all corrupt sstables from the affected node
> 3. Start the affected node
> 4. Run repair on the affected node
> 
> In my case, data is not written to Passive DC and has corrupt sstables.
> 
> Does the 2nd approach lead to data resurrection ?
> 
> I found in one of the threads in 'Mailing List' that data can ressurect if I 
> run 'nodetool repair'. Because of this reason, I considered this as a 
> secondary approach.
> 
> Do you have any other approach to fix this problem OR I can go with one of 
> the above approaches?
> 
> Regards,
> Pushpendra

Re: Is deleting live sstable safe in this scenario?

[Nitan Kainth]

Yeah, I meant the down node can’t participate in repairs


Regards,
Nitan
Cell: 510 449 9629

> On May 27, 2020, at 2:09 PM, Leon Zaruvinsky  wrote:
> 
> 
> Yep, Jeff is right, the intention would be to run a repair limited to the 
> available nodes.
> 
>> On Wed, May 27, 2020 at 2:59 PM Jeff Jirsa  wrote:
>> The "-hosts " flag tells cassandra to only compare trees/run repair on the 
>> hosts you specify, so if you have 3 replicas, but 1 replica is down, you can 
>> provide -hosts with the other two, and it will make sure those two are in 
>> sync (via merkle trees, etc), but ignore the third.
>> 
>> 
>> 
>>> On Wed, May 27, 2020 at 10:45 AM Nitan Kainth  wrote:
>>> Jeff,
>>> 
>>> If Cassandra is down how will it generate merkle tree to compare?
>>> 
>>> 
>>> Regards,
>>> Nitan
>>> Cell: 510 449 9629
>>> 
>>>>> On May 27, 2020, at 11:15 AM, Jeff Jirsa  wrote:
>>>>> 
>>>> 
>>>> You definitely can repair with a node down by passing `-hosts 
>>>> specific_hosts`
>>>> 
>>>>> On Wed, May 27, 2020 at 9:06 AM Nitan Kainth  
>>>>> wrote:
>>>>> I didn't get you Leon,
>>>>> 
>>>>> But, the simple thing is just to follow the steps and you will be fine. 
>>>>> You can't run the repair if the node is down.
>>>>> 
>>>>>> On Wed, May 27, 2020 at 10:34 AM Leon Zaruvinsky 
>>>>>>  wrote:
>>>>>> Hey Jeff/Nitan,
>>>>>> 
>>>>>> 1) this concern should not be a problem if the repair happens before the 
>>>>>> corrupted node is brought back online, right?
>>>>>> 2) in this case, is option (3) equivalent to replacing the node? where 
>>>>>> we repair the two live nodes and then bring up the third node with no 
>>>>>> data
>>>>>> 
>>>>>> Leon
>>>>>> 
>>>>>>> On Tue, May 26, 2020 at 10:11 PM Jeff Jirsa  wrote:
>>>>>>> There’s two problems with this approach if you need strict correctness 
>>>>>>> 
>>>>>>> 1) after you delete the sstable and before you repair you’ll violate 
>>>>>>> consistency, so you’ll potentially serve incorrect data for a while
>>>>>>> 
>>>>>>> 2) The sstable May have a tombstone past gc grace that’s shadowing data 
>>>>>>> in another sstable that’s not corrupt and deleting it may resurrect 
>>>>>>> that deleted data. 
>>>>>>> 
>>>>>>> The only strictly safe thing to do here, unfortunately, is to treat the 
>>>>>>> host as failed and rebuild it from it’s neighbors (and again being 
>>>>>>> pedantic here, that means stop the host, while it’s stopped repair the 
>>>>>>> surviving replicas, then bootstrap a replacement on top of the same 
>>>>>>> tokens)
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> > On May 26, 2020, at 4:46 PM, Leon Zaruvinsky 
>>>>>>> >  wrote:
>>>>>>> > 
>>>>>>> > 
>>>>>>> > Hi all,
>>>>>>> > 
>>>>>>> > I'm looking to understand Cassandra's behavior in an sstable 
>>>>>>> > corruption scenario, and what the minimum amount of work is that 
>>>>>>> > needs to be done to remove a bad sstable file.
>>>>>>> > 
>>>>>>> > Consider: 3 node, RF 3 cluster, reads/writes at quorum
>>>>>>> > SStable corruption exception on one node at 
>>>>>>> > keyspace1/table1/lb-1-big-Data.db
>>>>>>> > Sstablescrub does not work.
>>>>>>> > 
>>>>>>> > Is it safest to, after running a repair on the two live nodes,
>>>>>>> > 1) Delete only keyspace1/table1/lb-1-big-Data.db,
>>>>>>> > 2) Delete all files associated with that sstable (i.e., 
>>>>>>> > keyspace1/table1/lb-1-*),
>>>>>>> > 3) Delete all files under keyspace1/table1/, or
>>>>>>> > 4) Any of the above are the same from a correctness perspective.
>>>>>>> > 
>>>>>>> > Thanks,
>>>>>>> > Leon
>>>>>>> > 
>>>>>>> 
>>>>>>> -
>>>>>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>>>>>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>>>>>> 

Re: Is deleting live sstable safe in this scenario?

[Nitan Kainth]

Jeff,

If Cassandra is down how will it generate merkle tree to compare?


Regards,
Nitan
Cell: 510 449 9629

> On May 27, 2020, at 11:15 AM, Jeff Jirsa  wrote:
> 
> 
> You definitely can repair with a node down by passing `-hosts specific_hosts`
> 
>> On Wed, May 27, 2020 at 9:06 AM Nitan Kainth  wrote:
>> I didn't get you Leon,
>> 
>> But, the simple thing is just to follow the steps and you will be fine. You 
>> can't run the repair if the node is down.
>> 
>>> On Wed, May 27, 2020 at 10:34 AM Leon Zaruvinsky  
>>> wrote:
>>> Hey Jeff/Nitan,
>>> 
>>> 1) this concern should not be a problem if the repair happens before the 
>>> corrupted node is brought back online, right?
>>> 2) in this case, is option (3) equivalent to replacing the node? where we 
>>> repair the two live nodes and then bring up the third node with no data
>>> 
>>> Leon
>>> 
>>>> On Tue, May 26, 2020 at 10:11 PM Jeff Jirsa  wrote:
>>>> There’s two problems with this approach if you need strict correctness 
>>>> 
>>>> 1) after you delete the sstable and before you repair you’ll violate 
>>>> consistency, so you’ll potentially serve incorrect data for a while
>>>> 
>>>> 2) The sstable May have a tombstone past gc grace that’s shadowing data in 
>>>> another sstable that’s not corrupt and deleting it may resurrect that 
>>>> deleted data. 
>>>> 
>>>> The only strictly safe thing to do here, unfortunately, is to treat the 
>>>> host as failed and rebuild it from it’s neighbors (and again being 
>>>> pedantic here, that means stop the host, while it’s stopped repair the 
>>>> surviving replicas, then bootstrap a replacement on top of the same tokens)
>>>> 
>>>> 
>>>> 
>>>> > On May 26, 2020, at 4:46 PM, Leon Zaruvinsky  
>>>> > wrote:
>>>> > 
>>>> > 
>>>> > Hi all,
>>>> > 
>>>> > I'm looking to understand Cassandra's behavior in an sstable corruption 
>>>> > scenario, and what the minimum amount of work is that needs to be done 
>>>> > to remove a bad sstable file.
>>>> > 
>>>> > Consider: 3 node, RF 3 cluster, reads/writes at quorum
>>>> > SStable corruption exception on one node at 
>>>> > keyspace1/table1/lb-1-big-Data.db
>>>> > Sstablescrub does not work.
>>>> > 
>>>> > Is it safest to, after running a repair on the two live nodes,
>>>> > 1) Delete only keyspace1/table1/lb-1-big-Data.db,
>>>> > 2) Delete all files associated with that sstable (i.e., 
>>>> > keyspace1/table1/lb-1-*),
>>>> > 3) Delete all files under keyspace1/table1/, or
>>>> > 4) Any of the above are the same from a correctness perspective.
>>>> > 
>>>> > Thanks,
>>>> > Leon
>>>> > 
>>>> 
>>>> -
>>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>>> 

Re: Is deleting live sstable safe in this scenario?

[Nitan Kainth]

I didn't get you Leon,

But, the simple thing is just to follow the steps and you will be fine. You
can't run the repair if the node is down.

On Wed, May 27, 2020 at 10:34 AM Leon Zaruvinsky 
wrote:

> Hey Jeff/Nitan,
>
> 1) this concern should not be a problem if the repair happens before the
> corrupted node is brought back online, right?
> 2) in this case, is option (3) equivalent to replacing the node? where we
> repair the two live nodes and then bring up the third node with no data
>
> Leon
>
> On Tue, May 26, 2020 at 10:11 PM Jeff Jirsa  wrote:
>
>> There’s two problems with this approach if you need strict correctness
>>
>> 1) after you delete the sstable and before you repair you’ll violate
>> consistency, so you’ll potentially serve incorrect data for a while
>>
>> 2) The sstable May have a tombstone past gc grace that’s shadowing data
>> in another sstable that’s not corrupt and deleting it may resurrect that
>> deleted data.
>>
>> The only strictly safe thing to do here, unfortunately, is to treat the
>> host as failed and rebuild it from it’s neighbors (and again being pedantic
>> here, that means stop the host, while it’s stopped repair the surviving
>> replicas, then bootstrap a replacement on top of the same tokens)
>>
>>
>>
>> > On May 26, 2020, at 4:46 PM, Leon Zaruvinsky 
>> wrote:
>> >
>> > 
>> > Hi all,
>> >
>> > I'm looking to understand Cassandra's behavior in an sstable corruption
>> scenario, and what the minimum amount of work is that needs to be done to
>> remove a bad sstable file.
>> >
>> > Consider: 3 node, RF 3 cluster, reads/writes at quorum
>> > SStable corruption exception on one node at
>> keyspace1/table1/lb-1-big-Data.db
>> > Sstablescrub does not work.
>> >
>> > Is it safest to, after running a repair on the two live nodes,
>> > 1) Delete only keyspace1/table1/lb-1-big-Data.db,
>> > 2) Delete all files associated with that sstable (i.e.,
>> keyspace1/table1/lb-1-*),
>> > 3) Delete all files under keyspace1/table1/, or
>> > 4) Any of the above are the same from a correctness perspective.
>> >
>> > Thanks,
>> > Leon
>> >
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>

Re: Is deleting live sstable safe in this scenario?

[Nitan Kainth]

Stop the node
Delete as per option 2
Run repair 


Regards,
Nitan
Cell: 510 449 9629

> On May 26, 2020, at 6:46 PM, Leon Zaruvinsky  wrote:
> 
> 
> Hi all,
> 
> I'm looking to understand Cassandra's behavior in an sstable corruption 
> scenario, and what the minimum amount of work is that needs to be done to 
> remove a bad sstable file.
> 
> Consider: 3 node, RF 3 cluster, reads/writes at quorum
> SStable corruption exception on one node at keyspace1/table1/lb-1-big-Data.db
> Sstablescrub does not work.
> 
> Is it safest to, after running a repair on the two live nodes,
> 1) Delete only keyspace1/table1/lb-1-big-Data.db,
> 2) Delete all files associated with that sstable (i.e., 
> keyspace1/table1/lb-1-*),
> 3) Delete all files under keyspace1/table1/, or
> 4) Any of the above are the same from a correctness perspective.
> 
> Thanks,
> Leon
> 

Re: Nodetool clearsnapshot does not delete snapshot for dropped column_family

[Nitan Kainth]

I don't think it works like that. clearsnapshot --all would remove all
snapshots. Here is an example:

$ ls -l
/ss/xx/cassandra/data/ww/a-5bf825428b3811eabe0c6b7631a60bb0/snapshots/

total 8

drwxr-xr-x 2 cassandra cassandra 4096 Apr 30 23:17 dropped-1588288650821-a

drwxr-xr-x 2 cassandra cassandra 4096 Apr 30 23:17 manual

$ nodetool clearsnapshot --all

Requested clearing snapshot(s) for [all keyspaces] with [all snapshots]

$ ls -l
/ss/xx/cassandra/data/ww/a-5bf825428b3811eabe0c6b7631a60bb0/snapshots/

ls: cannot access
/ss/xx/cassandra/data/ww/a-5bf825428b3811eabe0c6b7631a60bb0/snapshots/: No
such file or directory

$


On Thu, Apr 30, 2020 at 5:44 PM Erick Ramirez 
wrote:

> Yes, you're right. It doesn't show up in listsnapshots nor does
> clearsnapshot remove the dropped snapshot because the table is no longer
> managed by C* (because it got dropped). So you will need to manually remove
> the dropped-* directories from the filesystem.
>
> Someone here will either correct me or hopefully provide a user-friendlier
> solution. Cheers!
>

Re: Disabling Swap for Cassandra

[Nitan Kainth]

Swap is controlled by OS and will use it when running short of memory. I don’t 
think you can disable at Cassandra level


Regards,
Nitan
Cell: 510 449 9629

> On Apr 16, 2020, at 5:50 PM, Kunal  wrote:
> 
> 
> Hello,
>  
> I need some suggestion from you all. I am new to Cassandra and was reading 
> Cassandra best practices. On one document, it was mentioned that Cassandra 
> should not be using swap, it degrades the performance. 
> My question is instead of disabling swap system wide, can we force Cassandra 
> not to use swap? Some documentation suggests to use memory_locking_policy in 
> cassandra.yaml. 
> 
> How do I check if our Cassandra already has this parameter and still uses 
> swap ? Is there any way i can check this. I already checked cassandra.yaml 
> and dont see this parameter. Is there any other place i can check and confirm?
> 
> Also, Can I set memlock parameter to unlimited (64kB default), so entire Heap 
> (Xms = Xmx) can be locked at node startup ? Will that help?
> 
> Or if you have any other suggestions, please let me know. 
>  
>  
> Regards,
> Kunal
>  

Re: Memory usage more than heap memory

[Nitan Kainth]

You can look at top command. There is column of memory


Regards,
Nitan
Cell: 510 449 9629

> On Apr 11, 2020, at 11:10 AM, HImanshu Sharma  
> wrote:
> 
> 
> Hi 
> 
> But I see memory not decreasing even if there is no traffic on cluster. How 
> can I find actual memory usage by Cassandra process. If it is  OS page cache 
> then how  to find how much is page  cache and how much is used by process?
> 
> Thanks
> Himanshu
> 
>> On Sat, Apr 11, 2020 at 9:07 PM Laxmikant Upadhyay  
>> wrote:
>> It is OS page cache used during read..your os will leverage memory if not 
>> being used by any other applications and it improves your read performance.
>> 
>>> On Sat, Apr 11, 2020, 12:47 PM HImanshu Sharma 
>>>  wrote:
>>> Hi 
>>> 
>>> I am very new to the use of cassandra. In a cassandra cluster of 3 nodes, I 
>>> am observing memory usage of cassandra process going more than the heap 
>>> memor allocated. As I understand, cassandra allocates off heap memory for 
>>> bloom filters, index summary etc. 
>>> 
>>> When I run nodetool info, I see off heap memory usage around 1.5G, my heap 
>>> is 8G. So memory usage should not go above  10G. But in running node it 
>>> goes as high as 18G with heavy loads. It comes down with different loads 
>>> but always more than 12G.
>>> 
>>> I would like to know why memory usage is high and is there way to control 
>>> it.
>>> 
>>> Thanks
>>> Himanshu Sharma

Re: Query data through python using IN clause

[Nitan Kainth]

Thanks Alex.

On Thu, Apr 2, 2020 at 1:39 AM Alex Ott  wrote:

> Hi
>
> Working code is below, but I want to warn you - prefer not to use IN with
> partition keys - because you'll have different partition key values,
> coordinator node will need to perform queries to other hosts that hold
> these partition keys, and this slow downs the operation, and adds an
> additional load to the coordinating node.  If you execute queries in
> parallel (using async) for every of combination of pk1 & pk2, and then
> consolidate data application side - this could be faster than query with
> IN.
>
> Answer:
>
> You need to pass list as value of temp - IN expects list there...
>
> query = session.prepare("select * from test.table1 where pk1 IN ? and
> pk2=0 and ck1 > ? AND ck1 < ?;")
> temp = [1,2,3]
>
> import dateutil.parser
>
> ck1 = dateutil.parser.parse('2020-01-01T00:00:00Z')
> ck2 = dateutil.parser.parse('2021-01-01T00:00:00Z')
>
> rows = session.execute(query, (temp, ck1, ck2))
> for row in rows:
> print row
>
>
>
>
> Nitan Kainth  at "Wed, 1 Apr 2020 18:21:54 -0500" wrote:
>  NK> Hi There,
>
>  NK> I am trying to read data from table as below structure:
>
>  NK> table1(
>  NK> pk1 bigint,
>  NK> pk2 bigint,
>  NK> ck1 timestamp,
>  NK> value text,
>  NK> primary key((pk1,pk2),ck1);
>
>  NK> query = session.prepare("select * from table1 where pk IN ? and pk2=0
> and ck1 > ? AND ck1 < ?;")
>
>  NK> temp = 1,2,3
>
>  NK> runq = session.execute(query2, (temp,ck1, ck1))
>
>  NK> TypeError: Received an argument of invalid type for column
> "in(bam_user)". Expected: ,
> Got:
>  NK> ; (cannot convert argument
> to integer)
>
>  NK> I found examples for prepared statements for inserts but couldn't
> find any for select and not able to make it to work.
>
>  NK> Any suggestions?
>
>
>
> --
> With best wishes,Alex Ott
> Principal Architect, DataStax
> http://datastax.com/
>

Re: sstableloader & num_tokens change

[Nitan Kainth]

Dsbulk works same as sstableloder.


Regards,
Nitan
Cell: 510 449 9629

> On Jan 24, 2020, at 10:40 AM, Sergio  wrote:
> 
> 
> I was wondering if that improvement for token allocation would work even with 
> just one rack. It should but I am not sure.
> 
> Does Dsbulk support migration cluster to cluster without CSV or JSON export?
> 
> Thanks and Regards
> 
>> On Fri, Jan 24, 2020, 8:34 AM Nitan Kainth  wrote:
>> Instead of sstableloader consider dsbulk by datastax. 
>> 
>>> On Fri, Jan 24, 2020 at 10:20 AM Reid Pinchback 
>>>  wrote:
>>> Jon Haddad has previously made the case for num_tokens=4.  His Accelerate 
>>> 2019 talk is available at:
>>> 
>>>  
>>> 
>>> https://www.youtube.com/watch?v=swL7bCnolkU
>>> 
>>>  
>>> 
>>> You might want to check that out.  Also I think the amount of effort you 
>>> put into evening out the token distribution increases as vnode count 
>>> shrinks.  The caveats are explored at:
>>> 
>>>  
>>> 
>>> https://thelastpickle.com/blog/2019/02/21/set-up-a-cluster-with-even-token-distribution.html
>>> 
>>>  
>>> 
>>>  
>>> 
>>> From: Voytek Jarnot 
>>> Reply-To: "user@cassandra.apache.org" 
>>> Date: Friday, January 24, 2020 at 10:39 AM
>>> To: "user@cassandra.apache.org" 
>>> Subject: sstableloader & num_tokens change
>>> 
>>>  
>>> 
>>> Message from External Sender
>>> 
>>> Running 3.11.x, 4 nodes RF=3, default 256 tokens; moving to a different 4 
>>> node RF=3 cluster.
>>> 
>>>  
>>> 
>>> I've read that 256 is not an optimal default num_tokens value, and that 32 
>>> is likely a better option.
>>> 
>>>  
>>> 
>>> We have the "opportunity" to switch, as we're migrating environments and 
>>> will likely be using sstableloader to do so. I'm curious if there are any 
>>> gotchas with using sstableloader to restore snapshots taken from 256-token 
>>> nodes into a cluster with 32-token nodes (otherwise same # of nodes and 
>>> same RF).
>>> 
>>>  
>>> 
>>> Thanks in advance.

Re: sstableloader & num_tokens change

[Nitan Kainth]

Instead of sstableloader consider dsbulk by datastax.

On Fri, Jan 24, 2020 at 10:20 AM Reid Pinchback 
wrote:

> Jon Haddad has previously made the case for num_tokens=4.  His Accelerate
> 2019 talk is available at:
>
>
>
> https://www.youtube.com/watch?v=swL7bCnolkU
>
>
>
> You might want to check that out.  Also I think the amount of effort you
> put into evening out the token distribution increases as vnode count
> shrinks.  The caveats are explored at:
>
>
>
>
> https://thelastpickle.com/blog/2019/02/21/set-up-a-cluster-with-even-token-distribution.html
>
>
>
>
>
> *From: *Voytek Jarnot 
> *Reply-To: *"user@cassandra.apache.org" 
> *Date: *Friday, January 24, 2020 at 10:39 AM
> *To: *"user@cassandra.apache.org" 
> *Subject: *sstableloader & num_tokens change
>
>
>
> *Message from External Sender*
>
> Running 3.11.x, 4 nodes RF=3, default 256 tokens; moving to a different 4
> node RF=3 cluster.
>
>
>
> I've read that 256 is not an optimal default num_tokens value, and that 32
> is likely a better option.
>
>
>
> We have the "opportunity" to switch, as we're migrating environments and
> will likely be using sstableloader to do so. I'm curious if there are any
> gotchas with using sstableloader to restore snapshots taken from 256-token
> nodes into a cluster with 32-token nodes (otherwise same # of nodes and
> same RF).
>
>
>
> Thanks in advance.
>

Re: Cassandra2.0.14 : Obsolete files not being deleted after compaction

[Nitan Kainth]

If you are certain that you don’t need data, your plan is good. Make sure to 
delete all the files for any given sequence number ie data, index, toc etc

Regards,
Nitan
Cell: 510 449 9629

> On Jan 21, 2020, at 5:36 AM, manish khandelwal  
> wrote:
> 
> 
> Hi Team
> 
> I am observing some obsolete files in Cassandra 2.0.14 which are already 
> compacted but not removed from the system after compaction. 
> As per CASSANDRA-7872 , after GC grace period has passed the sstables are 
> open for read again and can lead to data resurrection. I am facing disk 
> crunch  (90% full ) as well and need to remove those obsolete files ASAP.
> 
> 
> To avoid this what should be our strategy? I am thinking on following lines
> 1. Stop the Cassandra server.
> 2. Remove the obsolete files manually.
> 3. Start the Cassandra server.
> 
> Regards
> Manish
> 
> 
> 
> 

Re: *URGENT* Migration across different Cassandra cluster few having same keyspace/table names

[Nitan Kainth]

If the keyspace already exist, use copy command or sstableloader to merge data. 
If data volume it too big, consider spark or a custom java program 


Regards,
Nitan
Cell: 510 449 9629

> On Jan 16, 2020, at 10:26 PM, Ankit Gadhiya  wrote:
> 
> 
> Any leads on this ?
> 
> — Ankit
> 
>> On Thu, Jan 16, 2020 at 8:51 PM Ankit Gadhiya  wrote:
>> Hi Arvinder,
>> 
>> Thanks for your response.
>> 
>> Yes - Cluster B already has some data. Tables/KS names are identical ; for 
>> data - I still haven't got the clarity if it has identical data or no - I am 
>> assuming no since it's for different customers but need the confirmation.
>> 
>> Thanks & Regards,
>> Ankit Gadhiya
>> 
>> 
>> 
>>> On Thu, Jan 16, 2020 at 8:49 PM Arvinder Dhillon  
>>> wrote:
>>> So as I understand, Cluster B already has some data and not an empty 
>>> cluster.
>>> 
>>> When you say, clusters share same keyspace and table names, do you mean 
>>> both clusters have identical data on those ks/tables?
>>> 
>>> 
>>> -Arvi
>>> 
 On Thu, Jan 16, 2020, 5:27 PM Ankit Gadhiya  wrote:
 Hello Group,
 
 I have a requirement in one of the production systems where I need to be 
 able to migrate entire dataset from Cluster A (Azure Region A) to Cluster 
 B (Azure Region B). 
 
 Each cluster have 3 Cassandra nodes (RF=3) running used by different 
 applications. Few of the applications are common is Cluster A and Cluster 
 B thereby sharing same keyspace/table names. 
 Need suggestion for the best possible migration strategy here considering 
 - 1. No Application code changes possible - Minor config/infra changes can 
 be considered. 2. Zero data loss. 3. No/Minimal downtime.
 
 It'd be great to hear ideas from all of you based on your experiences.
 
 Cassandra Version - Cassandra 3.0.13 on both sides.
 Total Data size - Cluster A: 70 GB, Cluster B: 15 GB
 
 Thanks & Regards,
 Ankit Gadhiya
 
> -- 
> Thanks & Regards,
> Ankit Gadhiya
> 

Re: re-add column with same UDT fail

[Nitan Kainth]

I have seen this situation causing corruption due to different structure but 
same name column in old and new sstables.


Regards,
Nitan
Cell: 510 449 9629

> On Nov 21, 2019, at 7:59 AM, Tommy Stendahl 
>  wrote:
> 
> 
> Hi,
> 
> I run in to problem with 3.11.5, I think its related to "* Toughen up column 
> drop/recreate type validations (CASSANDRA-15204)"
> 
> I have a user defined type and I have a table with a column that has this UDF 
> as type, if I drop the column and recreate it with the same name it fails. I 
> think this should work, it did in 3.11.4, or I'm I missing something?
> 
> I recreated this in cqlsh:
> 
> cqlsh> CREATE KEYSPACE foo WITH replication = {'class': 'SimpleStrategy', 
> 'replication_factor': '1'};
> cqlsh> CREATE TYPE foo.my_type (a int, b int );
> cqlsh> CREATE TABLE foo.bar ( x int PRIMARY KEY, y int, z frozen );
> cqlsh> ALTER TABLE foo.bar DROP z ;
> cqlsh> ALTER TABLE foo.bar ADD z frozen;
> InvalidRequest: Error from server: code=2200 [Invalid query] message="Cannot 
> re-add previously dropped column 'z' of type frozen, incompatible 
> with previous type frozen>"
> cqlsh> 
> 
> Regards,
> Tommy

Re: Multiple C* instances on same machine

[Nitan Kainth]

Thank you folks! It was very helpful


Regards,
Nitan
Cell: 510 449 9629

> On Sep 20, 2019, at 8:04 PM, Jonathan Koppenhofer  
> wrote:
> 
> We do this without containers quite successfully. As precaution, we...
> - have dedicated disk per instance.
> - have lots of network bandwidth, but also throttle throughout defaults. Also 
> monitor network closely
> - share CPU completely. Limit Cassandra settings to limit CPU use (concurrent 
> threads, compaction throughput, etc) and monitor closely.
> - have plenty of memory on top of jvm allocation
> - never have 2 nodes from the same cluster on a single node.
> - use VIPs so each instance gets its own IP address.
> - use sane OS defaults as documented by Amy Tobey.
> 
> That said, using cgroups or containers would provide better isolation (but 
> worse bursting) when available.
> 
> 
>> On Fri, Sep 20, 2019, 8:42 PM Sandeep Nethi  wrote:
>> Hi Nitan,
>> 
>> You shouldn’t have any issues if you setup things properly.
>> 
>> Few possible issues could be (can become a bottleneck)
>> 
>> * CPU allocation (Instances can compete)
>> * Disk throughput & IOPS &
>> * Port allocations
>> * Network throughout 
>> * Consistency issues.
>> 
>> And we have work around for all above,
>> 
>> * CPU: Use jvm file to limit number of CPU cores for each instance.
>> * DISK: If possible allocate dedicated disks for each instance.
>> * NETWORK & Ports:   Have a secondary NIC (or equivalent to num of 
>> instances). This will give you the flexibility to have Cassandra on same 
>> ports with better networking operations.
>> * RACK: having multiple instances on one node can lead to consistency 
>> problems when a hosted node goes down for some reason with having RACK’s 
>> defined. So, this is very important to choose when going with this kind of 
>> setup.
>> 
>> Hope this helps!
>> 
>> Thanks,
>> Sandeep.Nethi
>> 
>> 
>> 
>>> On Sat, 21 Sep 2019 at 12:20 PM, Nitan Kainth  wrote:
>>> I am looking for possible issues doing this setup without containers. 
>>> 
>>> 
>>> Regards,
>>> Nitan
>>> Cell: 510 449 9629
>>> 
>>>> On Sep 20, 2019, at 5:22 PM, Elliott Sims  wrote:
>>>> 
>>>> A container of some sort gives you better isolation and less risk of a 
>>>> mistake that could cause the instances to conflict in some way.  Might be 
>>>> better for balancing resources between them as well, though using cgroups 
>>>> directly can also accomplish that.
>>>> 
>>>> On Fri, Sep 20, 2019, 8:27 AM Nitan Kainth  wrote:
>>>>> Hi There,
>>>>> 
>>>>> Any feedback pros/cons for having multiple instances of C* on the same 
>>>>> machine without Docker/container solution? 
>>>>> 
>>>>> The plan is to change the ports and run multiple C* processes, so we can 
>>>>> isolate two applications as two different clusters.

Re: Multiple C* instances on same machine

[Nitan Kainth]

I am looking for possible issues doing this setup without containers. 


Regards,
Nitan
Cell: 510 449 9629

> On Sep 20, 2019, at 5:22 PM, Elliott Sims  wrote:
> 
> A container of some sort gives you better isolation and less risk of a 
> mistake that could cause the instances to conflict in some way.  Might be 
> better for balancing resources between them as well, though using cgroups 
> directly can also accomplish that.
> 
> On Fri, Sep 20, 2019, 8:27 AM Nitan Kainth  wrote:
>> Hi There,
>> 
>> Any feedback pros/cons for having multiple instances of C* on the same 
>> machine without Docker/container solution? 
>> 
>> The plan is to change the ports and run multiple C* processes, so we can 
>> isolate two applications as two different clusters.

Multiple C* instances on same machine

[Nitan Kainth]

Hi There,

Any feedback pros/cons for having multiple instances of C* on the same
machine without Docker/container solution?

The plan is to change the ports and run multiple C* processes, so we can
isolate two applications as two different clusters.

Re: How can I add blank values instead of null values in cassandra ?

[Nitan Kainth]

You can set default values in driver but that also little code change


Regards,
Nitan
Cell: 510 449 9629

> On Sep 9, 2019, at 8:15 PM, buchi adddagada  wrote:
> 
> We are using DSE 5.1.0 & Spring boot Java.
> 
> While we are trying to insert data into cassandra , java by default inserts 
> null values in cassandra tables which is causing huge tombstones.
> 
> 
> 
> Instead of changing code in java to insert null values, can you control 
> anywhere at driver level ?
> 
> 
> 
> Thanks,
> 
> Buchi Babu

Re: Datafile Corruption

[Nitan Kainth]

Repair during upgrade have caused corruption too.

Also, dropping and adding columns with same name but different type


Regards,
Nitan
Cell: 510 449 9629

> On Aug 7, 2019, at 2:42 PM, Jeff Jirsa  wrote:
> 
> Is compression enabled?
> 
> If not, bit flips on disk can corrupt data files and reads + repair may send 
> that corruption to other hosts in the cluster 
> 
> 
>> On Aug 7, 2019, at 3:46 AM, Philip Ó Condúin  
>> wrote:
>> 
>> Hi All,
>> 
>> I am currently experiencing multiple datafile corruptions across most nodes 
>> in my cluster, there seems to be no pattern to the corruption.  I'm starting 
>> to think it might be a bug, we're using Cassandra 2.2.13.
>> 
>> Without going into detail about the issue I just want to confirm something.
>> 
>> Can someone share with me a list of scenarios that would cause corruption?
>> 
>> 1. OS failure
>> 2. Cassandra disturbed during the writing 
>> 
>> etc etc.
>> 
>> I need to investigate each scenario and don't want to leave any out.
>> 
>> -- 
>> Regards,
>> Phil

Re: Rebooting one Cassandra node caused all the application nodes go down

[Nitan Kainth]

You no corruption error or you see corruption error?


Regards,
Nitan
Cell: 510 449 9629

> On Jul 19, 2019, at 1:52 PM, Rahul Reddy  wrote:
> 
> Schema matches and corruption errors in system.log
> 
>> On Fri, Jul 19, 2019, 1:33 PM Nitan Kainth  wrote:
>> Do you see schemat in sync? Nodetool describecluster.
>> 
>> Check system log for any corruption.
>> 
>> 
>> Regards,
>> Nitan
>> Cell: 510 449 9629
>> 
>>> On Jul 19, 2019, at 12:32 PM, ZAIDI, ASAD A  wrote:
>>> 
>>> “aws asked to set nvme_timeout to higher number in etc/grub.conf.”
>>> 
>>>  
>>> 
>>> Did you ask AWS if setting higher value is real solution to bug - Is there 
>>> not any patch available to address the bug?   - just curios to know
>>> 
>>>  
>>> 
>>> From: Rahul Reddy [mailto:rahulreddy1...@gmail.com] 
>>> Sent: Friday, July 19, 2019 10:49 AM
>>> To: user@cassandra.apache.org
>>> Subject: Rebooting one Cassandra node caused all the application nodes go 
>>> down
>>> 
>>>  
>>> 
>>> Here ,
>>> 
>>>  
>>> 
>>> We have 6 nodes each in 2 data centers us-east-1 and us-west-2  . We have 
>>> RF 3 and  cl set to local quorum. And gossip snitch. All our instance are 
>>> c5.2xlarge and data files and comit logs are stored in gp2 ebs.  C5 
>>> instance type had a bug which aws asked to set nvme_timeout to higher 
>>> number in etc/grub.conf. after setting the parameter and did run nodetool 
>>> drain and reboot the node in east
>>> 
>>>  
>>> 
>>> Instance cameup but Cassandra didn't come up normal had to start the 
>>> Cassandra. Cassandra cameup but it shows other instances down. Even though 
>>> didn't reboot the other node down same was observed in one other node. How 
>>> could that happen and don't any errors in system.log which is set to info.
>>> 
>>> Without any intervention gossip settled in 10 mins entire cluster became 
>>> normal.
>>> 
>>>  
>>> 
>>> Tried same thing West it happened again
>>> 
>>>  
>>> 
>>>  
>>> 
>>>  
>>> 
>>> I'm concerned how to check what caused it and if a reboot happens again how 
>>> to avoid this.
>>> 
>>>  If I just  STOP Cassandra instead of reboot I don't see this issue.
>>> 
>>>  

Re: Rebooting one Cassandra node caused all the application nodes go down

[Nitan Kainth]

Do you see schemat in sync? Nodetool describecluster.

Check system log for any corruption.


Regards,
Nitan
Cell: 510 449 9629

> On Jul 19, 2019, at 12:32 PM, ZAIDI, ASAD A  wrote:
> 
> “aws asked to set nvme_timeout to higher number in etc/grub.conf.”
>  
> Did you ask AWS if setting higher value is real solution to bug - Is there 
> not any patch available to address the bug?   - just curios to know
>  
> From: Rahul Reddy [mailto:rahulreddy1...@gmail.com] 
> Sent: Friday, July 19, 2019 10:49 AM
> To: user@cassandra.apache.org
> Subject: Rebooting one Cassandra node caused all the application nodes go down
>  
> Here ,
>  
> We have 6 nodes each in 2 data centers us-east-1 and us-west-2  . We have RF 
> 3 and  cl set to local quorum. And gossip snitch. All our instance are 
> c5.2xlarge and data files and comit logs are stored in gp2 ebs.  C5 instance 
> type had a bug which aws asked to set nvme_timeout to higher number in 
> etc/grub.conf. after setting the parameter and did run nodetool drain and 
> reboot the node in east
>  
> Instance cameup but Cassandra didn't come up normal had to start the 
> Cassandra. Cassandra cameup but it shows other instances down. Even though 
> didn't reboot the other node down same was observed in one other node. How 
> could that happen and don't any errors in system.log which is set to info.
> Without any intervention gossip settled in 10 mins entire cluster became 
> normal.
>  
> Tried same thing West it happened again
>  
>  
>  
> I'm concerned how to check what caused it and if a reboot happens again how 
> to avoid this.
>  If I just  STOP Cassandra instead of reboot I don't see this issue.
>  

Re: commit_log way bigger than allowed

[Nitan Kainth]

You can delete commitlogs and start the node. But run repair on that node to 
sync any data mismatch.


Regards,
Nitan
Cell: 510 449 9629

> On Jun 25, 2019, at 4:37 AM, pwozniak  wrote:
> 
> Hi,
> 
> I have cluster of three Cassandra (v.2.1) machines. On one of the machines 
> files with commit logs filled up all available disk space (50 GB).
> 
> I haven't change 'commitlog_total_space_in_mb', so as far as I know It 
> shouldn't take more that 8GB of disc space.
> 
> I also haven't found any suspicious messages in log file and our cluster was 
> not hammered by huge amount of requests lately.
> 
> This machine (cassandra process) is not able to boot up now (it crashes while 
> replaying commit_log)
> 
> 
> 1. How can I find out what happened?
> 
> 2. Can I just delete all commit_logs, restart machine and run repair to have 
> consistent data?
> 
> 3. Maybe I can delete just part of the commit_log files so Cassandra will be 
> able to boot and clean (flush) all commit_log files?
> 
> 
> Regards,
> 
> Pawel
> 
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
> 

Re: Upgrade sstables vs major compaction

[Nitan Kainth]

Not sure where the binaries were obtained from. I am helping a friend who
is not part of this DL.

based on understanding, i proposed above mentioned plan wanted to hear
others' opinions.

On Mon, Jun 24, 2019 at 6:06 AM Oleksandr Shulgin <
oleksandr.shul...@zalando.de> wrote:

> On Fri, Jun 21, 2019 at 7:02 PM Nitan Kainth 
> wrote:
>
>>
>> we upgraded binaries from 3.0 to 4.0.
>>
>
> Where did you get the binaries for 4.0?  It is not released officially
> yet, so I guess you were using SVN trunk?  Or was there a pre-release?
>
> we run major compaction periodically for some valid reasons.
>>
>> Now, we are considering running major compaction instead of
>> upgradesstables. Repair is disabled, cluster have normal reads/writes (few
>> hundered/second).
>>
>> Plan is to run major compaction to clear tombstones and reduce data
>> volume. Once major compaction is done, we will run upgradesstable for any
>> leftover sstables if any.
>>
>> Can you please advise major compaction could cause issues instead of
>> upgradesstables? Or any other feedback in this action plan?
>>
>
> Sounds legit.
>
> --
> Alex
>
>

Upgrade sstables vs major compaction

[Nitan Kainth]

Hi,

we upgraded binaries from 3.0 to 4.0.
we run major compaction periodically for some valid reasons.

Now, we are considering running major compaction instead of
upgradesstables. Repair is disabled, cluster have normal reads/writes (few
hundered/second).

Plan is to run major compaction to clear tombstones and reduce data volume.
Once major compaction is done, we will run upgradesstable for any leftover
sstables if any.

Can you please advise major compaction could cause issues instead of
upgradesstables? Or any other feedback in this action plan?

Re: Compaction resume after node restart and upgrade

[Nitan Kainth]

Thanks Jeff.

On Mon, Jun 17, 2019 at 8:08 PM Jeff Jirsa  wrote:

> Yes  - the incomplete sstable will be deleted during startup (in 3.0 and
> newer there’s a transaction log of each compaction in progress - that gets
> cleaned during the startup process)
>
> On Jun 17, 2019, at 6:02 PM, Nitan Kainth  wrote:
>
> The interim/temp sstable will delete by itself?
>
>
> Regards,
>
> Nitan
>
> Cell: 510 449 9629
>
> On Jun 17, 2019, at 7:15 PM, Jeff Jirsa  wrote:
>
>
>
> On Jun 17, 2019, at 4:22 PM, Nitan Kainth  wrote:
>
> Hi,
>
> C* version 3.0.16.
> Two questions:
> 1. If major compaction is in progress and we restart the node, will
> compaction start from beginning or will it resume from where the node was
> shut down?
>
>
> You’ll have to restart it manually , and it’ll restart from the beginning
>
>
> 2. If we upgrade binaries to C* 4.0 while compaction is running, will it
> resume the compaction or start over or no compaction would kick in?
>
>
> Normal compactions will start over on restart but major compaction would
> need to be run again manually
>
>

Re: Compaction resume after node restart and upgrade

[Nitan Kainth]

The interim/temp sstable will delete by itself?


Regards,
Nitan
Cell: 510 449 9629

> On Jun 17, 2019, at 7:15 PM, Jeff Jirsa  wrote:
> 
> 
> 
>> On Jun 17, 2019, at 4:22 PM, Nitan Kainth  wrote:
>> 
>> Hi,
>> 
>> C* version 3.0.16.
>> Two questions:
>> 1. If major compaction is in progress and we restart the node, will 
>> compaction start from beginning or will it resume from where the node was 
>> shut down?
> 
> You’ll have to restart it manually , and it’ll restart from the beginning
> 
> 
>> 2. If we upgrade binaries to C* 4.0 while compaction is running, will it 
>> resume the compaction or start over or no compaction would kick in?
>> 
> 
> Normal compactions will start over on restart but major compaction would need 
> to be run again manually 

Re: Compaction resume after node restart and upgrade

[Nitan Kainth]

Thanks Jeff!


Regards,
Nitan
Cell: 510 449 9629

> On Jun 17, 2019, at 7:15 PM, Jeff Jirsa  wrote:
> 
> 
> 
>> On Jun 17, 2019, at 4:22 PM, Nitan Kainth  wrote:
>> 
>> Hi,
>> 
>> C* version 3.0.16.
>> Two questions:
>> 1. If major compaction is in progress and we restart the node, will 
>> compaction start from beginning or will it resume from where the node was 
>> shut down?
> 
> You’ll have to restart it manually , and it’ll restart from the beginning
> 
> 
>> 2. If we upgrade binaries to C* 4.0 while compaction is running, will it 
>> resume the compaction or start over or no compaction would kick in?
>> 
> 
> Normal compactions will start over on restart but major compaction would need 
> to be run again manually 

Compaction resume after node restart and upgrade

[Nitan Kainth]

Hi,

C* version 3.0.16.
Two questions:
1. If major compaction is in progress and we restart the node, will
compaction start from beginning or will it resume from where the node was
shut down?
2. If we upgrade binaries to C* 4.0 while compaction is running, will it
resume the compaction or start over or no compaction would kick in?

Re: Sstableloader

[Nitan Kainth]

If cassandra version is same, it should work


Regards,
Nitan
Cell: 510 449 9629

> On May 28, 2019, at 4:21 PM, Rahul Reddy  wrote:
> 
> Hello,
> 
> Does sstableloader works between datastax and Apache cassandra. I'm trying to 
> migrate dse 5.0.7 to Apache 3.11.1 ?

Re: Can sstable corruption cause schema mismatch?

[Nitan Kainth]

All ports are open.
We tried rolling restart and full cluster down and start one mode at a time.
Changes down were:
Storage addition
Ddl for column drop and recreate

Schema version is same for few nodes and few shows unavailable.

Network has been verified in detail and no severe packet drops.


Regards,
Nitan
Cell: 510 449 9629

> On May 29, 2019, at 10:04 AM, Alain RODRIGUEZ  wrote:
> 
> Ideas that come mind are:
> 
> - Rolling restart of the cluster
> - Use of 'nodetool resetlocalschema'  --> function name speaks for itself. 
> Note that this is to be ran on each node you think is having schema issues
> - Are all nodes showing a schema version showing the same one?
> - Port not fully open across all nodes?
> - Anything in the logs?
> 
> Do you know what triggered this situation in the first place?
> 
> C*heers,
> ---
> Alain Rodriguez - al...@thelastpickle.com
> France / Spain
> 
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
> 
>> Le mar. 28 mai 2019 à 18:28, Nitan Kainth  a écrit :
>> Thank you Alain.
>> 
>> Nodetool describecluster shows some nodes unreachable, different output from 
>> each node. 
>> Node1 can see all 4 nodes up.
>> Node 2 says node 4 and node 5 unreachable
>> Node 3 complains about node node 2 and node 1
>> 
>> Nodetool status shows all nodes up and read writes are working for most most 
>> operations. 
>> 
>> Network looks good. Any other ideas?
>> 
>> 
>> Regards,
>> Nitan
>> Cell: 510 449 9629
>> 
>>> On May 28, 2019, at 11:21 AM, Alain RODRIGUEZ  wrote:
>>> 
>>> Hello Nitan,
>>> 
>>>> 1. Can sstable corruption in application tables cause schema mismatch?
>>> 
>>> I would say it should not. I could imagine in the case that the corrupted 
>>> table hits some 'system' keyspace sstable. If not I don' see how corrupted 
>>> data can impact the schema on the node.
>>>  
>>>> 2. Do we need to disable repair while adding storage while Cassandra is 
>>>> down?
>>> 
>>> I think you don't have to, but that it's a good idea.
>>> Repairs would fail as soon/long as you have a node down that should be 
>>> involved (I think there is an option to change that behaviour now).
>>> Anyway, stopping repair and restarting it when all nodes are probably 
>>> allows you a better understanding/control of what's going on. Also, it 
>>> reduces the load in time of troubles or maintenance, when the cluster is 
>>> somewhat weaker.
>>> 
>>> C*heers,
>>> ---
>>> Alain Rodriguez - al...@thelastpickle.com
>>> France / Spain
>>> 
>>> The Last Pickle - Apache Cassandra Consulting
>>> http://www.thelastpickle.com
>>> 
>>> 
>>> 
>>>> Le mar. 28 mai 2019 à 17:13, Nitan Kainth  a écrit :
>>>> Hi,
>>>> 
>>>> Two questions:
>>>> 1. Can sstable corruption in application tables cause schema mismatch?
>>>> 2. Do we need to disable repair while adding storage while Cassandra is 
>>>> down?
>>>> 
>>>> 
>>>> Regards,
>>>> Nitan
>>>> Cell: 510 449 9629

Re: Can sstable corruption cause schema mismatch?

[Nitan Kainth]

Thank you Alain.

Nodetool describecluster shows some nodes unreachable, different output from 
each node. 
Node1 can see all 4 nodes up.
Node 2 says node 4 and node 5 unreachable
Node 3 complains about node node 2 and node 1

Nodetool status shows all nodes up and read writes are working for most most 
operations. 

Network looks good. Any other ideas?


Regards,
Nitan
Cell: 510 449 9629

> On May 28, 2019, at 11:21 AM, Alain RODRIGUEZ  wrote:
> 
> Hello Nitan,
> 
>> 1. Can sstable corruption in application tables cause schema mismatch?
> 
> I would say it should not. I could imagine in the case that the corrupted 
> table hits some 'system' keyspace sstable. If not I don' see how corrupted 
> data can impact the schema on the node.
>  
>> 2. Do we need to disable repair while adding storage while Cassandra is down?
> 
> I think you don't have to, but that it's a good idea.
> Repairs would fail as soon/long as you have a node down that should be 
> involved (I think there is an option to change that behaviour now).
> Anyway, stopping repair and restarting it when all nodes are probably allows 
> you a better understanding/control of what's going on. Also, it reduces the 
> load in time of troubles or maintenance, when the cluster is somewhat weaker.
> 
> C*heers,
> ---
> Alain Rodriguez - al...@thelastpickle.com
> France / Spain
> 
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
> 
> 
> 
>> Le mar. 28 mai 2019 à 17:13, Nitan Kainth  a écrit :
>> Hi,
>> 
>> Two questions:
>> 1. Can sstable corruption in application tables cause schema mismatch?
>> 2. Do we need to disable repair while adding storage while Cassandra is down?
>> 
>> 
>> Regards,
>> Nitan
>> Cell: 510 449 9629

Can sstable corruption cause schema mismatch?

[Nitan Kainth]

Hi,

Two questions:
1. Can sstable corruption in application tables cause schema mismatch?
2. Do we need to disable repair while adding storage while Cassandra is down?


Regards,
Nitan
Cell: 510 449 9629

Re: Corrupted sstables

[Nitan Kainth]

can Disk have bad sectors? fccheck or something similar can help.

Long shot: repair or any other operation conflicting. Would leave that to
others.

On Mon, May 6, 2019 at 3:50 PM Roy Burstein  wrote:

> It happens on the same column families and they have the same ddl (as
> already posted) . I did not check it after cleanup
> .
>
> On Mon, May 6, 2019, 23:43 Nitan Kainth  wrote:
>
>> This is strange, never saw this. does it happen to same column family?
>>
>> Does it happen after cleanup?
>>
>> On Mon, May 6, 2019 at 3:41 PM Roy Burstein 
>> wrote:
>>
>>> Yes.
>>>
>>> On Mon, May 6, 2019, 23:23 Nitan Kainth  wrote:
>>>
>>>> Roy,
>>>>
>>>> You mean all nodes show corruption when you add a node to cluster??
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Nitan
>>>>
>>>> Cell: 510 449 9629
>>>>
>>>> On May 6, 2019, at 2:48 PM, Roy Burstein 
>>>> wrote:
>>>>
>>>> It happened  on all the servers in the cluster every time I have added
>>>> node
>>>> .
>>>> This is new cluster nothing was upgraded here , we have a similar
>>>> cluster
>>>> running on C* 2.1.15 with no issues .
>>>> We are aware to the scrub utility just it reproduce every time we added
>>>> node to the cluster .
>>>>
>>>> We have many tables there
>>>>
>>>>

Re: Corrupted sstables

[Nitan Kainth]

This is strange, never saw this. does it happen to same column family?

Does it happen after cleanup?

On Mon, May 6, 2019 at 3:41 PM Roy Burstein  wrote:

> Yes.
>
> On Mon, May 6, 2019, 23:23 Nitan Kainth  wrote:
>
>> Roy,
>>
>> You mean all nodes show corruption when you add a node to cluster??
>>
>>
>> Regards,
>>
>> Nitan
>>
>> Cell: 510 449 9629
>>
>> On May 6, 2019, at 2:48 PM, Roy Burstein  wrote:
>>
>> It happened  on all the servers in the cluster every time I have added
>> node
>> .
>> This is new cluster nothing was upgraded here , we have a similar cluster
>> running on C* 2.1.15 with no issues .
>> We are aware to the scrub utility just it reproduce every time we added
>> node to the cluster .
>>
>> We have many tables there
>>
>>

Re: Corrupted sstables

[Nitan Kainth]

Roy,

You mean all nodes show corruption when you add a node to cluster??


Regards,
Nitan
Cell: 510 449 9629

> On May 6, 2019, at 2:48 PM, Roy Burstein  wrote:
> 
> It happened  on all the servers in the cluster every time I have added node
> .
> This is new cluster nothing was upgraded here , we have a similar cluster
> running on C* 2.1.15 with no issues .
> We are aware to the scrub utility just it reproduce every time we added
> node to the cluster .
> 
> We have many tables there

Re: Corrupted sstables

[Nitan Kainth]

Did you try sstablescrub?
If that doesn't work, you can delete all files of this sstable id and then
run repair -pr on this node.

On Mon, May 6, 2019 at 9:20 AM Roy Burstein  wrote:

> Hi ,
> We are having issues with Cassandra 3.11.4 , after adding node to the
> cluster we get many corrupted files across the cluster (almost all nodes)
> ,this is reproducible in our env.  .
> We  have 69 nodes in the cluster ,disk_access_mode: standard .
>
> The stack trace :
>
> WARN  [ReadStage-4] 2019-05-06 06:44:19,843 
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread 
> Thread[ReadStage-4,5,main]: {}
> java.lang.RuntimeException: 
> org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: 
> /var/lib/cassandra/data/disk1/sessions_rawdata/sessions_v2_2019_05_06-9cae0c20585411e99aa867a11519e31c/md-816-big-I
> ndex.db
> at 
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2588)
>  ~[apache-cassandra-3.11.4.jar:3.11.4]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> ~[na:1.8.0-zing_19.03.0.0]
> at 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
>  ~[apache-cassandra-3.11.4.jar:3.11.4]
> at 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
>  [apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114) 
> [apache-cassandra-3.11.4.jar:3.11.4]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0-zing_19.03.0.0]
> Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: 
> Corrupted: 
> /var/lib/cassandra/data/disk1/sessions_rawdata/sessions_v2_2019_05_06-9cae0c20585411e99aa867a11519e31c/md-816-big-Index.db
> at 
> org.apache.cassandra.io.sstable.format.big.BigTableReader.getPosition(BigTableReader.java:275)
>  ~[apache-cassandra-3.11.4.jar:3.11.4]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader.getPosition(SSTableReader.java:1586)
>  ~[apache-cassandra-3.11.4.jar:3.11.4]
> at 
> org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:64)
>  ~[apache-cassandra-3.11.4.jar:3.11.4]
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.initializeIterator(UnfilteredRowIteratorWithLowerBound.java:108)
>  ~[apache-cassandra-3.11.4.jar:3.11.4]
> at 
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48)
>  ~[apache-cassandra-3.11.4.jar:3.11.4]
> at 
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:99)
>  ~[apache-cassandra-3.11.4.jar:3.11.4]
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:119)
>  ~[apache-cassandra-3.11.4.jar:3.11.4]
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:48)
>  ~[apache-cassandra-3.11.4.jar:3.11.4]
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at 
> org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374)
>  ~[apache-cassandra-3.11.4.jar:3.11.4]
> at 
> org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186)
>  ~[apache-cassandra-3.11.4.jar:3.11.4]
> at 
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155)
>  ~[apache-cassandra-3.11.4.jar:3.11.4]
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:525)
>  ~[apache-cassandra-3.11.4.jar:3.11.4]
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:385)
>  ~[apache-cassandra-3.11.4.jar:3.11.4]
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:67)
>  ~[apache-cassandra-3.11.4.jar:3.11.4]
> at 
> org.apache.cassandra.db.SinglePartitionReadCommand.withSSTablesIterated(SinglePartitionReadCommand.java:853)
>  ~[apache-cassandra-3.11.4.jar:3.11.4]
> at 
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:797)
>  ~[apache-cassandra-3.11.4.jar:3.11.4]
> at 
> 

Re: Accidentaly removed SSTables of unneeded data

[Nitan Kainth]

You can run nodetool refresh and then sstablescrub to see if there is any
corruption.

On Thu, May 2, 2019 at 9:53 AM Simon ELBAZ  wrote:

> Hi,
>
> I am running Cassandra v2.1 on a 3 node cluster.
>
> *# yum list installed | grep cassa*
> *cassandra21.noarch2.1.12-1
> @datastax*
> *cassandra21-tools.noarch  2.1.12-1
> @datastax   *
>
> Unfortunately, I accidentally removed the SSTables (using rm) (older than
> 10 days) of a table on the 3 nodes.
>
> Running 'nodetool repair' on one of the 3 nodes returns error. Whereas, it
> does not on another.
>
> I don't need to recover the lost data but I would like 'nodetool repair'
> not returning an error.
>
> Thanks for any advice.
>
> Simon
>

Re: Increasing the size limits implications

[Nitan Kainth]

Increasing batch size could potentially lead to longer GC pauses.

Try to break you batch size. 300kb is a decent limit for most use cases.


Regards,
Nitan
Cell: 510 449 9629

> On Apr 29, 2019, at 12:17 PM, Bobbie Haynes  wrote:
> 
> Hi, 
>   I'm inserting into cassandra in batches(With each containing single PK 
> ).But my batch is failing and throwing exceptions.
> I want to know if we increase batch_size_warn_threshold_in_kb to 200KB and 
> batch_size_fail_threshold_in_kb to 300KB. What could be potential issues i 
> could be facing 
> 
> Thanks,
> Bobbie

Re: time tracking for down node for nodetool repair

[Nitan Kainth]

Valid suggestion. Stick to the plan, avoid downtime of a node more than hinted 
handoff window. OR increase window to a larger value, if you know it is going 
to take longer than current setting


Regards,
Nitan
Cell: 510 449 9629

> On Apr 8, 2019, at 8:43 PM, Soumya Jena  wrote:
> 
> Cassandra tracks it and no new hints will be created once the default 3 hours 
> window is passed  . However , cassandra will not automatically trigger a 
> repair if your node is down for more than 3 hours .Default settings of 3 
> hours for hints is defined in cassandra.yaml file . Look for 
> "max_hint_window_in_ms" in the cassandra.yaml file. Its configurable . Apart 
> from the periodic repair you should start a repair when you bring up a node 
> which has missed some writes .   
> 
> One more thing is  if node is down for long time and missed a lot of writes 
> sometimes it may be better to add that as a new fresh node rather than adding 
> it and then doing repair .
> 
>> On Mon, Apr 8, 2019 at 4:49 PM Stefan Miklosovic 
>>  wrote:
>> Ah I see it is the default for hinted handoffs. I was somehow thinking
>> its bigger figure I do not know why :)
>> 
>> I would say you should run repairs continuously / periodically so you
>> would not even have to do some thinking about that and it should run
>> in the background in a scheduled manner if possible.
>> 
>> Regards
>> 
>> On Tue, 9 Apr 2019 at 04:19, Kunal  wrote:
>> >
>> > Hello everyone..
>> >
>> >
>> >
>> > I have a 6 node Cassandra datacenter, 3 nodes on each datacenter. If one 
>> > of the node goes down and remain down for more than 3 hr, I have to run 
>> > nodetool repair. Just wanted to ask if Cassandra  automatically tracks the 
>> > time when one of the Cassandra node goes down or do I need to write code 
>> > to track the time and run repair when node comes back online after 3 hrs.
>> >
>> >
>> > Thanks in anticipation.
>> >
>> > Regards,
>> > Kunal Vaid
>> 
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>> 

Re: Audit in C*

[Nitan Kainth]

Good to know Bobbie 


Regards,
Nitan
Cell: 510 449 9629

> On Mar 13, 2019, at 3:21 PM, Bobbie Haynes  wrote:
> 
> Yes.we are using it and it is very helpful to u so far...!
> 
>> On Wed, Mar 13, 2019 at 11:38 AM Rahul Singh  
>> wrote:
>> Which version are you referring to?
>> 
>>> On Wed, Mar 13, 2019 at 10:28 AM Nitan Kainth  wrote:
>>> Hi,
>>> 
>>> Anybody have used auditing to find out failed login attempts, or 
>>> unauthorized access tries.
>>> 
>>> I found ecAudit by Ericsson, is it free to use? Has anybody tried it?
>>> 
>>> Ref: https://github.com/Ericsson/ecaudit

Re: Audit in C*

[Nitan Kainth]

3.11


Regards,
Nitan
Cell: 510 449 9629

> On Mar 14, 2019, at 3:18 AM, Per Otterström  
> wrote:
> 
> With ecAudit you can get audit records for login attempts and queries on 
> selected ks/tables. Currently there is no way to get audit records for 
> rejected attempts/queries _only_, but that’s an interesting feature for 
> future versions.
>  
> The plug-in is free to use under the Apache 2.0 license and comes pre-build 
> for Cassandra 3.0.x and Cassandra 3.11.x.
>  
> /pelle
>  
> From: Bobbie Haynes  
> Sent: den 13 mars 2019 20:21
> To: user@cassandra.apache.org
> Subject: Re: Audit in C*
>  
> Yes.we are using it and it is very helpful to u so far...!
>  
> On Wed, Mar 13, 2019 at 11:38 AM Rahul Singh  
> wrote:
> Which version are you referring to?
>  
> On Wed, Mar 13, 2019 at 10:28 AM Nitan Kainth  wrote:
> Hi,
>  
> Anybody have used auditing to find out failed login attempts, or unauthorized 
> access tries.
>  
> I found ecAudit by Ericsson, is it free to use? Has anybody tried it?
>  
> Ref: https://github.com/Ericsson/ecaudit

Re: Audit in C*

[Nitan Kainth]

Thank you Pelle


Regards,
Nitan
Cell: 510 449 9629

> On Mar 14, 2019, at 3:18 AM, Per Otterström  
> wrote:
> 
> With ecAudit you can get audit records for login attempts and queries on 
> selected ks/tables. Currently there is no way to get audit records for 
> rejected attempts/queries _only_, but that’s an interesting feature for 
> future versions.
>  
> The plug-in is free to use under the Apache 2.0 license and comes pre-build 
> for Cassandra 3.0.x and Cassandra 3.11.x.
>  
> /pelle
>  
> From: Bobbie Haynes  
> Sent: den 13 mars 2019 20:21
> To: user@cassandra.apache.org
> Subject: Re: Audit in C*
>  
> Yes.we are using it and it is very helpful to u so far...!
>  
> On Wed, Mar 13, 2019 at 11:38 AM Rahul Singh  
> wrote:
> Which version are you referring to?
>  
> On Wed, Mar 13, 2019 at 10:28 AM Nitan Kainth  wrote:
> Hi,
>  
> Anybody have used auditing to find out failed login attempts, or unauthorized 
> access tries.
>  
> I found ecAudit by Ericsson, is it free to use? Has anybody tried it?
>  
> Ref: https://github.com/Ericsson/ecaudit

Audit in C*

[Nitan Kainth]

Hi,

Anybody have used auditing to find out failed login attempts, or
unauthorized access tries.

I found ecAudit by Ericsson, is it free to use? Has anybody tried it?

Ref: https://github.com/Ericsson/ecaudit

Re: AxonOps - Cassandra operational management tool

[Nitan Kainth]

This is really cool!

will it be open source or licensed in near future?

On Thu, Feb 14, 2019 at 12:15 PM AxonOps  wrote:

> Hi folks,
>
> We are excited to announce AxonOps, an operational management tool for
> Apache Cassandra, is now ready for Beta testing.
>
> We'd be interested to hear you try this and let us know what you think!
>
> Please read the installation instructions on https://www.axonops.com
>
> AxonOps Team
>

Re: Max number of windows when using TWCS

[Nitan Kainth]

That’s right Jeff. That’s why I am thinking why not compaction gets rid of old 
exited sstables?


Regards,
Nitan
Cell: 510 449 9629

> On Feb 11, 2019, at 3:53 PM, Jeff Jirsa  wrote:
> 
> It's probably not safe. You shouldn't touch the underlying sstables unless 
> you're very sure you know what you're doing.
> 
> 
>> On Mon, Feb 11, 2019 at 1:05 PM Akash Gangil  wrote:
>> I have in the past tried to delete SSTables manually, but have noticed bits 
>> and pieces of that data still remain, even though the sstables of that 
>> window is deleted. So always wondered if playing directly with the 
>> underlying filesystem is a safe bet?
>> 
>> 
>>> On Mon, Feb 11, 2019 at 1:01 PM Jonathan Haddad  wrote:
>>> Deleting SSTables manually can be useful if you don't know your TTL up 
>>> front.  For example, you have an ETL process that moves your raw Cassandra 
>>> data into S3 as parquet files, and you want to be sure that process is 
>>> completed before you delete the data.  You could also start out without 
>>> setting a TTL and later realize you need one.  This is a remarkably common 
>>> problem.
>>> 
>>>> On Mon, Feb 11, 2019 at 12:51 PM Nitan Kainth  
>>>> wrote:
>>>> Jeff,
>>>> 
>>>> It means we have to delete sstables manually?
>>>> 
>>>> 
>>>> Regards,
>>>> Nitan
>>>> Cell: 510 449 9629
>>>> 
>>>>> On Feb 11, 2019, at 2:40 PM, Jeff Jirsa  wrote:
>>>>> 
>>>>> There's a bit of headache around overlapping sstables being strictly safe 
>>>>> to delete.  https://issues.apache.org/jira/browse/CASSANDRA-13418 was 
>>>>> added to allow the "I know it's not technically safe, but just delete it 
>>>>> anyway" use case. For a lot of people who started using TWCS before 
>>>>> 13418, "stop cassandra, remove stuff we know is expired, start cassandra" 
>>>>> is a not-uncommon pattern in very high-write, high-disk-space use cases. 
>>>>> 
>>>>> 
>>>>> 
>>>>>> On Mon, Feb 11, 2019 at 12:34 PM Nitan Kainth  
>>>>>> wrote:
>>>>>> Hi,
>>>>>> In regards to comment “Purging data is also straightforward, just 
>>>>>> dropping SSTables (by a script) where create date is older than a 
>>>>>> threshold, we don't even need to rely on TTL”
>>>>>> 
>>>>>> Doesn’t the old sstables drop by itself? One ttl and gc grace seconds 
>>>>>> past whole sstable will have only tombstones.
>>>>>> 
>>>>>> 
>>>>>> Regards,
>>>>>> Nitan
>>>>>> Cell: 510 449 9629
>>>>>> 
>>>>>>> On Feb 11, 2019, at 2:23 PM, DuyHai Doan  wrote:
>>>>>>> 
>>>>>>> Purging data is also straightforward, just dropping SSTables (by a 
>>>>>>> script) where create date is older than a threshold, we don't even need 
>>>>>>> to rely on TTL
>>> 
>>> 
>>> -- 
>>> Jon Haddad
>>> http://www.rustyrazorblade.com
>>> twitter: rustyrazorblade
>> 
>> 
>> -- 
>> Akash

Re: Max number of windows when using TWCS

[Nitan Kainth]

Jeff,

It means we have to delete sstables manually?


Regards,
Nitan
Cell: 510 449 9629

> On Feb 11, 2019, at 2:40 PM, Jeff Jirsa  wrote:
> 
> There's a bit of headache around overlapping sstables being strictly safe to 
> delete.  https://issues.apache.org/jira/browse/CASSANDRA-13418 was added to 
> allow the "I know it's not technically safe, but just delete it anyway" use 
> case. For a lot of people who started using TWCS before 13418, "stop 
> cassandra, remove stuff we know is expired, start cassandra" is a 
> not-uncommon pattern in very high-write, high-disk-space use cases. 
> 
> 
> 
>> On Mon, Feb 11, 2019 at 12:34 PM Nitan Kainth  wrote:
>> Hi,
>> In regards to comment “Purging data is also straightforward, just dropping 
>> SSTables (by a script) where create date is older than a threshold, we don't 
>> even need to rely on TTL”
>> 
>> Doesn’t the old sstables drop by itself? One ttl and gc grace seconds past 
>> whole sstable will have only tombstones.
>> 
>> 
>> Regards,
>> Nitan
>> Cell: 510 449 9629
>> 
>>> On Feb 11, 2019, at 2:23 PM, DuyHai Doan  wrote:
>>> 
>>> Purging data is also straightforward, just dropping SSTables (by a script) 
>>> where create date is older than a threshold, we don't even need to rely on 
>>> TTL

Re: Max number of windows when using TWCS

[Nitan Kainth]

Hi,
In regards to comment “Purging data is also straightforward, just dropping 
SSTables (by a script) where create date is older than a threshold, we don't 
even need to rely on TTL”

Doesn’t the old sstables drop by itself? One ttl and gc grace seconds past 
whole sstable will have only tombstones.


Regards,
Nitan
Cell: 510 449 9629

> On Feb 11, 2019, at 2:23 PM, DuyHai Doan  wrote:
> 
> Purging data is also straightforward, just dropping SSTables (by a script) 
> where create date is older than a threshold, we don't even need to rely on TTL

Re: datamodelling

[Nitan Kainth]

You could consider a sudo column like agent_type and make it a compound
partition key. It will limit break your partition into smaller ones but you
will have to query with agent_id and agent_type in that case.

On Tue, Feb 5, 2019 at 12:59 PM Bobbie Haynes  wrote:

> Hi Everyone,
>   Could you please help me in modeling my table
> below.I'm stuck here. My Partition key is agent_id and clustering column is
> rowid. Each agent can have a minimum of 1000 rows to 10M depends on how
> busy the agent .I'm facing large partition issue for my busy agents.
> I'm using SizeTieredCompaction here..The table has Writes/Reads (70/30
> ratio) and have deletes also in the table by agentid.
>
>
> CREATE TABLE IF NOT EXISTS XXX (
>  agent_id UUID,
>  row_id BIGINT,
>  path TEXT,
>  security_attributes TEXT,
>  actor TEXT,
>  PRIMARY KEY (agent_id,row_id)
> )
>

Re: Quick question about compaction progress

[Nitan Kainth]

Thank you Jeff


Regards,
Nitan
Cell: 510 449 9629

> On Jan 23, 2019, at 10:13 AM, Jeff Jirsa  wrote:
> 
> 
> 
>> On Jan 23, 2019, at 8:00 AM, Nitan Kainth  wrote:
>> 
>> Hi,
>> 
>> Why does nodetool compactionstats not show time remaining when 
>> compactionthroughput is set to 0?
> 
> Because we don’t have a good estimate if we’re not throttling (could be 
> added, just not tracked now)
> 
>> 
>> If the node is restarted during compaction, does it continue from where it 
>> is left or does it start over? 
> 
> Starts over - the incomplete files on disk will be removed during startup as 
> part of an incomplete lifecycle transaction
> 
>> 
>> If it starts over, what happens to the new sstable that was being used for 
>> compaction?
> 
> Removed.
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
> 

Quick question about compaction progress

[Nitan Kainth]

Hi,

Why does nodetool compactionstats not show time remaining when
compactionthroughput is set to 0?

If the node is restarted during compaction, does it continue from where it
is left or does it start over?

If it starts over, what happens to the new sstable that was being used for
compaction?

Re: Compact storage removal effect

[Nitan Kainth]

Thanks a lot Chris.

On Tue, Jan 22, 2019 at 11:08 AM Chris Lohfink 
wrote:

> In 3.x+ the format on disk is the same with compact storage on or off so
> you shouldn't expect much of a difference in table size with the new
> storage format compared to compact vs non compact in 2.x.
>
> Chris
>
> On Jan 22, 2019, at 10:21 AM, Nitan Kainth  wrote:
>
> hey Chris,
>
> We upgraded form 3.0.4 to 3.11. yes, I did run upgradesstables -a to
> migrate sstables.
> Here is the table structure:
>
> CREATE TABLE ks.cf1 ( key text, column1 timestamp, value blob, PRIMARY KEY
> (key, column1) ) WITH COMPACT STORAGE CREATE TABLE ks.cf2 ( key bigint,
> column1 text, value blob, PRIMARY KEY (key, column1) ) WITH COMPACT STORAGE
> CREATE TABLE ks.cf3 ( key text, column1 timestamp, value int, PRIMARY KEY
> (key, column1) ) WITH COMPACT STORAGE
>
> On Tue, Jan 22, 2019 at 10:07 AM Chris Lohfink 
> wrote:
>
>> What version are you running? Did you include an upgradesstables -a or
>> something to rebuild without the compact storage in your migration?
>>
>> After 3.0 the new format can be more or less the same size as the 2.x
>> compact storage tables depending on schema (which can impact things a lot).
>>
>> Chris
>>
>> > On Jan 22, 2019, at 9:58 AM, Nitan Kainth 
>> wrote:
>> >
>> > Hi,
>> >
>> > We are testing to migrate off from compact storage. After removing
>> compact storage, we were hoping to see an increase in disk usage but
>> nothing changed.
>> > any feedback, why didn't we see an increase in storage?
>>
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>
>

Re: Compact storage removal effect

[Nitan Kainth]

hey Chris,

We upgraded form 3.0.4 to 3.11. yes, I did run upgradesstables -a to
migrate sstables.
Here is the table structure:

CREATE TABLE ks.cf1 ( key text, column1 timestamp, value blob, PRIMARY KEY
(key, column1) ) WITH COMPACT STORAGE CREATE TABLE ks.cf2 ( key bigint,
column1 text, value blob, PRIMARY KEY (key, column1) ) WITH COMPACT STORAGE
CREATE TABLE ks.cf3 ( key text, column1 timestamp, value int, PRIMARY KEY
(key, column1) ) WITH COMPACT STORAGE

On Tue, Jan 22, 2019 at 10:07 AM Chris Lohfink 
wrote:

> What version are you running? Did you include an upgradesstables -a or
> something to rebuild without the compact storage in your migration?
>
> After 3.0 the new format can be more or less the same size as the 2.x
> compact storage tables depending on schema (which can impact things a lot).
>
> Chris
>
> > On Jan 22, 2019, at 9:58 AM, Nitan Kainth  wrote:
> >
> > Hi,
> >
> > We are testing to migrate off from compact storage. After removing
> compact storage, we were hoping to see an increase in disk usage but
> nothing changed.
> > any feedback, why didn't we see an increase in storage?
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Compact storage removal effect

[Nitan Kainth]

Hi,

We are testing to migrate off from compact storage. After removing compact
storage, we were hoping to see an increase in disk usage but nothing
changed.
any feedback, why didn't we see an increase in storage?

Re: Partition key with 300K rows can it be queried and distributed using Spark

[Nitan Kainth]

Not sure about spark data distribution but yeah spark can be used to retrieve 
such data from Cassandra.


Regards,
Nitan
Cell: 510 449 9629

> On Jan 17, 2019, at 2:15 PM, Goutham reddy  wrote:
> 
> Hi,
> As each partition key can hold up to 2 Billion rows, even then it is an 
> anti-pattern to have such huge data set for one partition key in our case it 
> is 300k rows only, but when trying to query for one particular key we are 
> getting timeout exception. If I use Spark to get the 300k rows for a 
> particular key does it solve the problem of timeouts and distribute the data 
> across the spark nodes or will it still throw timeout exceptions. Can you 
> please help me with the best practice to retrieve the data for the key with 
> 300k rows. Any help is highly appreciated.
> 
> Regards
> Goutham.

Re: is there an upper limit of the size tier of stcs?

[Nitan Kainth]

If you exceel 50% then you could be in trouble for compaction. No immediate
harm though, just that compaction will start failing and eventually, you
will see read latencies go high.

On Mon, Jan 14, 2019 at 4:59 AM Jinhua Luo  wrote:

> But what if the sstable file size of the largest tire is too large to
> compact (e.g. no spare disk room to merge two huge sstables).
> Not to mention that huge file is likely to bring troubles for some
> file systems, and it's hard to backup them.
>
> Nitan Kainth  于2019年1月14日周一 上午1:53写道：
> >
> > No, sstables keep compacting when they meet criteria. Look for
> compaction sub properties
> >
> >
> > Regards,
> >
> > Nitan
> >
> > Cell: 510 449 9629
> >
> >
> > On Jan 13, 2019, at 8:18 AM, Jinhua Luo  wrote:
> >
> > is there a largest tier where the sstables stop merging anymore? if so,
> then does it mean some tombstones at that last tier would not be removed?
> if not, then it's hard to imagine to compact sstables in huge size, like
> 10gb; after all, the total size of sstables would definitelly grow, so a
> huge sstable would be very likely possible, especially when the range of
> the keys is wide.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: is there an upper limit of the size tier of stcs?

[Nitan Kainth]

No, sstables keep compacting when they meet criteria. Look for compaction sub 
properties 


Regards,
Nitan
Cell: 510 449 9629

> On Jan 13, 2019, at 8:18 AM, Jinhua Luo  wrote:
> 
> is there a largest tier where the sstables stop merging anymore? if so, then 
> does it mean some tombstones at that last tier would not be removed? if not, 
> then it's hard to imagine to compact sstables in huge size, like 10gb; after 
> all, the total size of sstables would definitelly grow, so a huge sstable 
> would be very likely possible, especially when the range of the keys is wide.

Re: Does long latency affect Cassandra's performance

[Nitan Kainth]

Yes, it will have negative impact.
Avoid doing it if you can

Sent from my iPhone

> On Dec 15, 2018, at 12:18 PM, dayu  wrote:
> 
> Nitan,
> thanks for your reply.
>The new node in different DC is added as same DC, so i mean I use quorum 
> or all for three replicate not local quorum.
> does the long latency will block coordinate and affect the performance？
> 
> dayu
> 
>   
> dayu
> 邮箱：sdycre...@163.com
> Signature is customized by Netease Mail Master
> 
> On 12/16/2018 02:00, Nitan Kainth wrote:
> Dayu,
> 
> If you use local quorum it won't impact read and write latencies but there 
> will be delay in across DC replication; which may cause some application 
> issues if it is active active. But 2ms is probably negligible to see any 
> effect.
> 
>> On Sat, Dec 15, 2018 at 11:57 AM dayu  wrote:
>> Hi all，
>> I am adding a new node to cassandra，but the new node is at different 
>> DC，the ping latency between them is 2 ms.
>>I am wondering does the long latency would affect cluster's performance 
>> or thoughput？
>> 
>> Thank!
>> dayu
>> 
>>  
>> dayu
>> 邮箱：sdycre...@163.com
>> Signature is customized by Netease Mail Master

Re: Does long latency affect Cassandra's performance

[Nitan Kainth]

Dayu,

If you use local quorum it won't impact read and write latencies but there
will be delay in across DC replication; which may cause some application
issues if it is active active. But 2ms is probably negligible to see any
effect.

On Sat, Dec 15, 2018 at 11:57 AM dayu  wrote:

> Hi all，
> I am adding a new node to cassandra，but the new node is at different
> DC，the ping latency between them is 2 ms.
>I am wondering does the long latency would affect cluster's performance
> or thoughput？
>
> Thank!
> dayu
>
> dayu
> 邮箱：sdycre...@163.com
>
> 
>
> Signature is customized by Netease Mail Master
> 
>

Re: Unexplainable spikes of requests latency

[Nitan Kainth]

Latency and mismatch matches may not align but mismatch means data is not
in sync AND can cause read latency.

On Thu, Dec 13, 2018 at 2:34 AM Виталий Савкин 
wrote:

> Good catch. We ran repairs few times but don't do it on a regular basis.
> But I found no dependency between count of DigestMismatchExceptions and
> latency spikes (see attached graphs for example).
> One important point I didn't mention in the original mail is that all
> requests (both reads and writes) have CL=LOCAL_QUORUM.
>
> ср, 12 дек. 2018 г. в 19:49, Nitan Kainth :
>
>> DigestMismatchExceptions   --> could be due to data out of sync.Are you
>> running repairs?
>>
>> On Wed, Dec 12, 2018 at 11:39 AM Виталий Савкин 
>> wrote:
>>
>>> Hi everyone!
>>>
>>> Few times a day I see spikes of requests latencies on my cassandra
>>> clients. Usually 99thPercentile is below 100ms but that times it grows
>>> above 1 second.
>>> Type of request doesn't matter: different services are affected and I
>>> found that three absolutely identical requests (to the same partition key,
>>> issued in a three-second interval) completed in 1ms, 30ms and 1100ms. Also
>>> I found no correlation between spikes and patterns of load. G1 GC does not
>>> report any significant (>50ms) delays.
>>> Few suspicious things:
>>>
>>>- nodetool shows that there are dropped READs
>>>- there are DigestMismatchExceptions in logs
>>>- in tracing events I see that event "Executing single-partition
>>>query on *" sometimes happens right after "READ message received from
>>>/*.*.*.*" (in less than 100 micros) and sometimes after hundreds of
>>>milliseconds
>>>
>>> My cluster runs on six c5.2xlarge Amazon instances, data is stored on
>>> EBS. Cassandra version is 3.10.
>>> Any help in explaining this behavior is appreciated. I'm glad to share
>>> more details if needed.
>>>
>>> Thanks,
>>> Vitaliy Savkin.
>>>
>>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Unexplainable spikes of requests latency

[Nitan Kainth]

DigestMismatchExceptions   --> could be due to data out of sync.Are you
running repairs?

On Wed, Dec 12, 2018 at 11:39 AM Виталий Савкин 
wrote:

> Hi everyone!
>
> Few times a day I see spikes of requests latencies on my cassandra
> clients. Usually 99thPercentile is below 100ms but that times it grows
> above 1 second.
> Type of request doesn't matter: different services are affected and I
> found that three absolutely identical requests (to the same partition key,
> issued in a three-second interval) completed in 1ms, 30ms and 1100ms. Also
> I found no correlation between spikes and patterns of load. G1 GC does not
> report any significant (>50ms) delays.
> Few suspicious things:
>
>- nodetool shows that there are dropped READs
>- there are DigestMismatchExceptions in logs
>- in tracing events I see that event "Executing single-partition query
>on *" sometimes happens right after "READ message received from /*.*.*.*"
>(in less than 100 micros) and sometimes after hundreds of milliseconds
>
> My cluster runs on six c5.2xlarge Amazon instances, data is stored on EBS.
> Cassandra version is 3.10.
> Any help in explaining this behavior is appreciated. I'm glad to share
> more details if needed.
>
> Thanks,
> Vitaliy Savkin.
>

Re: Tracing in cassandra

[Nitan Kainth]

Using partition key is best way to query 

Sent from my iPhone

> On Oct 12, 2018, at 6:49 PM, Abdul Patel  wrote:
> 
> Yes with range queries its timing out, one question was the where condition 
> is primary key rather than clustering key.
> 
>> On Friday, October 12, 2018, Nitan Kainth  wrote:
>> Did it still timeout?
>> 
>> Sent from my iPhone
>> 
>>> On Oct 12, 2018, at 1:11 PM, Abdul Patel  wrote:
>>> 
>>> With limit 11 this is query..
>>> Select * from table where status=0 and tojen(user_id)  >=token(126838) and 
>>> token(user_id)  <= token>> 
>>>> On Friday, October 12, 2018, Abdul Patel  wrote:
>>>> Let me try with limit 11 ..we have 18 node cluster ..no nodes down..
>>>> 
>>>>> On Friday, October 12, 2018, Nitan Kainth  wrote:
>>>>> Try query with partition key selection in where clause. But time for 
>>>>> limit 11 shouldn’t fail. Are all nodes up? Do you see any corruption in 
>>>>> ay sstable?
>>>>> 
>>>>> Sent from my iPhone
>>>>> 
>>>>>> On Oct 12, 2018, at 11:40 AM, Abdul Patel  wrote:
>>>>>> 
>>>>>> Sean,
>>>>>> 
>>>>>> here it is :
>>>>>> CREATE TABLE Keyspave.tblname (
>>>>>> user_id bigint,
>>>>>> session_id text,
>>>>>> application_guid text,
>>>>>> last_access_time timestamp,
>>>>>> login_time timestamp,
>>>>>> status int,
>>>>>> terminated_by text,
>>>>>> update_time timestamp,
>>>>>> PRIMARY KEY (user_id, session_id)
>>>>>> ) WITH CLUSTERING ORDER BY (session_id ASC)
>>>>>> 
>>>>>> also they see timeouts with limit 11 as well, so is it better to remove 
>>>>>> with limit option ? or whats best to query such schema?
>>>>>> 
>>>>>>> On Fri, Oct 12, 2018 at 11:05 AM Durity, Sean R 
>>>>>>>  wrote:
>>>>>>> Cross-partition = multiple partitions
>>>>>>> 
>>>>>>>  
>>>>>>> 
>>>>>>> Simple example:
>>>>>>> 
>>>>>>> Create table customer (
>>>>>>> 
>>>>>>> Customerid int,
>>>>>>> 
>>>>>>> Name text,
>>>>>>> 
>>>>>>> Lastvisit date,
>>>>>>> 
>>>>>>> Phone text,
>>>>>>> 
>>>>>>> Primary key (customerid) );
>>>>>>> 
>>>>>>>  
>>>>>>> 
>>>>>>> Query
>>>>>>> 
>>>>>>> Select customerid from customer limit 5000;
>>>>>>> 
>>>>>>>  
>>>>>>> 
>>>>>>> The query is asking for 5000 different partitions to be selected across 
>>>>>>> the cluster. This is a very EXPENSIVE query for Cassandra, especially 
>>>>>>> as the number of nodes goes up. Typically, you want to query a single 
>>>>>>> partition. Read timeouts are usually caused by queries that are 
>>>>>>> selecting many partitions or a very large partition. That is why a 
>>>>>>> schema for the involved table could help.
>>>>>>> 
>>>>>>>  
>>>>>>> 
>>>>>>>  
>>>>>>> 
>>>>>>> Sean Durity
>>>>>>> 
>>>>>>>  
>>>>>>> 
>>>>>>> From: Abdul Patel  
>>>>>>> Sent: Friday, October 12, 2018 10:04 AM
>>>>>>> To: user@cassandra.apache.org
>>>>>>> Subject: [EXTERNAL] Re: Tracing in cassandra
>>>>>>> 
>>>>>>>  
>>>>>>> 
>>>>>>> Cpuld you elaborate cross partition query?
>>>>>>> 
>>>>>>> On Friday, October 12, 2018, Durity, Sean R 
>>>>>>>  wrote:
>>>>>>> 
>>>>>>> I suspect you are doing a cross-partition query, which will not scale 
>>>>>>> well (as you can see). What is the schema for the table involved?
>>>>>>> 
>>>>>>

Re: Tracing in cassandra

[Nitan Kainth]

Did it still timeout?

Sent from my iPhone

> On Oct 12, 2018, at 1:11 PM, Abdul Patel  wrote:
> 
> With limit 11 this is query..
> Select * from table where status=0 and tojen(user_id)  >=token(126838) and 
> token(user_id)  <= token 
>> On Friday, October 12, 2018, Abdul Patel  wrote:
>> Let me try with limit 11 ..we have 18 node cluster ..no nodes down..
>> 
>>> On Friday, October 12, 2018, Nitan Kainth  wrote:
>>> Try query with partition key selection in where clause. But time for limit 
>>> 11 shouldn’t fail. Are all nodes up? Do you see any corruption in ay 
>>> sstable?
>>> 
>>> Sent from my iPhone
>>> 
>>>> On Oct 12, 2018, at 11:40 AM, Abdul Patel  wrote:
>>>> 
>>>> Sean,
>>>> 
>>>> here it is :
>>>> CREATE TABLE Keyspave.tblname (
>>>> user_id bigint,
>>>> session_id text,
>>>> application_guid text,
>>>> last_access_time timestamp,
>>>> login_time timestamp,
>>>> status int,
>>>> terminated_by text,
>>>> update_time timestamp,
>>>> PRIMARY KEY (user_id, session_id)
>>>> ) WITH CLUSTERING ORDER BY (session_id ASC)
>>>> 
>>>> also they see timeouts with limit 11 as well, so is it better to remove 
>>>> with limit option ? or whats best to query such schema?
>>>> 
>>>>> On Fri, Oct 12, 2018 at 11:05 AM Durity, Sean R 
>>>>>  wrote:
>>>>> Cross-partition = multiple partitions
>>>>> 
>>>>>  
>>>>> 
>>>>> Simple example:
>>>>> 
>>>>> Create table customer (
>>>>> 
>>>>> Customerid int,
>>>>> 
>>>>> Name text,
>>>>> 
>>>>> Lastvisit date,
>>>>> 
>>>>> Phone text,
>>>>> 
>>>>> Primary key (customerid) );
>>>>> 
>>>>>  
>>>>> 
>>>>> Query
>>>>> 
>>>>> Select customerid from customer limit 5000;
>>>>> 
>>>>>  
>>>>> 
>>>>> The query is asking for 5000 different partitions to be selected across 
>>>>> the cluster. This is a very EXPENSIVE query for Cassandra, especially as 
>>>>> the number of nodes goes up. Typically, you want to query a single 
>>>>> partition. Read timeouts are usually caused by queries that are selecting 
>>>>> many partitions or a very large partition. That is why a schema for the 
>>>>> involved table could help.
>>>>> 
>>>>>  
>>>>> 
>>>>>  
>>>>> 
>>>>> Sean Durity
>>>>> 
>>>>>  
>>>>> 
>>>>> From: Abdul Patel  
>>>>> Sent: Friday, October 12, 2018 10:04 AM
>>>>> To: user@cassandra.apache.org
>>>>> Subject: [EXTERNAL] Re: Tracing in cassandra
>>>>> 
>>>>>  
>>>>> 
>>>>> Cpuld you elaborate cross partition query?
>>>>> 
>>>>> On Friday, October 12, 2018, Durity, Sean R  
>>>>> wrote:
>>>>> 
>>>>> I suspect you are doing a cross-partition query, which will not scale 
>>>>> well (as you can see). What is the schema for the table involved?
>>>>> 
>>>>>  
>>>>> 
>>>>>  
>>>>> 
>>>>> Sean Durity
>>>>> 
>>>>>  
>>>>> 
>>>>> From: Abdul Patel  
>>>>> Sent: Thursday, October 11, 2018 5:54 PM
>>>>> To: a...@instaclustr.com
>>>>> Cc: user@cassandra.apache.org
>>>>> Subject: [EXTERNAL] Re: Tracing in cassandra
>>>>> 
>>>>>  
>>>>> 
>>>>> Query :
>>>>> 
>>>>> SELECT * FROM keysoace.tablenameWHERE user_id = 390797583 LIMIT 5000; 
>>>>> 
>>>>> -Error: ReadTimeout: Error from server: code=1200 [Coordinator node timed 
>>>>> out waiting for  replica nodes' responses] message="Operation timed out - 
>>>>> received only 0 responses." info={'received_responses': 0, 
>>>>> 'required_responses': 1, 'consistency': 'ONE'}
>>>>> 
>>>>>  
>>>>> 
>>>>&g

Re: [EXTERNAL] Re: Tracing in cassandra

[Nitan Kainth]

Try query with partition key selection in where clause. But time for limit 11 
shouldn’t fail. Are all nodes up? Do you see any corruption in ay sstable?

Sent from my iPhone

> On Oct 12, 2018, at 11:40 AM, Abdul Patel  wrote:
> 
> Sean,
> 
> here it is :
> CREATE TABLE Keyspave.tblname (
> user_id bigint,
> session_id text,
> application_guid text,
> last_access_time timestamp,
> login_time timestamp,
> status int,
> terminated_by text,
> update_time timestamp,
> PRIMARY KEY (user_id, session_id)
> ) WITH CLUSTERING ORDER BY (session_id ASC)
> 
> also they see timeouts with limit 11 as well, so is it better to remove with 
> limit option ? or whats best to query such schema?
> 
>> On Fri, Oct 12, 2018 at 11:05 AM Durity, Sean R 
>>  wrote:
>> Cross-partition = multiple partitions
>> 
>>  
>> 
>> Simple example:
>> 
>> Create table customer (
>> 
>> Customerid int,
>> 
>> Name text,
>> 
>> Lastvisit date,
>> 
>> Phone text,
>> 
>> Primary key (customerid) );
>> 
>>  
>> 
>> Query
>> 
>> Select customerid from customer limit 5000;
>> 
>>  
>> 
>> The query is asking for 5000 different partitions to be selected across the 
>> cluster. This is a very EXPENSIVE query for Cassandra, especially as the 
>> number of nodes goes up. Typically, you want to query a single partition. 
>> Read timeouts are usually caused by queries that are selecting many 
>> partitions or a very large partition. That is why a schema for the involved 
>> table could help.
>> 
>>  
>> 
>>  
>> 
>> Sean Durity
>> 
>>  
>> 
>> From: Abdul Patel  
>> Sent: Friday, October 12, 2018 10:04 AM
>> To: user@cassandra.apache.org
>> Subject: [EXTERNAL] Re: Tracing in cassandra
>> 
>>  
>> 
>> Cpuld you elaborate cross partition query?
>> 
>> On Friday, October 12, 2018, Durity, Sean R  
>> wrote:
>> 
>> I suspect you are doing a cross-partition query, which will not scale well 
>> (as you can see). What is the schema for the table involved?
>> 
>>  
>> 
>>  
>> 
>> Sean Durity
>> 
>>  
>> 
>> From: Abdul Patel  
>> Sent: Thursday, October 11, 2018 5:54 PM
>> To: a...@instaclustr.com
>> Cc: user@cassandra.apache.org
>> Subject: [EXTERNAL] Re: Tracing in cassandra
>> 
>>  
>> 
>> Query :
>> 
>> SELECT * FROM keysoace.tablenameWHERE user_id = 390797583 LIMIT 5000; 
>> 
>> -Error: ReadTimeout: Error from server: code=1200 [Coordinator node timed 
>> out waiting for replica nodes' responses] message="Operation timed out - 
>> received only 0 responses." info={'received_responses': 0, 
>> 'required_responses': 1, 'consistency': 'ONE'}
>> 
>>  
>> 
>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70bd7c0-cd9e-11e8-8e99-15807bff4dfd 
>> | 
>> Parsing SELECT * FROM keysoace.tablenameWHERE user_id = 390797583 LIMIT 
>> 5000; | 10.54.145.32 |   4020 |   
>> Native-Transport-Requests-3
>> 
>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70bfed0-cd9e-11e8-8e99-15807bff4dfd 
>> |
>> Preparing statement | 
>> 10.54.145.32 |   5065 |   Native-Transport-Requests-3
>> 
>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c25e0-cd9e-11e8-8e99-15807bff4dfd 
>> |
>>  Executing single-partition query on roles | 
>> 10.54.145.32 |   6171 |   ReadStage-2
>> 
>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c4cf0-cd9e-11e8-8e99-15807bff4dfd 
>> |
>>Acquiring sstable references | 
>> 10.54.145.32 |   6362 |   ReadStage-2
>> 
>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c4cf1-cd9e-11e8-8e99-15807bff4dfd 
>> |  
>> Skipped 0/2 non-slice-intersecting sstables, included 0 due to tombstones | 
>> 10.54.145.32 |   6641 |   ReadStage-2
>> 
>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c4cf2-cd9e-11e8-8e99-15807bff4dfd 
>> |
>>   Key cache hit for sstable 346 | 
>> 10.54.145.32 |   6955 |   ReadStage-2
>> 
>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c4cf3-cd9e-11e8-8e99-15807bff4dfd 
>> |
>>Bloom filter allows skipping sstable 347 | 
>> 10.54.145.32 |   7202 |   ReadStage-2
>> 
>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c7400-cd9e-11e8-8e99-15807bff4dfd 
>> |  

Re: Compaction Strategy

[Nitan Kainth]

It’s not recommended to disable compaction, you will end up with hundreds to 
thousands of sstables and increased read latency. If your data is immitable, 
means no update/deletes it will have least impact.

Decreasing compaction throughput will release resources for application but 
don’t accumulate too many pending compaction tasks.

Sent from my iPhone

> On Sep 19, 2018, at 4:44 PM, rajasekhar kommineni  wrote:
> 
> Hello,
> 
> Can any one respond to my questions. Is it a good idea to disable auto 
> compaction and schedule it every 3 days. I am unable to control compaction 
> and it is causing timeouts. 
> 
> Also will reducing or increasing compaction_throughput_mb_per_sec eliminate 
> timeouts ?
> 
> Thanks,
> 
> 
>> On Sep 17, 2018, at 9:38 PM, rajasekhar kommineni  
>> wrote:
>> 
>> Hello Folks,
>> 
>> I need advice in deciding the compaction strategy for my C cluster. There 
>> are multiple jobs that will load the data with less inserts and more updates 
>> but no deletes. Currently I am using Size Tired compaction, but seeing auto 
>> compactions after the data load kicks, and also read timeouts during 
>> compaction.
>> 
>> Can anyone suggest good compaction strategy for my cluster which will reduce 
>> the timeouts.
>> 
>> 
>> Thanks,
>> 
> 
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Storage overhead for a row

[Nitan Kainth]

Hi,

We are trying to get to exact size of a row in table. All the data is
standard sized. As per dev team, partition size should be 10832 bytes. But
as per tablehistograms partition size is 17084
As per
https://docs.datastax.com/en/archived/cassandra/2.0/cassandra/architecture/architecturePlanningUserData_t.html
expected overhead is 23bytes per row per column.
Question:


   - Partition size is compressed or uncompressed?
   - compression ratio .40 means 40% reduction in size or 40% after
   compression?
   -

   primary_key_index overhead is stored in sstable or outside

   -

   Partition size in tablehistograms include overhead size or just actual data?


Here is the information about table.
--- table structure --

CREATE TABLE x.y (

key text,

column1 timestamp,

value blob,

PRIMARY KEY (key, column1)


--- tablehistogram ---


Percentile  SSTables Write Latency  Read LatencyPartition Size
  Cell Count

  (micros)  (micros)   (bytes)


50% 1.00 20.50152.32 17084
372

75% 1.00 20.50219.34 17084
372

95%17.00 24.60785.94 17084
372

98%17.00 29.52943.13 17084
372

99%17.00 35.43   1358.10 17084
372

Min 0.00  4.77 17.0943
  0

Max17.00219.34 155469.30 35425
770


--- table stats ---

Read Count: 4122704331

Read Latency: 0.36671174864540634 ms.

Write Count: 9972009145

Write Latency: 0.017737019208880798 ms.

Pending Flushes: 0

Table: raw_data

SSTable count: 16

Space used (live): 1109412902237

Space used (total): 1109412902237

Space used by snapshots (total): 0

Off heap memory used (total): 1160258564

SSTable Compression Ratio: 0.4096557120739874

Number of keys (estimate): 236017001

Memtable cell count: 318143

Memtable data size: 20781581

Memtable off heap memory used: 0

Memtable switch count: 55601

Local read count: 4122704332

Local read latency: 0.226 ms

Local write count: 9972009147

Local write latency: 0.020 ms

Pending flushes: 0

Bloom filter false positives: 163

Bloom filter false ratio: 0.0

Bloom filter space used: 741096200

Bloom filter off heap memory used: 741096072

Index summary off heap memory used: 90682196

Compression metadata off heap memory used: 328480296

Compacted partition minimum bytes: 43

Compacted partition maximum bytes: 35425

Compacted partition mean bytes: 11619

Average live cells per slice (last five minutes): 27.168724439058366

Maximum live cells per slice (last five minutes): 446

Average tombstones per slice (last five minutes): 1.0

Maximum tombstones per slice (last five minutes): 1

Re: default_time_to_live vs TTL on insert statement

[Nitan Kainth]

Okay so it means regular update and any ttl set with write overrides default 
setting. Which means datastax documentation is incorrect and should be updated.

Sent from my iPhone

> On Jul 12, 2018, at 9:35 AM, DuyHai Doan  wrote:
> 
> To set TTL on a column only and not on the whole CQL row, use UPDATE
> instead:
> 
> UPDATE  USING TTL xxx SET = WHERE partition=yyy
> 
>> On Thu, Jul 12, 2018 at 2:42 PM, Nitan Kainth  wrote:
>> 
>> Kurt,
>> 
>> It is same mentioned on apache docuemtation too, I am not able to f

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: default_time_to_live vs TTL on insert statement

[Nitan Kainth]

Kurt,

It is same mentioned on apache docuemtation too, I am not able to find it
right now.

But my question is:
How to set TTL for a whole column?

On Wed, Jul 11, 2018 at 11:36 PM, kurt greaves  wrote:

> The Datastax documentation is wrong. It won't error, and it shouldn't. If
> you want to fix that documentation I suggest contacting Datastax.
>
> On 11 July 2018 at 19:56, Nitan Kainth  wrote:
>
>> Hi DuyHai,
>>
>> Could you please explain in what case C* will error based on documented
>> statement:
>>
>> You can set a default TTL for an entire table by setting the table's
>> default_time_to_live
>> <https://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlCreateTable.html#tabProp__cqlTableDefaultTTL>
>>  property. If you try to set a TTL for a specific column that is longer
>> than the time defined by the table TTL, Cassandra returns an error.
>>
>>
>>
>> On Wed, Jul 11, 2018 at 2:34 PM, DuyHai Doan 
>> wrote:
>>
>>> default_time_to_live
>>> <https://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlCreateTable.html#tabProp__cqlTableDefaultTTL>
>>>  property applies if you don't specify any TTL on your CQL statement
>>>
>>> However you can always override the default_time_to_live
>>> <https://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlCreateTable.html#tabProp__cqlTableDefaultTTL>
>>>  property by specifying a custom value for each CQL statement
>>>
>>> The behavior is correct, nothing wrong here
>>>
>>> On Wed, Jul 11, 2018 at 7:31 PM, Nitan Kainth 
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> As per document: https://docs.datastax.com/en/cql/3.3/cql/cql_using
>>>> /useExpireExample.html
>>>>
>>>>
>>>>-
>>>>
>>>>You can set a default TTL for an entire table by setting the table's
>>>> default_time_to_live
>>>>
>>>> <https://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlCreateTable.html#tabProp__cqlTableDefaultTTL>
>>>> property. If you try to set a TTL for a specific column that is
>>>>longer than the time defined by the table TTL, Cassandra returns an 
>>>> error.
>>>>
>>>>
>>>> When I tried to test this statement, i found, we can insert data with
>>>> TTL greater than default_time_to_live. Is the document needs correction, or
>>>> am I mis-understanding it?
>>>>
>>>> CREATE TABLE test (
>>>>
>>>> name text PRIMARY KEY,
>>>>
>>>> description text
>>>>
>>>> ) WITH bloom_filter_fp_chance = 0.01
>>>>
>>>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>>>>
>>>> AND comment = ''
>>>>
>>>> AND compaction = {'class': 'org.apache.cassandra.db.compa
>>>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32',
>>>> 'min_threshold': '4'}
>>>>
>>>> AND compression = {'chunk_length_in_kb': '64', 'class': '
>>>> org.apache.cassandra.io.compress.LZ4Compressor'}
>>>>
>>>> AND crc_check_chance = 1.0
>>>>
>>>> AND dclocal_read_repair_chance = 0.1
>>>>
>>>> AND default_time_to_live = 240
>>>>
>>>> AND gc_grace_seconds = 864000
>>>>
>>>> AND max_index_interval = 2048
>>>>
>>>> AND memtable_flush_period_in_ms = 0
>>>>
>>>> AND min_index_interval = 128
>>>>
>>>> AND read_repair_chance = 0.0
>>>>
>>>> AND speculative_retry = '99PERCENTILE';
>>>>
>>>> insert into test (name, description) values ('name5', 'name
>>>> description5') using ttl 360;
>>>>
>>>> select * from test ;
>>>>
>>>>
>>>>  name  | description
>>>>
>>>> ---+---
>>>>
>>>>  name5 | name description5
>>>>
>>>>
>>>> SELECT TTL (description) from test;
>>>>
>>>>
>>>>  ttl(description)
>>>>
>>>> --
>>>>
>>>>  351
>>>>
>>>> Can someone please clear this for me?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>

Re: default_time_to_live vs TTL on insert statement

[Nitan Kainth]

Hi DuyHai,

Could you please explain in what case C* will error based on documented
statement:

You can set a default TTL for an entire table by setting the table's
default_time_to_live
<https://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlCreateTable.html#tabProp__cqlTableDefaultTTL>
 property. If you try to set a TTL for a specific column that is longer
than the time defined by the table TTL, Cassandra returns an error.



On Wed, Jul 11, 2018 at 2:34 PM, DuyHai Doan  wrote:

> default_time_to_live
> <https://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlCreateTable.html#tabProp__cqlTableDefaultTTL>
>  property applies if you don't specify any TTL on your CQL statement
>
> However you can always override the default_time_to_live
> <https://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlCreateTable.html#tabProp__cqlTableDefaultTTL>
>  property by specifying a custom value for each CQL statement
>
> The behavior is correct, nothing wrong here
>
> On Wed, Jul 11, 2018 at 7:31 PM, Nitan Kainth 
> wrote:
>
>> Hi,
>>
>> As per document: https://docs.datastax.com/en/cql/3.3/cql/cql_
>> using/useExpireExample.html
>>
>>
>>-
>>
>>You can set a default TTL for an entire table by setting the table's
>>default_time_to_live
>>
>> <https://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlCreateTable.html#tabProp__cqlTableDefaultTTL>
>> property. If you try to set a TTL for a specific column that is
>>longer than the time defined by the table TTL, Cassandra returns an error.
>>
>>
>> When I tried to test this statement, i found, we can insert data with TTL
>> greater than default_time_to_live. Is the document needs correction, or am
>> I mis-understanding it?
>>
>> CREATE TABLE test (
>>
>> name text PRIMARY KEY,
>>
>> description text
>>
>> ) WITH bloom_filter_fp_chance = 0.01
>>
>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>>
>> AND comment = ''
>>
>> AND compaction = {'class': 'org.apache.cassandra.db.compa
>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32',
>> 'min_threshold': '4'}
>>
>> AND compression = {'chunk_length_in_kb': '64', 'class': '
>> org.apache.cassandra.io.compress.LZ4Compressor'}
>>
>> AND crc_check_chance = 1.0
>>
>> AND dclocal_read_repair_chance = 0.1
>>
>> AND default_time_to_live = 240
>>
>> AND gc_grace_seconds = 864000
>>
>> AND max_index_interval = 2048
>>
>> AND memtable_flush_period_in_ms = 0
>>
>> AND min_index_interval = 128
>>
>> AND read_repair_chance = 0.0
>>
>> AND speculative_retry = '99PERCENTILE';
>>
>> insert into test (name, description) values ('name5', 'name
>> description5') using ttl 360;
>>
>> select * from test ;
>>
>>
>>  name  | description
>>
>> ---+---
>>
>>  name5 | name description5
>>
>>
>> SELECT TTL (description) from test;
>>
>>
>>  ttl(description)
>>
>> --
>>
>>  351
>>
>> Can someone please clear this for me?
>>
>>
>>
>>
>>
>>
>

default_time_to_live vs TTL on insert statement

[Nitan Kainth]

Hi,

As per document:
https://docs.datastax.com/en/cql/3.3/cql/cql_using/useExpireExample.html


   -

   You can set a default TTL for an entire table by setting the table's
   default_time_to_live
   

property. If you try to set a TTL for a specific column that is longer
   than the time defined by the table TTL, Cassandra returns an error.


When I tried to test this statement, i found, we can insert data with TTL
greater than default_time_to_live. Is the document needs correction, or am
I mis-understanding it?

CREATE TABLE test (

name text PRIMARY KEY,

description text

) WITH bloom_filter_fp_chance = 0.01

AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}

AND comment = ''

AND compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
'max_threshold': '32', 'min_threshold': '4'}

AND compression = {'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}

AND crc_check_chance = 1.0

AND dclocal_read_repair_chance = 0.1

AND default_time_to_live = 240

AND gc_grace_seconds = 864000

AND max_index_interval = 2048

AND memtable_flush_period_in_ms = 0

AND min_index_interval = 128

AND read_repair_chance = 0.0

AND speculative_retry = '99PERCENTILE';

insert into test (name, description) values ('name5', 'name description5')
using ttl 360;

select * from test ;


 name  | description

---+---

 name5 | name description5


SELECT TTL (description) from test;


 ttl(description)

--

 351

Can someone please clear this for me?

Re: Problem to activate mode DEBUG to see the slow queries

[Nitan Kainth]

You can also enable traceprobability: /opt/cassandra/bin/nodetool
settraceprobability 1
It will populate system_traces keyspace where you can see details on queries

On Thu, Jun 28, 2018 at 5:32 AM, Jean Carlo 
wrote:

> Thank you ahmed!
>
>
> Saludos
>
> Jean Carlo
>
> "The best way to predict the future is to invent it" Alan Kay
>
> On Thu, Jun 28, 2018 at 11:21 AM, Ahmed Eljami 
> wrote:
>
>> ​Hello Jean Carlo,
>>
>> To activate Debug mode,  you should edit "logback.xml " not
>> "log4j-server.properties"
>>
>>
>> Ahmed.
>>
>
>

Re: copy sstables while cassandra is running

[Nitan Kainth]

Yeah you are right Pradeep. I overlooked the point there. 

Sent from my iPhone

> On Jun 24, 2018, at 1:10 PM, Pradeep Chhetri  wrote:
> 
> I doubt mv will run instantly because copy is across two different filesystems
> 
>> On Sun, 24 Jun 2018 at 9:26 PM, Nitan Kainth  wrote:
>> To be safe you could follow below prices on each node one at a time:
>> Stop Cassandra
>> Move sstable— mv will do it instantly
>> Start Cassandra
>> 
>> If you do it online and a read trust comes for sane data that is being moved 
>> will fail.
>> 
>>> On Jun 23, 2018, at 11:23 PM, onmstester onmstester  
>>> wrote:
>>> 
>>> Hi 
>>> I'm using two directories on different disks as cassandra data storage, the 
>>> small disk is 90% full and the bigger diskis 30% full (the bigger one was 
>>> added later that we find out we need more storage!!),
>>> so i want to move all data to the big disk, one way is to stop my 
>>> application and copy all sstables from small disk to big one, but it would 
>>> take some hours and not acceptable due to QoS.
>>> I thought maybe i could copy the big sstables (the one that won't be 
>>> compact in weeks) to the big disk (near casssandra data but not right 
>>> there) while cassandra and my app are still running
>>> , then stop cassandra and my app, move big file to exact directory of 
>>> cassandra data on big disk (would take a few seconds) and then move 
>>> remained small sstables from  small disk to big one.
>>> Are there all of sstables related file immutable and (data, index, summary, 
>>> ...) would only be changed by compactions? Any better workaround for this 
>>> scenario would be appriciated?
>>> Thanks in Advance
>>> 
>>> Sent using Zoho Mail
>>> 
>>> 
>>> 

Re: copy sstables while cassandra is running

[Nitan Kainth]

To be safe you could follow below prices on each node one at a time:
Stop Cassandra
Move sstable— mv will do it instantly
Start Cassandra

If you do it online and a read trust comes for sane data that is being moved 
will fail.

> On Jun 23, 2018, at 11:23 PM, onmstester onmstester  
> wrote:
> 
> Hi 
> I'm using two directories on different disks as cassandra data storage, the 
> small disk is 90% full and the bigger diskis 30% full (the bigger one was 
> added later that we find out we need more storage!!),
> so i want to move all data to the big disk, one way is to stop my application 
> and copy all sstables from small disk to big one, but it would take some 
> hours and not acceptable due to QoS.
> I thought maybe i could copy the big sstables (the one that won't be compact 
> in weeks) to the big disk (near casssandra data but not right there) while 
> cassandra and my app are still running
> , then stop cassandra and my app, move big file to exact directory of 
> cassandra data on big disk (would take a few seconds) and then move remained 
> small sstables from  small disk to big one.
> Are there all of sstables related file immutable and (data, index, summary, 
> ...) would only be changed by compactions? Any better workaround for this 
> scenario would be appriciated?
> Thanks in Advance
> 
> Sent using Zoho Mail
> 
> 
> 

Re: Timestamp on hints file and system.hints table data

[Nitan Kainth]

Kurt,

Hint file matches UUID matches with another node in the cluster:

-rw-r--r--. 1 root root  6848246 May 13 23:37
1b694180-210a-4b75-8f2a-748f4a5b6a3d-1526254645089-1.hints

/opt/cassandra/bin/nodetool status |grep 1b694180

UN  x.x.x.   23.77 GiB  256  ?
1b694180-210a-4b75-8f2a-748f4a5b6a3d  RAC1



On Thu, Jun 14, 2018 at 12:45 AM, kurt greaves  wrote:

> Does the UUID on the filename correspond with a UUID in nodetool status?
>
> Sounds to me like it could be something weird with an old node that no
> longer exists, although hints for old nodes are meant to be cleaned up.
>
> On 14 June 2018 at 01:54, Nitan Kainth  wrote:
>
>> Kurt,
>>
>> No node is down for months. And yes, I am surprised to look at Unix
>> timestamp on files.
>>
>>
>>
>> On Jun 13, 2018, at 6:41 PM, kurt greaves  wrote:
>>
>> system.hints is not used in Cassandra 3. Can't explain the files though,
>> are you referring to the files timestamp or the Unix timestamp in the file
>> name? Is there a node that's been down for several months?
>>
>> On Wed., 13 Jun. 2018, 23:41 Nitan Kainth,  wrote:
>>
>>> Hi,
>>>
>>> I observed a strange behavior about stored hints.
>>>
>>> Time stamp of hints file shows several months old. I deleted them and
>>> saw new hints files created with same old date. Why is that?
>>>
>>> Also, I see hints files on disk but if I query system.hints table, it
>>> shows 0 rows. Why system.hints is not populated?
>>>
>>> Version 3.11-1
>>>
>>
>

Re: Options to replace hardware of the cluster

[Nitan Kainth]

Both are good options, a new data center in same cluster will work faster as 
you can stream multiple nodes.
Or you can add 8 nodes to existing data center and then decommission one node 
at a time but this is lengthy process including cleanup after decommission.
One node at a time is equally good. If data isn’t much this step can be 
efficient.

One approach we did was:
RSync a temporary storage volume online
Stop c*
Rsync delta
Detach temp storage
Start a new node with new hardware
Attach volume
Rsync to actual storage

Repeat process on each node one at a time


Sent from my iPhone

> On Jun 14, 2018, at 4:10 AM, Christian Lorenz  
> wrote:
> 
> Hi,
>  
> we need to move our existing cassandra cluster to new hardware nodes. 
> Currently the cluster size is 8 members, they need to be moved to 8 new 
> machines. Cassandra version in use is 3.11.1.  Unfortunately we use 
> materialized views in production. I know that they have been marked 
> retroactively as experimental.
> What is a good way to move to the new machines? One-by-One, or setup a new 
> cluster as a separate DC? The move should be done without downtime of the 
> application.
>  
> Do you have some advice for this kind of maintenance task?
>  
> Kind regards,
> Christian

Re: Timestamp on hints file and system.hints table data

[Nitan Kainth]

Kurt,

No node is down for months. And yes, I am surprised to look at Unix timestamp 
on files.


> On Jun 13, 2018, at 6:41 PM, kurt greaves  wrote:
> 
> system.hints is not used in Cassandra 3. Can't explain the files though, are 
> you referring to the files timestamp or the Unix timestamp in the file name? 
> Is there a node that's been down for several months?
> 
>> On Wed., 13 Jun. 2018, 23:41 Nitan Kainth,  wrote:
>> Hi,
>> 
>> I observed a strange behavior about stored hints. 
>> 
>> Time stamp of hints file shows several months old. I deleted them and saw 
>> new hints files created with same old date. Why is that?
>> 
>> Also, I see hints files on disk but if I query system.hints table, it shows 
>> 0 rows. Why system.hints is not populated?
>> 
>> Version 3.11-1

Timestamp on hints file and system.hints table data

[Nitan Kainth]

Hi,

I observed a strange behavior about stored hints.

Time stamp of hints file shows several months old. I deleted them and saw
new hints files created with same old date. Why is that?

Also, I see hints files on disk but if I query system.hints table, it shows
0 rows. Why system.hints is not populated?

Version 3.11-1

Re: Restoring snapshot

[Nitan Kainth]

Change RF fir k2 and then see.

Sent from my iPhone

> On Jun 13, 2018, at 7:05 AM,  
>  wrote:
> 
> For both K1 and K2, replication factor is 2 in the new cluster(although the 
> number of nodes is 1). I can understand the portion of the warning which says 
> that “only 1 replica could be found” but the question is, why is it giving 
> the name of keyspace K2 when I was restoring only K1(It should have given 
> warning for K1).
>  
> From: Nitan Kainth [mailto:nitankai...@gmail.com] 
> Sent: Wednesday, June 13, 2018 4:31 PM
> To: user@cassandra.apache.org
> Subject: Re: Restoring snapshot
>  
> Verify dc name and replication factor in create keyspace command in new 
> cluster.
> 
> Sent from my iPhone
> 
> On Jun 13, 2018, at 2:40 AM,  
>  wrote:
> 
> Dear Community,
>  
> I took a snapshot from a node which was part of a 2 node cluster. There were 
> 2 keyspaces in that cluster K1 and K2. I took snapshot of K1 only. Now I 
> create both keyspaces in some other cluster having only one node. When I 
> tried to restore the snapshot(of keyspace K1) in that cluster using 
> sstableloader, I got a warning:
>  
> “WARN  11:55:48,921 Error while computing token map for keyspace K2 with 
> datacenter dc1: could not achieve replication factor 2 (found 1 replicas 
> only), check your keyspace replication settings.”
>  
> Like I’ve said above, the new cluster contains only one node, therefore I can 
> understand the portion of the warning telling me that ‘it only found 1 
> replicas only’, but why is it computing token map for keyspace K2 when I was 
> restoring sstables of keyspace K1? Also, the same warning(regarding only K2) 
> is displayed whether I try to restore snapshot of K1 or K2.  Although, I’m 
> able to get the complete data, but I would appreciate if someone can explain 
> this observations.
>  
> Cassandra version: 3.11.2
>  
> Thanks and regards,
> Vishal Sharma
> 
> "Confidentiality Warning: This message and any attachments are intended only 
> for the use of the intended recipient(s), are confidential and may be 
> privileged. If you are not the intended recipient, you are hereby notified 
> that any review, re-transmission, conversion to hard copy, copying, 
> circulation or other use of this message and any attachments is strictly 
> prohibited. If you are not the intended recipient, please notify the sender 
> immediately by return email and delete this message and any attachments from 
> your system.
> 
> Virus Warning: Although the company has taken reasonable precautions to 
> ensure no viruses are present in this email. The company cannot accept 
> responsibility for any loss or damage arising from the use of this email or 
> attachment."
> 
> 
> "Confidentiality Warning: This message and any attachments are intended only 
> for the use of the intended recipient(s), are confidential and may be 
> privileged. If you are not the intended recipient, you are hereby notified 
> that any review, re-transmission, conversion to hard copy, copying, 
> circulation or other use of this message and any attachments is strictly 
> prohibited. If you are not the intended recipient, please notify the sender 
> immediately by return email and delete this message and any attachments from 
> your system.
> 
> Virus Warning: Although the company has taken reasonable precautions to 
> ensure no viruses are present in this email. The company cannot accept 
> responsibility for any loss or damage arising from the use of this email or 
> attachment."

Re: Restoring snapshot

[Nitan Kainth]

Verify dc name and replication factor in create keyspace command in new cluster.

Sent from my iPhone

> On Jun 13, 2018, at 2:40 AM,  
>  wrote:
> 
> Dear Community,
>  
> I took a snapshot from a node which was part of a 2 node cluster. There were 
> 2 keyspaces in that cluster K1 and K2. I took snapshot of K1 only. Now I 
> create both keyspaces in some other cluster having only one node. When I 
> tried to restore the snapshot(of keyspace K1) in that cluster using 
> sstableloader, I got a warning:
>  
> “WARN  11:55:48,921 Error while computing token map for keyspace K2 with 
> datacenter dc1: could not achieve replication factor 2 (found 1 replicas 
> only), check your keyspace replication settings.”
>  
> Like I’ve said above, the new cluster contains only one node, therefore I can 
> understand the portion of the warning telling me that ‘it only found 1 
> replicas only’, but why is it computing token map for keyspace K2 when I was 
> restoring sstables of keyspace K1? Also, the same warning(regarding only K2) 
> is displayed whether I try to restore snapshot of K1 or K2.  Although, I’m 
> able to get the complete data, but I would appreciate if someone can explain 
> this observations.
>  
> Cassandra version: 3.11.2
>  
> Thanks and regards,
> Vishal Sharma
> 
> "Confidentiality Warning: This message and any attachments are intended only 
> for the use of the intended recipient(s), are confidential and may be 
> privileged. If you are not the intended recipient, you are hereby notified 
> that any review, re-transmission, conversion to hard copy, copying, 
> circulation or other use of this message and any attachments is strictly 
> prohibited. If you are not the intended recipient, please notify the sender 
> immediately by return email and delete this message and any attachments from 
> your system.
> 
> Virus Warning: Although the company has taken reasonable precautions to 
> ensure no viruses are present in this email. The company cannot accept 
> responsibility for any loss or damage arising from the use of this email or 
> attachment."

Re: Read Latency Doubles After Shrinking Cluster and Never Recovers

[Nitan Kainth]

I think it would because it Cassandra will process more sstables to create 
response to read queries.

Now after clean if the data volume is same and compaction has been running, I 
can’t think of any more diagnostic step. Let’s wait for other experts to 
comment.

Can you also check sstable count for each table just to be sure that they are 
not extraordinarily high?

Sent from my iPhone

> On Jun 11, 2018, at 10:21 AM, Fd Habash  wrote:
> 
> Yes we did after adding the three nodes back and a full cluster repair as 
> well.
>  
> But even it we didn’t run cleanup, would it have impacted read latency the 
> fact that some nodes still have sstables that they no longer need?
>  
> Thanks
>  
> ----
> Thank you
>  
> From: Nitan Kainth
> Sent: Monday, June 11, 2018 10:18 AM
> To: user@cassandra.apache.org
> Subject: Re: Read Latency Doubles After Shrinking Cluster and Never Recovers
>  
> Did you run cleanup too? 
>  
> On Mon, Jun 11, 2018 at 10:16 AM, Fred Habash  wrote:
> I have hit dead-ends every where I turned on this issue. 
>  
> We had a 15-node cluster  that was doing 35 ms all along for years. At some 
> point, we made a decision to shrink it to 13. Read latency rose to near 70 
> ms. Shortly after, we decided this was not acceptable, so we added the three 
> nodes back in. Read latency dropped to near 50 ms and it has been hovering 
> around this value for over 6 months now.
>  
> Repairs run regularly, load on cluster nodes is even,  application activity 
> profile has not changed. 
>  
> Why are we unable to get back the same read latency now that the cluster is 
> 15 nodes large same as it was before?
>  
> --
>  
> 
> Thank you
> 
> 
>  
>  

Re: Read Latency Doubles After Shrinking Cluster and Never Recovers

[Nitan Kainth]

Did you run cleanup too?

On Mon, Jun 11, 2018 at 10:16 AM, Fred Habash  wrote:

> I have hit dead-ends every where I turned on this issue.
>
> We had a 15-node cluster  that was doing 35 ms all along for years. At
> some point, we made a decision to shrink it to 13. Read latency rose to
> near 70 ms. Shortly after, we decided this was not acceptable, so we added
> the three nodes back in. Read latency dropped to near 50 ms and it has been
> hovering around this value for over 6 months now.
>
> Repairs run regularly, load on cluster nodes is even,  application
> activity profile has not changed.
>
> Why are we unable to get back the same read latency now that the cluster
> is 15 nodes large same as it was before?
>
> --
>
> 
> Thank you
>
>
>

Re: How to identify which table causing Maximum Memory usage limit

[Nitan Kainth]

Sorry, I didn't mean to high jack the thread. But I have seen similar
issues and ignore it always because it wasn't really causing any issues.
But I am really curious on how to find these.

On Mon, Jun 11, 2018 at 9:45 AM, Nitan Kainth  wrote:

> thanks Martin.
>
> 99 percentile of all tables are even size. Max is always higher in all
> tables.
>
> The question is, How do I identify, which table is throwing this "Maximum
> memory usage reached (512.000MiB)" usage message?
>
> On Mon, Jun 11, 2018 at 5:59 AM, Martin Mačura  wrote:
>
>> Hi,
>> we've had this issue with large partitions (100 MB and more).  Use
>> nodetool tablehistograms to find partition sizes for each table.
>>
>> If you have enough heap space to spare, try increasing this parameter:
>> file_cache_size_in_mb: 512
>>
>> There's also the following parameter, but I did not test the impact yet:
>> buffer_pool_use_heap_if_exhausted: true
>>
>>
>> Regards,
>>
>> Martin
>>
>>
>> On Tue, Jun 5, 2018 at 3:54 PM, learner dba
>>  wrote:
>> > Hi,
>> >
>> > We see this message often, cluster has multiple keyspaces and column
>> > families;
>> > How do I know which CF is causing this?
>> > Or it could be something else?
>> > Do we need to worry about this message?
>> >
>> > INFO  [CounterMutationStage-1] 2018-06-05 13:36:35,983
>> NoSpamLogger.java:91
>> > - Maximum memory usage reached (512.000MiB), cannot allocate chunk of
>> > 1.000MiB
>> >
>> >
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>
>

Re: How to identify which table causing Maximum Memory usage limit

[Nitan Kainth]

thanks Martin.

99 percentile of all tables are even size. Max is always higher in all
tables.

The question is, How do I identify, which table is throwing this "Maximum
memory usage reached (512.000MiB)" usage message?

On Mon, Jun 11, 2018 at 5:59 AM, Martin Mačura  wrote:

> Hi,
> we've had this issue with large partitions (100 MB and more).  Use
> nodetool tablehistograms to find partition sizes for each table.
>
> If you have enough heap space to spare, try increasing this parameter:
> file_cache_size_in_mb: 512
>
> There's also the following parameter, but I did not test the impact yet:
> buffer_pool_use_heap_if_exhausted: true
>
>
> Regards,
>
> Martin
>
>
> On Tue, Jun 5, 2018 at 3:54 PM, learner dba
>  wrote:
> > Hi,
> >
> > We see this message often, cluster has multiple keyspaces and column
> > families;
> > How do I know which CF is causing this?
> > Or it could be something else?
> > Do we need to worry about this message?
> >
> > INFO  [CounterMutationStage-1] 2018-06-05 13:36:35,983
> NoSpamLogger.java:91
> > - Maximum memory usage reached (512.000MiB), cannot allocate chunk of
> > 1.000MiB
> >
> >
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: Cassandra 3.0.X migarte to VPC

[Nitan Kainth]

I meant migrating to gosspsnitch during adding new dc. New dc will be empty so 
all the data will be streamed based on snitch property chosen

Sent from my iPhone

> On Jun 7, 2018, at 12:23 PM, Jeff Jirsa  wrote:
> 
> I don't know about "new snitch protocol" (there is no snitch protocol), but 
> in the case of ec2snitch and gossipingpropertyfilesnitch (do not extrapolate 
> this to any other combination), you can switch them 1:1 before the move, and 
> everything should be fine, as long as care is taken to make the property file 
> match the aws region/az discovered.
> 
> If there were other snitches involved (especially SimpleSnitch), things get 
> much more complicated. 
> 
> 
> 
> 
>> On Thu, Jun 7, 2018 at 9:18 AM, Nitan Kainth  wrote:
>> Jeff, 
>> 
>> In this case, if Riccardo is adding new DC, then he can pickup new snitch 
>> protocol right?
>> 
>>> On Thu, Jun 7, 2018 at 12:15 PM, Jeff Jirsa  wrote:
>>> 
>>> 
>>> 
>>>> On Thu, Jun 7, 2018 at 9:12 AM, Nitan Kainth  wrote:
>>>> Riccardo,
>>>> 
>>>> Simplest method can be is to add VPC as an additional datacenter to 
>>>> existing cluster; once New DC has synced up all data, just switch over 
>>>> your application. Only caveat is that there should be enough network 
>>>> bandwidth between EC2 and VPC.
>>>> 
>>>> Horizontal scaling will help to some extent. 
>>>> 
>>>> Changing snitch will cause data movement but for new DC, I believe you can 
>>>> chose. Leaving others comment.
>>> 
>>> If you change the snitch, and the data ownership changes, you run the risk 
>>> of violating at least consistency guarantees, if not outright losing data.
>>> 
>>> Changing the snitch is only something you should attempt if you understand 
>>> the implications. Cassandra does NOT re-stream data to the right places 
>>> if/when you change a snitch. 
>>> 
>>>  
>>>> 
>>>> With EBS, use right instance where IOPs are as high as possible. 
>>>> Generally, SSDs IOPs are 75K; what is best for you depends on workload. I 
>>>> have seen 20K IOPs supporting production loads in reasonable cluster.
>>> 
>>> This is very much workload dependent. I know of people running thousands of 
>>> nodes / petabytes of EBS on 4T GP2 volumes which provide 10k iops, and they 
>>> RARELY go above 1k iops during normal use patterns.
>>> 
>>> -  Jeff
>>> 
>>>  
>>>> 
>>>>> On Thu, Jun 7, 2018 at 6:20 AM, Riccardo Ferrari  
>>>>> wrote:
>>>>> Dear list,
>>>>> 
>>>>> We have a small cluster on AWS EC2-Classic and we are planning to move it 
>>>>> to a VPC. 
>>>>> 
>>>>> I know this has been discussed few times already including here:
>>>>> https://mail-archives.apache.org/mod_mbox/cassandra-user/201406.mbox/%3cca+vsrlopop7th8nx20aoz3as75g2jrjm3ryx119deklynhq...@mail.gmail.com%3E
>>>>>  - Thanks Alain RODRIGUEZ!
>>>>> or here:
>>>>> http://grokbase.com/t/cassandra/user/1498y74hn7/moving-cassandra-from-ec2-classic-into-vpc
>>>>>  - Thanks Ben Bromhead!
>>>>> 
>>>>> However I can't find anyone suggesting ClassicLink to perform such 
>>>>> migration, maybe because was recent addition compared to those posts. So 
>>>>> current status is:
>>>>> 
>>>>> 3 nodes : m1.xlarge (ephemeral storage)
>>>>> Cassandra 3.0.8
>>>>> Few keyspaces with {'class': 'SimpleStrategy', 'replication_factor': '3'}
>>>>> endpoint_snitch: Ec2Snitch
>>>>> connectivity on private ips (local_address, rpc_address, no 
>>>>> broadcast_address)
>>>>> average load pretty high (we need to scale up)
>>>>> vnodes
>>>>> My idea is to migrate:
>>>>> Add intances on to the existing cluster (1 by 1):
>>>>> Same cassandra version
>>>>> Same cluster name
>>>>> Seed list from ec2 classic list
>>>>> Run repair and cleanup
>>>>> Update seed list (cassandar config and services)
>>>>> Decommision old instances (1 by 1)
>>>>> Run repair and cleanup again
>>>>> 
>>>>> Question time:
>>>>> What is the best approach, in 2018, for migrating a cluster from 
>>>>> EC2Classic to VPC with 0 downtime?
>>>>> Should we scale up before or during the migration? I know I should expect 
>>>>> some load from data streaming, at the same time we're adding capacity.
>>>>> We would like to migrato to GossipingPropertyFileSnitch, can we mix them?
>>>>> Any gothca?
>>>>> We are thinking to move to EBS (due to cluster size and snapshotting 
>>>>> caoabilities). Any hardware reccomendation?
>>>>> Any suggestion is much appreciated!
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 

Re: Cassandra 3.0.X migarte to VPC

[Nitan Kainth]

Jeff,

In this case, if Riccardo is adding new DC, then he can pickup new snitch
protocol right?

On Thu, Jun 7, 2018 at 12:15 PM, Jeff Jirsa  wrote:

>
>
>
> On Thu, Jun 7, 2018 at 9:12 AM, Nitan Kainth 
> wrote:
>
>> Riccardo,
>>
>> Simplest method can be is to add VPC as an additional datacenter to
>> existing cluster; once New DC has synced up all data, just switch over your
>> application. Only caveat is that there should be enough network bandwidth
>> between EC2 and VPC.
>>
>> Horizontal scaling will help to some extent.
>>
>> Changing snitch will cause data movement but for new DC, I believe you
>> can chose. Leaving others comment.
>>
>
> If you change the snitch, and the data ownership changes, you run the risk
> of violating at least consistency guarantees, if not outright losing data.
>
> Changing the snitch is only something you should attempt if you understand
> the implications. Cassandra does NOT re-stream data to the right places
> if/when you change a snitch.
>
>
>
>>
>> With EBS, use right instance where IOPs are as high as possible.
>> Generally, SSDs IOPs are 75K; what is best for you depends on workload. I
>> have seen 20K IOPs supporting production loads in reasonable cluster.
>>
>
> This is very much workload dependent. I know of people running thousands
> of nodes / petabytes of EBS on 4T GP2 volumes which provide 10k iops, and
> they RARELY go above 1k iops during normal use patterns.
>
> -  Jeff
>
>
>
>>
>> On Thu, Jun 7, 2018 at 6:20 AM, Riccardo Ferrari 
>> wrote:
>>
>>> Dear list,
>>>
>>> We have a small cluster on AWS EC2-Classic and we are planning to move
>>> it to a VPC.
>>>
>>> I know this has been discussed few times already including here:
>>> https://mail-archives.apache.org/mod_mbox/cassandra-user/201
>>> 406.mbox/%3CCA+VSrLopop7Th8nX20aOZ3As75g2jrJm3ryX119dekLYNHq
>>> f...@mail.gmail.com%3E - Thanks Alain RODRIGUEZ!
>>> or here:
>>> http://grokbase.com/t/cassandra/user/1498y74hn7/moving-cassa
>>> ndra-from-ec2-classic-into-vpc - Thanks Ben Bromhead!
>>>
>>> However I can't find anyone suggesting ClassicLink to perform such
>>> migration, maybe because was recent addition compared to those posts. So
>>> current status is:
>>>
>>>
>>>- 3 nodes : m1.xlarge (ephemeral storage)
>>>- Cassandra 3.0.8
>>>- Few keyspaces with {'class': 'SimpleStrategy',
>>>'replication_factor': '3'}
>>>- endpoint_snitch: Ec2Snitch
>>>- connectivity on private ips (local_address, rpc_address, no
>>>broadcast_address)
>>>- average load pretty high (we need to scale up)
>>>- vnodes
>>>
>>> My idea is to migrate:
>>>
>>>- Add intances on to the existing cluster (1 by 1):
>>>   - Same cassandra version
>>>   - Same cluster name
>>>   - Seed list from ec2 classic list
>>>- Run repair and cleanup
>>>- Update seed list (cassandar config and services)
>>>- Decommision old instances (1 by 1)
>>>- Run repair and cleanup again
>>>
>>>
>>> Question time:
>>>
>>>- What is the best approach, in 2018, for migrating a cluster from
>>>EC2Classic to VPC with 0 downtime?
>>>- Should we scale up before or during the migration? I know I should
>>>expect some load from data streaming, at the same time we're adding
>>>capacity.
>>>- We would like to migrato to GossipingPropertyFileSnitch, can we
>>>mix them?
>>>- Any gothca?
>>>- We are thinking to move to EBS (due to cluster size and
>>>snapshotting caoabilities). Any hardware reccomendation?
>>>
>>> Any suggestion is much appreciated!
>>>
>>>
>>>
>>>
>>
>

Re: Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.

[Nitan Kainth]

Check logs in your nodes, there could be GC or other issues causing
timeouts. This can happen for various reasons, network blips, node down,
node over loaded etc.

On Thu, Jun 7, 2018 at 6:16 AM, Saurav Suman  wrote:

> Hi Team,
>
>
> I am getting below error in all the nodes of cassandra . Can anyone let me
> know what could be the cause. I am using apache cassandra 3.0.10.
>
>
>
> Jun 06, 2018 2:59:04 PM com.google.common.cache.LocalCache$Segment$1 run
> WARNING: Exception thrown during refresh
> java.util.concurrent.ExecutionException: java.lang.RuntimeException:
> org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out
> - received only 0 responses.
> at com.google.common.util.concurrent.AbstractFuture$
> Sync.getValue(AbstractFuture.java:299)
> at com.google.common.util.concurrent.AbstractFuture$
> Sync.get(AbstractFuture.java:286)
> at com.google.common.util.concurrent.AbstractFuture.get(
> AbstractFuture.java:116)
> at com.google.common.util.concurrent.Uninterruptibles.
> getUninterruptibly(Uninterruptibles.java:137)
> at com.google.common.cache.LocalCache$Segment.
> getAndRecordStats(LocalCache.java:2348)
> at com.google.common.cache.LocalCache$Segment$1.run(
> LocalCache.java:2331)
> at com.google.common.util.concurrent.MoreExecutors$
> DirectExecutor.execute(MoreExecutors.java:457)
> at com.google.common.util.concurrent.ExecutionList.
> executeListener(ExecutionList.java:156)
> at com.google.common.util.concurrent.ExecutionList.
> execute(ExecutionList.java:145)
> at com.google.common.util.concurrent.AbstractFuture.
> setException(AbstractFuture.java:202)
> at com.google.common.util.concurrent.Futures$
> ChainingListenableFuture.run(Futures.java:902)
> at com.google.common.util.concurrent.MoreExecutors$
> DirectExecutor.execute(MoreExecutors.java:457)
> at com.google.common.util.concurrent.ExecutionList.
> executeListener(ExecutionList.java:156)
> at com.google.common.util.concurrent.ExecutionList.
> execute(ExecutionList.java:145)
> at com.google.common.util.concurrent.ListenableFutureTask.done(
> ListenableFutureTask.java:91)
> at java.util.concurrent.FutureTask.finishCompletion(
> FutureTask.java:384)
> at java.util.concurrent.FutureTask.setException(FutureTask.java:251)
> at java.util.concurrent.FutureTask.run(FutureTask.java:271)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:624)
> at org.apache.cassandra.concurrent.NamedThreadFactory.
> lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: 
> org.apache.cassandra.exceptions.ReadTimeoutException:
> Operation timed out - received only 0 responses.
> at org.apache.cassandra.auth.CassandraRoleManager.getRole(
> CassandraRoleManager.java:516)
> at org.apache.cassandra.auth.CassandraRoleManager.isSuper(
> CassandraRoleManager.java:305)
> at org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:52)
> at org.apache.cassandra.auth.AuthenticatedUser.isSuper(
> AuthenticatedUser.java:71)
> at org.apache.cassandra.auth.CassandraAuthorizer.authorize(
> CassandraAuthorizer.java:76)
> at org.apache.cassandra.auth.PermissionsCache$1$1.call(
> PermissionsCache.java:136)
> at org.apache.cassandra.auth.PermissionsCache$1$1.call(
> PermissionsCache.java:131)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 4 more
>
>
>
> *Saurav Suman*
> BigData Consultant
> Email- saurav.su...@widas.in
> 
> *WidasConcepts India Pvt. Ltd*
> UNIT II, SJR HUB, Sarjapur Road
> Bellandur, Bangalore 560103
>
>

Re: Cassandra 3.0.X migarte to VPC

[Nitan Kainth]

Riccardo,

Simplest method can be is to add VPC as an additional datacenter to
existing cluster; once New DC has synced up all data, just switch over your
application. Only caveat is that there should be enough network bandwidth
between EC2 and VPC.

Horizontal scaling will help to some extent.

Changing snitch will cause data movement but for new DC, I believe you can
chose. Leaving others comment.

With EBS, use right instance where IOPs are as high as possible. Generally,
SSDs IOPs are 75K; what is best for you depends on workload. I have seen
20K IOPs supporting production loads in reasonable cluster.

On Thu, Jun 7, 2018 at 6:20 AM, Riccardo Ferrari  wrote:

> Dear list,
>
> We have a small cluster on AWS EC2-Classic and we are planning to move it
> to a VPC.
>
> I know this has been discussed few times already including here:
> https://mail-archives.apache.org/mod_mbox/cassandra-user/
> 201406.mbox/%3CCA+VSrLopop7Th8nX20aOZ3As75g2jrJm
> 3ryx119deklynhq...@mail.gmail.com%3E - Thanks Alain RODRIGUEZ!
> or here:
> http://grokbase.com/t/cassandra/user/1498y74hn7/moving-cassandra-from-ec2-
> classic-into-vpc - Thanks Ben Bromhead!
>
> However I can't find anyone suggesting ClassicLink to perform such
> migration, maybe because was recent addition compared to those posts. So
> current status is:
>
>
>- 3 nodes : m1.xlarge (ephemeral storage)
>- Cassandra 3.0.8
>- Few keyspaces with {'class': 'SimpleStrategy', 'replication_factor':
>'3'}
>- endpoint_snitch: Ec2Snitch
>- connectivity on private ips (local_address, rpc_address, no
>broadcast_address)
>- average load pretty high (we need to scale up)
>- vnodes
>
> My idea is to migrate:
>
>- Add intances on to the existing cluster (1 by 1):
>   - Same cassandra version
>   - Same cluster name
>   - Seed list from ec2 classic list
>- Run repair and cleanup
>- Update seed list (cassandar config and services)
>- Decommision old instances (1 by 1)
>- Run repair and cleanup again
>
>
> Question time:
>
>- What is the best approach, in 2018, for migrating a cluster from
>EC2Classic to VPC with 0 downtime?
>- Should we scale up before or during the migration? I know I should
>expect some load from data streaming, at the same time we're adding
>capacity.
>- We would like to migrato to GossipingPropertyFileSnitch, can we mix
>them?
>- Any gothca?
>- We are thinking to move to EBS (due to cluster size and snapshotting
>caoabilities). Any hardware reccomendation?
>
> Any suggestion is much appreciated!
>
>
>
>

Re: how to immediately delete tombstones

[Nitan Kainth]

You can compact selective sstables using jmx Call.

Sent from my iPhone

> On Jun 2, 2018, at 12:04 AM, onmstester onmstester  
> wrote:
> 
> Thanks for your replies
> But my current situation is that i do not have enough free disk for my 
> biggest sstable, so i could not run major compaction or nodetool 
> garbagecollect
> 
> Sent using Zoho Mail
> 
> 
> 
>  On Thu, 31 May 2018 22:32:32 +0430 Alain RODRIGUEZ  
> wrote 
> 
> 

Re: Log application Queries

[Nitan Kainth]

this is awesome Jeff.


Regards,
Nitan K.
Cassandra and Oracle Architect/SME
Datastax Certified Cassandra expert
Oracle 10g Certified

On Fri, May 25, 2018 at 3:00 PM, Jeff Jirsa <jji...@gmail.com> wrote:

> https://issues.apache.org/jira/browse/CASSANDRA-13983 was added to
> cassandra 4.0
>
>
>
> On Fri, May 25, 2018 at 8:24 AM, Nitan Kainth <nitankai...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I would like to log all C* queries hitting cluster. Could someone please
>> tell me how can I do it at cluster level?
>> Will nodetool setlogginglevel work? If so, please share example with
>> library name.
>>
>> C* version 3.11
>>
>
>

Re: Log application Queries

[Nitan Kainth]

Thanks Alain


Regards,
Nitan K.
Cassandra and Oracle Architect/SME
Datastax Certified Cassandra expert
Oracle 10g Certified

On Fri, May 25, 2018 at 2:05 PM, Alain RODRIGUEZ <arodr...@gmail.com> wrote:

> Nate wrote a post about this exact topic. In case it is of some use:
> http://thelastpickle.com/blog/2016/02/10/locking-down-
> apache-cassandra-logging.html
>
> C*heers,
> ---
> Alain Rodriguez - @arodream - al...@thelastpickle.com
> France / Spain
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> 2018-05-25 18:00 GMT+01:00 Nitan Kainth <nitankai...@gmail.com>:
>
>> So settraceprobability is better option than nodetool :)
>>
>>
>> Regards,
>> Nitan K.
>> Cassandra and Oracle Architect/SME
>> Datastax Certified Cassandra expert
>> Oracle 10g Certified
>>
>> On Fri, May 25, 2018 at 12:15 PM, Surbhi Gupta <surbhi.gupt...@gmail.com>
>> wrote:
>>
>>> nodeool setlogginglevel is only valid for below :
>>>
>>>
>>>- org.apache.cassandra
>>>- org.apache.cassandra.db
>>>- org.apache.cassandra.service.StorageProxy
>>>
>>>
>>> On 25 May 2018 at 09:01, Nitan Kainth <nitankai...@gmail.com> wrote:
>>>
>>>> Thanks Surbhi. I found another way. I used nodetool
>>>> settraceprobability 1 and it is logging in system_traces.
>>>>
>>>> How is it different from nodeool setlogginglevel?
>>>>
>>>>
>>>> Regards,
>>>> Nitan K.
>>>> Cassandra and Oracle Architect/SME
>>>> Datastax Certified Cassandra expert
>>>> Oracle 10g Certified
>>>>
>>>> On Fri, May 25, 2018 at 11:41 AM, Surbhi Gupta <
>>>> surbhi.gupt...@gmail.com> wrote:
>>>>
>>>>> If using dse then u can enable in dse.yaml.
>>>>>
>>>>> # CQL slow log settings
>>>>> cql_slow_log_options:
>>>>> enabled: true
>>>>> threshold_ms: 0
>>>>> ttl_seconds: 259200
>>>>>
>>>>> As far as my understanding says setlogginglevel  is used for changing
>>>>> the logging level as below but not for slow query .
>>>>>
>>>>>- ALL
>>>>>- TRACE
>>>>>- DEBUG
>>>>>- INFO
>>>>>- WARN
>>>>>- ERROR
>>>>>- OFF
>>>>>
>>>>>
>>>>>
>>>>> On 25 May 2018 at 08:24, Nitan Kainth <nitankai...@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I would like to log all C* queries hitting cluster. Could someone
>>>>>> please tell me how can I do it at cluster level?
>>>>>> Will nodetool setlogginglevel work? If so, please share example with
>>>>>> library name.
>>>>>>
>>>>>> C* version 3.11
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Re: Log application Queries

[Nitan Kainth]

So settraceprobability is better option than nodetool :)


Regards,
Nitan K.
Cassandra and Oracle Architect/SME
Datastax Certified Cassandra expert
Oracle 10g Certified

On Fri, May 25, 2018 at 12:15 PM, Surbhi Gupta <surbhi.gupt...@gmail.com>
wrote:

> nodeool setlogginglevel is only valid for below :
>
>
>- org.apache.cassandra
>- org.apache.cassandra.db
>- org.apache.cassandra.service.StorageProxy
>
>
> On 25 May 2018 at 09:01, Nitan Kainth <nitankai...@gmail.com> wrote:
>
>> Thanks Surbhi. I found another way. I used nodetool settraceprobability
>> 1 and it is logging in system_traces.
>>
>> How is it different from nodeool setlogginglevel?
>>
>>
>> Regards,
>> Nitan K.
>> Cassandra and Oracle Architect/SME
>> Datastax Certified Cassandra expert
>> Oracle 10g Certified
>>
>> On Fri, May 25, 2018 at 11:41 AM, Surbhi Gupta <surbhi.gupt...@gmail.com>
>> wrote:
>>
>>> If using dse then u can enable in dse.yaml.
>>>
>>> # CQL slow log settings
>>> cql_slow_log_options:
>>> enabled: true
>>> threshold_ms: 0
>>> ttl_seconds: 259200
>>>
>>> As far as my understanding says setlogginglevel  is used for changing
>>> the logging level as below but not for slow query .
>>>
>>>- ALL
>>>- TRACE
>>>- DEBUG
>>>- INFO
>>>- WARN
>>>- ERROR
>>>- OFF
>>>
>>>
>>>
>>> On 25 May 2018 at 08:24, Nitan Kainth <nitankai...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I would like to log all C* queries hitting cluster. Could someone
>>>> please tell me how can I do it at cluster level?
>>>> Will nodetool setlogginglevel work? If so, please share example with
>>>> library name.
>>>>
>>>> C* version 3.11
>>>>
>>>
>>>
>>
>

Re: Log application Queries

[Nitan Kainth]

Thanks Surbhi. I found another way. I used nodetool settraceprobability 1
and it is logging in system_traces.

How is it different from nodeool setlogginglevel?


Regards,
Nitan K.
Cassandra and Oracle Architect/SME
Datastax Certified Cassandra expert
Oracle 10g Certified

On Fri, May 25, 2018 at 11:41 AM, Surbhi Gupta <surbhi.gupt...@gmail.com>
wrote:

> If using dse then u can enable in dse.yaml.
>
> # CQL slow log settings
> cql_slow_log_options:
> enabled: true
> threshold_ms: 0
> ttl_seconds: 259200
>
> As far as my understanding says setlogginglevel  is used for changing the
> logging level as below but not for slow query .
>
>- ALL
>- TRACE
>- DEBUG
>- INFO
>- WARN
>- ERROR
>- OFF
>
>
>
> On 25 May 2018 at 08:24, Nitan Kainth <nitankai...@gmail.com> wrote:
>
>> Hi,
>>
>> I would like to log all C* queries hitting cluster. Could someone please
>> tell me how can I do it at cluster level?
>> Will nodetool setlogginglevel work? If so, please share example with
>> library name.
>>
>> C* version 3.11
>>
>
>

Log application Queries

[Nitan Kainth]

Hi,

I would like to log all C* queries hitting cluster. Could someone please
tell me how can I do it at cluster level?
Will nodetool setlogginglevel work? If so, please share example with
library name.

C* version 3.11

Re: Snapshot SSTable modified??

[Nitan Kainth]

Jeff,

Shouldn't Snapshot get consistent state of sstables? -tmp file shouldn't
impact backup operation right?


Regards,
Nitan K.
Cassandra and Oracle Architect/SME
Datastax Certified Cassandra expert
Oracle 10g Certified

On Wed, May 23, 2018 at 6:26 PM, Jeff Jirsa  wrote:

> In versions before 3.0, sstables were written with a -tmp filename and
> copied/moved to the final filename when complete. This changes in 3.0 - we
> write into the file with the final name, and have a journal/log to let uss
> know when it's done/final/live.
>
> Therefore, you can no longer just watch for a -Data.db file to be created
> and uploaded - you have to watch the log to make sure it's not being
> written.
>
>
> On Wed, May 23, 2018 at 2:18 PM, Max C.  wrote:
>
>> Hi Everyone,
>>
>> We’ve noticed a few times in the last few weeks that when we’re doing
>> backups, tar has complained with messages like this:
>>
>> tar: /var/lib/cassandra/data/mars/test_instances_by_test_id-6a944
>> 0a04cc111e8878675f1041d7e1c/snapshots/backup_20180523_024502/mb-63-big-Data.db:
>> file changed as we read it
>>
>> Any idea what might be causing this?
>>
>> We’re running Cassandra 3.0.8 on RHEL 7.  Here’s rough pseudocode of our
>> backup process:
>>
>> 
>> SNAPSHOT_NAME=backup_YYYMMDD_HHMMSS
>> nodetool snapshot -t $SNAPSHOT_NAME
>>
>> for each keyspace
>> - dump schema to “schema.cql"
>> - tar -czf /file_server/backup_$HOSTNAME_$KEYSPACE_MMDD_HHMMSS.tgz
>> schema.cql /var/lib/cassandra/data/$KEYSPACE/*/snapshots/$SNAPSHOT_NAME
>>
>> nodetool clearsnapshot -t $SNAPSHOT_NAME
>>
>> Thanks.
>>
>> - Max
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>
>

Re: repair in C* 3.11.2 and anticompactions

[Nitan Kainth]

Jean,

Have you considered nodetool repair -pr (primary range) OR reaper? With
Reaper you can throttle repair load on system. These two uses ranges
anyway, so you may not run into anti-compaction.


Regards,
Nitan K.
Cassandra and Oracle Architect/SME
Datastax Certified Cassandra expert
Oracle 10g Certified

On Thu, May 24, 2018 at 5:44 AM, Jean Carlo 
wrote:

>
> Thanks alain and Lerh, It is clear now.
>
> In order to avoid problems and charge in the cluster doing
> anticompactions, I am going to use repair by sub ranges.
>
> I studied this tool called cassandra-list-subranges
>  it seems it
> still works for the versions 3.11.2. And I will take a look also to
> cassandra_range_repair
>  which it is more
> recent
>
> Do you have any remarks for cassandra-list-subranges
>  ?
>
>
>
> Saludos
>
> Jean Carlo
>
> "The best way to predict the future is to invent it" Alan Kay
>
> On Thu, May 24, 2018 at 11:12 AM, Alain RODRIGUEZ 
> wrote:
>
>> Hi Jean,
>>
>> Here is what Alexander wrote about it, a few months ago, in the comments
>> of the article mentioned above:
>>
>> "A full repair is an incremental one that doesn't skip repaired data.
>>> Performing anticompaction in that case too (regardless it is a valid
>>> approach or not) allows to mark as repaired SSTables that weren't before
>>> the full repair was started.
>>>
>>> It was clearly added with the intent of making full repair part of a
>>> routine where incremental repairs are also executed, leaving only subrange
>>> for people who do not want to use incremental.
>>>
>>> One major drawback is that by doing so, the project increased the
>>> operational complexity of running full repairs as it does not allow
>>> repairing the same keyspace from 2 nodes concurrently without risking some
>>> failures during validation compaction (when an SSTable is being
>>> anticompacted, it cannot go through validation compaction)."
>>>
>>>
>>  I hope this helps,
>>
>> C*heers,
>> ---
>> Alain Rodriguez - @arodream - al...@thelastpickle.com
>> France / Spain
>>
>> The Last Pickle - Apache Cassandra Consulting
>> http://www.thelastpickle.com
>>
>> 2018-05-23 21:48 GMT+01:00 Lerh Chuan Low :
>>
>>> Hey Jean,
>>>
>>> I think it still does anticompaction by default regardless, it will not
>>> do so only if you do subrange repair. TLP wrote a pretty good article on
>>> that: http://thelastpickle.com/blog/2017/12/14/should-you-us
>>> e-incremental-repair.html
>>>
>>> On 24 May 2018 at 00:42, Jean Carlo  wrote:
>>>
 Hello

 I just want to understand why, if I run a repair non incremental like
 this

 nodetool -h 127.0.0.1 -p 7100 repair -full -pr keyspace1 standard1

 Cassandra does anticompaction as the logs show

 INFO  [CompactionExecutor:20] 2018-05-23 16:36:27,598
 CompactionManager.java:1545 - Anticompacting [BigTableReader(path='/home/jr
 iveraura/.ccm/test/node1/data0/keyspace1/standard1-36a6ec405
 e9411e8b1d1b38a73559799/mc-2-big-Data.db')]

 As far as I understood the anticompactions are used to make the repair
 incremantals possible, so I was expecting no having anticompactions making
 repairs with the options  -pr -full

 Anyone knows why does cassandra make those anticompactions ?

 Thanks

 Jean Carlo

 "The best way to predict the future is to invent it" Alan Kay

>>>
>>>
>>
>

Re: Academic paper about Cassandra database compaction

[Nitan Kainth]

Hi Lucas,

I am not able to download. can you share as attachment in email?



Regards,
Nitan K.
Cassandra and Oracle Architect/SME
Datastax Certified Cassandra expert
Oracle 10g Certified

On Mon, May 14, 2018 at 9:12 AM, Lucas Benevides <
lu...@maurobenevides.com.br> wrote:

> Dear community,
>
> I want to tell you about my paper published in a conference in March. The
> title is " NoSQL Database Performance Tuning for IoT Data - Cassandra
> Case Study"  and it is available (not for free) in
> http://www.scitepress.org/DigitalLibrary/Link.aspx?doi=
> 10.5220/0006782702770284 .
>
> TWCS is used and compared with DTCS.
>
> I hope you can download it, unfortunately I cannot send copies as the
> publisher has its copyright.
>
> Lucas B. Dias
>
>
>

Re: Insert-only application repair

[Nitan Kainth]

If you have RF>CL then Repair needs to be run to make sure data is in sync. 

Sent from my iPhone

> On May 12, 2018, at 3:54 AM, onmstester onmstester  
> wrote:
> 
> 
> In an insert-only use case with TTL (6 months), should i run this command, 
> every 5-7 days on all the nodes of production cluster (according to this: 
> http://cassandra.apache.org/doc/latest/operating/repair.html )?
> nodetool repair -pr --full
> When none of the nodes was down in 4 months (ever since the cluster was 
> launched) and none of the rows been deleted, why should i run nodetool repair?
> 

Re: what happens for concurrent node add/removal?

[Nitan Kainth]

Seed node takes care of token re-assignment. But why do you even wanna do that? 
Add the node first and then remove.

Sent from my iPhone

> On Apr 28, 2018, at 5:10 AM, Jinhua Luo  wrote:
> 
> HI All,
> 
> For example, if at the same time, two nodes joining and there nodes removing.
> How c* handles such case? c* uses gossip to determine membership, and
> it does not have central leader to serialize the membership changes.
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org