Re: Cassandra node RAM amount vs data-per-node/total data?

2018-07-17 Thread onmstester onmstester
I actually never set Xmx > 32 GB, for any java application, unless it necessarily need more. Just because of the fact: "once you exceed this 32 GiB border JVM will stop using compressed object pointers, effectively reducing the available memory. That means increasing your JVM heap above 32 GiB

Re: cassandra cluser sizing

2018-07-17 Thread Jeff Jirsa
Depends on whether or not you’re seeing overlaps (do you have read repairs messing up the windows? Do you expire cleanly? Do you do non-ttl deletes?) When I wrote it I used to run about 70-80% full, but we were willing to do some manual cleanup when overlaps prevented dropping full sstables

Re: cassandra cluser sizing

2018-07-17 Thread Eunsu Kim
Can I ask you an additional question here? How much free space should I have if most tables use TimeWindowCompactionStrategy? > On 13 Jul 2018, at 10:09 PM, Vitaliy Semochkin wrote: > > Jeff, thank you very much for reply. > Will try to use 4TB per instance. > > If I understand it correctly

Re: System auth empty, how to populate it

2018-07-17 Thread Horia Mocioi
Cassandra allows to use custom authenticators so I would create a CustomPasswordAuthenticator. This would be a copy of the existing PasswordAuthenticator. I would add several debugging info like: provided username and password, the output of the checkpw function, what cql statement is executed

Re: concurrent_compactors via JMX

2018-07-17 Thread Alain RODRIGUEZ
Hello Riccardo, I noticed I have been writing a novel to answer a simple couple of questions again ¯\_(ツ)_/¯. So here is a short answer in the case that's what you were looking for :). Also, there is a warning that it might be counter-productive and stress the cluster even more to increase the

Re: Cassandra node RAM amount vs data-per-node/total data?

2018-07-17 Thread Rahul Singh
If you have a read-heavy cluster, even with LCS you can still optimize by having more of the key / row cache in memory. If you have a write heavy / read heavy , then you need more memory so that more data is available in the memtable as its written. Not having enough memory means not enough

Re: Cassandra Repair

2018-07-17 Thread Rahul Singh
I would recommend at least looking at reaper before trying to engineer another way. On Jul 17, 2018, 12:56 PM -0400, rajasekhar kommineni , wrote: > nodetool tablestats has an attribute for Percent repaired, can we target the > tables based the % given. > > > > > On Jul 17, 2018, at 4:45 AM,

Re: Cassandra Repair

2018-07-17 Thread rajasekhar kommineni
nodetool tablestats has an attribute for Percent repaired, can we target the tables based the % given. > On Jul 17, 2018, at 4:45 AM, Rahul Singh wrote: > > Have you considered looking into reaper project — could save you time in > figuring out your own strategy. >

concurrent_compactors via JMX

2018-07-17 Thread Riccardo Ferrari
Hi list, Cassandra 3.0.6 I'd like to test the change of concurrent compactors to see if it helps when the system is under stress. Can someone point me to the right mbean? I can not really find good docs about mbeans (or tools ...) Any suggestion much appreciated, best

Re: System auth empty, how to populate it

2018-07-17 Thread Thomas Lété
Thanks for your reply, - I have not defined role_manager in the config - I dropped the users table, it was present in the keyspace - Cassandra then created a record in the roles table, yay ! But when I do clash -u cassandra -p cassandra => Invalid credentials supplied. Authentication error on

Re: System auth empty, how to populate it

2018-07-17 Thread Simon Fontana Oscarsson
Very strange. CassandraRoleManager will create cassandra user if it doesn't already exist (look in the setupDefaultRole method). Perhaps this is an issue with consistency. After starting Cassandra, could you try do a nodetool repair on system_auth keyspace on all nodes. I'm starting to run out

Re: System auth empty, how to populate it

2018-07-17 Thread Sam Tunnicliffe
The default superuser is only created at startup if 3 conditions are met: i) The default role manager is configured. In cassandra.yaml, you should see "role_manager: CassandraRoleManager". This is also the default value, so unless you're explicitly using a custom role manager it should be good.

Re: Cassandra recommended server uptime?

2018-07-17 Thread Vsevolod Filaretov
@Jeff Jirsa >>Should not need regular reboots for Cassandra’s sake, but you should be regularly patching your servers Do you mean system updates/OS tuning and corresponding reloads, or some OS cache cleanups? Best regards, Vsevolod. вт, 17 июл. 2018 г., 17:18 Jeff Jirsa : > Should not need

Re: Cassandra node RAM amount vs data-per-node/total data?

2018-07-17 Thread Vsevolod Filaretov
@Rahul Singh thank you for the answer! What is your logic behind such RAM-per-node values? What symptoms usually suggest you that you need more RAM? Did you ever get C* node soft lockdowns/not-respondings due to node being loaded up to 100% of either ram/cpu/IO? If yes - under which conditions?

Re: Cassandra node RAM amount vs data-per-node/total data?

2018-07-17 Thread Rahul Singh
I usually don’t want to put more than 1.0-1.5 TB ( at the most ) per node. It makes streaming slow beyond my patience and keeps the repair / compaction processes lean. Memory depends on how much you plan to keep in memory in terms of key / row cache. For my uses, no less than 64GB if not more ~

Re: System auth empty, how to populate it

2018-07-17 Thread Thomas Lété
Yes I did that multiple time, always following the same procedure : stop Cassandra, on all nodes, remove data, update config then restart nodes one by one… I really don’t understand when I could have done wrong... > Le 17 juil. 2018 à 16:15, Simon Fontana Oscarsson > a écrit : > > This is

Re: Cassandra recommended server uptime?

2018-07-17 Thread Jeff Jirsa
Should not need regular reboots for Cassandra’s sake, but you should be regularly patching your servers -- Jeff Jirsa > On Jul 17, 2018, at 2:09 AM, Vsevolod Filaretov wrote: > > Good time of day everyone; > > Does Cassandra have a "recommended uptime"? I.e., does regular Cassandra node

Re: System auth empty, how to populate it

2018-07-17 Thread Simon Fontana Oscarsson
This is very strange behavior if Cassandra won't recreate the cassandra user when you delete the folder. So just to make sure, you are stopping Cassandra on all nodes and deleting the data directory? -- SIMON FONTANA OSCARSSON Software Developer Ericsson Ölandsgatan 1 37133 Karlskrona, Sweden

Re: System auth empty, how to populate it

2018-07-17 Thread Thomas Lété
It’s empty... > Le 17 juil. 2018 à 15:59, Horia Mocioi a écrit : > > Could you also send the output of "select * from system_auth.roles"? > (you will need to change authenticator to AllowAllAuthenticator and > authorizer to AllowAllAuthorizer) > > On tis, 2018-07-17 at 15:43 +0200, Thomas

Re: System auth empty, how to populate it

2018-07-17 Thread Horia Mocioi
Could you also send the output of "select * from system_auth.roles"? (you will need to change authenticator to AllowAllAuthenticator and authorizer to AllowAllAuthorizer)  On tis, 2018-07-17 at 15:43 +0200, Thomas Lété wrote: > Ok I tried that, nothing better (I already tried dropping the entire

Re: System auth empty, how to populate it

2018-07-17 Thread Thomas Lété
Ok I tried that, nothing better (I already tried dropping the entire system_auth folder that way, same result) When I open the log, I found nothing about « Password » and when I search for « roles », I only find that : DEBUG [main] 2018-07-17 15:37:39,420 CompactionStrategyManager.java:380 -

Re: System auth empty, how to populate it

2018-07-17 Thread Simon Fontana Oscarsson
Could you try the following steps? Stop Cassandra. Change authenticator in yaml to PasswordAuthenticator if not already done. Remove data directory with `rm -rf data/system_auth/roles-*` Start Cassandra. Login with `cqlsh -u cassandra -p cassandra` Works for me. -- SIMON FONTANA OSCARSSON

Re: Bind keyspace to specific data directory

2018-07-17 Thread Abdul Patel
The requirement is that we plan to have a new keyspace which will hist PII data and we wanted to see if we can have encrypted ssd then can we find the new keyspace with PII can be binded to it and no chanhes would required to current keyspaces. On Tuesday, July 17, 2018, Rahul Singh wrote: >

RE: [EXTERNAL] Re: Cassandra recommended server uptime?

2018-07-17 Thread Durity, Sean R
We do not have any scheduled, periodic node restarts. I have been working on Cassandra across many versions, and I have not seen a case where periodic restarts would solve any problem that I saw. There are certainly times when a node needs a restart – but those are because of specific reasons.

Re: System auth empty, how to populate it

2018-07-17 Thread Thomas Lété
The latest : 3.11.2, the same as yours :( > Le 17 juil. 2018 à 14:36, Horia Mocioi a écrit : > > What Cassandra version do you use? > > On tis, 2018-07-17 at 14:23 +0200, Thomas Lété wrote: >> Hi, >> >> Thanks I tried that, made a node tool repair system_auth and I get a >> new error now : >>

Re: System auth empty, how to populate it

2018-07-17 Thread Horia Mocioi
What Cassandra version do you use? On tis, 2018-07-17 at 14:23 +0200, Thomas Lété wrote: > Hi, > > Thanks I tried that, made a node tool repair system_auth and I get a > new error now : > > Connection error: ('Unable to connect to any servers', {'127.0.0.1': > AuthenticationFailed('Failed to

RE: Cassandra recommended server uptime?

2018-07-17 Thread Jonathan Baynes
I haven’t restarted a node (service) unless there are changes made to config, and for restarting the server its incredibly rare. In fact since Production has been up I think ive restrted the servers twice (both for DR occurences) in a Year. There is very little need to reboot the servers and

Cassandra node RAM amount vs data-per-node/total data?

2018-07-17 Thread Vsevolod Filaretov
What are general community and/or your personal experience viewpoints on cassandra node RAM amount vs data stored per node question? Thank you very much. Best regards, Vsevolod.

Re: Cassandra recommended server uptime?

2018-07-17 Thread Vsevolod Filaretov
@rahul.xavier.si...@gmail.com, @simon.fontana.oscars...@ericsson.com, Thank you for answers. I've got a really heavy data model, but no GC problems. Moreover, gclog shows very efficient GC process and hardly any GC pauses. So, the question remains open: how general is the practice of

Re: System auth empty, how to populate it

2018-07-17 Thread Thomas Lété
Hi, Thanks I tried that, made a node tool repair system_auth and I get a new error now : Connection error: ('Unable to connect to any servers', {'127.0.0.1': AuthenticationFailed('Failed to authenticate to 127.0.0.1: Error from server: code=0100 [Bad credentials] message="Provided username

RE: [EXTERNAL] New cluster vs Increasing nodes to already existed cluster

2018-07-17 Thread Rahul Singh
You can make new clusters or you can isolate with datacenters that don’t have a keyspace replicated. On Jul 16, 2018, 10:41 AM -0400, Durity, Sean R , wrote: > In most cases, we separate clusters by application. This does help with > isolating problems. A bad query in one application won’t

Re: Cassandra Repair

2018-07-17 Thread Rahul Singh
Have you considered looking into reaper project — could save you time in figuring out your own strategy.  https://github.com/thelastpickle/cassandra-reaper Otherwise you can always do a round robin of cron jobs per node once a week… Your repair cycle should repair all servers within a window

Re: Bind keyspace to specific data directory

2018-07-17 Thread Rahul Singh
What’s the goal, Abdul? Is it for security reasons or for organizational reasons. You could try prefixing / suffixing the keyspace names if its for organizational reasons (For now) if you don’t want to do the manual management of mounts as Anthony suggested . -- Rahul Singh

Re: Cassandra recommended server uptime?

2018-07-17 Thread Rahul Singh
It’s likely that if you have server stability issues its because of data model or compaction strategy configurations which lead to out of memory issues or massive GC pauses. Rebooting wouldn’t solve those issues. -- Rahul Singh rahul.si...@anant.us Anant Corporation On Jul 17, 2018, 7:28 AM

Re: Cassandra recommended server uptime?

2018-07-17 Thread Simon Fontana Oscarsson
Not anything that I'm aware of. Cassandra can run at months/years without rebooting. It is better to monitor your nodes and if you find anything abnormal a restart can help. -- SIMON FONTANA OSCARSSON Software Developer Ericsson Ölandsgatan 1 37133 Karlskrona, Sweden

Re: System auth empty, how to populate it

2018-07-17 Thread Horia Mocioi
Try executing in cqlsh: insert into system_auth.roles (role , can_login , is_superuser , salted_hash ) VALUES ( 'cassandra', True, True, '$2a$10$qQIh9pXDu0JNA7vQd7KRcO7VXryjbTu8XBQTC.tXcTpJHliH6S1aW'); The above hash encrypted value is for the default password cassandra After doing this you

Re: System auth empty, how to populate it

2018-07-17 Thread Thomas Lété
Hi Horia, Thanks for your reply :-) As the keyspace was corrupt, Cassandra didn’t boot… So as we were in a hurry and not needed that keyspace, we dropped the files and put AllowAllAuthorizer. Now even the roles table is empty. When I enable PasswordAuthenticator, I’m not able to authenticate,

Re: System auth empty, how to populate it

2018-07-17 Thread Horia Mocioi
Hello, Those tables are empty by default, except system_auth.roles table which contains one entry(by default): the cassandra user/role. CassandraRoleManager creates it on startup if it does not exist. Those tables will be populated once you start adding new users/roles and use authorization.

Cassandra recommended server uptime?

2018-07-17 Thread Vsevolod Filaretov
Good time of day everyone; Does Cassandra have a "recommended uptime"? I.e., does regular Cassandra node reboots help anything? Are periodical node reboots recommended for general system stability? Best regards, Vsevolod.

System auth empty, how to populate it

2018-07-17 Thread Thomas Lété
Hi everyone, Due to a crash, we lost the system_auth keyspace. It was not important at that time because this was a closed system, not communicating outside the structure so we used AllowAllAuthorizer. Now it is changing and we would like to recover that keyspace (re-create it from scratch).