Re: Hints folder missing in Cassandra

The problem is even though, I get that O_RDONLY WARN message, if I try to navigate to the path where the hints folder is stored, the folder is not present. I cannot check the permissions on that folder, its already missing, got deleted somehow. I believe everything runs as the root user. I do see

Re: Hints folder missing in Cassandra

> The environment is built using established images for Cassandra 3.10. > Unfortunately the debug log does not indicate any errors before I start > seeing the WARN for missing hints folder. I understand that hints file will > be deleted after replay is complete, but not sure of the root cause of wh

Re: Hints folder missing in Cassandra

The environment is built using established images for Cassandra 3.10. Unfortunately the debug log does not indicate any errors before I start seeing the WARN for missing hints folder. I understand that hints file will be deleted after replay is complete, but not sure of the root cause of why the hi

Re: Hints folder missing in Cassandra

Do you have any automation being applied via chef or likewise? Was the environment built using automation or using established images ? -- Rahul Singh rahul.si...@anant.us Anant Corporation On Feb 7, 2018, 10:37 AM -0600, test user , wrote: > The directory /var/lib/cassandra/hints was created,

Re: Add column if it does not exist?

Yah. I saw one such migration via Spark Job running concurrently and created 4 Cfids and migrated data. It was a nightmare to cleanup the duplicated sstables. Alter schema and migrate should always be different applications separate from the actual system. -- Rahul Singh rahul.si...@anant.us A

Re: Bootstrapping fails with < 128GB RAM ...

It would be extremely helpful to get some info about your heap. At a bare minimum, a histogram of the heap dump would be useful, but ideally a full heap dump would be best. jmap -dump:live,format=b,file=heap.bin PID Taking a look at that in YourKit should give some pretty quick insight into

Re: Add column if it does not exist?

All of the drivers also have keyspace / table metadata. For instance: https://datastax.github.io/python-driver/api/cassandra/metadata.html I’d be *really* careful how you use this. A lot of teams want to just deploy their c

Re: Hints folder missing in Cassandra

The directory */var/lib/cassandra/hints *was created, when I brought up the cluster. I am running into this issue until after the cluster is up and running for about 2 weeks. Regards, Cassandra User On Wed, Feb 7, 2018 at 1:28 AM, Kyrylo Lebediev wrote: > Hi, > > Not sure what could be the reas

Re: node restart causes application latency

No, I am not On Wed, Feb 7, 2018 at 11:35 AM, Jeff Jirsa wrote: > Are you using internode ssl? > > > -- > Jeff Jirsa > > > On Feb 7, 2018, at 8:24 AM, Mike Torra wrote: > > Thanks for the feedback guys. That example data model was indeed > abbreviated - the real queries have the partition key i

Re: node restart causes application latency

Are you using internode ssl? -- Jeff Jirsa > On Feb 7, 2018, at 8:24 AM, Mike Torra wrote: > > Thanks for the feedback guys. That example data model was indeed abbreviated > - the real queries have the partition key in them. I am using RF 3 on the > keyspace, so I don't think a node being

Re: node restart causes application latency

Thanks for the feedback guys. That example data model was indeed abbreviated - the real queries have the partition key in them. I am using RF 3 on the keyspace, so I don't think a node being down would mean the key I'm looking for would be unavailable. The load balancing policy of the driver seems

Re: Bootstrapping fails with < 128GB RAM ...

Ok then, following up on the wild guess : because you have quite a lot of concurrent compactors, maybe it is too much concurrent compactions for the jvm to deal with (taking into account that your load average of 106 seems really high IMHO) 55Gb of data is not that much, you can try to reduce thos

Re: Bootstrapping fails with < 128GB RAM ...

Hi Nicolas, Do you know how many sstables is this new node suppose to receive ? If I can find out this via nodetool netstats, then this would be 619 as following: # nodetool netstats Bootstrap b95371e0-0c0a-11e8-932b-f775227bf21c /192.168.1.215 - Receiving 71 files, 7744612158 bytes total.

Re: Bootstrapping fails with < 128GB RAM ...

Hi Jurgen, It does feel like some OOM during boostrap from previous C* v2, but that sould be fixed in your version. Do you know how many sstables is this new node suppose to receive ? Juste a wild guess : it may have something to do with compaction not keeping up because every other nodes are st

Bootstrapping fails with < 128GB RAM ...

Hi, I always face an issue when bootstrapping a Node having less than 184GB RAM (156GB JVM HEAP) on our 10 Node C* 3.11.1 Cluster. During bootstrap, when I watch the cassandra.log I observe a growth in JVM Heap Old Gen which gets not significantly freed any more. I know that JVM collects on Old Gen

Re: SSH remote access,permissions issue

Milenko, please reply to user@cassandra.apache.org not directly to me ;) You do not seem to have a cassandra process up and running, can you check with your friend what he did please ? Maybe something is going on on this cluster, do not try to start the node he might be working on it :) On 7

Re: SSH remote access,permissions issue

Milenko, Check your friends env variables, perhaps he has something set (port, etc) that you don't. Cheers. On Wed, Feb 7, 2018 at 11:29 AM, Matija Gobec wrote: > Hi Milenko, > > There is a high chance that JMX is listening on the instance IP instead of > locahost which is default when you run

Re: SSH remote access,permissions issue

Hi Milenko, There is a high chance that JMX is listening on the instance IP instead of locahost which is default when you run nodetool command. Try executing nodetool with -h nodeip or check netstat as Nicolas mentioned. If you run nodetool status on one of the existing nodes can you see that new

Re: SSH remote access,permissions issue

Hi Milenko, Can you check the JMX configuration in jvm.options file and make sure you can login without user/pwd ? Also, the node might be listening to a specific network interface, can you output 'netstat -an | grep 7199' for us ? (assuming your JMX port config is 7199) On 7 February 2018 at 1

SSH remote access,permissions issue

My friend has installed 4 nodes at his server in Norway.I wanted to add new nodes to cluster.I have logged in with ssh and password.ssh pi3@84.208.89.002 -p 22022 Last login: Wed Feb  7 06:59:05 2018 from 94.189.152.722 Later sudo su,and have repeated 100% exactly the same operations as he did.

Re: Hints folder missing in Cassandra

Hi, Not sure what could be the reason, but the issue is obvious: directory /var/lib/cassandra/hints is missing or cassandra user doesn't have enough permissions on it. Regards, Kyrill From: test user Sent: Wednesday, February 7, 2018 8:14:59 AM To: user@cass