Reading data from disk
Hello Team, Here is a simple query. Whenever a select query is being run with cluster columns in where clause, does it happen that the entire partition is being read from disk to memory and then iterated over to fetch the required result set. Or there are indexes in place which help read only specific data from the disk Thanks Allen
High frequency garbage collection leading to High load average
Hello team, On a given day , a node in 27 node cluster observed higher frequency of garbage collection. Mostly young gc. I have found below issues: 1. Higher number of slow queries being observed on that particular node for that particular day compared to other days 2. Higher outgoing traffic observed from the node , 10 times the average outbound traffic on that particular day 3. Higher number of cache requests hitting the key cache and chunk cache that other days on the particular node The cluster has large partition warning as well. My query is, which of the above is a likely cause of higher frequency of GC leading to High load average on the system.
Multi dc replication failing
Hello Team, I have a multi dc Cassandra setup. We write and read using local quorum to a particular datacenter A The replica definition in schema is correct. Sometimes we don't see data getting correctly replicated to datacenter B. I need to know what may be different ways to troubleshoot this. What are the key metrics, details, log patterns I may look for Thanks
Continuous hints getting generated even when nodes are Up
Hello Team, I'm using Cassandra 3.0.9. Recently I have observed that, all the nodes in my cluster are generated hints. All the nodes are up but I do see hints getting generated continuously. In the logs I see tombstone warnings, but frequency is less I also see frequent message Unknown column during serialization. In the nodetool tpstats o/p i do see hintsdispatcher has actiive thread as 1 and pending as 42. The cluster is spread over multiple datacenters as well. Kindly suggest what may be inferred from above. Thanks
Cross DC replication failing
Hello team, Greetings. Simple question Using Cassandra 3.0.8 Writing to DC-A using local_quorum Reading the same data from a DC-B using local quorum. It succeeds for a table and fails for other. Data written is not replicated immediately even after a long time. However if read is performed using a quorum we fetch data. I do understand what is happening Any idea 1. How to track how many messages are yet to be replicated 2. Any way to understand why they are not getting replicated across DC 3. I know I can force a sync by repair but this cross DC replication is expected to be automatic. Am I correct? Thanks
Full repair results in uneven data distribution
Hello Team, Sorry for this might be a simple question. I was working on Cassandra 2.1.14 Node1 -- 4.5 mb data Node2 -- 5.3 mb data Node3 -- 4.9 mb data Node3 was down since 90 days. I brought it up and it joined the cluster. To sync data I ran nodetool repair --full Repair was successful...however just to be sure that the data is in sync..I re-ran the repair process..expecting the process to exit and hence prove that there is nothing to repair. Each time I ran full repair.. repair did run completely and successfully...it didn't exit immediately as I expected. Running it 4 times I suddenly saw this Node 1 -- 43 mb Node2 -- 42 mb Node3 -- 6 mb I was clueless of this data growth on node1 and node2. May anyone pls help me understand why this happened. To bring back things to normal, I tried running nodetool repair -pr on all the hosts one after another.. repair ran successfully... Still there was difference in data size on 3 nodes. Hence I decided to decommission each node and re-add them one after another.. I did that. The data size is now Just wanted to understand is there anyway my data is lost..why was there a difference in data size after I ran full repair multiple times Thanks
Re: Facing issues while starting Cassandra
Hello Osman, Thanks for the suggestion. I did try "export LC_ALL=C" It didn't help. Thanks On Tue, 24 Dec, 2019, 12:05 PM Osman Yozgatlıoğlu, < osman.yozgatlio...@gmail.com> wrote: > I faced similar issues with different locale settings. > Could you try following command before running? > export LC_ALL=C; > > Regards, > Osman > > On Tue, 24 Dec 2019 at 09:01, Inquistive allen > wrote: > > > > Hello Jeff, > > > > Thanks for responding. > > I have validated the cassandra.yaml file with other hosts in the cluster. > > There is no difference. I copied a yaml file from other node to this > node and changed the required configs. Still facing the same issue. > > The server went down for patching and after coming back up, Cassandra > dosent seem to start. > > Having looked for solutions on google, I found that it might be a > problem with the /tmp directory where the classes are stored. > > Each time I try starting Cassandra, in the /tmp directory a new > directory is created, but nothing is inside the directory. After some time, > the node goes down. > > > > I believe there is something to do with the /tmp directory. > > Request you to comment on the same. > > > > Thanks > > > > On Tue, 24 Dec, 2019, 3:42 AM Jeff Jirsa, wrote: > >> > >> Are you able to share the yaml? Almost certainly something in it that’s > invalid. > >> > >> On Dec 23, 2019, at 12:51 PM, Inquistive allen > wrote: > >> > >> > >> Hello Team, > >> > >> I am facing issues while starting Cassandra. > >> > >> Caused by: org.apache.cassandra.exceptions.ConfigurationException : > Invalid yaml: file: /path/to/yaml > >> Error: null ; can't construct a java object for tag: > >> yaml.org,2002:org.apache.cassandra.config.Config; > exception= java.lang.reflect.InvocationTargetException > >> > >> Request to comment on how to resolve the issue. > >> > >> Thanks & Regards > >> Allen > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > >
Re: Facing issues while starting Cassandra
Hello Jeff, Thanks for responding. I have validated the cassandra.yaml file with other hosts in the cluster. There is no difference. I copied a yaml file from other node to this node and changed the required configs. Still facing the same issue. The server went down for patching and after coming back up, Cassandra dosent seem to start. Having looked for solutions on google, I found that it might be a problem with the /tmp directory where the classes are stored. Each time I try starting Cassandra, in the /tmp directory a new directory is created, but nothing is inside the directory. After some time, the node goes down. I believe there is something to do with the /tmp directory. Request you to comment on the same. Thanks On Tue, 24 Dec, 2019, 3:42 AM Jeff Jirsa, wrote: > Are you able to share the yaml? Almost certainly something in it that’s > invalid. > > On Dec 23, 2019, at 12:51 PM, Inquistive allen > wrote: > > > Hello Team, > > I am facing issues while starting Cassandra. > > Caused by: org.apache.cassandra.exceptions.ConfigurationException : > Invalid yaml: file: /path/to/yaml > Error: null ; can't construct a java object for tag: > yaml.org,2002:org.apache.cassandra.config.Config; > exception= java.lang.reflect.InvocationTargetException > > Request to comment on how to resolve the issue. > > Thanks & Regards > Allen > >
Facing issues while starting Cassandra
Hello Team, I am facing issues while starting Cassandra. Caused by: org.apache.cassandra.exceptions.ConfigurationException : Invalid yaml: file: /path/to/yaml Error: null ; can't construct a java object for tag: yaml.org,2002:org.apache.cassandra.config.Config; exception= java.lang.reflect.InvocationTargetException Request to comment on how to resolve the issue. Thanks & Regards Allen
Re: Cassandra is not showing a node up hours after restart
Hello , Check and compare everything parameters 1. Java version should ideally match across all nodes in the cluster 2. Check if port 7000 is open between the nodes. Use telnet or nc commands 3. You must see some clues in system logs, why the gossip is failing. Do confirm on the above things. Thanks On Tue, 26 Nov, 2019, 2:50 AM Paul Mena, wrote: > NTP was restarted on the Cassandra nodes, but unfortunately I’m still > getting the same result: the restarted node does not appear to be rejoining > the cluster. > > > > Here’s another data point: “nodetool gossipinfo”, when run from the > restarted node (“node001”) shows a status of “normal”: > > > > user@node001=> nodetool -u gossipinfo > > /192.168.187.121 > > generation:1574364410 > > heartbeat:209150 > > NET_VERSION:8 > > RACK:rack1 > > STATUS:NORMAL,-104847506331695918 > > RELEASE_VERSION:2.1.9 > > SEVERITY:0.0 > > LOAD:5.78684155614E11 > > HOST_ID:c99cf581-f4ae-4aa9-ab37-1a114ab2429b > > SCHEMA:fd2dcb4b-ca62-30df-b8f2-d3fd774f2801 > > DC:datacenter1 > > RPC_ADDRESS:192.168.185.121 > > > > When run from one of the other nodes, however, node001’s status is shown > as “shutdown”: > > > > user@node002=> nodetool gossipinfo > > /192.168.187.121 > > generation:1491825076 > > heartbeat:2147483647 > > STATUS:shutdown,true > > RACK:rack1 > > NET_VERSION:8 > > LOAD:5.78679987693E11 > > RELEASE_VERSION:2.1.9 > > DC:datacenter1 > > SCHEMA:fd2dcb4b-ca62-30df-b8f2-d3fd774f2801 > > HOST_ID:c99cf581-f4ae-4aa9-ab37-1a114ab2429b > > RPC_ADDRESS:192.168.185.121 > > SEVERITY:0.0 > > > > > > *Paul Mena* > > Senior Application Administrator > > WHOI - Information Services > > 508-289-3539 > > > > *From:* Paul Mena > *Sent:* Monday, November 25, 2019 9:29 AM > *To:* user@cassandra.apache.org > *Subject:* RE: Cassandra is not showing a node up hours after restart > > > > I’ve just discovered that NTP is not running on any of these Cassandra > nodes, and that the timestamps are all over the map. Could this be causing > my issue? > > > > user@remote=> ansible pre-prod-cassandra -a date > > node001.intra.myorg.org | CHANGED | rc=0 >> > > Mon Nov 25 13:58:17 UTC 2019 > > > > node004.intra.myorg.org | CHANGED | rc=0 >> > > Mon Nov 25 14:07:20 UTC 2019 > > > > node003.intra.myorg.org | CHANGED | rc=0 >> > > Mon Nov 25 13:57:06 UTC 2019 > > > > node001.intra.myorg.org | CHANGED | rc=0 >> > > Mon Nov 25 14:07:22 UTC 2019 > > > > *Paul Mena* > > Senior Application Administrator > > WHOI - Information Services > > 508-289-3539 > > > > *From:* Inquistive allen > *Sent:* Monday, November 25, 2019 2:46 AM > *To:* user@cassandra.apache.org > *Subject:* Re: Cassandra is not showing a node up hours after restart > > > > Hello team, > > > > Just to add on to the discussion, one may run, > > Nodetool disablebinary followed by a nodetool disablethrift followed by > nodetool drain. > > Nodetool drain also does the work of nodetool flush+ declaring in the > cluster that I'm down and not accepting traffic. > > > > Thanks > > > > > > On Mon, 25 Nov, 2019, 12:55 AM Surbhi Gupta, > wrote: > > Before Cassandra shutdown, nodetool drain should be executed first. As > soon as you do nodetool drain, others node will see this node down and no > new traffic will come to this node. > > I generally gives 10 seconds gap between nodetool drain and Cassandra > stop. > > > > On Sun, Nov 24, 2019 at 9:52 AM Paul Mena wrote: > > Thank you for the replies. I had made no changes to the config before the > rolling restart. > > > > I can try another restart but was wondering if I should do it differently. > I had simply done "service cassandra stop" followed by "service cassandra > start". Since then I've seen some suggestions to proceed the shutdown with > "nodetool disablegossip" and/or "nodetool drain". Are these commands > advisable? Are any other commands recommended either before the shutdown or > after the startup? > > > > Thanks again! > > > > Paul > -- > > *From:* Naman Gupta > *Sent:* Sunday, November 24, 2019 11:18:14 AM > *To:* user@cassandra.apache.org > *Subject:* Re: Cassandra is not showing a node up hours after restart > > > > Did you change the name of datacenter or any other config changes before > the rolling restart? >
Re: Cassandra is not showing a node up hours after restart
Hello team, Just to add on to the discussion, one may run, Nodetool disablebinary followed by a nodetool disablethrift followed by nodetool drain. Nodetool drain also does the work of nodetool flush+ declaring in the cluster that I'm down and not accepting traffic. Thanks On Mon, 25 Nov, 2019, 12:55 AM Surbhi Gupta, wrote: > Before Cassandra shutdown, nodetool drain should be executed first. As > soon as you do nodetool drain, others node will see this node down and no > new traffic will come to this node. > I generally gives 10 seconds gap between nodetool drain and Cassandra > stop. > > On Sun, Nov 24, 2019 at 9:52 AM Paul Mena wrote: > >> Thank you for the replies. I had made no changes to the config before the >> rolling restart. >> >> >> I can try another restart but was wondering if I should do it >> differently. I had simply done "service cassandra stop" followed by >> "service cassandra start". Since then I've seen some suggestions to >> proceed the shutdown with "nodetool disablegossip" and/or "nodetool drain". >> Are these commands advisable? Are any other commands recommended either >> before the shutdown or after the startup? >> >> >> Thanks again! >> >> >> Paul >> -- >> *From:* Naman Gupta >> *Sent:* Sunday, November 24, 2019 11:18:14 AM >> *To:* user@cassandra.apache.org >> *Subject:* Re: Cassandra is not showing a node up hours after restart >> >> Did you change the name of datacenter or any other config changes before >> the rolling restart? >> >> On Sun, Nov 24, 2019 at 8:49 PM Paul Mena wrote: >> >>> I am in the process of doing a rolling restart on a 4-node cluster >>> running Cassandra 2.1.9. I stopped and started Cassandra on node 1 via >>> "service cassandra stop/start", and noted nothing unusual in either >>> system.log or cassandra.log. Doing a "nodetool status" from node 1 shows >>> all four nodes up: >>> >>> user@node001=> nodetool status >>> Datacenter: datacenter1 >>> >>> === >>> Status=Up/Down >>> |/ State=Normal/Leaving/Joining/Moving >>> -- Address Load Tokens OwnsHost ID >>> Rack >>> UN 192.168.187.121 538.95 GB 256 ? >>> c99cf581-f4ae-4aa9-ab37-1a114ab2429b rack1 >>> UN 192.168.187.122 630.72 GB 256 ? >>> bfa07f47-7e37-42b4-9c0b-024b3c02e93f rack1 >>> UN 192.168.187.123 572.73 GB 256 ? >>> 273df9f3-e496-4c65-a1f2-325ed288a992 rack1 >>> UN 192.168.187.124 625.05 GB 256 ? >>> b8639cf1-5413-4ece-b882-2161bbb8a9c3 rack1 >>> >>> But doing the same command from any other of the 3 nodes shows node 1 >>> still down. >>> >>> user@node002=> nodetool status >>> Datacenter: datacenter1 >>> === >>> Status=Up/Down >>> |/ State=Normal/Leaving/Joining/Moving >>> -- Address Load Tokens OwnsHost ID >>> Rack >>> DN 192.168.187.121 538.94 GB 256 ? >>> c99cf581-f4ae-4aa9-ab37-1a114ab2429b rack1 >>> UN 192.168.187.122 630.72 GB 256 ? >>> bfa07f47-7e37-42b4-9c0b-024b3c02e93f rack1 >>> UN 192.168.187.123 572.73 GB 256 ? >>> 273df9f3-e496-4c65-a1f2-325ed288a992 rack1 >>> UN 192.168.187.124 625.04 GB 256 ? >>> b8639cf1-5413-4ece-b882-2161bbb8a9c3 rack1 >>> >>> Is there something I can do to remedy this current situation - so that I >>> can continue with the rolling restart? >>> >>>
Select statement in batch
Hello Team, Wanted to understand the impacted of using a select statement inside a batch. I keep seeing some slow queries frequently in the logs. Please comment on what may the impact of the same. Is it the right practice. Will a select statement in batch be lead to increase in read latency than a normal select prepared statement. Thanks, Allen
Re: Cassandra loglevel in 3.11.x version
Hello Jai, If you want to fetch queries from the logs. Kindly run the below command: nodetool setlogginglevel org.apache.cassandra.transport TRACE this will help you see all your queries in logs. Just beware that this fills up logs really fast. Thanks, Allen On Fri, 6 Sep, 2019, 10:57 AM Jai Bheemsen Rao Dhanwada, < jaibheem...@gmail.com> wrote: > Hello, > > I am trying to enable debug logs on the Cassandra cluster running with > 3.11.3, I do see some debug logs but I don't see the same level of DEBUG > logs I used to see with 2.1.16 version of cassandra. > > I am using the below command to set debug logging. > > $ nodetool setlogginglevel org.apache.cassandra DEBUG > > > In the 2.1.16 version of Cassandra I see the below logs when I make a > select query using CQL, but I don't see the same trace logs in 3.11.3. has > anything changed in the 3.11.3 version? > >> DEBUG [SharedPool-Worker-1] 2019-09-06 03:43:10,593 Message.java:437 - >> Received: QUERY SELECT * FROM ks.cf WHERE key=? AND id=?;, v=3 > > > Thanks in advance. >
Re: Consistent hashing vnodes and ring in cassandra
Hello Jeff, Thanks for the response. I just got it right. One Last thing, when a read request comes in to the coordinator node, the partition key is hashed and a node is located where corresponding data is previously stored. How does the coordinator node locate the replica nodes for this row. The first copy of this was written based on hash number, but the replica copies were written based on replication strategy. Will a hash of any partition key list out all nodes where data is present . Thanks On Sun, 18 Aug, 2019, 12:35 AM Jeff Jirsa, wrote: > > > > On Aug 17, 2019, at 10:53 AM, Inquistive allen > wrote: > > > > I am a newbie in cassandra. I have asked this question on various > platforms but never got a satisfying answer. > > Hence thought of bringing up the topic here. Sorry for this might be a > simple question. > > > > 1. I studied the paper on consistent hashing (which is being implemented > in Cassandra) > > 2. Cassandra has the concept of Vnodes. The vnodes( As I understand a > Vnode is a collection of Hashes) , are the basic blocks of replication in > cassandra. It is the vnodes which are replicated across the cluster. Please > do correct me I'm wrong > > Vnodes JUST mean each host has more than one token > > > 3. Suppose I have a Keyspace A with replication factor 3 and Keyspace B > with replication factor 2. > > 4. Is it that a Vnode is a collection of hashes of data from various > Keyspaces. > > 5. In that case, Keyspace with varying replication factors , replicating > them to other nodes would be a problem > > 6. Now from the consistent hashing paper, I get a feeling that , ach > Keyspace has a different ring. Also the name "KEYSPACE", points to a ring > of keys in the ring. > >So is it that each keyspace has a different ring. If it is so, > everything else like replicating vnodes among nodes in the cluster would > fall in place. > >Each Keyspace has a different ring ---> each Vnode has data of > various tables from a given keyspace> hence copies equal to RF is only > made in the cluster. > > > > I know I am missing something. This way of understanding thing might be > wrong. > > Kindly help me understand the same. As this would help me visualise > repair, bootstrap, adding cluster, streaming operations in a much better > way. > > > > The easiest way to visualize most cassandra operations is to draw the > tokens in a circle. Vnodes means extra tokens > > Replica sets are adjacent tokens. You steam from any node in the replica > set in the common replacement case, or the losing replica in the expansion > case > > > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > >
Consistent hashing vnodes and ring in cassandra
I am a newbie in cassandra. I have asked this question on various platforms but never got a satisfying answer. Hence thought of bringing up the topic here. Sorry for this might be a simple question. 1. I studied the paper on consistent hashing (which is being implemented in Cassandra) 2. Cassandra has the concept of Vnodes. The vnodes( As I understand a Vnode is a collection of Hashes) , are the basic blocks of replication in cassandra. It is the vnodes which are replicated across the cluster. Please do correct me I'm wrong 3. Suppose I have a Keyspace A with replication factor 3 and Keyspace B with replication factor 2. 4. Is it that a Vnode is a collection of hashes of data from various Keyspaces. 5. In that case, Keyspace with varying replication factors , replicating them to other nodes would be a problem 6. Now from the consistent hashing paper, I get a feeling that , ach Keyspace has a different ring. Also the name "KEYSPACE", points to a ring of keys in the ring. So is it that each keyspace has a different ring. If it is so, everything else like replicating vnodes among nodes in the cluster would fall in place. Each Keyspace has a different ring ---> each Vnode has data of various tables from a given keyspace> hence copies equal to RF is only made in the cluster. I know I am missing something. This way of understanding thing might be wrong. Kindly help me understand the same. As this would help me visualise repair, bootstrap, adding cluster, streaming operations in a much better way. - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
