java.lang.OutOfMemoryError: Java heap space
Hello I have a three nodes cluster. I noticed one node was always down. Restarting Cassandra fixes it but it will go down again after a couple of days. I'm pretty new to Cassandra so I'm wondering how I should troubleshoot it. Logs is as below. INFO [StorageServiceShutdownHook] 2014-04-28 13:21:05,091 ThriftServer.java (line 141) Stop listening to thrift clients ERROR [GossipStage:4] 2014-04-28 13:11:59,877 CassandraDaemon.java (line 196) Exception in thread Thread[GossipStage:4,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [ACCEPT-/10.20.132.44] 2014-04-28 13:06:10,261 CassandraDaemon.java (line 196) Exception in thread Thread[ACCEPT-/10.20.132.44,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [GossipStage:3] 2014-04-28 12:54:02,116 CassandraDaemon.java (line 196) Exception in thread Thread[GossipStage:3,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [GossipStage:2] 2014-04-28 12:52:27,644 CassandraDaemon.java (line 196) Exception in thread Thread[GossipStage:2,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [Thread-222] 2014-04-28 12:50:18,689 CassandraDaemon.java (line 196) Exception in thread Thread[Thread-222,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [GossipTasks:1] 2014-04-28 12:47:12,879 CassandraDaemon.java (line 196) Exception in thread Thread[GossipTasks:1,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [GossipTasks:1] 2014-04-28 13:24:59,113 CassandraDaemon.java (line 196) Exception in thread Thread[GossipTasks:1,5,main] java.lang.IllegalThreadStateException at java.lang.Thread.start(Thread.java:704) at org.apache.cassandra.service.CassandraDaemon$2.uncaughtException(CassandraDaemon.java:202) at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.handleOrLog(DebuggableThreadPoolExecutor.java:220) at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:79) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Server.java (line 181) Stop listening for CQL clients INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Gossiper.java (line 1251) Announcing shutdown INFO [StorageServiceShutdownHook] 2014-04-28 13:26:12,524 MessagingService.java (line 667) Waiting for messaging service to quiesce Thanks Gary
Re: java.lang.OutOfMemoryError: Java heap space
BTW, the CPU usage on this node is pretty high, but data size is pretty small. PID USERNAME THR PRI NICE SIZE RES SHR STATE TIMECPU COMMAND 28674 cassandr 89 250 9451M 8970M 525M sleep 32.1H 329% java UN 8.92 GB256 35.4% c2d9d02e-bdb3-47cb-af1b-eabc2eeb503b rack1 UN 5.32 GB256 33.1% 874a269f-96bc-4db6-9d15-7cac6771a2ea rack1 DN 339.94 MB 256 31.5% fba5b071-aee1-451f-ae42-49c3bbe70bbe rack1 On Mon, Apr 28, 2014 at 10:25 AM, Gary Zhao garyz...@gmail.com wrote: Hello I have a three nodes cluster. I noticed one node was always down. Restarting Cassandra fixes it but it will go down again after a couple of days. I'm pretty new to Cassandra so I'm wondering how I should troubleshoot it. Logs is as below. INFO [StorageServiceShutdownHook] 2014-04-28 13:21:05,091 ThriftServer.java (line 141) Stop listening to thrift clients ERROR [GossipStage:4] 2014-04-28 13:11:59,877 CassandraDaemon.java (line 196) Exception in thread Thread[GossipStage:4,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [ACCEPT-/10.20.132.44] 2014-04-28 13:06:10,261 CassandraDaemon.java (line 196) Exception in thread Thread[ACCEPT-/10.20.132.44,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [GossipStage:3] 2014-04-28 12:54:02,116 CassandraDaemon.java (line 196) Exception in thread Thread[GossipStage:3,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [GossipStage:2] 2014-04-28 12:52:27,644 CassandraDaemon.java (line 196) Exception in thread Thread[GossipStage:2,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [Thread-222] 2014-04-28 12:50:18,689 CassandraDaemon.java (line 196) Exception in thread Thread[Thread-222,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [GossipTasks:1] 2014-04-28 12:47:12,879 CassandraDaemon.java (line 196) Exception in thread Thread[GossipTasks:1,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [GossipTasks:1] 2014-04-28 13:24:59,113 CassandraDaemon.java (line 196) Exception in thread Thread[GossipTasks:1,5,main] java.lang.IllegalThreadStateException at java.lang.Thread.start(Thread.java:704) at org.apache.cassandra.service.CassandraDaemon$2.uncaughtException(CassandraDaemon.java:202) at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.handleOrLog(DebuggableThreadPoolExecutor.java:220) at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:79) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Server.java (line 181) Stop listening for CQL clients INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Gossiper.java (line 1251) Announcing shutdown INFO [StorageServiceShutdownHook] 2014-04-28 13:26:12,524 MessagingService.java (line 667) Waiting for messaging service to quiesce Thanks Gary
Re: java.lang.OutOfMemoryError: Java heap space
Are the virtual machines? The last time I had this issues was because of VMWare ballooning. If not, what versions of Cassandra and Java are you using? On Mon, Apr 28, 2014 at 6:30 PM, Gary Zhao garyz...@gmail.com wrote: BTW, the CPU usage on this node is pretty high, but data size is pretty small. PID USERNAME THR PRI NICE SIZE RES SHR STATE TIMECPU COMMAND 28674 cassandr 89 250 9451M 8970M 525M sleep 32.1H 329% java UN 8.92 GB256 35.4% c2d9d02e-bdb3-47cb-af1b-eabc2eeb503b rack1 UN 5.32 GB256 33.1% 874a269f-96bc-4db6-9d15-7cac6771a2ea rack1 DN 339.94 MB 256 31.5% fba5b071-aee1-451f-ae42-49c3bbe70bbe rack1 On Mon, Apr 28, 2014 at 10:25 AM, Gary Zhao garyz...@gmail.com wrote: Hello I have a three nodes cluster. I noticed one node was always down. Restarting Cassandra fixes it but it will go down again after a couple of days. I'm pretty new to Cassandra so I'm wondering how I should troubleshoot it. Logs is as below. INFO [StorageServiceShutdownHook] 2014-04-28 13:21:05,091 ThriftServer.java (line 141) Stop listening to thrift clients ERROR [GossipStage:4] 2014-04-28 13:11:59,877 CassandraDaemon.java (line 196) Exception in thread Thread[GossipStage:4,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [ACCEPT-/10.20.132.44] 2014-04-28 13:06:10,261 CassandraDaemon.java (line 196) Exception in thread Thread[ACCEPT-/ 10.20.132.44,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [GossipStage:3] 2014-04-28 12:54:02,116 CassandraDaemon.java (line 196) Exception in thread Thread[GossipStage:3,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [GossipStage:2] 2014-04-28 12:52:27,644 CassandraDaemon.java (line 196) Exception in thread Thread[GossipStage:2,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [Thread-222] 2014-04-28 12:50:18,689 CassandraDaemon.java (line 196) Exception in thread Thread[Thread-222,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [GossipTasks:1] 2014-04-28 12:47:12,879 CassandraDaemon.java (line 196) Exception in thread Thread[GossipTasks:1,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [GossipTasks:1] 2014-04-28 13:24:59,113 CassandraDaemon.java (line 196) Exception in thread Thread[GossipTasks:1,5,main] java.lang.IllegalThreadStateException at java.lang.Thread.start(Thread.java:704) at org.apache.cassandra.service.CassandraDaemon$2.uncaughtException(CassandraDaemon.java:202) at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.handleOrLog(DebuggableThreadPoolExecutor.java:220) at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:79) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Server.java (line 181) Stop listening for CQL clients INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Gossiper.java (line 1251) Announcing shutdown INFO [StorageServiceShutdownHook] 2014-04-28 13:26:12,524 MessagingService.java (line 667) Waiting for messaging service to quiesce Thanks Gary
Discount Codes for Cassandra Trainings this week!
Hey Cassandra Community, We still have open seats to the Developer trainings this week and are offering discounts if you are interested in attending. 10% off 1 ticket 20% off 2 tickets 30% off 3 tickets http://www.datastax.com/what-we-offer/products-services/training If you are interested in attending, please let me know. Best, Steph
Cassandra data retention policy
Hi guys, We have a processing system that just uses the data for the past six months in Cassandra. Any suggestions on the best way to manage the old data in order to save disk space? We want to keep it as backup but it will not be used unless we need to do recovery. Thanks in advance! -John
RE: Cassandra data retention policy
CQL lets you specify a default TTL per column family/table: and default_time_to_live=86400 . From: Redmumba [mailto:redmu...@gmail.com] Sent: Monday, April 28, 2014 12:51 PM To: user@cassandra.apache.org Subject: Re: Cassandra data retention policy Have you looked into using a TTL? You can set this per insert (unfortunately, it can't be set per CF) and values will be tombstoned after that amount of time. I.e., INSERT INTO VALUES ... TTL 15552000 Keep in mind, after the values have expired, they will essentially become tombstones--so you will still need to run clean-ups (probably daily) to clear up space. Does this help? One caveat is that this is difficult to apply to existing rows--i.e., you can't bulk-update a bunch of rows with this data. As such, another good suggestion is to simply have a secondary index on a date field of some kind, and run a bulk remove (and subsequent clean-up) daily/weekly/whatever. On Mon, Apr 28, 2014 at 11:31 AM, Han Jia johnideal...@gmail.commailto:johnideal...@gmail.com wrote: Hi guys, We have a processing system that just uses the data for the past six months in Cassandra. Any suggestions on the best way to manage the old data in order to save disk space? We want to keep it as backup but it will not be used unless we need to do recovery. Thanks in advance! -John
Re: Cassandra data retention policy
TTL is good for this but I have no idea how you will ever be able to restore data removed from disk like that. Perhaps one could make a snapshot and then delete everything with timestamp older than a date and then run compaction on every node to reclaim the disk. 2014-04-28 21:57 GMT+02:00 Donald Smith donald.sm...@audiencescience.com: CQL lets you specify a default TTL per column family/table: and default_time_to_live=86400 . *From:* Redmumba [mailto:redmu...@gmail.com] *Sent:* Monday, April 28, 2014 12:51 PM *To:* user@cassandra.apache.org *Subject:* Re: Cassandra data retention policy Have you looked into using a TTL? You can set this per insert (unfortunately, it can't be set per CF) and values will be tombstoned after that amount of time. I.e., INSERT INTO VALUES ... TTL 15552000 Keep in mind, after the values have expired, they will essentially become tombstones--so you will still need to run clean-ups (probably daily) to clear up space. Does this help? One caveat is that this is difficult to apply to existing rows--i.e., you can't bulk-update a bunch of rows with this data. As such, another good suggestion is to simply have a secondary index on a date field of some kind, and run a bulk remove (and subsequent clean-up) daily/weekly/whatever. On Mon, Apr 28, 2014 at 11:31 AM, Han Jia johnideal...@gmail.com wrote: Hi guys, We have a processing system that just uses the data for the past six months in Cassandra. Any suggestions on the best way to manage the old data in order to save disk space? We want to keep it as backup but it will not be used unless we need to do recovery. Thanks in advance! -John
Re: Cassandra data retention policy
He said below that he’d like to keep the old data, so that might rule out TTLs in any case. You’ve got a few options that I can think of off the top of my head. The easiest from a management perspective is to use one table per month. WhateverData042014 would be this months. It’s easy enough to back up sstables, you just copy them off somewhere. You could compact the previous month’s table at the beginning of the following month, and copy the stables off for archiving, in s3 or something similar. Depending on where you end up moving the data, it might be more trouble than it’s worth, since you might need to come up with a backup plan, and now you’ll have 2 things to back up instead of just 1. Also restoring the data is more of a pain than just querying it. On Apr 28, 2014, at 12:57 PM, Donald Smith donald.sm...@audiencescience.com wrote: CQL lets you specify a default TTL per column family/table: and default_time_to_live=86400 . From: Redmumba [mailto:redmu...@gmail.com] Sent: Monday, April 28, 2014 12:51 PM To: user@cassandra.apache.org Subject: Re: Cassandra data retention policy Have you looked into using a TTL? You can set this per insert (unfortunately, it can't be set per CF) and values will be tombstoned after that amount of time. I.e., INSERT INTO VALUES ... TTL 15552000 Keep in mind, after the values have expired, they will essentially become tombstones--so you will still need to run clean-ups (probably daily) to clear up space. Does this help? One caveat is that this is difficult to apply to existing rows--i.e., you can't bulk-update a bunch of rows with this data. As such, another good suggestion is to simply have a secondary index on a date field of some kind, and run a bulk remove (and subsequent clean-up) daily/weekly/whatever. On Mon, Apr 28, 2014 at 11:31 AM, Han Jia johnideal...@gmail.com wrote: Hi guys, We have a processing system that just uses the data for the past six months in Cassandra. Any suggestions on the best way to manage the old data in order to save disk space? We want to keep it as backup but it will not be used unless we need to do recovery. Thanks in advance! -John
Re: Load balancing issue with virtual nodes
Hello all Some update about the issue. After wiping completely all sstable/commitlog/saved_caches folder and restart the cluster from scratch, we still experience weird figures. After the restart, nodetool status does not show an exact balance of 50% of data for each node : Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN host1 48.57 KB 256 *51.6%* d00de0d1-836f-4658-af64-3a12c00f47d6 rack1 UN host2 48.57 KB 256 *48.4%* e9d2505b-7ba7-414c-8b17-af3bbe79ed9c rack1 As you can see, the % is very close to 50% but not exactly 50% What can explain that ? Can it be network connection issue during token initial shuffle phase ? P.S: both host1 and host2 are supposed to have exactly the same hardware Regards Duy Hai DOAN On Thu, Apr 24, 2014 at 11:20 PM, Batranut Bogdan batra...@yahoo.comwrote: I don't know about hector but the datastax java driver needs just one ip from the cluster and it will discover the rest of the nodes. Then by default it will do a round robin when sending requests. So if Hector does the same the patterb will againg appear. Did you look at the size of the dirs? That documentation is for C* 0.8. It's old. But depending on your boxes you might reach CPU bottleneck. Might want to google for write path in cassandra.. According to that, there is not much to do when writes come in... On Friday, April 25, 2014 12:00 AM, DuyHai Doan doanduy...@gmail.com wrote: I did some experiments. Let's say we have node1 and node2 First, I configured Hector with node1 node2 as hosts and I saw that only node1 has high CPU load To eliminate the client connection issue, I re-test with only node2 provided as host for Hector. Same pattern. CPU load is above 50% on node1 and below 10% on node2. It means that node2 is playing as coordinator and forward many write/read request to node1 Why did I look at CPU load and not iostat al ? Because I have a very intensive write work load with read-only-once pattern. I've read here ( http://www.datastax.com/docs/0.8/cluster_architecture/cluster_planning) that heavy write in C* is more CPU bound but maybe the info may be outdated and no longer true Regards Duy Hai DOAN On Thu, Apr 24, 2014 at 10:00 PM, Michael Shuler mich...@pbandjelly.orgwrote: On 04/24/2014 10:29 AM, DuyHai Doan wrote: Client used = Hector 1.1-4 Default Load Balancing connection policy Both nodes addresses are provided to Hector so according to its connection policy, the client should switch alternatively between both nodes OK, so is only one connection being established to one node for one bulk write operation? Or are multiple connections being made to both nodes and writes performed on both? -- Michael
JDK 8
I've been searching around, but cannot find any information as to whether Cassandra runs on JRE 8. Any information on that? Thanks, Mitchell
Re: java.lang.OutOfMemoryError: Java heap space
Yes, they are virtual machines, but we are using KVM. Is there any solutions for this issue or we should use physical machines? On Mon, Apr 28, 2014 at 10:38 AM, Prem Yadav ipremya...@gmail.com wrote: Are the virtual machines? The last time I had this issues was because of VMWare ballooning. If not, what versions of Cassandra and Java are you using? On Mon, Apr 28, 2014 at 6:30 PM, Gary Zhao garyz...@gmail.com wrote: BTW, the CPU usage on this node is pretty high, but data size is pretty small. PID USERNAME THR PRI NICE SIZE RES SHR STATE TIMECPU COMMAND 28674 cassandr 89 250 9451M 8970M 525M sleep 32.1H 329% java UN 8.92 GB256 35.4% c2d9d02e-bdb3-47cb-af1b-eabc2eeb503b rack1 UN 5.32 GB256 33.1% 874a269f-96bc-4db6-9d15-7cac6771a2ea rack1 DN 339.94 MB 256 31.5% fba5b071-aee1-451f-ae42-49c3bbe70bbe rack1 On Mon, Apr 28, 2014 at 10:25 AM, Gary Zhao garyz...@gmail.com wrote: Hello I have a three nodes cluster. I noticed one node was always down. Restarting Cassandra fixes it but it will go down again after a couple of days. I'm pretty new to Cassandra so I'm wondering how I should troubleshoot it. Logs is as below. INFO [StorageServiceShutdownHook] 2014-04-28 13:21:05,091 ThriftServer.java (line 141) Stop listening to thrift clients ERROR [GossipStage:4] 2014-04-28 13:11:59,877 CassandraDaemon.java (line 196) Exception in thread Thread[GossipStage:4,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [ACCEPT-/10.20.132.44] 2014-04-28 13:06:10,261 CassandraDaemon.java (line 196) Exception in thread Thread[ACCEPT-/ 10.20.132.44,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [GossipStage:3] 2014-04-28 12:54:02,116 CassandraDaemon.java (line 196) Exception in thread Thread[GossipStage:3,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [GossipStage:2] 2014-04-28 12:52:27,644 CassandraDaemon.java (line 196) Exception in thread Thread[GossipStage:2,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [Thread-222] 2014-04-28 12:50:18,689 CassandraDaemon.java (line 196) Exception in thread Thread[Thread-222,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [GossipTasks:1] 2014-04-28 12:47:12,879 CassandraDaemon.java (line 196) Exception in thread Thread[GossipTasks:1,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [GossipTasks:1] 2014-04-28 13:24:59,113 CassandraDaemon.java (line 196) Exception in thread Thread[GossipTasks:1,5,main] java.lang.IllegalThreadStateException at java.lang.Thread.start(Thread.java:704) at org.apache.cassandra.service.CassandraDaemon$2.uncaughtException(CassandraDaemon.java:202) at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.handleOrLog(DebuggableThreadPoolExecutor.java:220) at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:79) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Server.java (line 181) Stop listening for CQL clients INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Gossiper.java (line 1251) Announcing shutdown INFO [StorageServiceShutdownHook] 2014-04-28 13:26:12,524 MessagingService.java (line 667) Waiting for messaging service to quiesce Thanks Gary
Re: Load balancing issue with virtual nodes
Some imbalance is expected and considered normal: See http://wiki.apache.org/cassandra/VirtualNodes/Balance As well as https://issues.apache.org/jira/browse/CASSANDRA-7032 Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr | +61 415 936 359 On 29 Apr 2014, at 7:30 am, DuyHai Doan doanduy...@gmail.com wrote: Hello all Some update about the issue. After wiping completely all sstable/commitlog/saved_caches folder and restart the cluster from scratch, we still experience weird figures. After the restart, nodetool status does not show an exact balance of 50% of data for each node : Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN host1 48.57 KB 256 51.6% d00de0d1-836f-4658-af64-3a12c00f47d6 rack1 UN host2 48.57 KB 256 48.4% e9d2505b-7ba7-414c-8b17-af3bbe79ed9c rack1 As you can see, the % is very close to 50% but not exactly 50% What can explain that ? Can it be network connection issue during token initial shuffle phase ? P.S: both host1 and host2 are supposed to have exactly the same hardware Regards Duy Hai DOAN On Thu, Apr 24, 2014 at 11:20 PM, Batranut Bogdan batra...@yahoo.com wrote: I don't know about hector but the datastax java driver needs just one ip from the cluster and it will discover the rest of the nodes. Then by default it will do a round robin when sending requests. So if Hector does the same the patterb will againg appear. Did you look at the size of the dirs? That documentation is for C* 0.8. It's old. But depending on your boxes you might reach CPU bottleneck. Might want to google for write path in cassandra.. According to that, there is not much to do when writes come in... On Friday, April 25, 2014 12:00 AM, DuyHai Doan doanduy...@gmail.com wrote: I did some experiments. Let's say we have node1 and node2 First, I configured Hector with node1 node2 as hosts and I saw that only node1 has high CPU load To eliminate the client connection issue, I re-test with only node2 provided as host for Hector. Same pattern. CPU load is above 50% on node1 and below 10% on node2. It means that node2 is playing as coordinator and forward many write/read request to node1 Why did I look at CPU load and not iostat al ? Because I have a very intensive write work load with read-only-once pattern. I've read here (http://www.datastax.com/docs/0.8/cluster_architecture/cluster_planning) that heavy write in C* is more CPU bound but maybe the info may be outdated and no longer true Regards Duy Hai DOAN On Thu, Apr 24, 2014 at 10:00 PM, Michael Shuler mich...@pbandjelly.org wrote: On 04/24/2014 10:29 AM, DuyHai Doan wrote: Client used = Hector 1.1-4 Default Load Balancing connection policy Both nodes addresses are provided to Hector so according to its connection policy, the client should switch alternatively between both nodes OK, so is only one connection being established to one node for one bulk write operation? Or are multiple connections being made to both nodes and writes performed on both? -- Michael
Can the seeds list be changed at runtime?
Hi, All, I wonder if I can change the seeds list at runtime. i.e. without change the yaml file and restart DB service? Thanks Boying
RE: java.lang.OutOfMemoryError: Java heap space
If you don’t have row caching turned on, you can try changing settings to get your memtables to flush faster (read more at the datastax documention at [1]). Btw, if you are using a pre-1.2 version, your bloom filter might come into play as well. All said and done, if your write request rate is too high for your ring to handle, you just can’t flush fast enough, and you’ll need to move to physical machines and/or add nodes to your ring. -Arindam [1] http://www.datastax.com/documentation/cassandra/2.0/cassandra/troubleshooting/trblshootDyingNodes_r.html From: Gary Zhao [mailto:garyz...@gmail.com] Sent: Monday, April 28, 2014 5:34 PM To: user@cassandra.apache.org Subject: Re: java.lang.OutOfMemoryError: Java heap space Yes, they are virtual machines, but we are using KVM. Is there any solutions for this issue or we should use physical machines? On Mon, Apr 28, 2014 at 10:38 AM, Prem Yadav ipremya...@gmail.commailto:ipremya...@gmail.com wrote: Are the virtual machines? The last time I had this issues was because of VMWare ballooning. If not, what versions of Cassandra and Java are you using? On Mon, Apr 28, 2014 at 6:30 PM, Gary Zhao garyz...@gmail.commailto:garyz...@gmail.com wrote: BTW, the CPU usage on this node is pretty high, but data size is pretty small. PID USERNAME THR PRI NICE SIZE RES SHR STATE TIMECPU COMMAND 28674 cassandr 89 250 9451M 8970M 525M sleep 32.1H 329% java UN 8.92 GB256 35.4% c2d9d02e-bdb3-47cb-af1b-eabc2eeb503b rack1 UN 5.32 GB256 33.1% 874a269f-96bc-4db6-9d15-7cac6771a2ea rack1 DN 339.94 MB 256 31.5% fba5b071-aee1-451f-ae42-49c3bbe70bbe rack1 On Mon, Apr 28, 2014 at 10:25 AM, Gary Zhao garyz...@gmail.commailto:garyz...@gmail.com wrote: Hello I have a three nodes cluster. I noticed one node was always down. Restarting Cassandra fixes it but it will go down again after a couple of days. I'm pretty new to Cassandra so I'm wondering how I should troubleshoot it. Logs is as below. INFO [StorageServiceShutdownHook] 2014-04-28 13:21:05,091 ThriftServer.java (line 141) Stop listening to thrift clients ERROR [GossipStage:4] 2014-04-28 13:11:59,877 CassandraDaemon.java (line 196) Exception in thread Thread[GossipStage:4,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [ACCEPT-/10.20.132.44http://10.20.132.44] 2014-04-28 13:06:10,261 CassandraDaemon.java (line 196) Exception in thread Thread[ACCEPT-/10.20.132.44http://10.20.132.44,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [GossipStage:3] 2014-04-28 12:54:02,116 CassandraDaemon.java (line 196) Exception in thread Thread[GossipStage:3,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [GossipStage:2] 2014-04-28 12:52:27,644 CassandraDaemon.java (line 196) Exception in thread Thread[GossipStage:2,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [Thread-222] 2014-04-28 12:50:18,689 CassandraDaemon.java (line 196) Exception in thread Thread[Thread-222,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [GossipTasks:1] 2014-04-28 12:47:12,879 CassandraDaemon.java (line 196) Exception in thread Thread[GossipTasks:1,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [GossipTasks:1] 2014-04-28 13:24:59,113 CassandraDaemon.java (line 196) Exception in thread Thread[GossipTasks:1,5,main] java.lang.IllegalThreadStateException at java.lang.Thread.start(Thread.java:704) at org.apache.cassandra.service.CassandraDaemon$2.uncaughtException(CassandraDaemon.java:202) at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.handleOrLog(DebuggableThreadPoolExecutor.java:220) at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:79) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Server.java (line 181) Stop listening for CQL clients INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Gossiper.java (line 1251) Announcing shutdown INFO [StorageServiceShutdownHook] 2014-04-28 13:26:12,524 MessagingService.java (line 667) Waiting for messaging service to quiesce Thanks Gary
row caching for frequently updated column
I am wondering if there is any negative impact on Cassandra write operation, if I turn on row caching for a table that has mostly 'static columns' but few frequently write columns (like timestamp). The application will frequently write to a few columns, and the application will also frequently query entire row. How Cassandra handle update column to a cached row? does it update both memtables value and also the row cached row's column(which dealing with memory update so it is very fast) ? or in order to update the cached row, entire row need to read back from sstable? thanks
Re: Recommended Approach for Config Changes
Thanks for all the good info. We have found that running drain first before restarting should always be done, even if there is not much data or I/O. Also, we've found that node tool drain returns often before it's finished, so it's important to watch the logs (or opscenter) for it and any compaction tasks to complete before restarting. On Apr 25, 2014 1:09 PM, Jon Haddad j...@jonhaddad.com wrote: You might want to take a peek at what’s happening in the process via strace -p or tcpdump. I can’t remember ever waiting an hour for a node to rejoin. On Apr 25, 2014, at 8:59 AM, Tyler Hobbs ty...@datastax.com wrote: On Fri, Apr 25, 2014 at 10:43 AM, Phil Burress philburress...@gmail.comwrote: Thanks. I made a change to a single node and it took almost an hour to rejoin the cluster (go from DN to UP in nodetool status). The cluster is pretty much idle right now and has a very small dataset. Is that normal? Not unless it had to replay a lot of commitlogs on startup. If you look at your logs and see that that's the case, you may want to run 'nodetool drain' before stopping the node. -- Tyler Hobbs DataStax http://datastax.com/
Migrating Cassandra to New Nodes
I have a new set of nodes and I'd like to migrate my entire cluster onto them without any downtime. I believe that I can launch the new cluster and have them join the ring and then use nodetool to decommission the old nodes one at a time. But, I'm wondering what is the safest way to update the seeds in the cassandra.yaml files? AFAICT, there is nothing particularly special about the choices of seeds? So, prior to starting decom, I was figuring I could update all the seeds to some subset of the new cluster. Is that reliable? TIA, --nash
