gossiper problem
All: I have four cassandra servers in cluster. I do not restart any one of the servers, why the following print show the four servers restart many times? What is the possible reason? The connection between the four server's is good. Swap may be used, because there are other applications run with cassandra server. 10.63.61.71 log INFO [Timer-0] 2011-07-13 10:44:55,732 Gossiper.java (line 181) InetAddress /10.63.61.74 is now dead. INFO [GMFD:1] 2011-07-13 10:44:57,748 Gossiper.java (line 579) InetAddress /10.63.61.74 is now UP INFO [Timer-0] 2011-07-13 15:56:44,630 Gossiper.java (line 181) InetAddress /10.63.61.74 is now dead. INFO [GMFD:1] 2011-07-13 15:56:44,653 Gossiper.java (line 579) InetAddress /10.63.61.74 is now UP INFO [Timer-0] 2011-07-13 16:03:24,391 Gossiper.java (line 181) InetAddress /10.63.61.72 is now dead. INFO [GMFD:1] 2011-07-13 16:03:24,405 Gossiper.java (line 579) InetAddress /10.63.61.72 is now UP INFO [Timer-0] 2011-07-13 22:21:41,246 Gossiper.java (line 181) InetAddress /10.63.61.74 is now dead. INFO [Timer-0] 2011-07-13 22:22:45,602 Gossiper.java (line 181) InetAddress /10.63.61.73 is now dead. INFO [Timer-0] 2011-07-13 22:22:45,602 Gossiper.java (line 181) InetAddress /10.63.61.72 is now dead. INFO [GMFD:1] 2011-07-13 22:22:45,993 Gossiper.java (line 579) InetAddress /10.63.61.73 is now UP INFO [GMFD:1] 2011-07-13 22:22:46,107 Gossiper.java (line 579) InetAddress /10.63.61.72 is now UP INFO [GMFD:1] 2011-07-13 22:22:46,107 Gossiper.java (line 579) InetAddress /10.63.61.74 is now UP INFO [Timer-0] 2011-07-13 22:24:08,812 Gossiper.java (line 181) InetAddress /10.63.61.74 is now dead. INFO [GMFD:1] 2011-07-13 22:24:08,920 Gossiper.java (line 579) InetAddress /10.63.61.74 is now UP 10.63.61.72 log INFO [Timer-0] 2011-07-13 02:06:03,941 Gossiper.java (line 181) InetAddress /10.63.61.71 is now dead. INFO [GMFD:1] 2011-07-13 02:06:05,109 Gossiper.java (line 579) InetAddress /10.63.61.71 is now UP INFO [Timer-0] 2011-07-13 03:39:41,918 Gossiper.java (line 181) InetAddress /10.63.61.71 is now dead. INFO [GMFD:1] 2011-07-13 03:39:45,536 Gossiper.java (line 579) InetAddress /10.63.61.71 is now UP INFO [Timer-0] 2011-07-13 10:10:17,449 Gossiper.java (line 181) InetAddress /10.63.61.74 is now dead. INFO [Timer-0] 2011-07-13 10:10:17,471 Gossiper.java (line 181) InetAddress /10.63.61.71 is now dead. INFO [GMFD:1] 2011-07-13 10:10:18,451 Gossiper.java (line 579) InetAddress /10.63.61.74 is now UP INFO [GMFD:1] 2011-07-13 10:10:18,451 Gossiper.java (line 579) InetAddress /10.63.61.71 is now UP INFO [Timer-0] 2011-07-13 10:44:36,140 Gossiper.java (line 181) InetAddress /10.63.61.71 is now dead. INFO [GMFD:1] 2011-07-13 10:44:57,417 Gossiper.java (line 579) InetAddress /10.63.61.71 is now UP INFO [Timer-0] 2011-07-13 10:45:10,141 Gossiper.java (line 181) InetAddress /10.63.61.71 is now dead. INFO [GMFD:1] 2011-07-13 10:45:14,478 Gossiper.java (line 579) InetAddress /10.63.61.71 is now UP INFO [Timer-0] 2011-07-13 15:14:44,044 Gossiper.java (line 181) InetAddress /10.63.61.71 is now dead. INFO [GMFD:1] 2011-07-13 15:14:47,610 Gossiper.java (line 579) InetAddress /10.63.61.71 is now UP INFO [Timer-0] 2011-07-13 15:56:36,857 Gossiper.java (line 181) InetAddress /10.63.61.71 is now dead. INFO [GMFD:1] 2011-07-13 15:56:44,417 Gossiper.java (line 579) InetAddress /10.63.61.71 is now UP INFO [Timer-0] 2011-07-13 16:02:37,260 Gossiper.java (line 181) InetAddress /10.63.61.71 is now dead. INFO [GMFD:1] 2011-07-13 16:02:52,651 Gossiper.java (line 579) InetAddress /10.63.61.71 is now UP INFO [Timer-0] 2011-07-13 16:03:05,289 Gossiper.java (line 181) InetAddress /10.63.61.71 is now dead. INFO [GMFD:1] 2011-07-13 16:03:11,260 Gossiper.java (line 579) InetAddress /10.63.61.71 is now UP INFO [Timer-0] 2011-07-13 16:08:47,666 Gossiper.java (line 181) InetAddress /10.63.61.71 is now dead. INFO [GMFD:1] 2011-07-13 16:08:48,668 Gossiper.java (line 579) InetAddress /10.63.61.71 is now UP INFO [Timer-0] 2011-07-13 17:38:32,569 Gossiper.java (line 181) InetAddress /10.63.61.71 is now dead. INFO [GMFD:1] 2011-07-13 17:38:34,572 Gossiper.java (line 579) InetAddress /10.63.61.71 is now UP INFO [Timer-0] 2011-07-13 22:20:45,706 Gossiper.java (line 181) InetAddress /10.63.61.71 is now dead. INFO [GMFD:1] 2011-07-13 22:22:46,143 Gossiper.java (line 579) InetAddress /10.63.61.71 is now UP INFO [Timer-0] 2011-07-13 22:23:32,875 Gossiper.java (line 181) InetAddress /10.63.61.71 is now dead. INFO [GMFD:1] 2011-07-13 22:24:08,948 Gossiper.java (line 579) InetAddress /10.63.61.71 is now UP INFO [Timer-0] 2011-07-13 22:32:37,421 Gossiper.java (line 181) InetAddress /10.63.61.71 is now dead. INFO [GMFD:1] 2011-07-13 22:32:38,036 Gossiper.java (line 579) InetAddress /10.63.61.71 is now UP 10.63.61.73 log INFO [Timer-0] 2011-07-13 03:39:42,066 Gossiper.java (line 181) InetAddress /10.63.61.71 is now dead. INFO [GMFD:1] 2011-07-13
Re: gossiper problem
How about GC logs, what are your pause times? JVM settings might help If you are not sure how to enable GC logs check cassandra.yaml look for application pause times. it is highly recommended not to swap -- include JNA jar. Regards, /VJ On Thu, Jul 14, 2011 at 1:42 AM, Donna Li donna...@utstar.com wrote: All: I have four cassandra servers in cluster. I do not restart any one of the servers, why the following print show the four servers restart many times? What is the possible reason? The connection between the four server’s is good. Swap may be used, because there are other applications run with cassandra server. ** ** 10.63.61.71 log INFO [Timer-0] 2011-07-13 10:44:55,732 Gossiper.java (line 181) InetAddress /10.63.61.74 is now dead. INFO [GMFD:1] 2011-07-13 10:44:57,748 Gossiper.java (line 579) InetAddress /10.63.61.74 is now UP INFO [Timer-0] 2011-07-13 15:56:44,630 Gossiper.java (line 181) InetAddress /10.63.61.74 is now dead. INFO [GMFD:1] 2011-07-13 15:56:44,653 Gossiper.java (line 579) InetAddress /10.63.61.74 is now UP INFO [Timer-0] 2011-07-13 16:03:24,391 Gossiper.java (line 181) InetAddress /10.63.61.72 is now dead. INFO [GMFD:1] 2011-07-13 16:03:24,405 Gossiper.java (line 579) InetAddress /10.63.61.72 is now UP INFO [Timer-0] 2011-07-13 22:21:41,246 Gossiper.java (line 181) InetAddress /10.63.61.74 is now dead. INFO [Timer-0] 2011-07-13 22:22:45,602 Gossiper.java (line 181) InetAddress /10.63.61.73 is now dead. INFO [Timer-0] 2011-07-13 22:22:45,602 Gossiper.java (line 181) InetAddress /10.63.61.72 is now dead. INFO [GMFD:1] 2011-07-13 22:22:45,993 Gossiper.java (line 579) InetAddress /10.63.61.73 is now UP INFO [GMFD:1] 2011-07-13 22:22:46,107 Gossiper.java (line 579) InetAddress /10.63.61.72 is now UP INFO [GMFD:1] 2011-07-13 22:22:46,107 Gossiper.java (line 579) InetAddress /10.63.61.74 is now UP INFO [Timer-0] 2011-07-13 22:24:08,812 Gossiper.java (line 181) InetAddress /10.63.61.74 is now dead. INFO [GMFD:1] 2011-07-13 22:24:08,920 Gossiper.java (line 579) InetAddress /10.63.61.74 is now UP ** ** 10.63.61.72 log INFO [Timer-0] 2011-07-13 02:06:03,941 Gossiper.java (line 181) InetAddress /10.63.61.71 is now dead. INFO [GMFD:1] 2011-07-13 02:06:05,109 Gossiper.java (line 579) InetAddress /10.63.61.71 is now UP INFO [Timer-0] 2011-07-13 03:39:41,918 Gossiper.java (line 181) InetAddress /10.63.61.71 is now dead. INFO [GMFD:1] 2011-07-13 03:39:45,536 Gossiper.java (line 579) InetAddress /10.63.61.71 is now UP INFO [Timer-0] 2011-07-13 10:10:17,449 Gossiper.java (line 181) InetAddress /10.63.61.74 is now dead. INFO [Timer-0] 2011-07-13 10:10:17,471 Gossiper.java (line 181) InetAddress /10.63.61.71 is now dead. INFO [GMFD:1] 2011-07-13 10:10:18,451 Gossiper.java (line 579) InetAddress /10.63.61.74 is now UP INFO [GMFD:1] 2011-07-13 10:10:18,451 Gossiper.java (line 579) InetAddress /10.63.61.71 is now UP INFO [Timer-0] 2011-07-13 10:44:36,140 Gossiper.java (line 181) InetAddress /10.63.61.71 is now dead. INFO [GMFD:1] 2011-07-13 10:44:57,417 Gossiper.java (line 579) InetAddress /10.63.61.71 is now UP INFO [Timer-0] 2011-07-13 10:45:10,141 Gossiper.java (line 181) InetAddress /10.63.61.71 is now dead. INFO [GMFD:1] 2011-07-13 10:45:14,478 Gossiper.java (line 579) InetAddress /10.63.61.71 is now UP INFO [Timer-0] 2011-07-13 15:14:44,044 Gossiper.java (line 181) InetAddress /10.63.61.71 is now dead. INFO [GMFD:1] 2011-07-13 15:14:47,610 Gossiper.java (line 579) InetAddress /10.63.61.71 is now UP INFO [Timer-0] 2011-07-13 15:56:36,857 Gossiper.java (line 181) InetAddress /10.63.61.71 is now dead. INFO [GMFD:1] 2011-07-13 15:56:44,417 Gossiper.java (line 579) InetAddress /10.63.61.71 is now UP INFO [Timer-0] 2011-07-13 16:02:37,260 Gossiper.java (line 181) InetAddress /10.63.61.71 is now dead. INFO [GMFD:1] 2011-07-13 16:02:52,651 Gossiper.java (line 579) InetAddress /10.63.61.71 is now UP INFO [Timer-0] 2011-07-13 16:03:05,289 Gossiper.java (line 181) InetAddress /10.63.61.71 is now dead. INFO [GMFD:1] 2011-07-13 16:03:11,260 Gossiper.java (line 579) InetAddress /10.63.61.71 is now UP INFO [Timer-0] 2011-07-13 16:08:47,666 Gossiper.java (line 181) InetAddress /10.63.61.71 is now dead. INFO [GMFD:1] 2011-07-13 16:08:48,668 Gossiper.java (line 579) InetAddress /10.63.61.71 is now UP INFO [Timer-0] 2011-07-13 17:38:32,569 Gossiper.java (line 181) InetAddress /10.63.61.71 is now dead. INFO [GMFD:1] 2011-07-13 17:38:34,572 Gossiper.java (line 579) InetAddress /10.63.61.71 is now UP INFO [Timer-0] 2011-07-13 22:20:45,706 Gossiper.java (line 181) InetAddress /10.63.61.71 is now dead. INFO [GMFD:1] 2011-07-13 22:22:46,143 Gossiper.java (line 579)
Re: gossiper problem
well I am not a JVM guru, but it seem server has memory problem. 13 10:44:57,748 Gossiper.java (line 579) InetAddress /10.63.61.74 is now UP INFO [Timer-0] 2011-07-13 15:56:44,630 Gossiper.java (line 181) InetAddress /10.63.61.74 is now dead. INFO [GMFD:1] 2011-07-13 15:56:44,653 Gossiper.java (line 579) InetAddress /10.63.61.74 is now UP INFO [Timer-0] 2011-07-13 16:03:24,391 Gossiper.java (line 181) InetAddress /10.63.61.72 is now dead. It is swapping due to memory need, recommended!! disable swap. rather die with OOM than swapping. INFO [GC inspection] 2011-07-13 03:12:06,153 GCInspector.java (line 110) GC for ConcurrentMarkSweep: 1097 ms, 371528920 reclaimed leaving 17677528 used; max is 118784 INFO [GC inspection] 2011-07-13 03:12:07,351 GCInspector.java (line 110) GC for ParNew: 466 ms, 20619976 reclaimed leaving 157240232 used; max is 118784 INFO [GC inspection] 2011-07-13 03:25:54,378 GCInspector.java (line 110) GC for ParNew: 283 ms, 26850072 reclaimed leaving 154180424 used; max is 118784 INFO [GC inspection] 2011-07-13 06:29:58,092 GCInspector.java (line 110) GC for ParNew: 538 ms, 17358792 reclaimed leaving My cassandra version is **0.6.3**, and the configuration about gc on storage_conf.xml is GCGraceSeconds864000/GCGraceSeconds JVM configuration is as following: JVM_OPTS= \ -ea \ -Xms**256M** \ -Xmx**1G** \ -XX:+UseParNewGC \ Can I decrease the JVM_OPTS to –Xms**128M** –Xmx**512M** to avoid swap, the data saved in cassandra is small, I do not need so much memory. Reducing max head size wont solve problem, i think it will do more swapping. data only does not only count for memory requirement, but no. of memtables, as each CF has separate memtable and its size, compaction, caching, read You should upgrade to 0.7 or later. /samal
