Re: High CPU usage as of 8pm eastern time
Thank you for the mail. Same here, but I restarted the affected server before I noticed your mail. It affected both OpenJDK Java 6 (packaged with Ubuntu 10.04) and Oracle Java 7 processes. Ubuntu 32 bit servers had no issues, only a 64 bit machine. Likely it is related to the leap second introduced today. On 2012.07.01. 5:11, Mina Naguib wrote: Hi folks Our cassandra (and other java-based apps) started experiencing extremely high CPU usage as of 8pm eastern time (midnight UTC). The issue appears to be related to specific versions of java + linux + ntpd There are many solutions floating around on IRC, twitter, stackexchange, LKML. The simplest one that worked for us is simply to run this command on each affected machine: date; date `date +%m%d%H%M%C%y.%S`; date; CPU drop was instantaneous - there was no need to restart the server, ntpd, or any of the affected JVMs.
Re: High CPU usage as of 8pm eastern time
More information for others that were affected. Our installation of java: [root@inv4 conf]# java -version java version 1.6.0_30 Java(TM) SE Runtime Environment (build 1.6.0_30-b12) Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode) [root@inv4 conf]# uname -a Linux inv4 2.6.32-220.4.2.el6.x86_64 #1 SMP Tue Feb 14 04:00:16 GMT 2012 x86_64 x86_64 x86_64 GNU/Linux Jonathan pointed out a Linux bug that may be related: https://issues.apache.org/jira/browse/CASSANDRA-4066 In my case only the Java process went nuts, as seems to be the case in many other reports: https://bugzilla.mozilla.org/show_bug.cgi?id=769972 http://www.wired.com/wiredenterprise/2012/07/leap-second-bug-wreaks-havoc-with-java-linux/ I hope everyone got enough sleep! - David On Sun, Jul 1, 2012 at 4:49 AM, Hontvári József Levente hontv...@flyordie.com wrote: Thank you for the mail. Same here, but I restarted the affected server before I noticed your mail. It affected both OpenJDK Java 6 (packaged with Ubuntu 10.04) and Oracle Java 7 processes. Ubuntu 32 bit servers had no issues, only a 64 bit machine. Likely it is related to the leap second introduced today. On 2012.07.01. 5:11, Mina Naguib wrote: Hi folks Our cassandra (and other java-based apps) started experiencing extremely high CPU usage as of 8pm eastern time (midnight UTC). The issue appears to be related to specific versions of java + linux + ntpd There are many solutions floating around on IRC, twitter, stackexchange, LKML. The simplest one that worked for us is simply to run this command on each affected machine: date; date `date +%m%d%H%M%C%y.%S`; date; CPU drop was instantaneous - there was no need to restart the server, ntpd, or any of the affected JVMs.
High CPU usage as of 8pm eastern time
Hi folks Our cassandra (and other java-based apps) started experiencing extremely high CPU usage as of 8pm eastern time (midnight UTC). The issue appears to be related to specific versions of java + linux + ntpd There are many solutions floating around on IRC, twitter, stackexchange, LKML. The simplest one that worked for us is simply to run this command on each affected machine: date; date `date +%m%d%H%M%C%y.%S`; date; CPU drop was instantaneous - there was no need to restart the server, ntpd, or any of the affected JVMs.
Re: High CPU usage as of 8pm eastern time
YES! This happened to me as well. RUNNING THAT COMMAND FIXED THE PROBLEM! THANK YOU SO MUCH On Sat, Jun 30, 2012 at 11:11 PM, Mina Naguib mina.nag...@bloomdigital.com wrote: Hi folks Our cassandra (and other java-based apps) started experiencing extremely high CPU usage as of 8pm eastern time (midnight UTC). The issue appears to be related to specific versions of java + linux + ntpd There are many solutions floating around on IRC, twitter, stackexchange, LKML. The simplest one that worked for us is simply to run this command on each affected machine: date; date `date +%m%d%H%M%C%y.%S`; date; CPU drop was instantaneous - there was no need to restart the server, ntpd, or any of the affected JVMs. -- David Daeschler
