Question #244564 on Graphite changed: https://answers.launchpad.net/graphite/+question/244564
D DC gave more information on the question: Adding more information: I started carbon-cache in debug mode and it ran for 3 days without any problem. Max cache-size reached 480k (configured as MAX_CACHE_SIZE = 25000000, so no limit problem during that time). Yesterday at 22:30 the cache size started to increase drastically, no data written on storage anymore, and eventually the 25M cache limit was reached in a matter of few hours, as per the log info: 19/07/2014 22:05:58 :: [console] Sorted 28049 cache queues in 0.014046 seconds 19/07/2014 22:17:50 :: [console] Sorted 28122 cache queues in 0.016169 seconds <- last line in console.log. Same for the debug messages, the updates were fine until that time, and then there was a huge delay until the next message: Max memory reached, see here: 19/07/2014 22:28:02 :: [updates] wrote 13 datapoints for server.x.disk-sda.disk_time.read in 0.00916 seconds 19/07/2014 22:28:02 :: [updates] wrote 13 datapoints for server.y.disk-sdad.disk_ops.write in 0.00929 seconds 19/07/2014 22:28:02 :: [updates] wrote 13 datapoints for server.x.disk-sdp.disk_octets.read in 0.00886 seconds 20/07/2014 12:30:18 :: [console] MetricCache is full: self.size=25000000 20/07/2014 12:30:18 :: [console] MetricCache is full: self.size=25000001 20/07/2014 12:30:18 :: [console] MetricCache is full: self.size=25000002 20/07/2014 12:30:18 :: [console] MetricCache is full: self.size=25000003 .... and so on... As we can see there is a huge 14 hours gap, while no activity occurred on the storage, and the cache memory increased to the limit of 25M. In the end the process hang. -- You received this question notification because you are a member of graphite-dev, which is an answer contact for Graphite. _______________________________________________ Mailing list: https://launchpad.net/~graphite-dev Post to : graphite-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~graphite-dev More help : https://help.launchpad.net/ListHelp
