Bug#923464: carbon-cache unusable in Buster
severity 923464 important thanks Hi, we also ran into this bug and I can confirm the attached patch by Mark fixed the issue for us. This should definitely be fixed in unstable/testing and then also in buster. I set the severity to important as the repeated crashes of carbon-cache caused regular metric data loss on our server as well as oom situations (where was not clear what was happening in the beginning) and fast filling up of /var/log due to excessive traceback logging. Thanks, Maxi On Sat, 11 Jan 2020 12:05:17 + Mark Hymers wrote: > On Thu, 07, Nov, 2019 at 10:57:23AM +0100, Morten Brekkevold spoke thus.. > > The carbon-cache package in Debian Buster is definitely BROKEN - it just > > keeps crashing. > > Hi, > > We've been running with the attached patch since December which fixes > the problem. > > Thorsten (as you're the last uploader) - I'm happy to prepare a stable > update to fix this, but the version is the same in stable/testing so > can't do so currently. There are two point release (1.1.5 and 1.1.6) > for this package but they seem to be tied into other graphite-* package > updates too. Would you accept an NMU to unstable with just this patch > and then an accompanying stable update? This problem basically makes > graphite-carbon unusable in buster. > > Thanks. > > Mark > > -- > Mark Hymers signature.asc Description: This is a digitally signed message part.
Bug#923464: carbon-cache unusable in Buster
On Thu, 07, Nov, 2019 at 10:57:23AM +0100, Morten Brekkevold spoke thus..
> The carbon-cache package in Debian Buster is definitely BROKEN - it just
> keeps crashing.
Hi,
We've been running with the attached patch since December which fixes
the problem.
Thorsten (as you're the last uploader) - I'm happy to prepare a stable
update to fix this, but the version is the same in stable/testing so
can't do so currently. There are two point release (1.1.5 and 1.1.6)
for this package but they seem to be tied into other graphite-* package
updates too. Would you accept an NMU to unstable with just this patch
and then an accompanying stable update? This problem basically makes
graphite-carbon unusable in buster.
Thanks.
Mark
--
Mark Hymers
--- graphite-carbon/lib/carbon/cache.py 2020-01-11 11:58:15.877162033 +
+++ cache.py 2020-01-11 11:55:19.516768965 +
@@ -184,7 +185,8 @@
if not self:
return (None, [])
if self.strategy:
- metric = self.strategy.choose_item()
+ with self.lock:
+metric = self.strategy.choose_item()
else:
# Avoid .keys() as it dumps the whole list
metric = next(iter(self))
@@ -206,18 +208,18 @@
def store(self, metric, datapoint):
timestamp, value = datapoint
-if timestamp not in self[metric]:
- # Not a duplicate, hence process if cache is not full
- if self.is_full:
-log.msg("MetricCache is full: self.size=%d" % self.size)
-events.cacheFull()
- else:
-with self.lock:
+with self.lock:
+ if timestamp not in self[metric]:
+# Not a duplicate, hence process if cache is not full
+if self.is_full:
+ log.msg("MetricCache is full: self.size=%d" % self.size)
+ events.cacheFull()
+else:
self.size += 1
self[metric][timestamp] = value
-else:
- # Updating a duplicate does not increase the cache size
- self[metric][timestamp] = value
+ else:
+# Updating a duplicate does not increase the cache size
+self[metric][timestamp] = value
_Cache = None
signature.asc
Description: PGP signature
Bug#923464: carbon-cache unusable in Buster
On Sat, 11, Jan, 2020 at 12:05:17PM +, Mark Hymers spoke thus.. > On Thu, 07, Nov, 2019 at 10:57:23AM +0100, Morten Brekkevold spoke thus.. > > The carbon-cache package in Debian Buster is definitely BROKEN - it just > > keeps crashing. > > Hi, > > We've been running with the attached patch since December which fixes > the problem. > > Thorsten (as you're the last uploader) - I'm happy to prepare a stable Urgh - sorry, Thomas, not Thorsten. My brain said one thing and my fingers typed another. Sorry. Mark -- Mark Hymers signature.asc Description: PGP signature
Bug#923464: carbon-cache unusable in Buster
The carbon-cache package in Debian Buster is definitely BROKEN - it just keeps crashing. I would say: Severity: grave -- mvh Morten Brekkevold Uninett
Bug#923464: carbon-cache unusable in Buster
Hi, Since Buster installation carbon-cache daemon is very often freezing with this logs : Traceback (most recent call last): File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext result = inContext.theWork() File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in inContext.theWork = lambda: context.call(ctx, func, *args, **kw) File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext return func(*args,**kw) --- --- File "/usr/lib/python3/dist-packages/carbon/writer.py", line 189, in writeForever writeCachedDataPoints() File "/usr/lib/python3/dist-packages/carbon/writer.py", line 98, in writeCachedDataPoints (metric, datapoints) = cache.drain_metric() File "/usr/lib/python3/dist-packages/carbon/cache.py", line 187, in drain_metric metric = self.strategy.choose_item() File "/usr/lib/python3/dist-packages/carbon/cache.py", line 116, in choose_item return next(self.queue) builtins.StopIteration: Is seems to be related to Github issues. carbon-cache daemon has to be restarted very, very often… -- Sylvain
