jiangpengcheng commented on PR #5251:
URL: https://github.com/apache/openwhisk/pull/5251#issuecomment-1139194671
> I have an issue where the `namespaceContainer` metric still emits
containers for a namespace even after no activations are being run in a long
time for that namespace. The value just remains constant forever until I
restart the scheduler in which it then correctly goes to 0 not emit for that
namespace. My thought was something getting stuck in memory with the memory
queue even after it should have been shut down since the metric is reported
from that actor. And it's weird that it would still report that there are
containers even if there are none in etcd for the namespace even if the memory
queue wasn't properly shut down as I assume it would get updated with the
correct value when still emitting unless it's stuck in a zombie state or
something. Do you think this could be the same issue?
do you mean these metrics in `MemoryQueue.scala`?
```
MetricEmitter.emitGaugeMetric(
LoggingMarkers.SCHEDULER_NAMESPACE_CONTAINER(invocationNamespace),
namespaceContainerCount.existingContainerNumByNamespace)
MetricEmitter.emitGaugeMetric(
LoggingMarkers.SCHEDULER_NAMESPACE_INPROGRESS_CONTAINER(invocationNamespace),
namespaceContainerCount.inProgressContainerNumByNamespace)
MetricEmitter.emitGaugeMetric(
LoggingMarkers.SCHEDULER_ACTION_CONTAINER(invocationNamespace,
action.asString),
containers.size)
MetricEmitter.emitGaugeMetric(
LoggingMarkers.SCHEDULER_ACTION_INPROGRESS_CONTAINER(invocationNamespace,
action.asString),
creationIds.size)
```
looks like some memory queue under the namespace are not terminated
the `Shcduler` provides a http api `queue/status` which return memory queue
status inside it, you can check whether all queues are terminated when error
happens
this issue is caused by `MemoryQueue` is removed while leader key in etcd is
not, so they are not related
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
