[Bug 1153077] Re: rabbitmq queue fills up and celery stops executing tasks when upload_dhcp_leases is done every minute (by default)
I've just hit this again overnight. Logs are attached, command output is below. $ sudo rabbitmqctl list_queues -p /maas_workers name messages_unacknowledged Listing queues ... c1d352af-8060-4bf4-b820-f468b22a97fb4 celery 0 maas-juju.celeryd.pidbox0 master 0 ...done. $ sudo service maas-cluster-celery status maas-cluster-celery start/running, process 6499 $ sudo service maas-region-celery status maas-region-celery start/running, process 6555 ** Attachment added: celery-logs.tar.gz https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1153077/+attachment/3689937/+files/celery-logs.tar.gz -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to maas in Ubuntu. https://bugs.launchpad.net/bugs/1153077 Title: rabbitmq queue fills up and celery stops executing tasks when upload_dhcp_leases is done every minute (by default) To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1153077/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1153077] Re: rabbitmq queue fills up and celery stops executing tasks when upload_dhcp_leases is done every minute (by default)
I've just hit this again overnight. Logs are attached, command output is below. $ sudo rabbitmqctl list_queues -p /maas_workers name messages_unacknowledged Listing queues ... c1d352af-8060-4bf4-b820-f468b22a97fb4 celery 0 maas-juju.celeryd.pidbox0 master 0 ...done. $ sudo service maas-cluster-celery status maas-cluster-celery start/running, process 6499 $ sudo service maas-region-celery status maas-region-celery start/running, process 6555 ** Attachment added: celery-logs.tar.gz https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1153077/+attachment/3689937/+files/celery-logs.tar.gz -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1153077 Title: rabbitmq queue fills up and celery stops executing tasks when upload_dhcp_leases is done every minute (by default) To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1153077/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1153077] Re: rabbitmq queue fills up and celery stops executing tasks when upload_dhcp_leases is done every minute (by default)
Leases file is attached. I am working on figuring out why there are so many abandoned leases, but hopefully MAAS can be made to not choke on such a large file. ** Attachment added: dhcpd.leases.gz https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1153077/+attachment/3689992/+files/dhcpd.leases.gz -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1153077 Title: rabbitmq queue fills up and celery stops executing tasks when upload_dhcp_leases is done every minute (by default) To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1153077/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1153077] Re: rabbitmq queue fills up and celery stops executing tasks when upload_dhcp_leases is done every minute (by default)
I think I'm seeing the same issue. I have a celeryd process that appears to be stuck in a loop and a queue in the vhost /maas_workers named c1d352af-8060-4bf4-b820-f468b22a97fb. One thing that stands out to me is that the queue has 4 unacknowledged messages at all times. New ones seem to come in and get acked, but the 4 remain. Here are a couple example messages. The content-type is application/x -python-serialize (dp1 S'retries' p2 I0 sS'task' p3 S'provisioningserver.tasks.upload_dhcp_leases' p4 sS'kwargs' p5 (dp6 sS'eta' p7 NsS'args' p8 (lp9 sS'id' p10 S'ac7c0755-43ab-40b8-bc39-5315236a3639' p11 sS'expires' p12 NsS'utc' p13 I00 s. -- (dp1 S'retries' p2 I0 sS'task' p3 S'provisioningserver.tasks.report_boot_images' p4 sS'kwargs' p5 (dp6 sS'eta' p7 NsS'args' p8 (lp9 sS'id' p10 S'd5447c64-8317-4d13-a5ce-d1c818c2efc0' p11 sS'expires' p12 NsS'utc' p13 I00 s. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to maas in Ubuntu. https://bugs.launchpad.net/bugs/1153077 Title: rabbitmq queue fills up and celery stops executing tasks when upload_dhcp_leases is done every minute (by default) To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1153077/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1153077] Re: rabbitmq queue fills up and celery stops executing tasks when upload_dhcp_leases is done every minute (by default)
I think I'm seeing the same issue. I have a celeryd process that appears to be stuck in a loop and a queue in the vhost /maas_workers named c1d352af-8060-4bf4-b820-f468b22a97fb. One thing that stands out to me is that the queue has 4 unacknowledged messages at all times. New ones seem to come in and get acked, but the 4 remain. Here are a couple example messages. The content-type is application/x -python-serialize (dp1 S'retries' p2 I0 sS'task' p3 S'provisioningserver.tasks.upload_dhcp_leases' p4 sS'kwargs' p5 (dp6 sS'eta' p7 NsS'args' p8 (lp9 sS'id' p10 S'ac7c0755-43ab-40b8-bc39-5315236a3639' p11 sS'expires' p12 NsS'utc' p13 I00 s. -- (dp1 S'retries' p2 I0 sS'task' p3 S'provisioningserver.tasks.report_boot_images' p4 sS'kwargs' p5 (dp6 sS'eta' p7 NsS'args' p8 (lp9 sS'id' p10 S'd5447c64-8317-4d13-a5ce-d1c818c2efc0' p11 sS'expires' p12 NsS'utc' p13 I00 s. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1153077 Title: rabbitmq queue fills up and celery stops executing tasks when upload_dhcp_leases is done every minute (by default) To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1153077/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
