Posting my own follow up, I got this issue resolved. At some point during
the build-out of the Foreman server, the file system where MongoDB is
installed ran out of space due to a typo on my end. Once the FS was
extended everything seems to have been fine, but it looks like there was a
persistent problem with the 'Listen on candlepin events' and 'Monitor Event
Queue' tasks. While they were in state running, they ever seemed to do
anything. After digging into the code and not being able to figure out why
they might get hung in a way that persists across reboots, I crossed my
fingers and destroyed the tasks from the foreman-rake console and restarted
foreman-tasks. Both tasks came right back up and started processing
information! It took more than 24 hours for the candlepin events to catch
up with the past 4 months of events, but it eventually did and the queue
size is down to 0. Errata are now showing as applicable, and can be
searched and managed as expected.
It hasn't been running long enough to tell, but I believe the memory leak I
mentioned in another posting may be fixed as well. I no longer seem to have
a dynflow_executor as my top memory consumer on the server.
So, TL;DR, if the mongodb file system fills up, it is possible for the long
running tasks to get in a "stuck" state where they're running, but not
doing any useful work, even across reboots.
Leaving this here in case anyone else runs into the same issues.
--
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to foreman-users+unsubscr...@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
