Posting my own follow up, I got this issue resolved. At some point during the build-out of the Foreman server, the file system where MongoDB is installed ran out of space due to a typo on my end. Once the FS was extended everything seems to have been fine, but it looks like there was a persistent problem with the 'Listen on candlepin events' and 'Monitor Event Queue' tasks. While they were in state running, they ever seemed to do anything. After digging into the code and not being able to figure out why they might get hung in a way that persists across reboots, I crossed my fingers and destroyed the tasks from the foreman-rake console and restarted foreman-tasks. Both tasks came right back up and started processing information! It took more than 24 hours for the candlepin events to catch up with the past 4 months of events, but it eventually did and the queue size is down to 0. Errata are now showing as applicable, and can be searched and managed as expected.
It hasn't been running long enough to tell, but I believe the memory leak I mentioned in another posting may be fixed as well. I no longer seem to have a dynflow_executor as my top memory consumer on the server. So, TL;DR, if the mongodb file system fills up, it is possible for the long running tasks to get in a "stuck" state where they're running, but not doing any useful work, even across reboots. Leaving this here in case anyone else runs into the same issues. -- You received this message because you are subscribed to the Google Groups "Foreman users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/foreman-users. For more options, visit https://groups.google.com/d/optout.
