Updating here for posterity in the list: I ultimately had to go nuclear and destroy all the tasks using pulp-admin. This got me unstuck enough to be able to go in and take care of the rest of the 'stuck' tasks (orphaned stuff from hosts now deleted, etc).
On Thu, Jun 15, 2017 at 10:26 AM John Mitsch <[email protected]> wrote: > Neil, > > Pulp is really backed up with tasks, my guess would be there was some > sort of deadlock in qpid and it "unlocked", releasing all of these tasks. > My advice would be first try stopping then starting these services > pulp_celerybeat.service pulp_workers.service pulp_resource_manager.service, > and then restarting qpidd and httpd. Wait a bit and see if tasks start > clearing out. > > If that doesn't work and you want to go a more forceful route, you can try > deleting the tasks in pulp. This can cause some serious issues if you have > open tasks that you actually care about (like a repo sync), so if you do > wind up doing this, make sure any tasks you care about are finished up. > You've been warned :) > > pulp-admin -u admin -p $pulpAdminPassword tasks list | grep "Task Id"| awk > '{print $3}' | xargs -I % pulp-admin -u admin -p $pulpAdminPassword tasks > cancel --task-id % > > If you get unstuck, I would also advise upgrading to 3.4. Let me know how > it goes! > > -John > > > John Mitsch > Red Hat Engineering > (860)-967-7285 <(860)%20967-7285> > irc: jomitsch > > On Thu, Jun 15, 2017 at 10:13 AM, Neil Hanlon <[email protected]> wrote: > >> Thanks for taking a look for me. Here's the output of the tasks list. >> >> >> On Thu, Jun 15, 2017 at 10:07 AM John Mitsch <[email protected]> wrote: >> >>> Seems like something funny is going on with the pulp tasks, can you >>> install pulp-admin and take a look at the tasks? We may need to kill >>> something at that level >>> >>> to install: >>> https://gist.github.com/johnpmitsch/866fb430780ba0e95a5554566b989416 >>> >>> -John >>> >>> John Mitsch >>> Red Hat Engineering >>> (860)-967-7285 <(860)%20967-7285> >>> irc: jomitsch >>> >>> On Tue, Jun 13, 2017 at 5:06 PM, Neil Hanlon <[email protected]> wrote: >>> >>>> Certainly. Going to let that run overnight :) >>>> >>>> On Tue, Jun 13, 2017 at 5:00 PM John Mitsch <[email protected]> >>>> wrote: >>>> >>>>> That is very strange, this may be a good candidate for a foreman-debug >>>>> so we can dig through logs. Can you create one and send it to me >>>>> privately? >>>>> >>>>> -John >>>>> >>>>> John Mitsch >>>>> Red Hat Engineering >>>>> (860)-967-7285 <(860)%20967-7285> >>>>> irc: jomitsch >>>>> >>>>> On Tue, Jun 13, 2017 at 3:34 PM, <[email protected]> wrote: >>>>> >>>>>> So mostly we use foreman/katello just for syncing operating systems >>>>>> out to capsules/smart proxies, and for managing system lifecycle >>>>>> (basically, just kickstarts). I went last week to add a XenServer 7.2 OS >>>>>> into my Katello Products, and started having tons of issues, specifically >>>>>> in that I had 20k+ 'Actions::Katello::Host::UploadPackageProfile' >>>>>> tasks sitting in my list as state = running, result = pending--but not >>>>>> doing anything at all. >>>>>> >>>>>> To be honest, I don't really care why they're not working--I just >>>>>> want them to stop being scheduled at all. Today I've shut off >>>>>> foreman-tasks >>>>>> and pulp_workers and tried to set all these tasks which were >>>>>> running/pending to stopped/error, and that worked. Seemed to help out >>>>>> with >>>>>> the issues of Foreman. However I still had some tasks that weren't being >>>>>> scheduled, so I checked and sure enough, foreman tasks was running, as >>>>>> was >>>>>> pulp, but it was complaining about not having a dynflow environment. >>>>>> Restarting foreman tasks and waiting a few minutes helped, at which >>>>>> point I >>>>>> started seeing stuff like this spam my production.log, and tons of >>>>>> Actions::Katello::Host::UploadPackageProfile >>>>>> tasks being scheduled again--with dates from *months* ago. >>>>>> >>>>>> 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step >>>>>> a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 got event >>>>>> Dynflow::Action::Polling::Poll >>>>>> 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step >>>>>> c0e798bf-a9f9-40c7-9d55-75d686454f42: 4 suspended >> running in phase >>>>>> Run Actions::Pulp::Consumer::GenerateApplicability >>>>>> 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step >>>>>> df8fcf4c-0de7-4cca-8aaf-f38adcd2b6ec: 4 running >> suspended in phase >>>>>> Run Actions::Pulp::Consumer::GenerateApplicability >>>>>> 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step >>>>>> a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 suspended >> running in phase >>>>>> Run Actions::Pulp::Consumer::GenerateApplicability >>>>>> 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step >>>>>> f96f15cb-5933-4b18-956d-a2e4c7052004: 4 suspended >> running in phase >>>>>> Run Actions::Pulp::Consumer::GenerateApplicability >>>>>> 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step >>>>>> dc8976be-99e7-4dc4-8a3f-893d3687a1cc: 4 running >> suspended in phase >>>>>> Run Actions::Pulp::Consumer::GenerateApplicability >>>>>> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " >>>>>> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/41b06756-a517-415b-bcb9-9002dc0e5fec/", >>>>>> "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, >>>>>> deflate", "accept"=>"application/json", >>>>>> "content_type"=>"application/json" >>>>>> | \nRestClient.get >>>>>> "https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/", >>>>>> "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, >>>>>> deflate", "accept"=>"application/json", >>>>>> "content_type"=>"application/json" >>>>>> | \n# => 200 OK | application/json 326 bytes >>>>>> | >>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>>>>> c0e798bf-a9f9-40c7-9d55-75d686454f42: 4 running >> suspended in phase >>>>>> Run Actions::Pulp::Consumer::GenerateApplicability >>>>>> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " >>>>>> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/41b06756-a517-415b-bcb9-9002dc0e5fec/", >>>>>> "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, >>>>>> deflate", "accept"=>"application/json", >>>>>> "content_type"=>"application/json" >>>>>> | \nRestClient.get >>>>>> "https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/", >>>>>> "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, >>>>>> deflate", "accept"=>"application/json", >>>>>> "content_type"=>"application/json" >>>>>> | \n# => 200 OK | application/json 326 bytes >>>>>> | \n# => 200 OK | application/json 328 bytes >>>>>> | >>>>>> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " >>>>>> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/41b06756-a517-415b-bcb9-9002dc0e5fec/", >>>>>> "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, >>>>>> deflate", "accept"=>"application/json", >>>>>> "content_type"=>"application/json" >>>>>> | \nRestClient.get >>>>>> "https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/", >>>>>> "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, >>>>>> deflate", "accept"=>"application/json", >>>>>> "content_type"=>"application/json" >>>>>> | \n# => 200 OK | application/json 326 bytes >>>>>> | \n# => 200 OK | application/json 328 bytes >>>>>> | \n# => 200 OK | application/json 328 bytes >>>>>> | >>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>>>>> f96f15cb-5933-4b18-956d-a2e4c7052004: 4 running >> suspended in phase >>>>>> Run Actions::Pulp::Consumer::GenerateApplicability >>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>>>>> a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 running >> suspended in phase >>>>>> Run Actions::Pulp::Consumer::GenerateApplicability >>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>>>>> 1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 got event >>>>>> Dynflow::Action::Polling::Poll >>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>>>>> 1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 suspended >> running in phase >>>>>> Run Actions::Pulp::Consumer::GenerateApplicability >>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>>>>> 2392d32f-0886-4473-ae5a-7595504acb39: 4 got event >>>>>> Dynflow::Action::Polling::Poll >>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>>>>> 2392d32f-0886-4473-ae5a-7595504acb39: 4 suspended >> running in phase >>>>>> Run Actions::Pulp::Consumer::GenerateApplicability >>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>>>>> a03805a6-1725-4a92-b3f5-05514cc52c81: 4 got event >>>>>> Dynflow::Action::Polling::Poll >>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>>>>> a03805a6-1725-4a92-b3f5-05514cc52c81: 4 suspended >> running in phase >>>>>> Run Actions::Pulp::Consumer::GenerateApplicability >>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>>>>> 96b10629-bfed-4cc5-a9b8-b80e1fc5b302: 4 got event >>>>>> Dynflow::Action::Polling::Poll >>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>>>>> 96b10629-bfed-4cc5-a9b8-b80e1fc5b302: 4 suspended >> running in phase >>>>>> Run Actions::Pulp::Consumer::GenerateApplicability >>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>>>>> 903f497f-a539-45af-a220-4cf3d7dd70e2: 4 got event >>>>>> Dynflow::Action::Polling::Poll >>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>>>>> 903f497f-a539-45af-a220-4cf3d7dd70e2: 4 suspended >> running in phase >>>>>> Run Actions::Pulp::Consumer::GenerateApplicability >>>>>> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " >>>>>> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/", >>>>>> "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, >>>>>> deflate", "accept"=>"application/json", >>>>>> "content_type"=>"application/json" >>>>>> | \n# => 200 OK | application/json 329 bytes >>>>>> | >>>>>> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " >>>>>> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/", >>>>>> "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, >>>>>> deflate", "accept"=>"application/json", >>>>>> "content_type"=>"application/json" >>>>>> | \n# => 200 OK | application/json 329 bytes >>>>>> | \n# => 200 OK | application/json 329 bytes >>>>>> | >>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>>>>> 1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 running >> suspended in phase >>>>>> Run Actions::Pulp::Consumer::GenerateApplicability >>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>>>>> 2392d32f-0886-4473-ae5a-7595504acb39: 4 running >> suspended in phase >>>>>> Run Actions::Pulp::Consumer::GenerateApplicability >>>>>> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " >>>>>> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/", >>>>>> "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, >>>>>> deflate", "accept"=>"application/json", >>>>>> "content_type"=>"application/json" >>>>>> | \n# => 200 OK | application/json 329 bytes >>>>>> | \n# => 200 OK | application/json 329 bytes >>>>>> | \n# => 200 OK | application/json 330 bytes >>>>>> | >>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>>>>> a03805a6-1725-4a92-b3f5-05514cc52c81: 4 running >> suspended in phase >>>>>> Run Actions::Pulp::Consumer::GenerateApplicability >>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>>>>> 3c827032-6d3b-4a5b-975c-cf78ec6268bd: 4 got event >>>>>> Dynflow::Action::Polling::Poll >>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>>>>> 3c827032-6d3b-4a5b-975c-cf78ec6268bd: 4 suspended >> running in phase >>>>>> Run Actions::Pulp::Consumer::GenerateApplicability >>>>>> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " >>>>>> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/", >>>>>> "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, >>>>>> deflate", "accept"=>"application/json", >>>>>> "content_type"=>"application/json" >>>>>> | \n# => 200 OK | application/json 329 bytes >>>>>> | \n# => 200 OK | application/json 329 bytes >>>>>> | \n# => 200 OK | application/json 330 bytes >>>>>> | \n# => 200 OK | application/json 327 bytes >>>>>> | >>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>>>>> 903f497f-a539-45af-a220-4cf3d7dd70e2: 4 running >> suspended in phase >>>>>> Run Actions::Pulp::Consumer::GenerateApplicability >>>>>> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " >>>>>> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/", >>>>>> "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, >>>>>> deflate", "accept"=>"application/json", >>>>>> "content_type"=>"application/json" >>>>>> | \n# => 200 OK | application/json 329 bytes >>>>>> | \n# => 200 OK | application/json 329 bytes >>>>>> >>>>>> >>>>>> >>>>>> <https://lh3.googleusercontent.com/-sEewHxc4UuA/WUA9z6RVaWI/AAAAAAAABJM/mWeOvhRvpgknjIOZs8Unwt1l2791tugbACLcBGAs/s1600/Selection_133.png> >>>>>> >>>>>> >>>>>> >>>>>> I'm happy to upload any logs anyone might want to look at to help me >>>>>> with this.. but I'm at my wits end with this. I'm about ready to tear it >>>>>> all down and just rebuild it from scratch... >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Foreman users" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To post to this group, send email to [email protected]. >>>>>> Visit this group at https://groups.google.com/group/foreman-users. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Foreman users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at https://groups.google.com/group/foreman-users. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Foreman users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at https://groups.google.com/group/foreman-users. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Foreman users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at https://groups.google.com/group/foreman-users. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Foreman users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/foreman-users. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Foreman users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/foreman-users. For more options, visit https://groups.google.com/d/optout.
