Seems like something funny is going on with the pulp tasks, can you install pulp-admin and take a look at the tasks? We may need to kill something at that level
to install: https://gist.github.com/johnpmitsch/866fb430780ba0e95a5554566b989416 -John John Mitsch Red Hat Engineering (860)-967-7285 irc: jomitsch On Tue, Jun 13, 2017 at 5:06 PM, Neil Hanlon <[email protected]> wrote: > Certainly. Going to let that run overnight :) > > On Tue, Jun 13, 2017 at 5:00 PM John Mitsch <[email protected]> wrote: > >> That is very strange, this may be a good candidate for a foreman-debug so >> we can dig through logs. Can you create one and send it to me privately? >> >> -John >> >> John Mitsch >> Red Hat Engineering >> (860)-967-7285 <(860)%20967-7285> >> irc: jomitsch >> >> On Tue, Jun 13, 2017 at 3:34 PM, <[email protected]> wrote: >> >>> So mostly we use foreman/katello just for syncing operating systems out >>> to capsules/smart proxies, and for managing system lifecycle (basically, >>> just kickstarts). I went last week to add a XenServer 7.2 OS into my >>> Katello Products, and started having tons of issues, specifically in that I >>> had 20k+ 'Actions::Katello::Host::UploadPackageProfile' tasks sitting >>> in my list as state = running, result = pending--but not doing anything at >>> all. >>> >>> To be honest, I don't really care why they're not working--I just want >>> them to stop being scheduled at all. Today I've shut off foreman-tasks and >>> pulp_workers and tried to set all these tasks which were running/pending to >>> stopped/error, and that worked. Seemed to help out with the issues of >>> Foreman. However I still had some tasks that weren't being scheduled, so I >>> checked and sure enough, foreman tasks was running, as was pulp, but it was >>> complaining about not having a dynflow environment. Restarting foreman >>> tasks and waiting a few minutes helped, at which point I started seeing >>> stuff like this spam my production.log, and tons of >>> Actions::Katello::Host::UploadPackageProfile tasks being scheduled >>> again--with dates from *months* ago. >>> >>> 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step >>> a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 got event >>> Dynflow::Action::Polling::Poll >>> 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step >>> c0e798bf-a9f9-40c7-9d55-75d686454f42: 4 suspended >> running in phase >>> Run Actions::Pulp::Consumer::GenerateApplicability >>> 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step >>> df8fcf4c-0de7-4cca-8aaf-f38adcd2b6ec: 4 running >> suspended in phase >>> Run Actions::Pulp::Consumer::GenerateApplicability >>> 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step >>> a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 suspended >> running in phase >>> Run Actions::Pulp::Consumer::GenerateApplicability >>> 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step >>> f96f15cb-5933-4b18-956d-a2e4c7052004: 4 suspended >> running in phase >>> Run Actions::Pulp::Consumer::GenerateApplicability >>> 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step >>> dc8976be-99e7-4dc4-8a3f-893d3687a1cc: 4 running >> suspended in phase >>> Run Actions::Pulp::Consumer::GenerateApplicability >>> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " >>> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ >>> 41b06756-a517-415b-bcb9-9002dc0e5fec/", "Accept"=>"*/*; q=0.5, >>> application/xml", "Accept-Encoding"=>"gzip, deflate", >>> "accept"=>"application/json", "content_type"=>"application/json" >>> | \nRestClient.get "https://c4katello02.c4.< >>> redacted>/pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/", >>> "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, >>> deflate", "accept"=>"application/json", "content_type"=>"application/ >>> json" >>> | \n# => 200 OK | application/json 326 bytes >>> | >>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>> c0e798bf-a9f9-40c7-9d55-75d686454f42: 4 running >> suspended in phase >>> Run Actions::Pulp::Consumer::GenerateApplicability >>> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " >>> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ >>> 41b06756-a517-415b-bcb9-9002dc0e5fec/", "Accept"=>"*/*; q=0.5, >>> application/xml", "Accept-Encoding"=>"gzip, deflate", >>> "accept"=>"application/json", "content_type"=>"application/json" >>> | \nRestClient.get "https://c4katello02.c4.< >>> redacted>/pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/", >>> "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, >>> deflate", "accept"=>"application/json", "content_type"=>"application/ >>> json" >>> | \n# => 200 OK | application/json 326 bytes >>> | \n# => 200 OK | application/json 328 bytes >>> | >>> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " >>> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ >>> 41b06756-a517-415b-bcb9-9002dc0e5fec/", "Accept"=>"*/*; q=0.5, >>> application/xml", "Accept-Encoding"=>"gzip, deflate", >>> "accept"=>"application/json", "content_type"=>"application/json" >>> | \nRestClient.get "https://c4katello02.c4.< >>> redacted>/pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/", >>> "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, >>> deflate", "accept"=>"application/json", "content_type"=>"application/ >>> json" >>> | \n# => 200 OK | application/json 326 bytes >>> | \n# => 200 OK | application/json 328 bytes >>> | \n# => 200 OK | application/json 328 bytes >>> | >>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>> f96f15cb-5933-4b18-956d-a2e4c7052004: 4 running >> suspended in phase >>> Run Actions::Pulp::Consumer::GenerateApplicability >>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>> a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 running >> suspended in phase >>> Run Actions::Pulp::Consumer::GenerateApplicability >>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>> 1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 got event >>> Dynflow::Action::Polling::Poll >>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>> 1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 suspended >> running in phase >>> Run Actions::Pulp::Consumer::GenerateApplicability >>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>> 2392d32f-0886-4473-ae5a-7595504acb39: 4 got event >>> Dynflow::Action::Polling::Poll >>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>> 2392d32f-0886-4473-ae5a-7595504acb39: 4 suspended >> running in phase >>> Run Actions::Pulp::Consumer::GenerateApplicability >>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>> a03805a6-1725-4a92-b3f5-05514cc52c81: 4 got event >>> Dynflow::Action::Polling::Poll >>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>> a03805a6-1725-4a92-b3f5-05514cc52c81: 4 suspended >> running in phase >>> Run Actions::Pulp::Consumer::GenerateApplicability >>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>> 96b10629-bfed-4cc5-a9b8-b80e1fc5b302: 4 got event >>> Dynflow::Action::Polling::Poll >>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>> 96b10629-bfed-4cc5-a9b8-b80e1fc5b302: 4 suspended >> running in phase >>> Run Actions::Pulp::Consumer::GenerateApplicability >>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>> 903f497f-a539-45af-a220-4cf3d7dd70e2: 4 got event >>> Dynflow::Action::Polling::Poll >>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>> 903f497f-a539-45af-a220-4cf3d7dd70e2: 4 suspended >> running in phase >>> Run Actions::Pulp::Consumer::GenerateApplicability >>> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " >>> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ >>> f2f90682-f0c9-404d-98d7-8227fceda2b7/", "Accept"=>"*/*; q=0.5, >>> application/xml", "Accept-Encoding"=>"gzip, deflate", >>> "accept"=>"application/json", "content_type"=>"application/json" >>> | \n# => 200 OK | application/json 329 bytes >>> | >>> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " >>> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ >>> f2f90682-f0c9-404d-98d7-8227fceda2b7/", "Accept"=>"*/*; q=0.5, >>> application/xml", "Accept-Encoding"=>"gzip, deflate", >>> "accept"=>"application/json", "content_type"=>"application/json" >>> | \n# => 200 OK | application/json 329 bytes >>> | \n# => 200 OK | application/json 329 bytes >>> | >>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>> 1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 running >> suspended in phase >>> Run Actions::Pulp::Consumer::GenerateApplicability >>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>> 2392d32f-0886-4473-ae5a-7595504acb39: 4 running >> suspended in phase >>> Run Actions::Pulp::Consumer::GenerateApplicability >>> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " >>> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ >>> f2f90682-f0c9-404d-98d7-8227fceda2b7/", "Accept"=>"*/*; q=0.5, >>> application/xml", "Accept-Encoding"=>"gzip, deflate", >>> "accept"=>"application/json", "content_type"=>"application/json" >>> | \n# => 200 OK | application/json 329 bytes >>> | \n# => 200 OK | application/json 329 bytes >>> | \n# => 200 OK | application/json 330 bytes >>> | >>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>> a03805a6-1725-4a92-b3f5-05514cc52c81: 4 running >> suspended in phase >>> Run Actions::Pulp::Consumer::GenerateApplicability >>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>> 3c827032-6d3b-4a5b-975c-cf78ec6268bd: 4 got event >>> Dynflow::Action::Polling::Poll >>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>> 3c827032-6d3b-4a5b-975c-cf78ec6268bd: 4 suspended >> running in phase >>> Run Actions::Pulp::Consumer::GenerateApplicability >>> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " >>> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ >>> f2f90682-f0c9-404d-98d7-8227fceda2b7/", "Accept"=>"*/*; q=0.5, >>> application/xml", "Accept-Encoding"=>"gzip, deflate", >>> "accept"=>"application/json", "content_type"=>"application/json" >>> | \n# => 200 OK | application/json 329 bytes >>> | \n# => 200 OK | application/json 329 bytes >>> | \n# => 200 OK | application/json 330 bytes >>> | \n# => 200 OK | application/json 327 bytes >>> | >>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >>> 903f497f-a539-45af-a220-4cf3d7dd70e2: 4 running >> suspended in phase >>> Run Actions::Pulp::Consumer::GenerateApplicability >>> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " >>> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ >>> f2f90682-f0c9-404d-98d7-8227fceda2b7/", "Accept"=>"*/*; q=0.5, >>> application/xml", "Accept-Encoding"=>"gzip, deflate", >>> "accept"=>"application/json", "content_type"=>"application/json" >>> | \n# => 200 OK | application/json 329 bytes >>> | \n# => 200 OK | application/json 329 bytes >>> >>> >>> >>> <https://lh3.googleusercontent.com/-sEewHxc4UuA/WUA9z6RVaWI/AAAAAAAABJM/mWeOvhRvpgknjIOZs8Unwt1l2791tugbACLcBGAs/s1600/Selection_133.png> >>> >>> >>> >>> I'm happy to upload any logs anyone might want to look at to help me >>> with this.. but I'm at my wits end with this. I'm about ready to tear it >>> all down and just rebuild it from scratch... >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Foreman users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at https://groups.google.com/group/foreman-users. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Foreman users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/foreman-users. >> For more options, visit https://groups.google.com/d/optout. >> > -- > You received this message because you are subscribed to the Google Groups > "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Foreman users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/foreman-users. For more options, visit https://groups.google.com/d/optout.
