Certainly. Going to let that run overnight :) On Tue, Jun 13, 2017 at 5:00 PM John Mitsch <[email protected]> wrote:
> That is very strange, this may be a good candidate for a foreman-debug so > we can dig through logs. Can you create one and send it to me privately? > > -John > > John Mitsch > Red Hat Engineering > (860)-967-7285 <(860)%20967-7285> > irc: jomitsch > > On Tue, Jun 13, 2017 at 3:34 PM, <[email protected]> wrote: > >> So mostly we use foreman/katello just for syncing operating systems out >> to capsules/smart proxies, and for managing system lifecycle (basically, >> just kickstarts). I went last week to add a XenServer 7.2 OS into my >> Katello Products, and started having tons of issues, specifically in that I >> had 20k+ 'Actions::Katello::Host::UploadPackageProfile' tasks sitting in >> my list as state = running, result = pending--but not doing anything at all. >> >> To be honest, I don't really care why they're not working--I just want >> them to stop being scheduled at all. Today I've shut off foreman-tasks and >> pulp_workers and tried to set all these tasks which were running/pending to >> stopped/error, and that worked. Seemed to help out with the issues of >> Foreman. However I still had some tasks that weren't being scheduled, so I >> checked and sure enough, foreman tasks was running, as was pulp, but it was >> complaining about not having a dynflow environment. Restarting foreman >> tasks and waiting a few minutes helped, at which point I started seeing >> stuff like this spam my production.log, and tons of >> Actions::Katello::Host::UploadPackageProfile >> tasks being scheduled again--with dates from *months* ago. >> >> 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step >> a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 got event >> Dynflow::Action::Polling::Poll >> 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step >> c0e798bf-a9f9-40c7-9d55-75d686454f42: 4 suspended >> running in phase >> Run Actions::Pulp::Consumer::GenerateApplicability >> 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step >> df8fcf4c-0de7-4cca-8aaf-f38adcd2b6ec: 4 running >> suspended in phase >> Run Actions::Pulp::Consumer::GenerateApplicability >> 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step >> a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 suspended >> running in phase >> Run Actions::Pulp::Consumer::GenerateApplicability >> 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step >> f96f15cb-5933-4b18-956d-a2e4c7052004: 4 suspended >> running in phase >> Run Actions::Pulp::Consumer::GenerateApplicability >> 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step >> dc8976be-99e7-4dc4-8a3f-893d3687a1cc: 4 running >> suspended in phase >> Run Actions::Pulp::Consumer::GenerateApplicability >> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " >> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/41b06756-a517-415b-bcb9-9002dc0e5fec/", >> "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, >> deflate", "accept"=>"application/json", "content_type"=>"application/json" >> | \nRestClient.get >> "https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/", >> "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, >> deflate", "accept"=>"application/json", "content_type"=>"application/json" >> | \n# => 200 OK | application/json 326 bytes >> | >> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >> c0e798bf-a9f9-40c7-9d55-75d686454f42: 4 running >> suspended in phase >> Run Actions::Pulp::Consumer::GenerateApplicability >> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " >> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/41b06756-a517-415b-bcb9-9002dc0e5fec/", >> "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, >> deflate", "accept"=>"application/json", "content_type"=>"application/json" >> | \nRestClient.get >> "https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/", >> "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, >> deflate", "accept"=>"application/json", "content_type"=>"application/json" >> | \n# => 200 OK | application/json 326 bytes >> | \n# => 200 OK | application/json 328 bytes >> | >> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " >> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/41b06756-a517-415b-bcb9-9002dc0e5fec/", >> "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, >> deflate", "accept"=>"application/json", "content_type"=>"application/json" >> | \nRestClient.get >> "https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/", >> "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, >> deflate", "accept"=>"application/json", "content_type"=>"application/json" >> | \n# => 200 OK | application/json 326 bytes >> | \n# => 200 OK | application/json 328 bytes >> | \n# => 200 OK | application/json 328 bytes >> | >> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >> f96f15cb-5933-4b18-956d-a2e4c7052004: 4 running >> suspended in phase >> Run Actions::Pulp::Consumer::GenerateApplicability >> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >> a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 running >> suspended in phase >> Run Actions::Pulp::Consumer::GenerateApplicability >> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >> 1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 got event >> Dynflow::Action::Polling::Poll >> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >> 1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 suspended >> running in phase >> Run Actions::Pulp::Consumer::GenerateApplicability >> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >> 2392d32f-0886-4473-ae5a-7595504acb39: 4 got event >> Dynflow::Action::Polling::Poll >> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >> 2392d32f-0886-4473-ae5a-7595504acb39: 4 suspended >> running in phase >> Run Actions::Pulp::Consumer::GenerateApplicability >> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >> a03805a6-1725-4a92-b3f5-05514cc52c81: 4 got event >> Dynflow::Action::Polling::Poll >> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >> a03805a6-1725-4a92-b3f5-05514cc52c81: 4 suspended >> running in phase >> Run Actions::Pulp::Consumer::GenerateApplicability >> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >> 96b10629-bfed-4cc5-a9b8-b80e1fc5b302: 4 got event >> Dynflow::Action::Polling::Poll >> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >> 96b10629-bfed-4cc5-a9b8-b80e1fc5b302: 4 suspended >> running in phase >> Run Actions::Pulp::Consumer::GenerateApplicability >> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >> 903f497f-a539-45af-a220-4cf3d7dd70e2: 4 got event >> Dynflow::Action::Polling::Poll >> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >> 903f497f-a539-45af-a220-4cf3d7dd70e2: 4 suspended >> running in phase >> Run Actions::Pulp::Consumer::GenerateApplicability >> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " >> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/", >> "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, >> deflate", "accept"=>"application/json", "content_type"=>"application/json" >> | \n# => 200 OK | application/json 329 bytes >> | >> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " >> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/", >> "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, >> deflate", "accept"=>"application/json", "content_type"=>"application/json" >> | \n# => 200 OK | application/json 329 bytes >> | \n# => 200 OK | application/json 329 bytes >> | >> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >> 1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 running >> suspended in phase >> Run Actions::Pulp::Consumer::GenerateApplicability >> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >> 2392d32f-0886-4473-ae5a-7595504acb39: 4 running >> suspended in phase >> Run Actions::Pulp::Consumer::GenerateApplicability >> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " >> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/", >> "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, >> deflate", "accept"=>"application/json", "content_type"=>"application/json" >> | \n# => 200 OK | application/json 329 bytes >> | \n# => 200 OK | application/json 329 bytes >> | \n# => 200 OK | application/json 330 bytes >> | >> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >> a03805a6-1725-4a92-b3f5-05514cc52c81: 4 running >> suspended in phase >> Run Actions::Pulp::Consumer::GenerateApplicability >> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >> 3c827032-6d3b-4a5b-975c-cf78ec6268bd: 4 got event >> Dynflow::Action::Polling::Poll >> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >> 3c827032-6d3b-4a5b-975c-cf78ec6268bd: 4 suspended >> running in phase >> Run Actions::Pulp::Consumer::GenerateApplicability >> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " >> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/", >> "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, >> deflate", "accept"=>"application/json", "content_type"=>"application/json" >> | \n# => 200 OK | application/json 329 bytes >> | \n# => 200 OK | application/json 329 bytes >> | \n# => 200 OK | application/json 330 bytes >> | \n# => 200 OK | application/json 327 bytes >> | >> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step >> 903f497f-a539-45af-a220-4cf3d7dd70e2: 4 running >> suspended in phase >> Run Actions::Pulp::Consumer::GenerateApplicability >> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " >> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/", >> "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, >> deflate", "accept"=>"application/json", "content_type"=>"application/json" >> | \n# => 200 OK | application/json 329 bytes >> | \n# => 200 OK | application/json 329 bytes >> >> >> >> <https://lh3.googleusercontent.com/-sEewHxc4UuA/WUA9z6RVaWI/AAAAAAAABJM/mWeOvhRvpgknjIOZs8Unwt1l2791tugbACLcBGAs/s1600/Selection_133.png> >> >> >> >> I'm happy to upload any logs anyone might want to look at to help me with >> this.. but I'm at my wits end with this. I'm about ready to tear it all >> down and just rebuild it from scratch... >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Foreman users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/foreman-users. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Foreman users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/foreman-users. For more options, visit https://groups.google.com/d/optout.
