That is very strange, this may be a good candidate for a foreman-debug so we can dig through logs. Can you create one and send it to me privately?
-John John Mitsch Red Hat Engineering (860)-967-7285 irc: jomitsch On Tue, Jun 13, 2017 at 3:34 PM, <[email protected]> wrote: > So mostly we use foreman/katello just for syncing operating systems out to > capsules/smart proxies, and for managing system lifecycle (basically, just > kickstarts). I went last week to add a XenServer 7.2 OS into my Katello > Products, and started having tons of issues, specifically in that I had > 20k+ 'Actions::Katello::Host::UploadPackageProfile' tasks sitting in my > list as state = running, result = pending--but not doing anything at all. > > To be honest, I don't really care why they're not working--I just want > them to stop being scheduled at all. Today I've shut off foreman-tasks and > pulp_workers and tried to set all these tasks which were running/pending to > stopped/error, and that worked. Seemed to help out with the issues of > Foreman. However I still had some tasks that weren't being scheduled, so I > checked and sure enough, foreman tasks was running, as was pulp, but it was > complaining about not having a dynflow environment. Restarting foreman > tasks and waiting a few minutes helped, at which point I started seeing > stuff like this spam my production.log, and tons of > Actions::Katello::Host::UploadPackageProfile tasks being scheduled > again--with dates from *months* ago. > > 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step > a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 got event > Dynflow::Action::Polling::Poll > 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step > c0e798bf-a9f9-40c7-9d55-75d686454f42: 4 suspended >> running in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step > df8fcf4c-0de7-4cca-8aaf-f38adcd2b6ec: 4 running >> suspended in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step > a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 suspended >> running in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step > f96f15cb-5933-4b18-956d-a2e4c7052004: 4 suspended >> running in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step > dc8976be-99e7-4dc4-8a3f-893d3687a1cc: 4 running >> suspended in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " > https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ > 41b06756-a517-415b-bcb9-9002dc0e5fec/", "Accept"=>"*/*; q=0.5, > application/xml", "Accept-Encoding"=>"gzip, deflate", > "accept"=>"application/json", "content_type"=>"application/json" > | \nRestClient.get "https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ > ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/", "Accept"=>"*/*; q=0.5, > application/xml", "Accept-Encoding"=>"gzip, deflate", > "accept"=>"application/json", "content_type"=>"application/json" > | \n# => 200 OK | application/json 326 bytes > | > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > c0e798bf-a9f9-40c7-9d55-75d686454f42: 4 running >> suspended in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " > https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ > 41b06756-a517-415b-bcb9-9002dc0e5fec/", "Accept"=>"*/*; q=0.5, > application/xml", "Accept-Encoding"=>"gzip, deflate", > "accept"=>"application/json", "content_type"=>"application/json" > | \nRestClient.get "https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ > ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/", "Accept"=>"*/*; q=0.5, > application/xml", "Accept-Encoding"=>"gzip, deflate", > "accept"=>"application/json", "content_type"=>"application/json" > | \n# => 200 OK | application/json 326 bytes > | \n# => 200 OK | application/json 328 bytes > | > 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " > https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ > 41b06756-a517-415b-bcb9-9002dc0e5fec/", "Accept"=>"*/*; q=0.5, > application/xml", "Accept-Encoding"=>"gzip, deflate", > "accept"=>"application/json", "content_type"=>"application/json" > | \nRestClient.get "https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ > ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/", "Accept"=>"*/*; q=0.5, > application/xml", "Accept-Encoding"=>"gzip, deflate", > "accept"=>"application/json", "content_type"=>"application/json" > | \n# => 200 OK | application/json 326 bytes > | \n# => 200 OK | application/json 328 bytes > | \n# => 200 OK | application/json 328 bytes > | > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > f96f15cb-5933-4b18-956d-a2e4c7052004: 4 running >> suspended in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 running >> suspended in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > 1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 got event > Dynflow::Action::Polling::Poll > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > 1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 suspended >> running in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > 2392d32f-0886-4473-ae5a-7595504acb39: 4 got event > Dynflow::Action::Polling::Poll > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > 2392d32f-0886-4473-ae5a-7595504acb39: 4 suspended >> running in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > a03805a6-1725-4a92-b3f5-05514cc52c81: 4 got event > Dynflow::Action::Polling::Poll > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > a03805a6-1725-4a92-b3f5-05514cc52c81: 4 suspended >> running in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > 96b10629-bfed-4cc5-a9b8-b80e1fc5b302: 4 got event > Dynflow::Action::Polling::Poll > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > 96b10629-bfed-4cc5-a9b8-b80e1fc5b302: 4 suspended >> running in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > 903f497f-a539-45af-a220-4cf3d7dd70e2: 4 got event > Dynflow::Action::Polling::Poll > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > 903f497f-a539-45af-a220-4cf3d7dd70e2: 4 suspended >> running in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " > https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ > f2f90682-f0c9-404d-98d7-8227fceda2b7/", "Accept"=>"*/*; q=0.5, > application/xml", "Accept-Encoding"=>"gzip, deflate", > "accept"=>"application/json", "content_type"=>"application/json" > | \n# => 200 OK | application/json 329 bytes > | > 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " > https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ > f2f90682-f0c9-404d-98d7-8227fceda2b7/", "Accept"=>"*/*; q=0.5, > application/xml", "Accept-Encoding"=>"gzip, deflate", > "accept"=>"application/json", "content_type"=>"application/json" > | \n# => 200 OK | application/json 329 bytes > | \n# => 200 OK | application/json 329 bytes > | > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > 1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 running >> suspended in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > 2392d32f-0886-4473-ae5a-7595504acb39: 4 running >> suspended in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " > https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ > f2f90682-f0c9-404d-98d7-8227fceda2b7/", "Accept"=>"*/*; q=0.5, > application/xml", "Accept-Encoding"=>"gzip, deflate", > "accept"=>"application/json", "content_type"=>"application/json" > | \n# => 200 OK | application/json 329 bytes > | \n# => 200 OK | application/json 329 bytes > | \n# => 200 OK | application/json 330 bytes > | > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > a03805a6-1725-4a92-b3f5-05514cc52c81: 4 running >> suspended in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > 3c827032-6d3b-4a5b-975c-cf78ec6268bd: 4 got event > Dynflow::Action::Polling::Poll > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > 3c827032-6d3b-4a5b-975c-cf78ec6268bd: 4 suspended >> running in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " > https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ > f2f90682-f0c9-404d-98d7-8227fceda2b7/", "Accept"=>"*/*; q=0.5, > application/xml", "Accept-Encoding"=>"gzip, deflate", > "accept"=>"application/json", "content_type"=>"application/json" > | \n# => 200 OK | application/json 329 bytes > | \n# => 200 OK | application/json 329 bytes > | \n# => 200 OK | application/json 330 bytes > | \n# => 200 OK | application/json 327 bytes > | > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > 903f497f-a539-45af-a220-4cf3d7dd70e2: 4 running >> suspended in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " > https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ > f2f90682-f0c9-404d-98d7-8227fceda2b7/", "Accept"=>"*/*; q=0.5, > application/xml", "Accept-Encoding"=>"gzip, deflate", > "accept"=>"application/json", "content_type"=>"application/json" > | \n# => 200 OK | application/json 329 bytes > | \n# => 200 OK | application/json 329 bytes > > > > <https://lh3.googleusercontent.com/-sEewHxc4UuA/WUA9z6RVaWI/AAAAAAAABJM/mWeOvhRvpgknjIOZs8Unwt1l2791tugbACLcBGAs/s1600/Selection_133.png> > > > > I'm happy to upload any logs anyone might want to look at to help me with > this.. but I'm at my wits end with this. I'm about ready to tear it all > down and just rebuild it from scratch... > > -- > You received this message because you are subscribed to the Google Groups > "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Foreman users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/foreman-users. For more options, visit https://groups.google.com/d/optout.
