Hello, it seems that when the proxy is making a callback to Foreman, it doesn't reach Foreman at all. Could you please set set log_level to DEBUG in /etc/smart_proxy_dynflow_core/settings.yml, watch the log in /var/log/foreman-proxy/smart_proxy_dynflow_core.log and try to rerun the job? Some useful information might show up there.
Adam On Mon, Nov 21, 2016 at 5:25 PM, susanne.bachmann5 via Foreman users < [email protected]> wrote: > Hi Adam, > > i checked to logs, but i didn't find the entry. > Here are an outlook from the production.log: > > 2016-11-21 16:28:28 e00d853a [app] [I] Started GET "/job_invocations/new" > for .... at 2016-11-21 16:28:28 +0100 > 2016-11-21 16:28:28 e00d853a [app] [I] Processing by > JobInvocationsController#new as HTML > 2016-11-21 16:28:29 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/template_inputs/_invocation_form.html.erb > (1.6ms) > 2016-11-21 16:28:29 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/job_invocations/_description_fields.html.erb > (2.0ms) > 2016-11-21 16:28:29 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks- > 0.8.2/app/views/common/_trigger_form.html.erb (12.8ms) > 2016-11-21 16:28:29 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/job_invocations/_preview_hosts_modal.html.erb > (0.9ms) > 2016-11-21 16:28:29 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/job_invocations/_form.html.erb (46.7ms) > 2016-11-21 16:28:29 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/job_invocations/new.html.erb within > layouts/application (48.5ms) > 2016-11-21 16:28:29 e00d853a [app] [I] Rendered > layouts/_application_content.html.erb (0.3ms) > 2016-11-21 16:28:29 e00d853a [app] [I] Rendered > home/_user_dropdown.html.erb (2.1ms) > 2016-11-21 16:28:29 e00d853a [app] [I] Read fragment > views/tabs_and_title_records-4 (0.3ms) > 2016-11-21 16:28:29 e00d853a [app] [I] Rendered home/_topbar.html.erb > (6.2ms) > 2016-11-21 16:28:29 e00d853a [app] [I] Rendered layouts/base.html.erb > (7.8ms) > 2016-11-21 16:28:29 e00d853a [app] [I] Completed 200 OK in 71ms (Views: > 56.4ms | ActiveRecord: 3.8ms) > 2016-11-21 16:28:29 e00d853a [app] [I] Started GET > "/assets/template_invocation-bf20f4154d0c5fb85a297aa4b930a7 > 3bc8bbfde6d77962de84d883686ac2f323.css" for .. > 2016-11-21 16:28:29 e00d853a [app] [I] Started GET "/assets/foreman_tasks/ > trigger_form-d7cb707d980496ae04aadc63bfe6812f9e596f6cef83253df91ee4f4a11c57c7.css" > for .. > 2016-11-21 16:28:29 e00d853a [app] [I] Started GET "/assets/modal_window- > 7117ccc83cf81f90d031ea2faeba27ddaefe176603551d25e300b59c2fdd947b.css" for > .. > 2016-11-21 16:28:29 e00d853a [app] [I] Started GET > "/assets/template_invocation-9117a6b7dcc7ac0b02f590094eea76 > 7faa2568d6a58e497eb9543477b7eece76.js" for .. > 2016-11-21 16:28:29 e00d853a [app] [I] Started GET "/assets/trigger_form- > 9434ffad72fa0fc76bfd09f66a0fc4710a24b243aa57d3cd954ada6852552162.js" for > .. > 2016-11-21 16:28:33 e00d853a [app] [I] Started POST > "/job_invocations/refresh" for ... > 2016-11-21 16:28:33 e00d853a [app] [I] Processing by > JobInvocationsController#refresh as JS > 2016-11-21 16:28:33 e00d853a [app] [I] Parameters: {"utf8"=>"✓", > "authenticity_token"=>"<long string>", > "job_invocation"=>{"job_category"=>"Commands", > "providers"=>{"SSH"=>{"job_template_id"=>"144", > "job_templates"=>{"144"=>{"input_values"=>"[FILTERED]", > "effective_user"=>""}}}}, "description"=>"Run ", > "description_override"=>"Run %{command}", "description_format"=>"Run > %{command}", "concurrency_level"=>"", "time_span"=>""}, > "targeting"=>{"bookmark_id"=>"", "search_query"=>"<our Server search > string>", "targeting_type"=>"static_query"}, > "triggering"=>{"mode"=>"immediate", > "start_at_raw"=>"2016-11-21 15:28", "start_before_raw"=>"", > "input_type"=>"daily", "cronline"=>"", "days"=>"", > "days_of_week"=>{"1"=>"0", "2"=>"0", "3"=>"0", "4"=>"0", "5"=>"0", > "6"=>"0", "7"=>"0"}, "time"=>{"time(1i)"=>"2016", "time(2i)"=>"11", > "time(3i)"=>"21", "time(4i)"=>"15", "time(5i)"=>"28"}, "max_iteration"=>"", > "end_time_limited"=>"false", "end_time"=>{"end_time(1i)"=>"2016", > "end_time(2i)"=>"11", "end_time(3i)"=>"21", "end_time(4i)"=>"15", > "end_time(5i)"=>"28"}}} > 2016-11-21 16:28:33 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/template_inputs/_invocation_form.html.erb > (1.5ms) > 2016-11-21 16:28:33 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/job_invocations/_description_fields.html.erb > (1.6ms) > 2016-11-21 16:28:33 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks- > 0.8.2/app/views/common/_trigger_form.html.erb (12.8ms) > 2016-11-21 16:28:33 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/job_invocations/_preview_hosts_modal.html.erb > (0.5ms) > 2016-11-21 16:28:33 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/job_invocations/_form.html.erb (96.9ms) > 2016-11-21 16:28:33 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/job_invocations/refresh.js.erb (100.3ms) > 2016-11-21 16:28:33 e00d853a [app] [I] Completed 200 OK in 124ms (Views: > 56.2ms | ActiveRecord: 47.4ms) > 2016-11-21 16:28:44 e00d853a [app] [I] Started POST "/job_invocations" for > ... > 2016-11-21 16:28:44 e00d853a [app] [I] Processing by > JobInvocationsController#create as HTML > 2016-11-21 16:28:44 e00d853a [app] [I] Parameters: {"utf8"=>"✓", > "authenticity_token"=>"<long string>", > "job_invocation"=>{"job_category"=>"Commands", > "providers"=>{"SSH"=>{"job_template_id"=>"144", > "job_templates"=>{"144"=>{"input_values"=>"[FILTERED]", > "effective_user"=>""}}}}, "description"=>"Run echo job_run", > "description_override"=>"Run %{command}", "description_format"=>"Run > %{command}", "concurrency_level"=>"", "time_span"=>""}, > "targeting"=>{"bookmark_id"=>"", "search_query"=>"<our server search > query string>", "targeting_type"=>"static_query"}, > "triggering"=>{"mode"=>"immediate", "start_at_raw"=>"2016-11-21 15:28", > "start_before_raw"=>"", "input_type"=>"daily", "cronline"=>"", "days"=>"", > "days_of_week"=>{"1"=>"0", "2"=>"0", "3"=>"0", "4"=>"0", "5"=>"0", > "6"=>"0", "7"=>"0"}, "time"=>{"time(1i)"=>"2016", "time(2i)"=>"11", > "time(3i)"=>"21", "time(4i)"=>"15", "time(5i)"=>"28"}, "max_iteration"=>"", > "end_time_limited"=>"false", "end_time"=>{"end_time(1i)"=>"2016", > "end_time(2i)"=>"11", "end_time(3i)"=>"21", "end_time(4i)"=>"15", > "end_time(5i)"=>"28"}}, "commit"=>"Submit"} > 2016-11-21 16:28:44 e00d853a [app] [I] Redirected to https://<your > foreman server>/job_invocations/33 > 2016-11-21 16:28:44 e00d853a [app] [I] Completed 302 Found in 190ms > (ActiveRecord: 66.4ms) > 2016-11-21 16:28:44 [foreman-tasks/dynflow] [D] ExecutionPlan > ee51fdbb-5c6e-45d4-92c0-2749c45c0c90 planned >> running > 2016-11-21 16:28:44 e00d853a [app] [I] Started GET "/job_invocations/33" > for ... > 2016-11-21 16:28:44 e00d853a [app] [I] Processing by > JobInvocationsController#show as HTML > 2016-11-21 16:28:44 e00d853a [app] [I] Parameters: {"id"=>"33"} > 2016-11-21 16:28:44 [foreman-tasks/dynflow] [D] Step > ee51fdbb-5c6e-45d4-92c0-2749c45c0c90: 2 pending >> running in > phase Run Actions::RemoteExecution::RunHostsJob > 2016-11-21 16:28:44 [app] [D] Setting current user thread-local variable > to <user that is logged in> > 2016-11-21 16:28:45 [foreman-tasks/dynflow] [D] ExecutionPlan > d3d3c62c-217a-4884-b5ce-8e1349a26ea2 pending >> planning > 2016-11-21 16:28:45 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/job_invocations/_tab_overview.html.erb > (22.9ms) > 2016-11-21 16:28:45 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/job_invocations/_host_name_td.html.erb > (0.7ms) > 2016-11-21 16:28:45 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/job_invocations/_host_status_td.html.erb > (0.5ms) > 2016-11-21 16:28:45 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/job_invocations/_host_actions_td.html.erb > (2.2ms) > 2016-11-21 16:28:45 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/job_invocations/_tab_hosts.html.erb > (7.3ms) > 2016-11-21 16:28:45 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/job_invocations/show.html.erb within > layouts/application (40.1ms) > 2016-11-21 16:28:45 e00d853a [app] [I] Rendered > layouts/_application_content.html.erb (0.4ms) > 2016-11-21 16:28:45 [foreman-tasks/dynflow] [D] Step > d3d3c62c-217a-4884-b5ce-8e1349a26ea2: 1 pending >> running in > phase Plan Actions::RemoteExecution::RunHostJob > 2016-11-21 16:28:45 e00d853a [app] [I] Rendered > home/_user_dropdown.html.erb (2.1ms) > 2016-11-21 16:28:45 e00d853a [app] [I] Read fragment > views/tabs_and_title_records-4 (0.3ms) > 2016-11-21 16:28:45 e00d853a [app] [I] Rendered home/_topbar.html.erb > (7.9ms) > 2016-11-21 16:28:45 e00d853a [app] [I] Rendered layouts/base.html.erb > (9.5ms) > 2016-11-21 16:28:45 e00d853a [app] [I] Completed 200 OK in 63ms (Views: > 47.2ms | ActiveRecord: 5.3ms) > 2016-11-21 16:28:45 e00d853a [app] [I] Started GET > "/assets/job_invocations-f079e92744b55af1d247e064a15a97 > 11f56c06165fea3cd2a7724ef2bf5b011a.css" for ... > 2016-11-21 16:28:45 e00d853a [app] [I] Started GET > "/assets/template_invocation-9117a6b7dcc7ac0b02f590094eea76 > 7faa2568d6a58e497eb9543477b7eece76.js" for ... > 2016-11-21 16:28:45 [app] [D] Cache read: remote_execution_fallback_proxy > 2016-11-21 16:28:45 [app] [D] Cache read: remote_execution_global_proxy > 2016-11-21 16:28:45 [app] [D] Cache read: proxy_request_timeout > 2016-11-21 16:28:45 [app] [D] Cache read: ssl_certificate > 2016-11-21 16:28:45 [app] [D] Cache read: ssl_ca_file > 2016-11-21 16:28:45 [app] [D] Cache read: ssl_priv_key > 2016-11-21 16:28:45 [app] [D] Cache read: safemode_render > 2016-11-21 16:28:45 [app] [D] Cache read: Enable_Smart_Variables_in_ENC > 2016-11-21 16:28:45 [app] [D] Cache read: remote_execution_fallback_proxy > 2016-11-21 16:28:45 [app] [D] Cache read: remote_execution_global_proxy > 2016-11-21 16:28:45 [app] [D] Cache read: remote_execution_ssh_user > 2016-11-21 16:28:45 [app] [D] Cache read: remote_execution_effective_ > user_method > 2016-11-21 16:28:45 [app] [D] Cache read: Enable_Smart_Variables_in_ENC > 2016-11-21 16:28:45 [app] [D] Cache read: remote_execution_fallback_proxy > 2016-11-21 16:28:45 [app] [D] Cache read: remote_execution_global_proxy > 2016-11-21 16:28:45 [foreman-tasks/dynflow] [D] Step > d3d3c62c-217a-4884-b5ce-8e1349a26ea2: 2 pending >> running in > phase Plan Actions::ProxyAction > 2016-11-21 16:28:45 [app] [D] Cache read: foreman_tasks_proxy_action_ > retry_interval > 2016-11-21 16:28:45 [app] [D] Cache read: foreman_tasks_proxy_action_ > retry_count > 2016-11-21 16:28:45 [app] [D] Cache read: foreman_tasks_proxy_action_ > start_timeout > 2016-11-21 16:28:45 [foreman-tasks/dynflow] [D] Step > d3d3c62c-217a-4884-b5ce-8e1349a26ea2: 2 running >> success in > phase Plan Actions::ProxyAction > 2016-11-21 16:28:45 [foreman-tasks/dynflow] [D] Step > d3d3c62c-217a-4884-b5ce-8e1349a26ea2: 1 running >> success in > phase Plan Actions::RemoteExecution::RunHostJob > 2016-11-21 16:28:45 [foreman-tasks/dynflow] [D] ExecutionPlan > d3d3c62c-217a-4884-b5ce-8e1349a26ea2 planning >> planned > 2016-11-21 16:28:45 [app] [D] Setting current user thread-local variable > to nil > 2016-11-21 16:28:45 [foreman-tasks/dynflow] [D] Step > ee51fdbb-5c6e-45d4-92c0-2749c45c0c90: 2 running >> suspended in > phase Run Actions::RemoteExecution::RunHostsJob > 2016-11-21 16:28:45 [foreman-tasks/dynflow] [D] ExecutionPlan > d3d3c62c-217a-4884-b5ce-8e1349a26ea2 planned >> running > 2016-11-21 16:28:45 [foreman-tasks/dynflow] [D] Step > d3d3c62c-217a-4884-b5ce-8e1349a26ea2: 3 pending >> running in > phase Run Actions::ProxyAction > 2016-11-21 16:28:45 [app] [D] Cache read: proxy_request_timeout > 2016-11-21 16:28:45 [app] [D] Cache read: ssl_certificate > 2016-11-21 16:28:45 [app] [D] Cache read: ssl_ca_file > 2016-11-21 16:28:45 [app] [D] Cache read: ssl_priv_key > 2016-11-21 16:28:45 [foreman-tasks/dynflow] [D] Step > d3d3c62c-217a-4884-b5ce-8e1349a26ea2: 3 running >> suspended in > phase Run Actions::ProxyAction > 2016-11-21 16:28:46 e00d853a [app] [I] Started GET > "/job_invocations/33?hosts_needs_refresh=&host_ids_ > needing_name_update%5B%5D=336&host_ids_needing_status_update%5B%5D=336&_=1479742125314" > ... > 2016-11-21 16:28:46 e00d853a [app] [I] Processing by > JobInvocationsController#show as JS > 2016-11-21 16:28:46 e00d853a [app] [I] Parameters: > {"hosts_needs_refresh"=>"", "host_ids_needing_name_update"=>["336"], > "host_ids_needing_status_update"=>["336"], "_"=>"1479742125314", > "id"=>"33"} > 2016-11-21 16:28:46 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/job_invocations/_host_name_td.html.erb > (0.8ms) > 2016-11-21 16:28:46 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/job_invocations/_host_status_td.html.erb > (0.6ms) > 2016-11-21 16:28:46 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/job_invocations/show.js.erb (19.1ms) > 2016-11-21 16:28:46 e00d853a [app] [I] Completed 200 OK in 34ms (Views: > 19.0ms | ActiveRecord: 2.6ms) > ..... > last 5 lines are repeated every second > .... > > 2016-11-21 16:29:45 [foreman-tasks/dynflow] [D] Step > d3d3c62c-217a-4884-b5ce-8e1349a26ea2: 3 got event > Dynflow::Action::Timeouts::Timeout > 2016-11-21 16:29:45 [foreman-tasks/dynflow] [D] Step > d3d3c62c-217a-4884-b5ce-8e1349a26ea2: 3 suspended >> running in > phase Run Actions::ProxyAction > 2016-11-21 16:29:45 [app] [D] Cache read: proxy_request_timeout > 2016-11-21 16:29:45 [app] [D] Cache read: ssl_certificate > 2016-11-21 16:29:45 [app] [D] Cache read: ssl_ca_file > 2016-11-21 16:29:45 [app] [D] Cache read: ssl_priv_key > 2016-11-21 16:29:45 [foreman-tasks/action] [E] ERF42-6299 > [Foreman::Exception]: The smart proxy task > a017ac8b-07c7-4772-9d16-dee3d69b375f > failed. (Foreman::Exception) > > | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks- > 0.8.2/app/lib/actions/proxy_action.rb:59:in `check_task_status' > | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks- > 0.8.2/app/lib/actions/proxy_action.rb:37:in `block in run' > | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks- > 0.8.2/app/lib/actions/proxy_action.rb:159:in `with_connection_error_ > handling' > | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks- > 0.8.2/app/lib/actions/proxy_action.rb:21:in `run' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/action.rb:506:in `block (3 levels) in execute_run' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/middleware/stack.rb:26:in `call' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/middleware/stack.rb:26:in `pass' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/middleware.rb:17:in `pass' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/action/progress.rb:30:in `with_progress_calculation' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/action/progress.rb:16:in `run' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/middleware/stack.rb:22:in `call' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/middleware/stack.rb:26:in `pass' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/middleware.rb:17:in `pass' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/middleware.rb:30:in `run' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/middleware/stack.rb:22:in `call' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/middleware/world.rb:30:in `execute' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/action.rb:505:in `block (2 levels) in execute_run' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/action.rb:504:in `catch' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/action.rb:504:in `block in execute_run' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/action.rb:419:in `call' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/action.rb:419:in `block in with_error_handling' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/action.rb:419:in `catch' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/action.rb:419:in `with_error_handling' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/action.rb:499:in `execute_run' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/action.rb:260:in `execute' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 > levels) in execute' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/execution_plan/steps/abstract.rb:155:in > `with_meta_calculation' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in > execute' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in > `open_action' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/executors/parallel/worker.rb:15:in `block in on_message' > | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7. > 3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns' > | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7. > 3/lib/algebrick/matchers/abstract.rb:73:in `tap' > | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7. > 3/lib/algebrick/matchers/abstract.rb:73:in `assigns' > | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7. > 3/lib/algebrick/matching.rb:56:in `match_value' > | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7. > 3/lib/algebrick/matching.rb:36:in `block in match?' > | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7. > 3/lib/algebrick/matching.rb:35:in `each' > | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7. > 3/lib/algebrick/matching.rb:35:in `match?' > | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7. > 3/lib/algebrick/matching.rb:23:in `match' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/executors/parallel/worker.rb:12:in `on_message' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in `on_envelope' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:in > `on_envelope' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/actor.rb:26:in > `on_envelope' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/awaits.rb:15:in > `on_envelope' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in > `on_envelope' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:38:in > `process_envelope' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:31:in > `process_envelopes?' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:20:in > `on_envelope' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/termination.rb:55:in > `on_envelope' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/removes_child.rb:10:in > `on_envelope' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in > `on_envelope' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/core.rb:161:in `process_envelope' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/core.rb:95:in `block in on_envelope' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/core.rb:118:in `block (2 levels) in > schedule_execution' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in > `block in synchronize' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in > `synchronize' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in > `synchronize' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/core.rb:115:in `block in > schedule_execution' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in `call' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in `call' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:96:in `work' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:77:in `block > in call_job' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in > `call' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in > `run_task' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:in > `block (3 levels) in create_worker' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in > `loop' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in > `block (2 levels) in create_worker' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in > `catch' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in > `block in create_worker' > | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/ > lib/logging/diagnostic_context.rb:323:in `call' > | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/ > lib/logging/diagnostic_context.rb:323:in `block in > create_with_logging_context' > 2016-11-21 16:29:45 [foreman-tasks/dynflow] [D] Step > d3d3c62c-217a-4884-b5ce-8e1349a26ea2: 3 running >> error in > phase Run Actions::ProxyAction > 2016-11-21 16:29:45 [foreman-tasks/dynflow] [D] ExecutionPlan > d3d3c62c-217a-4884-b5ce-8e1349a26ea2 running >> paused > 2016-11-21 16:29:45 [foreman-tasks/dynflow] [D] ExecutionPlan > d3d3c62c-217a-4884-b5ce-8e1349a26ea2 paused >> stopped > 2016-11-21 16:29:45 [foreman-tasks/dynflow] [D] Step > ee51fdbb-5c6e-45d4-92c0-2749c45c0c90: 2 got event > Dynflow::Action::WithSubPlans::SubPlanFinished[execution_plan_id: > d3d3c62c-217a-4884-b5ce-8e1349a26ea2, success: false] > 2016-11-21 16:29:45 [foreman-tasks/dynflow] [D] Step > ee51fdbb-5c6e-45d4-92c0-2749c45c0c90: 2 suspended >> running in > phase Run Actions::RemoteExecution::RunHostsJob > 2016-11-21 16:29:45 [app] [D] Setting current user thread-local variable > to <logged in user> > 2016-11-21 16:29:45 [app] [D] Setting current user thread-local variable > to nil > 2016-11-21 16:29:45 [foreman-tasks/action] [E] A sub task failed > (RuntimeError) > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/action/with_sub_plans.rb:206:in `check_for_errors!' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/action/with_sub_plans.rb:125:in `try_to_finish' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/action/with_sub_plans.rb:21:in `block in run' > | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7. > 3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns' > | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7. > 3/lib/algebrick/matchers/abstract.rb:73:in `tap' > | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7. > 3/lib/algebrick/matchers/abstract.rb:73:in `assigns' > | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7. > 3/lib/algebrick/matching.rb:56:in `match_value' > | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7. > 3/lib/algebrick/matching.rb:36:in `block in match?' > | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7. > 3/lib/algebrick/matching.rb:35:in `each' > | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7. > 3/lib/algebrick/matching.rb:35:in `match?' > | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7. > 3/lib/algebrick/matching.rb:23:in `match' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/action/with_sub_plans.rb:11:in `run' > | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/lib/actions/remote_execution/run_hosts_job.rb:47:in > `run' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/action.rb:506:in `block (3 levels) in execute_run' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/middleware/stack.rb:26:in `call' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/middleware/stack.rb:26:in `pass' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/middleware.rb:17:in `pass' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/middleware.rb:30:in `run' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/middleware/stack.rb:22:in `call' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/middleware/stack.rb:26:in `pass' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/middleware.rb:17:in `pass' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/middleware.rb:30:in `run' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/middleware/stack.rb:22:in `call' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/middleware/stack.rb:26:in `pass' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/middleware.rb:17:in `pass' > | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks- > 0.8.2/app/lib/actions/middleware/keep_current_user.rb:30:in `block in run' > | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks- > 0.8.2/app/lib/actions/middleware/keep_current_user.rb:53:in > `restore_curent_user' > | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks- > 0.8.2/app/lib/actions/middleware/keep_current_user.rb:30:in `run' > > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/middleware/stack.rb:22:in `call' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/middleware/stack.rb:26:in `pass' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/middleware.rb:17:in `pass' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/action/progress.rb:30:in `with_progress_calculation' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/action/progress.rb:16:in `run' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/middleware/stack.rb:22:in `call' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/middleware/stack.rb:26:in `pass' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/middleware.rb:17:in `pass' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/middleware.rb:30:in `run' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/middleware/stack.rb:22:in `call' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/middleware/world.rb:30:in `execute' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/action.rb:505:in `block (2 levels) in execute_run' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/action.rb:504:in `catch' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/action.rb:504:in `block in execute_run' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/action.rb:419:in `call' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/action.rb:419:in `block in with_error_handling' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/action.rb:419:in `catch' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/action.rb:419:in `with_error_handling' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/action.rb:499:in `execute_run' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/action.rb:260:in `execute' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 > levels) in execute' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/execution_plan/steps/abstract.rb:155:in > `with_meta_calculation' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in > execute' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in > `open_action' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/executors/parallel/worker.rb:15:in `block in on_message' > | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7. > 3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns' > | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7. > 3/lib/algebrick/matchers/abstract.rb:73:in `tap' > | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7. > 3/lib/algebrick/matchers/abstract.rb:73:in `assigns' > | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7. > 3/lib/algebrick/matching.rb:56:in `match_value' > | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7. > 3/lib/algebrick/matching.rb:36:in `block in match?' > | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7. > 3/lib/algebrick/matching.rb:35:in `each' > | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7. > 3/lib/algebrick/matching.rb:35:in `match?' > | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7. > 3/lib/algebrick/matching.rb:23:in `match' > | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8. > 15/lib/dynflow/executors/parallel/worker.rb:12:in `on_message' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in `on_envelope' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:in > `on_envelope' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/actor.rb:26:in > `on_envelope' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/awaits.rb:15:in > `on_envelope' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in > `on_envelope' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:38:in > `process_envelope' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:31:in > `process_envelopes?' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:20:in > `on_envelope' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/termination.rb:55:in > `on_envelope' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/removes_child.rb:10:in > `on_envelope' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in > `on_envelope' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/core.rb:161:in `process_envelope' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/core.rb:95:in `block in on_envelope' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/core.rb:118:in `block (2 levels) in > schedule_execution' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in > `block in synchronize' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in > `synchronize' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in > `synchronize' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-edge-0.2.0/lib/concurrent/actor/core.rb:115:in `block in > schedule_execution' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in `call' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in `call' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:96:in `work' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:77:in `block > in call_job' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in > `call' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in > `run_task' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:in > `block (3 levels) in create_worker' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in > `loop' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in > `block (2 levels) in create_worker' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in > `catch' > | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent- > ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in > `block in create_worker' > | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/ > lib/logging/diagnostic_context.rb:323:in `call' > | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/ > lib/logging/diagnostic_context.rb:323:in `block in > create_with_logging_context' > 2016-11-21 16:29:45 [foreman-tasks/dynflow] [D] Step > ee51fdbb-5c6e-45d4-92c0-2749c45c0c90: 2 running >> error in > phase Run Actions::RemoteExecution::RunHostsJob > 2016-11-21 16:29:45 [foreman-tasks/dynflow] [D] ExecutionPlan > ee51fdbb-5c6e-45d4-92c0-2749c45c0c90 running >> paused > 2016-11-21 16:29:45 [foreman-tasks/dynflow] [D] ExecutionPlan > ee51fdbb-5c6e-45d4-92c0-2749c45c0c90 paused >> running > 2016-11-21 16:29:45 [app] [D] Setting current user thread-local variable > to <logged in user> > 2016-11-21 16:29:45 [app] [D] Setting current user thread-local variable > to nil > 2016-11-21 16:29:45 [foreman-tasks/dynflow] [D] Step > ee51fdbb-5c6e-45d4-92c0-2749c45c0c90: 2 skipping >> skipped in > phase Run Actions::RemoteExecution::RunHostsJob > 2016-11-21 16:29:45 [foreman-tasks/dynflow] [D] ExecutionPlan > ee51fdbb-5c6e-45d4-92c0-2749c45c0c90 running >> stopped > 2016-11-21 16:29:45 e00d853a [app] [I] Started GET > "/template_invocations/66?since=1479742126.712004&line_counter=1&_=1479742153685" > for ... > 2016-11-21 16:29:45 e00d853a [app] [I] Processing by > TemplateInvocationsController#show as JS > 2016-11-21 16:29:45 e00d853a [app] [I] Parameters: > {"since"=>"1479742126.712004", "line_counter"=>"1", "_"=>"1479742153685", > "id"=>"66"} > 2016-11-21 16:29:45 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/template_invocations/_output_line_set.html.erb > (0.0ms) > 2016-11-21 16:29:45 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/template_invocations/_refresh.js.erb > (0.2ms) > 2016-11-21 16:29:45 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/template_invocations/show.js.erb (1.0ms) > 2016-11-21 16:29:45 e00d853a [app] [I] Completed 200 OK in 142ms (Views: > 1.8ms | ActiveRecord: 1.4ms) > 2016-11-21 16:29:57 e00d853a [app] [I] Started GET "/job_invocations/33" > for ... > 2016-11-21 16:29:57 e00d853a [app] [I] Processing by > JobInvocationsController#show as HTML > 2016-11-21 16:29:57 e00d853a [app] [I] Parameters: {"id"=>"33"} > 2016-11-21 16:29:57 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/job_invocations/_tab_overview.html.erb > (25.3ms) > 2016-11-21 16:29:57 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/job_invocations/_host_name_td.html.erb > (1.0ms) > 2016-11-21 16:29:57 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/job_invocations/_host_status_td.html.erb > (4.5ms) > 2016-11-21 16:29:57 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/job_invocations/_host_actions_td.html.erb > (2.3ms) > 2016-11-21 16:29:57 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/job_invocations/_tab_hosts.html.erb > (11.9ms) > 2016-11-21 16:29:57 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/job_invocations/show.html.erb within > layouts/application (49.0ms) > 2016-11-21 16:29:57 e00d853a [app] [I] Rendered > layouts/_application_content.html.erb (0.4ms) > 2016-11-21 16:29:57 e00d853a [app] [I] Rendered > home/_user_dropdown.html.erb (2.1ms) > 2016-11-21 16:29:57 e00d853a [app] [I] Read fragment > views/tabs_and_title_records-4 (0.3ms) > 2016-11-21 16:29:57 e00d853a [app] [I] Rendered home/_topbar.html.erb > (7.9ms) > 2016-11-21 16:29:57 e00d853a [app] [I] Rendered layouts/base.html.erb > (9.6ms) > 2016-11-21 16:29:57 e00d853a [app] [I] Completed 200 OK in 73ms (Views: > 57.3ms | ActiveRecord: 4.6ms) > 2016-11-21 16:29:57 e00d853a [app] [I] Started GET "/job_invocations/33" > ... > 2016-11-21 16:29:57 e00d853a [app] [I] Processing by > JobInvocationsController#show as HTML > 2016-11-21 16:29:57 e00d853a [app] [I] Parameters: {"id"=>"33"} > 2016-11-21 16:29:57 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/job_invocations/_tab_overview.html.erb > (24.3ms) > 2016-11-21 16:29:57 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/job_invocations/_host_name_td.html.erb > (0.9ms) > 2016-11-21 16:29:57 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/job_invocations/_host_status_td.html.erb > (4.5ms) > 2016-11-21 16:29:57 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/job_invocations/_host_actions_td.html.erb > (2.1ms) > 2016-11-21 16:29:57 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/job_invocations/_tab_hosts.html.erb > (11.3ms) > 2016-11-21 16:29:57 e00d853a [app] [I] Rendered > /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ > remote_execution-1.2.1/app/views/job_invocations/show.html.erb within > layouts/application (45.9ms) > 2016-11-21 16:29:57 e00d853a [app] [I] Rendered > layouts/_application_content.html.erb (0.4ms) > 2016-11-21 16:29:57 e00d853a [app] [I] Rendered > home/_user_dropdown.html.erb (2.1ms) > 2016-11-21 16:29:57 e00d853a [app] [I] Read fragment > views/tabs_and_title_records-4 (0.3ms) > 2016-11-21 16:29:57 e00d853a [app] [I] Rendered home/_topbar.html.erb > (6.8ms) > 2016-11-21 16:29:57 e00d853a [app] [I] Rendered layouts/base.html.erb > (8.5ms) > 2016-11-21 16:29:57 e00d853a [app] [I] Completed 200 OK in 68ms (Views: > 52.9ms | ActiveRecord: 4.3ms) > 2016-11-21 16:29:57 e00d853a [app] [I] Started GET > "/assets/job_invocations-f079e92744b55af1d247e064a15a97 > 11f56c06165fea3cd2a7724ef2bf5b011a.css" for .... > 2016-11-21 16:29:57 e00d853a [app] [I] Started GET > "/assets/template_invocation-9117a6b7dcc7ac0b02f590094eea76 > 7faa2568d6a58e497eb9543477b7eece76.js" for ... > > > proxy.log seems to be fine: > D, [2016-11-21T10:25:44.977974 #2479] DEBUG -- : accept: <foreman Server > IP>:34338 > D, [2016-11-21T10:25:44.979924 #2479] DEBUG -- : Rack::Handler::WEBrick is > invoked. > D, [2016-11-21T10:25:44.980663 #2479] DEBUG -- : verifying remote client > <foreman Server IP> against trusted_hosts ["<foreman Hostname>"] > D, [2016-11-21T10:25:44.981101 #2479] DEBUG -- : Proxy request from > <foreman > Hostname>:8443/dynflow/tasks/c1e8e502-bcdc-4030-90a3-b3e7cd092067/status > to https://<foreman Hostname>:8008/tasks/c1e8e502- > bcdc-4030-90a3-b3e7cd092067/status > D, [2016-11-21T10:25:45.028398 #2479] DEBUG -- : Proxy request status 200 > - #<Net::HTTPOK:0x007f2ff02d9610> > I, [2016-11-21T10:25:45.028710 #2479] INFO -- : <foreman Server IP> - - > [21/Nov/2016:10:25:45 +0100] "GET > /dynflow/tasks/c1e8e502-bcdc-4030-90a3-b3e7cd092067/status > HTTP/1.1" 200 3064 0.0482 > > D, [2016-11-21T10:25:45.069262 #2479] DEBUG -- : close: <foreman Server > IP>:34338 > > > > Thx > > Susanne > > On Monday, November 21, 2016 at 11:03:03 AM UTC+1, Adam Ruzicka wrote: >> >> Hello Susanne, >> if you check the production log, I guess you should have a line looking >> similar to this one >> >> [app] [W] No smart proxy server found on ["localhost", >> "localhost.localdomain", "localhost4", "localhost4.localdomain4"] and is >> not in trusted_puppetmaster_hosts >> >> It means that the smart proxy tried to report back to Foreman, but >> Foreman rejected it. If this is the case, you could fix this by logging >> into Foreman, going into Administration > Settings > Authentication and >> adding one of the names listed in the warning to the Trusted puppetmaster >> hosts array. >> >> Hope this helps, let us know how it goes >> >> Adam >> >> On Mon, Nov 21, 2016 at 10:46 AM, susanne.bachmann5 via Foreman users < >> [email protected]> wrote: >> >>> proxy_action >>> >>> -- > You received this message because you are subscribed to the Google Groups > "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Foreman users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/foreman-users. For more options, visit https://groups.google.com/d/optout.
