While I know it's deprecated, I've started looking at pulp-admin:
https://www.mankier.com/1/pulp-admin#Tasks
The task - in the GUI, the id, which I think is the Foreman task id, is
dcbef693-85c3-4418-a189-6ebdb262d99f
This returns
[root@vmpr-res-utils ~]# pulp-admin -u admin tasks details --task-id
dcbef693-85c3-4418-a189-6ebdb262d99f
Enter password:
+----------------------------------------------------------------------+
Task Details
+----------------------------------------------------------------------+
The following resource(s) could not be found:
dcbef693-85c3-4418-a189-6ebdb262d99f (resource_id)
So this is the wrong id. I go looking in the dynaflow and see this in the
"suspended" task
pulp_task_group:
group_id: 9a8074c1-60ad-460d-80ad-9c6148d739ca
[root@vmpr-res-utils ~]# pulp-admin -u admin tasks details --task-id
9a8074c1-60ad-460d-80ad-9c6148d739ca
+----------------------------------------------------------------------+
Task Details
+----------------------------------------------------------------------+
The following resource(s) could not be found:
9a8074c1-60ad-460d-80ad-9c6148d739ca (resource_id)
Hmm. That's not ideal. Maybe I'm wrong - it does say "pulp_task_group" not
"pulp_task":
[root@vmpr-res-utils ~]# pulp-admin -u admin tasks list
+----------------------------------------------------------------------+
Tasks
+----------------------------------------------------------------------+
No tasks found
Hooh boy.
Ok. So I've started seeing hints about a "force unlock" in some of the
pulp/katello docs and bug reports, but can't find anything definitive.
In desperation, I did, as per https://access.redhat.com/solutions/1381053 :
foreman-rake console
ForemanTasks::Task.where(:state => :running).where(:label =>
"Actions::Pulp::Repository::RegenerateApplicability").destroy_all
And the sync status page in the GUI is returning 404.
And now sync is failing on failed subtasks: RuntimeError: A sub task failed.
pulp-admin returns
Traceback (most recent call last):
File "/usr/bin/pulp-admin", line 9, in <module>
load_entry_point('pulp-client-admin==2.9.3', 'console_scripts',
'pulp-admin')()
File "/usr/lib/python2.7/site-packages/pulp/client/admin/__init__.py",
line 9, in main
exit_code = launcher.main(read_config(),
exception_handler_class=AdminExceptionHandler)
File "/usr/lib/python2.7/site-packages/pulp/client/launcher.py", line
115, in main
code = cli.run(args)
File "/usr/lib/python2.7/site-packages/pulp/client/extensions/core.py",
line 477, in run
code = self.context.exception_handler.handle_exception(e)
File
"/usr/lib/python2.7/site-packages/pulp/client/extensions/exceptions.py",
line 98, in handle_exception
exit_code = handle_func(e)
File
"/usr/lib/python2.7/site-packages/pulp_rpm/extensions/admin/rpm_admin_consumer/bind.py",
line 64, in handle_not_found
ExceptionHandler.handle_not_found(self, e)
File
"/usr/lib/python2.7/site-packages/pulp/client/extensions/exceptions.py",
line 163, in handle_not_found
for resource_type, resource_id in e.extra_data['resources'].items():
KeyError: 'resources'
foreman-rake returns
[root@vmpr-res-utils ~]# foreman-rake katello:delete_orphaned_content
--trace
** Invoke katello:delete_orphaned_content (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute katello:delete_orphaned_content
rake aborted!
RestClient::ResourceNotFound: 404 Resource Not Found
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/abstract_response.rb:74:in
`return!'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.9.0/lib/runcible/base.rb:79:in
`block in get_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/request.rb:493:in
`call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/request.rb:493:in
`process_result'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/request.rb:421:in
`block in transmit'
/opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:853:in `start'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/request.rb:413:in
`transmit'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/request.rb:176:in
`execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/request.rb:41:in
`execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/resource.rb:94:in
`delete'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.9.0/lib/runcible/base.rb:78:in
`get_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.9.0/lib/runcible/base.rb:66:in
`call'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.9.0/lib/runcible/resources/content.rb:89:in
`remove_orphans'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.2.1.1/app/models/katello/glue/pulp/repo.rb:45:in
`delete_orphaned_content'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.2.1.1/lib/katello/tasks/delete_orphaned_content.rake:4:in
`block (2 levels) in <top (required)>'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:240:in
`call'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:240:in
`block in execute'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:235:in
`each'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:235:in
`execute'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:179:in
`block in invoke_with_call_chain'
/opt/rh/rh-ruby22/root/usr/share/ruby/monitor.rb:211:in `mon_synchronize'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:172:in
`invoke_with_call_chain'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:165:in
`invoke'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:150:in
`invoke_task'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in
`block (2 levels) in top_level'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in
`each'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in
`block in top_level'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:115:in
`run_with_threads'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:100:in
`top_level'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:78:in
`block in run'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:176:in
`standard_exception_handling'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:75:in
`run'
/opt/rh/rh-ruby22/root/usr/bin/rake:33:in `<main>'
Tasks: TOP => katello:delete_orphaned_content
The backtrace is:
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action/with_sub_plans.rb:206:in
`check_for_errors!'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action/with_sub_plans.rb:119:in
`wait_for_sub_plans'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action/with_sub_plans.rb:39:in
`initiate'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action/with_sub_plans.rb:16:in
`block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in
`block in assigns'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in
`tap'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in
`assigns'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in
`match_value'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in
`block in match?'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in
`each'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in
`match?'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in
`match'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action/with_sub_plans.rb:11:in
`run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.8.2/app/lib/actions/bulk_action.rb:20:in
`run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action.rb:506:in
`block (3 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:26:in
`call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:26:in
`pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware.rb:17:in
`pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.8.2/app/lib/actions/middleware/keep_current_user.rb:30:in
`block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.8.2/app/lib/actions/middleware/keep_current_user.rb:53:in
`restore_curent_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.8.2/app/lib/actions/middleware/keep_current_user.rb:30:in
`run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:22:in
`call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:26:in
`pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware.rb:17:in
`pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action/progress.rb:30:in
`with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action/progress.rb:16:in
`run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:22:in
`call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:26:in
`pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware.rb:17:in
`pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.2.1.1/app/lib/actions/middleware/keep_locale.rb:11:in
`block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.2.1.1/app/lib/actions/middleware/keep_locale.rb:22:in
`with_locale'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.2.1.1/app/lib/actions/middleware/keep_locale.rb:11:in
`run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:22:in
`call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:26:in
`pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware.rb:17:in
`pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware.rb:30:in
`run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:22:in
`call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/world.rb:30:in
`execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action.rb:505:in
`block (2 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action.rb:504:in
`catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action.rb:504:in
`block in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action.rb:419:in
`call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action.rb:419:in
`block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action.rb:419:in
`catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action.rb:419:in
`with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action.rb:499:in
`execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action.rb:260:in
`execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in
`block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/execution_plan/steps/abstract.rb:155:in
`call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/execution_plan/steps/abstract.rb:155:in
`with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in
`block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in
`open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in
`execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/executors/parallel/worker.rb:15:in
`block in on_message'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in
`block in assigns'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in
`tap'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in
`assigns'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in
`match_value'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in
`block in match?'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in
`each'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in
`match?'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in
`match'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/executors/parallel/worker.rb:12:in
`on_message'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in
`on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:in
`on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
`pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/actor.rb:26:in
`on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
`pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/awaits.rb:15:in
`on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
`pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in
`on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
`pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:38:in
`process_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:31:in
`process_envelopes?'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:20:in
`on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
`pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/termination.rb:55:in
`on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
`pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/removes_child.rb:10:in
`on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
`pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in
`on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:161:in
`process_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:95:in
`block in on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:118:in
`block (2 levels) in schedule_execution'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in
`block in synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in
`synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in
`synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:115:in
`block in schedule_execution'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in
`call'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in
`call'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:96:in
`work'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:77:in
`block in call_job'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in
`call'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in
`run_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:in
`block (3 levels) in create_worker'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in
`loop'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in
`block (2 levels) in create_worker'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in
`catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in
`block in create_worker'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in
`call'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in
`block in create_with_logging_context'
I've broken pulp, haven't I.
L.
------
The most dangerous phrase in the language is, "We've always done it this
way."
- Grace Hopper
On 28 February 2017 at 09:29, Lachlan Musicman <[email protected]> wrote:
> Thankyou - that worked in so much as it didn't throw an error.
>
> I still have the problem of four synchronisations that have stopped but
> aren't completing.
>
> By stopped I mean "are not advancing".
>
> As far as I can tell, they are in a running state - which is why the steps
> above didn't work, they were only for killing tasks in a paused state.
>
> Label: Actions::Katello::Repository::Sync
> Name: Synchronize
> Owner: foreman_admin
> Execution type: Delayed
> Start at: 2017-02-25 03:01:08 +1100
> Start before: -
> Started at: 2017-02-25 03:01:08 +1100
> Ended at:
> State: running
>
> So the task started a three days ago and hasn't advanced any more. This
> job in particular is blocking the other three from completing according to
> the relevant logs.
>
> It looks like I have a suspended Actions::Pulp::Repository::
> RegenerateApplicability:
>
>
> 16: Actions::Pulp::Repository::RegenerateApplicability (suspended) [
> 282330.29s / 1592.86s ]
>
> but I see no documentation on how I might get it out of suspension?
>
> cheers
> L.
>
>
> ------
> The most dangerous phrase in the language is, "We've always done it this
> way."
>
> - Grace Hopper
>
> On 27 February 2017 at 18:07, Sean O'Keeffe <[email protected]> wrote:
>
>> Try foreman-rake katello:reimport
>>
>> foreman-rake katello:reindex Has been renamed recently.
>>
>> On Mon, 27 Feb 2017 at 00:49, Lachlan Musicman <[email protected]> wrote:
>>
>>> Search turned up this potential solution:
>>>
>>> https://access.redhat.com/discussions/2065363
>>>
>>> I tried restarting Katello service as noted, without success. I ended up
>>> here
>>>
>>> https://access.redhat.com/solutions/1381053
>>>
>>> There's an error thrown on the last step:
>>>
>>> [root@vmpr-res-utils ~]# foreman-rake katello:reindex --trace
>>> rake aborted!
>>> Don't know how to build task 'katello:reindex'
>>> /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task_manager.rb:62:in
>>> `[]'
>>> /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:149:in
>>> `invoke_task'
>>> /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in
>>> `block (2 levels) in top_level'
>>> /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in
>>> `each'
>>> /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in
>>> `block in top_level'
>>> /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:115:in
>>> `run_with_threads'
>>> /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:100:in
>>> `top_level'
>>> /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:78:in
>>> `block in run'
>>> /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:176:in
>>> `standard_exception_handling'
>>> /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:75:in
>>> `run'
>>> /opt/rh/rh-ruby22/root/usr/bin/rake:33:in `<main>'
>>>
>>> That will teach me for trying to solve my own problems.
>>>
>>> The problem seems to have persisted - where do I look next?
>>>
>>> cheers
>>> L.
>>>
>>>
>>>
>>> ------
>>> The most dangerous phrase in the language is, "We've always done it this
>>> way."
>>>
>>> - Grace Hopper
>>>
>>> On 27 February 2017 at 09:39, Lachlan Musicman <[email protected]>
>>> wrote:
>>>
>>> CentOS 7.3, Katello 3.2, Foreman 1.13.4
>>>
>>> And it's cascading - there are four other sync's that haven't finished
>>> because one seems to be in a permanent pending state.
>>>
>>> I don't see any obvious cancel/resume function? The cancel link on the
>>> sync status page bounces when clicked but...nothing.
>>>
>>> The logs/Dynaflow show no errors, it's just stopped at 71%.
>>>
>>> How to solve?
>>>
>>> (raw details below)
>>>
>>> cheers
>>> L.
>>>
>>> {"repository"=>{"id"=>5, "name"=>"epel", "label"=>"epel"},
>>> "product"=>
>>> {"id"=>2,
>>> "name"=>"Extra Packages for Enterprise Linux 7",
>>> "label"=>"Extra_Packages_for_Enterprise_Linux_7_-_x86_64",
>>> "cp_id"=>"1464869283627"},
>>> "provider"=>{"id"=>1, "name"=>"Anonymous"},
>>> "organization"=>
>>> {"id"=>1,
>>> "name"=>"Peter MacCallum Cancer Centre",
>>> "label"=>"Peter_MacCallum_Cancer_Centre"},
>>> "services_checked"=>["pulp", "pulp_auth"],
>>> "id"=>5,
>>> "sync_result"=>
>>> {"contents_changed"=>true,
>>> "pulp_tasks"=>
>>> [{"exception"=>nil,
>>> "task_type"=>"pulp.server.managers.repo.sync.sync",
>>> "_href"=>"/pulp/api/v2/tasks/5460a086-d7e8-4510-9549-c75766ec8fc2/",
>>> "task_id"=>"5460a086-d7e8-4510-9549-c75766ec8fc2",
>>> "tags"=>
>>>
>>> ["pulp:repository:Peter_MacCallum_Cancer_Centre-Extra_Packages_for_Enterprise_Linux_7_-_x86_64-epel",
>>> "pulp:action:sync"],
>>> "finish_time"=>"2017-02-24T16:07:18Z",
>>> "_ns"=>"task_status",
>>> "start_time"=>"2017-02-24T16:01:23Z",
>>> "traceback"=>nil,
>>> "spawned_tasks"=>
>>>
>>> [{"_href"=>"/pulp/api/v2/tasks/a9843fac-51de-4fb0-a06d-31c0db9b108e/",
>>> "task_id"=>"a9843fac-51de-4fb0-a06d-31c0db9b108e"}],
>>> "progress_report"=>
>>> {"yum_importer"=>
>>> {"content"=>
>>> {"items_total"=>22,
>>> "state"=>"FINISHED",
>>> "error_details"=>[],
>>> "details"=>
>>> {"rpm_total"=>22,
>>> "rpm_done"=>22,
>>> "drpm_total"=>0,
>>> "drpm_done"=>0},
>>> "size_total"=>16914540,
>>> "size_left"=>0,
>>> "items_left"=>0},
>>> "comps"=>{"state"=>"FINISHED"},
>>> "purge_duplicates"=>{"state"=>"FINISHED"},
>>> "distribution"=>
>>> {"items_total"=>0,
>>> "state"=>"FINISHED",
>>> "error_details"=>[],
>>> "items_left"=>0},
>>> "errata"=>{"state"=>"FINISHED"},
>>> "metadata"=>{"state"=>"FINISHED"}}},
>>> "queue"=>
>>> "reserved_resource_worke...@vmpr-res-utils.unix.petermac.org.au.dq",
>>> "state"=>"finished",
>>> "worker_name"=>
>>> "[email protected]",
>>> "result"=>
>>> {"result"=>"success",
>>> "importer_id"=>"yum_importer",
>>> "exception"=>nil,
>>> "repo_id"=>
>>>
>>> "Peter_MacCallum_Cancer_Centre-Extra_Packages_for_Enterprise_Linux_7_-_x86_64-epel",
>>> "traceback"=>nil,
>>> "started"=>"2017-02-24T16:01:23Z",
>>> "_ns"=>"repo_sync_results",
>>> "completed"=>"2017-02-24T16:07:18Z",
>>> "importer_type_id"=>"yum_importer",
>>> "error_message"=>nil,
>>> "summary"=>
>>> {"content"=>{"state"=>"FINISHED"},
>>> "comps"=>{"state"=>"FINISHED"},
>>> "purge_duplicates"=>{"state"=>"FINISHED"},
>>> "distribution"=>{"state"=>"FINISHED"},
>>> "errata"=>{"state"=>"FINISHED"},
>>> "metadata"=>{"state"=>"FINISHED"}},
>>> "added_count"=>29,
>>> "removed_count"=>-25,
>>> "updated_count"=>14872,
>>> "id"=>"58b05a36bdb1d60a7024bf11",
>>> "details"=>
>>> {"content"=>
>>> {"size_total"=>16914540,
>>> "items_left"=>0,
>>> "items_total"=>22,
>>> "state"=>"FINISHED",
>>> "size_left"=>0,
>>> "details"=>
>>> {"rpm_total"=>22,
>>> "rpm_done"=>22,
>>> "drpm_total"=>0,
>>> "drpm_done"=>0},
>>> "error_details"=>[]},
>>> "comps"=>{"state"=>"FINISHED"},
>>> "purge_duplicates"=>{"state"=>"FINISHED"},
>>> "distribution"=>
>>> {"items_total"=>0,
>>> "state"=>"FINISHED",
>>> "error_details"=>[],
>>> "items_left"=>0},
>>> "errata"=>{"state"=>"FINISHED"},
>>> "metadata"=>{"state"=>"FINISHED"}}},
>>> "error"=>nil,
>>> "_id"=>{"$oid"=>"58b058a1c47a37aeec805a43"},
>>> "id"=>"58b058a1c47a37aeec805a43"},
>>> {"exception"=>nil,
>>> "task_type"=>"pulp.server.managers.repo.publish.publish",
>>> "_href"=>"/pulp/api/v2/tasks/a9843fac-51de-4fb0-a06d-31c0db9b108e/",
>>> "task_id"=>"a9843fac-51de-4fb0-a06d-31c0db9b108e",
>>> "tags"=>
>>>
>>> ["pulp:repository:Peter_MacCallum_Cancer_Centre-Extra_Packages_for_Enterprise_Linux_7_-_x86_64-epel",
>>> "pulp:action:publish"],
>>> "finish_time"=>"2017-02-24T16:09:45Z",
>>> "_ns"=>"task_status",
>>> "start_time"=>"2017-02-24T16:07:21Z",
>>> "traceback"=>nil,
>>> "spawned_tasks"=>[],
>>> "progress_report"=>
>>>
>>> {"Peter_MacCallum_Cancer_Centre-Extra_Packages_for_Enterprise_Linux_7_-_x86_64-epel"=>
>>> [{"num_success"=>1,
>>> "description"=>"Initializing repo metadata",
>>> "step_type"=>"initialize_repo_metadata",
>>> "items_total"=>1,
>>> "state"=>"FINISHED",
>>> "error_details"=>[],
>>> "details"=>"",
>>> "num_failures"=>0,
>>> "step_id"=>"5a446a30-7f4f-4fdc-beed-6a8fd2889a80",
>>> "num_processed"=>1},
>>> {"num_success"=>0,
>>> "description"=>"Publishing Distribution files",
>>> "step_type"=>"distribution",
>>> "items_total"=>0,
>>> "state"=>"FINISHED",
>>> "error_details"=>[],
>>> "details"=>"",
>>> "num_failures"=>0,
>>> "step_id"=>"ed6fbc4a-5bce-40f5-95a0-8138a58ca633",
>>> "num_processed"=>0},
>>> {"num_success"=>11239,
>>> "description"=>"Publishing RPMs",
>>> "step_type"=>"rpms",
>>> "items_total"=>11239,
>>> "state"=>"FINISHED",
>>> "error_details"=>[],
>>> "details"=>"",
>>> "num_failures"=>0,
>>> "step_id"=>"5634cc70-459d-4f08-9ca0-6f9783af239c",
>>> "num_processed"=>11239},
>>> {"num_success"=>0,
>>> "description"=>"Publishing Delta RPMs",
>>> "step_type"=>"drpms",
>>> "items_total"=>1,
>>> "state"=>"SKIPPED",
>>> "error_details"=>[],
>>> "details"=>"",
>>> "num_failures"=>0,
>>> "step_id"=>"9c172b59-01dd-4913-8d0d-40c367d970ca",
>>> "num_processed"=>0},
>>> {"num_success"=>3448,
>>> "description"=>"Publishing Errata",
>>> "step_type"=>"errata",
>>> "items_total"=>3448,
>>> "state"=>"FINISHED",
>>> "error_details"=>[],
>>> "details"=>"",
>>> "num_failures"=>0,
>>> "step_id"=>"76c9b431-7cf9-4edb-a1d0-555cecec03b7",
>>> "num_processed"=>3448},
>>> {"num_success"=>214,
>>> "description"=>"Publishing Comps file",
>>> "step_type"=>"comps",
>>> "items_total"=>214,
>>> "state"=>"FINISHED",
>>> "error_details"=>[],
>>> "details"=>"",
>>> "num_failures"=>0,
>>> "step_id"=>"7c3df37c-419c-44c2-adc7-2c74bd537263",
>>> "num_processed"=>214},
>>> {"num_success"=>0,
>>> "description"=>"Publishing Metadata.",
>>> "step_type"=>"metadata",
>>> "items_total"=>0,
>>> "state"=>"FINISHED",
>>> "error_details"=>[],
>>> "details"=>"",
>>> "num_failures"=>0,
>>> "step_id"=>"a5f913dd-d33c-4777-aa13-b25e086b4dfb",
>>> "num_processed"=>0},
>>> {"num_success"=>1,
>>> "description"=>"Closing repo metadata",
>>> "step_type"=>"close_repo_metadata",
>>> "items_total"=>1,
>>> "state"=>"FINISHED",
>>> "error_details"=>[],
>>> "details"=>"",
>>> "num_failures"=>0,
>>> "step_id"=>"0f71d80f-5e6d-4759-8e69-4b0c3bb559f2",
>>> "num_processed"=>1},
>>> {"num_success"=>0,
>>> "description"=>"Generating sqlite files",
>>> "step_type"=>"generate sqlite",
>>> "items_total"=>1,
>>> "state"=>"SKIPPED",
>>> "error_details"=>[],
>>> "details"=>"",
>>> "num_failures"=>0,
>>> "step_id"=>"12c2b669-a180-44a5-9b75-e0fffaf3dd49",
>>> "num_processed"=>0},
>>> {"num_success"=>0,
>>> "description"=>"Generating HTML files",
>>> "step_type"=>"repoview",
>>> "items_total"=>1,
>>> "state"=>"SKIPPED",
>>> "error_details"=>[],
>>> "details"=>"",
>>> "num_failures"=>0,
>>> "step_id"=>"c3d3f16a-727a-4b7c-839c-45a95a9888a7",
>>> "num_processed"=>0},
>>> {"num_success"=>1,
>>> "description"=>"Publishing files to web",
>>> "step_type"=>"publish_directory",
>>> "items_total"=>1,
>>> "state"=>"FINISHED",
>>> "error_details"=>[],
>>> "details"=>"",
>>> "num_failures"=>0,
>>> "step_id"=>"cb0246f9-c2ef-49b3-9bd2-da53e71d692a",
>>> "num_processed"=>1},
>>> {"num_success"=>1,
>>> "description"=>"Writing Listings File",
>>> "step_type"=>"initialize_repo_metadata",
>>> "items_total"=>1,
>>> "state"=>"FINISHED",
>>> "error_details"=>[],
>>> "details"=>"",
>>> "num_failures"=>0,
>>> "step_id"=>"0f12a784-557d-4f4f-8d5b-f34db4dd59ec",
>>> "num_processed"=>1},
>>> {"num_success"=>1,
>>> "description"=>"Writing Listings File",
>>> "step_type"=>"initialize_repo_metadata",
>>> "items_total"=>1,
>>> "state"=>"FINISHED",
>>> "error_details"=>[],
>>> "details"=>"",
>>> "num_failures"=>0,
>>> "step_id"=>"3dcbf71e-6950-45a1-b834-42ccfc2e8e0f",
>>> "num_processed"=>1}]},
>>> "queue"=>
>>> "reserved_resource_worke...@vmpr-res-utils.unix.petermac.org.au.dq",
>>> "state"=>"finished",
>>> "worker_name"=>
>>> "[email protected]",
>>> "result"=>
>>> {"result"=>"success",
>>> "exception"=>nil,
>>> "repo_id"=>
>>>
>>> "Peter_MacCallum_Cancer_Centre-Extra_Packages_for_Enterprise_Linux_7_-_x86_64-epel",
>>> "started"=>"2017-02-24T16:07:21Z",
>>> "_ns"=>"repo_publish_results",
>>> "completed"=>"2017-02-24T16:09:45Z",
>>> "traceback"=>nil,
>>> "distributor_type_id"=>"yum_distributor",
>>> "summary"=>
>>> {"generate sqlite"=>"SKIPPED",
>>> "initialize_repo_metadata"=>"FINISHED",
>>> "rpms"=>"FINISHED",
>>> "close_repo_metadata"=>"FINISHED",
>>> "drpms"=>"SKIPPED",
>>> "comps"=>"FINISHED",
>>> "distribution"=>"FINISHED",
>>> "repoview"=>"SKIPPED",
>>> "publish_directory"=>"FINISHED",
>>> "errata"=>"FINISHED",
>>> "metadata"=>"FINISHED"},
>>> "error_message"=>nil,
>>> "distributor_id"=>
>>>
>>> "Peter_MacCallum_Cancer_Centre-Extra_Packages_for_Enterprise_Linux_7_-_x86_64-epel",
>>> "id"=>"58b05ac9bdb1d60a7024c08c",
>>> "details"=>
>>> [{"num_success"=>1,
>>> "description"=>"Initializing repo metadata",
>>> "step_type"=>"initialize_repo_metadata",
>>> "items_total"=>1,
>>> "state"=>"FINISHED",
>>> "error_details"=>[],
>>> "details"=>"",
>>> "num_failures"=>0,
>>> "step_id"=>"5a446a30-7f4f-4fdc-beed-6a8fd2889a80",
>>> "num_processed"=>1},
>>> {"num_success"=>0,
>>> "description"=>"Publishing Distribution files",
>>> "step_type"=>"distribution",
>>> "items_total"=>0,
>>> "state"=>"FINISHED",
>>> "error_details"=>[],
>>> "details"=>"",
>>> "num_failures"=>0,
>>> "step_id"=>"ed6fbc4a-5bce-40f5-95a0-8138a58ca633",
>>> "num_processed"=>0},
>>> {"num_success"=>11239,
>>> "description"=>"Publishing RPMs",
>>> "step_type"=>"rpms",
>>> "items_total"=>11239,
>>> "state"=>"FINISHED",
>>> "error_details"=>[],
>>> "details"=>"",
>>> "num_failures"=>0,
>>> "step_id"=>"5634cc70-459d-4f08-9ca0-6f9783af239c",
>>> "num_processed"=>11239},
>>> {"num_success"=>0,
>>> "description"=>"Publishing Delta RPMs",
>>> "step_type"=>"drpms",
>>> "items_total"=>1,
>>> "state"=>"SKIPPED",
>>> "error_details"=>[],
>>> "details"=>"",
>>> "num_failures"=>0,
>>> "step_id"=>"9c172b59-01dd-4913-8d0d-40c367d970ca",
>>> "num_processed"=>0},
>>> {"num_success"=>3448,
>>> "description"=>"Publishing Errata",
>>> "step_type"=>"errata",
>>> "items_total"=>3448,
>>> "state"=>"FINISHED",
>>> "error_details"=>[],
>>> "details"=>"",
>>> "num_failures"=>0,
>>> "step_id"=>"76c9b431-7cf9-4edb-a1d0-555cecec03b7",
>>> "num_processed"=>3448},
>>> {"num_success"=>214,
>>> "description"=>"Publishing Comps file",
>>> "step_type"=>"comps",
>>> "items_total"=>214,
>>> "state"=>"FINISHED",
>>> "error_details"=>[],
>>> "details"=>"",
>>> "num_failures"=>0,
>>> "step_id"=>"7c3df37c-419c-44c2-adc7-2c74bd537263",
>>> "num_processed"=>214},
>>> {"num_success"=>0,
>>> "description"=>"Publishing Metadata.",
>>> "step_type"=>"metadata",
>>> "items_total"=>0,
>>> "state"=>"FINISHED",
>>> "error_details"=>[],
>>> "details"=>"",
>>> "num_failures"=>0,
>>> "step_id"=>"a5f913dd-d33c-4777-aa13-b25e086b4dfb",
>>> "num_processed"=>0},
>>> {"num_success"=>1,
>>> "description"=>"Closing repo metadata",
>>> "step_type"=>"close_repo_metadata",
>>> "items_total"=>1,
>>> "state"=>"FINISHED",
>>> "error_details"=>[],
>>> "details"=>"",
>>> "num_failures"=>0,
>>> "step_id"=>"0f71d80f-5e6d-4759-8e69-4b0c3bb559f2",
>>> "num_processed"=>1},
>>> {"num_success"=>0,
>>> "description"=>"Generating sqlite files",
>>> "step_type"=>"generate sqlite",
>>> "items_total"=>1,
>>> "state"=>"SKIPPED",
>>> "error_details"=>[],
>>> "details"=>"",
>>> "num_failures"=>0,
>>> "step_id"=>"12c2b669-a180-44a5-9b75-e0fffaf3dd49",
>>> "num_processed"=>0},
>>> {"num_success"=>0,
>>> "description"=>"Generating HTML files",
>>> "step_type"=>"repoview",
>>> "items_total"=>1,
>>> "state"=>"SKIPPED",
>>> "error_details"=>[],
>>> "details"=>"",
>>> "num_failures"=>0,
>>> "step_id"=>"c3d3f16a-727a-4b7c-839c-45a95a9888a7",
>>> "num_processed"=>0},
>>> {"num_success"=>1,
>>> "description"=>"Publishing files to web",
>>> "step_type"=>"publish_directory",
>>> "items_total"=>1,
>>> "state"=>"FINISHED",
>>> "error_details"=>[],
>>> "details"=>"",
>>> "num_failures"=>0,
>>> "step_id"=>"cb0246f9-c2ef-49b3-9bd2-da53e71d692a",
>>> "num_processed"=>1},
>>> {"num_success"=>1,
>>> "description"=>"Writing Listings File",
>>> "step_type"=>"initialize_repo_metadata",
>>> "items_total"=>1,
>>> "state"=>"FINISHED",
>>> "error_details"=>[],
>>> "details"=>"",
>>> "num_failures"=>0,
>>> "step_id"=>"0f12a784-557d-4f4f-8d5b-f34db4dd59ec",
>>> "num_processed"=>1},
>>> {"num_success"=>1,
>>> "description"=>"Writing Listings File",
>>> "step_type"=>"initialize_repo_metadata",
>>> "items_total"=>1,
>>> "state"=>"FINISHED",
>>> "error_details"=>[],
>>> "details"=>"",
>>> "num_failures"=>0,
>>> "step_id"=>"3dcbf71e-6950-45a1-b834-42ccfc2e8e0f",
>>> "num_processed"=>1}]},
>>> "error"=>nil,
>>> "_id"=>{"$oid"=>"58b05a36c47a37aeec805b06"},
>>> "id"=>"58b05a36c47a37aeec805b06"}],
>>> "poll_attempts"=>{"total"=>29, "failed"=>0}},
>>> "user_id"=>1,
>>> "contents_changed"=>true,
>>> "current_user_id"=>1}
>>>
>>>
>>>
>>> ------
>>> The most dangerous phrase in the language is, "We've always done it this
>>> way."
>>>
>>> - Grace Hopper
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Foreman users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> To post to this group, send email to [email protected].
>>> Visit this group at https://groups.google.com/group/foreman-users.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Foreman users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To post to this group, send email to [email protected].
>> Visit this group at https://groups.google.com/group/foreman-users.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
--
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
