Are the broken hosts still blocked by a stopped task? If so, can you resume that task? If the task creating them hasn't completed, then deleting them may cause some issues.
John Mitsch Red Hat Engineering (860)-967-7285 irc: jomitsch On Thu, Jul 28, 2016 at 12:40 PM, <[email protected]> wrote: > certificate and key were missing from /etc/pki/pulp. Seems like this step > was missed/failed during the install - maybe due to system clock being out? > > I've put some in place (and fixed system clock) and can register new hosts > now, but the two broken hosts are completely wedged. Any way to forcibly > remove them? > > > On Thursday, 28 July 2016 15:55:20 UTC+1, John Mitsch wrote: >> >> Do you see anything related in /var/log/messages? >> >> John Mitsch >> Red Hat Engineering >> (860)-967-7285 >> irc: jomitsch >> >> On Thu, Jul 28, 2016 at 10:37 AM, <[email protected]> wrote: >> >>> The Action is Actions::Pulp::Consumer::Create. I tried doing a resume on >>> the task and the error is now 409 Conflict. There is a large backtrace. >>> >>> /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in >>> `return!' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.2/lib/runcible/base.rb:79:in >>> `block in get_response' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in >>> `call' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in >>> `process_result' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.37/lib/restclient_ext/request.rb:50:in >>> `block in transmit' >>> /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:853:in `start' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.37/lib/restclient_ext/request.rb:44:in >>> `transmit' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in >>> `execute' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in >>> `execute' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:67:in >>> `post' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.2/lib/runcible/base.rb:78:in >>> `get_response' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.2/lib/runcible/base.rb:66:in >>> `call' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.2/lib/runcible/resources/consumer.rb:20:in >>> `create' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/pulp/consumer/create.rb:13:in >>> `run' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:506:in >>> `block (3 levels) in execute_run' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in >>> `call' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in >>> `pass' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in >>> `pass' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:30:in >>> `run' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in >>> `call' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in >>> `pass' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in >>> `pass' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/remote_action.rb:16:in >>> `block in run' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/remote_action.rb:40:in >>> `block in as_remote_user' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/models/katello/concerns/user_extensions.rb:20:in >>> `cp_config' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/remote_action.rb:27:in >>> `as_cp_user' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/remote_action.rb:39:in >>> `as_remote_user' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/remote_action.rb:16:in >>> `run' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in >>> `call' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in >>> `pass' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in >>> `pass' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/progress.rb:30:in >>> `with_progress_calculation' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/progress.rb:16:in >>> `run' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in >>> `call' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in >>> `pass' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in >>> `pass' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/keep_locale.rb:11:in >>> `block in run' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/keep_locale.rb:22:in >>> `with_locale' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/actions/middleware/keep_locale.rb:11:in >>> `run' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in >>> `call' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in >>> `pass' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in >>> `pass' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:30:in >>> `run' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in >>> `call' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/world.rb:30:in >>> `execute' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:505:in >>> `block (2 levels) in execute_run' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:504:in >>> `catch' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:504:in >>> `block in execute_run' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in >>> `call' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in >>> `block in with_error_handling' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in >>> `catch' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in >>> `with_error_handling' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:499:in >>> `execute_run' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:260:in >>> `execute' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in >>> `block (2 levels) in execute' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract.rb:155:in >>> `call' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract.rb:155:in >>> `with_meta_calculation' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in >>> `block in execute' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in >>> `open_action' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in >>> `execute' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/executors/parallel/worker.rb:15:in >>> `block in on_message' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in >>> `block in assigns' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in >>> `tap' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in >>> `assigns' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in >>> `match_value' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in >>> `block in match?' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in >>> `each' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in >>> `match?' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in >>> `match' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/executors/parallel/worker.rb:12:in >>> `on_message' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in >>> `on_envelope' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:in >>> `on_envelope' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in >>> `pass' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/actor.rb:26:in >>> `on_envelope' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in >>> `pass' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/awaits.rb:15:in >>> `on_envelope' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in >>> `pass' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in >>> `on_envelope' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in >>> `pass' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:38:in >>> `process_envelope' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:31:in >>> `process_envelopes?' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:20:in >>> `on_envelope' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in >>> `pass' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/termination.rb:55:in >>> `on_envelope' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in >>> `pass' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/removes_child.rb:10:in >>> `on_envelope' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in >>> `pass' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in >>> `on_envelope' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:161:in >>> `process_envelope' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:95:in >>> `block in on_envelope' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:118:in >>> `block (2 levels) in schedule_execution' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in >>> `block in synchronize' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in >>> `synchronize' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in >>> `synchronize' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:115:in >>> `block in schedule_execution' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in >>> `call' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in >>> `call' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:96:in >>> `work' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:77:in >>> `block in call_job' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in >>> `call' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in >>> `run_task' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:in >>> `block (3 levels) in create_worker' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in >>> `loop' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in >>> `block (2 levels) in create_worker' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in >>> `catch' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in >>> `block in create_worker' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in >>> `call' >>> /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in >>> `block in create_with_logging_context' >>> >>> >>> >>> On Thursday, 28 July 2016 15:01:58 UTC+1, John Mitsch wrote: >>>> >>>> Alan, >>>> >>>> Can you navigate to the task that it is conflicting with and look for >>>> what is wrong with that task? The errored task is blocking your content >>>> host deletions from running. You can find it with the URL in the error >>>> message or by searching 'id = ca67ff65-c365-441d-92ca-06b0f8c23d0c' in the >>>> tasks page. >>>> >>>> Thanks, >>>> >>>> John Mitsch >>>> Red Hat Engineering >>>> (860)-967-7285 >>>> irc: jomitsch >>>> >>>> On Thu, Jul 28, 2016 at 9:55 AM, <[email protected]> wrote: >>>> >>>>> Hi, >>>>> >>>>> Just done a fresh install of Katello 3 in the lab and I cannot >>>>> register any content hosts to it. >>>>> >>>>> The initial request for a host fails with >>>>> >>>>> Task ca67ff65-c365-441d-92ca-06b0f8c23d0c: >>>>> RestClient::InternalServerError: 500 Internal Server Error >>>>> >>>>> And any subsequent request gets: >>>>> >>>>> Required lock is already taken by other running tasks. >>>>> Please inspect their state, fix their errors and resume them. >>>>> >>>>> Required lock: read >>>>> Conflicts with tasks: >>>>> - >>>>> https://nyn-katello-01.abu.inmarsat.com/foreman_tasks/tasks/ca67ff65-c365-441d-92ca-06b0f8c23d0c >>>>> >>>>> Attempting to delete the host in the UI results in the same error. >>>>> >>>>> Any suggestions? >>>>> >>>>> Thanks, >>>>> >>>>> Alan >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Foreman users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at https://groups.google.com/group/foreman-users. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Foreman users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at https://groups.google.com/group/foreman-users. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Foreman users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/foreman-users. For more options, visit https://groups.google.com/d/optout.
