I'm running foreman 1.14.2, and Katello 3.3.0. On the foreman server, I'm seeing these messages: [Wed Mar 15 01:57:02.739257 2017] [ssl:error] [pid 18720] [client 10.9.0.1:42382] AH02039: Certificate Verification: Error (20): unable to get local issuer certificate ... eventually followed by a burst of something like this: 2017-03-15 01:57:02 [foreman-tasks/action] [E] RPM1004: Error retrieving metadata: Not found (Katello::Errors::PulpError) | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/pulp/abstract_async_task.rb:121:in `block in external_task=' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/pulp/abstract_async_task.rb:119:in `each' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/pulp/abstract_async_task.rb:119:in `external_task=' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action/polling.rb:98:in `poll_external_task_with_rescue' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action/polling.rb:21:in `run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action/cancellable.rb:9:in `run' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/pulp/abstract_async_task.rb:45:in `run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action.rb:506:in `block (3 levels) in execute_run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:26:in `call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:26:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware.rb:17:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware.rb:30:in `run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:22:in `call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:26:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware.rb:17:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/middleware/remote_action.rb:16:in `block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/middleware/remote_action.rb:16:in `run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:22:in `call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:26:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware.rb:17:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action/progress.rb:30:in `with_progress_calculation' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action/progress.rb:16:in `run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:22:in `call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:26:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware.rb:17:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/middleware/keep_locale.rb:11:in `block in run' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/middleware/keep_locale.rb:22:in `with_locale' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/middleware/keep_locale.rb:11:in `run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:22:in `call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:26:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware.rb:17:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware.rb:30:in `run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:22:in `call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/world.rb:30:in `execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action.rb:505:in `block (2 levels) in execute_run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action.rb:504:in `catch' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action.rb:504:in `block in execute_run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action.rb:419:in `call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action.rb:419:in `block in with_error_handling' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action.rb:419:in `catch' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action.rb:419:in `with_error_handling' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action.rb:499:in `execute_run' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action.rb:260:in `execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/director.rb:55:in `execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/executors/parallel/worker.rb:11:in `on_message' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in `on_envelope' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/actor.rb:26:in `on_envelope' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/termination.rb:55:in `on_envelope' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:161:in `process_envelope' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:95:in `block in on_envelope' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:118:in `block (2 levels) in schedule_execution' | /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `block in synchronize' | /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize' | /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:115:in `block in schedule_execution' | /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in `call' | /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in `call' | /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:96:in `work' | /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:77:in `block in call_job' | /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in `call' | /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in `run_task' | /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:in `block (3 levels) in create_worker' | /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in `loop' | /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in `block (2 levels) in create_worker' | /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in `catch' | /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in `block in create_worker' | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `call' | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
On the proxy side, I see this: Mar 15 01:56:58 smart-proxy-02 pulp: nectar.downloaders.threaded:ERROR: Skipping requests to <foreman server> due to repeated connection failures: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:579) ... eventually followed by this: Mar 15 01:57:04 smart-proxy-02 pulp: pulp_puppet.plugins.importers.forge:ERROR: (31246-98880) Exception while retrieving metadata for repository <blah blah> Mar 15 01:57:04 smart-proxy-02 pulp: pulp_puppet.plugins.importers.forge:ERROR: (31246-98880) Traceback (most recent call last): Mar 15 01:57:04 smart-proxy-02 pulp: pulp_puppet.plugins.importers.forge:ERROR: (31246-98880) File "/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/forge.py", line 113, in _parse_metadata Mar 15 01:57:04 smart-proxy-02 pulp: pulp_puppet.plugins.importers.forge:ERROR: (31246-98880) metadata_json_docs = downloader.retrieve_metadata(self.progress_report) Mar 15 01:57:04 smart-proxy-02 pulp: pulp_puppet.plugins.importers.forge:ERROR: (31246-98880) File "/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/downloaders/web.py", line 57, in retrieve_metadata Mar 15 01:57:04 smart-proxy-02 pulp: pulp_puppet.plugins.importers.forge:ERROR: (31246-98880) raise exceptions.FileRetrievalException(report.error_msg) Mar 15 01:57:04 smart-proxy-02 pulp: pulp_puppet.plugins.importers.forge:ERROR: (31246-98880) FileRetrievalException: FileRetrievalException: A connection error occurred Mar 15 01:57:04 smart-proxy-02 pulp: pulp_puppet.plugins.importers.forge:ERROR: (31234-04096) Exception while retrieving metadata for repository <nuance_mobility-Production-Smart-Proxy> Mar 15 01:57:04 smart-proxy-02 pulp: pulp_puppet.plugins.importers.forge:ERROR: (31234-04096) Traceback (most recent call last): Mar 15 01:57:04 smart-proxy-02 pulp: pulp_puppet.plugins.importers.forge:ERROR: (31234-04096) File "/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/forge.py", line 113, in _parse_metadata Mar 15 01:57:04 smart-proxy-02 pulp: pulp_puppet.plugins.importers.forge:ERROR: (31234-04096) metadata_json_docs = downloader.retrieve_metadata(self.progress_report) Mar 15 01:57:04 smart-proxy-02 pulp: pulp_puppet.plugins.importers.forge:ERROR: (31234-04096) File "/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/downloaders/web.py", line 57, in retrieve_metadata Mar 15 01:57:04 smart-proxy-02 pulp: pulp_puppet.plugins.importers.forge:ERROR: (31234-04096) raise exceptions.FileRetrievalException(report.error_msg) Mar 15 01:57:04 smart-proxy-02 pulp: pulp_puppet.plugins.importers.forge:ERROR: (31234-04096) FileRetrievalException: FileRetrievalException: A connection error occurred On my foreman server, pulp is configured with this: [security] cacert: /etc/pki/pulp/ca.crt cakey: /etc/pki/pulp/ca.key And the proxy is configured with this: [security] cacert: /etc/pki/katello/certs/katello-default-ca.crt cakey: /etc/pki/pulp/ca.key *Every* single proxy is experiencing the same error. I installed a new proxy to test it, and it fails to sync with the same error as well. It's probably worth noting that I did change the name of the foreman server about a week ago (this was succeeding prior to that), and I used a new script: https://github.com/Katello/katello-packaging/pull/323/commits Everything seemed to be fine after that, but I only recently got around to checking out the proxies. >From the proxy, I could run: openssl s_client -connect foreman-01.prod.mcs.som.mob.nuance.com:443 -CAfile /etc/pki/katello/certs/katello-default-ca.crt ...and it completes successfully. Thanks for any help pointing me in the right direction! -- You received this message because you are subscribed to the Google Groups "Foreman users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/foreman-users. For more options, visit https://groups.google.com/d/optout.
