Unfortunately that's not enough. I've regenerated the certificate tarballs, but in that same thought, I installed a new smart-proxy from scratch - same story on a new build. I believe the tarballs are still using something incorrectly, or pulp on my foreman server is configured incorrectly.
Today I ran "pulp-gen-ca-certificate", regenerated the certificate tarballs, restarted everything, and still no luck. I'm not sure if this is related, but I've had an issue with pulp giving me a 404. When I checked that out, /etc/httpd/conf.d/pulp.conf was blank. Someone pointed me to this: https://gist.github.com/dLobatog/4053b17713135fae26748b9c2ec7d466, which I installed and everything started out OK. Whatever is managing pulp configs, though, is fighting me, and it reverts to a blank config. On Tuesday, March 14, 2017 at 10:12:12 PM UTC-4, jpavel wrote: > > I'm running foreman 1.14.2, and Katello 3.3.0. > > On the foreman server, I'm seeing these messages: > [Wed Mar 15 01:57:02.739257 2017] [ssl:error] [pid 18720] [client > 10.9.0.1:42382] AH02039: Certificate Verification: Error (20): unable to > get local issuer certificate > ... > eventually followed by a burst of something like this: > 2017-03-15 01:57:02 [foreman-tasks/action] [E] RPM1004: Error retrieving > metadata: Not found (Katello::Errors::PulpError) > | > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/pulp/abstract_async_task.rb:121:in > > `block in external_task=' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/pulp/abstract_async_task.rb:119:in > > `each' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/pulp/abstract_async_task.rb:119:in > > `external_task=' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action/polling.rb:98:in > > `poll_external_task_with_rescue' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action/polling.rb:21:in > > `run' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action/cancellable.rb:9:in > > `run' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/pulp/abstract_async_task.rb:45:in > > `run' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action.rb:506:in > > `block (3 levels) in execute_run' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:26:in > > `call' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:26:in > > `pass' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware.rb:17:in > > `pass' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware.rb:30:in > > `run' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:22:in > > `call' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:26:in > > `pass' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware.rb:17:in > > `pass' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/middleware/remote_action.rb:16:in > > `block in run' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/middleware/remote_action.rb:40:in > > `block in as_remote_user' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/models/katello/concerns/user_extensions.rb:21:in > > `cp_config' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/middleware/remote_action.rb:27:in > > `as_cp_user' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/middleware/remote_action.rb:39:in > > `as_remote_user' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/middleware/remote_action.rb:16:in > > `run' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:22:in > > `call' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:26:in > > `pass' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware.rb:17:in > > `pass' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action/progress.rb:30:in > > `with_progress_calculation' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action/progress.rb:16:in > > `run' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:22:in > > `call' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:26:in > > `pass' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware.rb:17:in > > `pass' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/middleware/keep_locale.rb:11:in > > `block in run' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/middleware/keep_locale.rb:22:in > > `with_locale' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/middleware/keep_locale.rb:11:in > > `run' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:22:in > > `call' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:26:in > > `pass' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware.rb:17:in > > `pass' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware.rb:30:in > > `run' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:22:in > > `call' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/world.rb:30:in > > `execute' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action.rb:505:in > > `block (2 levels) in execute_run' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action.rb:504:in > > `catch' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action.rb:504:in > > `block in execute_run' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action.rb:419:in > > `call' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action.rb:419:in > > `block in with_error_handling' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action.rb:419:in > > `catch' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action.rb:419:in > > `with_error_handling' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action.rb:499:in > > `execute_run' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action.rb:260:in > > `execute' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in > > `block (2 levels) in execute' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/execution_plan/steps/abstract.rb:155:in > > `call' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/execution_plan/steps/abstract.rb:155:in > > `with_meta_calculation' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in > > `block in execute' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in > > `open_action' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in > > `execute' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/director.rb:55:in > > `execute' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/executors/parallel/worker.rb:11:in > > `on_message' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in > > `on_envelope' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:in > > `on_envelope' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in > > `pass' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/actor.rb:26:in > > `on_envelope' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in > > `pass' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/awaits.rb:15:in > > `on_envelope' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in > > `pass' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in > > `on_envelope' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in > > `pass' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:38:in > > `process_envelope' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:31:in > > `process_envelopes?' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:20:in > > `on_envelope' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in > > `pass' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/termination.rb:55:in > > `on_envelope' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in > > `pass' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/removes_child.rb:10:in > > `on_envelope' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in > > `pass' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in > > `on_envelope' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:161:in > > `process_envelope' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:95:in > > `block in on_envelope' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:118:in > > `block (2 levels) in schedule_execution' > | > /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in > > `block in synchronize' > | > /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in > > `synchronize' > | > /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in > > `synchronize' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:115:in > > `block in schedule_execution' > | > /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in > > `call' > | > /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in > > `call' > | > /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:96:in > > `work' > | > /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:77:in > > `block in call_job' > | > /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in > > `call' > | > /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in > > `run_task' > | > /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:in > > `block (3 levels) in create_worker' > | > /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in > > `loop' > | > /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in > > `block (2 levels) in create_worker' > | > /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in > > `catch' > | > /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in > > `block in create_worker' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in > > `call' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in > > `block in create_with_logging_context' > > > On the proxy side, I see this: > Mar 15 01:56:58 smart-proxy-02 pulp: nectar.downloaders.threaded:ERROR: > Skipping requests to <foreman server> due to repeated connection failures: > [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:579) > ... > eventually followed by this: > Mar 15 01:57:04 smart-proxy-02 pulp: > pulp_puppet.plugins.importers.forge:ERROR: (31246-98880) Exception while > retrieving metadata for repository <blah blah> > Mar 15 01:57:04 smart-proxy-02 pulp: > pulp_puppet.plugins.importers.forge:ERROR: (31246-98880) Traceback (most > recent call last): > Mar 15 01:57:04 smart-proxy-02 pulp: > pulp_puppet.plugins.importers.forge:ERROR: (31246-98880) File > "/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/forge.py", > line 113, in _parse_metadata > Mar 15 01:57:04 smart-proxy-02 pulp: > pulp_puppet.plugins.importers.forge:ERROR: (31246-98880) > metadata_json_docs = downloader.retrieve_metadata(self.progress_report) > Mar 15 01:57:04 smart-proxy-02 pulp: > pulp_puppet.plugins.importers.forge:ERROR: (31246-98880) File > "/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/downloaders/web.py", > > line 57, in retrieve_metadata > Mar 15 01:57:04 smart-proxy-02 pulp: > pulp_puppet.plugins.importers.forge:ERROR: (31246-98880) raise > exceptions.FileRetrievalException(report.error_msg) > Mar 15 01:57:04 smart-proxy-02 pulp: > pulp_puppet.plugins.importers.forge:ERROR: (31246-98880) > FileRetrievalException: FileRetrievalException: A connection error occurred > Mar 15 01:57:04 smart-proxy-02 pulp: > pulp_puppet.plugins.importers.forge:ERROR: (31234-04096) Exception while > retrieving metadata for repository <nuance_mobility-Production-Smart-Proxy> > Mar 15 01:57:04 smart-proxy-02 pulp: > pulp_puppet.plugins.importers.forge:ERROR: (31234-04096) Traceback (most > recent call last): > Mar 15 01:57:04 smart-proxy-02 pulp: > pulp_puppet.plugins.importers.forge:ERROR: (31234-04096) File > "/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/forge.py", > line 113, in _parse_metadata > Mar 15 01:57:04 smart-proxy-02 pulp: > pulp_puppet.plugins.importers.forge:ERROR: (31234-04096) > metadata_json_docs = downloader.retrieve_metadata(self.progress_report) > Mar 15 01:57:04 smart-proxy-02 pulp: > pulp_puppet.plugins.importers.forge:ERROR: (31234-04096) File > "/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/downloaders/web.py", > > line 57, in retrieve_metadata > Mar 15 01:57:04 smart-proxy-02 pulp: > pulp_puppet.plugins.importers.forge:ERROR: (31234-04096) raise > exceptions.FileRetrievalException(report.error_msg) > Mar 15 01:57:04 smart-proxy-02 pulp: > pulp_puppet.plugins.importers.forge:ERROR: (31234-04096) > FileRetrievalException: FileRetrievalException: A connection error occurred > > On my foreman server, pulp is configured with this: > [security] > cacert: /etc/pki/pulp/ca.crt > cakey: /etc/pki/pulp/ca.key > > And the proxy is configured with this: > [security] > cacert: /etc/pki/katello/certs/katello-default-ca.crt > cakey: /etc/pki/pulp/ca.key > > *Every* single proxy is experiencing the same error. I installed a new > proxy to test it, and it fails to sync with the same error as well. > > It's probably worth noting that I did change the name of the foreman > server about a week ago (this was succeeding prior to that), and I used a > new script: > https://github.com/Katello/katello-packaging/pull/323/commits > > Everything seemed to be fine after that, but I only recently got around to > checking out the proxies. > > From the proxy, I could run: > openssl s_client -connect foreman-01.prod.mcs.som.mob.nuance.com:443 > -CAfile /etc/pki/katello/certs/katello-default-ca.crt > ...and it completes successfully. > > Thanks for any help pointing me in the right direction! > -- You received this message because you are subscribed to the Google Groups "Foreman users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/foreman-users. For more options, visit https://groups.google.com/d/optout.
