I think that I figured out what is wrong, I just need to figure out how to resolve it.
Last November, I upgraded the SW server to v2.6, and a few months ago, I changed jabberd to use the Postgres DB. However, the proxy server is still on v2.4 (I forgot all about upgrading it when I did the primary server). I started reviewing the jabberd config files, c2s.xml, s2s.xml and sm.xml and noticed that the current config files are significatly differnet from the dist files, plus they had the changes that I made when switching to the Postgres DB. My guess is that SW must have updated the config files when I made the changes a few months ago. I copied the .dist files back to being the primary config file and changed the ID and password everywhere that I found the settings, but I'm still not able to connect. Besides the server ID and DB password, is there anything else in the config file that I need to change? Do I need to upgrade to v2.6 and install Postgres on the proxy to make everything work? Thanks Daryl ________________________________ From: [email protected] <[email protected]> on behalf of Daryl Rose <[email protected]> Sent: Tuesday, June 6, 2017 10:10 AM To: [email protected] Subject: Re: [Spacewalk-list] Inactive Systems Yes I did and I get: service osad start Starting osad: 2017-06-06 15:08:27 jabber_lib.main: Unable to connect to jabber servers, sleeping 101 seconds Any thoughts? Thank you. Daryl ________________________________ From: [email protected] <[email protected]> on behalf of Wilkinson, Matthew <[email protected]> Sent: Monday, June 5, 2017 2:31 PM To: [email protected] Subject: Re: [Spacewalk-list] Inactive Systems Did you rm the /etc/sysconfig/rhn/osad-auth.conf file and then restart jabberd? --Matthew Wilkinson From: [email protected] [mailto:[email protected]] On Behalf Of Daryl Rose Sent: Monday, June 05, 2017 14:26 To: [email protected] Subject: [Spacewalk-list] Inactive Systems [This is an external email. Be cautious with links, attachments and responses.] ________________________________ Last week 218 systems became "Inactive Systems". All of these systems are in the DMZ and connect via a proxy server. I discovered that /var on the proxy had filled, and I'm guessing corrupted the Berkeley Database. I shutdown Jabberd, cleaned everything out, restarted jabberd, and tried to re-authenticate the inactive systems. However, I'm not able to get them to re-authenticate. There are multiple errors in multiple error logs. This is a snippet from the /var/log/httpd/error log: [Mon Jun 05 10:41:19 2017] [error] Exception reported from <FQDN proxy server> [Mon Jun 05 10:41:19 2017] [error] Time: Mon Jun 5 10:41:19 2017 [Mon Jun 05 10:41:19 2017] [error] Exception type <type 'exceptions.EOFError'> [Mon Jun 05 10:41:19 2017] [error] Exception while handling function handler [Mon Jun 05 10:41:19 2017] [error] Request object information: [Mon Jun 05 10:41:19 2017] [error] URI: /XMLRPC [Mon Jun 05 10:41:19 2017] [error] Remote Host: <web-server> [Mon Jun 05 10:41:19 2017] [error] Server Name: <FQDN proxy server>:443 [Mon Jun 05 10:41:19 2017] [error] Headers passed in: [Mon Jun 05 10:41:19 2017] [error] \tAccept-Encoding: identity [Mon Jun 05 10:41:19 2017] [error] \tCONTENT_LENGTH: 122 [Mon Jun 05 10:41:19 2017] [error] \tCONTENT_TYPE: text/xml [Mon Jun 05 10:41:19 2017] [error] \tDOCUMENT_ROOT: /var/www/html [Mon Jun 05 10:41:19 2017] [error] \tGATEWAY_INTERFACE: CGI/1.1 [Mon Jun 05 10:41:19 2017] [error] \tHTTPS: 1 [Mon Jun 05 10:41:19 2017] [error] \tHTTP_ACCEPT_ENCODING: identity [Mon Jun 05 10:41:19 2017] [error] \tHTTP_HOST: <FQDN proxy server> [Mon Jun 05 10:41:19 2017] [error] \tHTTP_USER_AGENT: rhn.rpclib.py/2.5.75-1.el6 [Mon Jun 05 10:41:19 2017] [error] \tHTTP_X_CLIENT_VERSION: 1 [Mon Jun 05 10:41:19 2017] [error] \tHTTP_X_INFO: RPC Processor (C) Red Hat, Inc (version 2.5.75-1.el6) [Mon Jun 05 10:41:19 2017] [error] \tHTTP_X_RHN_TRANSPORT_CAPABILITY: follow-redirects=3 [Mon Jun 05 10:41:19 2017] [error] \tHTTP_X_TRANSPORT_INFO: Extended Capabilities Transport (C) Red Hat, Inc (version 2.5.75-1.el6) [Mon Jun 05 10:41:19 2017] [error] \tHost: <FQDN proxy server> [Mon Jun 05 10:41:19 2017] [error] \tPATH_INFO: [Mon Jun 05 10:41:19 2017] [error] \tQUERY_STRING: [Mon Jun 05 10:41:19 2017] [error] \tREMOTE_ADDR: 172.19.216.14 [Mon Jun 05 10:41:19 2017] [error] \tREMOTE_PORT: 43218 [Mon Jun 05 10:41:19 2017] [error] \tREQUEST_METHOD: POST [Mon Jun 05 10:41:19 2017] [error] \tREQUEST_URI: /XMLRPC [Mon Jun 05 10:41:19 2017] [error] \tSCRIPT_FILENAME: /usr/share/rhn/wsgi/xmlrpc.py [Mon Jun 05 10:41:19 2017] [error] \tSCRIPT_NAME: /XMLRPC [Mon Jun 05 10:41:19 2017] [error] \tSCRIPT_URI: https://<FQDN<https://%3cFQDN> proxy server>/XMLRPC [Mon Jun 05 10:41:19 2017] [error] \tSCRIPT_URL: /XMLRPC [Mon Jun 05 10:41:19 2017] [error] \tSERVER_ADDR: 172.17.210.59 [Mon Jun 05 10:41:19 2017] [error] \tSERVER_ADMIN: root@localhost [Mon Jun 05 10:41:19 2017] [error] \tSERVER_NAME: <FQDN proxy server> [Mon Jun 05 10:41:19 2017] [error] \tSERVER_PORT: 443 [Mon Jun 05 10:41:19 2017] [error] \tSERVER_PROTOCOL: HTTP/1.1 [Mon Jun 05 10:41:19 2017] [error] \tSERVER_SIGNATURE: <address>Apache/2.2.15 (Red Hat) Server at <FQDN proxy server> Port 443</address> [Mon Jun 05 10:41:19 2017] [error] [Mon Jun 05 10:41:19 2017] [error] \tSERVER_SOFTWARE: Apache/2.2.15 (Red Hat) [Mon Jun 05 10:41:19 2017] [error] \tUser-Agent: rhn.rpclib.py/2.5.75-1.el6 [Mon Jun 05 10:41:19 2017] [error] \tX-Client-Version: 1 [Mon Jun 05 10:41:19 2017] [error] \tX-Info: RPC Processor (C) Red Hat, Inc (version 2.5.75-1.el6) [Mon Jun 05 10:41:19 2017] [error] \tX-RHN-Transport-Capability: follow-redirects=3 [Mon Jun 05 10:41:19 2017] [error] \tX-Transport-Info: Extended Capabilities Transport (C) Red Hat, Inc (version 2.5.75-1.el6) [Mon Jun 05 10:41:19 2017] [error] \tmod_ssl.is_https: <built-in method ssl_is_https of mod_wsgi.Adapter object at 0x7fd673532a08> [Mon Jun 05 10:41:19 2017] [error] \tmod_ssl.var_lookup: <built-in method ssl_var_lookup of mod_wsgi.Adapter object at 0x7fd673532a08> [Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.application_group: corp-spwlkprxy-prod01.dtndmz.com|/xmlrpc [Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.callable_object: application [Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.handler_script: [Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.input_chunked: 0 [Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.listener_host: [Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.listener_port: 443 [Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.process_group: [Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.request_handler: wsgi-script [Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.script_reloading: 1 [Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.version: (3, 2) [Mon Jun 05 10:41:19 2017] [error] \twsgi.errors: <mod_wsgi.Log object at 0x7fd673415ab0> [Mon Jun 05 10:41:19 2017] [error] \twsgi.file_wrapper: <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7fd673532a08> [Mon Jun 05 10:41:19 2017] [error] \twsgi.input: <cStringIO.StringO object at 0x7fd6733ffab0> [Mon Jun 05 10:41:19 2017] [error] \twsgi.multiprocess: True [Mon Jun 05 10:41:19 2017] [error] \twsgi.multithread: False [Mon Jun 05 10:41:19 2017] [error] \twsgi.run_once: False [Mon Jun 05 10:41:19 2017] [error] \twsgi.url_scheme: https [Mon Jun 05 10:41:19 2017] [error] \twsgi.version: (1, 1) [Mon Jun 05 10:41:19 2017] [error] Extra information about this error: [Mon Jun 05 10:41:19 2017] [error] Unhandled exception type [Mon Jun 05 10:41:19 2017] [error] [Mon Jun 05 10:41:19 2017] [error] Exception Handler Information [Mon Jun 05 10:41:19 2017] [error] Traceback (most recent call last): [Mon Jun 05 10:41:19 2017] [error] File "/usr/share/rhn/proxy/apacheServer.py", line 63, in __call__ [Mon Jun 05 10:41:19 2017] [error] ret = f(req) [Mon Jun 05 10:41:19 2017] [error] File "/usr/share/rhn/proxy/apacheHandler.py", line 357, in handler [Mon Jun 05 10:41:19 2017] [error] ret = handlerObj.handler() [Mon Jun 05 10:41:19 2017] [error] File "/usr/share/rhn/proxy/broker/rhnBroker.py", line 220, in handler [Mon Jun 05 10:41:19 2017] [error] authToken = self.proxyAuth.check_cached_token() [Mon Jun 05 10:41:19 2017] [error] File "/usr/share/rhn/proxy/rhnProxyAuth.py", line 130, in check_cached_token [Mon Jun 05 10:41:19 2017] [error] oldToken = self.get_cached_token() [Mon Jun 05 10:41:19 2017] [error] File "/usr/share/rhn/proxy/rhnProxyAuth.py", line 147, in get_cached_token [Mon Jun 05 10:41:19 2017] [error] return shelf[key] [Mon Jun 05 10:41:19 2017] [error] File "/usr/share/rhn/proxy/rhnProxyAuth.py", line 450, in __getitem__ [Mon Jun 05 10:41:19 2017] [error] val = rhnCache.get(rkey, missing_is_null=0) [Mon Jun 05 10:41:19 2017] [error] File "/usr/lib/python2.6/site-packages/spacewalk/common/rhnCache.py", line 70, in get [Mon Jun 05 10:41:19 2017] [error] return cache.get(name, modified) [Mon Jun 05 10:41:19 2017] [error] File "/usr/lib/python2.6/site-packages/spacewalk/common/rhnCache.py", line 366, in get [Mon Jun 05 10:41:19 2017] [error] return cPickle.loads(pickled) [Mon Jun 05 10:41:19 2017] [error] EOFError This is from one of the clients that can't connect: 2017-06-05 12:46:48 rhn_log.log_error: Error caught: 2017-06-05 12:46:48 rhn_log.log_error: Traceback (most recent call last): File "/usr/share/rhn/osad/jabber_lib.py", line 127, in main self.fix_connection(c) File "/usr/share/rhn/osad/osad.py", line 202, in fix_connection server.registration.welcome_message() File "/usr/lib64/python2.6/site-packages/rhn/rpclib.py", line 662, in __call__ return self._send(self._name, args) File "/usr/lib64/python2.6/site-packages/rhn/rpclib.py", line 394, in _request self._handler, request, verbose=self._verbose) File "/usr/lib64/python2.6/site-packages/rhn/transports.py", line 177, in request headers, fd = req.send_http(host, handler) File "/usr/lib64/python2.6/site-packages/rhn/transports.py", line 740, in send_http response.status, response.reason, response.msg) ProtocolError: <ProtocolError for <FQDN Proxy Server> /XMLRPC: 500 Internal Server Error> My inital thoughts turn to the certifiacation, but It that shouldn't be the case, because this has been working fine for over a year. As a matter of fact, I can see that the client estabilishes a connection on with the proxy server on port 5222. Anyone have any thoughts on what could be causing this? Thank you. Daryl
_______________________________________________ Spacewalk-list mailing list [email protected] https://www.redhat.com/mailman/listinfo/spacewalk-list
