More interesting things: Using the reactivation feature doesn't work.
[root@zbx-prx-elk-04 log]# /usr/sbin/rhnreg_ks --force --serverUrl= http://spacewalk.companyname.com/XMLRPC --activationkey=9b601306ac1616daf1510d93a5084bf6 Error communicating with server. The message was: Internal Server Error Give it the wrong token, and you get an actual error: [root@zbx-prx-elk-04 log]# /usr/sbin/rhnreg_ks --force --serverUrl= http://spacewalk.companyname.com/XMLRPC --activationkey=1-9b601306ac1616daf1510d93a5084bf6 Error Message: Could not find token '1-9b601306ac1616daf1510d93a5084bf6' Error Class Code: 60 Error Class Info: The activation token specified could not be found on the server. Please retry with a valid key. On Tue, Jan 27, 2015 at 10:36 AM, Chris Swingler <[email protected]> wrote: > Yeah, osad/rhnsd don't seem to have any bearing on this problem, turning > them on/off and trying again was one of the first things I tried. No > change; so it's not like osad is somehow half-picking up the task and then > not finishing it. > > Worth mentioning, if it's not clear: Spacewalk never sees the task even > get acknowledged by the remote system. It just stays at: > > This action will be executed after 1/27/15 10:28:00 AM CST > This action's status is: Queued. > This action has not yet been picked up. > > Of course, brand new VMs I set up to try to replicate this issue work fine. > > Paulo: That's an idea, but not necessarily a *good* one... :) > > On Tue, Jan 27, 2015 at 10:29 AM, Paulo Silva <[email protected]> wrote: > >> I've found an ugly fix for this, all I had to do was to use rhnreg_ks to >> re-register the system and now events on the new profile are being picked >> up by rhn_check. >> >> 2015-01-27 16:07 GMT+00:00 Paulo Silva <[email protected]>: >> >>> Hi Glen, >>> >>> Can't try that since I'm using rhnsd and not osad. >>> >>> Thanks >>> >>> 2015-01-27 15:28 GMT+00:00 Glen Collins <[email protected]>: >>> >>>> Hi Paulo, >>>> >>>> Why don't you try this on the client.... >>>> >>>> service osad stop >>>> rm -rf /etc/sysconfig/rhn/osad-auth.conf >>>> service osad start >>>> (Wait for 2 minutes) and run (On the client) >>>> /usr/sbin/rhn_check >>>> >>>> I've had the same issues and just restarting OSAD didn't do it for me. >>>> But when I removed the auth file, that seemed to fix things. Not elegant in >>>> any sort of means but it seems to work. Have not tried it long term since >>>> I'm putting this "fix" into place right now. I guess this is what happens >>>> when you cobble a product together and "make it work". I have found jabber >>>> to be a PITA! >>>> >>>> Anyways, hope this helps! >>>> >>>> Glen Collins >>>> >>>> ------------------------------ >>>> It's not permissions: >>>> >>>> [root@zbx-prx-elk-04 ~]# rhn-actions-control --report >>>> deploy is enabled >>>> diff is enabled >>>> upload is enabled >>>> mtime_upload is enabled >>>> run is enabled >>>> >>>> It's not SSL either, we have our systems configured to speak in the >>>> clear while troubleshooting this issue. Flipping over to HTTPS doesn't >>>> change anything. >>>> >>>> This particular server has three tasks pending now: two config file >>>> pushes and one remote command (just to run "date"). Here's the output of >>>> rhn_check -vvvv: >>>> >>>> [root@zbx-prx-elk-04 ~]# rhn_check -vvvv >>>> D: opening db environment /var/lib/rpm cdb:mpool:joinenv >>>> D: opening db index /var/lib/rpm/Packages rdonly mode=0x0 >>>> D: locked db index /var/lib/rpm/Packages >>>> D: loading keyring from pubkeys in /var/lib/rpm/pubkeys/*.key >>>> D: couldn't find any keys in /var/lib/rpm/pubkeys/*.key >>>> D: loading keyring from rpmdb >>>> D: opening db index /var/lib/rpm/Name rdonly mode=0x0 >>>> D: added key gpg-pubkey-b6f8c6bf-48853124 to keyring >>>> D: added key gpg-pubkey-8db2536f-4ade0b9d to keyring >>>> D: added key gpg-pubkey-66fd4949-4803fe57 to keyring >>>> D: added key gpg-pubkey-c105b9de-4e0fd3a3 to keyring >>>> D: added key gpg-pubkey-217521f6-45e8a532 to keyring >>>> D: added key gpg-pubkey-0608b895-4bd22942 to keyring >>>> D: added key gpg-pubkey-863a853d-4f55f54d to keyring >>>> D: added key gpg-pubkey-79ea5ed4-508d25a6 to keyring >>>> D: Using legacy gpg-pubkey(s) from rpmdb >>>> D: opening db index /var/lib/rpm/Providename rdonly mode=0x0 >>>> D: do_call packages.checkNeedUpdate('rhnsd=1',){} >>>> D: opening db environment /var/lib/rpm cdb:mpool:joinenv >>>> D: opening db index /var/lib/rpm/Packages rdonly mode=0x0 >>>> D: loading keyring from pubkeys in /var/lib/rpm/pubkeys/*.key >>>> D: couldn't find any keys in /var/lib/rpm/pubkeys/*.key >>>> D: loading keyring from rpmdb >>>> D: opening db index /var/lib/rpm/Name rdonly mode=0x0 >>>> D: added key gpg-pubkey-b6f8c6bf-48853124 to keyring >>>> D: added key gpg-pubkey-8db2536f-4ade0b9d to keyring >>>> D: added key gpg-pubkey-66fd4949-4803fe57 to keyring >>>> D: added key gpg-pubkey-c105b9de-4e0fd3a3 to keyring >>>> D: added key gpg-pubkey-217521f6-45e8a532 to keyring >>>> D: added key gpg-pubkey-0608b895-4bd22942 to keyring >>>> D: added key gpg-pubkey-863a853d-4f55f54d to keyring >>>> D: added key gpg-pubkey-79ea5ed4-508d25a6 to keyring >>>> D: Using legacy gpg-pubkey(s) from rpmdb >>>> D: opening db index /var/lib/rpm/Providename rdonly mode=0x0 >>>> D: closed db index /var/lib/rpm/Providename >>>> D: closed db index /var/lib/rpm/Name >>>> D: closed db index /var/lib/rpm/Packages >>>> D: closed db environment /var/lib/rpm >>>> Loaded plugins: fastestmirror, rhnplugin >>>> Config time: 0.040 >>>> D: login(forceUpdate=False) invoked >>>> D: readCachedLogin invoked >>>> D: Checking pickled loginInfo, currentTime=1422370609.99, >>>> createTime=1422370587.99, expire-offset=3600.0 >>>> D: readCachedLogin(): using pickled loginInfo set to expire at >>>> 1422374187.99 >>>> D: rpcServer: Calling XMLRPC up2date.listChannels >>>> Setting up Package Sacks >>>> Loading mirror speeds from cached hostfile >>>> pkgsack time: 0.071 >>>> rpmdb time: 0.000 >>>> Loading mirror speeds from cached hostfile >>>> repo time: 0.000 >>>> D: local action status: (0, 'rpm database not modified since last >>>> update (or package list recently updated)', {}) >>>> D: rpcServer: Calling XMLRPC registration.welcome_message >>>> D: closed db index /var/lib/rpm/Providename >>>> D: closed db index /var/lib/rpm/Name >>>> D: closed db index /var/lib/rpm/Packages >>>> D: closed db environment /var/lib/rpm >>>> >>>> Paulo: To answer your question about the API, spacecmd calls it >>>> extensively; and you can look up pending tasks by ID. >>>> >>>> spacecmd {SSM:0}> schedule_details 578593 >>>> ID: 578593 >>>> Action: Remote Command on zbx-prx-elk-04.company.com. >>>> User: admincswingler >>>> Date: 20150126T16:36:00 >>>> >>>> Completed: 0 >>>> Failed: 0 >>>> Pending: 1 >>>> >>>> Pending Systems >>>> --------------- >>>> zbx-prx-elk-04.company.com >>>> spacecmd {SSM:0}> schedule_details 578588 >>>> ID: 578588 >>>> Action: Deploy config files to system >>>> User: admincswingler >>>> Date: 20150126T16:22:00 >>>> >>>> Completed: 0 >>>> Failed: 0 >>>> Pending: 1 >>>> >>>> Pending Systems >>>> --------------- >>>> zbx-prx-elk-04.company.com >>>> spacecmd {SSM:0}> schedule_details 578586 >>>> ID: 578586 >>>> Action: Deploy config files to system >>>> User: admincswingler >>>> Date: 20150126T16:18:00 >>>> >>>> Completed: 0 >>>> Failed: 0 >>>> Pending: 1 >>>> >>>> Pending Systems >>>> --------------- >>>> zbx-prx-elk-04.company.com >>>> spacecmd {SSM:0}> >>>> >>>> Still scratching my head on this one, and getting rather frustrated :) >>>> >>>> On Tue, Jan 27, 2015 at 7:02 AM, Paulo Silva <[email protected]> >>>> wrote: >>>> >>>>> I'm checking pending events using the web interface (URL >>>>> /rhn/systems/details/history/Pending.do?sid=1000010296) and it shows 2 >>>>> pending events: and Hardware Refresh and an Errata Update. >>>>> >>>>> The sid seams to be correct with the server I'm running my commands: >>>>> >>>>> # grep 1000010296 /etc/sysconfig/rhn/systemid >>>>> <value><string>ID-1000010296</string></value> >>>>> >>>>> Is there a way to check for pending events using the API? >>>>> >>>>> 2015-01-27 12:45 GMT+00:00 Alexander Innes <[email protected]>: >>>>> >>>>>> Just to make sure im not thinking of the wrong feture are you trying >>>>>> to use to remote commands? >>>>>> Systems -> System -> somethin -> remote command (or a different one >>>>>> so i can test in our environment :) ) >>>>>> >>>>>> from that output its not seeing anything as you said :o >>>>>> >>>>>> On 27 January 2015 at 12:16, Paulo Silva <[email protected]> wrote: >>>>>> >>>>>>> I'm having the same issue, running rhn_check does this: >>>>>>> >>>>>>> # rhn_check -vvv >>>>>>> D: opening db environment /var/lib/rpm cdb:mpool:joinenv >>>>>>> D: opening db index /var/lib/rpm/Packages rdonly mode=0x0 >>>>>>> D: locked db index /var/lib/rpm/Packages >>>>>>> D: loading keyring from pubkeys in /var/lib/rpm/pubkeys/*.key >>>>>>> D: couldn't find any keys in /var/lib/rpm/pubkeys/*.key >>>>>>> D: loading keyring from rpmdb >>>>>>> D: opening db index /var/lib/rpm/Name rdonly mode=0x0 >>>>>>> D: added key gpg-pubkey-c105b9de-4e0fd3a3 to keyring >>>>>>> D: added key gpg-pubkey-b5c61460-41667588 to keyring >>>>>>> D: added key gpg-pubkey-6b8d79e6-3f49313d to keyring >>>>>>> D: added key gpg-pubkey-0608b895-4bd22942 to keyring >>>>>>> D: added key gpg-pubkey-863a853d-4f55f54d to keyring >>>>>>> D: Using legacy gpg-pubkey(s) from rpmdb >>>>>>> D: opening db index /var/lib/rpm/Providename rdonly mode=0x0 >>>>>>> D: do_call packages.checkNeedUpdate('rhnsd=1',){} >>>>>>> D: opening db environment /var/lib/rpm cdb:mpool:joinenv >>>>>>> D: opening db index /var/lib/rpm/Packages rdonly mode=0x0 >>>>>>> D: loading keyring from pubkeys in /var/lib/rpm/pubkeys/*.key >>>>>>> D: couldn't find any keys in /var/lib/rpm/pubkeys/*.key >>>>>>> D: loading keyring from rpmdb >>>>>>> D: opening db index /var/lib/rpm/Name rdonly mode=0x0 >>>>>>> D: added key gpg-pubkey-c105b9de-4e0fd3a3 to keyring >>>>>>> D: added key gpg-pubkey-b5c61460-41667588 to keyring >>>>>>> D: added key gpg-pubkey-6b8d79e6-3f49313d to keyring >>>>>>> D: added key gpg-pubkey-0608b895-4bd22942 to keyring >>>>>>> D: added key gpg-pubkey-863a853d-4f55f54d to keyring >>>>>>> D: Using legacy gpg-pubkey(s) from rpmdb >>>>>>> D: opening db index /var/lib/rpm/Providename rdonly mode=0x0 >>>>>>> D: closed db index /var/lib/rpm/Providename >>>>>>> D: closed db index /var/lib/rpm/Name >>>>>>> D: closed db index /var/lib/rpm/Packages >>>>>>> D: closed db environment /var/lib/rpm >>>>>>> Loaded plugins: etckeeper, fastestmirror, rhnplugin >>>>>>> Config time: 0.073 >>>>>>> D: login(forceUpdate=False) invoked >>>>>>> D: readCachedLogin invoked >>>>>>> D: Checking pickled loginInfo, currentTime=1422360920.07, >>>>>>> createTime=1422360336.74, expire-offset=3600.0 >>>>>>> D: readCachedLogin(): using pickled loginInfo set to expire at >>>>>>> 1422363936.74 >>>>>>> D: rpcServer: Calling XMLRPC up2date.listChannels >>>>>>> This system is receiving updates from RHN Classic or Red Hat >>>>>>> Satellite. >>>>>>> rpmdb time: 0.000 >>>>>>> Loading mirror speeds from cached hostfile >>>>>>> repo time: 0.002 >>>>>>> Setting up Package Sacks >>>>>>> pkgsack time: 0.253 >>>>>>> D: local action status: (0, 'rpm database not modified since last >>>>>>> update (or package list recently updated)', {}) >>>>>>> D: rpcServer: Calling XMLRPC registration.welcome_message >>>>>>> D: closed db index /var/lib/rpm/Providename >>>>>>> D: closed db index /var/lib/rpm/Name >>>>>>> D: closed db index /var/lib/rpm/Packages >>>>>>> D: closed db environment /var/lib/rpm >>>>>>> >>>>>>> >>>>>>> 2015-01-27 10:53 GMT+00:00 Alexander Innes <[email protected]>: >>>>>>> >>>>>>>> Schedual the action then on the server do rhn-chck -vvv. It will >>>>>>>> tell you what the client's doing, my gut feeling is either SSL errors >>>>>>>> OR >>>>>>>> permsions (rhn-actions-control is it?) >>>>>>>> >>>>>>>> On 26 January 2015 at 23:16, Chris Swingler < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hey Spacewalk-list! >>>>>>>>> >>>>>>>>> I have quite a few systems (I haven't dug into seeing if there is >>>>>>>>> any consistency in CentOS versions or the RHN tools) that just plain >>>>>>>>> do not >>>>>>>>> seem to get tasks back from the Spacewalk server. We're running >>>>>>>>> Spacewalk >>>>>>>>> 2.2 on CentOS release 6.5 (Final). >>>>>>>>> >>>>>>>>> I can create a simple task in Spacewalk, like just print the >>>>>>>>> output of "date", and an rhn_check never seems to execute it. It >>>>>>>>> shows up >>>>>>>>> under Events > Pending, the SystemIDs match, but it never moves. >>>>>>>>> >>>>>>>>> Running a packet capture to watch the transaction, it looks like >>>>>>>>> Spacewalk itself isn't even sending the task to the remote system. >>>>>>>>> Spacewalk happily replies with an HTTP 200 and an empty payload. >>>>>>>>> >>>>>>>>> HTTP/1.1 200 OK >>>>>>>>> Date: Mon, 26 Jan 2015 22:47:55 GMT >>>>>>>>> Server: Apache >>>>>>>>> Content-Length: 126 >>>>>>>>> X-RHN-Server-Capability: registration.finish_message(1)=1 >>>>>>>>> X-RHN-Server-Capability: registration.remaining_subscriptions(1)=1 >>>>>>>>> X-RHN-Server-Capability: abrt(1)=1 >>>>>>>>> X-RHN-Server-Capability: registration.update_contact_info(1)=1 >>>>>>>>> X-RHN-Server-Capability: staging_content(1)=1 >>>>>>>>> X-RHN-Server-Capability: applet.has_base_channel(1)=1 >>>>>>>>> X-RHN-Server-Capability: registration.smbios(1)=1 >>>>>>>>> X-RHN-Server-Capability: registration.extended_update_support(1)=1 >>>>>>>>> X-RHN-Server-Capability: rhncfg.filetype.directory(1)=1 >>>>>>>>> X-RHN-Server-Capability: registration.update_systemid(1)=1 >>>>>>>>> X-RHN-Server-Capability: registration.register_osad(1)=1 >>>>>>>>> X-RHN-Server-Capability: registration.delta_packages(1)=1 >>>>>>>>> X-RHN-Server-Capability: cpu_sockets(1)=1 >>>>>>>>> X-RHN-Server-Capability: ipv6(1)=1 >>>>>>>>> X-RHN-Server-Capability: rhncfg.content.base64_decode(1)=1 >>>>>>>>> X-RHN-Server-Capability: xmlrpc.packages.extended_profile(1-2)=1 >>>>>>>>> X-RHN-Server-Capability: xmlrpc.errata.patch_names(1)=1 >>>>>>>>> X-RHN-Server-Capability: xmlrpc.packages.checksums(1)=1 >>>>>>>>> X-RHN-Server-Capability: xmlrpc.login.extra_data(1)=1 >>>>>>>>> X-RHN-Proxy-Version: 0 >>>>>>>>> X-Transport-Info: Extended Capabilities Transport (C) Red Hat, Inc >>>>>>>>> (version 2.5.72-1.el6) >>>>>>>>> X-RHN-Client-Version: 1 >>>>>>>>> Connection: close >>>>>>>>> Content-Type: text/xml >>>>>>>>> >>>>>>>>> <?xml version='1.0'?> >>>>>>>>> <methodResponse> >>>>>>>>> <params> >>>>>>>>> <param> >>>>>>>>> <value><string></string></value> >>>>>>>>> </param> >>>>>>>>> </params> >>>>>>>>> </methodResponse> >>>>>>>>> >>>>>>>>> >>>>>>>>> Full transaction at >>>>>>>>> >>>>>>>>> https://gist.github.com/cswingler/f718abcb9ce290adec29 >>>>>>>>> >>>>>>>>> rhn_server_xmlrpc.log simply shows: >>>>>>>>> >>>>>>>>> 2015/01/26 16:52:07 -05:00 30997 172.29.7.14: >>>>>>>>> xmlrpc/queue.get(1000015056, 2, 'checkins enabled') >>>>>>>>> 2015/01/26 16:52:07 -05:00 30998 172.29.7.14: >>>>>>>>> xmlrpc/up2date.listChannels(1000015056,) >>>>>>>>> 2015/01/26 16:52:08 -05:00 30995 172.29.7.14: >>>>>>>>> xmlrpc/registration.welcome_message('lang: None',) >>>>>>>>> >>>>>>>>> Any basic things I should be checking? I've tried updating the >>>>>>>>> spacewalk/rhn tools on the agent, restarting Spacewalk itself, and >>>>>>>>> nothing >>>>>>>>> seems to make a change. This issue, as far as I can recall, did not >>>>>>>>> seem to >>>>>>>>> appear until after we upgraded to 2.2. >>>>>>>>> >>>>>>>>> Some systems seem just fine, and I can't seem to find a consistent >>>>>>>>> reason why some succeed and others fail. >>>>>>>>> >>>>>>>>> Thanks! >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Spacewalk-list mailing list >>>>>>>>> [email protected] >>>>>>>>> https://www.redhat.com/mailman/listinfo/spacewalk-list >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Spacewalk-list mailing list >>>>>>>> [email protected] >>>>>>>> https://www.redhat.com/mailman/listinfo/spacewalk-list >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Paulo Silva <[email protected]> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Spacewalk-list mailing list >>>>>>> [email protected] >>>>>>> https://www.redhat.com/mailman/listinfo/spacewalk-list >>>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Spacewalk-list mailing list >>>>>> [email protected] >>>>>> https://www.redhat.com/mailman/listinfo/spacewalk-list >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Paulo Silva <[email protected]> >>>>> >>>>> _______________________________________________ >>>>> Spacewalk-list mailing list >>>>> [email protected] >>>>> https://www.redhat.com/mailman/listinfo/spacewalk-list >>>>> >>>> >>>> >>>> _______________________________________________ >>>> Spacewalk-list mailing list >>>> [email protected] >>>> https://www.redhat.com/mailman/listinfo/spacewalk-list >>>> >>>> >>>> _______________________________________________ >>>> Spacewalk-list mailing list >>>> [email protected] >>>> https://www.redhat.com/mailman/listinfo/spacewalk-list >>>> >>> >>> >>> >>> -- >>> Paulo Silva <[email protected]> >>> >> >> >> >> -- >> Paulo Silva <[email protected]> >> >> _______________________________________________ >> Spacewalk-list mailing list >> [email protected] >> https://www.redhat.com/mailman/listinfo/spacewalk-list >> > >
_______________________________________________ Spacewalk-list mailing list [email protected] https://www.redhat.com/mailman/listinfo/spacewalk-list
