Hi Michael, Disregard previous, our dev env nightly build wasn't updating nightly. Manual push of .26 resolved the issue. Thanks a ton!
Jody McIvor Sr. Systems Administrator, Integration Les Services D'intégration [TEL] (250) 852-5202 [email protected] BCLC.com -----Original Message----- From: [email protected] <[email protected]> On Behalf Of [email protected] Sent: February 14, 2020 9:16 AM To: [email protected] Subject: Spacewalk-list Digest, Vol 141, Issue 19 Send Spacewalk-list mailing list submissions to [email protected] To subscribe or unsubscribe via the World Wide Web, visit https://www.redhat.com/mailman/listinfo/spacewalk-list or, via email, send a message with subject or body 'help' to [email protected] You can reach the person managing the list at [email protected] When replying, please edit your Subject line so it is more specific than "Re: Contents of Spacewalk-list digest..." Today's Topics: 1. Spacewalk 2.10 Failing to sync repo (Michael Mraka) (Jody McIvor) 2. Re: "Invalid function call attempted (code 6)" (Paul Greene) ---------------------------------------------------------------------- Message: 1 Date: Fri, 14 Feb 2020 17:03:25 +0000 From: Jody McIvor <[email protected]> To: "[email protected]" <[email protected]> Subject: [Spacewalk-list] Spacewalk 2.10 Failing to sync repo (Michael Mraka) Message-ID: <[email protected]> Content-Type: text/plain; charset=WINDOWS-1252 Hi Michael, I ran your suggested command to peek at the header info of one of (most recent) the RPMs: ==================================================== [jmcivor@<MYREPOSERVER>]$ rpm -qip <PACKAGENAME>.rpm Name : <PACKAGENAME>-v2.15.01 Epoch : 0 Version : 1 Release : 1 Architecture: x86_64 Install Date: (not installed) Group : BCLC Size : 8121032 License : 2017 BCLC Signature : (none) Source RPM : terminal-source Build Date : Thu 06 Feb 2020 01:44:20 PM PST Build Host : <BUILDHOST> Relocations : (not relocatable) Packager : <USERNAMEOFPACKAGER> Vendor : BCLC URL : n/a Summary : terminal Description : <APPDESCRIPTION> ==================================================== And ran manual repo sync: ==================================================== [root@<SPACEWALKSERVER>]# spacewalk-repo-sync --channel <CHANNELNAME> --type yum 08:42:02 ====================================== 08:42:02 | Channel: <CHANNELNAME> 08:42:02 ====================================== 08:42:02 Sync of channel started. 08:42:02 08:42:02 Processing repository with URL: http://<REPOSERVER>/rhel/x86_64/<CHANNELNAME> / 08:42:03 Packages in repo: 111 08:42:03 Packages already synced: 65 08:42:03 Packages to sync: 14 08:42:03 New packages to download: 14 08:42:03 Downloading packages: 08:42:03 1/14 : <PACKAGENAME>.rpm 08:42:03 2/14 : <PACKAGENAME>.rpm 08:42:03 3/14 : <PACKAGENAME>.rpm 08:42:03 4/14 : <PACKAGENAME>.rpm 08:42:03 5/14 : <PACKAGENAME>.rpm 08:42:03 6/14 : <PACKAGENAME>.rpm 08:42:04 7/14 : <PACKAGENAME>.rpm 08:42:04 8/14 : <PACKAGENAME>.rpm 08:42:04 9/14 : <PACKAGENAME>.rpm 08:42:04 10/14 : <PACKAGENAME>.rpm 08:42:04 11/14 : <PACKAGENAME>.rpm 08:42:04 12/14 : <PACKAGENAME>.rpm 08:42:04 13/14 : <PACKAGENAME>.rpm 08:42:04 14/14 : <PACKAGENAME>.rpm 08:42:05 08:42:05 Importing packages to DB: Importing packages: |##################################################| 100.0% 08:42:05 08:42:05 Errata in repo: 0. 08:42:05 Sync of channel completed in 0:00:03. 08:42:05 Total time: 0:00:03 ==================================================== As usual, it reports completion, but digging into the actual log for this channels reposync (var/log/rhn/reposync/<CHANNELNAME>.log) gives us the details I provided previously: ==================================================== 2020/02/14 08:42:05 -07:00 Sync of channel completed in 0:00:03. 2020/02/14 08:48:08 -07:00 Command: ['/bin/spacewalk-repo-sync', '--channel', '<CHANNELNAME>', '--type', 'yum'] 2020/02/14 08:48:08 -07:00 Sync of channel started. 2020/02/14 08:48:08 -07:00 2020/02/14 08:48:08 -07:00 Processing repository with URL: http://<REPOSERVER>/rhel/x86_64/<CHANNELNAME>/ 2020/02/14 08:48:08 -07:00 Packages in repo: 111 2020/02/14 08:48:08 -07:00 Packages already synced: 65 2020/02/14 08:48:08 -07:00 Packages to sync: 14 2020/02/14 08:48:08 -07:00 New packages to download: 14 2020/02/14 08:48:08 -07:00 Downloading packages: 2020/02/14 08:48:09 -07:00 1/14 : <PACKAGENAME>.rpm 2020/02/14 08:48:09 -07:00 2/14 : <PACKAGENAME>.rpm 2020/02/14 08:48:09 -07:00 3/14 : <PACKAGENAME>.rpm 2020/02/14 08:48:09 -07:00 4/14 : <PACKAGENAME>.rpm 2020/02/14 08:48:09 -07:00 5/14 : <PACKAGENAME>.rpm 2020/02/14 08:48:09 -07:00 6/14 : <PACKAGENAME>.rpm 2020/02/14 08:48:09 -07:00 7/14 : <PACKAGENAME>.rpm 2020/02/14 08:48:09 -07:00 8/14 : <PACKAGENAME>.rpm 2020/02/14 08:48:09 -07:00 9/14 : <PACKAGENAME>.rpm 2020/02/14 08:48:09 -07:00 10/14 : <PACKAGENAME>.rpm 2020/02/14 08:48:09 -07:00 11/14 : <PACKAGENAME>.rpm 2020/02/14 08:48:09 -07:00 12/14 : <PACKAGENAME>.rpm 2020/02/14 08:48:09 -07:00 13/14 : <PACKAGENAME>.rpm 2020/02/14 08:48:09 -07:00 14/14 : <PACKAGENAME>.rpm 2020/02/14 08:48:10 -07:00 Importing packages started. 2020/02/14 08:48:10 -07:00 2020/02/14 08:48:10 -07:00 Importing packages to DB: 2020/02/14 08:48:10 -07:00 unknown header tag 2020/02/14 08:48:11 -07:00 unknown header tag 2020/02/14 08:48:11 -07:00 unknown header tag 2020/02/14 08:48:11 -07:00 unknown header tag 2020/02/14 08:48:11 -07:00 unknown header tag 2020/02/14 08:48:11 -07:00 unknown header tag 2020/02/14 08:48:11 -07:00 unknown header tag 2020/02/14 08:48:11 -07:00 unknown header tag 2020/02/14 08:48:11 -07:00 unknown header tag 2020/02/14 08:48:11 -07:00 unknown header tag 2020/02/14 08:48:11 -07:00 unknown header tag 2020/02/14 08:48:11 -07:00 unknown header tag 2020/02/14 08:48:11 -07:00 unknown header tag 2020/02/14 08:48:11 -07:00 unknown header tag 2020/02/14 08:48:11 -07:00 Importing packages finished. 2020/02/14 08:48:11 -07:00 2020/02/14 08:48:11 -07:00 Errata in repo: 0. 2020/02/14 08:48:11 -07:00 Sync of channel completed in 0:00:03. ==================================================== If I don't hear from you today, have a great V-day :) Thanks, Jody McIvor Sr. Systems Administrator, Integration Les Services D'int?gration [TEL] (250) 852-5202 [email protected] BCLC.com -----Original Message----- From: [email protected] <[email protected]> On Behalf Of [email protected] Sent: February 14, 2020 1:06 AM To: [email protected] Subject: Spacewalk-list Digest, Vol 141, Issue 16 Send Spacewalk-list mailing list submissions to [email protected] To subscribe or unsubscribe via the World Wide Web, visit https://www.redhat.com/mailman/listinfo/spacewalk-list or, via email, send a message with subject or body 'help' to [email protected] You can reach the person managing the list at [email protected] When replying, please edit your Subject line so it is more specific than "Re: Contents of Spacewalk-list digest..." Today's Topics: 1. "Invalid function call attempted (code 6)" (Paul Greene) 2. Re: "Invalid function call attempted (code 6)" (Paul Greene) 3. Re: "Invalid function call attempted (code 6)" (Ezequiel Sozzi) 4. Re: Spacewalk 2.10 Failing to sync repo (Michael Mraka) ---------------------------------------------------------------------- Message: 1 Date: Thu, 13 Feb 2020 17:25:30 -0500 From: Paul Greene <[email protected]> To: [email protected] Subject: [Spacewalk-list] "Invalid function call attempted (code 6)" Message-ID: <camakkvpzcubjjpo8jrphssboxsua4fkaw8t9owl3x35qtt0...@mail.gmail.com> Content-Type: text/plain; charset="utf-8" I have a spacewalk 2.9 server with CentOS 7 clients. When I run a scheduled remote command on 50 systems, usually about half of the systems will get marked as "failed" with the error "Invalid function call attempted (code 6)". They all have the same configuration, and every line put in the remote command will run just fine from a command prompt. If I go into a system that has been marked "failed" and manually verify if the command did what it was supposed to do, many times it actually did succeed, but was still marked "failed". And there are some that did in fact fail. How can I address this error to get rid of the false "failed" messages? I looked in /var/log/up2date on the clients that failed and get just these messages at the time the scheduled task failed: up2date updateLoginfo() login info up2date logging into up2date server up2date successfully retrieved authentication token from up2date server -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://www.redhat.com/archives/spacewalk-list/attachments/20200213/cee09dea/attachment.html> ------------------------------ Message: 2 Date: Thu, 13 Feb 2020 18:26:16 -0500 From: Paul Greene <[email protected]> To: [email protected] Subject: Re: [Spacewalk-list] "Invalid function call attempted (code 6)" Message-ID: <camakkvorhbxcqe2e+u5jtc5hhyynt9wcylo_epwqyodkonq...@mail.gmail.com> Content-Type: text/plain; charset="utf-8" Just to add some more info that might be relevant. The spacewalk server itself is running CentOS 7.7, kernel 3.10.0-1062.9.1.el7.x86_64, and most of the clients that are failing are running the same kernel. There's some machines running an older kernel - 3.10.0-862.14.4.el7.x86_64 - and on these machines they seem to reliably run the task without problems. A small number of machines on the newer kernel succeed with the task, but most don't. On Thu, Feb 13, 2020 at 5:25 PM Paul Greene <[email protected]> wrote: > I have a spacewalk 2.9 server with CentOS 7 clients. When I run a > scheduled remote command on 50 systems, usually about half of the > systems will get marked as "failed" with the error "Invalid function > call attempted (code 6)". > > They all have the same configuration, and every line put in the remote > command will run just fine from a command prompt. If I go into a > system that has been marked "failed" and manually verify if the > command did what it was supposed to do, many times it actually did > succeed, but was still marked "failed". And there are some that did in fact > fail. > > How can I address this error to get rid of the false "failed" messages? > > I looked in /var/log/up2date on the clients that failed and get just > these messages at the time the scheduled task failed: > > up2date updateLoginfo() login info > up2date logging into up2date server > up2date successfully retrieved authentication token from up2date > server > -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://www.redhat.com/archives/spacewalk-list/attachments/20200213/d301b339/attachment.html> ------------------------------ Message: 3 Date: Thu, 13 Feb 2020 21:22:23 -0300 From: Ezequiel Sozzi <[email protected]> To: [email protected] Subject: Re: [Spacewalk-list] "Invalid function call attempted (code 6)" Message-ID: <calqknvl0m+zpnuzw-ftukyc9m82sqhtftm4208nraai25vh...@mail.gmail.com> Content-Type: text/plain; charset="utf-8" Hi Paul, This a more often issue than everybody things, in order to fix this, what we do is to run the next commands on the client side: Disable all the plugins to disable rhnplugin: sed -i 's/plugins=1/plugins=0/g' /etc/yum.conf Disable all the external repositores: yum-config-manager --disable \* Re-enable all the plugins to enable rhnplugin: sed -i 's/plugins=0/plugins=1/g' /etc/yum.conf Update all the packages related to rpm, rhn, and yum: yum update rpm* rhn* yum* -y This fix the issue. At least that's my experience. Hope this helps. BR, Ezequiel El jue., 13 de febrero de 2020 7:26 p. m., Paul Greene < [email protected]> escribi?: > I have a spacewalk 2.9 server with CentOS 7 clients. When I run a > scheduled remote command on 50 systems, usually about half of the > systems will get marked as "failed" with the error "Invalid function > call attempted (code 6)". > > They all have the same configuration, and every line put in the remote > command will run just fine from a command prompt. If I go into a > system that has been marked "failed" and manually verify if the > command did what it was supposed to do, many times it actually did > succeed, but was still marked "failed". And there are some that did in fact > fail. > > How can I address this error to get rid of the false "failed" messages? > > I looked in /var/log/up2date on the clients that failed and get just > these messages at the time the scheduled task failed: > > up2date updateLoginfo() login info > up2date logging into up2date server > up2date successfully retrieved authentication token from up2date > server _______________________________________________ > Spacewalk-list mailing list > [email protected] > https://www.redhat.com/mailman/listinfo/spacewalk-list -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://www.redhat.com/archives/spacewalk-list/attachments/20200213/aa87e3fe/attachment.html> ------------------------------ Message: 4 Date: Fri, 14 Feb 2020 10:06:01 +0100 From: Michael Mraka <[email protected]> To: [email protected] Subject: Re: [Spacewalk-list] Spacewalk 2.10 Failing to sync repo Message-ID: <[email protected]> Content-Type: text/plain; charset=utf-8 Jody McIvor: ... > 2020/02/13 08:09:02 -07:00 Command: ['/usr/bin/spacewalk-repo-sync', > '--channel', <MY CHANNEL NAME>', '--type', 'yum'] > 2020/02/13 08:09:02 -07:00 Sync of channel started. > 2020/02/13 08:09:02 -07:00 > 2020/02/13 08:09:02 -07:00 Processing repository with URL: > http://<MYREPOSERVER>/rhel/x86_64/<MYREPOFOLDER/ > 2020/02/13 08:09:03 -07:00 Packages in repo: 111 > 2020/02/13 08:09:04 -07:00 Packages already synced: 65 > 2020/02/13 08:09:04 -07:00 Packages to sync: 14 > 2020/02/13 08:09:04 -07:00 New packages to download: 14 > 2020/02/13 08:09:04 -07:00 Downloading packages: > 2020/02/13 08:09:04 -07:00 1/14 : <PACKAGENAME>.rpm ... > 2020/02/13 08:09:08 -07:00 Importing packages to DB: > 2020/02/13 08:09:08 -07:00 unknown header tag Hello, I guess rpm packages you try to sync contain a new header which is not recognized by rpm on your server. Try to run from commandline (on server) # rpm -qip <PACKAGENAME>.rpm # /usr/bin/spacewalk-repo-sync --channel <MY CHANNEL NAME> and watch the errors. Regards, -- Michael Mr?ka System Management Engineering, Red Hat ------------------------------ _______________________________________________ Spacewalk-list mailing list [email protected] https://www.redhat.com/mailman/listinfo/spacewalk-list End of Spacewalk-list Digest, Vol 141, Issue 16 *********************************************** ________________________________ This email is intended only for the addressee. It may contain confidential or proprietary information that cannot be disclosed without BCLC's permission. If you have received this email in error, please notify the sender immediately and delete the email. ------------------------------ Message: 2 Date: Fri, 14 Feb 2020 12:15:17 -0500 From: Paul Greene <[email protected]> To: [email protected] Subject: Re: [Spacewalk-list] "Invalid function call attempted (code 6)" Message-ID: <CAMAkkvOWeDkQDWBkU7wTUzuCb6=GkgGy=iyGQAM==nacb8a...@mail.gmail.com> Content-Type: text/plain; charset="utf-8" Ezequiel, When I run rhn_check -vvvv, the only line in there that looks like an error message is "couldn't find any keys in /var/lib/rpm/pubkeys/*.key", (the pubkey folder doesn't exist) but then the next line is "loading keyring from rpmdb", so it looks like it gets the key from there. I get those same messages though on both the machines that fail and the ones that succeed. Paul On Fri, Feb 14, 2020 at 11:51 AM Paul Greene <[email protected]> wrote: > What version of 7 are you running? (including kernel #?) > > On Fri, Feb 14, 2020 at 11:17 AM Ezequiel Sozzi <[email protected]> wrote: > >> Hi Paul, >> >> Versions should not be the problem, I'm managing almost 4000 servers >> with spacewalk and 35% are Centos6 while the other 65% are Centos7. >> have you tried to perform a rhn_check -vvvv from the client? That >> could bring you more information. >> >> BR, >> >> El vie., 14 de feb. de 2020 a la(s) 13:12, Paul Greene ( >> [email protected]) escribi?: >> >>> Ezequiel, >>> >>> I tried it but it didn't seem to do anything. ? >>> These systems have no connection to the internet - our repositories >>> are all internal to the network (one repo for base, one for updates, >>> and one for EPEL), and they have all the latest updates anyway, so >>> there was nothing to update. >>> Not sure where to go with this. >>> Just to add to my second post - older versions of CentOS 7 aren't >>> having issues, and there's many systems still on CentOS 6 that don't >>> have any issues either. So that leads me to believe there's >>> something about the differences in OS versions that are the root of the >>> problem. >>> >>> Paul >>> >>> On Thu, Feb 13, 2020 at 7:23 PM Ezequiel Sozzi <[email protected]> wrote: >>> >>>> Hi Paul, >>>> >>>> This a more often issue than everybody things, in order to fix >>>> this, what we do is to run the next commands on the client side: >>>> >>>> Disable all the plugins to disable rhnplugin: sed -i >>>> 's/plugins=1/plugins=0/g' /etc/yum.conf >>>> >>>> Disable all the external repositores: yum-config-manager --disable >>>> \* >>>> >>>> Re-enable all the plugins to enable rhnplugin: sed -i >>>> 's/plugins=0/plugins=1/g' /etc/yum.conf Update all the packages >>>> related to rpm, rhn, and yum: yum update rpm* >>>> rhn* yum* -y >>>> >>>> This fix the issue. At least that's my experience. Hope this helps. >>>> >>>> BR, >>>> >>>> Ezequiel >>>> >>>> >>>> >>>> El jue., 13 de febrero de 2020 7:26 p. m., Paul Greene < >>>> [email protected]> escribi?: >>>> >>>>> I have a spacewalk 2.9 server with CentOS 7 clients. When I run a >>>>> scheduled remote command on 50 systems, usually about half of the >>>>> systems will get marked as "failed" with the error "Invalid >>>>> function call attempted (code 6)". >>>>> >>>>> They all have the same configuration, and every line put in the >>>>> remote command will run just fine from a command prompt. If I go >>>>> into a system that has been marked "failed" and manually verify if >>>>> the command did what it was supposed to do, many times it actually >>>>> did succeed, but was still marked "failed". And there are some that did >>>>> in fact fail. >>>>> >>>>> How can I address this error to get rid of the false "failed" messages? >>>>> >>>>> I looked in /var/log/up2date on the clients that failed and get >>>>> just these messages at the time the scheduled task failed: >>>>> >>>>> up2date updateLoginfo() login info up2date logging into up2date >>>>> server up2date successfully retrieved authentication token from >>>>> up2date server _______________________________________________ >>>>> Spacewalk-list mailing list >>>>> [email protected] >>>>> https://www.redhat.com/mailman/listinfo/spacewalk-list >>>> >>>> _______________________________________________ >>>> Spacewalk-list mailing list >>>> [email protected] >>>> https://www.redhat.com/mailman/listinfo/spacewalk-list >>> >>> _______________________________________________ >>> Spacewalk-list mailing list >>> [email protected] >>> https://www.redhat.com/mailman/listinfo/spacewalk-list >> >> _______________________________________________ >> Spacewalk-list mailing list >> [email protected] >> https://www.redhat.com/mailman/listinfo/spacewalk-list > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://www.redhat.com/archives/spacewalk-list/attachments/20200214/07d90de0/attachment.html> ------------------------------ _______________________________________________ Spacewalk-list mailing list [email protected] https://www.redhat.com/mailman/listinfo/spacewalk-list End of Spacewalk-list Digest, Vol 141, Issue 19 *********************************************** ________________________________ This email is intended only for the addressee. It may contain confidential or proprietary information that cannot be disclosed without BCLC's permission. If you have received this email in error, please notify the sender immediately and delete the email. _______________________________________________ Spacewalk-list mailing list [email protected] https://www.redhat.com/mailman/listinfo/spacewalk-list
