Thank you so much for giving this some attention and patching, Michael!
Sadly, the issue remains. First thing this morning I logged into the instance
of Spacewalk experiencing this issue and ensured that it had indeed received
spacewalk-backend-tools-2.10.25 overnight, and it had. So I immediately
attempted a repository sync and at first glance it seems all the same errors
are still occurring. Oddly enough, Error code 14 seems to not exist (I've
googled the heck out of it a few times in recent months/weeks). Attempting to
sync a channel that was created a few weeks ago during testing returns error 5.
- Email rcvd from spacewalk taskomatic:
"Taskomatic bunch repo-sync-bunch was scheduled to run within the
single-repo-sync-bunch-1 schedule. Subtask repo-sync failed. For more
information check
/var/log/rhn/tasko/org1/repo-sync-bunch/repo-sync_9902924_err."
- Contents of repo-sync_9902924_err:
2020-02-13 08:09:09,034 [DefaultQuartzScheduler_Worker-3] ERROR
com.redhat.rhn.taskomatic.task.RepoSyncTask - Executing a task threw an
exception: org.quartz.JobExecutionException
2020-02-13 08:09:09,034 [DefaultQuartzScheduler_Worker-3] ERROR
com.redhat.rhn.taskomatic.task.RepoSyncTask - Message: Command
'[/usr/bin/spacewalk-repo-sync, --channel, <MY CHANNEL NAME>, --type, yum]'
exited with error code 14
2020-02-13 08:09:09,034 [DefaultQuartzScheduler_Worker-3] ERROR
com.redhat.rhn.taskomatic.task.RepoSyncTask - Cause: null
2020-02-13 08:09:09,050 [DefaultQuartzScheduler_Worker-3] ERROR
com.redhat.rhn.taskomatic.task.RepoSyncTask - Stack
trace:org.quartz.JobExecutionException: Command '[/usr/bin/spacewalk-repo-sync,
--channel, <MY CHANNEL NAME>, --type, yum]' exited with error code 14
at
com.redhat.rhn.taskomatic.task.RhnJavaJob.executeExtCmd(RhnJavaJob.java:103)
at
com.redhat.rhn.taskomatic.task.RepoSyncTask.execute(RepoSyncTask.java:70)
at com.redhat.rhn.taskomatic.task.RhnJavaJob.execute(RhnJavaJob.java:88)
at com.redhat.rhn.taskomatic.TaskoJob.execute(TaskoJob.java:186)
at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
-Contents of <MY CHANNEL NAME>.log:
2020/02/13 08:09:02 -07:00 Command: ['/usr/bin/spacewalk-repo-sync',
'--channel', <MY CHANNEL NAME>', '--type', 'yum']
2020/02/13 08:09:02 -07:00 Sync of channel started.
2020/02/13 08:09:02 -07:00
2020/02/13 08:09:02 -07:00 Processing repository with URL:
http://<MYREPOSERVER>/rhel/x86_64/<MYREPOFOLDER/
2020/02/13 08:09:03 -07:00 Packages in repo: 111
2020/02/13 08:09:04 -07:00 Packages already synced: 65
2020/02/13 08:09:04 -07:00 Packages to sync: 14
2020/02/13 08:09:04 -07:00 New packages to download: 14
2020/02/13 08:09:04 -07:00 Downloading packages:
2020/02/13 08:09:04 -07:00 1/14 : <PACKAGENAME>.rpm
2020/02/13 08:09:04 -07:00 2/14 : <PACKAGENAME>.rpm
2020/02/13 08:09:04 -07:00 3/14 : <PACKAGENAME>.rpm
2020/02/13 08:09:04 -07:00 4/14 : <PACKAGENAME>.rpm
2020/02/13 08:09:05 -07:00 5/14 : <PACKAGENAME>.rpm
2020/02/13 08:09:05 -07:00 6/14 : <PACKAGENAME>.rpm
2020/02/13 08:09:06 -07:00 7/14 : <PACKAGENAME>.rpm
2020/02/13 08:09:06 -07:00 8/14 : <PACKAGENAME>.rpm
2020/02/13 08:09:06 -07:00 9/14 : <PACKAGENAME>.rpm
2020/02/13 08:09:07 -07:00 10/14 : <PACKAGENAME>.rpm
2020/02/13 08:09:07 -07:00 11/14 : <PACKAGENAME>.rpm
2020/02/13 08:09:08 -07:00 12/14 : <PACKAGENAME>.rpm
2020/02/13 08:09:08 -07:00 13/14 : <PACKAGENAME>.rpm
2020/02/13 08:09:08 -07:00 14/14 : <PACKAGENAME>.rpm
2020/02/13 08:09:08 -07:00 Importing packages started.
2020/02/13 08:09:08 -07:00
2020/02/13 08:09:08 -07:00 Importing packages to DB:
2020/02/13 08:09:08 -07:00 unknown header tag
2020/02/13 08:09:08 -07:00 unknown header tag
2020/02/13 08:09:08 -07:00 unknown header tag
2020/02/13 08:09:08 -07:00 unknown header tag
2020/02/13 08:09:08 -07:00 unknown header tag
2020/02/13 08:09:08 -07:00 unknown header tag
2020/02/13 08:09:08 -07:00 unknown header tag
2020/02/13 08:09:08 -07:00 unknown header tag
2020/02/13 08:09:08 -07:00 unknown header tag
2020/02/13 08:09:08 -07:00 unknown header tag
2020/02/13 08:09:08 -07:00 unknown header tag
2020/02/13 08:09:08 -07:00 unknown header tag
2020/02/13 08:09:08 -07:00 unknown header tag
2020/02/13 08:09:08 -07:00 unknown header tag
2020/02/13 08:09:08 -07:00 Importing packages finished.
2020/02/13 08:09:08 -07:00
2020/02/13 08:09:08 -07:00 Errata in repo: 0.
2020/02/13 08:09:08 -07:00 Sync of channel completed in 0:00:06.
-Contents of rhn_taskomatic_daemon.log:
INFO | jvm 1 | 2020/02/13 08:09:01 | 2020-02-13 08:09:01,753
[DefaultQuartzScheduler_Worker-3] INFO
com.redhat.rhn.taskomatic.task.RepoSyncTask - Syncing repos for channel: <MY
CHANNEL NAME>
INFO | jvm 1 | 2020/02/13 08:09:09 | 2020-02-13 08:09:09,029 [Thread-51]
INFO com.redhat.rhn.taskomatic.task.RepoSyncTask - 08:09:02
======================================
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:02 | Channel: <MY CHANNEL NAME>
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:02
======================================
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:02 Sync of channel started.
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:02
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:02 Processing repository with
URL: http://<MYREPOSERVER>/rhel/x86_64/<MYREPOFOLDER/
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:03 Packages in repo:
111
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:04 Packages already synced:
65
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:04 Packages to sync:
14
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:04 New packages to
download: 14
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:04 Downloading packages:
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:04 1/14 : <PACKAGENAME>.rpm
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:04 2/14 : <PACKAGENAME>.rpm
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:04 3/14 : <PACKAGENAME>.rpm
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:04 4/14 : <PACKAGENAME>.rpm
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:05 5/14 : <PACKAGENAME>.rpm
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:05 6/14 : <PACKAGENAME>.rpm
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:06 7/14 : <PACKAGENAME>.rpm
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:06 8/14 : <PACKAGENAME>.rpm
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:06 9/14 : <PACKAGENAME>.rpm
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:07 10/14 : <PACKAGENAME>.rpm
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:07 11/14 : <PACKAGENAME>.rpm
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:08 12/14 : <PACKAGENAME>.rpm
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:08 13/14 : <PACKAGENAME>.rpm
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:08 14/14 : <PACKAGENAME>.rpm
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:08
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:08 Importing packages to DB:
INFO | jvm 1 | 2020/02/13 08:09:09 |
INFO | jvm 1 | 2020/02/13 08:09:09 | Importing packages:
|####----------------------------------------------| 7.14%
INFO | jvm 1 | 2020/02/13 08:09:09 | Importing packages:
|#######-------------------------------------------| 14.29%
INFO | jvm 1 | 2020/02/13 08:09:09 | Importing packages:
|###########---------------------------------------| 21.43%
INFO | jvm 1 | 2020/02/13 08:09:09 | Importing packages:
|##############------------------------------------| 28.57%
INFO | jvm 1 | 2020/02/13 08:09:09 | Importing packages:
|##################--------------------------------| 35.71%
INFO | jvm 1 | 2020/02/13 08:09:09 | Importing packages:
|#####################-----------------------------| 42.86%
INFO | jvm 1 | 2020/02/13 08:09:09 | Importing packages:
|#########################-------------------------| 50.0%
INFO | jvm 1 | 2020/02/13 08:09:09 | Importing packages:
|#############################---------------------| 57.14%
INFO | jvm 1 | 2020/02/13 08:09:09 | Importing packages:
|################################------------------| 64.29%
INFO | jvm 1 | 2020/02/13 08:09:09 | Importing packages:
|####################################--------------| 71.43%
INFO | jvm 1 | 2020/02/13 08:09:09 | Importing packages:
|#######################################-----------| 78.57%
INFO | jvm 1 | 2020/02/13 08:09:09 | Importing packages:
|###########################################-------| 85.71%
INFO | jvm 1 | 2020/02/13 08:09:09 | Importing packages:
|##############################################----| 92.86%
INFO | jvm 1 | 2020/02/13 08:09:09 | Importing packages:
|##################################################| 100.0%
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:08
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:08 Errata in repo: 0.
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:08 Sync of channel completed in
0:00:06.
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:08 Total time: 0:00:06
INFO | jvm 1 | 2020/02/13 08:09:09 |
INFO | jvm 1 | 2020/02/13 08:09:09 | 2020-02-13 08:09:09,034
[DefaultQuartzScheduler_Worker-3] ERROR
com.redhat.rhn.taskomatic.task.RepoSyncTask - Executing a task threw an
exception: org.quartz.JobExecutionException
INFO | jvm 1 | 2020/02/13 08:09:09 | 2020-02-13 08:09:09,034
[DefaultQuartzScheduler_Worker-3] ERROR
com.redhat.rhn.taskomatic.task.RepoSyncTask - Message: Command
'[/usr/bin/spacewalk-repo-sync, --channel, <MY CHANNEL NAME>, --type, yum]'
exited with error code 14
INFO | jvm 1 | 2020/02/13 08:09:09 | 2020-02-13 08:09:09,034
[DefaultQuartzScheduler_Worker-3] ERROR
com.redhat.rhn.taskomatic.task.RepoSyncTask - Cause: null
INFO | jvm 1 | 2020/02/13 08:09:09 | 2020-02-13 08:09:09,050
[DefaultQuartzScheduler_Worker-3] ERROR
com.redhat.rhn.taskomatic.task.RepoSyncTask - Stack
trace:org.quartz.JobExecutionException: Command '[/usr/bin/spacewalk-repo-sync,
--channel, <MY CHANNEL NAME>, --type, yum]' exited with error code 14
INFO | jvm 1 | 2020/02/13 08:09:09 | at
com.redhat.rhn.taskomatic.task.RhnJavaJob.executeExtCmd(RhnJavaJob.java:103)
INFO | jvm 1 | 2020/02/13 08:09:09 | at
com.redhat.rhn.taskomatic.task.RepoSyncTask.execute(RepoSyncTask.java:70)
INFO | jvm 1 | 2020/02/13 08:09:09 | at
com.redhat.rhn.taskomatic.task.RhnJavaJob.execute(RhnJavaJob.java:88)
INFO | jvm 1 | 2020/02/13 08:09:09 | at
com.redhat.rhn.taskomatic.TaskoJob.execute(TaskoJob.java:186)
INFO | jvm 1 | 2020/02/13 08:09:09 | at
org.quartz.core.JobRunShell.run(JobRunShell.java:216)
INFO | jvm 1 | 2020/02/13 08:09:09 | at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
Jody McIvor
Sr. Systems Administrator, Integration
Les Services D'intégration
[TEL] (250) 852-5202
[email protected]
BCLC.com
-----Original Message-----
From: [email protected] <[email protected]> On
Behalf Of [email protected]
Sent: February 12, 2020 9:00 AM
To: [email protected]
Subject: Spacewalk-list Digest, Vol 141, Issue 14
Send Spacewalk-list mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
https://www.redhat.com/mailman/listinfo/spacewalk-list
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific than "Re:
Contents of Spacewalk-list digest..."
Today's Topics:
1. Re: Spacewalk 2.10 Failing to sync repo (Michael Mraka)
----------------------------------------------------------------------
Message: 1
Date: Wed, 12 Feb 2020 16:18:40 +0100
From: Michael Mraka <[email protected]>
To: [email protected]
Subject: Re: [Spacewalk-list] Spacewalk 2.10 Failing to sync repo
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8
Ree, Jan-Albert van:
> Good afternoon,
> While testing spacewalk-nightly we saw the same behaviour :
>
> - packages get downloaded just fine from remote repositories
> - building local repo data fails however
>
> Scenario's tested were both a clean install of spacewalk-nightly and an
> upgrade of 2.9 to nightly. With the upgrade from 2.9 to nightly, package sync
> was working fine before upgrading to nightly so it's definately something
> wrong in the nightly repo creation.
>
> Hope this helps, willing to test/debug further but need some pointers
> on what to test/report
Hello,
Most likely it's a bug https://bugzilla.redhat.com/show_bug.cgi?id=1802135.
I've fixed it today and updated backend packages are alredy in nightly repo.
Regards,
> --
> Jan-Albert van Ree
>
>
> ________________________________________
> From: [email protected]
> <[email protected]> on behalf of Jody McIvor
> <[email protected]>
> Sent: Monday, January 13, 2020 17:22
> To: [email protected]
> Subject: [Spacewalk-list] Spacewalk 2.10 Failing to sync repo
>
> Hi Guys,
>
> Our DEV environment instance of spacewalk 2.10 seems to be failing to sync
> repositories. Running it via SSH reports success, and running it via HTTP
> interface reports success, yet logs show otherwise. As in the log snippet
> below, it is finding the packages, downloading them (seemingly) but failing
> to update the software channel(s).
...
> V- This issue seems to be affecting ALL software channels
> - Original repository location likely ruled out as the problem, as
> other repo's hosted on same machine, using same createrepo process are
> all working fine
> - Spacewalk shows proper "Last Modified" date, but "Last Repo Build"
> shows as way outdated. (Channels>Select Channel>DetailsTab)
> - Adding original repository location on the client (Effectively
> bypassing spacewalk) succeeds and allows access to all the latest RPMs
>
> Taskomatic log after attempting sync:
> INFO | jvm 1 | 2020/01/13 08:03:30 | 2020-01-13 08:03:30,760 [Thread-53]
> INFO com.redhat.rhn.taskomatic.task.RepoSyncTask - 08:03:29
> ======================================
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:29 | Channel: ? ? ?
> ?-channel-dev
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:29
> ======================================
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:29 Sync of channel started.
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:29
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:29 Processing repository
> with URL: http:// ? ? ? ?/rhel/x86_64/? ? ? ?-dev/
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:29 Packages in repo:
> 88
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:29 Packages already
> synced: 66
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:29 Packages to sync:
> 5
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:29 New packages to
> download: 5
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:29 Downloading packages:
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:29 1/5 : ? ? ?
> ?-1.02-4.x86_64.rpm
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:29 2/5 : ? ? ?
> ?-2.11.03.3.rpm
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:29 3/5 : ? ? ?
> ?-2.13.01.95.rpm
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:29 4/5 : ? ? ?
> ?-v2.13.03-1-1.x86_64.rpm
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:29 5/5 : ? ? ?
> ?-v1.03.01-1-1.x86_64.rpm
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:30
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:30 Importing packages to DB:
> INFO | jvm 1 | 2020/01/13 08:03:30 |
> INFO | jvm 1 | 2020/01/13 08:03:30 | Importing packages:
> |##########----------------------------------------| 20.0%
> INFO | jvm 1 | 2020/01/13 08:03:30 | Importing packages:
> |####################------------------------------| 40.0%
> INFO | jvm 1 | 2020/01/13 08:03:30 | Importing packages:
> |##############################--------------------| 60.0%
> INFO | jvm 1 | 2020/01/13 08:03:30 | Importing packages:
> |########################################----------| 80.0%
> INFO | jvm 1 | 2020/01/13 08:03:30 | Importing packages:
> |##################################################| 100.0%
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:30
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:30 Errata in repo: 0.
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:30 Sync of channel completed
> in 0:00:01.
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:30 Total time: 0:00:01
> INFO | jvm 1 | 2020/01/13 08:03:30 |
> INFO | jvm 1 | 2020/01/13 08:03:30 | 2020-01-13 08:03:30,763
> [DefaultQuartzScheduler_Worker-9] ERROR
> com.redhat.rhn.taskomatic.task.RepoSyncTask - Executing a task threw an
> exception: org.quartz.JobExecutionException
> INFO | jvm 1 | 2020/01/13 08:03:30 | 2020-01-13 08:03:30,763
> [DefaultQuartzScheduler_Worker-9] ERROR
> com.redhat.rhn.taskomatic.task.RepoSyncTask - Message: Command
> '[/usr/bin/spacewalk-repo-sync, --channel, ? ? ? ?-channel-dev, --type, yum]'
> exited with error code 5
> INFO | jvm 1 | 2020/01/13 08:03:30 | 2020-01-13 08:03:30,763
> [DefaultQuartzScheduler_Worker-9] ERROR
> com.redhat.rhn.taskomatic.task.RepoSyncTask - Cause: null
> INFO | jvm 1 | 2020/01/13 08:03:30 | 2020-01-13 08:03:30,763
> [DefaultQuartzScheduler_Worker-9] ERROR
> com.redhat.rhn.taskomatic.task.RepoSyncTask - Stack
> trace:org.quartz.JobExecutionException: Command
> '[/usr/bin/spacewalk-repo-sync, --channel, ? ? ? ?-channel-dev, --type, yum]'
> exited with error code 5
> INFO | jvm 1 | 2020/01/13 08:03:30 | at
> com.redhat.rhn.taskomatic.task.RhnJavaJob.executeExtCmd(RhnJavaJob.java:103)
> INFO | jvm 1 | 2020/01/13 08:03:30 | at
> com.redhat.rhn.taskomatic.task.RepoSyncTask.execute(RepoSyncTask.java:70)
> INFO | jvm 1 | 2020/01/13 08:03:30 | at
> com.redhat.rhn.taskomatic.task.RhnJavaJob.execute(RhnJavaJob.java:88)
> INFO | jvm 1 | 2020/01/13 08:03:30 | at
> com.redhat.rhn.taskomatic.TaskoJob.execute(TaskoJob.java:186)
> INFO | jvm 1 | 2020/01/13 08:03:30 | at
> org.quartz.core.JobRunShell.run(JobRunShell.java:216)
> INFO | jvm 1 | 2020/01/13 08:03:30 | at
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
>
>
> Been poking at this one for a while now, any help appreciated!
>
> Thanks,
>
> Jody McIvor
> Sr. Systems Administrator, Integration Les Services D?int?gration
> [TEL] (250) 852-5202
>
> _______________________________________________
> Spacewalk-list mailing list
> [email protected]
> https://www.redhat.com/mailman/listinfo/spacewalk-list
--
Michael Mr?ka
System Management Engineering, Red Hat
------------------------------
_______________________________________________
Spacewalk-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/spacewalk-list
End of Spacewalk-list Digest, Vol 141, Issue 14
***********************************************
________________________________
This email is intended only for the addressee. It may contain confidential or
proprietary information that cannot be disclosed without BCLC's permission. If
you have received this email in error, please notify the sender immediately and
delete the email.
_______________________________________________
Spacewalk-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/spacewalk-list
