On 05.07.18 16:05, Matt Moldvan wrote:
How is the server utilization with respect to disk I/O (something like
iotop or htop might help here)? Maybe there is something else blocking
My server is basically idle. 99% idle, little disk i/o. It doesn't do anything
really.
and the server doesn't have enough resources to complete. Have you
tried running an strace against the running process?
If it doesn't have enough resources shouldn't there be an exception?
For me, it looks more like something doesn't make it into the database and thus into the
persistent state. For instance, I now have the repodata task at "RUNNING" for
three days:
Channel Repodata: 2018-07-02 08:13:10 CEST RUNNING
The log file shows this regarding repodata:
# fgrep -i repodata rhn_taskomatic_daemon.log
INFO | jvm 1 | 2018/07/02 08:13:10 | 2018-07-02 08:13:10,584 [Thread-12] INFO com.redhat.rhn.taskomatic.TaskoQuartzHelper - Job single-channel-repodata-bunch-0 scheduled succesfully.
INFO | jvm 1 | 2018/07/02 08:13:10 | 2018-07-02 08:13:10,636
[DefaultQuartzScheduler_Worker-8] INFO com.redhat.rhn.taskomatic.TaskoJob -
single-channel-repodata-bunch-0: bunch channel-repodata-bunch STARTED
INFO | jvm 1 | 2018/07/02 08:13:10 | 2018-07-02 08:13:10,651
[DefaultQuartzScheduler_Worker-8] DEBUG com.redhat.rhn.taskomatic.TaskoJob -
single-channel-repodata-bunch-0: task channel-repodata started
INFO | jvm 1 | 2018/07/02 08:13:10 | 2018-07-02 08:13:10,793
[DefaultQuartzScheduler_Worker-8] INFO
com.redhat.rhn.taskomatic.task.ChannelRepodata - In the queue: 4
INFO | jvm 1 | 2018/07/02 08:13:11 | 2018-07-02 08:13:11,102
[DefaultQuartzScheduler_Worker-8] DEBUG com.redhat.rhn.taskomatic.TaskoJob -
channel-repodata (single-channel-repodata-bunch-0) ... running
INFO | jvm 1 | 2018/07/02 08:13:11 | 2018-07-02 08:13:11,103
[DefaultQuartzScheduler_Worker-8] INFO com.redhat.rhn.taskomatic.TaskoJob -
single-channel-repodata-bunch-0: bunch channel-repodata-bunch FINISHED
So according to the logs the repodata bunch has finished. According to the web
interface it has not. Nothing has been updated in /var/cache/rhn/repodata/
either. In addition, those four channels which were still updated haven't been
updated either now.
Thanks,
Gerald
I also had an (well, many) issue(s) with our Spacewalk server before
disabling snapshots in /etc/rhn/rhn.conf. I also increased the number
of workers and max repodata work items:
# system snapshots enabled
enable_snapshots = 0
...
taskomatic.maxmemory=6144
taskomatic.errata_cache_max_work_items = 500
taskomatic.channel_repodata_max_work_items = 50
taskomatic.channel_repodata_workers = 5
On Thu, Jul 5, 2018 at 4:38 AM Florence Savary
<[email protected] <mailto:[email protected]>> wrote:
Hello,
Thanks for sharing your configuration files. They differ very little
from mine. I just changed the number of workers in rhn.conf, but it
didn't change anything.
I deleted all the channels clones not used by any system and dating
back from before May 2018, in order to lower the number of channels
in the queue. There were 127 channels in the queue before these
deletion (indicated in /var/log/rhn/rhn_taskomatic_daemon.log), and
there are 361 of them now ... I must admit I'm confused... I hoped
it would reduce the number of channels to process and thus "help"
taskomatic, but obviously I was wrong.
I also noticed that the repodata regeneration seems to work fine for
existing channels that are not clones, but it is not working for new
channels that are not clones (and not working for new clones but
nothing new here).
Has anyone got any other idea (even the tiniest) ?
Regards,
Florence
2018-07-04 15:21 GMT+02:00 Paul Dias - BCX <[email protected]
<mailto:[email protected]>>:
Hi,____
__ __
Let me post my settings that I have on my CentOS6 server. Can’t
remember but I have one or two others, but his is from the top
of my head.____
__ __
/etc/rhn/rhn.conf____
# Added by paul dias increase number of taskomatic workers
20180620____
taskomatic.channel_repodata_workers = 3____
taskomatic.java.maxmemory=4096____
__ __
/etc/sysconfig/tomcat6____
JAVA_OPTS="-ea -Xms256m -Xmx512m -Djava.awt.headless=true
-Dorg.xml.sax.driver=org.apache.xerces.parsers.SAXParser
-Dorg.apache.tomcat.util.http.Parameters.MAX_COUNT=1024
-XX:MaxNewSize=256 -XX:-UseConcMarkSweepGC
-Dnet.sf.ehcache.skipUpdateCheck=true
-Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory"____
__ __
/etc/tomcat/server.xml____
<!-- Define an AJP 1.3 Connector on port 8009 -->____
<Connector port="8009" protocol="AJP/1.3"
redirectPort="8443" URIEncoding="UTF-8" address="127.0.0.1"
maxThreads="256" connectionTimeout="20000"/>____
__ __
<Connector port="8009" protocol="AJP/1.3"
redirectPort="8443" URIEncoding="UTF-8" address="::1"
maxThreads="256" connectionTimeout="20000"/>____
__ __
/usr/share/rhn/config-defaults/rhn_taskomatic_daemon.conf____
# Initial Java Heap Size (in MB)____
wrapper.java.initmemory=512____
__ __
# Maximum Java Heap Size (in MB)____
wrapper.java.maxmemory=1512____
# Adjusted by paul 20180620____
__ __
wrapper.ping.timeout=0____
# # adjusted paul dias 20180620____
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
Regards,____
*Paul Dias____*
Technical Consultant____
6^th Floor, 8 Boundary Road____
Newlands____
Cape Town____
7700____
T: +27 (0) 21 681 3149 <tel:+27%2021%20681%203149>____
*Meet your future today.____*
*__ __*
__BCX______
__ __
__ __
__ __
__Social-facebook
<https://www.facebook.com/BCXworld>____Social-twitter
<https://twitter.com/BCXworld>____Social-linkdin
<https://za.linkedin.com/BCX>____Social-youtube
<https://www.youtube.com/BCXworld>______
__ __
__ __
This e-mail is subject to the BCX electronic communication legal
notice, available at:
https://www.bcx.co.za/disclaimers____
/__ __/
/__ __/
__ __
*From:*Paul Dias - BCX
*Sent:* 02 July 2018 06:53 PM
*To:* [email protected] <mailto:[email protected]>
*Subject:* Re: [Spacewalk-list] Taskomatic runs indefinitely
without ever generating repodata____
__ __
What I have noticed, if you use
"spacecmd softchannel_generateyumcache <channel name>" and then
go to tasks and run single repodata bunch, you will see it will
actually start and generate your channel cache for you on the
channel you used the spacecmd on, this works every time.____
__ __
But yes the task logs just show repodata bunch running forever.____
__ __
Regards,____
*Paul Dias*____
6^th Floor, 8 Boundary Road____
Newlands____
Cape Town____
7700____
T: +27 (0) 21 681 3149 <tel:+27%2021%20681%203149>____
__ __
*Meet your future today.*____
**____
BCX____
__ __
------------------------------------------------------------------------
*From:*Gerald Vogt <[email protected] <mailto:[email protected]>>
*Sent:* Monday, 02 July 2018 9:45 AM
*To:* [email protected] <mailto:[email protected]>
*Subject:* Re: [Spacewalk-list] Taskomatic runs indefinitely
without ever generating repodata____
____
After letting the upgraded server sit for a while it seems only
a few of
the task schedules actually finish. By now, only those tasks
show up in
in the task engine status page:
Changelog Cleanup: 2018-07-01 23:00:00 CEST FINISHED
Clean Log History: 2018-07-01 23:00:00 CEST FINISHED
Compare Config Files: 2018-07-01 23:00:00 CEST FINISHED
Daily Summary Mail: 2018-07-01 23:00:00 CEST FINISHED
Daily Summary Queue: 2018-07-01 23:00:00 CEST FINISHED
All the other tasks have disappeared from the list by now.
The repo-sync tasks seem to work. New packages appear in the
channel.
However, the repo build is not running or better it seems to never
properly finish.
If I start it manually, it seems to do its work:
> INFO | jvm 1 | 2018/07/02 08:13:10 | 2018-07-02 08:13:10,584
[Thread-12] INFO com.redhat.rhn.taskomatic.TaskoQuartzHelper - Job
single-channel-repodata-bunch-0 scheduled succesfully.
> INFO | jvm 1 | 2018/07/02 08:13:10 | 2018-07-02 08:13:10,636
[DefaultQuartzScheduler_Worker-8] INFO com.redhat.rhn.taskomatic.TaskoJob -
single-channel-repodata-bunch-0: bunch channel-repodata-bunch STARTED
> INFO | jvm 1 | 2018/07/02 08:13:10 | 2018-07-02 08:13:10,651
[DefaultQuartzScheduler_Worker-8] DEBUG com.redhat.rhn.taskomatic.TaskoJob -
single-channel-repodata-bunch-0: task channel-repodata started
> INFO | jvm 1 | 2018/07/02 08:13:10 | 2018-07-02 08:13:10,793
[DefaultQuartzScheduler_Worker-8] INFO
com.redhat.rhn.taskomatic.task.ChannelRepodata - In the queue: 4
> INFO | jvm 1 | 2018/07/02 08:13:11 | 2018-07-02 08:13:11,102
[DefaultQuartzScheduler_Worker-8] DEBUG com.redhat.rhn.taskomatic.TaskoJob -
channel-repodata (single-channel-repodata-bunch-0) ... running
> INFO | jvm 1 | 2018/07/02 08:13:11 | 2018-07-02 08:13:11,103
[DefaultQuartzScheduler_Worker-8] INFO com.redhat.rhn.taskomatic.TaskoJob -
single-channel-repodata-bunch-0: bunch channel-repodata-bunch FINISHED
> INFO | jvm 1 | 2018/07/02 08:13:11 | 2018-07-02 08:13:11,137
[Thread-677] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - File
Modified Date:2018-06-23 03:48:50 CEST
> INFO | jvm 1 | 2018/07/02 08:13:11 | 2018-07-02 08:13:11,137
[Thread-677] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter -
Channel Modified Date:2018-07-02 03:45:39 CEST
> INFO | jvm 1 | 2018/07/02 08:13:11 | 2018-07-02 08:13:11,211
[Thread-678] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - File
Modified Date:2018-06-23 04:09:51 CEST
> INFO | jvm 1 | 2018/07/02 08:13:11 | 2018-07-02 08:13:11,213
[Thread-678] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter -
Channel Modified Date:2018-07-02 03:47:55 CEST
> INFO | jvm 1 | 2018/07/02 08:13:19 | 2018-07-02 08:13:19,062
[Thread-677] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter -
Generating new repository metadata for channel 'epel6-centos6-x86_64'(sha1) 14401
packages, 11613 errata
> INFO | jvm 1 | 2018/07/02 08:13:21 | 2018-07-02 08:13:21,193
[Thread-678] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter -
Generating new repository metadata for channel 'epel7-centos7-x86_64'(sha1) 16282
packages, 10176 errata
> INFO | jvm 1 | 2018/07/02 08:40:12 | 2018-07-02 08:40:12,351
[Thread-677] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter -
Repository metadata generation for 'epel6-centos6-x86_64' finished in 1613 seconds
> INFO | jvm 1 | 2018/07/02 08:40:12 | 2018-07-02 08:40:12,457
[Thread-677] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - File
Modified Date:2018-06-19 06:28:57 CEST
> INFO | jvm 1 | 2018/07/02 08:40:12 | 2018-07-02 08:40:12,457
[Thread-677] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter -
Channel Modified Date:2018-07-02 04:30:05 CEST
> INFO | jvm 1 | 2018/07/02 08:40:12 | 2018-07-02 08:40:12,691
[Thread-677] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter -
Generating new repository metadata for channel
'postgresql96-centos7-x86_64'(sha256) 1032 packages, 0 errata
> INFO | jvm 1 | 2018/07/02 08:41:51 | 2018-07-02 08:41:51,710
[Thread-677] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter -
Repository metadata generation for 'postgresql96-centos7-x86_64' finished in 98
seconds
> INFO | jvm 1 | 2018/07/02 08:41:51 | 2018-07-02 08:41:51,803
[Thread-677] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - File
Modified Date:2018-06-20 05:08:38 CEST
> INFO | jvm 1 | 2018/07/02 08:41:51 | 2018-07-02 08:41:51,803
[Thread-677] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter -
Channel Modified Date:2018-07-02 04:00:00 CEST
> INFO | jvm 1 | 2018/07/02 08:41:51 | 2018-07-02 08:41:51,923
[Thread-677] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter -
Generating new repository metadata for channel
'postgresql10-centos6-x86_64'(sha512) 436 packages, 0 errata
> INFO | jvm 1 | 2018/07/02 08:42:26 | 2018-07-02 08:42:26,479
[Thread-677] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter -
Repository metadata generation for 'postgresql10-centos6-x86_64' finished in 34
seconds
> INFO | jvm 1 | 2018/07/02 08:45:01 | 2018-07-02 08:45:01,697
[Thread-678] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter -
Repository metadata generation for 'epel7-centos7-x86_64' finished in 1900 seconds
yet, the task remains in RUNNING. And for whatever reason it
only seems
to work some channels. I find a total of 20 repos syncing in the
logs of
the updated server compared to 42 repos syncing in the logs of
the old.
I don't really see the difference between those 20 repos syncing
and
those other 22 not. First I suspected channels with custom quartz
schedules, but then I found channels in both groups.
So I don't know how to troubleshoot this any further. The
repodata task
which I have started 1,5 hours ago is still at "RUNNING". The
channels
for which the sync works have been updated. I don't know why it
is still
running. Server load is back down...
Thanks,
Gerald
On 22.06.18 19:12, Gerald Vogt wrote:
> I have the same problem after upgrading from 2.6 to 2.8 on CentOS 6.9. I
> have even increased the memory as suggested by that link but it makes no
> differences. None of the scheduled tasks are running. I can run a bunch
> manually. But the scheduler doesn't seem to work. Last execution times
> on the task engine status pages are still at timestamps from before the
> upgrade. -Gerald
>
>
>
> On 22.06.18 14:15, Avi Miller wrote:
>> Hi,
>>
>>> On 22 Jun 2018, at 5:51 pm, Florence Savary
>>> <[email protected]
<mailto:[email protected]>> wrote:
>>>
>>> When using taskotop, we can see a line for the channel-repodata task,
>>> we see it is running, but there is never any channel displayed in the
>>> Channel column. We can also see the task marked as running in the
>>> Admin tab of the WebUI, but if we let it, it never stops. The task
>>> runs indefinitely, whithout ever doing anything.
>>
>> If you've never modified the default memory settings, Taskomatic is
>> probably running out of memory and task is crashing. This is a known
>> issue, particularly when you sync large repos.
>>
>> I would suggest increasing the memory assigned to Taskomatic to see if
>> that resolves the issue. You will need to restart it after making
>> these changes:
>> https://docs.oracle.com/cd/E92593_01/E90695/html/swk24-issues-memory.html
>>
>> Cheers,
>> Avi
>>
>> --
>> Oracle <http://www.oracle.com>
>> Avi Miller | Product Management Director | +61 (3) 8616 3496
<tel:+61%203%208616%203496>
>> Oracle Linux and Virtualization
>> 417 St Kilda Road, Melbourne, Victoria 3004 Australia
>>
>>
>> _______________________________________________
>> Spacewalk-list mailing list
>> [email protected] <mailto:[email protected]>
>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>>
>
> _______________________________________________
> Spacewalk-list mailing list
> [email protected] <mailto:[email protected]>
> https://www.redhat.com/mailman/listinfo/spacewalk-list
____
_______________________________________________
Spacewalk-list mailing list
[email protected] <mailto:[email protected]>
https://www.redhat.com/mailman/listinfo/spacewalk-list
_______________________________________________
Spacewalk-list mailing list
[email protected] <mailto:[email protected]>
https://www.redhat.com/mailman/listinfo/spacewalk-list
_______________________________________________
Spacewalk-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/spacewalk-list
_______________________________________________
Spacewalk-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/spacewalk-list
