On 08/03/2016 07:41 AM, Rick Langston wrote:
> Things seem to stem from pulp and a connection
cool, this narrows it down some:)
Let's try this to see if it captures the issue:
# stop everything
katello-service stop
# start mongodb
systemctl start mongod
# become qpidd user
sudo su - qpidd -s /bin/bash (to become qpid user)
#start qpid with trace output
/usr/sbin/qpidd --config /etc/qpid/qpidd.conf -t 2>&1 | tee /tmp/qpid.out
in another window:
systemctl start pulp_celerybeat
systemctl start pulp_resource_manager
systemctl start pulp_workers
<wait a minute or so>
then stop the three pulp services, ^C to exit qpidd, and do a
"katello-service start" to spin everything up again.
Hopefully this will reproduce the issue, and we'll have the full qpid
log in /tmp/qpid.out to see what happened. The log is at trace setting
and will be extremely verbose.
>
> [root@servername ~]# /bin/systemctl status pulp_celerybeat.service
> ● pulp_celerybeat.service - Pulp's Celerybeat
> Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
> enabled; vendor preset: disabled)
> Active: inactive (dead) since Wed 2016-08-03 06:32:14 CDT; 1s ago
> Process: 5388 ExecStart=/usr/bin/celery beat
> --app=pulp.server.async.celery_instance.celery
> --scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
> status=0/SUCCESS)
> Main PID: 5388 (code=exited, status=0/SUCCESS)
>
> Aug 03 06:32:13 servername.corp.company.net pulp[5388]:
> celery.beat:CRITICAL: (5388-64256) Timeout: Connection attach timed out
> Aug 03 06:32:13 servername.corp.company.net celery[5388]: celery beat
> v3.1.11 (Cipater) is starting.
> Aug 03 06:32:13 servername.corp.company.net celery[5388]: __ - ... __
> - _
> Aug 03 06:32:13 servername.corp.company.net celery[5388]: Configuration ->
> Aug 03 06:32:13 servername.corp.company.net celery[5388]: . broker ->
> qpid://servername.corp.company.net:5671//
> Aug 03 06:32:13 servername.corp.company.net celery[5388]: . loader ->
> celery.loaders.app.AppLoader
> Aug 03 06:32:13 servername.corp.company.net celery[5388]: . scheduler ->
> pulp.server.async.scheduler.Scheduler
> Aug 03 06:32:13 servername.corp.company.net celery[5388]: . logfile ->
> [stderr]@%INFO
> Aug 03 06:32:13 servername.corp.company.net celery[5388]: . maxinterval ->
> now (0s)
> Aug 03 06:32:14 servername.corp.company.net pulp[5388]:
> qpid.messaging:DEBUG: poll(1 fds, timeout=4000.0)
>
> On Tuesday, August 2, 2016 at 10:12:41 AM UTC-5, Chris Duryee wrote:
>>
>>
>>
>> On 08/02/2016 10:23 AM, Rick Langston wrote:
>>>
>>>
>>> On Tuesday, August 2, 2016 at 7:30:07 AM UTC-5, Chris Duryee wrote:
>>>>
>>>> and it will eventually die as it did last night.
>>>>
>>>
>>> Where would I send the debug file?
>>
>> I guess the easiest way is to just email me direct with the attachment,
>> but we can keep the discussion on-list.
>>
>>>
>>>
>>>>
>>>> On 08/01/2016 04:14 PM, Rick Langston wrote:
>>>>> Strange, If I restart the service "/bin/systemctl status
>>>>> pulp_celerybeat.service" it will eventually start. Any ideas?
>>>>
>>>> I'm glad it eventually restarts:) that is strange though that fails at
>>>> first but eventually works.
>>>>
>>>> Can you send the output of foreman-debug? This will create an an
>> archive
>>>> with system information and log files that we can peruse to debug
>> further.
>>>>
>>>>>
>>>>>
>>>>> On Monday, August 1, 2016 at 12:19:27 PM UTC-5, Chris Duryee wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 08/01/2016 10:52 AM, Chris Duryee wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 08/01/2016 09:49 AM, Rick Langston wrote:
>>>>>>>> Thank you for help on this
>>>>>>>>
>>>>>>>> yes here is the actual capture
>>>>>>>>
>>>>>>>
>>>>>>> Thanks for the capture of the logs.
>>>>>>>
>>>>>>> What is the output of systemctl status qpidd.service? I am curious
>> if
>>>>>>> only celerybeat is unable to connect, or if there is some other
>> issue.
>>>>>>
>>>>>> one more thing to check, as root please run:
>>>>>>
>>>>>> qpid-stat --ssl-certificate /etc/pki/katello/certs/java-client.crt
>>>>>> --ssl-key /etc/pki/katello/private/java-client.key -b
>>>>>> "amqps://$(hostname -f):5671" -q
>>>>>>
>>>>>> If qpid is accepting connections, it should output some stuff about
>>>>>> queues.
>>>>>>
>>>>>>>
>>>>>>> If you want to restart qpidd, the best way is to just do a
>>>>>>> katello-service stop. A number of other services depend on qpid, so
>>>> it's
>>>>>>> easiest to just down everything, ensure qpidd is stopped, then bring
>>>> it
>>>>>>> all back up via the script.
>>>>>>>
>>>>>>>> command line output
>>>>>>>> [root@dscaprv01 tmp]# systemctl restart pulp_celerybeat.service
>>>>>>>> [root@dscaprv01 tmp]# systemctl status pulp_celerybeat.service
>>>>>>>> ● pulp_celerybeat.service - Pulp's Celerybeat
>>>>>>>> Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
>>>>>>>> enabled; vendor preset: disabled)
>>>>>>>> Active: active (running) since Mon 2016-08-01 08:34:20 CDT; 7s
>> ago
>>>>>>>> Main PID: 5887 (celery)
>>>>>>>> CGroup: /system.slice/pulp_celerybeat.service
>>>>>>>> └─5887 /usr/bin/python /usr/bin/celery beat
>>>>>>>> --app=pulp.server.async.celery_instance.celery
>>>>>>>> --scheduler=pulp.server.async.scheduler.Scheduler
>>>>>>>>
>>>>>>>> Aug 01 08:34:20 dscaprv01.corp.acxiom.net systemd[1]: Started
>> Pulp's
>>>>>>>> Celerybeat.
>>>>>>>> Aug 01 08:34:20 dscaprv01.corp.acxiom.net systemd[1]: Starting
>>>> Pulp's
>>>>>>>> Celerybeat...
>>>>>>>> Aug 01 08:34:25 dscaprv01.corp.acxiom.net pulp[5887]:
>>>>>> celery.beat:INFO:
>>>>>>>> beat: Starting...
>>>>>>>> Aug 01 08:34:25 dscaprv01.corp.acxiom.net pulp[5887]:
>>>>>>>> pulp.server.db.connection:INFO: Attempting to connect to
>>>>>> localhost:27017
>>>>>>>> Aug 01 08:34:25 dscaprv01.corp.acxiom.net pulp[5887]:
>>>>>>>> pulp.server.async.scheduler:INFO: Worker Timeout Monitor Started
>>>>>>>> Aug 01 08:34:25 dscaprv01.corp.acxiom.net pulp[5887]:
>>>>>>>> pulp.server.db.connection:INFO: Attempting to connect to
>>>>>> localhost:27017
>>>>>>>> Aug 01 08:34:26 dscaprv01.corp.acxiom.net pulp[5887]:
>>>>>>>> pulp.server.db.connection:INFO: Write concern for Mongo connection:
>>>> {}
>>>>>>>> [root@dscaprv01 tmp]# systemctl status pulp_celerybeat.service
>>>>>>>> ● pulp_celerybeat.service - Pulp's Celerybeat
>>>>>>>> Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
>>>>>>>> enabled; vendor preset: disabled)
>>>>>>>> Active: inactive (dead) since Mon 2016-08-01 08:34:31 CDT; 18s
>> ago
>>>>>>>> Process: 5887 ExecStart=/usr/bin/celery beat
>>>>>>>> --app=pulp.server.async.celery_instance.celery
>>>>>>>> --scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
>>>>>>>> status=0/SUCCESS)
>>>>>>>> Main PID: 5887 (code=exited, status=0/SUCCESS)
>>>>>>>>
>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net pulp[5887]:
>>>>>> celery.beat:CRITICAL:
>>>>>>>> (5887-79264) raise Timeout("Connection attach timed out")
>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net pulp[5887]:
>>>>>> celery.beat:CRITICAL:
>>>>>>>> (5887-79264) Timeout: Connection attach timed out
>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: celery
>> beat
>>>>>> v3.1.11
>>>>>>>> (Cipater) is starting.
>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: __ -
>>>> ...
>>>>>> __
>>>>>>>> - _
>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]:
>>>> Configuration
>>>>>> ->
>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . broker
>> ->
>>>>>>>> qpid://dscaprv01.corp.acxiom.net:5671//
>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . loader
>> ->
>>>>>>>> celery.loaders.app.AppLoader
>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
>> scheduler
>>>> ->
>>>>>>>> pulp.server.async.scheduler.Scheduler
>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . logfile
>> ->
>>>>>>>> [stderr]@%INFO
>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
>>>> maxinterval
>>>>>> ->
>>>>>>>> now (0s)
>>>>>>>>
>>>>>>>> /var/log/messages output
>>>>>>>>
>>>>>>>> ==> /var/log/messages <==
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>>> beat
>>>>>>>> raised exception <class 'qpid.messaging.exceptions.Timeout'>:
>>>>>>>> Timeout('Connection attach timed out',)
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>>>>>>> Traceback (most recent call last):
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>>> File
>>>>>>>> "/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112,
>> in
>>>>>>>> start_scheduler
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>
>>>>>>>> beat.start()
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>>> File
>>>>>>>> "/usr/lib/python2.7/site-packages/celery/beat.py", line 462, in
>> start
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>
>>>>>>>> interval = self.scheduler.tick()
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>>> File
>>>>>>>> "/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py",
>>>> line
>>>>>>>> 265, in tick
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>
>>>>>> ret
>>>>>>>> = self.call_tick(self, celerybeat_name)
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>>> File
>>>>>>>> "/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py",
>>>> line
>>>>>>>> 230, in call_tick
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>
>>>>>> ret
>>>>>>>> = super(Scheduler, self).tick()
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>>> File
>>>>>>>> "/usr/lib/python2.7/site-packages/celery/beat.py", line 220, in
>> tick
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>
>>>>>>>> next_time_to_run = self.maybe_due(entry, self.publisher)
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>>> File
>>>>>>>> "/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line
>> 325,
>>>>>> in
>>>>>>>> __get__
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>
>>>>>>>> value = obj.__dict__[self.__name__] = self.__get(obj)
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>>> File
>>>>>>>> "/usr/lib/python2.7/site-packages/celery/beat.py", line 342, in
>>>>>> publisher
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>
>>>>>>>> return self.Publisher(self._ensure_connected())
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>>> File
>>>>>>>> "/usr/lib/python2.7/site-packages/celery/beat.py", line 326, in
>>>>>>>> _ensure_connected
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>
>>>>>>>> _error_handler, self.app.conf.BROKER_CONNECTION_MAX_RETRIES
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>>> File
>>>>>>>> "/usr/lib/python2.7/site-packages/kombu/connection.py", line 369,
>> in
>>>>>>>> ensure_connection
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>
>>>>>>>> interval_start, interval_step, interval_max, callback)
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>>> File
>>>>>>>> "/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line
>> 246,
>>>>>> in
>>>>>>>> retry_over_time
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>
>>>>>>>> return fun(*args, **kwargs)
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>>> File
>>>>>>>> "/usr/lib/python2.7/site-packages/kombu/connection.py", line 237,
>> in
>>>>>> connect
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>
>>>>>>>> return self.connection
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>>> File
>>>>>>>> "/usr/lib/python2.7/site-packages/kombu/connection.py", line 741,
>> in
>>>>>>>> connection
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>
>>>>>>>> self._connection = self._establish_connection()
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>>> File
>>>>>>>> "/usr/lib/python2.7/site-packages/kombu/connection.py", line 696,
>> in
>>>>>>>> _establish_connection
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>
>>>>>> conn
>>>>>>>> = self.transport.establish_connection()
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>>> File
>>>>>>>> "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line
>>>> 1600,
>>>>>> in
>>>>>>>> establish_connection
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>
>>>>>> conn
>>>>>>>> = self.Connection(**opts)
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>>> File
>>>>>>>> "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line
>>>> 1261,
>>>>>> in
>>>>>>>> __init__
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>
>>>>>>>> self._qpid_conn = establish(**self.connection_options)
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>>> File
>>>>>>>> "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py",
>> line
>>>>>> 112,
>>>>>>>> in establish
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>
>>>>>>>> conn.open(timeout=timeout)
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>>> File
>>>>>>>> "<string>", line 6, in open
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>>> File
>>>>>>>> "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py",
>> line
>>>>>> 323,
>>>>>>>> in open
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>
>>>>>>>> self.attach(timeout=timeout)
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>>> File
>>>>>>>> "<string>", line 6, in attach
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>>> File
>>>>>>>> "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py",
>> line
>>>>>> 343,
>>>>>>>> in attach
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>
>>>>
>>>>>>>> raise Timeout("Connection attach timed out")
>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
>>>>>> Timeout:
>>>>>>>> Connection attach timed out
>>>>>>>> Aug 1 08:45:03 dscaprv01 celery: celery beat v3.1.11 (Cipater) is
>>>>>> starting.
>>>>>>>> Aug 1 08:45:03 dscaprv01 celery: __ - ... __ - _
>>>>>>>> Aug 1 08:45:03 dscaprv01 celery: Configuration ->
>>>>>>>> Aug 1 08:45:03 dscaprv01 celery: . broker ->
>>>>>>>> qpid://dscaprv01.corp.acxiom.net:5671//
>>>>>>>> Aug 1 08:45:03 dscaprv01 celery: . loader ->
>>>>>> celery.loaders.app.AppLoader
>>>>>>>> Aug 1 08:45:03 dscaprv01 celery: . scheduler ->
>>>>>>>> pulp.server.async.scheduler.Scheduler
>>>>>>>> Aug 1 08:45:03 dscaprv01 celery: . logfile -> [stderr]@%INFO
>>>>>>>> Aug 1 08:45:03 dscaprv01 celery: . maxinterval -> now (0s)
>>>>>>>>
>>>>>>>>
>>>>>>>> On Monday, August 1, 2016 at 8:43:45 AM UTC-5, Chris Duryee wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 08/01/2016 09:39 AM, Rick Langston wrote:
>>>>>>>>>> The backend service all say ok but when a run a katelli-service
>>>>>> status I
>>>>>>>>>> can see that celery.bet fails to status. if I restart the server
>> it
>>>>>> and
>>>>>>>>>> immediately check the status it says running but checking the
>>>> status
>>>>>>>>> again
>>>>>>>>>> shows it timed out.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Is this the sequence of events?
>>>>>>>>>
>>>>>>>>> * service pulp_celerybeat start (outputs success)
>>>>>>>>> * service pulp_celerybeat status (outputs success)
>>>>>>>>> * wait some number of seconds
>>>>>>>>> * service pulp_celerybeat status (outputs error)
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> No memory errros noted
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> [root@dscaprv01 tmp]# systemctl status pulp_celerybeat.service
>>>>>>>>>> ● pulp_celerybeat.service - Pulp's Celerybeat
>>>>>>>>>> Loaded: loaded
>> (/usr/lib/systemd/system/pulp_celerybeat.service;
>>>>>>>>>> enabled; vendor preset: disabled)
>>>>>>>>>> Active: inactive (dead) since Mon 2016-08-01 08:34:31 CDT; 18s
>>>> ago
>>>>>>>>>> Process: 5887 ExecStart=/usr/bin/celery beat
>>>>>>>>>> --app=pulp.server.async.celery_instance.celery
>>>>>>>>>> --scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
>>>>>>>>>> status=0/SUCCESS)
>>>>>>>>>> Main PID: 5887 (code=exited, status=0/SUCCESS)
>>>>>>>>>>
>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net pulp[5887]:
>>>>>>>>> celery.beat:CRITICAL:
>>>>>>>>>> (5887-79264) raise Timeout("Connection attach timed out")
>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net pulp[5887]:
>>>>>>>>> celery.beat:CRITICAL:
>>>>>>>>>> (5887-79264) Timeout: Connection attach timed out
>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: celery
>>>> beat
>>>>>>>>> v3.1.11
>>>>>>>>>> (Cipater) is starting.
>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: __ -
>>
>>>>>> ...
>>>>>>>>> __
>>>>>>>>>> - _
>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]:
>>>>>> Configuration
>>>>>>>>> ->
>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . broker
>>>> ->
>>>>>>>>>> qpid://dscaprv01.corp.acxiom.net:5671//
>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . loader
>>>> ->
>>>>>>>>>> celery.loaders.app.AppLoader
>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
>>>> scheduler
>>>>>> ->
>>>>>>>>>> pulp.server.async.scheduler.Scheduler
>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
>> logfile
>>>> ->
>>>>>>>>>> [stderr]@%INFO
>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
>>>>>> maxinterval
>>>>>>>>> ->
>>>>>>>>>> now (0s)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Monday, August 1, 2016 at 8:22:46 AM UTC-5, Chris Duryee
>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 08/01/2016 08:50 AM, Rick Langston wrote:
>>>>>>>>>>>> I do see this issue in messages but not sure if its related
>>>>>>>>>>>>
>>>>>>>>>>>> Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL:
>>>> (28691-76416)
>>>>>>>>> beat
>>>>>>>>>>>> raised exception <class 'qpid.messaging.exceptions.Timeout'>:
>>>>>>>>>>>> Timeout('Connection attach timed out',)
>>>>>>>>>>>> Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL:
>>>> (28691-76416)
>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>> Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL:
>>>> (28691-76416)
>>>>>>
>>>>>>>>>>> File
>>>>>>>>>>>> "/usr/lib/python2.7/site-packages/celery/apps/beat.py", line
>> 112,
>>>>>> in
>>>>>>>>>>>> start_scheduler
>>>>>>>>>>>> Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL:
>>>> (28691-76416)
>>>>>>
>>>>>>>>>
>>>>>>>>>>>> beat.start()
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> That is the likely culprit:)
>>>>>>>>>>>
>>>>>>>>>>> Next time your task hangs, check in the "/about" page on your
>>>>>> Katello
>>>>>>>>>>> instance and ensure everything under "Backend System Status"
>> says
>>>>>> "OK"
>>>>>>>>>>> with no further message.
>>>>>>>>>>>
>>>>>>>>>>> If there are pulp errors, a possible quick fix is to ensure
>> qpidd
>>>> is
>>>>>>>>>>> still running, then restart pulp_workers, pulp_celerybeat and
>>>>>>>>>>> pulp_resource_manager. I suspect your task will get picked up
>>>> after
>>>>>>>>> that.
>>>>>>>>>>>
>>>>>>>>>>> Also, please check dmesg for out-of-memory errors. There are
>> some
>>>>>> other
>>>>>>>>>>> possible things we can check, but I would be curious first about
>>>> the
>>>>>>>>>>> backend system status output.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Monday, August 1, 2016 at 7:12:26 AM UTC-5, Chris Duryee
>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 08/01/2016 07:54 AM, Rick Langston wrote:
>>>>>>>>>>>>>> Hello
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I seem to be having some odd behavior with this version.
>> With
>>>> a
>>>>>>>>> fresh
>>>>>>>>>>>>>> install on centos 7 I have setup a product which completes
>>>>>> normally
>>>>>>>>>>> but
>>>>>>>>>>>>>> when I discover a repo and save them i get these meta data
>> task
>>>>>> that
>>>>>>>>>>>>> seem
>>>>>>>>>>>>>> to just wait forever. Any ideas what can be the culprit
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Are there any related errors in /var/log/messages?
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Action:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Actions::Pulp::Repository::DistributorPublish
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> State: waiting for Pulp to start the task
>>>>>>>>>>>>>> Input:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> {"pulp_id"=>"test-centos-6_updates_x86_64",
>>>>>>>>>>>>>> "distributor_type_id"=>"yum_distributor",
>>>>>>>>>>>>>> "source_pulp_id"=>nil,
>>>>>>>>>>>>>> "dependency"=>nil,
>>>>>>>>>>>>>> "remote_user"=>"admin",
>>>>>>>>>>>>>> "remote_cp_user"=>"admin",
>>>>>>>>>>>>>> "locale"=>"en"}
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Output:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> {"pulp_tasks"=>
>>>>>>>>>>>>>> [{"exception"=>nil,
>>>>>>>>>>>>>> "task_type"=>"pulp.server.managers.repo.publish.publish",
>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>> "_href"=>"/pulp/api/v2/tasks/a40815d5-9ba4-463a-8216-338cdcc4b1cc/",
>>>>>>>>>>>>>> "task_id"=>"a40815d5-9ba4-463a-8216-338cdcc4b1cc",
>>>>>>>>>>>>>> "tags"=>
>>>>>>>>>>>>>> ["pulp:repository:test-centos-6_updates_x86_64",
>>>>>>>>>>>>> "pulp:action:publish"],
>>>>>>>>>>>>>> "finish_time"=>nil,
>>>>>>>>>>>>>> "_ns"=>"task_status",
>>>>>>>>>>>>>> "start_time"=>nil,
>>>>>>>>>>>>>> "traceback"=>nil,
>>>>>>>>>>>>>> "spawned_tasks"=>[],
>>>>>>>>>>>>>> "progress_report"=>{},
>>>>>>>>>>>>>> "queue"=>"None.dq",
>>>>>>>>>>>>>> "state"=>"waiting",
>>>>>>>>>>>>>> "worker_name"=>nil,
>>>>>>>>>>>>>> "result"=>nil,
>>>>>>>>>>>>>> "error"=>nil,
>>>>>>>>>>>>>> "_id"=>{"$oid"=>"579f32aa95c48c6a54257674"},
>>>>>>>>>>>>>> "id"=>"579f32aa95c48c6a54257674"}],
>>>>>>>>>>>>>> "poll_attempts"=>{"total"=>100, "failed"=>0}}
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
--
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
