On 08/04/2016 07:38 AM, Rick Langston wrote:
> Well this is strange. it looks as if the process has stabilized. I didn't
> see anything in the capture that jumped out at me but will try and
> normalize the build today and see if it continues to work.
>
Well, I am glad it works now at least:) If it becomes unstable again let
us know and we can look more.
> Thank you for your help with this
No problem, thanks for debugging with us.
>
> On Wednesday, August 3, 2016 at 10:41:49 AM UTC-5, Chris Duryee wrote:
>>
>>
>>
>> On 08/03/2016 07:41 AM, Rick Langston wrote:
>>> Things seem to stem from pulp and a connection
>>
>> cool, this narrows it down some:)
>>
>> Let's try this to see if it captures the issue:
>>
>> # stop everything
>> katello-service stop
>>
>> # start mongodb
>> systemctl start mongod
>>
>> # become qpidd user
>> sudo su - qpidd -s /bin/bash (to become qpid user)
>>
>> #start qpid with trace output
>> /usr/sbin/qpidd --config /etc/qpid/qpidd.conf -t 2>&1 | tee /tmp/qpid.out
>>
>> in another window:
>>
>> systemctl start pulp_celerybeat
>> systemctl start pulp_resource_manager
>> systemctl start pulp_workers
>>
>> <wait a minute or so>
>>
>> then stop the three pulp services, ^C to exit qpidd, and do a
>> "katello-service start" to spin everything up again.
>>
>> Hopefully this will reproduce the issue, and we'll have the full qpid
>> log in /tmp/qpid.out to see what happened. The log is at trace setting
>> and will be extremely verbose.
>>
>>>
>>> [root@servername ~]# /bin/systemctl status pulp_celerybeat.service
>>> ● pulp_celerybeat.service - Pulp's Celerybeat
>>> Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
>>> enabled; vendor preset: disabled)
>>> Active: inactive (dead) since Wed 2016-08-03 06:32:14 CDT; 1s ago
>>> Process: 5388 ExecStart=/usr/bin/celery beat
>>> --app=pulp.server.async.celery_instance.celery
>>> --scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
>>> status=0/SUCCESS)
>>> Main PID: 5388 (code=exited, status=0/SUCCESS)
>>>
>>> Aug 03 06:32:13 servername.corp.company.net pulp[5388]:
>>> celery.beat:CRITICAL: (5388-64256) Timeout: Connection attach timed out
>>> Aug 03 06:32:13 servername.corp.company.net celery[5388]: celery beat
>>> v3.1.11 (Cipater) is starting.
>>> Aug 03 06:32:13 servername.corp.company.net celery[5388]: __ -
>> ... __
>>> - _
>>> Aug 03 06:32:13 servername.corp.company.net celery[5388]: Configuration
>> ->
>>> Aug 03 06:32:13 servername.corp.company.net celery[5388]: . broker ->
>>> qpid://servername.corp.company.net:5671//
>>> Aug 03 06:32:13 servername.corp.company.net celery[5388]: . loader ->
>>> celery.loaders.app.AppLoader
>>> Aug 03 06:32:13 servername.corp.company.net celery[5388]: . scheduler
>> ->
>>> pulp.server.async.scheduler.Scheduler
>>> Aug 03 06:32:13 servername.corp.company.net celery[5388]: . logfile ->
>>> [stderr]@%INFO
>>> Aug 03 06:32:13 servername.corp.company.net celery[5388]: . maxinterval
>> ->
>>> now (0s)
>>> Aug 03 06:32:14 servername.corp.company.net pulp[5388]:
>>> qpid.messaging:DEBUG: poll(1 fds, timeout=4000.0)
>>>
>>> On Tuesday, August 2, 2016 at 10:12:41 AM UTC-5, Chris Duryee wrote:
>>>>
>>>>
>>>>
>>>> On 08/02/2016 10:23 AM, Rick Langston wrote:
>>>>>
>>>>>
>>>>> On Tuesday, August 2, 2016 at 7:30:07 AM UTC-5, Chris Duryee wrote:
>>>>>>
>>>>>> and it will eventually die as it did last night.
>>>>>>
>>>>>
>>>>> Where would I send the debug file?
>>>>
>>>> I guess the easiest way is to just email me direct with the attachment,
>>>> but we can keep the discussion on-list.
>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> On 08/01/2016 04:14 PM, Rick Langston wrote:
>>>>>>> Strange, If I restart the service "/bin/systemctl status
>>>>>>> pulp_celerybeat.service" it will eventually start. Any ideas?
>>>>>>
>>>>>> I'm glad it eventually restarts:) that is strange though that fails
>> at
>>>>>> first but eventually works.
>>>>>>
>>>>>> Can you send the output of foreman-debug? This will create an an
>>>> archive
>>>>>> with system information and log files that we can peruse to debug
>>>> further.
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Monday, August 1, 2016 at 12:19:27 PM UTC-5, Chris Duryee wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 08/01/2016 10:52 AM, Chris Duryee wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 08/01/2016 09:49 AM, Rick Langston wrote:
>>>>>>>>>> Thank you for help on this
>>>>>>>>>>
>>>>>>>>>> yes here is the actual capture
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks for the capture of the logs.
>>>>>>>>>
>>>>>>>>> What is the output of systemctl status qpidd.service? I am curious
>>>> if
>>>>>>>>> only celerybeat is unable to connect, or if there is some other
>>>> issue.
>>>>>>>>
>>>>>>>> one more thing to check, as root please run:
>>>>>>>>
>>>>>>>> qpid-stat --ssl-certificate /etc/pki/katello/certs/java-client.crt
>>>>>>>> --ssl-key /etc/pki/katello/private/java-client.key -b
>>>>>>>> "amqps://$(hostname -f):5671" -q
>>>>>>>>
>>>>>>>> If qpid is accepting connections, it should output some stuff about
>>>>>>>> queues.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> If you want to restart qpidd, the best way is to just do a
>>>>>>>>> katello-service stop. A number of other services depend on qpid,
>> so
>>>>>> it's
>>>>>>>>> easiest to just down everything, ensure qpidd is stopped, then
>> bring
>>>>>> it
>>>>>>>>> all back up via the script.
>>>>>>>>>
>>>>>>>>>> command line output
>>>>>>>>>> [root@dscaprv01 tmp]# systemctl restart pulp_celerybeat.service
>>>>>>>>>> [root@dscaprv01 tmp]# systemctl status pulp_celerybeat.service
>>>>>>>>>> ● pulp_celerybeat.service - Pulp's Celerybeat
>>>>>>>>>> Loaded: loaded
>> (/usr/lib/systemd/system/pulp_celerybeat.service;
>>>>>>>>>> enabled; vendor preset: disabled)
>>>>>>>>>> Active: active (running) since Mon 2016-08-01 08:34:20 CDT; 7s
>>>> ago
>>>>>>>>>> Main PID: 5887 (celery)
>>>>>>>>>> CGroup: /system.slice/pulp_celerybeat.service
>>>>>>>>>> └─5887 /usr/bin/python /usr/bin/celery beat
>>>>>>>>>> --app=pulp.server.async.celery_instance.celery
>>>>>>>>>> --scheduler=pulp.server.async.scheduler.Scheduler
>>>>>>>>>>
>>>>>>>>>> Aug 01 08:34:20 dscaprv01.corp.acxiom.net systemd[1]: Started
>>>> Pulp's
>>>>>>>>>> Celerybeat.
>>>>>>>>>> Aug 01 08:34:20 dscaprv01.corp.acxiom.net systemd[1]: Starting
>>>>>> Pulp's
>>>>>>>>>> Celerybeat...
>>>>>>>>>> Aug 01 08:34:25 dscaprv01.corp.acxiom.net pulp[5887]:
>>>>>>>> celery.beat:INFO:
>>>>>>>>>> beat: Starting...
>>>>>>>>>> Aug 01 08:34:25 dscaprv01.corp.acxiom.net pulp[5887]:
>>>>>>>>>> pulp.server.db.connection:INFO: Attempting to connect to
>>>>>>>> localhost:27017
>>>>>>>>>> Aug 01 08:34:25 dscaprv01.corp.acxiom.net pulp[5887]:
>>>>>>>>>> pulp.server.async.scheduler:INFO: Worker Timeout Monitor Started
>>>>>>>>>> Aug 01 08:34:25 dscaprv01.corp.acxiom.net pulp[5887]:
>>>>>>>>>> pulp.server.db.connection:INFO: Attempting to connect to
>>>>>>>> localhost:27017
>>>>>>>>>> Aug 01 08:34:26 dscaprv01.corp.acxiom.net pulp[5887]:
>>>>>>>>>> pulp.server.db.connection:INFO: Write concern for Mongo
>> connection:
>>>>>> {}
>>>>>>>>>> [root@dscaprv01 tmp]# systemctl status pulp_celerybeat.service
>>>>>>>>>> ● pulp_celerybeat.service - Pulp's Celerybeat
>>>>>>>>>> Loaded: loaded
>> (/usr/lib/systemd/system/pulp_celerybeat.service;
>>>>>>>>>> enabled; vendor preset: disabled)
>>>>>>>>>> Active: inactive (dead) since Mon 2016-08-01 08:34:31 CDT; 18s
>>>> ago
>>>>>>>>>> Process: 5887 ExecStart=/usr/bin/celery beat
>>>>>>>>>> --app=pulp.server.async.celery_instance.celery
>>>>>>>>>> --scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
>>>>>>>>>> status=0/SUCCESS)
>>>>>>>>>> Main PID: 5887 (code=exited, status=0/SUCCESS)
>>>>>>>>>>
>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net pulp[5887]:
>>>>>>>> celery.beat:CRITICAL:
>>>>>>>>>> (5887-79264) raise Timeout("Connection attach timed out")
>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net pulp[5887]:
>>>>>>>> celery.beat:CRITICAL:
>>>>>>>>>> (5887-79264) Timeout: Connection attach timed out
>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: celery
>>>> beat
>>>>>>>> v3.1.11
>>>>>>>>>> (Cipater) is starting.
>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: __ -
>>
>>>>>> ...
>>>>>>>> __
>>>>>>>>>> - _
>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]:
>>>>>> Configuration
>>>>>>>> ->
>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . broker
>>>> ->
>>>>>>>>>> qpid://dscaprv01.corp.acxiom.net:5671//
>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . loader
>>>> ->
>>>>>>>>>> celery.loaders.app.AppLoader
>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
>>>> scheduler
>>>>>> ->
>>>>>>>>>> pulp.server.async.scheduler.Scheduler
>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
>> logfile
>>>> ->
>>>>>>>>>> [stderr]@%INFO
>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
>>>>>> maxinterval
>>>>>>>> ->
>>>>>>>>>> now (0s)
>>>>>>>>>>
>>>>>>>>>> /var/log/messages output
>>>>>>>>>>
>>>>>>>>>> ==> /var/log/messages <==
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>>> beat
>>>>>>>>>> raised exception <class 'qpid.messaging.exceptions.Timeout'>:
>>>>>>>>>> Timeout('Connection attach timed out',)
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>>> File
>>>>>>>>>> "/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112,
>>>> in
>>>>>>>>>> start_scheduler
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>
>>>>>>>>>> beat.start()
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>>> File
>>>>>>>>>> "/usr/lib/python2.7/site-packages/celery/beat.py", line 462, in
>>>> start
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>
>>>>>>>>>> interval = self.scheduler.tick()
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>>> File
>>>>>>>>>>
>> "/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py",
>>>>>> line
>>>>>>>>>> 265, in tick
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>
>>>>>>>> ret
>>>>>>>>>> = self.call_tick(self, celerybeat_name)
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>>> File
>>>>>>>>>>
>> "/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py",
>>>>>> line
>>>>>>>>>> 230, in call_tick
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>
>>>>>>>> ret
>>>>>>>>>> = super(Scheduler, self).tick()
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>>> File
>>>>>>>>>> "/usr/lib/python2.7/site-packages/celery/beat.py", line 220, in
>>>> tick
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>
>>>>>>>>>> next_time_to_run = self.maybe_due(entry, self.publisher)
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>>> File
>>>>>>>>>> "/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line
>>>> 325,
>>>>>>>> in
>>>>>>>>>> __get__
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>
>>>>>>>>>> value = obj.__dict__[self.__name__] = self.__get(obj)
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>>> File
>>>>>>>>>> "/usr/lib/python2.7/site-packages/celery/beat.py", line 342, in
>>>>>>>> publisher
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>
>>>>>>>>>> return self.Publisher(self._ensure_connected())
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>>> File
>>>>>>>>>> "/usr/lib/python2.7/site-packages/celery/beat.py", line 326, in
>>>>>>>>>> _ensure_connected
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>
>>>>>>>>>> _error_handler, self.app.conf.BROKER_CONNECTION_MAX_RETRIES
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>>> File
>>>>>>>>>> "/usr/lib/python2.7/site-packages/kombu/connection.py", line 369,
>>>> in
>>>>>>>>>> ensure_connection
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>
>>>>>>>>>> interval_start, interval_step, interval_max, callback)
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>>> File
>>>>>>>>>> "/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line
>>>> 246,
>>>>>>>> in
>>>>>>>>>> retry_over_time
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>
>>>>>>>>>> return fun(*args, **kwargs)
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>>> File
>>>>>>>>>> "/usr/lib/python2.7/site-packages/kombu/connection.py", line 237,
>>>> in
>>>>>>>> connect
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>
>>>>>>>>>> return self.connection
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>>> File
>>>>>>>>>> "/usr/lib/python2.7/site-packages/kombu/connection.py", line 741,
>>>> in
>>>>>>>>>> connection
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>
>>>>>>>>>> self._connection = self._establish_connection()
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>>> File
>>>>>>>>>> "/usr/lib/python2.7/site-packages/kombu/connection.py", line 696,
>>>> in
>>>>>>>>>> _establish_connection
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>
>>>>>>>> conn
>>>>>>>>>> = self.transport.establish_connection()
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>>> File
>>>>>>>>>> "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line
>>>>>> 1600,
>>>>>>>> in
>>>>>>>>>> establish_connection
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>
>>>>>>>> conn
>>>>>>>>>> = self.Connection(**opts)
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>>> File
>>>>>>>>>> "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line
>>>>>> 1261,
>>>>>>>> in
>>>>>>>>>> __init__
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>
>>>>>>>>>> self._qpid_conn = establish(**self.connection_options)
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>>> File
>>>>>>>>>> "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py",
>>>> line
>>>>>>>> 112,
>>>>>>>>>> in establish
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>
>>>>>>>>>> conn.open(timeout=timeout)
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>>> File
>>>>>>>>>> "<string>", line 6, in open
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>>> File
>>>>>>>>>> "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py",
>>>> line
>>>>>>>> 323,
>>>>>>>>>> in open
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>
>>>>>>>>>> self.attach(timeout=timeout)
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>>> File
>>>>>>>>>> "<string>", line 6, in attach
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>>> File
>>>>>>>>>> "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py",
>>>> line
>>>>>>>> 343,
>>>>>>>>>> in attach
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>
>>>>>>
>>>>>>>>>> raise Timeout("Connection attach timed out")
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
>> (7070-59168)
>>>>>>>> Timeout:
>>>>>>>>>> Connection attach timed out
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 celery: celery beat v3.1.11 (Cipater)
>> is
>>>>>>>> starting.
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 celery: __ - ... __ - _
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 celery: Configuration ->
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 celery: . broker ->
>>>>>>>>>> qpid://dscaprv01.corp.acxiom.net:5671//
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 celery: . loader ->
>>>>>>>> celery.loaders.app.AppLoader
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 celery: . scheduler ->
>>>>>>>>>> pulp.server.async.scheduler.Scheduler
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 celery: . logfile -> [stderr]@%INFO
>>>>>>>>>> Aug 1 08:45:03 dscaprv01 celery: . maxinterval -> now (0s)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Monday, August 1, 2016 at 8:43:45 AM UTC-5, Chris Duryee
>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 08/01/2016 09:39 AM, Rick Langston wrote:
>>>>>>>>>>>> The backend service all say ok but when a run a katelli-service
>>>>>>>> status I
>>>>>>>>>>>> can see that celery.bet fails to status. if I restart the
>> server
>>>> it
>>>>>>>> and
>>>>>>>>>>>> immediately check the status it says running but checking the
>>>>>> status
>>>>>>>>>>> again
>>>>>>>>>>>> shows it timed out.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Is this the sequence of events?
>>>>>>>>>>>
>>>>>>>>>>> * service pulp_celerybeat start (outputs success)
>>>>>>>>>>> * service pulp_celerybeat status (outputs success)
>>>>>>>>>>> * wait some number of seconds
>>>>>>>>>>> * service pulp_celerybeat status (outputs error)
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> No memory errros noted
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [root@dscaprv01 tmp]# systemctl status pulp_celerybeat.service
>>>>>>>>>>>> ● pulp_celerybeat.service - Pulp's Celerybeat
>>>>>>>>>>>> Loaded: loaded
>>>> (/usr/lib/systemd/system/pulp_celerybeat.service;
>>>>>>>>>>>> enabled; vendor preset: disabled)
>>>>>>>>>>>> Active: inactive (dead) since Mon 2016-08-01 08:34:31 CDT;
>> 18s
>>>>>> ago
>>>>>>>>>>>> Process: 5887 ExecStart=/usr/bin/celery beat
>>>>>>>>>>>> --app=pulp.server.async.celery_instance.celery
>>>>>>>>>>>> --scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
>>>>>>>>>>>> status=0/SUCCESS)
>>>>>>>>>>>> Main PID: 5887 (code=exited, status=0/SUCCESS)
>>>>>>>>>>>>
>>>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net pulp[5887]:
>>>>>>>>>>> celery.beat:CRITICAL:
>>>>>>>>>>>> (5887-79264) raise Timeout("Connection attach timed out")
>>>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net pulp[5887]:
>>>>>>>>>>> celery.beat:CRITICAL:
>>>>>>>>>>>> (5887-79264) Timeout: Connection attach timed out
>>>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: celery
>>>>>> beat
>>>>>>>>>>> v3.1.11
>>>>>>>>>>>> (Cipater) is starting.
>>>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: __
>> -
>>>>
>>>>>>>> ...
>>>>>>>>>>> __
>>>>>>>>>>>> - _
>>>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]:
>>>>>>>> Configuration
>>>>>>>>>>> ->
>>>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
>> broker
>>>>>> ->
>>>>>>>>>>>> qpid://dscaprv01.corp.acxiom.net:5671//
>>>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
>> loader
>>>>>> ->
>>>>>>>>>>>> celery.loaders.app.AppLoader
>>>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
>>>>>> scheduler
>>>>>>>> ->
>>>>>>>>>>>> pulp.server.async.scheduler.Scheduler
>>>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
>>>> logfile
>>>>>> ->
>>>>>>>>>>>> [stderr]@%INFO
>>>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
>>>>>>>> maxinterval
>>>>>>>>>>> ->
>>>>>>>>>>>> now (0s)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Monday, August 1, 2016 at 8:22:46 AM UTC-5, Chris Duryee
>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 08/01/2016 08:50 AM, Rick Langston wrote:
>>>>>>>>>>>>>> I do see this issue in messages but not sure if its related
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL:
>>>>>> (28691-76416)
>>>>>>>>>>> beat
>>>>>>>>>>>>>> raised exception <class 'qpid.messaging.exceptions.Timeout'>:
>>>>>>>>>>>>>> Timeout('Connection attach timed out',)
>>>>>>>>>>>>>> Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL:
>>>>>> (28691-76416)
>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>> Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL:
>>>>>> (28691-76416)
>>>>>>>>
>>>>>>>>>>>>> File
>>>>>>>>>>>>>> "/usr/lib/python2.7/site-packages/celery/apps/beat.py", line
>>>> 112,
>>>>>>>> in
>>>>>>>>>>>>>> start_scheduler
>>>>>>>>>>>>>> Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL:
>>>>>> (28691-76416)
>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>>> beat.start()
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> That is the likely culprit:)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Next time your task hangs, check in the "/about" page on your
>>>>>>>> Katello
>>>>>>>>>>>>> instance and ensure everything under "Backend System Status"
>>>> says
>>>>>>>> "OK"
>>>>>>>>>>>>> with no further message.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If there are pulp errors, a possible quick fix is to ensure
>>>> qpidd
>>>>>> is
>>>>>>>>>>>>> still running, then restart pulp_workers, pulp_celerybeat and
>>>>>>>>>>>>> pulp_resource_manager. I suspect your task will get picked up
>>>>>> after
>>>>>>>>>>> that.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Also, please check dmesg for out-of-memory errors. There are
>>>> some
>>>>>>>> other
>>>>>>>>>>>>> possible things we can check, but I would be curious first
>> about
>>>>>> the
>>>>>>>>>>>>> backend system status output.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Monday, August 1, 2016 at 7:12:26 AM UTC-5, Chris Duryee
>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 08/01/2016 07:54 AM, Rick Langston wrote:
>>>>>>>>>>>>>>>> Hello
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I seem to be having some odd behavior with this version.
>>>> With
>>>>>> a
>>>>>>>>>>> fresh
>>>>>>>>>>>>>>>> install on centos 7 I have setup a product which completes
>>>>>>>> normally
>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>> when I discover a repo and save them i get these meta data
>>>> task
>>>>>>>> that
>>>>>>>>>>>>>>> seem
>>>>>>>>>>>>>>>> to just wait forever. Any ideas what can be the culprit
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Are there any related errors in /var/log/messages?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Action:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Actions::Pulp::Repository::DistributorPublish
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> State: waiting for Pulp to start the task
>>>>>>>>>>>>>>>> Input:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> {"pulp_id"=>"test-centos-6_updates_x86_64",
>>>>>>>>>>>>>>>> "distributor_type_id"=>"yum_distributor",
>>>>>>>>>>>>>>>> "source_pulp_id"=>nil,
>>>>>>>>>>>>>>>> "dependency"=>nil,
>>>>>>>>>>>>>>>> "remote_user"=>"admin",
>>>>>>>>>>>>>>>> "remote_cp_user"=>"admin",
>>>>>>>>>>>>>>>> "locale"=>"en"}
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Output:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> {"pulp_tasks"=>
>>>>>>>>>>>>>>>> [{"exception"=>nil,
>>>>>>>>>>>>>>>>
>> "task_type"=>"pulp.server.managers.repo.publish.publish",
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>> "_href"=>"/pulp/api/v2/tasks/a40815d5-9ba4-463a-8216-338cdcc4b1cc/",
>>>>>>>>>>>>>>>> "task_id"=>"a40815d5-9ba4-463a-8216-338cdcc4b1cc",
>>>>>>>>>>>>>>>> "tags"=>
>>>>>>>>>>>>>>>> ["pulp:repository:test-centos-6_updates_x86_64",
>>>>>>>>>>>>>>> "pulp:action:publish"],
>>>>>>>>>>>>>>>> "finish_time"=>nil,
>>>>>>>>>>>>>>>> "_ns"=>"task_status",
>>>>>>>>>>>>>>>> "start_time"=>nil,
>>>>>>>>>>>>>>>> "traceback"=>nil,
>>>>>>>>>>>>>>>> "spawned_tasks"=>[],
>>>>>>>>>>>>>>>> "progress_report"=>{},
>>>>>>>>>>>>>>>> "queue"=>"None.dq",
>>>>>>>>>>>>>>>> "state"=>"waiting",
>>>>>>>>>>>>>>>> "worker_name"=>nil,
>>>>>>>>>>>>>>>> "result"=>nil,
>>>>>>>>>>>>>>>> "error"=>nil,
>>>>>>>>>>>>>>>> "_id"=>{"$oid"=>"579f32aa95c48c6a54257674"},
>>>>>>>>>>>>>>>> "id"=>"579f32aa95c48c6a54257674"}],
>>>>>>>>>>>>>>>> "poll_attempts"=>{"total"=>100, "failed"=>0}}
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
--
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
