Well the rebuild came back with the same issue. This issue seems to be
tied to this hardware somehow as I can build using the same process in a VM
and everything is as it should be. Diags have been ran on the hardware
with no issues found. we also removed the external storage and mounted
everything local with no change.
Are there any tunes that we need to be aware of in regards to
CPU,disk,memory or open files sockets etc?
Thank you
On Thursday, August 4, 2016 at 12:03:13 PM UTC-5, Chris Duryee wrote:
>
>
>
> On 08/04/2016 07:38 AM, Rick Langston wrote:
> > Well this is strange. it looks as if the process has stabilized. I
> didn't
> > see anything in the capture that jumped out at me but will try and
> > normalize the build today and see if it continues to work.
> >
>
> Well, I am glad it works now at least:) If it becomes unstable again let
> us know and we can look more.
>
> > Thank you for your help with this
>
> No problem, thanks for debugging with us.
>
> >
> > On Wednesday, August 3, 2016 at 10:41:49 AM UTC-5, Chris Duryee wrote:
> >>
> >>
> >>
> >> On 08/03/2016 07:41 AM, Rick Langston wrote:
> >>> Things seem to stem from pulp and a connection
> >>
> >> cool, this narrows it down some:)
> >>
> >> Let's try this to see if it captures the issue:
> >>
> >> # stop everything
> >> katello-service stop
> >>
> >> # start mongodb
> >> systemctl start mongod
> >>
> >> # become qpidd user
> >> sudo su - qpidd -s /bin/bash (to become qpid user)
> >>
> >> #start qpid with trace output
> >> /usr/sbin/qpidd --config /etc/qpid/qpidd.conf -t 2>&1 | tee
> /tmp/qpid.out
> >>
> >> in another window:
> >>
> >> systemctl start pulp_celerybeat
> >> systemctl start pulp_resource_manager
> >> systemctl start pulp_workers
> >>
> >> <wait a minute or so>
> >>
> >> then stop the three pulp services, ^C to exit qpidd, and do a
> >> "katello-service start" to spin everything up again.
> >>
> >> Hopefully this will reproduce the issue, and we'll have the full qpid
> >> log in /tmp/qpid.out to see what happened. The log is at trace setting
> >> and will be extremely verbose.
> >>
> >>>
> >>> [root@servername ~]# /bin/systemctl status pulp_celerybeat.service
> >>> ● pulp_celerybeat.service - Pulp's Celerybeat
> >>> Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
> >>> enabled; vendor preset: disabled)
> >>> Active: inactive (dead) since Wed 2016-08-03 06:32:14 CDT; 1s ago
> >>> Process: 5388 ExecStart=/usr/bin/celery beat
> >>> --app=pulp.server.async.celery_instance.celery
> >>> --scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
> >>> status=0/SUCCESS)
> >>> Main PID: 5388 (code=exited, status=0/SUCCESS)
> >>>
> >>> Aug 03 06:32:13 servername.corp.company.net pulp[5388]:
> >>> celery.beat:CRITICAL: (5388-64256) Timeout: Connection attach timed
> out
> >>> Aug 03 06:32:13 servername.corp.company.net celery[5388]: celery beat
> >>> v3.1.11 (Cipater) is starting.
> >>> Aug 03 06:32:13 servername.corp.company.net celery[5388]: __ -
> >> ... __
> >>> - _
> >>> Aug 03 06:32:13 servername.corp.company.net celery[5388]:
> Configuration
> >> ->
> >>> Aug 03 06:32:13 servername.corp.company.net celery[5388]: . broker ->
> >>> qpid://servername.corp.company.net:5671//
> >>> Aug 03 06:32:13 servername.corp.company.net celery[5388]: . loader ->
> >>> celery.loaders.app.AppLoader
> >>> Aug 03 06:32:13 servername.corp.company.net celery[5388]: . scheduler
> >> ->
> >>> pulp.server.async.scheduler.Scheduler
> >>> Aug 03 06:32:13 servername.corp.company.net celery[5388]: . logfile
> ->
> >>> [stderr]@%INFO
> >>> Aug 03 06:32:13 servername.corp.company.net celery[5388]: .
> maxinterval
> >> ->
> >>> now (0s)
> >>> Aug 03 06:32:14 servername.corp.company.net pulp[5388]:
> >>> qpid.messaging:DEBUG: poll(1 fds, timeout=4000.0)
> >>>
> >>> On Tuesday, August 2, 2016 at 10:12:41 AM UTC-5, Chris Duryee wrote:
> >>>>
> >>>>
> >>>>
> >>>> On 08/02/2016 10:23 AM, Rick Langston wrote:
> >>>>>
> >>>>>
> >>>>> On Tuesday, August 2, 2016 at 7:30:07 AM UTC-5, Chris Duryee wrote:
> >>>>>>
> >>>>>> and it will eventually die as it did last night.
> >>>>>>
> >>>>>
> >>>>> Where would I send the debug file?
> >>>>
> >>>> I guess the easiest way is to just email me direct with the
> attachment,
> >>>> but we can keep the discussion on-list.
> >>>>
> >>>>>
> >>>>>
> >>>>>>
> >>>>>> On 08/01/2016 04:14 PM, Rick Langston wrote:
> >>>>>>> Strange, If I restart the service "/bin/systemctl status
> >>>>>>> pulp_celerybeat.service" it will eventually start. Any ideas?
> >>>>>>
> >>>>>> I'm glad it eventually restarts:) that is strange though that fails
> >> at
> >>>>>> first but eventually works.
> >>>>>>
> >>>>>> Can you send the output of foreman-debug? This will create an an
> >>>> archive
> >>>>>> with system information and log files that we can peruse to debug
> >>>> further.
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Monday, August 1, 2016 at 12:19:27 PM UTC-5, Chris Duryee
> wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 08/01/2016 10:52 AM, Chris Duryee wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On 08/01/2016 09:49 AM, Rick Langston wrote:
> >>>>>>>>>> Thank you for help on this
> >>>>>>>>>>
> >>>>>>>>>> yes here is the actual capture
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Thanks for the capture of the logs.
> >>>>>>>>>
> >>>>>>>>> What is the output of systemctl status qpidd.service? I am
> curious
> >>>> if
> >>>>>>>>> only celerybeat is unable to connect, or if there is some other
> >>>> issue.
> >>>>>>>>
> >>>>>>>> one more thing to check, as root please run:
> >>>>>>>>
> >>>>>>>> qpid-stat --ssl-certificate
> /etc/pki/katello/certs/java-client.crt
> >>>>>>>> --ssl-key /etc/pki/katello/private/java-client.key -b
> >>>>>>>> "amqps://$(hostname -f):5671" -q
> >>>>>>>>
> >>>>>>>> If qpid is accepting connections, it should output some stuff
> about
> >>>>>>>> queues.
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>> If you want to restart qpidd, the best way is to just do a
> >>>>>>>>> katello-service stop. A number of other services depend on qpid,
> >> so
> >>>>>> it's
> >>>>>>>>> easiest to just down everything, ensure qpidd is stopped, then
> >> bring
> >>>>>> it
> >>>>>>>>> all back up via the script.
> >>>>>>>>>
> >>>>>>>>>> command line output
> >>>>>>>>>> [root@dscaprv01 tmp]# systemctl restart pulp_celerybeat.service
> >>>>>>>>>> [root@dscaprv01 tmp]# systemctl status pulp_celerybeat.service
> >>>>>>>>>> ● pulp_celerybeat.service - Pulp's Celerybeat
> >>>>>>>>>> Loaded: loaded
> >> (/usr/lib/systemd/system/pulp_celerybeat.service;
> >>>>>>>>>> enabled; vendor preset: disabled)
> >>>>>>>>>> Active: active (running) since Mon 2016-08-01 08:34:20 CDT;
> 7s
> >>>> ago
> >>>>>>>>>> Main PID: 5887 (celery)
> >>>>>>>>>> CGroup: /system.slice/pulp_celerybeat.service
> >>>>>>>>>> └─5887 /usr/bin/python /usr/bin/celery beat
> >>>>>>>>>> --app=pulp.server.async.celery_instance.celery
> >>>>>>>>>> --scheduler=pulp.server.async.scheduler.Scheduler
> >>>>>>>>>>
> >>>>>>>>>> Aug 01 08:34:20 dscaprv01.corp.acxiom.net systemd[1]: Started
> >>>> Pulp's
> >>>>>>>>>> Celerybeat.
> >>>>>>>>>> Aug 01 08:34:20 dscaprv01.corp.acxiom.net systemd[1]: Starting
> >>>>>> Pulp's
> >>>>>>>>>> Celerybeat...
> >>>>>>>>>> Aug 01 08:34:25 dscaprv01.corp.acxiom.net pulp[5887]:
> >>>>>>>> celery.beat:INFO:
> >>>>>>>>>> beat: Starting...
> >>>>>>>>>> Aug 01 08:34:25 dscaprv01.corp.acxiom.net pulp[5887]:
> >>>>>>>>>> pulp.server.db.connection:INFO: Attempting to connect to
> >>>>>>>> localhost:27017
> >>>>>>>>>> Aug 01 08:34:25 dscaprv01.corp.acxiom.net pulp[5887]:
> >>>>>>>>>> pulp.server.async.scheduler:INFO: Worker Timeout Monitor
> Started
> >>>>>>>>>> Aug 01 08:34:25 dscaprv01.corp.acxiom.net pulp[5887]:
> >>>>>>>>>> pulp.server.db.connection:INFO: Attempting to connect to
> >>>>>>>> localhost:27017
> >>>>>>>>>> Aug 01 08:34:26 dscaprv01.corp.acxiom.net pulp[5887]:
> >>>>>>>>>> pulp.server.db.connection:INFO: Write concern for Mongo
> >> connection:
> >>>>>> {}
> >>>>>>>>>> [root@dscaprv01 tmp]# systemctl status pulp_celerybeat.service
> >>>>>>>>>> ● pulp_celerybeat.service - Pulp's Celerybeat
> >>>>>>>>>> Loaded: loaded
> >> (/usr/lib/systemd/system/pulp_celerybeat.service;
> >>>>>>>>>> enabled; vendor preset: disabled)
> >>>>>>>>>> Active: inactive (dead) since Mon 2016-08-01 08:34:31 CDT;
> 18s
> >>>> ago
> >>>>>>>>>> Process: 5887 ExecStart=/usr/bin/celery beat
> >>>>>>>>>> --app=pulp.server.async.celery_instance.celery
> >>>>>>>>>> --scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
> >>>>>>>>>> status=0/SUCCESS)
> >>>>>>>>>> Main PID: 5887 (code=exited, status=0/SUCCESS)
> >>>>>>>>>>
> >>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net pulp[5887]:
> >>>>>>>> celery.beat:CRITICAL:
> >>>>>>>>>> (5887-79264) raise Timeout("Connection attach timed out")
> >>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net pulp[5887]:
> >>>>>>>> celery.beat:CRITICAL:
> >>>>>>>>>> (5887-79264) Timeout: Connection attach timed out
> >>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: celery
> >>>> beat
> >>>>>>>> v3.1.11
> >>>>>>>>>> (Cipater) is starting.
> >>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: __
> -
> >>
> >>>>>> ...
> >>>>>>>> __
> >>>>>>>>>> - _
> >>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]:
> >>>>>> Configuration
> >>>>>>>> ->
> >>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
> broker
> >>>> ->
> >>>>>>>>>> qpid://dscaprv01.corp.acxiom.net:5671//
> >>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
> loader
> >>>> ->
> >>>>>>>>>> celery.loaders.app.AppLoader
> >>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
> >>>> scheduler
> >>>>>> ->
> >>>>>>>>>> pulp.server.async.scheduler.Scheduler
> >>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
> >> logfile
> >>>> ->
> >>>>>>>>>> [stderr]@%INFO
> >>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
> >>>>>> maxinterval
> >>>>>>>> ->
> >>>>>>>>>> now (0s)
> >>>>>>>>>>
> >>>>>>>>>> /var/log/messages output
> >>>>>>>>>>
> >>>>>>>>>> ==> /var/log/messages <==
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>>> beat
> >>>>>>>>>> raised exception <class 'qpid.messaging.exceptions.Timeout'>:
> >>>>>>>>>> Timeout('Connection attach timed out',)
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>>>>>>> Traceback (most recent call last):
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>>> File
> >>>>>>>>>> "/usr/lib/python2.7/site-packages/celery/apps/beat.py", line
> 112,
> >>>> in
> >>>>>>>>>> start_scheduler
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>
> >>>>>>>>>> beat.start()
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>>> File
> >>>>>>>>>> "/usr/lib/python2.7/site-packages/celery/beat.py", line 462, in
> >>>> start
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>
> >>>>>>>>>> interval = self.scheduler.tick()
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>>> File
> >>>>>>>>>>
> >> "/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py",
> >>>>>> line
> >>>>>>>>>> 265, in tick
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>
> >>>>>>>> ret
> >>>>>>>>>> = self.call_tick(self, celerybeat_name)
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>>> File
> >>>>>>>>>>
> >> "/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py",
> >>>>>> line
> >>>>>>>>>> 230, in call_tick
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>
> >>>>>>>> ret
> >>>>>>>>>> = super(Scheduler, self).tick()
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>>> File
> >>>>>>>>>> "/usr/lib/python2.7/site-packages/celery/beat.py", line 220, in
> >>>> tick
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>
> >>>>>>>>>> next_time_to_run = self.maybe_due(entry, self.publisher)
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>>> File
> >>>>>>>>>> "/usr/lib/python2.7/site-packages/kombu/utils/__init__.py",
> line
> >>>> 325,
> >>>>>>>> in
> >>>>>>>>>> __get__
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>
> >>>>>>>>>> value = obj.__dict__[self.__name__] = self.__get(obj)
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>>> File
> >>>>>>>>>> "/usr/lib/python2.7/site-packages/celery/beat.py", line 342, in
> >>>>>>>> publisher
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>
> >>>>>>>>>> return self.Publisher(self._ensure_connected())
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>>> File
> >>>>>>>>>> "/usr/lib/python2.7/site-packages/celery/beat.py", line 326, in
> >>>>>>>>>> _ensure_connected
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>
> >>>>>>>>>> _error_handler, self.app.conf.BROKER_CONNECTION_MAX_RETRIES
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>>> File
> >>>>>>>>>> "/usr/lib/python2.7/site-packages/kombu/connection.py", line
> 369,
> >>>> in
> >>>>>>>>>> ensure_connection
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>
> >>>>>>>>>> interval_start, interval_step, interval_max, callback)
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>>> File
> >>>>>>>>>> "/usr/lib/python2.7/site-packages/kombu/utils/__init__.py",
> line
> >>>> 246,
> >>>>>>>> in
> >>>>>>>>>> retry_over_time
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>
> >>>>>>>>>> return fun(*args, **kwargs)
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>>> File
> >>>>>>>>>> "/usr/lib/python2.7/site-packages/kombu/connection.py", line
> 237,
> >>>> in
> >>>>>>>> connect
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>
> >>>>>>>>>> return self.connection
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>>> File
> >>>>>>>>>> "/usr/lib/python2.7/site-packages/kombu/connection.py", line
> 741,
> >>>> in
> >>>>>>>>>> connection
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>
> >>>>>>>>>> self._connection = self._establish_connection()
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>>> File
> >>>>>>>>>> "/usr/lib/python2.7/site-packages/kombu/connection.py", line
> 696,
> >>>> in
> >>>>>>>>>> _establish_connection
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>
> >>>>>>>> conn
> >>>>>>>>>> = self.transport.establish_connection()
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>>> File
> >>>>>>>>>> "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py",
> line
> >>>>>> 1600,
> >>>>>>>> in
> >>>>>>>>>> establish_connection
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>
> >>>>>>>> conn
> >>>>>>>>>> = self.Connection(**opts)
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>>> File
> >>>>>>>>>> "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py",
> line
> >>>>>> 1261,
> >>>>>>>> in
> >>>>>>>>>> __init__
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>
> >>>>>>>>>> self._qpid_conn = establish(**self.connection_options)
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>>> File
> >>>>>>>>>> "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py",
> >>>> line
> >>>>>>>> 112,
> >>>>>>>>>> in establish
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>
> >>>>>>>>>> conn.open(timeout=timeout)
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>>> File
> >>>>>>>>>> "<string>", line 6, in open
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>>> File
> >>>>>>>>>> "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py",
> >>>> line
> >>>>>>>> 323,
> >>>>>>>>>> in open
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>
> >>>>>>>>>> self.attach(timeout=timeout)
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>>> File
> >>>>>>>>>> "<string>", line 6, in attach
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>>> File
> >>>>>>>>>> "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py",
> >>>> line
> >>>>>>>> 343,
> >>>>>>>>>> in attach
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>
> >>>>>>
> >>>>>>>>>> raise Timeout("Connection attach timed out")
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL:
> >> (7070-59168)
> >>>>>>>> Timeout:
> >>>>>>>>>> Connection attach timed out
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 celery: celery beat v3.1.11 (Cipater)
> >> is
> >>>>>>>> starting.
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 celery: __ - ... __ -
> _
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 celery: Configuration ->
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 celery: . broker ->
> >>>>>>>>>> qpid://dscaprv01.corp.acxiom.net:5671//
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 celery: . loader ->
> >>>>>>>> celery.loaders.app.AppLoader
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 celery: . scheduler ->
> >>>>>>>>>> pulp.server.async.scheduler.Scheduler
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 celery: . logfile -> [stderr]@%INFO
> >>>>>>>>>> Aug 1 08:45:03 dscaprv01 celery: . maxinterval -> now (0s)
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Monday, August 1, 2016 at 8:43:45 AM UTC-5, Chris Duryee
> >> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On 08/01/2016 09:39 AM, Rick Langston wrote:
> >>>>>>>>>>>> The backend service all say ok but when a run a
> katelli-service
> >>>>>>>> status I
> >>>>>>>>>>>> can see that celery.bet fails to status. if I restart the
> >> server
> >>>> it
> >>>>>>>> and
> >>>>>>>>>>>> immediately check the status it says running but checking
> the
> >>>>>> status
> >>>>>>>>>>> again
> >>>>>>>>>>>> shows it timed out.
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Is this the sequence of events?
> >>>>>>>>>>>
> >>>>>>>>>>> * service pulp_celerybeat start (outputs success)
> >>>>>>>>>>> * service pulp_celerybeat status (outputs success)
> >>>>>>>>>>> * wait some number of seconds
> >>>>>>>>>>> * service pulp_celerybeat status (outputs error)
> >>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> No memory errros noted
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> [root@dscaprv01 tmp]# systemctl status
> pulp_celerybeat.service
> >>>>>>>>>>>> ● pulp_celerybeat.service - Pulp's Celerybeat
> >>>>>>>>>>>> Loaded: loaded
> >>>> (/usr/lib/systemd/system/pulp_celerybeat.service;
> >>>>>>>>>>>> enabled; vendor preset: disabled)
> >>>>>>>>>>>> Active: inactive (dead) since Mon 2016-08-01 08:34:31 CDT;
> >> 18s
> >>>>>> ago
> >>>>>>>>>>>> Process: 5887 ExecStart=/usr/bin/celery beat
> >>>>>>>>>>>> --app=pulp.server.async.celery_instance.celery
> >>>>>>>>>>>> --scheduler=pulp.server.async.scheduler.Scheduler
> (code=exited,
> >>>>>>>>>>>> status=0/SUCCESS)
> >>>>>>>>>>>> Main PID: 5887 (code=exited, status=0/SUCCESS)
> >>>>>>>>>>>>
> >>>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net pulp[5887]:
> >>>>>>>>>>> celery.beat:CRITICAL:
> >>>>>>>>>>>> (5887-79264) raise Timeout("Connection attach timed out")
> >>>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net pulp[5887]:
> >>>>>>>>>>> celery.beat:CRITICAL:
> >>>>>>>>>>>> (5887-79264) Timeout: Connection attach timed out
> >>>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]:
> celery
> >>>>>> beat
> >>>>>>>>>>> v3.1.11
> >>>>>>>>>>>> (Cipater) is starting.
> >>>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: __
> >> -
> >>>>
> >>>>>>>> ...
> >>>>>>>>>>> __
> >>>>>>>>>>>> - _
> >>>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]:
> >>>>>>>> Configuration
> >>>>>>>>>>> ->
> >>>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
> >> broker
> >>>>>> ->
> >>>>>>>>>>>> qpid://dscaprv01.corp.acxiom.net:5671//
> >>>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
> >> loader
> >>>>>> ->
> >>>>>>>>>>>> celery.loaders.app.AppLoader
> >>>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
> >>>>>> scheduler
> >>>>>>>> ->
> >>>>>>>>>>>> pulp.server.async.scheduler.Scheduler
> >>>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
> >>>> logfile
> >>>>>> ->
> >>>>>>>>>>>> [stderr]@%INFO
> >>>>>>>>>>>> Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
> >>>>>>>> maxinterval
> >>>>>>>>>>> ->
> >>>>>>>>>>>> now (0s)
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Monday, August 1, 2016 at 8:22:46 AM UTC-5, Chris Duryee
> >>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 08/01/2016 08:50 AM, Rick Langston wrote:
> >>>>>>>>>>>>>> I do see this issue in messages but not sure if its related
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL:
> >>>>>> (28691-76416)
> >>>>>>>>>>> beat
> >>>>>>>>>>>>>> raised exception <class
> 'qpid.messaging.exceptions.Timeout'>:
> >>>>>>>>>>>>>> Timeout('Connection attach timed out',)
> >>>>>>>>>>>>>> Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL:
> >>>>>> (28691-76416)
> >>>>>>>>>>>>>> Traceback (most recent call last):
> >>>>>>>>>>>>>> Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL:
> >>>>>> (28691-76416)
> >>>>>>>>
> >>>>>>>>>>>>> File
> >>>>>>>>>>>>>> "/usr/lib/python2.7/site-packages/celery/apps/beat.py",
> line
> >>>> 112,
> >>>>>>>> in
> >>>>>>>>>>>>>> start_scheduler
> >>>>>>>>>>>>>> Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL:
> >>>>>> (28691-76416)
> >>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>>> beat.start()
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> That is the likely culprit:)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Next time your task hangs, check in the "/about" page on
> your
> >>>>>>>> Katello
> >>>>>>>>>>>>> instance and ensure everything under "Backend System Status"
> >>>> says
> >>>>>>>> "OK"
> >>>>>>>>>>>>> with no further message.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> If there are pulp errors, a possible quick fix is to ensure
> >>>> qpidd
> >>>>>> is
> >>>>>>>>>>>>> still running, then restart pulp_workers, pulp_celerybeat
> and
> >>>>>>>>>>>>> pulp_resource_manager. I suspect your task will get picked
> up
> >>>>>> after
> >>>>>>>>>>> that.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Also, please check dmesg for out-of-memory errors. There are
> >>>> some
> >>>>>>>> other
> >>>>>>>>>>>>> possible things we can check, but I would be curious first
> >> about
> >>>>>> the
> >>>>>>>>>>>>> backend system status output.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Monday, August 1, 2016 at 7:12:26 AM UTC-5, Chris Duryee
> >>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 08/01/2016 07:54 AM, Rick Langston wrote:
> >>>>>>>>>>>>>>>> Hello
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I seem to be having some odd behavior with this version.
> >>>> With
> >>>>>> a
> >>>>>>>>>>> fresh
> >>>>>>>>>>>>>>>> install on centos 7 I have setup a product which
> completes
> >>>>>>>> normally
> >>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>> when I discover a repo and save them i get these meta
> data
> >>>> task
> >>>>>>>> that
> >>>>>>>>>>>>>>> seem
> >>>>>>>>>>>>>>>> to just wait forever. Any ideas what can be the culprit
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Are there any related errors in /var/log/messages?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Action:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Actions::Pulp::Repository::DistributorPublish
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> State: waiting for Pulp to start the task
> >>>>>>>>>>>>>>>> Input:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> {"pulp_id"=>"test-centos-6_updates_x86_64",
> >>>>>>>>>>>>>>>> "distributor_type_id"=>"yum_distributor",
> >>>>>>>>>>>>>>>> "source_pulp_id"=>nil,
> >>>>>>>>>>>>>>>> "dependency"=>nil,
> >>>>>>>>>>>>>>>> "remote_user"=>"admin",
> >>>>>>>>>>>>>>>> "remote_cp_user"=>"admin",
> >>>>>>>>>>>>>>>> "locale"=>"en"}
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Output:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> {"pulp_tasks"=>
> >>>>>>>>>>>>>>>> [{"exception"=>nil,
> >>>>>>>>>>>>>>>>
> >> "task_type"=>"pulp.server.managers.repo.publish.publish",
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>
> "_href"=>"/pulp/api/v2/tasks/a40815d5-9ba4-463a-8216-338cdcc4b1cc/",
> >>>>>>>>>>>>>>>> "task_id"=>"a40815d5-9ba4-463a-8216-338cdcc4b1cc",
> >>>>>>>>>>>>>>>> "tags"=>
> >>>>>>>>>>>>>>>> ["pulp:repository:test-centos-6_updates_x86_64",
> >>>>>>>>>>>>>>> "pulp:action:publish"],
> >>>>>>>>>>>>>>>> "finish_time"=>nil,
> >>>>>>>>>>>>>>>> "_ns"=>"task_status",
> >>>>>>>>>>>>>>>> "start_time"=>nil,
> >>>>>>>>>>>>>>>> "traceback"=>nil,
> >>>>>>>>>>>>>>>> "spawned_tasks"=>[],
> >>>>>>>>>>>>>>>> "progress_report"=>{},
> >>>>>>>>>>>>>>>> "queue"=>"None.dq",
> >>>>>>>>>>>>>>>> "state"=>"waiting",
> >>>>>>>>>>>>>>>> "worker_name"=>nil,
> >>>>>>>>>>>>>>>> "result"=>nil,
> >>>>>>>>>>>>>>>> "error"=>nil,
> >>>>>>>>>>>>>>>> "_id"=>{"$oid"=>"579f32aa95c48c6a54257674"},
> >>>>>>>>>>>>>>>> "id"=>"579f32aa95c48c6a54257674"}],
> >>>>>>>>>>>>>>>> "poll_attempts"=>{"total"=>100, "failed"=>0}}
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
>
--
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
