After trying the things from my other e-mail. When your system is in its "bad state" show the `qpid-stat -q` output too please.
The `qpid-stat -q` output from this e-mail shows all of the queues are empty with msgIn and msgOut being the same. -Brian On 05/04/2016 05:07 AM, Mallick, Samiron wrote: > *Seems the queue has something. I deleted them up one-by-one, and tried > starting the stuck repo sync. and it works for the repo finally. But bad > news is, the queue again regenerates lot of tasks automatically where no > sync tasks running. So while trying to run another repo, it went to > Waiting again. Is there any workaround yet?* > > > > # qpid-stat -q > > Queues > > queue > dur autoDel excl msg msgIn msgOut bytes bytesIn bytesOut cons > bind > > > ==================================================================================================================================================== > > > 1a4b6e57-3ecc-406d-84cd-29b24a0a6610:1.0 Y > > Y 0 2 2 0 486 486 1 2 > > > 36e7ca4b-5a0d-4f5c-9f94-a22016390562:1.0 Y > > Y 0 8 8 0 4.91k 4.91k 1 2 > > > 36e7ca4b-5a0d-4f5c-9f94-a22016390562:2.0 Y > > Y 0 4 4 0 2.50k 2.50k 1 2 > > > 3de2643d-bb8d-4e98-94d1-d8ed4e1bdf11:1.0 Y > > Y 0 8 8 0 4.88k 4.88k 1 2 > > > 3de2643d-bb8d-4e98-94d1-d8ed4e1bdf11:2.0 Y > > Y 0 4 4 0 2.52k 2.52k 1 2 > > > 43099b2b-cc78-4b96-a1a9-50d94517c1e2:1.0 Y > > Y 0 2 2 0 486 486 1 2 > > > 4409c371-0d54-44c4-94b7-ec0bb7ecfd45:1.0 Y > > Y 0 0 0 0 0 0 1 2 > > 680eb17a-8285-450c-b8b9-51d107b4ff2d:0.0 > Y Y 0 0 0 0 0 > 0 1 2 > > > bcbc1fa3-8157-403d-8f33-252fe057587a:1.0 Y > > Y 0 5 5 0 2.67k 2.67k 1 2 > > celery > Y > 0 0 0 0 0 0 1 2 > > > celeryev.4021d653-24bf-4f06-9aee-aa457c579c4b Y > > 0 12 12 0 10.0k 10.0k 1 2 > > pulp.task > Y 0 0 0 0 0 0 > 3 1 > > [email protected] > <mailto:[email protected]> > Y 0 0 0 0 0 0 1 2 > > [email protected] > <mailto:[email protected]> Y > Y 0 0 0 0 0 0 1 2 > > resource_manager > Y 0 0 0 0 0 0 > 1 2 > > [email protected] > <mailto:[email protected]> > Y 0 0 0 0 0 0 1 2 > > [email protected] > <mailto:[email protected]> Y > Y 0 0 0 0 0 0 1 2 > > # pulp-admin tasks list > > +----------------------------------------------------------------------+ > > Tasks > > +----------------------------------------------------------------------+ > > > > No tasks found > > > > > > > > # qpid-tool > > Management Tool for QPID > > qpid: list > > Summary of Objects by Type: > > qpid: help > > Management Tool for QPID > > > > Commands: > > agents - Print a list of the known Agents > > list - Print summary of existing objects > by class > > list <className> - Print list of objects of the > specified class > > list <className> active - Print list of non-deleted objects > of the specified class > > show <ID> - Print contents of an object (infer > className) > > call <ID> <methodName> [<args>] - Invoke a method on an object > > schema - Print summary of object classes > seen on the target > > schema <className> - Print details of an object class > > set time-format short - Select short timestamp format > (default) > > set time-format long - Select long timestamp format > > quit or ^D - Exit the program > > > > qpid: list > > Summary of Objects by Type: > > Package Class Active Deleted > > ============================================================ > > org.apache.qpid.broker binding 43 12 > > org.apache.qpid.broker broker 1 0 > > org.apache.qpid.broker memory 1 0 > > org.apache.qpid.broker system 1 0 > > org.apache.qpid.linearstore store 1 0 > > org.apache.qpid.broker subscription 23 5 > > org.apache.qpid.broker connection 14 1 > > org.apache.qpid.broker session 19 1 > > org.apache.qpid.linearstore journal 5 0 > > org.apache.qpid.acl acl 1 0 > > org.apache.qpid.broker queue 21 5 > > org.apache.qpid.broker exchange 13 0 > > org.apache.qpid.broker vhost 1 0 > > qpid: list queue > > Object Summary: > > ID Created Destroyed Index > > > ============================================================================================================================ > > 114 06:24:09 06:24:41 > org.apache.qpid.broker:queue:topic-mysrv.3108.1 > > 115 06:24:09 06:24:41 > org.apache.qpid.broker:queue:reply-mysrv.3108.1 > > 116 06:24:09 06:24:41 > org.apache.qpid.broker:queue:qmfc-v2-ui-mysrv.3108.1 > > 117 06:24:09 06:24:41 > org.apache.qpid.broker:queue:qmfc-v2-mysrv.3108.1 > > 118 06:24:09 06:24:41 > org.apache.qpid.broker:queue:qmfc-v2-hb-mysrv.3108.1 > > 198 06:16:36 - > org.apache.qpid.broker:queue:1a4b6e57-3ecc-406d-84cd-29b24a0a6610:1.0 > > 199 06:16:36 - > org.apache.qpid.broker:queue:36e7ca4b-5a0d-4f5c-9f94-a22016390562:1.0 > > 200 06:16:38 - > org.apache.qpid.broker:queue:36e7ca4b-5a0d-4f5c-9f94-a22016390562:2.0 > > 201 06:16:36 - > org.apache.qpid.broker:queue:3de2643d-bb8d-4e98-94d1-d8ed4e1bdf11:1.0 > > 202 06:16:37 - > org.apache.qpid.broker:queue:3de2643d-bb8d-4e98-94d1-d8ed4e1bdf11:2.0 > > 203 06:16:36 - > org.apache.qpid.broker:queue:43099b2b-cc78-4b96-a1a9-50d94517c1e2:1.0 > > 204 06:16:33 - > org.apache.qpid.broker:queue:4409c371-0d54-44c4-94b7-ec0bb7ecfd45:1.0 > > 205 06:16:33 - > org.apache.qpid.broker:queue:bcbc1fa3-8157-403d-8f33-252fe057587a:1.0 > > 206 06:16:33 - org.apache.qpid.broker:queue:celery > > 207 06:16:33 - > org.apache.qpid.broker:queue:celeryev.4021d653-24bf-4f06-9aee-aa457c579c4b > > 208 06:16:33 - org.apache.qpid.broker:queue:pulp.task > > 209 06:24:43 - > org.apache.qpid.broker:queue:qmfc-v2-hb-mysrv.3122.1 > > 210 06:24:43 - > org.apache.qpid.broker:queue:qmfc-v2-mysrv.3122.1 > > 211 06:24:43 - > org.apache.qpid.broker:queue:qmfc-v2-ui-mysrv.3122.1 > > 212 06:24:43 - > org.apache.qpid.broker:queue:reply-mysrv.3122.1 > > 213 06:16:37 - > org.apache.qpid.broker:queue:[email protected] > > 214 06:16:36 - > org.apache.qpid.broker:queue:[email protected] > > 215 06:16:33 - org.apache.qpid.broker:queue:resource_manager > > 216 06:16:38 - > org.apache.qpid.broker:queue:[email protected] > > 217 06:16:37 - > org.apache.qpid.broker:queue:[email protected] > > 218 06:24:43 - > org.apache.qpid.broker:queue:topic-mysrv.3122.1 > > > > # qpid-config del queue 1a4b6e57-3ecc-406d-84cd-29b24a0a6610:1.0 > > Failed: Exception: Exception from Agent: {u'error_code': 7, > u'error_text': 'precondition-failed: Cannot delete queue > 1a4b6e57-3ecc-406d-84cd-29b24a0a6610:1.0; queue in use > (/builddir/build/BUILD/qpid-cpp-0.34/src/qpid/broker/Broker.cpp:1068)'} > > > > # qpid-config del queue [email protected] > <mailto:[email protected]> --force > > > > qpid: list > > Summary of Objects by Type: > > Package Class Active Deleted > > ============================================================ > > org.apache.qpid.broker binding 16 0 > > org.apache.qpid.broker broker 1 0 > > org.apache.qpid.broker memory 1 0 > > org.apache.qpid.broker system 1 0 > > org.apache.qpid.linearstore store 1 0 > > org.apache.qpid.broker subscription 7 0 > > org.apache.qpid.broker connection 13 0 > > org.apache.qpid.broker session 269 0 > > org.apache.qpid.acl acl 1 0 > > org.apache.qpid.broker queue 7 0 > > org.apache.qpid.broker exchange 13 0 > > org.apache.qpid.broker vhost 1 0 > > qpid: list queue > > Object Summary: > > ID Created Destroyed Index > > > ====================================================================================================== > > 146 08:47:30 - > org.apache.qpid.broker:queue:2d1a7c8f-bc3b-4d54-bbe6-b7b264530506:1.0 > > 147 08:47:30 - > org.apache.qpid.broker:queue:celeryev.d45c6bc2-2449-4700-b3bb-bbbbf0b2990b > > 148 08:52:24 - > org.apache.qpid.broker:queue:qmfc-v2-hb-mysrv.4080.1 > > 149 08:52:24 - > org.apache.qpid.broker:queue:qmfc-v2-mysrv.4080.1 > > 150 08:52:24 - > org.apache.qpid.broker:queue:qmfc-v2-ui-mysrv.4080.1 > > 151 08:52:24 - > org.apache.qpid.broker:queue:reply-mysrv.4080.1 > > 152 08:52:24 - > org.apache.qpid.broker:queue:topic-mysrv.4080.1 > > > > # pulp-admin tasks list > > +----------------------------------------------------------------------+ > > Tasks > > +----------------------------------------------------------------------+ > > > > No tasks found > > > > # pulp-admin rpm repo sync run --repo-id=rhel-6-server-rpms > > +----------------------------------------------------------------------+ > > Synchronizing Repository [rhel-6-server-rpms] > > +----------------------------------------------------------------------+ > > > > This command may be exited via ctrl+c without affecting the request. > > > > > > [\] > > *** STUCK *** > > > > On Wed, May 4, 2016 at 9:37 AM, Mallick, Samiron > <[email protected] <mailto:[email protected]>> wrote: > > Hey Brian, thanks for the reply. > > *From the below output I could see "resource_worker-1" is > responsible for this task and I have 4 worker displayed on the > server.____* > > __ __ > > # pulp-admin tasks list____ > > > +----------------------------------------------------------------------+____ > > Tasks____ > > > +----------------------------------------------------------------------+____ > > __ __ > > Operations: sync____ > > Resources: rhel-6-server-supplementary-rpms (repository)____ > > State: Waiting____ > > Start Time: Unstarted____ > > Finish Time: Incomplete____ > > Task Id: 49b83f70-e6d6-4cdb-9c5a-93c20c31d697____ > > __ __ > > __ __ > > # pulp-admin -vv tasks details --task-id > 49b83f70-e6d6-4cdb-9c5a-93c20c31d697____ > > > +----------------------------------------------------------------------+____ > > Task Details____ > > > +----------------------------------------------------------------------+____ > > __ __ > > 2016-05-04 04:55:33,231 - DEBUG - sending GET request to > /pulp/api/v2/tasks/49b83f70-e6d6-4cdb-9c5a-93c20c31d697/____ > > 2016-05-04 04:55:33,362 - INFO - GET request to > /pulp/api/v2/tasks/49b83f70-e6d6-4cdb-9c5a-93c20c31d697/ with > parameters None____ > > 2016-05-04 04:55:33,362 - INFO - Response status : 200____ > > __ __ > > 2016-05-04 04:55:33,363 - INFO - Response body :____ > > {____ > > "exception": null,____ > > "task_type": "pulp.server.managers.repo.sync.sync",____ > > "_href": > "/pulp/api/v2/tasks/49b83f70-e6d6-4cdb-9c5a-93c20c31d697/",____ > > "task_id": "49b83f70-e6d6-4cdb-9c5a-93c20c31d697",____ > > "tags": [____ > > "pulp:repository:rhel-6-server-supplementary-rpms",____ > > "pulp:action:sync"____ > > ],____ > > "finish_time": null,____ > > "_ns": "task_status",____ > > "start_time": null,____ > > "traceback": null,____ > > "spawned_tasks": [],____ > > "progress_report": {},____ > > "queue": "[email protected]",____ > > "state": "waiting",____ > > "worker_name": "reserved_resource_worker-1@mysrv",____ > > "result": null,____ > > "error": null,____ > > "_id": {____ > > "$oid": "572964399b70a2ea1d2694aa"____ > > },____ > > "id": "572964399b70a2ea1d2694aa"____ > > }____ > > __ __ > > Operations: sync____ > > Resources: rhel-6-server-supplementary-rpms (repository)____ > > State: Waiting____ > > Start Time: Unstarted____ > > Finish Time: Incomplete____ > > Result: Incomplete____ > > Task Id: 49b83f70-e6d6-4cdb-9c5a-93c20c31d697____ > > Progress Report:____ > > __ __ > > __ __ > > # pulp-admin status____ > > > +----------------------------------------------------------------------+____ > > Status of the server____ > > > +----------------------------------------------------------------------+____ > > __ __ > > Api Version: 2____ > > Database Connection:____ > > Connected: True____ > > Known Workers:____ > > _id: scheduler@mysrv____ > > _ns: workers____ > > Last Heartbeat: 2016-05-04T02:53:34Z____ > > _id: reserved_resource_worker-3@mysrv____ > > _ns: workers____ > > Last Heartbeat: 2016-05-04T02:54:00Z____ > > _id: reserved_resource_worker-2@mysrv____ > > _ns: workers____ > > Last Heartbeat: 2016-05-04T02:54:00Z____ > > _id: resource_manager@mysrv____ > > _ns: workers____ > > Last Heartbeat: 2016-05-04T02:54:00Z____ > > _id: reserved_resource_worker-1@mysrv____ > > _ns: workers____ > > Last Heartbeat: 2016-05-04T02:54:01Z____ > > _id: reserved_resource_worker-0@mysrv____ > > _ns: workers____ > > Last Heartbeat: 2016-05-04T02:54:03Z____ > > Messaging Connection:____ > > Connected: True____ > > Versions:____ > > Platform Version: 2.8.2____ > > __ __ > > # ps -awfux | grep celery____ > > root 4637 0.0 0.0 112644 960 pts/0 S+ 04:56 > 0:00 \_ grep --color=auto celery____ > > apache 1592 0.0 1.4 667716 56368 ? Ssl May03 0:26 > /usr/bin/python /usr/bin/celery worker -A pulp.server.async.app -n > resource_manager@%h -Q resource_manager -c 1 --events --umask 18 > --pidfile=/var/run/pulp/resource_manager.pid --heartbeat-interval=30____ > > apache 2921 0.0 1.4 667664 54296 ? Sl May03 0:13 \_ > /usr/bin/python /usr/bin/celery worker -A pulp.server.async.app -n > resource_manager@%h -Q resource_manager -c 1 --events --umask 18 > --pidfile=/var/run/pulp/resource_manager.pid --heartbeat-interval=30____ > > apache 1616 0.0 1.4 667996 56400 ? Ssl May03 0:27 > /usr/bin/python /usr/bin/celery worker -n > reserved_resource_worker-0@%h -A pulp.server.async.app -c 1 --events > --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-0.pid > --heartbeat-interval=30____ > > apache 2919 0.0 1.4 741536 54564 ? Sl May03 0:11 \_ > /usr/bin/python /usr/bin/celery worker -n > reserved_resource_worker-0@%h -A pulp.server.async.app -c 1 --events > --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-0.pid > --heartbeat-interval=30____ > > apache 1626 0.0 1.5 668560 59524 ? Ssl May03 0:29 > /usr/bin/python /usr/bin/celery worker -n > reserved_resource_worker-1@%h -A pulp.server.async.app -c 1 --events > --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-1.pid > --heartbeat-interval=30____ > > apache 4561 0.0 1.4 668560 56260 ? S 04:47 0:00 \_ > /usr/bin/python /usr/bin/celery worker -n > reserved_resource_worker-1@%h -A pulp.server.async.app -c 1 --events > --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-1.pid > --heartbeat-interval=30____ > > apache 1631 0.0 1.5 667748 58508 ? Ssl May03 0:27 > /usr/bin/python /usr/bin/celery worker -n > reserved_resource_worker-2@%h -A pulp.server.async.app -c 1 --events > --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-2.pid > --heartbeat-interval=30____ > > apache 2922 4.2 8.0 1042956 311476 ? Sl May03 48:25 \_ > /usr/bin/python /usr/bin/celery worker -n > reserved_resource_worker-2@%h -A pulp.server.async.app -c 1 --events > --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-2.pid > --heartbeat-interval=30____ > > apache 1637 0.0 1.4 667744 56368 ? Ssl May03 0:27 > /usr/bin/python /usr/bin/celery worker -n > reserved_resource_worker-3@%h -A pulp.server.async.app -c 1 --events > --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-3.pid > --heartbeat-interval=30____ > > apache 2920 0.0 1.4 815420 54760 ? Sl May03 0:13 \_ > /usr/bin/python /usr/bin/celery worker -n > reserved_resource_worker-3@%h -A pulp.server.async.app -c 1 --events > --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-3.pid > --heartbeat-interval=30____ > > apache 4620 6.5 0.8 663652 31432 ? Ssl 04:56 0:00 > /usr/bin/python /usr/bin/celery beat > --app=pulp.server.async.celery_instance.celery > --scheduler=pulp.server.async.scheduler.Scheduler____ > > __ __ > > __ __ > > *As I saw errors in output of pulp_worker-1, I restarted each worker > individually, and all error seems gone.____* > > __ __ > > # systemctl status pulp_workers.service____ > > ● pulp_workers.service - Pulp Celery Workers____ > > Loaded: loaded (/usr/lib/systemd/system/pulp_workers.service; > enabled; vendor preset: disabled)____ > > Active: active (exited) since Wed 2016-05-04 05:36:38 CEST; 3s > ago____ > > Process: 5717 ExecStop=/usr/bin/python -m > pulp.server.async.manage_workers stop (code=exited, > status=0/SUCCESS)____ > > Process: 5731 ExecStart=/usr/bin/python -m > pulp.server.async.manage_workers start (code=exited, > status=0/SUCCESS)____ > > Main PID: 5731 (code=exited, status=0/SUCCESS)____ > > __ __ > > May 04 05:36:38 mysrv systemd[1]: Starting Pulp Celery Workers...____ > > May 04 05:36:38 mysrv systemd[1]: Started Pulp Celery Workers.____ > > __ __ > > # systemctl status pulp_worker-0____ > > ? pulp_worker-0.service - Pulp Worker #0____ > > Loaded: loaded (/run/systemd/system/pulp_worker-0.service; > static; vendor preset: disabled)____ > > Active: active (running) since Wed 2016-05-04 05:10:44 CEST; 1min > 26s ago____ > > Main PID: 4753 (celery)____ > > CGroup: /system.slice/pulp_worker-0.service____ > > +-4753 /usr/bin/python /usr/bin/celery worker -n > reserved_resource_worker-0@%h -A pulp.server.async.app -c 1 --events > --umask 18 --pidfile=/var...____ > > +-4766 /usr/bin/python /usr/bin/celery worker -n > reserved_resource_worker-0@%h -A pulp.server.async.app -c 1 --events > --umask 18 --pidfile=/var...____ > > __ __ > > May 04 05:10:46 mysrv celery[4753]: - ** ---------- .> transport: > qpid://mysrv:5672// <http://srpadinf0244.mgt.insim.biz:5672//>____ > > May 04 05:10:46 mysrv celery[4753]: - ** ---------- .> results: > disabled____ > > May 04 05:10:46 mysrv celery[4753]: - *** --- * --- .> concurrency: > 1 (prefork)____ > > May 04 05:10:46 mysrv celery[4753]: -- ******* ----____ > > May 04 05:10:46 mysrv celery[4753]: --- ***** ----- [queues]____ > > May 04 05:10:46 mysrv celery[4753]: -------------- .> > celery exchange=celery(direct) key=celery____ > > May 04 05:10:46 mysrv celery[4753]: .> > [email protected] exchange=C.dq(direct) > key=rese...s <http://sim.biz/>rv____ > > May 04 05:10:46 mysrv pulp[4753]: kombu.transport.qpid:INFO: > Connected to qpid with SASL mechanism ANONYMOUS____ > > May 04 05:10:46 mysrv pulp[4753]: celery.worker.consumer:INFO: > Connected to qpid://mysrv:5672// > <http://srpadinf0244.mgt.insim.biz:5672//>____ > > May 04 05:10:46 mysrv pulp[4753]: kombu.transport.qpid:INFO: > Connected to qpid with SASL mechanism ANONYMOUS____ > > Hint: Some lines were ellipsized, use -l to show in full.____ > > __ __ > > __ __ > > # systemctl status pulp_worker-1____ > > ? pulp_worker-1.service - Pulp Worker #1____ > > Loaded: loaded (/run/systemd/system/pulp_worker-1.service; > static; vendor preset: disabled)____ > > Active: active (running) since Wed 2016-05-04 05:08:16 CEST; 3min > 57s ago____ > > Main PID: 4718 (celery)____ > > CGroup: /system.slice/pulp_worker-1.service____ > > +-4718 /usr/bin/python /usr/bin/celery worker -n > reserved_resource_worker-1@%h -A pulp.server.async.app -c 1 --events > --umask 18 --pidfile=/var...____ > > +-4733 /usr/bin/python /usr/bin/celery worker -n > reserved_resource_worker-1@%h -A pulp.server.async.app -c 1 --events > --umask 18 --pidfile=/var...____ > > __ __ > > May 04 05:08:23 mysrv pulp[4718]: celery.worker.strategy:INFO: > Received task: > pulp.server.controllers.repository.download_def...3cc3c36]____ > > May 04 05:08:23 mysrv pulp[4718]: celery.worker.strategy:INFO: > Received task: > pulp.server.controllers.repository.download_def...ce7430b]____ > > May 04 05:08:23 mysrv pulp[4718]: celery.worker.job:INFO: Task > pulp.server.controllers.repository.download_deferred[aad88f32-...9s: > None____ > > May 04 05:08:23 mysrv pulp[4718]: celery.worker.strategy:INFO: > Received task: > pulp.server.db.reaper.reap_expired_documents[02...8322faa]____ > > May 04 05:08:23 mysrv pulp[4718]: celery.worker.strategy:INFO: > Received task: > pulp.server.controllers.repository.download_def...ddadf87]____ > > May 04 05:08:23 mysrv pulp[4718]: celery.worker.strategy:INFO: > Received task: > pulp.server.controllers.repository.download_def...d0cf8c6]____ > > May 04 05:08:23 mysrv pulp[4718]: celery.worker.strategy:INFO: > Received task: > pulp.server.controllers.repository.download_def...72edf98]____ > > May 04 05:08:23 mysrv pulp[4718]: celery.worker.strategy:INFO: > Received task: > pulp.server.controllers.repository.download_def...1e9e4bc]____ > > May 04 05:08:23 mysrv pulp[4718]: celery.worker.strategy:INFO: > Received task: > pulp.server.controllers.repository.download_def...30f8627]____ > > May 04 05:08:23 mysrv pulp[4718]: celery.worker.strategy:INFO: > Received task: > pulp.server.controllers.repository.queue_downlo...fd23e13]____ > > Hint: Some lines were ellipsized, use -l to show in full.____ > > __ __ > > __ __ > > # systemctl status pulp_worker-2____ > > ? pulp_worker-2.service - Pulp Worker #2____ > > Loaded: loaded (/run/systemd/system/pulp_worker-2.service; > static; vendor preset: disabled)____ > > Active: active (running) since Wed 2016-05-04 05:11:06 CEST; 1min > 10s ago____ > > Main PID: 4776 (celery)____ > > CGroup: /system.slice/pulp_worker-2.service____ > > +-4776 /usr/bin/python /usr/bin/celery worker -n > reserved_resource_worker-2@%h -A pulp.server.async.app -c 1 --events > --umask 18 --pidfile=/var...____ > > +-4789 /usr/bin/python /usr/bin/celery worker -n > reserved_resource_worker-2@%h -A pulp.server.async.app -c 1 --events > --umask 18 --pidfile=/var...____ > > __ __ > > May 04 05:11:07 mysrv celery[4776]: - ** ---------- .> transport: > qpid://mysrv:5672// <http://srpadinf0244.mgt.insim.biz:5672//>____ > > May 04 05:11:07 mysrv celery[4776]: - ** ---------- .> results: > disabled____ > > May 04 05:11:07 mysrv celery[4776]: - *** --- * --- .> concurrency: > 1 (prefork)____ > > May 04 05:11:07 mysrv celery[4776]: -- ******* ----____ > > May 04 05:11:07 mysrv celery[4776]: --- ***** ----- [queues]____ > > May 04 05:11:07 mysrv celery[4776]: -------------- .> > celery exchange=celery(direct) key=celery____ > > May 04 05:11:07 mysrv celery[4776]: .> > [email protected] > exchange=C.dq(direct) key=rese...s <http://sim.biz/>rv____ > > May 04 05:11:07 mysrv pulp[4776]: kombu.transport.qpid:INFO: > Connected to qpid with SASL mechanism ANONYMOUS____ > > May 04 05:11:07 mysrv pulp[4776]: celery.worker.consumer:INFO: > Connected to qpid://mysrv:5672// > <http://srpadinf0244.mgt.insim.biz:5672//>____ > > May 04 05:11:07 mysrv pulp[4776]: kombu.transport.qpid:INFO: > Connected to qpid with SASL mechanism ANONYMOUS____ > > Hint: Some lines were ellipsized, use -l to show in full.____ > > __ __ > > __ __ > > # systemctl status pulp_worker-3____ > > ? pulp_worker-3.service - Pulp Worker #3____ > > Loaded: loaded (/run/systemd/system/pulp_worker-3.service; > static; vendor preset: disabled)____ > > Active: active (running) since Wed 2016-05-04 05:11:21 CEST; 59s > ago____ > > Main PID: 4798 (celery)____ > > CGroup: /system.slice/pulp_worker-3.service____ > > +-4798 /usr/bin/python /usr/bin/celery worker -n > reserved_resource_worker-3@%h -A pulp.server.async.app -c 1 --events > --umask 18 --pidfile=/var...____ > > +-4811 /usr/bin/python /usr/bin/celery worker -n > reserved_resource_worker-3@%h -A pulp.server.async.app -c 1 --events > --umask 18 --pidfile=/var...____ > > __ __ > > May 04 05:11:22 mysrv celery[4798]: - ** ---------- .> transport: > qpid://mysrv:5672// <http://srpadinf0244.mgt.insim.biz:5672//>____ > > May 04 05:11:22 mysrv celery[4798]: - ** ---------- .> results: > disabled____ > > May 04 05:11:22 mysrv celery[4798]: - *** --- * --- .> concurrency: > 1 (prefork)____ > > May 04 05:11:22 mysrv celery[4798]: -- ******* ----____ > > May 04 05:11:22 mysrv celery[4798]: --- ***** ----- [queues]____ > > May 04 05:11:22 mysrv celery[4798]: -------------- .> > celery exchange=celery(direct) key=celery____ > > May 04 05:11:22 mysrv celery[4798]: .> > [email protected] > exchange=C.dq(direct) key=rese...s <http://sim.biz/>rv____ > > May 04 05:11:22 mysrv pulp[4798]: kombu.transport.qpid:INFO: > Connected to qpid with SASL mechanism ANONYMOUS____ > > May 04 05:11:22 mysrv pulp[4798]: celery.worker.consumer:INFO: > Connected to qpid://mysrv:5672// > <http://srpadinf0244.mgt.insim.biz:5672//>____ > > May 04 05:11:22 mysrv pulp[4798]: kombu.transport.qpid:INFO: > Connected to qpid with SASL mechanism ANONYMOUS____ > > Hint: Some lines were ellipsized, use -l to show in full.____ > > __ __ > > *Now I have no tasks queued at all. I ran the repo sync again. and > again it went to waiting.....____* > > __ __ > > # pulp-admin tasks list____ > > > +----------------------------------------------------------------------+____ > > Tasks____ > > > +----------------------------------------------------------------------+____ > > __ __ > > No tasks found____ > > __ __ > > # pulp-admin rpm repo sync run > --repo-id=rhel-6-server-supplementary-rpms____ > > > +----------------------------------------------------------------------+____ > > Synchronizing Repository [rhel-6-server-supplementary-rpms]____ > > > +----------------------------------------------------------------------+____ > > __ __ > > This command may be exited via ctrl+c without affecting the request.____ > > __ __ > > __ __ > > [/]____ > > Waiting to begin... > > > > *//* > > > On Wed, May 4, 2016 at 1:52 AM, Brian Bouterse <[email protected] > <mailto:[email protected]>> wrote: > > Kodiak is right that the second task stuck at "Waiting to Begin" is > likely waiting behind another operation on that same repo. > Canceling the > one prior will likely allow the later one to start. > > How many workers are running and how many do you expect? You can see > what Pulp thinks with: `pulp-admin status` > > You can compare that to your pulp processes on all of your Pulp > servers > with `sudo ps -awfux | grep celery`. > > Also you can look at the task details with -vv to see the worker the > halted task is assigned to. Something like `pulp-admin -vv tasks > details > --task-id 03842c9d-e053-4a6f-a4c4-2d7302be9c8c.` > > Unfortunately you'll have to see the worker in the raw response > with -vv > because of [0]. > > [0]: https://pulp.plan.io/issues/1832 > > -Brian > > > On 05/03/2016 11:53 AM, Kodiak Firesmith wrote: > > I believe you may need to cancel the pending repo sync task before > you > > can delete the repo. Maybe try: > > pulp-admin tasks cancel > --task-id=2d776d63-fd8a-4e0a-8f32-d2276c85187c > > pulp-admin tasks cancel > --task-id=03842c9d-e053-4a6f-a4c4-2d7302be9c8c > > > > Then: > > pulp-admin rpm repo delete > --repo-id=rhel-6-server-supplementary-rpms > > > > > > On Tue, May 3, 2016 at 11:47 AM, Mallick, Samiron > > <[email protected] <mailto:[email protected]> > <mailto:[email protected] > <mailto:[email protected]>>> wrote: > > > > Could anyone please tell me what went wrong with the > repository. One > > of my EL7 server registered and was able to fetch contents > from CDN. > > Recently I found one of the repo stuck after downloading > RPMs. It’s > > never ending. I rebooted my server, cancelled tasks, > deleted the > > repo and recreated, but no luck. No if I run sync, it > directly going > > to waiting stage. Earlier I observed it was starting the > task but > > the start time was same as I ran the job first time. Even > now I am > > not able to delete the repo as well as it is showing > “Waiting to > > begin”. I am running Pulp v2.8. Any idea would be greatly > appreciated. > > > > > > > > # rpm -qa pulp-server > > > > pulp-server-2.8.2-1.el7.noarch > > > > > > > > # pulp-admin rpm repo sync run > > --repo-id=rhel-6-server-supplementary-rpms > > > > > > +----------------------------------------------------------------------+ > > > > Synchronizing Repository > [rhel-6-server-supplementary-rpms] > > > > > > +----------------------------------------------------------------------+ > > > > > > > > This command may be exited via ctrl+c without affecting > the request. > > > > > > > > > > > > Downloading metadata... > > > > [\] > > > > ... completed > > > > > > > > Downloading repository content... > > > > [-] > > > > [==================================================] 100% > > > > RPMs: 0/0 items > > > > Delta RPMs: 0/0 items > > > > > > > > ... completed > > > > > > > > Downloading distribution files... > > > > [==================================================] 100% > > > > Distributions: 0/0 items > > > > ... completed > > > > > > > > Importing errata... > > > > [/] > > > > ... completed > > > > > > > > Importing package groups/categories... > > > > [-] > > > > ... completed > > > > > > > > Cleaning duplicate packages... > > > > [|] > > > > ... completed > > > > > > > > *** AND STUCK HERE *** > > > > > > > > # pulp-admin tasks list > > > > > > +----------------------------------------------------------------------+ > > > > Tasks > > > > > > +----------------------------------------------------------------------+ > > > > > > > > Operations: sync > > > > Resources: rhel-6-server-supplementary-rpms (repository) > > > > State: Running > > > > Start Time: 2016-05-03T07:06:36Z > > > > Finish Time: Incomplete > > > > Task Id: 2d776d63-fd8a-4e0a-8f32-d2276c85187c > > > > > > > > Operations: publish > > > > Resources: rhel-6-server-supplementary-rpms (repository) > > > > State: Waiting > > > > Start Time: Unstarted > > > > Finish Time: Incomplete > > > > Task Id: 03842c9d-e053-4a6f-a4c4-2d7302be9c8c > > > > > > > > # date > > > > Tue May 3 09:22:30 CEST 2016 > > > > # pulp-admin rpm repo sync schedules list > > --repo-id=rhel-6-server-supplementary-rpms > > > > > > +----------------------------------------------------------------------+ > > > > Schedules > > > > > > +----------------------------------------------------------------------+ > > > > > > > > There are no schedules defined for this operation. > > > > > > > > # pulp-admin rpm repo delete > --repo-id=rhel-6-server-supplementary-rpms > > > > This command may be exited via ctrl+c without affecting > the request. > > > > > > > > > > > > [-] > > > > Running... > > > > [-] > > > > Waiting to begin... > > > > > > > > *** AND STUCK HERE *** > > > > > > _______________________________________________ > > Pulp-list mailing list > > [email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>> > > https://www.redhat.com/mailman/listinfo/pulp-list > > > > > > > > > > _______________________________________________ > > Pulp-list mailing list > > [email protected] <mailto:[email protected]> > > https://www.redhat.com/mailman/listinfo/pulp-list > > > > _______________________________________________ > Pulp-list mailing list > [email protected] <mailto:[email protected]> > https://www.redhat.com/mailman/listinfo/pulp-list > > > _______________________________________________ Pulp-list mailing list [email protected] https://www.redhat.com/mailman/listinfo/pulp-list
