Nevermind, things are looking prettier now. I unregistered that consumer again, deleted a ton of tasks that were in Waiting state, and rebooted the pulp server for good measure. I restarted goferd on the consumer and reregistered it. Seems fine now. It was like that particular consumer was filling up the qpid message queue or something along those lines. Also, before I rebooted the pulp server, I tried restarting pulp_workers again, but it was spinning its wheels/waiting trying to shut down one of the workers.
From: [email protected] [mailto:[email protected]] On Behalf Of Ashby, Jason (IMS) Sent: Thursday, November 13, 2014 4:13 PM To: [email protected] Subject: [Pulp-list] ConnectionError: Enqueue capacity threshold exceeded on queue Consumer and server: Pulp 2.4.3, CentOS 6.5. The consumers were upgraded in-place from 2.3. The server is a fresh 2.4.3 install. Hi guys, I keep hitting this error that seems to be clogging up my pulp server and preventing consumers from communicating with the server. This ERROR seems to complain about the same consumer (log08.example.com), so I've tried unregistering and reregistering that consumer from both the pulp server and consumer sides, as well as yum reinstalling pulp and gofer, but no luck. I see some old discussions<https://www.redhat.com/archives/pulp-list/2011-March/msg00092.html> about this, but no clear solution. Sounds like an bug that was resolved in newer versions of qpidd (>0.24-ish) Any ideas, or troubleshooting tips? Jason Here is most of the traceback from /var/log/messages on the server: ...snip... Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: value = method(self, *args, **kwargs) Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/controllers/consumers.py", line 238, in POST Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: consumer_id, repo_id, distributor_id, notify_agent, binding_config, options) Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/tasks/consumer.py", line 52, in bind Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: task = agent_manager.bind(consumer_id, repo_id, distributor_id, agent_options) Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/managers/consumer/agent.py", line 98, in bind Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: agent.consumer.bind(context, agent_bindings, options) Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/agent/direct/pulpagent.py", line 140, in bind Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: consumer.bind(bindings, options) Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/gofer/rmi/stub.py", line 72, in __call__ Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: return self.stub._send(request, opts) Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/gofer/rmi/stub.py", line 142, in _send Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: return self.__send(request, options) Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/gofer/rmi/stub.py", line 173, in __send Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: any=opts.any) Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/gofer/rmi/policy.py", line 354, in send Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: trigger() Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/gofer/rmi/policy.py", line 468, in __call__ Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: self.__send() Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/gofer/rmi/policy.py", line 450, in __send Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: queue.declare(policy.url) Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/gofer/transport/qpid/model.py", line 156, in declare Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: sender = session.sender(self.address()) Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "<string>", line 6, in sender Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 606, in sender Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: sender._ewait(lambda: sender.linked) Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 814, in _ewait Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: result = self.session._ewait(lambda: self.error or predicate(), timeout) Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 580, in _ewait Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: result = self.connection._ewait(lambda: self.error or predicate(), timeout) Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 219, in _ewait Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: self.check_error() Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 212, in check_error Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: raise e Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: ConnectionError: Enqueue capacity threshold exceeded on queue "pulp.agent.log08.example.com". (/builddir/build/BUILD/qpid-0.26/cpp/src/qpid/legacystore/JournalImpl.cpp:594)(501) ________________________________ Information in this e-mail may be confidential. It is intended only for the addressee(s) identified above. If you are not the addressee(s), or an employee or agent of the addressee(s), please note that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify the sender of the error. ________________________________ Information in this e-mail may be confidential. It is intended only for the addressee(s) identified above. If you are not the addressee(s), or an employee or agent of the addressee(s), please note that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify the sender of the error.
_______________________________________________ Pulp-list mailing list [email protected] https://www.redhat.com/mailman/listinfo/pulp-list
