So this came up briefly at the tripleo sprint, and since I can't seem to find a /why/ document (https://wiki.openstack.org/wiki/Marconi/Incubation#Raised_Questions_.2B_Answers and https://wiki.openstack.org/wiki/Marconi#Design don't supply this) we decided at the TC meeting that I should raise it here.
Firstly, let me check my facts :) - Marconi is backed by a modular 'storage' layer which places some conceptual design constraints on the storage backends that are possible (e.g. I rather expect a 0mq implementation to be very tricky, at best (vs the RPC style front end https://wiki.openstack.org/wiki/Marconi/specs/zmq/api/v1 )), and has a hybrid control/data plane API implementation where one can call into it to make queues etc, and to consume them. The API for the queues is very odd from a queueing perspective - https://wiki.openstack.org/wiki/Marconi/specs/api/v1#Get_a_Specific_Message - you don't subscribe to the queue, you enumerate and ask for a single message. And the implementations in tree are mongodb (which is at best contentious, due to the AGPL and many folks reasonable concerns about it), and mysq. My desires around Marconi are: - to make sure the queue we have is suitable for use by OpenStack itself: we have a very strong culture around consolidating technology choices, and it would be extremely odd to have Marconi be something that isn't suitable to replace rabbitmq etc as the queue abstraction in the fullness of time. - to make sure that deployers with scale / performance needs can have that met by Marconi - to make my life easy as a deployer ;) So my questions are: - why isn't the API a queue friendly API (e.g. like https://github.com/twitter/kestrel - kestrel which uses the memcache API, puts put into the queue, gets get from the queue). The current API looks like pretty much the worst case scenario there - CRUD rather than submit/retrieve with blocking requests (e.g. longpoll vs poll). - wouldn't it be better to expose other existing implementations of HTTP message queues like nova does with hypervisors, rather than creating our own one? E.g. HTTPSQS, RestMQ, Kestrel, queues.io. - or even do what Trove does and expose the actual implementation directly? - whats the plan to fix the API? - is there a plan / desire to back onto actual queue services (e.g. AMQP, $anyof the http ones above, etc) - what is the current performance - how many usecs does it take to put a message, and get one back, in real world use? How many concurrent clients can a single Marconi API server with one backing server deliver today? As background, 'implement a message queue in a SQL DB' is such a horrid antipattern its been a standing joke in many organisations I've been in - and yet we're preparing to graduate *exactly that* which is frankly perplexing. -Rob -- Robert Collins <rbtcoll...@hp.com> Distinguished Technologist HP Converged Cloud _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
