nice.
On Mar 8, 2011, at 1:14 PM, Eric Day wrote:
Hi Curtis,
On Tue, Mar 08, 2011 at 04:31:59PM +, Curtis Carter wrote:
Couple of questions.
Where did you come up with 300 bytes per message for ets?
Raw message = key(16) + ttl(8) + hide(8) + body(100) = 132 bytes
I inserted the
Hi everyone,
I added a sqlite backend to the prototype and ran some tests. Initially
things were very slow, but after some further testing I was able
to figure out where the time was being spent. In order to do this I
added a very simple binary protocol interface to insert only. These
tests are
...@oddments.org]
Sent: Tuesday, March 08, 2011 5:58 PM
To: Todd Willey
Cc: openstack@lists.launchpad.net
Subject: Re: [Openstack] Queue Service Implementation Thoughts
On Tue, Mar 08, 2011 at 04:47:38PM -0500, Todd Willey wrote:
With this switch to python, does it make sense to revisit the concept
I ran the tests again to verify:
500k requests - 10 processes each running 50k requests.
time req/s cs us sy id
2 thread/proc
echo c++ 7.19 69541 142182 23 77 0
echo erlang 9.53 52465 105871 39 61 0
echo python 9.58 52192 108420 42 58 0
2 thread/proc
wsgi
Eric, Thanks for your experimentation and analysis. Somewhat illustrates the point about premature optimization. First cut, have to agree with you and conclude that python implementation is effective, overall. As you said,if we find performance bottlenecks, especially as the payload size increases
: openstack@lists.launchpad.net
Subject: Re: [Openstack] Queue Service, next steps
Eric,
Thank you.
You raise lots of interest points. In no particular order:-
AMQP Observations
Your comments about AMQP seem to mostly be appropriate for one of the older
versions, eg 0-8, and I don't think
Hi Raphael,
On Mon, Feb 28, 2011 at 10:01:55AM +, Raphael Cohn wrote:
AMQP Observations
Your comments about AMQP seem to mostly be appropriate for one of the
older versions, eg 0-8, and I don't think they particularly apply to later
versions, eg 1-0. AMQP 0-8 did have some
Eric Team,
OpenStack's QueueService seems very interesting. As we have an existing
message queue implementation, we'd be happy to help you guys out. We're
about making messaging cloud-scale, so that everyone benefits.
However, it worries us that you're planning to implement a REST API for
Hi Raphael,
On Sun, Feb 27, 2011 at 11:18:35AM +, Raphael Cohn wrote:
OpenStack's QueueService seems very interesting. As we have an existing
message queue implementation, we'd be happy to help you guys out. We're
about making messaging cloud-scale, so that everyone benefits.
On Tue, Feb 22, 2011 at 12:55:32AM -0800, Vishvananda Ishaya wrote:
Awesome stuff!
I had to do the following patch to get it to run (on python 2.6.1 on my mac)
Yeah, the version in my junk repo was broken for a couple minutes
last night, you must have pulled at just the right moment. :)
I
; openstack@lists.launchpad.net
Subject: Re: [Openstack] Queue Service, next steps
Please bear in mind the long term maintainability of the openstack package.
One of the attractive features at the moment is that there are not
significant pre-reqs to set up the environment and most mass market
On Feb 22, 2011, at 8:58 AM, Eric Day wrote:
On Tue, Feb 22, 2011 at 12:55:32AM -0800, Vishvananda Ishaya wrote:
Awesome stuff!
I had to do the following patch to get it to run (on python 2.6.1 on my mac)
Yeah, the version in my junk repo was broken for a couple minutes
last night, you
Quick comment on the languages from a dev community perspective:
The Nova/Swift community is Python-based, so you won't directly reuse
it, whether you use C++ or Erlang. For new contributors, you either
target very experienced C++ developers (to get concurrency/scalability
right), or
I am very concerned about seeing a lot of I'd love to learn Erlang on
something we probably don't want to screw up. At the same time, I know there's
a number of us very experienced with C/C++ (even with respect to
concurrency/scalability).
- Chris
On Feb 21, 2011, at 2:40 AM, Thierry Carrez
Please bear in mind the long term maintainability of the openstack package.
One of the attractive features at the moment is that there are not significant
pre-reqs to set up the environment and most mass market environments can
support it.
Using C++ would not significantly change this
When you want to modify messages in a queue and also return those
that were modified. This is the atomic get/set needed by workers.
POST /account/queue?hide=60 HTTP/1.1
This will return all messages currently in /account/queue, but also
mark them as hidden for 60 seconds. This ensures only one
Hi Todd,
That's the multicast example, for a normal 1-1 queue, look at the
first example:
Worker: POST /account/queue?wait=60hide=60detail=all
(long-polling worker, request blocks until a message is ready)
Client: PUT /account/queue
(message inserted, returns unique id that was created)
Hi Mike,
On Sat, Feb 19, 2011 at 07:42:12PM -0600, Michael Barton wrote:
On Fri, Feb 18, 2011 at 9:38 PM, Eric Day e...@oddments.org wrote:
Hi Mike,
You make a good point, I apologize for not documenting some of the
ideas sooner. The architecture I had in mind borrows from other
queue
I have been meaning to reply with the exact same thing.
On Feb 18, 2011, at 10:12 AM, Jay Pipes jaypi...@gmail.com wrote:
Personally, I'd prefer C++ since that's what I'm used to, but I'd be
open to learning Erlang, too. Been wanting to learn it for a while
now.
-jay
On Fri, Feb 18,
would be easier.Cheersk/
Original Message
Subject: Re: [Openstack] Queue Service, next steps
From: Daniel Salinas imsplit...@gmail.com
Date: Fri, February 18, 2011 8:20 am
To: "openstack@lists.launchpad.net" openstack@lists.launchpad.net
I would totally be
Personally, I've been interested in learning Erlang as well. Eric, could you
explain more about why you anticipate that the queuing service will CPU-bound?
In either case, I've heard that Erlang can actually make better use of
multi-procs than C++, but I don't have any experience to speak from.
On 2/17/11 4:21 PM, Eric Day wrote:
Thanks to everyone who gave great feedback on the first queue service
thread. I've updated the wiki page to include the suggestions.
http://wiki.openstack.org/QueueService
Pondering this, I have a few thoughts:
I'll note that Erlang seems a realistic
On Fri, Feb 18, 2011 at 11:20:23AM -0600, Monsyne Dragon wrote:
Secondly, could we leverage existing opensource codebases to
accomplish this? I know AMQP was having issues over WAN links, and
likely isn't suited for disconnected operations (I've seen one
multitenant hosted queueing service
@lists.launchpad.net
Subject: Re: [Openstack] Queue Service, next steps
I checked out the CDMI spec, and while it covers the basics, it doesn't
seem to include features for visibility, concurrency control (multiple
workers grabbing messages via atomic operations), or long-polling. We
can still support
On Fri, Feb 18, 2011 at 5:22 PM, Eric Day e...@oddments.org wrote:
The main question right now is where to land on the spectrum of service
efficiency vs ease of development (C/C++ on one end and Python on
the other). It seems we're landing in the middle with Erlang. :)
Maybe I'm describing a
On Fri, Feb 18, 2011 at 6:23 PM, Eric Day e...@oddments.org wrote:
Perhaps I've been assuming some things, but I thought everyone
understood that is what we are looking to build (fault-tolerant,
horizontally scalable, ...). We're certainly not looking to build
a clustered queue (like RabbitMQ)
Hi Muriel,
On Tue, Feb 15, 2011 at 10:04:20AM +0100, Muriel wrote:
i'm completely new to openstack but i started working on something
similar to eucalyptus. In the coming months we will begin a project
to bring our work on openstack and share it with you.
Cool, looking forward to it!
The
On Mon, 2011-02-14 at 09:51 -0800, Eric Day wrote:
There are possible solutions to build an AMQP based service,
but AMQP brings complexity and a protocol not optimized for high
latency and intermittent connectivity.
Regardless of whether the system uses AMQP, I am a fan of AMQP's
flexible
Eric,
Ordered delivery of messages is a feature that I recognize is central to the
design of a queue service. A queue must be durable (persistent) in order to
provide reliable ordering capability. Having this feature simplifies
implementation of applications that consume the queue. I agree
Pete,
Queues are essential for highly concurrent use cases, and batch processing.
It's the core of what you need to build a cloud-specific application. Reference:
http://www.eecs.harvard.edu/~mdw/proj/seda/
Adrian
On Feb 14, 2011, at 10:20 AM, Pete Zaitcev wrote:
On Mon, 14 Feb 2011
Hi Pete,
Two initial use cases are a comparable service to what other public
cloud providers already have (similar HTTP/REST API, but eventually
with more features) and a flexible, high-performance queue/event system
for backend log and event processing for other OpenStack services (see
the
Eric,
Just looking at the http operations. Shouldn't the calls be around the
account then the queue?
GET /$account_id/queue/id to list all the queues
GET /$account_id/queue/id/message/id
Am I off here? Thoughts?
pvo
On 2/14/11 5:07 PM, Eric Day e...@oddments.org wrote:
I've summarized
Yeah, for anonymous access that would be the case. Those are not needed
when the request comes in with OpenStack Auth headers (like nova).
Do you think we should be repeating the account id in the URI when
the auth headers are present?
-Eric
On Mon, Feb 14, 2011 at 11:44:58PM +, Paul Voccio
On Tue, Feb 15, 2011 at 12:49:01AM +, Paul Voccio wrote:
Dropping the account_id, would this also assume that there can be more
than one queue per account?
Yeah. Think of the queue name as an extra namespace layer much like
a swift container, except you don't create or delete them, they
Looking at the swift docs, they reference a container like so:
METHOD /v1/account/container HTTP/1.1
http://docs.rackspacecloud.com/files/api/v1/cf-devguide-20110112.pdf
For the openstack api, it also includes the account id in the request:
POST /v1.1/214412/images HTTP/1.1
Host:
On Mon, Feb 14, 2011 at 7:57 PM, Paul Voccio paul.voc...@rackspace.com wrote:
Looking at the swift docs, they reference a container like so:
METHOD /v1/account/container HTTP/1.1
Yeah, this has worked out well for us. Delegated access,
authentication methods that don't provide the account
36 matches
Mail list logo
