Thanks for the responses Flavio, Roland.
Some background on why I'm asking: we're using Kafka as the message queue for
a stream processing service we're building, which we're delivering to our
internal customers as a service along with OpenStack. We're considering
building a high throughput ingest API to get the clients' data streams into the
stream processing service. It occurs to me that this API is simply a messaging
API, and so I'm wondering if we should consider building this high throughput
API as part of the Marconi project.
Has this topic come up in the Marconi team's discussions, and would it fit into
the vision of the Marconi roadmap?
Thanks,
Keith Newstadt
keith_newst...@symantec.com
@knewstadt
Date: Sun, 1 Jun 2014 15:01:40 +
From: "Hochmuth, Roland M"
To: OpenStack List
Subject: Re: [openstack-dev] [Marconi] Kafka support and high
throughput
Message-ID:
Content-Type: text/plain; charset="us-ascii"
There are some folks in HP evaluating different messaging technologies for
Marconi, such as RabbitMQ and Kafka. I'll ping them and maybe they can
share
some information.
On a related note, the Monitoring as a Service solution we are working
on uses Kafka. This was just open-sourced at,
https://github.com/hpcloud-mon,
and will be moving over to StackForge starting next week. The architecture
is at,
https://github.com/hpcloud-mon/mon-arch.
I haven't really looked at Marconi. If you are interested in
throughput, low latency, durability, scale and fault-tolerance Kafka
seems like a great choice.
It has been also pointed out from various sources that possibly Kafka
could be another oslo.messaging transport. Are you looking into that as
that would be very interesting to me and something that is on my task
list that I haven't gotten to yet.
On 5/30/14, 7:03 AM, "Keith Newstadt" wrote:
>Has anyone given thought to using Kafka to back Marconi? And has there
>been discussion about adding high throughput APIs to Marconi.
>
>We're looking at providing Kafka as a messaging service for our
>customers, in a scenario where throughput is a priority. We've had good
>luck using both streaming HTTP interfaces and long poll interfaces to get
>high throughput for other web services we've built. Would this use case
>be appropriate in the context of the Marconi roadmap?
>
>Thanks,
>Keith Newstadt
>keith_newst...@symantec.com
>
Keith Newstadt
Cloud Services Architect
Cloud Platform Engineering
Symantec Corporation
www.symantec.com
Office: (781) 530-2299 Mobile: (617) 513-1321
Email: keith_newst...@symantec.com
Twitter: @knewstadt
This message (including any attachments) is intended only for the use of the
individual or entity to which it is addressed and may contain information that
is non-public, proprietary, privileged, confidential, and exempt from
disclosure under applicable law or may constitute as attorney work product. If
you are not the intended recipient, you are hereby notified that any use,
dissemination, distribution, or copying of this communication is strictly
prohibited. If you have received this communication in error, notify us
immediately by telephone and (i) destroy this message if a facsimile or (ii)
delete this message immediately if this is an electronic communication.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
